Erik's Engineering

something alliterative

edit:

I got the reasons for things slowing down wrong. Things get slower, but not for the reasons I thought. José Valim explains what's actually going on. I can confirm that if I take the largest of my test apps and nuke all the helpers it will speed up to roughly the speed of the 200 scaffold version, though the response times are pretty inconsistent. If I clear out the routes file, startup times and response times go down to the same level as the tiniest 1 scaffold app.


This is mainly a follow-up to my previous post.   Several people asked about the performance of Ruby Enterprise Edition, and one person mentioned start-up times.

I thought I'd make use of the test applications I'd created and gather a little more data, this time looking at start-up times and including REE.

REE Dev performance

Here's a quick update of the graph from my previous post with REE added.   You can see that in this respect it performs almost identically to Ruby 1.9.2.  jRuby is scaling a lot better than the others, though MRI outperforms it for smaller applications.

 

rails-scaling-ree-1000

 

Startup Times

Startup times are a bit different.  I timed these using a quick test script:

#!/bin/bash
echo puts Time.now > test_script
time rails runner test_script
time rails runner test_script
time rails runner test_script

 

You can see that this basically grabs the time taken to start up the rails environment, do a tiny bit of work, and shut down.  I'm assuming that shutdown is near-instantaneous so that the bulk of what we're observing is start up time.

Here's what I came up with:

rails-startup-performance-graph

It looks like jRuby isn't really in the running.  For a bare-minimum rails app, jRuby needs 5.5 seconds, while ruby 1.9.2 needs only 1 second.  jRuby then scales better than Ruby 1.8.7, but you have to get really big before jRuby starts actually being faster.  Even then it's not as fast as Ruby 1.9.2.  REE starts out mirroring Ruby 1.8.7, but it looks like towards the end it's doing noticeably worse.  That might just be noise in the data, though.  We'd probably have to go up to 1500 or 2000 models to say for sure.

All in all, it looks like Ruby 1.9.2 is the king of startup times.   It seems that start up time scales differently than dev mode response time, and that a workflow that avoids having to re-start the ruby interpreter is a big win.

A Note About Scale

I'm looking at how things change when you go from a tiny app to a really huge one. The smallest app I'm talking about takes up about 600k on disk, and the largest is about 55M. 55M is absolutely huge, especially since it doesn't have any images or other assets bulking it up.

# scaffolds size (k)
1 596
50 3460
100 6268
200 11880
300 17504
500 28716
750 42732
1000 56756

No matter how you slice it, 56M is a lot of code. Hopefully this will make it a little easier to compare my numbers which are based on synthetic test applications to real-world applications.

Published on 19/02/2011 at 18h15 .

5 comments

edit:

I got the reasons for things slowing down wrong.  Things get slower, but not for the reasons I thought.  José Valim explains what's actually going on.  I can confirm that if I take the largest of my test apps and nuke all the helpers it will speed up to roughly the speed of the 200 scaffold version, though the response times are pretty inconsistent.   If I clear out the routes file,  startup times and response times go down to the same level as the tiniest 1 scaffold app.

tl;dr

The faster your Rails app runs in dev mode, the better. As your app gets bigger, it will get slower. Jruby doesn't slow down as much. For larger codebases it blows the doors off of other implementations.

The Long Version

I'd like to talk about performance. Development performance.

This is something rather dear to my heart. Optimizing development performance can greatly improve development productivity. Getting new features faster is one of the reasons we like Rails. The faster a developer can work, the more features come out of the sausage factory.

When I was in college, I did a senior project where we programmed a microprocessor by burning our code onto an EPROM. Erasing the EPROM meant putting it under a UV light for 15-20 minutes. Then you'd burn your code to it, plug it into your board and see if that new version of your keypad debouncing code worked. My teammates were in awe of my foresight in ordering 2 EPROMs, so we only had to wait 10 minutes on average between test runs.

10 minutes to test every code change. That sucked. I should have ordered a dozen, even if they did cost $4 apiece.

I spent > 10 years programming in perl.  Under mod_perl I'd make a code change, then restart my webserver and test to see if it worked.  Elapsed time somewhere ~ 5-10 seconds. The same kind of test/fix/verify cycle could be done in 10 seconds. Much better.

On Rails, we can program in the development environment where it will automatically recompile most of our application on every page view.  For small apps this might as well be instantaneous.  This is AWESOME. You can do things almost as fast as you can think and type.

Until It's Not

Well, it's awesome as long as it stays fast.  In order to make sure all your changes show up in development mode, Rails recompiles all the controllers and models on each page view.  This means that the more code in your app, the longer it will take to compile. Eventually you end up clicking and waiting 5 or 10 seconds for a response. This is even worse than back in the perl world, because at least then you were switching to a console and running 'sudo apachectl restart' or some such, so you had something to do during that time. Bored programmers start checking Hacker News and productivity suffers.

At work we've got several Rails apps but we're still in the process of transitioning from monolithic to service oriented architecture. The original app is huge.  Hundreds of models and controllers. Megabytes of code.  That's a lot of code to compile on each request and we end up with a painfully slow minimum response time.

We run on jruby. For most of us this is the first jruby app we've worked on. Naturally, we all blamed jruby and grumbled a bit.  Charles Nutter approached me about it and we ended up hypothesizing that it was purely the big recompile adding so much to the time.  I figured I should test that hypothesis.

Testing methodology

I made a Rails 3.0.3 app and used 'rails g scaffold' to add more and more controllers and models to it. I used 'rails s' to start a webrick-based development server (it's the default) and 'time wget http://localhost:3000/' to test response time (grab the "real" value). You can't trust the reported response time on the server console because that doesn't include recompile time.  No matter how slow the response time, that pretty much always showed 20-30ms.

RVM was a lifesaver. RVM plus some helper scripts to swap Gemfiles let me test 5 different Ruby versions.

I did 5 test runs with each ruby version at each of 8 different application sizes.  A typical dataset looks something like this:

ruby-1.8.7-p302
scaffolds 1 2 3 4 5 avg
1 1.693 0.101 0.078 0.071 0.085 0.08375
50 2.347 0.156 0.157 0.147 0.157 0.15425
100 2.487 0.279 0.269 0.225 0.225 0.2495
200 3.339 0.509 0.466 0.445 0.412 0.458
300 3.682 0.667 0.652 0.672 0.611 0.6505
500 5.130 1.232 1.199 1.188 1.130 1.18725
750 7.536 2.246 2.228 2.138 2.166 2.1945
1000 8.949 3.118 2.965 3.060 3.023 3.0415

The sharp eyed among you have probably realized that the averages listed aren't for all 5 test runs. The first one was always much slower than the others and I'm mainly interested in the subsequent runs, so I'm only averaging the 4 remaining runs.

I did this for Ruby 1.8.7, Ruby 1.9.2, Jruby 1.5.6, Jruby 1.6.0 and Rubinius (rbx) 1.2.0. That's about 200 readings after I got everything all set up and automated.

Now, it should probably be noted that these times are optimistic. The code generated by Rails' scaffold generator is pretty simple. Not a lot of complex control structures and the inheritance hierarchies are very straightforward. Real life code is almost certainly harder to compile.

The Big Picture

Rails Dev scaling graph, 0 - 1000 scaffolds

Well look at that. We've got response time in seconds on the Y axis, with number of scaffolds in the app on the X axis.

First of all, Rubinius is just a lot slower than the others all around. Further, none of them scale linearly. Ruby 1.9.2 is much faster than ruby 1.8.7, but both versions of jruby are even faster than that. There's a nearly 12 second difference in response time between the fastest and slowest implementations and at 1000 scaffolds jruby is over twice as fast as the next fastest implementation.

They all curve upward a bit, meaning that as you add more code it has a greater effect on response times. Jruby is MUCH closer to linear, though.

But there's something else there.

The Smaller Picture

Rails Dev scaling graph, 0-200 scaffolds

This looks a little different. Ruby 1.9.2 wins until around 160 scaffolds. I'm not sure how important this is. Both MRI and Jruby are running at 300ms or less at that point, so I doubt the differences really make a difference to a developer. It IS interesting to note that jruby seems to have a little more fixed overhead but makes up for it by scaling better as you add more code.

Conclusion

For smaller apps, this probably doesn't make any difference whatsoever. For bigger apps, you can help maximize developer effectiveness by picking a ruby that will help them work faster. Jruby seems pretty good, with ruby 1.9.2 coming in second. Stay away from ruby 1.8.7 or rubinius if you're working with larger codebases.

Now, what I'd really like is a way to avoid recompiling everything every time. If I could have Rails recompile just the model or controller I'm working on and skip all the others, that'd be grand. I've taken a couple stabs at it, but I haven't succeeded yet.

Breaking larger apps down into a bunch of smaller apps that use a service oriented architecture will effectively give you that. Each one has a smaller codebase so the recompile time isn't as big of an issue, especially if you set cache_classes = true for all the apps you're not actively working on.

Published on 03/02/2011 at 04h14 under , .

7 comments

How do you keep up with developments in programming?  What about keeping yourself engaged as a developer?

I read books.  I read a lot of blogs.  In fact, my Google Reader is so packed with blogs that I can't always keep up.  I end up going through the list of articles much the same way you might scan over the list of articles on cnn.com or a local newspaper.

Luckily, I've got another technique up my sleeve.  I've got a commute.  I drive for about an hour total each day, to and from work.  I can't read blogs then, but I can listen to podcasts.  Podcasts are a bit weird.  You can't skim them very well - you have to actually listen to them front to back.  That's also the downside to podcasts - you can't skim them well, so you need be a bit more selective about them.  Here are my choices for programming related podcasts (other podcasts will come in a future post).

I listen to two kinds of programming podcasts.  The first are aimed at current events - either new software releases or community happenings.  These have a lot of overlap with the blogs I read, but the enforced pacing of a podcast means that sometimes I'll end up learning important things about something I'd skimmed over in the RSS reader.  The other kind are more about the programming lifestyle, and I haven't found a good counterpart for these in the blog world.

 

Current Events Podcasts

These supplement my selection of RSS feeds.  They push me to learn a bit more about things that I skipped over in the RSS reader.

The Changelog http://thechangelog.com/

Wynn Netherland and Adam Stacoviak covering "what's fresh and new in Open Source".  Each week features an interview with someone from the open source community as well as a rundown of notable releases.  They're big on node.js and javascript, but I'll forgive them for that.  They've almost convinced me to try node.  I do wish they'd learn to ask questions during interviews rather than instructing people to tell us a little about X, but that's just a pet peeve of mine.

Ruby 5 http://ruby5.envylabs.com/

A rotating cast of developers give a twice-weekly roundup of new releases and cool stuff in the Ruby world.  5 minutes long and moves really quickly.  This is a pretty good way to hear about new releases with a little bit of flavor to help you know when something's interesting to you in particular.  They pretty obviously write a script beforehand, so jokes and interactions between the presenters can seem a bit stilted sometimes.

The Ruby Show http://rubyshow.com/

Silly.  Juvenile.  Entertaining.  Peter Cooper and Jason Seifer cover recent happenings in Ruby land.  The subject matter has a lot of overlaps with The Changelog and Ruby 5, but somehow all three of these will pick up subjects that the others miss.  The Ruby Show tends to focus a bit more on community and interesting learning opportunities than the others do.  Listen for any time Peter has to say a word with an initial "thr" sound.

 

Programming Lifestyle Podcasts

These are more meditative - talking about being a programmer in general.

Coderpath http://coderpath.com

Miles Forrest and Curtis McHale interview other developers and talk about getting better at programming.  Coderpath doesn't update as often as I'd like, but when it does it's great.

This Developer's Life http://thisdeveloperslife.com/

Rob Conery and Scott Hanselman do a podcast about different aspects of being a developer.  The focus here isn't on code, but on things like work/life balance, horrible projects and getting fired.  Stories are generally told via anecdotes from other developers that they've interviewed.  Listen to this to hear some entertaining stories that help you understand that you're not alone.

 

All in all, I have enough podcasts to keep my commutes covered pretty well.  I feel like they play an important part in keeping me informed and inspired.

Published on 17/01/2011 at 02h27 .

0 comments

Powered by Typo – Thème Frédéric de Villamil | Photo Glenn