Erik's Engineering

something alliterative

Ruby Performance in the Rails Development Environment

edit:

I got the reasons for things slowing down wrong.  Things get slower, but not for the reasons I thought.  José Valim explains what's actually going on.  I can confirm that if I take the largest of my test apps and nuke all the helpers it will speed up to roughly the speed of the 200 scaffold version, though the response times are pretty inconsistent.   If I clear out the routes file,  startup times and response times go down to the same level as the tiniest 1 scaffold app.

tl;dr

The faster your Rails app runs in dev mode, the better. As your app gets bigger, it will get slower. Jruby doesn't slow down as much. For larger codebases it blows the doors off of other implementations.

The Long Version

I'd like to talk about performance. Development performance.

This is something rather dear to my heart. Optimizing development performance can greatly improve development productivity. Getting new features faster is one of the reasons we like Rails. The faster a developer can work, the more features come out of the sausage factory.

When I was in college, I did a senior project where we programmed a microprocessor by burning our code onto an EPROM. Erasing the EPROM meant putting it under a UV light for 15-20 minutes. Then you'd burn your code to it, plug it into your board and see if that new version of your keypad debouncing code worked. My teammates were in awe of my foresight in ordering 2 EPROMs, so we only had to wait 10 minutes on average between test runs.

10 minutes to test every code change. That sucked. I should have ordered a dozen, even if they did cost $4 apiece.

I spent > 10 years programming in perl.  Under mod_perl I'd make a code change, then restart my webserver and test to see if it worked.  Elapsed time somewhere ~ 5-10 seconds. The same kind of test/fix/verify cycle could be done in 10 seconds. Much better.

On Rails, we can program in the development environment where it will automatically recompile most of our application on every page view.  For small apps this might as well be instantaneous.  This is AWESOME. You can do things almost as fast as you can think and type.

Until It's Not

Well, it's awesome as long as it stays fast.  In order to make sure all your changes show up in development mode, Rails recompiles all the controllers and models on each page view.  This means that the more code in your app, the longer it will take to compile. Eventually you end up clicking and waiting 5 or 10 seconds for a response. This is even worse than back in the perl world, because at least then you were switching to a console and running 'sudo apachectl restart' or some such, so you had something to do during that time. Bored programmers start checking Hacker News and productivity suffers.

At work we've got several Rails apps but we're still in the process of transitioning from monolithic to service oriented architecture. The original app is huge.  Hundreds of models and controllers. Megabytes of code.  That's a lot of code to compile on each request and we end up with a painfully slow minimum response time.

We run on jruby. For most of us this is the first jruby app we've worked on. Naturally, we all blamed jruby and grumbled a bit.  Charles Nutter approached me about it and we ended up hypothesizing that it was purely the big recompile adding so much to the time.  I figured I should test that hypothesis.

Testing methodology

I made a Rails 3.0.3 app and used 'rails g scaffold' to add more and more controllers and models to it. I used 'rails s' to start a webrick-based development server (it's the default) and 'time wget http://localhost:3000/' to test response time (grab the "real" value). You can't trust the reported response time on the server console because that doesn't include recompile time.  No matter how slow the response time, that pretty much always showed 20-30ms.

RVM was a lifesaver. RVM plus some helper scripts to swap Gemfiles let me test 5 different Ruby versions.

I did 5 test runs with each ruby version at each of 8 different application sizes.  A typical dataset looks something like this:

ruby-1.8.7-p302
scaffolds 1 2 3 4 5 avg
1 1.693 0.101 0.078 0.071 0.085 0.08375
50 2.347 0.156 0.157 0.147 0.157 0.15425
100 2.487 0.279 0.269 0.225 0.225 0.2495
200 3.339 0.509 0.466 0.445 0.412 0.458
300 3.682 0.667 0.652 0.672 0.611 0.6505
500 5.130 1.232 1.199 1.188 1.130 1.18725
750 7.536 2.246 2.228 2.138 2.166 2.1945
1000 8.949 3.118 2.965 3.060 3.023 3.0415

The sharp eyed among you have probably realized that the averages listed aren't for all 5 test runs. The first one was always much slower than the others and I'm mainly interested in the subsequent runs, so I'm only averaging the 4 remaining runs.

I did this for Ruby 1.8.7, Ruby 1.9.2, Jruby 1.5.6, Jruby 1.6.0 and Rubinius (rbx) 1.2.0. That's about 200 readings after I got everything all set up and automated.

Now, it should probably be noted that these times are optimistic. The code generated by Rails' scaffold generator is pretty simple. Not a lot of complex control structures and the inheritance hierarchies are very straightforward. Real life code is almost certainly harder to compile.

The Big Picture

Rails Dev scaling graph, 0 - 1000 scaffolds

Well look at that. We've got response time in seconds on the Y axis, with number of scaffolds in the app on the X axis.

First of all, Rubinius is just a lot slower than the others all around. Further, none of them scale linearly. Ruby 1.9.2 is much faster than ruby 1.8.7, but both versions of jruby are even faster than that. There's a nearly 12 second difference in response time between the fastest and slowest implementations and at 1000 scaffolds jruby is over twice as fast as the next fastest implementation.

They all curve upward a bit, meaning that as you add more code it has a greater effect on response times. Jruby is MUCH closer to linear, though.

But there's something else there.

The Smaller Picture

Rails Dev scaling graph, 0-200 scaffolds

This looks a little different. Ruby 1.9.2 wins until around 160 scaffolds. I'm not sure how important this is. Both MRI and Jruby are running at 300ms or less at that point, so I doubt the differences really make a difference to a developer. It IS interesting to note that jruby seems to have a little more fixed overhead but makes up for it by scaling better as you add more code.

Conclusion

For smaller apps, this probably doesn't make any difference whatsoever. For bigger apps, you can help maximize developer effectiveness by picking a ruby that will help them work faster. Jruby seems pretty good, with ruby 1.9.2 coming in second. Stay away from ruby 1.8.7 or rubinius if you're working with larger codebases.

Now, what I'd really like is a way to avoid recompiling everything every time. If I could have Rails recompile just the model or controller I'm working on and skip all the others, that'd be grand. I've taken a couple stabs at it, but I haven't succeeded yet.

Breaking larger apps down into a bunch of smaller apps that use a service oriented architecture will effectively give you that. Each one has a smaller codebase so the recompile time isn't as big of an issue, especially if you set cache_classes = true for all the apps you're not actively working on.

Published on 03/02/2011 at 04h14 under , .

  • By Alan Brown 04/02/2011 at 07h01

    I suspect the setup on jruby is jit-related.

    Wouldn’t it be cool if you could capture the java code involved for the most CPU intensive stuff and just use that code until the ruby code its based on changes? Or is that already happening to some degree?


  • By Erik 04/02/2011 at 17h40

    I think jruby does what you describe normally, but it can’t do it in dev mode. The problem is that Rails flags it all as needing a recompile, whether it changed or not. That’s part of Rails, so the different ruby interpreters don’t have much choice.

    I’d love to work out a way to have a development mode that only applied to certain controllers and models.


  • By Matt Jones 04/02/2011 at 18h39

    @Erik:

    I believe development mode used to work like that, but it was abandoned as it lead to really obscure bugs under certain circumstances.


  • By thedarkone 04/02/2011 at 20h14

    You should definitely check out rails-dev-boost plugin.

    It patches Rails to do selective constant reloading in development mode. There are a few caveats that you need to be aware of (ie: don’t use class-level variables referencing other “unloadable” classes, etc.), but the payback is worth it: you get production level responsiveness in development mode.


  • By Joe Van Dyk 05/02/2011 at 06h25

    Hm, on my larger Rails 2.3 application, JRuby 1.5 was 50-100% slower than REE 1.8.7 + Passenger, both with class caching turned off and on.

    What app server did you use for jruby?


  • By Erik 05/02/2011 at 15h44

    Joe: I used webrick. I was going for things as “stock” as humanly possible. Literally just ‘rails s’.

    At work we use jetty-rails. I haven’t benchmarked them against each other. For this round I was mainly wanting to prove to myself that

    1. bigger rails apps slow down in dev mode (i.e. it’s not just us)
    2. it happens on all the rubies (i.e. jruby isn’t the problem)

    I just did a quick test with ree, and it clocked in at 7.5s for 1000 scaffolds, which is faster than ruby 1.8.7, but slower than 1.9.2.

    @krobertson brought up startup time, which looks like jruby is slower than 1.9.2, but faster than ree. Maybe worth a second post to add ree to the graphs and document startup times.


  • By Rich Morin 05/02/2011 at 16h48

    While you’re playing with this, you might want to look into the kind of “predictive compilation” hackery that Xcode employs.

    http://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/XcodeBuildSystem/800-ReducingBuildTimes/bsspeedup_build.html


Comment Ruby Performance in the Rails Development Environment

Powered by Typo – Thème Frédéric de Villamil | Photo Glenn