I got the reasons for things slowing down wrong. Things get slower, but not for the reasons I thought. José Valim explains what's actually going on. I can confirm that if I take the largest of my test apps and nuke all the helpers it will speed up to roughly the speed of the 200 scaffold version, though the response times are pretty inconsistent. If I clear out the routes file, startup times and response times go down to the same level as the tiniest 1 scaffold app.
The faster your Rails app runs in dev mode, the better. As your app gets bigger, it will get slower. Jruby doesn't slow down as much. For larger codebases it blows the doors off of other implementations.
The Long Version
I'd like to talk about performance. Development performance.
This is something rather dear to my heart. Optimizing development performance can greatly improve development productivity. Getting new features faster is one of the reasons we like Rails. The faster a developer can work, the more features come out of the sausage factory.
When I was in college, I did a senior project where we programmed a microprocessor by burning our code onto an EPROM. Erasing the EPROM meant putting it under a UV light for 15-20 minutes. Then you'd burn your code to it, plug it into your board and see if that new version of your keypad debouncing code worked. My teammates were in awe of my foresight in ordering 2 EPROMs, so we only had to wait 10 minutes on average between test runs.
10 minutes to test every code change. That sucked. I should have ordered a dozen, even if they did cost $4 apiece.
I spent > 10 years programming in perl. Under mod_perl I'd make a code change, then restart my webserver and test to see if it worked. Elapsed time somewhere ~ 5-10 seconds. The same kind of test/fix/verify cycle could be done in 10 seconds. Much better.
On Rails, we can program in the development environment where it will automatically recompile most of our application on every page view. For small apps this might as well be instantaneous. This is AWESOME. You can do things almost as fast as you can think and type.
Until It's Not
Well, it's awesome as long as it stays fast. In order to make sure all your changes show up in development mode, Rails recompiles all the controllers and models on each page view. This means that the more code in your app, the longer it will take to compile. Eventually you end up clicking and waiting 5 or 10 seconds for a response. This is even worse than back in the perl world, because at least then you were switching to a console and running 'sudo apachectl restart' or some such, so you had something to do during that time. Bored programmers start checking Hacker News and productivity suffers.
At work we've got several Rails apps but we're still in the process of transitioning from monolithic to service oriented architecture. The original app is huge. Hundreds of models and controllers. Megabytes of code. That's a lot of code to compile on each request and we end up with a painfully slow minimum response time.
We run on jruby. For most of us this is the first jruby app we've worked on. Naturally, we all blamed jruby and grumbled a bit. Charles Nutter approached me about it and we ended up hypothesizing that it was purely the big recompile adding so much to the time. I figured I should test that hypothesis.
I made a Rails 3.0.3 app and used 'rails g scaffold' to add more and more controllers and models to it. I used 'rails s' to start a webrick-based development server (it's the default) and 'time wget http://localhost:3000/' to test response time (grab the "real" value). You can't trust the reported response time on the server console because that doesn't include recompile time. No matter how slow the response time, that pretty much always showed 20-30ms.
RVM was a lifesaver. RVM plus some helper scripts to swap Gemfiles let me test 5 different Ruby versions.
I did 5 test runs with each ruby version at each of 8 different application sizes. A typical dataset looks something like this:
The sharp eyed among you have probably realized that the averages listed aren't for all 5 test runs. The first one was always much slower than the others and I'm mainly interested in the subsequent runs, so I'm only averaging the 4 remaining runs.
I did this for Ruby 1.8.7, Ruby 1.9.2, Jruby 1.5.6, Jruby 1.6.0 and Rubinius (rbx) 1.2.0. That's about 200 readings after I got everything all set up and automated.
Now, it should probably be noted that these times are optimistic. The code generated by Rails' scaffold generator is pretty simple. Not a lot of complex control structures and the inheritance hierarchies are very straightforward. Real life code is almost certainly harder to compile.
The Big Picture
Well look at that. We've got response time in seconds on the Y axis, with number of scaffolds in the app on the X axis.
First of all, Rubinius is just a lot slower than the others all around. Further, none of them scale linearly. Ruby 1.9.2 is much faster than ruby 1.8.7, but both versions of jruby are even faster than that. There's a nearly 12 second difference in response time between the fastest and slowest implementations and at 1000 scaffolds jruby is over twice as fast as the next fastest implementation.
They all curve upward a bit, meaning that as you add more code it has a greater effect on response times. Jruby is MUCH closer to linear, though.
But there's something else there.
The Smaller Picture
This looks a little different. Ruby 1.9.2 wins until around 160 scaffolds. I'm not sure how important this is. Both MRI and Jruby are running at 300ms or less at that point, so I doubt the differences really make a difference to a developer. It IS interesting to note that jruby seems to have a little more fixed overhead but makes up for it by scaling better as you add more code.
For smaller apps, this probably doesn't make any difference whatsoever. For bigger apps, you can help maximize developer effectiveness by picking a ruby that will help them work faster. Jruby seems pretty good, with ruby 1.9.2 coming in second. Stay away from ruby 1.8.7 or rubinius if you're working with larger codebases.
Now, what I'd really like is a way to avoid recompiling everything every time. If I could have Rails recompile just the model or controller I'm working on and skip all the others, that'd be grand. I've taken a couple stabs at it, but I haven't succeeded yet.
Breaking larger apps down into a bunch of smaller apps that use a service oriented architecture will effectively give you that. Each one has a smaller codebase so the recompile time isn't as big of an issue, especially if you set cache_classes = true for all the apps you're not actively working on.