Is Ruby Fast Yet?

 

Practical, comparative benchmarks of Ruby implementations. Thanks to RVM for Ruby setup and compiling. Github

Rails Requests Per Second Steady State

More is better. Single-threaded; warm JIT (warmup requires around 50,000 requests for JRuby and RBX).

1 week

 

2week


Rails Requests Per Second

More is better. Single-threaded; cold start (3 seconds of warmup, 15 seconds of testing).


Rails Startup Time

Less is better. Note that JRuby does not use Drip, Nailgun, or Theine for this benchmark.


Rails Memory Usage

Less is better if you have reason to care about memory usage.

So, Is Ruby Fast Yet?

I like to think that “fast” is, somewhat arbitrarily, either five times or ten times faster than Ruby 1.8.7. Both 5x and 10x better than 1.8.7 are significant milestones, but neither has been reached. So, no. Ruby is not fast yet.

Ruby is, however, getting faster with each release. These benchmarks give a good indication of how much faster.

About the Benchmarks

This is intended as a Ruby benchmark rather than a Rails one; Rails just happens to be the best thing out there to benchmark. Rails present the greatest challenge to Ruby implementations.

As a Ruby benchmark, the Rails version is locked to 3.1.3 and the server is WEBrick.

I would love to compare Ruby to other dynamic languages. However, I don’t know of any sufficiently complex, real-world applications that are also cross-language. If you discover any, please let me know.

Hardware

Mid-2007 MacBook Pro
2.2GHz Core 2 Duo
6GB 667MHz DDR2 SDRAM
WD Scorpio Black 7200RPM 320GB Hard Drive
Ubuntu Server 14.04
Fans preemptively sped up with macfanctld.

Updates

  1. 2014-12-26 OS Switched to Ubuntu Server 14.04 from Mac OS X 10.6. JRuby is now run in OpenJDK 7. (The upgrade was required because newer rubies no longer compiled on Mac OS X 10.6, and upgrading to Yosemite introduced a hard-to-diagnose performance regression that may have been in the network stack. Ubuntu shows some weird regressions in the newer MRIs, but at least runs the reference Ruby (MRI 1.8.7) at the same speed as before. Additionally, many people use Ubuntu 14.04 in production.)
  2. 2014-09-21 Older head rubies are no longer benchmarked: 1.9.2-head, 1.9.3-head, and 2.0.0-head. Note that RVM 1.8.7 actually builds 1.8.7-head for security reasons. Ruby uses a more semantic versioning scheme as of 2.1.0, so 2.1.0 and 2.1.0-head have been replaced by 2.1 and 2.1-head.
  3. 2014-01-02 Java updated from 1.6.0_51-b11-457 to 1.7.0_45.
  4. 2013-05-24 Rails Requests Per Second Steady State benchmark added to show JIT performance.
  5. 2012-01-12 First results.