JRuby/Glassfish Investigation

I’ve been keeping an eye on the JRuby scene for a while now and I recently went about benchmarking again. JRuby is interesting because it has the potential to bring robust Java deployment and tools to the Ruby world.

I made a simple widgets application as outlined here and installed JRuby/Glassfish as outlined here (both great articles at AD TechFL) and then proceeded to benchmark:

Glassfish/Jruby on httperf:

Elena:~/Applications/jruby/lib vishnu$ time httperf --client=0/1 --server=localhost --port=8080 --uri=/jruby-demo/widgets/list --send-buffer=4096 --recv-buffer=16384 --num-conns=480 --rate=60
httperf --client=0/1 --server=localhost --port=8080 --uri=/jruby-demo/widgets/list --rate=60 --send-buffer=4096 --recv-buffer=16384 --num-conns=480 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 2

Total: connections 480 requests 480 replies 480 test-duration 7.999 s

Connection rate: 60.0 conn/s (16.7 ms/conn, <=7 concurrent connections)
Connection time [ms]: min 1.4 avg 24.6 max 251.3 median 15.5 stddev 35.7
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 1.000

Request rate: 60.0 req/s (16.7 ms/req)
Request size [B]: 83.0

Reply rate [replies/s]: min 60.0 avg 60.0 max 60.0 stddev 0.0 (1 samples)
Reply time [ms]: response 24.5 transfer 0.0
Reply size [B]: header 360.0 content 1077.0 footer 0.0 (total 1437.0)
Reply status: 1xx=0 2xx=438 3xx=0 4xx=0 5xx=42

CPU time [s]: user 1.42 system 5.79 (user 17.7% system 72.4% total 90.1%)
Net I/O: 89.1 KB/s (0.7*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

real    0m8.008s
user    0m1.418s
sys     0m5.799s

Glassfish/Jruby on ab:

Elena:~/Applications/jruby/lib vishnu$ ab -n400 http://localhost:8080/jruby-demo/widgets/show/1
This is ApacheBench, Version 1.3d  apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Finished 400 requests
Server Software:        Sun
Server Hostname:        localhost
Server Port:            8080

Document Path:          /jruby-demo/widgets/show/1
Document Length:        611 bytes

Concurrency Level:      1
Time taken for tests:   8.551 seconds
Complete requests:      400
Failed requests:        0
Broken pipe errors:     0
Total transferred:      397600 bytes
HTML transferred:       244400 bytes
Requests per second:    46.78 [#/sec] (mean)
Time per request:       21.38 [ms] (mean)
Time per request:       21.38 [ms] (mean, across all concurrent requests)
Transfer rate:          46.50 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.0      0     0
Processing:    13    21   52.6     16  1019
Waiting:       13    21   52.6     16  1019
Total:         13    21   52.6     16  1019

Percentage of the requests served within a certain time (ms)
  50%     16
  66%     17
  75%     17
  80%     17
  90%     18
  95%     20
  98%     37
  99%    116
 100%   1019 (last request)

Let’s compare. Here’s how a single Mongrel/C does with httperf:

Elena:~/Applications/jruby/lib vishnu$ time httperf --client=0/1 --server=localhost --port=3000 --uri=/widgets/list --send-buffer=4096 --recv-buffer=16384 --num-conns=480 --rate=30
httperf --client=0/1 --server=localhost --port=3000 --uri=/widgets/list --rate=30 --send-buffer=4096 --recv-buffer=16384 --num-conns=480 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 3

Total: connections 480 requests 480 replies 480 test-duration 16.009 s

Connection rate: 30.0 conn/s (33.4 ms/conn, <=18 concurrent connections)
Connection time [ms]: min 9.0 avg 56.4 max 607.6 median 9.5 stddev 99.4
Connection time [ms]: connect 0.2
Connection length [replies/conn]: 1.000

Request rate: 30.0 req/s (33.4 ms/req)
Request size [B]: 72.0

Reply rate [replies/s]: min 29.4 avg 29.8 max 30.0 stddev 0.3 (3 samples)
Reply time [ms]: response 55.7 transfer 0.5
Reply size [B]: header 278.0 content 1011.0 footer 0.0 (total 1289.0)
Reply status: 1xx=0 2xx=480 3xx=0 4xx=0 5xx=0

CPU time [s]: user 3.91 system 11.35 (user 24.4% system 70.9% total 95.3%)
Net I/O: 39.9 KB/s (0.3*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

real    0m16.017s
user    0m3.912s
sys     0m11.354s

And Mongrel/C with ab:

Elena:~/Applications/jruby/lib vishnu$ ab -n400 http://localhost:3000/widgets/list
This is ApacheBench, Version 1.3d  apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Finished 400 requests
Server Software:        Mongrel
Server Hostname:        localhost
Server Port:            3000

Document Path:          /widgets/list
Document Length:        1011 bytes

Concurrency Level:      1
Time taken for tests:   7.974 seconds
Complete requests:      400
Failed requests:        0
Broken pipe errors:     0
Total transferred:      515600 bytes
HTML transferred:       404400 bytes
Requests per second:    50.16 [#/sec] (mean)
Time per request:       19.93 [ms] (mean)
Time per request:       19.93 [ms] (mean, across all concurrent requests)
Transfer rate:          64.66 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0     0    0.0      0     0
Processing:     9    19   63.4     11  1167
Waiting:        9    19   63.4     11  1167
Total:          9    19   63.4     11  1167

Percentage of the requests served within a certain time (ms)
  50%     11
  66%     12
  75%     12
  80%     12
  90%     13
  95%     15
  98%    113
  99%    149
 100%   1167 (last request)

Observations

  • Advantage: For the same number of connections, Glassfish/Jruby versus a single instance Mongrel/C has twice the reply rate in half the benchmark time.
  • Disadvantage: Request latency is more for Glassfish/Jruby. (around 5ms av. more)
  • Disadvantage: JRuby Ruby/Rails support is not perfect but very good.
  • Advantage: Much simpler to set up. Copy the war file to the server and deployment is done, autodeployment is also possible w/o restarting Glassfish.
  • Advantage: Much simpler to install and get going on the server, just requires a JDK, glassfish installed anywhere, etc.
  • Advantage: Runs on Java, so Java deployment experience will do.
  • Disadvantage: Does not support C library extensions for Ruby, question: how many of them are there & useful (memcached?) ?
  • Advantage: Supports Java libraries kind of easily, again question: how many of them are there, useful and relevant?
  • Question: This benchmark is versus a single mongrel, how does cluster vs. cluster compare?
  • Question: Is glassfish clustering easy? Observation: mongrel clustering is cumbersome at present. Maybe swiftiply will help there with dynamic addition of mongrels?

11 responses

  1. Work at home….

    Free work at home. Work at home business. Legitimate work from home jobs. Work at home….

  2. Report on JRuby On Glassfish@Silicon Valley JUG…

    I was at Silicon Valley JUG (Java User’s Group) to talk about Ruby on Rails in GlassFish. The slides used in this presentation can be found here.

    The main theme of my presentation was on highlighting Rails development and deployment on GlassFish v2 a…

  3. CG: It is cumbersome because dynamically adding mongrels is tougher than it should be.

    Matt: I do this often as a hobby and the test rig is always subversioned so that I can come back to it, will continue playing around with it.

  4. Hi Vishnu, I agree that generally speaking, things should be about the same. The reason I took the time to post though is that with ab I was unable to really push a server to it’s limits. Latency, in particular, was an area that ab didn’t measure very well at all. I was able to induce big changes in an a benchmark run with just slight changes in the flags.

    Do you still have your test rig? It’d be interesting to see what ab does with some different flags or with a simple faban or flood driver.

  5. Most of this article is very good but:
    >> mongrel clustering is cumbersome at present

  6. I gather that a Mongrel cluster on a single machine is closer to a single Glassfish instance, due to the way both approach concurrency, although a Glassfish cluster on one machine would be a reasonable comparison.

    I’m curious to see the results of that.

  7. […] my article on benchmarking Glassfish got picked up by the Aquarium. Cool # glassfish | me outside | […]

  8. Hi Matt, I’m aware of ab’s problems, but it was the simplest benchmark to run. Deficiencies should be the same across benchmarks and app-servers rt? All of the above is measured after reaching a ‘steady-state’ situation.

    But yeah, this benchmark is not that scientific, and was meant just as a quick comparison. I want to read up on this peepcode and perform a proper comparison sometime.

  9. I don’t know about httpperf, but ab is not a very good tool to measure latency of requests with. I’ve personally been able to shift benchmarks results pretty dramatically with slight changes to the -c flag to ab. In effect, you’re benchmarking ab’s ability to walk through the sockets with a single thread, counting the latency it takes ab’s single thread to walk through the fd’s.

    Scott Oaks writes about the problem here:
    http://weblogs.java.net/blog/sdo/archive/2007/03/ab_considered_h.html

    When I last worked with a customer who had shown a similar problem (it too was RonR), we shifted to Faban, which prides itself on being stastically correct.

    I nearly ran a similar benchmark. Perhaps I will, once I have the time. 🙂

    Good stuff though…

  10. That seems cool Albert, thanks!

  11. I would imagine that Ehcache might make a good replacement for memcached. You might have to write some JRuby/Java specific code though (or maybe someone already has).

Leave a Reply

Create a website or blog at WordPress.com