Hi mongrelians, I found myself with an unloaded full production server recently which presented an opportunity to run some performance tests against Mongrel and nginx. I thought this list might have some interest in the results of my tinkering! I actually ran two sets of studies, and Wayne Seguin kindly provided a lot of guidance and support in getting them running. Thanks Wayne! Also, EngineYard provided the hardware for the test. The first test looked at Mongrels running with nginx fair proxy vs. without it: http://www.misuse.org/science/2008/04/07/thin-ruby-on-rails-nginx-fair-proxy-performance-testing/ The second test looked at Mongrels vs Thin as the application handler for Rails: http://www.misuse.org/science/2008/04/07/thin-vs-mongrel-a-ruby-on-rails-performance-shootout/ Overall the results were not so surprising but at least there''s a little more hard data for people to look at while making decisions about this stuff. Summary 1: Fair proxy can boost performance significantly on sites that have a mix of 95% fast (<1s) and 5% slow-ish (4s) pages. Summary 2: Unix sockets are probably 4% faster than IP sockets (and therefore Thin is about 4% faster than Mongrel). One thing I did notice was that under heavy load nginx fair proxy caused some weird "clustering" of requests - the graphs (unpublished) show it pretty clearly while the statistics (published above) do not. So while fair proxy looks pretty good overall, if you are heavily loaded, I might recommend from the study that you do not use fair proxy, at least the version I tested (current stable release). Your comments and feedback are welcome. I hope the studies'' information and quality are usable to you. It''s a very small contribution back to this great community and the people who have contributed so much more. Steve
On Mon, 07 Apr 2008 23:14:47 -0700 Steve Midgley <public at misuse.org> wrote:> Hi mongrelians, > > http://www.misuse.org/science/2008/04/07/thin-ruby-on-rails-nginx-fair-proxy-performance-testing/Yay! You mentioned standard deviations. Bad! You didn''t give any. That''d help people figure out if the 4% is even worth it.> The second test looked at Mongrels vs Thin as the application handler > for Rails: > > http://www.misuse.org/science/2008/04/07/thin-vs-mongrel-a-ruby-on-rails-performance-shootout/"Thin is the newest incarnation of Zed Shaw?s (and now community managed) masterwork Mongrel." That statement isn''t correct. Please don''t make it since I don''t think the author of Thin or the authors of Mongrel would say that. Oh, well they''d say my stuff is a masterwork, just not that Thin is an incarnation of Mongrel. :-)> Overall the results were not so surprising but at least there''s a > little more hard data for people to look at while making decisions > about this stuff. > > Summary 1: Fair proxy can boost performance significantly on sites that > have a mix of 95% fast (<1s) and 5% slow-ish (4s) pages. > > Summary 2: Unix sockets are probably 4% faster than IP sockets (and > therefore Thin is about 4% faster than Mongrel).I don''t think it''s unix sockets vs. ip sockets. These days those aren''t really that much faster than a localhost connection thanks to advancements in performance for internal IPC. Still, 4% increase for all that work done in Thin is kind of lame. These kinds of results keep bringing me back to the real problem: Ruby. It''s not what IO loop you use, or threading model, or socket type, but more just that Ruby is slow and it''s processing sucks. Another comment I''d make is, why not also release the methodology you gave. I''m actually working on a presentation for RuPy so if you''d like to hack on a repeatable study with an automated report let me know. Could be a good test for what I''m writing for the conference. -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/
On 8 Apr 2008, at 08:20, Zed A. Shaw wrote:> > I don''t think it''s unix sockets vs. ip sockets. These days those > aren''t really that much faster than a localhost connection thanks to > advancements in performance for internal IPC. > > Still, 4% increase for all that work done in Thin is kind of lame. > These kinds of results keep bringing me back to the real problem: > Ruby. It''s not what IO loop you use, or threading model, or socket > type, but more just that Ruby is slow and it''s processing sucks.I believe this is all correct, however there are other advantages to unix sockets for certain setups. They''re easier to secure (this is not always relevant, but it can be). In some scenarios they can be easier to manage too (you can glob them, it''s harder to ''glob'' using netstat). I know that was a discussion about performance, this is just my $0.02 :-)
> I''m actually working on a presentation for RuPy so if you''d like > to hack on a repeatable study with an automated report let me know.ebb includes four different benchmarks that i often use to measure against thin, mongrel, fastcgi, and evented mongrel. http://github.com/ry/ebb/tree/master/benchmark but i will present these at RuPy too! ry
>On Mon, 07 Apr 2008 23:14:47 -0700 > > > > > http://www.misuse.org/science/2008/04/07/thin-ruby-on-rails-nginx-fair-proxy-performance-testing/ > >Yay! You mentioned standard deviations. Bad! You didn''t give any. >That''d help people figure out if the 4% is even worth it.I did! I did! The raw data (including std dev''s) were linked off the main post. That link is: http://www.misuse.org/downloads/thin_mongrel_test_results.html - I thought that level of info on the main post would be too distracting/confusing for many readers.. I tried to be meticulous in collecting these data, and Zed, for every std dev cut and paste out of JMeter I thought off you. :)>[snip] >"Thin is the newest incarnation of Zed Shaw''s (and now community >managed) masterwork Mongrel." > >That statement isn''t correct. Please don''t make it since I don''t >think >the author of Thin or the authors of Mongrel would say that. Oh, well >they''d say my stuff is a masterwork, just not that Thin is an >incarnation of Mongrel. :-)Ok - that''s fixed. I said that b/c I thought Thin is using your/mongrel''s ragel http states. But you know (way) more than me on this. So I changed the language.>[snip] >I don''t think it''s unix sockets vs. ip sockets. These days those >aren''t really that much faster than a localhost connection thanks to >advancements in performance for internal IPC.The Thin author Marc said the same thing. I''ll adjust that in the article soon.>[snip] >Another comment I''d make is, why not also release the methodology you >gave. I''m actually working on a presentation for RuPy so if you''d >like >to hack on a repeatable study with an automated report let me know. >Could be a good test for what I''m writing for the conference.Thanks for the encouragement. I just published my methodology, which was the same for both tests, at the end of the second report: http://www.misuse.org/science/2008/04/07/thin-vs-mongrel-a-ruby-on-rails-performance-shootout/ Feedback and criticism are welcome on that methodology. I have done a fair bit of performance testing in the past, but not in such a general sense as comparing two different application servers, so I may have done something wrong. I''d also be open to performing more tests using this methodology and codebase, but I don''t have access to an unloaded high performance webserver anymore. I could probably talk Ezra or Wayne into loaning me one though. :) Drop me a line here or personally if that sounds useful. Steve