Can any of you recommend an open source web stress/load testing tool, preferably one that runs on OS X/Linux, but Windows is OK as a fall-back. I''d like to load up my Rails application, to make sure it doesn''t have any failure modes under heavy load. Thanks, Dave
For basic load-testing you can use ab (apache benchmark) It''s already on OS X/Linux if apache (apache-devel?) is installed. On Jul 25, 2005, at 5:51 PM, Dave Ringoen wrote:> Can any of you recommend an open source web stress/load testing tool, > preferably one that runs on OS X/Linux, but Windows is OK as a fall- > back. > > I''d like to load up my Rails application, to make sure it doesn''t > have any > failure modes under heavy load. > > Thanks, > Dave > > > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >
> Can any of you recommend an open source web stress/load testing tool, > preferably one that runs on OS X/Linux, but Windows is OK as a fall-back. > > I''d like to load up my Rails application, to make sure it doesn''t have any > failure modes under heavy load.There''s a good set of notes on this in the Deployment chapter of the Rails book, but otherwise I''d recommend reading up on siege: http://www.joedog.org/siege/ -- David Heinemeier Hansson http://www.loudthinking.com -- Broadcasting Brain http://www.basecamphq.com -- Online project management http://www.backpackit.com -- Personal information manager http://www.rubyonrails.com -- Web-application framework
The microsoft web stress tool obviously only runs on windows and it''s not open source, but it is free and a really great application. http://www.microsoft.com/technet/archive/itsolutions/intranet/downloads/webstres.mspx Andrew ----- Original Message ----- From: "Dave Ringoen" <email-aMBdsrFCgW+KfZpkr/XK+Q@public.gmane.org> To: <rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org> Sent: Tuesday, July 26, 2005 7:51 AM Subject: [Rails] Web Stress/Load testing tool Can any of you recommend an open source web stress/load testing tool, preferably one that runs on OS X/Linux, but Windows is OK as a fall-back. I''d like to load up my Rails application, to make sure it doesn''t have any failure modes under heavy load. Thanks, Dave _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails
Single url pounding: Seige http://www.joedog.org/siege/ Path testing: JMeter http://jakarta.apache.org/jmeter/ -Brian On Jul 25, 2005, at 5:51 PM, Dave Ringoen wrote:> Can any of you recommend an open source web stress/load testing tool, > preferably one that runs on OS X/Linux, but Windows is OK as a fall- > back. > > I''d like to load up my Rails application, to make sure it doesn''t > have any > failure modes under heavy load. > > Thanks, > Dave > > > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails > >
Alright! Something I actually know! Time to start a flame war. :-) Seriously though, this is a big part of my "for fun" research, so here''s the short list of gear I''ve used and some advice. First, some quick definitions: - functional testing -- Trying to break a web application by shoving crap into the forms and clicking buttons the developers don''t click. - load testing -- Trying to break a web application by pounding the hell out of it with tons of simultaneous connections. - performance testing -- Trying to analyze the performance of a web application so that you can compare it against: a previous release, another application, a base "perfect" case, or a requirement. The tools for all three: * httperf -- http://www.hpl.hp.com/research/linux/httperf/ My personal favorite since it''s the only one that gives detailed statistics of request, response, connection, transfers, etc. and even separates the headers from the body to spot additional problems. Command line only. Additionally it satisfies my fetish that a tool without a standard deviation is useless. * TestMaker -- http://www.pushtotest.com/ More of a testing tool which uses Jython to let you write up pretty complex test scenarios against a wide range or protocols. It''s kind of difficult to use, but if you need to do deep automated testing then it''s the best out there. It can also do "performance" testing, but what they call "performance" testing is more like load testing. * Siege -- Already mentioned before. This is also a good load testing tool, but it won''t tell you anything useful for performance comparison. Like other tools if it doesn''t have a standard deviation then it''s useless for any real work. Siege also doesn''t tell you useful stuff like actual concurrent connections made (httperf does). * JMeter -- Again already mentioned. I know people are going to hate me for saying this, but please stay away from JMeter. It has the absolute worst statistics of all the tools, is difficult to use, has bad misleading graphs, complicated terminology, never defines "user", and it can''t figure out if it is a functional testing, load testing, or performance testing tool. * Watir -- http://wtr.rubyforge.org/ My absolute freaking favorite tool for functional testing. It actually uses the real deal Internet Explorer to do it''s tests, so you know that your site works with the majority of your users. It''s also damn easy to use. I had an intern at work using it in an hour and she''d never used Ruby before. You can also do a bit of load testing by launching a ton of independent test cases in a massive loop. You might be able to squeeze some performance metrics out of it by timing how long a page takes to render, which is actually a nice measure of what the users think of your site, but YMMV. * curl -- http://curl.haxx.se/ Nothing beats running curl in a massive loop or creating a ton of curl processes to pound a server. You won''t get performance metrics out of it, but it will surely tell you if your system can handle the load. It''s also easy to use for quick tests. When I''m testing a Rails deploy I just hit it with curl to see if it''s working well or not. * SLAMD -- http://www.slamd.com/ Never used this, but heard of this one guy who was this big time UNIX guru in Switzerland who managed to get a SLAMD cluster working one time before he shot himself in misery. :-) Also doesn''t produce accurate statistics so performance evaluation is useless, but good for load testing, if you can get it running. * JWebUnit -- Actually, Java has a lot of functional test tools that were crammed into the JUnit framework. Rock on if you like coding your tests in Java. * webunit -- http://mechanicalcat.net/tech/webunit/ Kind of the same thing as JWebUnit but for Python. Python has a few tools that I''ve used in the past, but most of them were pre-ALPHA quality. None of them were good for performance testing, just functional or load testing. A common theme you may notice is that all of these tools have very useless statistics, so they don''t help you compare the performance of systems. It may sound like anal academic nonsense, but without at least a standard deviation or range calculation you''ll never be able to find out if your system is performing *consistently* under load. My recommendation is to use httperf to check the performance of systems and to make sure that your proposed performance enhancements actually do improve things. Other than that you''re pretty much just at the mercy of whatever is available. Zed A. Shaw http://www.zedshaw.com/ On Mon, 2005-07-25 at 15:51 -0600, Dave Ringoen wrote:> Can any of you recommend an open source web stress/load testing tool, > preferably one that runs on OS X/Linux, but Windows is OK as a fall-back. > > I''d like to load up my Rails application, to make sure it doesn''t have any > failure modes under heavy load. > > Thanks, > Dave
Thanks, Zed! That''s a great write-up! Dave On 7/25/05 6:37 PM, "Zed A. Shaw" <zedshaw-dd7LMGGEL7NBDgjK7y7TUQ@public.gmane.org> wrote:> Alright! Something I actually know! Time to start a flame war. :-) > Seriously though, this is a big part of my "for fun" research, so here''s > the short list of gear I''ve used and some advice. > > First, some quick definitions: > > - functional testing -- Trying to break a web application by shoving > crap into the forms and clicking buttons the developers don''t click. > - load testing -- Trying to break a web application by pounding the hell > out of it with tons of simultaneous connections. > - performance testing -- Trying to analyze the performance of a web > application so that you can compare it against: a previous release, > another application, a base "perfect" case, or a requirement. > > > The tools for all three: > > * httperf -- http://www.hpl.hp.com/research/linux/httperf/ > My personal favorite since it''s the only one that gives detailed > statistics of request, response, connection, transfers, etc. and even > separates the headers from the body to spot additional problems. > Command line only. Additionally it satisfies my fetish that a tool > without a standard deviation is useless. > > * TestMaker -- http://www.pushtotest.com/ > More of a testing tool which uses Jython to let you write up pretty > complex test scenarios against a wide range or protocols. It''s kind of > difficult to use, but if you need to do deep automated testing then it''s > the best out there. It can also do "performance" testing, but what they > call "performance" testing is more like load testing. > > * Siege -- Already mentioned before. This is also a good load testing > tool, but it won''t tell you anything useful for performance comparison. > Like other tools if it doesn''t have a standard deviation then it''s > useless for any real work. Siege also doesn''t tell you useful stuff > like actual concurrent connections made (httperf does). > > * JMeter -- Again already mentioned. I know people are going to hate me > for saying this, but please stay away from JMeter. It has the absolute > worst statistics of all the tools, is difficult to use, has bad > misleading graphs, complicated terminology, never defines "user", and it > can''t figure out if it is a functional testing, load testing, or > performance testing tool. > > * Watir -- http://wtr.rubyforge.org/ > My absolute freaking favorite tool for functional testing. It actually > uses the real deal Internet Explorer to do it''s tests, so you know that > your site works with the majority of your users. It''s also damn easy to > use. I had an intern at work using it in an hour and she''d never used > Ruby before. You can also do a bit of load testing by launching a ton > of independent test cases in a massive loop. You might be able to > squeeze some performance metrics out of it by timing how long a page > takes to render, which is actually a nice measure of what the users > think of your site, but YMMV. > > * curl -- http://curl.haxx.se/ > Nothing beats running curl in a massive loop or creating a ton of curl > processes to pound a server. You won''t get performance metrics out of > it, but it will surely tell you if your system can handle the load. > It''s also easy to use for quick tests. When I''m testing a Rails deploy > I just hit it with curl to see if it''s working well or not. > > * SLAMD -- http://www.slamd.com/ > Never used this, but heard of this one guy who was this big time UNIX > guru in Switzerland who managed to get a SLAMD cluster working one time > before he shot himself in misery. :-) Also doesn''t produce accurate > statistics so performance evaluation is useless, but good for load > testing, if you can get it running. > > * JWebUnit -- Actually, Java has a lot of functional test tools that > were crammed into the JUnit framework. Rock on if you like coding your > tests in Java. > > * webunit -- http://mechanicalcat.net/tech/webunit/ > Kind of the same thing as JWebUnit but for Python. Python has a few > tools that I''ve used in the past, but most of them were pre-ALPHA > quality. None of them were good for performance testing, just > functional or load testing. > > > A common theme you may notice is that all of these tools have very > useless statistics, so they don''t help you compare the performance of > systems. It may sound like anal academic nonsense, but without at least > a standard deviation or range calculation you''ll never be able to find > out if your system is performing *consistently* under load. My > recommendation is to use httperf to check the performance of systems and > to make sure that your proposed performance enhancements actually do > improve things. Other than that you''re pretty much just at the mercy of > whatever is available. > > Zed A. Shaw > http://www.zedshaw.com/ > > > On Mon, 2005-07-25 at 15:51 -0600, Dave Ringoen wrote: >> Can any of you recommend an open source web stress/load testing tool, >> preferably one that runs on OS X/Linux, but Windows is OK as a fall-back. >> >> I''d like to load up my Rails application, to make sure it doesn''t have any >> failure modes under heavy load. >> >> Thanks, >> Dave > > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >
Similar to below, my experience has led me to use a few tools, in various situations. But, httperf is excellent as mentioned. Also, as someone else mentioned the MS Web Stress tool is, even if I sort of hate to recommend it, actually pretty darn good. Mostly I find it useful for just pounding the heck out of an app to see if it breaks at various load levels. I used this a ton when testing various Java servlet containers, as you can''t trust the benchmarks you see (IMHO they are only applicable to that particular application - we had considerably different results for some containers with our application). Also, as mentioned below, I second TestMaker. This is a really sweet tool. Httperf is great for just doing a lot of load testing, and relatively brain dead URL pounding kinds of stuff. But, TestMaker is more of an interactive testing tool. It does better following chains of actions or flows through an app. As said, there is a bit of a learning curve, but they have a recording feature which will help get you up and running. We used this tool extensively on the last project I did for my day job. On 7/25/05 5:37 PM, "Zed A. Shaw" <zedshaw-dd7LMGGEL7NBDgjK7y7TUQ@public.gmane.org> wrote:> Alright! Something I actually know! Time to start a flame war. :-) > Seriously though, this is a big part of my "for fun" research, so here''s > the short list of gear I''ve used and some advice. > > First, some quick definitions: > > - functional testing -- Trying to break a web application by shoving > crap into the forms and clicking buttons the developers don''t click. > - load testing -- Trying to break a web application by pounding the hell > out of it with tons of simultaneous connections. > - performance testing -- Trying to analyze the performance of a web > application so that you can compare it against: a previous release, > another application, a base "perfect" case, or a requirement. > > > The tools for all three: > > * httperf -- http://www.hpl.hp.com/research/linux/httperf/ > My personal favorite since it''s the only one that gives detailed > statistics of request, response, connection, transfers, etc. and even > separates the headers from the body to spot additional problems. > Command line only. Additionally it satisfies my fetish that a tool > without a standard deviation is useless. > > * TestMaker -- http://www.pushtotest.com/ > More of a testing tool which uses Jython to let you write up pretty > complex test scenarios against a wide range or protocols. It''s kind of > difficult to use, but if you need to do deep automated testing then it''s > the best out there. It can also do "performance" testing, but what they > call "performance" testing is more like load testing. > > * Siege -- Already mentioned before. This is also a good load testing > tool, but it won''t tell you anything useful for performance comparison. > Like other tools if it doesn''t have a standard deviation then it''s > useless for any real work. Siege also doesn''t tell you useful stuff > like actual concurrent connections made (httperf does). > > * JMeter -- Again already mentioned. I know people are going to hate me > for saying this, but please stay away from JMeter. It has the absolute > worst statistics of all the tools, is difficult to use, has bad > misleading graphs, complicated terminology, never defines "user", and it > can''t figure out if it is a functional testing, load testing, or > performance testing tool. > > * Watir -- http://wtr.rubyforge.org/ > My absolute freaking favorite tool for functional testing. It actually > uses the real deal Internet Explorer to do it''s tests, so you know that > your site works with the majority of your users. It''s also damn easy to > use. I had an intern at work using it in an hour and she''d never used > Ruby before. You can also do a bit of load testing by launching a ton > of independent test cases in a massive loop. You might be able to > squeeze some performance metrics out of it by timing how long a page > takes to render, which is actually a nice measure of what the users > think of your site, but YMMV. > > * curl -- http://curl.haxx.se/ > Nothing beats running curl in a massive loop or creating a ton of curl > processes to pound a server. You won''t get performance metrics out of > it, but it will surely tell you if your system can handle the load. > It''s also easy to use for quick tests. When I''m testing a Rails deploy > I just hit it with curl to see if it''s working well or not. > > * SLAMD -- http://www.slamd.com/ > Never used this, but heard of this one guy who was this big time UNIX > guru in Switzerland who managed to get a SLAMD cluster working one time > before he shot himself in misery. :-) Also doesn''t produce accurate > statistics so performance evaluation is useless, but good for load > testing, if you can get it running. > > * JWebUnit -- Actually, Java has a lot of functional test tools that > were crammed into the JUnit framework. Rock on if you like coding your > tests in Java. > > * webunit -- http://mechanicalcat.net/tech/webunit/ > Kind of the same thing as JWebUnit but for Python. Python has a few > tools that I''ve used in the past, but most of them were pre-ALPHA > quality. None of them were good for performance testing, just > functional or load testing. > > > A common theme you may notice is that all of these tools have very > useless statistics, so they don''t help you compare the performance of > systems. It may sound like anal academic nonsense, but without at least > a standard deviation or range calculation you''ll never be able to find > out if your system is performing *consistently* under load. My > recommendation is to use httperf to check the performance of systems and > to make sure that your proposed performance enhancements actually do > improve things. Other than that you''re pretty much just at the mercy of > whatever is available. > > Zed A. Shaw > http://www.zedshaw.com/ > > > On Mon, 2005-07-25 at 15:51 -0600, Dave Ringoen wrote: >> Can any of you recommend an open source web stress/load testing tool, >> preferably one that runs on OS X/Linux, but Windows is OK as a fall-back. >> >> I''d like to load up my Rails application, to make sure it doesn''t have any >> failure modes under heavy load. >> >> Thanks, >> Dave > > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails