Just wanted to pick some smart people about this topic: What are you guys doing to increase the speed of your specs? I''m a big fan of autotest, but right now my current project has 438 specs (for rails). Most of them are in the model, and for all of them we are hitting the database (they are more functional specs than unit-tests). The whole suite takes 112 seconds (meaning the red to green cycle leaves me learning how to juggle, just like Jim Weirich with make). Any recommendations, short of mocking/stubbing to get around this? Some of the suggestions I''ve heard are: - In Memory Database w/ Sqlite. As far as I know, no one has gotten this to work (with rspec, at least) - Using faster hardware. I don''t have the budget right now for a better machine. One interesting idea is this one: http:// www.dcmanges.com/blog/deep-test-preview, although as far as I know, it doesn''t work for rspec. - Only running specs per file, and using continous integration to do the rest. Right now this isn''t an option for testing my User model, which has 92 examples, and takes 20 seconds. Also, I''d really rather not commit before I know the whole test suite passes - Other ideas? Scott
Scott Taylor wrote:> Just wanted to pick some smart people about this topic: What are you > guys doing to increase the speed of your specs? > > I''m a big fan of autotest, but right now my current project has 438 > specs (for rails). Most of them are in the model, and for all of > them we are hitting the database (they are more functional specs than > unit-tests). The whole suite takes 112 seconds (meaning the red to > green cycle leaves me learning how to juggle, just like Jim Weirich > with make). Any recommendations, short of mocking/stubbing to get > around this? Some of the suggestions I''ve heard are: > > - In Memory Database w/ Sqlite. As far as I know, no one has gotten > this to work (with rspec, at least) > - Using faster hardware. I don''t have the budget right now for a > better machine. One interesting idea is this one: http:// > www.dcmanges.com/blog/deep-test-preview, although as far as I know, > it doesn''t work for rspec. > - Only running specs per file, and using continous integration to do > the rest. Right now this isn''t an option for testing my User model, > which has 92 examples, and takes 20 seconds. Also, I''d really rather > not commit before I know the whole test suite passes > - Other ideas? > > Scott > > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >1. Use the Ruby profiler to see where it''s spending its time. 2. Recompile the Ruby interpreter with full optimization if you haven''t already. Assuming you''re on a platform supported by gcc, download the Ruby source and type export CFLAGS=''-O3 -march=<whatever>'' before the "./configure". <whatever> is your chip architecture, for example "pentium3", "pentium4", "athlon-xp", ... If your test suite and application are fully open source and you don''t mind someone playing with it, I''m collecting benchmark suites for the Ruby interpreter and would love to have a nice heavy "rspec" run as an example of real-world Ruby usage. Contact me off-list if you''re interested, or if you want me to show you how to profile the Ruby interpreter itself on your test case.
> > If your test suite and application are fully open source and you don''t > mind someone playing with it, I''m collecting benchmark suites for the > Ruby interpreter and would love to have a nice heavy "rspec" run as an > example of real-world Ruby usage. Contact me off-list if you''re > interested, or if you want me to show you how to profile the Ruby > interpreter itself on your test case. > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-usersUnfortunately, the app isn''t open source, although I do have one other app, and a rails plugins which both have specs: http://rubyforge.org/projects/replacefixtures/ http://rubyforge.org/projects/datedbackup/ Those don''t hit a database, so they are incredibly fast: Finished in 0.19639 seconds 257 examples, 2 failures, 4 pending I''m going to give the profiler a shot, plus compiling ruby from source (I''m using the fink package manager, after having ditched darwin ports). I''ll see where that leaves me. As for running the profiler - is there an easy way to wrap the blocks/ closures given to an ''it'' example with the profiler? (I haven''t contacted you off list since I''m sure others would benefit from this information). Other, more general ideas are still welcome... Scott
Scott Taylor wrote:>> If your test suite and application are fully open source and you don''t >> mind someone playing with it, I''m collecting benchmark suites for the >> Ruby interpreter and would love to have a nice heavy "rspec" run as an >> example of real-world Ruby usage. Contact me off-list if you''re >> interested, or if you want me to show you how to profile the Ruby >> interpreter itself on your test case. >> _______________________________________________ >> rspec-users mailing list >> rspec-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/rspec-users > > Unfortunately, the app isn''t open source, although I do have one > other app, and a rails plugins which both have specs: > > http://rubyforge.org/projects/replacefixtures/ > http://rubyforge.org/projects/datedbackup/ > > Those don''t hit a database, so they are incredibly fast: > > Finished in 0.19639 seconds > > 257 examples, 2 failures, 4 pending > > I''m going to give the profiler a shot, plus compiling ruby from > source (I''m using the fink package manager, after having ditched > darwin ports). I''ll see where that leaves me. > > As for running the profiler - is there an easy way to wrap the blocks/ > closures given to an ''it'' example with the profiler? (I haven''t > contacted you off list since I''m sure others would benefit from this > information). > > Other, more general ideas are still welcome... > > Scott > > > > > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >Ah ... you''re on a Mac ... Intel or PPC? If Intel, you might want to look into their VTune tool (I think it runs on MacOS). Even if you don''t have the latest whiz-bang multi-core Intel chip, VTune will tell you stuff you probably never knew you should care about. (I''m an Intel-free zone, and so I use AMD''s CodeAnalyst.) :) Database, eh? Maybe you should be recompiling the database from source too. PostgreSQL? MySQL? There are *lots* of tweaking gizmos for them, plus a benchmark called "MySQL Super Smack".
Okay - Now I think we are getting off topic, but...> > Ah ... you''re on a Mac ... Intel or PPC? If Intel, you might want to > look into their VTune tool (I think it runs on MacOS). Even if you > don''t > have the latest whiz-bang multi-core Intel chip, VTune will tell you > stuff you probably never knew you should care about. >I''ll check that out.> (I''m an Intel-free zone, and so I use AMD''s CodeAnalyst.) :) > > Database, eh? Maybe you should be recompiling the database from source > too. PostgreSQL? MySQL? There are *lots* of tweaking gizmos for them, > plus a benchmark called "MySQL Super Smack".I''ve heard of it. Any to recommend? As an aside: how correlated do you think performance of specs is to real performance of a production rails app? I''m asking this as knowing almost nothing about mysql (I learned about the query_cache just the other day) - as a developer who just wants my app to run (it crashes at least once a day). Scott
In-memory with sqlite worked fine with rspec-0.8 (it''s been a while since I did this!). Google for instructions or drop me a line. Don''t forget autotest/zentest to run only those tests which have ben affected by changes. Rgds, Jerry Scott Taylor wrote:> Just wanted to pick some smart people about this topic: What are you > guys doing to increase the speed of your specs? > > I''m a big fan of autotest, but right now my current project has 438 > specs (for rails). Most of them are in the model, and for all of > them we are hitting the database (they are more functional specs than > unit-tests). The whole suite takes 112 seconds (meaning the red to > green cycle leaves me learning how to juggle, just like Jim Weirich > with make). Any recommendations, short of mocking/stubbing to get > around this? Some of the suggestions I''ve heard are: > > - In Memory Database w/ Sqlite. As far as I know, no one has gotten > this to work (with rspec, at least) > - Using faster hardware. I don''t have the budget right now for a > better machine. One interesting idea is this one: http:// > www.dcmanges.com/blog/deep-test-preview, although as far as I know, > it doesn''t work for rspec. > - Only running specs per file, and using continous integration to do > the rest. Right now this isn''t an option for testing my User model, > which has 92 examples, and takes 20 seconds. Also, I''d really rather > not commit before I know the whole test suite passes > - Other ideas? > > Scott > > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >
On Oct 4, 2007, at 6:11 AM, Jerry West wrote:> In-memory with sqlite worked fine with rspec-0.8 (it''s been a while > since I did this!). Google for instructions or drop me a line. Don''t > forget autotest/zentest to run only those tests which have ben > affected > by changes.Actually, on my current rails project I can''t use sqlite, because of a bug in rails: http://dev.rubyonrails.org/ticket/9385 So out goes that idea (at least for the forseeable future). What speed difference do you see with in-memory database? I''m already using ZenTest. When going from red -> (subset) green => (all) green, the last step is killing me (because I have to wait around for 2 minutes for all the specs to fail. This usually results in me spending a lot of time responding to emails while I should be developing ;) Scott
On Oct 4, 2007, at 6:11 AM, Jerry West wrote:> In-memory with sqlite worked fine with rspec-0.8 (it''s been a while > since I did this!). Google for instructions or drop me a line. Don''t > forget autotest/zentest to run only those tests which have ben > affected > by changes.Actually, on my current rails project I can''t use sqlite, because of a bug in rails: http://dev.rubyonrails.org/ticket/9385 So out goes that idea (at least for the forseeable future). What speed difference do you see with in-memory database? I''m already using ZenTest. When going from red -> (subset) green => (all) green, the last step is killing me (because I have to wait around for 2 minutes for all the specs to fail. This usually results in me spending a lot of time responding to emails while I should be developing ;) Scott
Scott, I don''t really have a lot to contribute on how to make it faster, other than to outline what we''ve been doing on our projects. On one of our current projects we have the following 2570 examples that run in ~70 seconds on our pairing stations (mac minis, 1.83 c2d). In general across our various machines is at or a little more than a minute for specs for controllers, models, helpers, lib, views, and plugins. Our Story suite takes longer, but it''s still under development so I don''t really count it at this point. We have Ruby 1.8.6 installed from MacPorts on all machines, as well as MySQL 5. (current as of a month ago) from macports. We make good use of mocking and stubbing through our controller tests, and little use of fixtures. We primarily use the spec_attribute_helper (or factory method) as Luke Redpath and Dan Manges have outlined in their respective blog articles. I''ve been looking at deep test, or possible spec_distributed as a way to speed things up more. Our main issue is our precommit task (rake cruise which our ci server also runs) executes rcov to test for full coverage and adds 15-25 seconds to the whole thing bringing it up to a minute and a half. We also make heavy use of some simple custom rcov tasks ala rake app:coverage:models, rake app:coverage:lib, etc to help when we are working on slices of the apps. So far we have found that regardless of the wait (and the increased chances of wandering about to other tasks) we always want to run the full stack locally and not just a slice. Then again all this strikes me as rather funny as I remember waiting 5 or 6 minutes back in my .NET days for a quarter our tests to run, with no coverage report. I guess ruby and rspec spoiled me :) - Chad NOTE: Our project is on EdgeRails (few revisions back) and Trunk RSpec On Oct 6, 2007, at 11:16 PM, Scott Taylor wrote:> > On Oct 4, 2007, at 6:11 AM, Jerry West wrote: > >> In-memory with sqlite worked fine with rspec-0.8 (it''s been a while >> since I did this!). Google for instructions or drop me a line. >> Don''t >> forget autotest/zentest to run only those tests which have ben >> affected >> by changes. > > > Actually, on my current rails project I can''t use sqlite, because of > a bug in rails: > > http://dev.rubyonrails.org/ticket/9385 > > So out goes that idea (at least for the forseeable future). What > speed difference do you see with in-memory database? > > I''m already using ZenTest. When going from red -> (subset) green => > (all) green, the last step is killing me (because I have to wait > around for 2 minutes for all the specs to fail. This usually results > in me spending a lot of time responding to emails while I should be > developing ;) > > Scott > > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users
On Oct 7, 2007, at 12:31 AM, Chad Humphries wrote:> Scott, > > I don''t really have a lot to contribute on how to make it faster, > other than to outline what we''ve been doing on our projects. > > On one of our current projects we have the following 2570 examples > that run in ~70 seconds on our pairing stations (mac minis, 1.83I assume your "pairing stations" are two separate mac-minis, in which you practice pair programming? Or is this a cluster of two mac-minis? But this sounds great - 70 seconds for 2500 specs. How many of those are model specs (that hit the database)?> c2d). In general across our various machines is at or a little more > than a minute for specs for controllers, models, helpers, lib, views, > and plugins. Our Story suite takes longer, but it''s still under > development so I don''t really count it at this point. We have Ruby > 1.8.6 installed from MacPorts on all machines, as well as MySQL 5. > (current as of a month ago) from macports. > > We make good use of mocking and stubbing through our controller > tests, and little use of fixtures. We primarily use the > spec_attribute_helper (or factory method) as Luke Redpath and Dan > Manges have outlined in their respective blog articles. I''ve been > looking at deep test, or possible spec_distributed as a way to speed > things up more. Our main issue is our precommit task (rake cruiseThe factory method (or attribute_helper) still hits the database. I don''t see it as any sort of performance gain. In fact, I''ve even developed a plugin around the Factory idea, and it was only when I started using it in all of my tests that the speed really started to affect me (I was using mocking/stubbing, with much frustration prior to that point). But to me it''s pretty clear the plugin (or the factory) is not the problem - the hit is the database. DHH saw this hit, and since they were using fixtures,he found that creating the fixtures, and then wrapping each test in a transaction was a huge performance gain. I wonder if the same would be true with setups/ before(:each)... The obvious thing to do to solve the performance problem is to remove the hit to the database. The question is: At what level of abstraction should this be done? The one camp (which would include fellows like Jay Fields), would mock/stub everything they don''t write. For me, I see testing as more than testing - it''s the documentation to my code which never lies to me (this documentation is so good, that I can give it to my boss, who is not a programmer). For that reason (testing is not about testing), I could never stub/ mock AR inline - that is, in the tests themselves (although if I had some sort of external plugin to do this for me, I may feel differently about it). The other alternatives seem to be: b) writing a sql parser, and c) speeding up the database (deeptest, in-memory databases, more hardware, etc). The sql parser is a big job, and speeding up the database doesn''t seem to give the performance improvement that I''m looking for. As for deeptest, I wouldn''t mind helping out with getting it to work for rspec (email me off-list at scott at railsnewbie.com, if you are interested in my help). I haven''t explored spec_distributed at all (in fact, I didn''t even know it existed, although I had a similar thought a few days ago).> which our ci server also runs) executes rcov to test for full > coverage and adds 15-25 seconds to the whole thing bringing it up to > a minute and a half.Coverage is not a big point for me. I''m happy with running rcov only once a week, assuming that I have 100% right now, and I develop everything test-first. Right now I''m no where near 100%, so I''m not too worried about it. Regards, Scott
On 10/6/07, Scott Taylor <mailing_lists at railsnewbie.com> wrote:> > On Oct 7, 2007, at 12:31 AM, Chad Humphries wrote: > > > Scott, > > > > I don''t really have a lot to contribute on how to make it faster, > > other than to outline what we''ve been doing on our projects. > > > > On one of our current projects we have the following 2570 examples > > that run in ~70 seconds on our pairing stations (mac minis, 1.83 > > I assume your "pairing stations" are two separate mac-minis, in which > you practice pair programming? Or is this a cluster of two mac-minis? > > But this sounds great - 70 seconds for 2500 specs. How many of those > are model specs (that hit the database)? > > > c2d). In general across our various machines is at or a little more > > than a minute for specs for controllers, models, helpers, lib, views, > > and plugins. Our Story suite takes longer, but it''s still under > > development so I don''t really count it at this point. We have Ruby > > 1.8.6 installed from MacPorts on all machines, as well as MySQL 5. > > (current as of a month ago) from macports. > > > > We make good use of mocking and stubbing through our controller > > tests, and little use of fixtures. We primarily use the > > spec_attribute_helper (or factory method) as Luke Redpath and Dan > > Manges have outlined in their respective blog articles. I''ve been > > looking at deep test, or possible spec_distributed as a way to speed > > things up more. Our main issue is our precommit task (rake cruise > > The factory method (or attribute_helper) still hits the database. I > don''t see it as any sort of performance gain. In fact, I''ve even > developed a plugin around the Factory idea, and it was only when I > started using it in all of my tests that the speed really started to > affect me (I was using mocking/stubbing, with much frustration prior > to that point). But to me it''s pretty clear the plugin (or the > factory) is not the problem - the hit is the database. DHH saw this > hit, and since they were using fixtures,he found that creating the > fixtures, and then wrapping each test in a transaction was a huge > performance gain. I wonder if the same would be true with setups/ > before(:each)... > > The obvious thing to do to solve the performance problem is to remove > the hit to the database. The question is: At what level of > abstraction should this be done? The one camp (which would include > fellows like Jay Fields), would mock/stub everything they don''t > write. For me, I see testing as more than testing - it''s the > documentation to my code which never lies to me (this documentation > is so good, that I can give it to my boss, who is not a programmer).I think you''d be far better served by user stories in this case. The user story is the tool we use to communicate with non-geeks, be it customers or our boss. Low-level specs fall squarely in the developer realm, as far as I''m concerned. They''re easy to read because we don''t like to torture ourselves. I think that they''re fantastic for communicating intent with other developers, but that''s where it ends. So, when you move documentation as a goal from specs over to stories, that frees specs to take advantage of techniques like mocks to help with your design. Documentation is a pleasant side effect of BDD, not a goal. Pat
On Oct 7, 2007, at 1:47 AM, Pat Maddox wrote:> On 10/6/07, Scott Taylor <mailing_lists at railsnewbie.com> wrote: >> >> On Oct 7, 2007, at 12:31 AM, Chad Humphries wrote: >> >>> Scott, >>> >>> I don''t really have a lot to contribute on how to make it faster, >>> other than to outline what we''ve been doing on our projects. >>> >>> On one of our current projects we have the following 2570 examples >>> that run in ~70 seconds on our pairing stations (mac minis, 1.83 >> >> I assume your "pairing stations" are two separate mac-minis, in which >> you practice pair programming? Or is this a cluster of two mac- >> minis? >> >> But this sounds great - 70 seconds for 2500 specs. How many of those >> are model specs (that hit the database)? >> >>> c2d). In general across our various machines is at or a little more >>> than a minute for specs for controllers, models, helpers, lib, >>> views, >>> and plugins. Our Story suite takes longer, but it''s still under >>> development so I don''t really count it at this point. We have Ruby >>> 1.8.6 installed from MacPorts on all machines, as well as MySQL 5. >>> (current as of a month ago) from macports. >>> >>> We make good use of mocking and stubbing through our controller >>> tests, and little use of fixtures. We primarily use the >>> spec_attribute_helper (or factory method) as Luke Redpath and Dan >>> Manges have outlined in their respective blog articles. I''ve >>> been >>> looking at deep test, or possible spec_distributed as a way to speed >>> things up more. Our main issue is our precommit task (rake cruise >> >> The factory method (or attribute_helper) still hits the database. I >> don''t see it as any sort of performance gain. In fact, I''ve even >> developed a plugin around the Factory idea, and it was only when I >> started using it in all of my tests that the speed really started to >> affect me (I was using mocking/stubbing, with much frustration prior >> to that point). But to me it''s pretty clear the plugin (or the >> factory) is not the problem - the hit is the database. DHH saw this >> hit, and since they were using fixtures,he found that creating the >> fixtures, and then wrapping each test in a transaction was a huge >> performance gain. I wonder if the same would be true with setups/ >> before(:each)... >> >> The obvious thing to do to solve the performance problem is to remove >> the hit to the database. The question is: At what level of >> abstraction should this be done? The one camp (which would include >> fellows like Jay Fields), would mock/stub everything they don''t >> write. For me, I see testing as more than testing - it''s the >> documentation to my code which never lies to me (this documentation >> is so good, that I can give it to my boss, who is not a programmer). > > I think you''d be far better served by user stories in this case. The > user story is the tool we use to communicate with non-geeks, be it > customers or our boss. Low-level specs fall squarely in the developer > realm, as far as I''m concerned. They''re easy to read because we don''t > like to torture ourselves. I think that they''re fantastic for > communicating intent with other developers, but that''s where it ends. > > So, when you move documentation as a goal from specs over to stories, > that frees specs to take advantage of techniques like mocks to help > with your design. Documentation is a pleasant side effect of BDD, not > a goal.Documentation is certainly a goal of BDD (Just not necessarily for the business user - as you pointed out). I don''t write my specs to be understandable by the business user, although it just often happens that they are (at least in this rails context, since even most of the fields of the database are more-or-less user facing). The point that I was trying to hammer home is that mocking/stubbing adds a lot of extra noise to a rails test, where it doesn''t in the other projects that I''ve worked on. But whether or not you call the specs "stories" or "integration specs" (it is, after all, all a continuum of what is considered an "integration" test), I still have no idea how to run them any faster. Scott
The Stories (ie Story Runner full integration tests that hit the db) are fairly slow by comparison, I agree. We have those run on our CI server, and only locally when we modify them. That''s our approach for handling integration testing. Re: The mac mini''s, yeah they were pairing stations (one per pair of users, no distributed spec running yet). Our main specs are built around heavier use of mocking/stubbing. - Chad On Oct 7, 2007, at 2:14 AM, Scott Taylor wrote:> > On Oct 7, 2007, at 1:47 AM, Pat Maddox wrote: > >> On 10/6/07, Scott Taylor <mailing_lists at railsnewbie.com> wrote: >>> >>> On Oct 7, 2007, at 12:31 AM, Chad Humphries wrote: >>> >>>> Scott, >>>> >>>> I don''t really have a lot to contribute on how to make it faster, >>>> other than to outline what we''ve been doing on our projects. >>>> >>>> On one of our current projects we have the following 2570 examples >>>> that run in ~70 seconds on our pairing stations (mac minis, 1.83 >>> >>> I assume your "pairing stations" are two separate mac-minis, in >>> which >>> you practice pair programming? Or is this a cluster of two mac- >>> minis? >>> >>> But this sounds great - 70 seconds for 2500 specs. How many of >>> those >>> are model specs (that hit the database)? >>> >>>> c2d). In general across our various machines is at or a little >>>> more >>>> than a minute for specs for controllers, models, helpers, lib, >>>> views, >>>> and plugins. Our Story suite takes longer, but it''s still under >>>> development so I don''t really count it at this point. We have >>>> Ruby >>>> 1.8.6 installed from MacPorts on all machines, as well as MySQL 5. >>>> (current as of a month ago) from macports. >>>> >>>> We make good use of mocking and stubbing through our controller >>>> tests, and little use of fixtures. We primarily use the >>>> spec_attribute_helper (or factory method) as Luke Redpath and Dan >>>> Manges have outlined in their respective blog articles. I''ve >>>> been >>>> looking at deep test, or possible spec_distributed as a way to >>>> speed >>>> things up more. Our main issue is our precommit task (rake cruise >>> >>> The factory method (or attribute_helper) still hits the database. I >>> don''t see it as any sort of performance gain. In fact, I''ve even >>> developed a plugin around the Factory idea, and it was only when I >>> started using it in all of my tests that the speed really started to >>> affect me (I was using mocking/stubbing, with much frustration prior >>> to that point). But to me it''s pretty clear the plugin (or the >>> factory) is not the problem - the hit is the database. DHH saw this >>> hit, and since they were using fixtures,he found that creating the >>> fixtures, and then wrapping each test in a transaction was a huge >>> performance gain. I wonder if the same would be true with setups/ >>> before(:each)... >>> >>> The obvious thing to do to solve the performance problem is to >>> remove >>> the hit to the database. The question is: At what level of >>> abstraction should this be done? The one camp (which would include >>> fellows like Jay Fields), would mock/stub everything they don''t >>> write. For me, I see testing as more than testing - it''s the >>> documentation to my code which never lies to me (this documentation >>> is so good, that I can give it to my boss, who is not a programmer). >> >> I think you''d be far better served by user stories in this case. The >> user story is the tool we use to communicate with non-geeks, be it >> customers or our boss. Low-level specs fall squarely in the >> developer >> realm, as far as I''m concerned. They''re easy to read because we >> don''t >> like to torture ourselves. I think that they''re fantastic for >> communicating intent with other developers, but that''s where it ends. >> >> So, when you move documentation as a goal from specs over to stories, >> that frees specs to take advantage of techniques like mocks to help >> with your design. Documentation is a pleasant side effect of BDD, >> not >> a goal. > > Documentation is certainly a goal of BDD (Just not necessarily for > the business user - as you pointed out). I don''t write my specs to > be understandable by the business user, although it just often > happens that they are (at least in this rails context, since even > most of the fields of the database are more-or-less user facing). > The point that I was trying to hammer home is that mocking/stubbing > adds a lot of extra noise to a rails test, where it doesn''t in the > other projects that I''ve worked on. > > But whether or not you call the specs "stories" or "integration > specs" (it is, after all, all a continuum of what is considered an > "integration" test), I still have no idea how to run them any faster. > > Scott > > > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users
On 10/6/07, Scott Taylor <mailing_lists at railsnewbie.com> wrote:> > On Oct 7, 2007, at 1:47 AM, Pat Maddox wrote: > > > On 10/6/07, Scott Taylor <mailing_lists at railsnewbie.com> wrote: > >> > >> On Oct 7, 2007, at 12:31 AM, Chad Humphries wrote: > >> > >>> Scott, > >>> > >>> I don''t really have a lot to contribute on how to make it faster, > >>> other than to outline what we''ve been doing on our projects. > >>> > >>> On one of our current projects we have the following 2570 examples > >>> that run in ~70 seconds on our pairing stations (mac minis, 1.83 > >> > >> I assume your "pairing stations" are two separate mac-minis, in which > >> you practice pair programming? Or is this a cluster of two mac- > >> minis? > >> > >> But this sounds great - 70 seconds for 2500 specs. How many of those > >> are model specs (that hit the database)? > >> > >>> c2d). In general across our various machines is at or a little more > >>> than a minute for specs for controllers, models, helpers, lib, > >>> views, > >>> and plugins. Our Story suite takes longer, but it''s still under > >>> development so I don''t really count it at this point. We have Ruby > >>> 1.8.6 installed from MacPorts on all machines, as well as MySQL 5. > >>> (current as of a month ago) from macports. > >>> > >>> We make good use of mocking and stubbing through our controller > >>> tests, and little use of fixtures. We primarily use the > >>> spec_attribute_helper (or factory method) as Luke Redpath and Dan > >>> Manges have outlined in their respective blog articles. I''ve > >>> been > >>> looking at deep test, or possible spec_distributed as a way to speed > >>> things up more. Our main issue is our precommit task (rake cruise > >> > >> The factory method (or attribute_helper) still hits the database. I > >> don''t see it as any sort of performance gain. In fact, I''ve even > >> developed a plugin around the Factory idea, and it was only when I > >> started using it in all of my tests that the speed really started to > >> affect me (I was using mocking/stubbing, with much frustration prior > >> to that point). But to me it''s pretty clear the plugin (or the > >> factory) is not the problem - the hit is the database. DHH saw this > >> hit, and since they were using fixtures,he found that creating the > >> fixtures, and then wrapping each test in a transaction was a huge > >> performance gain. I wonder if the same would be true with setups/ > >> before(:each)... > >> > >> The obvious thing to do to solve the performance problem is to remove > >> the hit to the database. The question is: At what level of > >> abstraction should this be done? The one camp (which would include > >> fellows like Jay Fields), would mock/stub everything they don''t > >> write. For me, I see testing as more than testing - it''s the > >> documentation to my code which never lies to me (this documentation > >> is so good, that I can give it to my boss, who is not a programmer). > > > > I think you''d be far better served by user stories in this case. The > > user story is the tool we use to communicate with non-geeks, be it > > customers or our boss. Low-level specs fall squarely in the developer > > realm, as far as I''m concerned. They''re easy to read because we don''t > > like to torture ourselves. I think that they''re fantastic for > > communicating intent with other developers, but that''s where it ends. > > > > So, when you move documentation as a goal from specs over to stories, > > that frees specs to take advantage of techniques like mocks to help > > with your design. Documentation is a pleasant side effect of BDD, not > > a goal. > > Documentation is certainly a goal of BDD (Just not necessarily for > the business user - as you pointed out). I don''t write my specs to > be understandable by the business user, although it just often > happens that they are (at least in this rails context, since even > most of the fields of the database are more-or-less user facing). > The point that I was trying to hammer home is that mocking/stubbing > adds a lot of extra noise to a rails test, where it doesn''t in the > other projects that I''ve worked on.Well, one of the main parts of programming is making tradeoffs. You''re probably not going to run 2500 tests that hit the db in under 30 seconds. So you either don''t hit the db or you don''t run all the tests all the time. You can certainly change autotest to not automatically run the full suite after you go red->green. Only have it run the specs for the files you changed. Break your specs into fast and slow groups. Run the fast ones continuously and the slow ones left often. Work on moving one or two slow specs into the fast group every day. You''ll probably discover some techniques that you can share with all of us :) As far as mocks adding a lot of extra noise, I find in general that when mocks are painful for me it''s because my code is too tightly coupled. Just like my body tells me something is wrong through pain, so does my code.> But whether or not you call the specs "stories" or "integration > specs" (it is, after all, all a continuum of what is considered an > "integration" test), I still have no idea how to run them any faster.Well, I don''t want to beat this to death, but I think it''s dangerous to look at specs and stories as simply different degrees of the same concept. The fundamental difference is that you should not change a story without consulting a customer, and you should not have to consult a customer to change a spec. Stories represent assumptions and business value. Assumptions are learned through conversations with the customer, and only the customer can define business value. Specs are merely a tool to help us generate that business value, and velocity goes way down if we have to worry that changing some specs will impact the customer in some way. Worse, if we don''t worry about it, we may end up changing some assumptions around and thus deliver the wrong product. In the ever popular analogy of car mechanics, I would want my mechanic to ask me before replacing an axle on my car, but I would not examine the wrench he chooses to perform the maintenance. Pat
On Oct 7, 2007, at 11:42 AM, Pat Maddox wrote:> On 10/6/07, Scott Taylor <mailing_lists at railsnewbie.com> wrote: >> >> On Oct 7, 2007, at 1:47 AM, Pat Maddox wrote: >> >>> On 10/6/07, Scott Taylor <mailing_lists at railsnewbie.com> wrote: >>>> >>>> On Oct 7, 2007, at 12:31 AM, Chad Humphries wrote: >>>> >>>>> Scott, >>>>> >>>>> I don''t really have a lot to contribute on how to make it faster, >>>>> other than to outline what we''ve been doing on our projects. >>>>> >>>>> On one of our current projects we have the following 2570 examples >>>>> that run in ~70 seconds on our pairing stations (mac minis, 1.83 >>>> >>>> I assume your "pairing stations" are two separate mac-minis, in >>>> which >>>> you practice pair programming? Or is this a cluster of two mac- >>>> minis? >>>> >>>> But this sounds great - 70 seconds for 2500 specs. How many of >>>> those >>>> are model specs (that hit the database)? >>>> >>>>> c2d). In general across our various machines is at or a little >>>>> more >>>>> than a minute for specs for controllers, models, helpers, lib, >>>>> views, >>>>> and plugins. Our Story suite takes longer, but it''s still under >>>>> development so I don''t really count it at this point. We have >>>>> Ruby >>>>> 1.8.6 installed from MacPorts on all machines, as well as MySQL 5. >>>>> (current as of a month ago) from macports. >>>>> >>>>> We make good use of mocking and stubbing through our controller >>>>> tests, and little use of fixtures. We primarily use the >>>>> spec_attribute_helper (or factory method) as Luke Redpath and Dan >>>>> Manges have outlined in their respective blog articles. I''ve >>>>> been >>>>> looking at deep test, or possible spec_distributed as a way to >>>>> speed >>>>> things up more. Our main issue is our precommit task (rake >>>>> cruise >>>> >>>> The factory method (or attribute_helper) still hits the >>>> database. I >>>> don''t see it as any sort of performance gain. In fact, I''ve even >>>> developed a plugin around the Factory idea, and it was only when I >>>> started using it in all of my tests that the speed really >>>> started to >>>> affect me (I was using mocking/stubbing, with much frustration >>>> prior >>>> to that point). But to me it''s pretty clear the plugin (or the >>>> factory) is not the problem - the hit is the database. DHH saw >>>> this >>>> hit, and since they were using fixtures,he found that creating the >>>> fixtures, and then wrapping each test in a transaction was a huge >>>> performance gain. I wonder if the same would be true with setups/ >>>> before(:each)... >>>> >>>> The obvious thing to do to solve the performance problem is to >>>> remove >>>> the hit to the database. The question is: At what level of >>>> abstraction should this be done? The one camp (which would include >>>> fellows like Jay Fields), would mock/stub everything they don''t >>>> write. For me, I see testing as more than testing - it''s the >>>> documentation to my code which never lies to me (this documentation >>>> is so good, that I can give it to my boss, who is not a >>>> programmer). >>> >>> I think you''d be far better served by user stories in this case. >>> The >>> user story is the tool we use to communicate with non-geeks, be it >>> customers or our boss. Low-level specs fall squarely in the >>> developer >>> realm, as far as I''m concerned. They''re easy to read because we >>> don''t >>> like to torture ourselves. I think that they''re fantastic for >>> communicating intent with other developers, but that''s where it >>> ends. >>> >>> So, when you move documentation as a goal from specs over to >>> stories, >>> that frees specs to take advantage of techniques like mocks to help >>> with your design. Documentation is a pleasant side effect of >>> BDD, not >>> a goal. >> >> Documentation is certainly a goal of BDD (Just not necessarily for >> the business user - as you pointed out). I don''t write my specs to >> be understandable by the business user, although it just often >> happens that they are (at least in this rails context, since even >> most of the fields of the database are more-or-less user facing). >> The point that I was trying to hammer home is that mocking/stubbing >> adds a lot of extra noise to a rails test, where it doesn''t in the >> other projects that I''ve worked on. > > Well, one of the main parts of programming is making tradeoffs. > You''re probably not going to run 2500 tests that hit the db in under > 30 seconds. So you either don''t hit the db or you don''t run all the > tests all the time. You can certainly change autotest to not > automatically run the full suite after you go red->green. Only have > it run the specs for the files you changed. > > Break your specs into fast and slow groups. Run the fast ones > continuously and the slow ones left often. Work on moving one or two > slow specs into the fast group every day. You''ll probably discover > some techniques that you can share with all of us :)My current thought is this: 1. Use sqlite in memory database when developing (but use the CI server to run the specs against mysql) 2. Start breaking up the spec files into more managable groups. Right now 100 of our 500 specs are in one file (the user_spec.rb). It''s a social networking app, so it makes sense that most of the specs would be centered around the user. But the description blocks should show some sort of logical grouping, which can then be split up into different spec files (user_with_items_spec.rb, user_with_comments_spec.rb, and so on). 3. Use autotest (or a custom tool) to rerun this one file that I''m changing. It''s pretty obvious that the mapping to autotest is flawed (Dave Astels has talked about the flaws of the one-to-one mapping of spec file to lib file (or app) several times). Maybe some sort of tool which could inflect that any change to the user_with_items_spec.rb should rerun with a change either to the User or to the Item model, as well as the spec file.> > As far as mocks adding a lot of extra noise, I find in general that > when mocks are painful for me it''s because my code is too tightly > coupled. Just like my body tells me something is wrong through pain, > so does my code. >Uhum...I don''t think It''s my code that''s too tightly coupled (*clears throat* rails). Seriously, though, I tend to agree with you - but here I don''t have an option of rewriting rails test-first to be more decoupled.> >> But whether or not you call the specs "stories" or "integration >> specs" (it is, after all, all a continuum of what is considered an >> "integration" test), I still have no idea how to run them any faster. > > Well, I don''t want to beat this to death, but I think it''s dangerous > to look at specs and stories as simply different degrees of the same > concept. The fundamental difference is that you should not change a > story without consulting a customer, and you should not have to > consult a customer to change a spec. Stories represent assumptions > and business value. Assumptions are learned through conversations > with the customer, and only the customer can define business value. > Specs are merely a tool to help us generate that business value, and > velocity goes way down if we have to worry that changing some specs > will impact the customer in some way. Worse, if we don''t worry about > it, we may end up changing some assumptions around and thus deliver > the wrong product.I agree with what you''ve said. But with rails (and a fat model), the specs for the model almost sound like a user story (albeit, not a full stack one). Here is a tiny part of the specdoc which I sent to my boss: The User - should only find the one item which is not ignored, if there are two items in the queue, and one is ignored - should only find one item in the review queue, if there are two items in the queue, but only one is ignored - should see all items in the review queue, if none have yet been ignored (and two are in the queue) - should not be allowed to ignore items from review queue if he has already reviewed 20 items - should raise an error if a nil item is given to be skipped - should raise an error if no item is given to be skipped - should not skip the item if the user is not allowed to skip the item Obviously you can see I didn''t write these things for him - (the ones at the end prove it. A "nil item" doesn''t mean anything to my boss. But the other ones clearly show business value. Once again, (some of these) specs just happen to be things which my boss can read. I wrote those specs without mocking and stubbing, but they could as just as well been mocked/stubbed to death. Scott
> > The factory method (or attribute_helper) still hits the database. I > don''t see it as any sort of performance gain. In fact, I''ve even > developed a plugin around the Factory idea, and it was only when I > started using it in all of my tests that the speed really started to > affect me (I was using mocking/stubbing, with much frustration prior > to that point). But to me it''s pretty clear the plugin (or the > factory) is not the problem - the hit is the database. DHH saw this > hit, and since they were using fixtures,he found that creating the > fixtures, and then wrapping each test in a transaction was a huge > performance gain. I wonder if the same would be true with setups/ > before(:each)... > >Scott, Where is this factory plugin you speak of? Thanks, Ben
On Oct 7, 2007, at 3:01 PM, Ben Mabey wrote:> >> >> The factory method (or attribute_helper) still hits the database. I >> don''t see it as any sort of performance gain. In fact, I''ve even >> developed a plugin around the Factory idea, and it was only when I >> started using it in all of my tests that the speed really started to >> affect me (I was using mocking/stubbing, with much frustration prior >> to that point). But to me it''s pretty clear the plugin (or the >> factory) is not the problem - the hit is the database. DHH saw this >> hit, and since they were using fixtures,he found that creating the >> fixtures, and then wrapping each test in a transaction was a huge >> performance gain. I wonder if the same would be true with setups/ >> before(:each)... >> >> > > Scott, > > Where is this factory plugin you speak of?http://replacefixtures.rubyforge.org/ There is also a screencast at the top of that page. I would highly advise you watch it. Scott