Hi all, We did a spike last week to de-couple our view and controller tests from the database, using NullDb. It didn''t go too well. I realise that this plugin isn''t part of RSpec, but I thought others on this list might have experiences to share. Here''s a summary of my colleague''s investigations:> I installed the plugin from http://github.com/jakehow/nulldb/tree/master > > Changed spec/spec_helper to set > "ActiveRecord::Base.establish_connection(:adapter => :nulldb)" by > default. > > The specs that complained of the lack of db, I included a before(all) > that changed the connection to the test database (include > NeedsDatabase - copied from Ben Mabey''s Functional module) > By default, I also included the db for all models (config.include > NeedsDatabase, :type => :model) > > Running spec spec/views with or without nulldb takes about the same > time (a couple of seconds less with nulldb). However, running "rake > spec" with nulldb takes 10 times longer!! > > Another weird thing was that three tests failed in the controllers > (venues and concerts) in a very strange way, even when I included the > NeedsDatabase in the tests that needed it. The weird bit is, when I > run: "spec spec/controllers/venues_controller_spec.rb" it doesn''t > fail. But when I run: "spec spec/controllers/users_controller_spec.rb > spec/controllers/venues_controller_spec.rb" It does fail... That is, > the user_controller test is influencing the results of the > venue_controller test! > The same weird behaviour happens in lib/sk/find > > This made me had to include NeedsDatabase for all lib and controllers > tests as well.The barrier for us was the shockingly poor performance of ''rake spec'' on the view specs - it really means we just can''t use it, and actually only barely improved the performance of the specs at all. I was disappointed that the view specs didn''t get any faster. My guess is that stub_model is the problem - as it has to do quite a bit of work to set up the attributes on the models. So, can anyone tell us what we might have been doing wrong? Or did I just have unrealistic expectations of how this might help? Matt Wynne http://blog.mattwynne.net http://www.songkick.com
On 1/12/09 7:09 AM, Matt Wynne wrote:> Hi all, > > We did a spike last week to de-couple our view and controller tests > from the database, using NullDb. It didn''t go too well. I realise that > this plugin isn''t part of RSpec, but I thought others on this list > might have experiences to share. > > Here''s a summary of my colleague''s investigations: > >> I installed the plugin from http://github.com/jakehow/nulldb/tree/master >> >> Changed spec/spec_helper to set >> "ActiveRecord::Base.establish_connection(:adapter => :nulldb)" by >> default. >> >> The specs that complained of the lack of db, I included a before(all) >> that changed the connection to the test database (include >> NeedsDatabase - copied from Ben Mabey''s Functional module) >> By default, I also included the db for all models (config.include >> NeedsDatabase, :type => :model) >> >> Running spec spec/views with or without nulldb takes about the same >> time (a couple of seconds less with nulldb). However, running "rake >> spec" with nulldb takes 10 times longer!! >> >> Another weird thing was that three tests failed in the controllers >> (venues and concerts) in a very strange way, even when I included the >> NeedsDatabase in the tests that needed it. The weird bit is, when I >> run: "spec spec/controllers/venues_controller_spec.rb" it doesn''t >> fail. But when I run: "spec spec/controllers/users_controller_spec.rb >> spec/controllers/venues_controller_spec.rb" It does fail... That is, >> the user_controller test is influencing the results of the >> venue_controller test! >> The same weird behaviour happens in lib/sk/find >> >> This made me had to include NeedsDatabase for all lib and controllers >> tests as well. > > The barrier for us was the shockingly poor performance of ''rake spec'' > on the view specs - it really means we just can''t use it, and actually > only barely improved the performance of the specs at all. > > I was disappointed that the view specs didn''t get any faster. My guess > is that stub_model is the problem - as it has to do quite a bit of > work to set up the attributes on the models. > > So, can anyone tell us what we might have been doing wrong? Or did I > just have unrealistic expectations of how this might help? > > Matt Wynne > http://blog.mattwynne.net > http://www.songkick.com > > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-usersHey Matt, I had similar experiences when I started using NullDB. I think that the reason why the overall spec suite runs slow is AR is switching from one adapter to another, and in the case of the mysql one it probably is relatively expensive to set up a new connection. Switching over to using NullDB on an existing project of any sizable size is going to be large task IMO. On my last project we started using NullDB by default and the result has been very worthwhile and noticeable. In this project DB connectivity was the exception, not the norm (even on our model specs) and so that is why I created a module that would turn on the DB temporarily. In your case the opposite approach may be needed. Going back to the problem at hand... Based on the email you are using a module like this: unless Object.const_defined?(:NeedsDatabase) share_as :NeedsDatabase do before :all do ActiveRecord::Base.establish_connection(:test) end after :all do ActiveRecord::Base.establish_connection(:adapter => :nulldb) end end end Since you are including this on all of your model specs (and more) you are incurring the setup and teardown cost of the DB connection repeatedly. Like I said, you may want to adopt the opposite policy of turning it off temporarily... You could also make the the changing smarter by only changing it when you need to.. something like: share_as :NeedsDatabase do before :all do ActiveRecord::Base.establish_connection(:test) if ActiveRecord::Base.connection.class.to_s =~ /NullDB/ end end share_as :DontNeedDatabase do before :all do ActiveRecord::Base.establish_connection(:adapter => :nulldb) unless ActiveRecord::Base.connection.class.to_s =~ /NullDB/ end end With this approach you would need to specify whether or not to use the DB for each example group... just a thought. Of course, an even better solution would be to somehow modify AR to maintain an active connection to the real DB even when it isn''t the active adapter. re: the view specs I wouldn''t expect to see too much gain here. The big win for nullDB is in models IMO. Of course, having it on the view and controller specs helps prevent accidental DB calls. re: the random controller spec failures I only guess is that you are somehow relying on the DB in your controller specs. I did this too in my controllers when I called class methods on models that would get a list from the DB... I can''t really help without seeing any specs though. I hope this helps and you can find a suitable policy on how to manage the connections/adapters. Just ask if you have any other questions. -Ben
On 12 Jan 2009, at 17:36, Ben Mabey wrote:> On 1/12/09 7:09 AM, Matt Wynne wrote: >> Hi all, >> >> We did a spike last week to de-couple our view and controller tests >> from the database, using NullDb. It didn''t go too well. I realise >> that this plugin isn''t part of RSpec, but I thought others on this >> list might have experiences to share. >> >> Here''s a summary of my colleague''s investigations: >> >>> I installed the plugin from http://github.com/jakehow/nulldb/tree/master >>> >>> Changed spec/spec_helper to set >>> "ActiveRecord::Base.establish_connection(:adapter => :nulldb)" by >>> default. >>> >>> The specs that complained of the lack of db, I included a >>> before(all) >>> that changed the connection to the test database (include >>> NeedsDatabase - copied from Ben Mabey''s Functional module) >>> By default, I also included the db for all models (config.include >>> NeedsDatabase, :type => :model) >>> >>> Running spec spec/views with or without nulldb takes about the same >>> time (a couple of seconds less with nulldb). However, running "rake >>> spec" with nulldb takes 10 times longer!! >>> >>> Another weird thing was that three tests failed in the controllers >>> (venues and concerts) in a very strange way, even when I included >>> the >>> NeedsDatabase in the tests that needed it. The weird bit is, when I >>> run: "spec spec/controllers/venues_controller_spec.rb" it doesn''t >>> fail. But when I run: "spec spec/controllers/ >>> users_controller_spec.rb >>> spec/controllers/venues_controller_spec.rb" It does fail... That is, >>> the user_controller test is influencing the results of the >>> venue_controller test! >>> The same weird behaviour happens in lib/sk/find >>> >>> This made me had to include NeedsDatabase for all lib and >>> controllers >>> tests as well. >> >> The barrier for us was the shockingly poor performance of ''rake >> spec'' on the view specs - it really means we just can''t use it, and >> actually only barely improved the performance of the specs at all. >> >> I was disappointed that the view specs didn''t get any faster. My >> guess is that stub_model is the problem - as it has to do quite a >> bit of work to set up the attributes on the models. >> >> So, can anyone tell us what we might have been doing wrong? Or did >> I just have unrealistic expectations of how this might help? >> >> Matt Wynne >> http://blog.mattwynne.net >> http://www.songkick.com >> >> _______________________________________________ >> rspec-users mailing list >> rspec-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/rspec-users > > Hey Matt, > I had similar experiences when I started using NullDB. I think that > the reason why the overall spec suite runs slow is AR is switching > from one adapter to another, and in the case of the mysql one it > probably is relatively expensive to set up a new connection. > Switching over to using NullDB on an existing project of any sizable > size is going to be large task IMO. On my last project we started > using NullDB by default and the result has been very worthwhile and > noticeable. In this project DB connectivity was the exception, not > the norm (even on our model specs) and so that is why I created a > module that would turn on the DB temporarily. In your case the > opposite approach may be needed. > > Going back to the problem at hand... Based on the email you are > using a module like this: > unless Object.const_defined?(:NeedsDatabase) > share_as :NeedsDatabase do > > before :all do > ActiveRecord::Base.establish_connection(:test) > end > > after :all do > ActiveRecord::Base.establish_connection(:adapter => :nulldb) > end > end > end > > Since you are including this on all of your model specs (and more) > you are incurring the setup and teardown cost of the DB connection > repeatedly. Like I said, you may want to adopt the opposite policy > of turning it off temporarily... You could also make the the > changing smarter by only changing it when you need to.. something > like: > > share_as :NeedsDatabase do > before :all do > ActiveRecord::Base.establish_connection(:test) if > ActiveRecord::Base.connection.class.to_s =~ /NullDB/ > end > end > share_as :DontNeedDatabase do > before :all do > ActiveRecord::Base.establish_connection(:adapter => :nulldb) > unless ActiveRecord::Base.connection.class.to_s =~ /NullDB/ > end > end > > With this approach you would need to specify whether or not to use > the DB for each example group... just a thought. Of course, an even > better solution would be to somehow modify AR to maintain an active > connection to the real DB even when it isn''t the active adapter. > > re: the view specs > I wouldn''t expect to see too much gain here. The big win for > nullDB is in models IMO. Of course, having it on the view and > controller specs helps prevent accidental DB calls. > > re: the random controller spec failures > I only guess is that you are somehow relying on the DB in your > controller specs. I did this too in my controllers when I called > class methods on models that would get a list from the DB... I can''t > really help without seeing any specs though. > > I hope this helps and you can find a suitable policy on how to > manage the connections/adapters. Just ask if you have any other > questions. > > -BenThanks Ben. I had a brief look at the AR code and it did appear to be caching the adapters so I''d assumed the changeover wasn''t expensive, but it did feel like something like that, so that''s probably it. Ho hum. No quick wins for speeding up our specs then! Matt Wynne http://blog.mattwynne.net http://www.songkick.com
On 1/12/09 1:05 PM, Matt Wynne wrote:> > On 12 Jan 2009, at 17:36, Ben Mabey wrote: > >> On 1/12/09 7:09 AM, Matt Wynne wrote: >>> Hi all, >>> >>> We did a spike last week to de-couple our view and controller tests >>> from the database, using NullDb. It didn''t go too well. I realise >>> that this plugin isn''t part of RSpec, but I thought others on this >>> list might have experiences to share. >>> >>> Here''s a summary of my colleague''s investigations: >>> >>>> I installed the plugin from >>>> http://github.com/jakehow/nulldb/tree/master >>>> >>>> Changed spec/spec_helper to set >>>> "ActiveRecord::Base.establish_connection(:adapter => :nulldb)" by >>>> default. >>>> >>>> The specs that complained of the lack of db, I included a before(all) >>>> that changed the connection to the test database (include >>>> NeedsDatabase - copied from Ben Mabey''s Functional module) >>>> By default, I also included the db for all models (config.include >>>> NeedsDatabase, :type => :model) >>>> >>>> Running spec spec/views with or without nulldb takes about the same >>>> time (a couple of seconds less with nulldb). However, running "rake >>>> spec" with nulldb takes 10 times longer!! >>>> >>>> Another weird thing was that three tests failed in the controllers >>>> (venues and concerts) in a very strange way, even when I included the >>>> NeedsDatabase in the tests that needed it. The weird bit is, when I >>>> run: "spec spec/controllers/venues_controller_spec.rb" it doesn''t >>>> fail. But when I run: "spec spec/controllers/users_controller_spec.rb >>>> spec/controllers/venues_controller_spec.rb" It does fail... That is, >>>> the user_controller test is influencing the results of the >>>> venue_controller test! >>>> The same weird behaviour happens in lib/sk/find >>>> >>>> This made me had to include NeedsDatabase for all lib and controllers >>>> tests as well. >>> >>> The barrier for us was the shockingly poor performance of ''rake >>> spec'' on the view specs - it really means we just can''t use it, and >>> actually only barely improved the performance of the specs at all. >>> >>> I was disappointed that the view specs didn''t get any faster. My >>> guess is that stub_model is the problem - as it has to do quite a >>> bit of work to set up the attributes on the models. >>> >>> So, can anyone tell us what we might have been doing wrong? Or did I >>> just have unrealistic expectations of how this might help? >>> >>> Matt Wynne >>> http://blog.mattwynne.net >>> http://www.songkick.com >>> >>> _______________________________________________ >>> rspec-users mailing list >>> rspec-users at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/rspec-users >> >> Hey Matt, >> I had similar experiences when I started using NullDB. I think that >> the reason why the overall spec suite runs slow is AR is switching >> from one adapter to another, and in the case of the mysql one it >> probably is relatively expensive to set up a new connection. >> Switching over to using NullDB on an existing project of any sizable >> size is going to be large task IMO. On my last project we started >> using NullDB by default and the result has been very worthwhile and >> noticeable. In this project DB connectivity was the exception, not >> the norm (even on our model specs) and so that is why I created a >> module that would turn on the DB temporarily. In your case the >> opposite approach may be needed. >> >> Going back to the problem at hand... Based on the email you are using >> a module like this: >> unless Object.const_defined?(:NeedsDatabase) >> share_as :NeedsDatabase do >> >> before :all do >> ActiveRecord::Base.establish_connection(:test) >> end >> >> after :all do >> ActiveRecord::Base.establish_connection(:adapter => :nulldb) >> end >> end >> end >> >> Since you are including this on all of your model specs (and more) >> you are incurring the setup and teardown cost of the DB connection >> repeatedly. Like I said, you may want to adopt the opposite policy >> of turning it off temporarily... You could also make the the changing >> smarter by only changing it when you need to.. something like: >> >> share_as :NeedsDatabase do >> before :all do >> ActiveRecord::Base.establish_connection(:test) if >> ActiveRecord::Base.connection.class.to_s =~ /NullDB/ >> end >> end >> share_as :DontNeedDatabase do >> before :all do >> ActiveRecord::Base.establish_connection(:adapter => :nulldb) >> unless ActiveRecord::Base.connection.class.to_s =~ /NullDB/ >> end >> end >> >> With this approach you would need to specify whether or not to use >> the DB for each example group... just a thought. Of course, an even >> better solution would be to somehow modify AR to maintain an active >> connection to the real DB even when it isn''t the active adapter. >> >> re: the view specs >> I wouldn''t expect to see too much gain here. The big win for nullDB >> is in models IMO. Of course, having it on the view and controller >> specs helps prevent accidental DB calls. >> >> re: the random controller spec failures >> I only guess is that you are somehow relying on the DB in your >> controller specs. I did this too in my controllers when I called >> class methods on models that would get a list from the DB... I can''t >> really help without seeing any specs though. >> >> I hope this helps and you can find a suitable policy on how to manage >> the connections/adapters. Just ask if you have any other questions. >> >> -Ben > > Thanks Ben. I had a brief look at the AR code and it did appear to be > caching the adapters so I''d assumed the changeover wasn''t expensive, > but it did feel like something like that, so that''s probably it.Oh really? Hmm.. well, I never did any real testing to verify my assumption about the the changeover. I just noticed that suites that had to switch adapters a lot ran very slowly.> > Ho hum. No quick wins for speeding up our specs then!Unfortunately not. :(> > > Matt Wynne > http://blog.mattwynne.net > http://www.songkick.com > > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users