rspec users - Jun 2009 - Testing with activerecord at ThoughtWorks (Stubbing vs real DB)

I am reading the post by Fowler, ''Ruby at ThoughtWorks''.
http://martinfowler.com/articl/rubyAtThoughtWorks.html#WasRubyTheRightChoice

He talks about testing with activerecord:

"Right at the beginning of our use of Ruby, there was a debate on how
best to organize testing in the presence of the Active Record database
layer in Rails. The basic problem is that most of the time,
performance of enterprise applications is dominated by database
access. We''ve found that by using a Test Double we can greatly speed
up our tests. Having fast tests is crucial to our test-intensive
development process. Kent Beck recommends a basic commit build of
under ten minutes. Most of our projects manage this these days, and
using a database double is a vital part of achieving it.

The problem with Active Record is that by combining database access
code with business logic, it''s rather harder to create a database
double. The Mingle team''s reaction to this was to accept that Rails
binds the database tightly and thus run all the commit tests against a
real database.

The contrary view was advocated most firmly by the Atlanta and Jersey
teams. Ruby has a powerful feature that allows you to redefine methods
at run-time. You can use this to take an active record class, and
redefine the the database access methods in that class as stubs. The
team started the gem unitrecord to help with this.

In the three years, we''ve not seen a generally accepted victor in this
debate. The Mingle team run a couple of thousand tests against a real
postgres database in around 8 minutes. (They parallelize the tests to
make use of multiple cores.) The Atlanta and Jersey teams consider it
valuable that their commit test runs in 2 minutes with stubs versus 8
minutes without. The trade-off is the simplicity of the direct
database tests versus the faster commit build of the stubbed tests.

While both teams are broadly happy with their positions in this
debate, the use of stubbing has led to another issue for the Atlanta/
Jersey teams. As the teams became familiar with using method stubbing,
they used it more and more - falling into the inevitable over-usage
where unit tests would stub out every method other than the one being
tested. The problem here, as often with using doubles, is brittle
tests. As you change the behavior of the application, you also have to
change lots of doubles that are mimicking the old behavior. This over-
usage has led both teams to move away from stubbed unit tests and to
use more rails-style functional tests with direct database access."

What do you think of the two tactics?
Also, What does it mean and where can I see examples of "running all
the commit tests against a real database" ?

Can I find it in "the rspec book" ?

Thanks!

oren wrote:
> I am reading the post by Fowler, ''Ruby at ThoughtWorks''.
>
http://martinfowler.com/articl/rubyAtThoughtWorks.html#WasRubyTheRightChoice
>
> He talks about testing with activerecord:
>
> "Right at the beginning of our use of Ruby, there was a debate on how
> best to organize testing in the presence of the Active Record database
> layer in Rails. The basic problem is that most of the time,
> performance of enterprise applications is dominated by database
> access. We''ve found that by using a Test Double we can greatly
speed
> up our tests. Having fast tests is crucial to our test-intensive
> development process. Kent Beck recommends a basic commit build of
> under ten minutes. Most of our projects manage this these days, and
> using a database double is a vital part of achieving it.
>
> The problem with Active Record is that by combining database access
> code with business logic, it''s rather harder to create a database
> double. The Mingle team''s reaction to this was to accept that
Rails
> binds the database tightly and thus run all the commit tests against a
> real database.
>
> The contrary view was advocated most firmly by the Atlanta and Jersey
> teams. Ruby has a powerful feature that allows you to redefine methods
> at run-time. You can use this to take an active record class, and
> redefine the the database access methods in that class as stubs. The
> team started the gem unitrecord to help with this.
>
> In the three years, we''ve not seen a generally accepted victor in
this
> debate. The Mingle team run a couple of thousand tests against a real
> postgres database in around 8 minutes. (They parallelize the tests to
> make use of multiple cores.) The Atlanta and Jersey teams consider it
> valuable that their commit test runs in 2 minutes with stubs versus 8
> minutes without. The trade-off is the simplicity of the direct
> database tests versus the faster commit build of the stubbed tests.
>
> While both teams are broadly happy with their positions in this
> debate, the use of stubbing has led to another issue for the Atlanta/
> Jersey teams. As the teams became familiar with using method stubbing,
> they used it more and more - falling into the inevitable over-usage
> where unit tests would stub out every method other than the one being
> tested. The problem here, as often with using doubles, is brittle
> tests. As you change the behavior of the application, you also have to
> change lots of doubles that are mimicking the old behavior. This over-
> usage has led both teams to move away from stubbed unit tests and to
> use more rails-style functional tests with direct database access."
>
> What do you think of the two tactics?
>   
I have done both on past projects.  I did not use any of the TW gems to 
do it.  I used Avdi Grimm''s nice NullDB plugin to do it.  IMO it is a 
much better and cleaner approach than the ones talked about in the 
article.  I gave a lightning talk about the tradeoffs of both approaches 
and how I used NullDB in my project.  Here are the slides for the talk:
http://www.slideshare.net/bmabey/disconnecting-the-database-with-activerecord

The slides go through the benefits and tradeoffs of both approaches.  
Let me know if you want me to elaborate on anything else.  I should 
point out that another, but less extreme, solution to stubbing out the 
DB is to use an in-memory DB (like sqllite) to get a speed boost.  With 
this approach you may still need to switch to your real DB adapter for 
DB specific SQL but it all the basic AR stuff with work just
fine.
> Also, What does it mean and where can I see examples of "running all
> the commit tests against a real database" ?
>   
Sounds like they have configured there project to to allow you to hot 
swap which strategy to use.  So, for development you can use the 
stubbing approach and get faster feedback and then swap in the real 
database on the CI server to make sure there aren''t any surprises when 
it is actually executed against the DB.  This is a good approach and can 
be done quite easily with NullDB.
> Can I find it in "the rspec book" ?
>   
I don''t think so.  To be honest, I don''t think it is done that
much.  I
think most teams just let there model tests hit the DB. (And thus are 
not "real" unit tests by some people''s definition....)


-Ben

Awesome presentation, thanks for clarifying it Ben.