rspec users - Feb 2007 - We can''t 100% remove our unit tests from the database, can we?

I hope this isn''t too rambly.  This is sort of a brain dump of a
subject I''ve been thinking about for months as I''ve used
RSpec.

Let''s say we''ve got a simple query method, that will find all
the
users in the DB older than 18.  Our model could look like

class User < ActiveRecord::Base
  def self.find_older_than(age)
    find :all, :conditions => ["age > ?", age]
  end
end

What would our spec look like?

specify "should find all users over the given age" do
  User.should_receive(:find).with(:all, :conditions => ["age >
?", 18])
  User.find_older_than 18
end

I don''t know about you, but to me that sucks.  There is no TDD rhythm
there.  We write a failing spec, but it''s a fairly complex spec.  On
top of that we''ve basically implemented the method from within the
spec.  That''s not TDD.

We also can''t refactor.  Let''s say that at some point we
decided to
change the method to

def User.find_older_than(age)
  find(:all).select {|u| u.age > age }
end

Our spec would break, even though the semantics of
User.find_older_than hasn''t changed.

Okay, so it''s just a crappy spec.  But how do we change it?  Lots of
people suggest stubbing out DB calls.  There''s no sense in testing the
same thing over and over - namely that you have a connection and your
ORM tool is working correctly.

I''m beginning to think that most of the time, you don''t want
to use a
test DB at all.  When you''re testing an AR model, just use an
in-memory record and stub any associations you need.  But when you do
do something that interacts with the database - like doing a custom
find - you need to use the database to make sure that you''re getting
the right results.  Whether you use fixtures or create some records in
the setup method, you have to actually hit the db.

Let me know what you think.  I''m completely open to the idea that
I''ve
missed something pathetically obvious.

Pat

On 22-Feb-07, at 12:41 PM, Pat Maddox wrote:
> I hope this isn''t too rambly.  This is sort of a brain dump of a
> subject I''ve been thinking about for months as I''ve used
RSpec.
>
> Let''s say we''ve got a simple query method, that will find
all the
> users in the DB older than 18.  Our model could look like
The way you describe is the way to do this.  It sucks, but it''s not  
our fault, it''s ActiveRecord''s fault.  AcitveRecord has extra
suckage, baked in for free.  You don''t even have to pay extra.

-Steven

On 2/22/07, Pat Maddox <pergesu at gmail.com>
wrote:
> I hope this isn''t too rambly.  This is sort of a brain dump of a
> subject I''ve been thinking about for months as I''ve used
RSpec.
>
> Let''s say we''ve got a simple query method, that will find
all the
> users in the DB older than 18.  Our model could look like
>
> class User < ActiveRecord::Base
>   def self.find_older_than(age)
>     find :all, :conditions => ["age > ?", age]
>   end
> end
>
> What would our spec look like?
>
> specify "should find all users over the given age" do
>   User.should_receive(:find).with(:all, :conditions => ["age >
?", 18])
>   User.find_older_than 18
> end
>
> I don''t know about you, but to me that sucks.  There is no TDD
rhythm
> there.  We write a failing spec, but it''s a fairly complex spec. 
On
> top of that we''ve basically implemented the method from within the
> spec.  That''s not TDD.
>
> We also can''t refactor.  Let''s say that at some point we
decided to
> change the method to
>
> def User.find_older_than(age)
>   find(:all).select {|u| u.age > age }
> end
>
> Our spec would break, even though the semantics of
> User.find_older_than hasn''t changed.
>
> Okay, so it''s just a crappy spec.  But how do we change it?  Lots
of
> people suggest stubbing out DB calls.  There''s no sense in testing
the
> same thing over and over - namely that you have a connection and your
> ORM tool is working correctly.
>
> I''m beginning to think that most of the time, you don''t
want to use a
> test DB at all.  When you''re testing an AR model, just use an
> in-memory record and stub any associations you need.  But when you do
> do something that interacts with the database - like doing a custom
> find - you need to use the database to make sure that you''re
getting
> the right results.  Whether you use fixtures or create some records in
> the setup method, you have to actually hit the db.
>
> Let me know what you think.  I''m completely open to the idea that
I''ve
> missed something pathetically obvious.
>
Only mock AR when you''re speccing code that *interacts* with AR
(controllers and views).

Don''t mock AR in specs for AR subclasses - let them go against a real
database.

Don''t mock APIs you don''t own.
See http://www.jmock.org/oopsla2004.pdf for more on this

HTH,
Aslak
> Pat
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

On 2/22/07, Pat Maddox <pergesu at gmail.com>
wrote:
>
>
> specify "should find all users over the given age" do
>   User.should_receive(:find).with(:all, :conditions => ["age >
?", 18])
>   User.find_older_than 18
> end

I don''t have a good suggestion here, but an observation is that the
spec is
too fine-grained and brittle, which, as you mention below, inhibits
refactoring and TDD flow, etc.

I don''t know about you, but to me that sucks.  There is no TDD
rhythm
> there.  We write a failing spec, but it''s a fairly complex spec. 
On
> top of that we''ve basically implemented the method from within the
> spec.  That''s not TDD.
>
> We also can''t refactor.  Let''s say that at some point we
decided to
> change the method to
>
> def User.find_older_than(age)
>   find(:all).select {|u| u.age > age }
> end
>
> Our spec would break, even though the semantics of
> User.find_older_than hasn''t changed.

I have experienced similar troubles as you have in the past, and the usage
of mock-based testing in these scenarios still hasn''t fully clicked for
me
either.  I know something''s not right, but I can''t put my
finger on what it
is.  I suspect it''s that the APIs you''re working with are too
data-intensive
and not message-intensive (i.e., "tell don''t ask"), which is
where mock
testing really shines.  I''ll defer to this post from Steve Freeman
(there
are probably other entries too that are similar), and the quote that
apparently originates from Joe Walnes, "Don''t mock types you
don''t own".

http://stevef.truemesh.com/archives/000194.html

I''m beginning to think that most of the time, you don''t want
to use a
> test DB at all.  When you''re testing an AR model, just use an
> in-memory record and stub any associations you need.  But when you do
> do something that interacts with the database - like doing a custom
> find - you need to use the database to make sure that you''re
getting
> the right results.  Whether you use fixtures or create some records in
> the setup method, you have to actually hit the db.

I don''t have a better answer, so maybe your instinct is right here --
don''t
fight with brittle mocks, if the database works for you and doesn''t
slow you
down too much.

I would think Dave, David and Aslak would have good input on this topic, and
hope they have a moment to share their experiences.

Cheers,
/Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/rspec-users/attachments/20070222/0dfbdffa/attachment-0001.html

On 2/22/07, Pat Maddox <pergesu at gmail.com>
wrote:
> I hope this isn''t too rambly.  This is sort of a brain dump of a
> subject I''ve been thinking about for months as I''ve used
RSpec.
>
> Let''s say we''ve got a simple query method, that will find
all the
> users in the DB older than 18.  Our model could look like
>
> class User < ActiveRecord::Base
>   def self.find_older_than(age)
>     find :all, :conditions => ["age > ?", age]
>   end
> end
>
> What would our spec look like?
>
> specify "should find all users over the given age" do
>   User.should_receive(:find).with(:all, :conditions => ["age >
?", 18])
>   User.find_older_than 18
> end
>
> I don''t know about you, but to me that sucks.  There is no TDD
rhythm
> there.  We write a failing spec, but it''s a fairly complex spec. 
On
> top of that we''ve basically implemented the method from within the
> spec.  That''s not TDD.
>
> We also can''t refactor.  Let''s say that at some point we
decided to
> change the method to
>
> def User.find_older_than(age)
>   find(:all).select {|u| u.age > age }
> end
>
> Our spec would break, even though the semantics of
> User.find_older_than hasn''t changed.
>
> Okay, so it''s just a crappy spec.  But how do we change it?  Lots
of
> people suggest stubbing out DB calls.  There''s no sense in testing
the
> same thing over and over - namely that you have a connection and your
> ORM tool is working correctly.
>
> I''m beginning to think that most of the time, you don''t
want to use a
> test DB at all.  When you''re testing an AR model, just use an
> in-memory record and stub any associations you need.  But when you do
> do something that interacts with the database - like doing a custom
> find - you need to use the database to make sure that you''re
getting
> the right results.  Whether you use fixtures or create some records in
> the setup method, you have to actually hit the db.
>
> Let me know what you think.  I''m completely open to the idea that
I''ve
> missed something pathetically obvious.
This is a tricky thing in Rails. Michael Feathers sets some good
boundaries around unit testing in a blog entry:
http://www.artima.com/weblogs/viewpost.jsp?thread=126923. You''ll note
that the first thing it says is "A test is not a unit test if: it
talks to a database."

So along comes Rails and Rails says "unit tests test models, which
talk to the database". So we have some confusion to get past right
away.

So let''s say we agree that unit tests should never talk to a database.
If we agree on that definition, then we can agree that testing that
custom finds work correctly can not really be a unit test. So then the
question becomes not whether we should use the database for those
tests, but rather what we call those tests and where they go in our
test or spec directories.

One option is to leave them under test/units or spec/models. If the
time to run the suite isn''t impeding your work, then this might be OK.

Another option is to separate out model specs that interact w/ the DB
from those that don''t in two separate directories and set up rake
tasks to run all the non-db stuff or include the db stuff at your
whim.

Keep in mind that the important thing about all of this is that the
quicker you can run specs the more often you will, and tighter the
feedback loop, and the less time you''ll spend debugging code that you
just wrote!

Hope that''s helpful.

Cheers,
David
>
> Pat
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

On 2/22/07, aslak hellesoy <aslak.hellesoy at gmail.com>
wrote:
> On 2/22/07, Pat Maddox <pergesu at gmail.com> wrote:
> > I hope this isn''t too rambly.  This is sort of a brain dump
of a
> > subject I''ve been thinking about for months as I''ve
used RSpec.
> >
> > Let''s say we''ve got a simple query method, that will
find all the
> > users in the DB older than 18.  Our model could look like
> >
> > class User < ActiveRecord::Base
> >   def self.find_older_than(age)
> >     find :all, :conditions => ["age > ?", age]
> >   end
> > end
> >
> > What would our spec look like?
> >
> > specify "should find all users over the given age" do
> >   User.should_receive(:find).with(:all, :conditions => ["age
> ?", 18])
> >   User.find_older_than 18
> > end
> >
> > I don''t know about you, but to me that sucks.  There is no
TDD rhythm
> > there.  We write a failing spec, but it''s a fairly complex
spec.  On
> > top of that we''ve basically implemented the method from
within the
> > spec.  That''s not TDD.
> >
> > We also can''t refactor.  Let''s say that at some
point we decided to
> > change the method to
> >
> > def User.find_older_than(age)
> >   find(:all).select {|u| u.age > age }
> > end
> >
> > Our spec would break, even though the semantics of
> > User.find_older_than hasn''t changed.
> >
> > Okay, so it''s just a crappy spec.  But how do we change it? 
Lots of
> > people suggest stubbing out DB calls.  There''s no sense in
testing the
> > same thing over and over - namely that you have a connection and your
> > ORM tool is working correctly.
> >
> > I''m beginning to think that most of the time, you
don''t want to use a
> > test DB at all.  When you''re testing an AR model, just use an
> > in-memory record and stub any associations you need.  But when you do
> > do something that interacts with the database - like doing a custom
> > find - you need to use the database to make sure that you''re
getting
> > the right results.  Whether you use fixtures or create some records in
> > the setup method, you have to actually hit the db.
> >
> > Let me know what you think.  I''m completely open to the idea
that I''ve
> > missed something pathetically obvious.
> >
>
> Only mock AR when you''re speccing code that *interacts* with AR
> (controllers and views).
>
> Don''t mock AR in specs for AR subclasses - let them go against a
real database.
>
> Don''t mock APIs you don''t own.
> See http://www.jmock.org/oopsla2004.pdf for more on this
>
> HTH,
> Aslak
I agree w/ Aslak in principle.

The problem is that in an effort to make life seemingly easier for you
and me, AR forces us to couple our business logic with our persistence
logic, which puts us in a catch 22 vis a vis pre-Rails thinking about
testing and mocking. We can either test these things separately, which
requires that we mock code we didn''t write, or we adhere to only
mocking code we don''t own, which requires us to test two concepts
together.

Alternatively, we could add concepts to our rails apps to decouple
these things - non-AR model classes that delegate to AR backed models
- ModelFactories so we don''t call class methods on the model classes,
etc. My instinct is that it would be more complicated that its worth.

Has anyone experimented w/ anything like that?
>
> > Pat
> > _______________________________________________
> > rspec-users mailing list
> > rspec-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/rspec-users
> >
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

Three replies for the price of one...

I was about to reply to this:

On 22 Feb 2007, at 22:08, Nick Sieger wrote:
> Joe Walnes, "Don''t mock types you don''t own"
Idiot question... doesn''t this preclude stubbing methods like  
ActiveRecord::Base.find?  Or have I misunderstood?


Then I saw this, which kinda answers it:

On 22 Feb 2007, at 22:32, David Chelimsky wrote:
> The problem is that in an effort to make life seemingly easier for you
> and me, AR forces us to couple our business logic with our persistence
> logic, which puts us in a catch 22 vis a vis pre-Rails thinking about
> testing and mocking. We can either test these things separately, which
> requires that we mock code we didn''t write, or we adhere to only
> mocking code we don''t own, which requires us to test two concepts
> together.
>
> Alternatively, we could add concepts to our rails apps to decouple
> these things - non-AR model classes that delegate to AR backed models
> - ModelFactories so we don''t call class methods on the model
classes,
> etc. My instinct is that it would be more complicated that its worth.
>
> Has anyone experimented w/ anything like that?
Not tried that, but I''ve been thinking for a while.  Is it time  
ActiveRecord became a data mapper, that defaulted to model -> table  
mapping?  I mean something like,

   class Person
     persistence_mapper :auto

     # usual stuff
     validates_presence_of :name
   end

creates dynamically:

   PersonMapper < ActiveRecord::Base
   end

and if you want more control,

   class FinanceProposal
     # make it call ComplexFinanceProposalMapper.new or whatever
     persistence_mapper :complex_finance_proposal_mapper
   end

   class FinanceProposalMapper
     # all the overcomplicated stuff you need to do to save finance  
proposals,
     # possibly using ActiveRecord::Base subclasses to save to tables
   end

This way you get separation of validation so you can test it  
independently, without losing any of the out-of-the-box simplicity.


On 22 Feb 2007, at 21:33, Steven Baker wrote:
> The way you describe is the way to do this.  It sucks, but it''s
not
> our fault, it''s ActiveRecord''s fault.  AcitveRecord has
extra
> suckage, baked in for free.  You don''t even have to pay extra.
It seems a bit unfair to throw too much criticism at ActiveRecord.   
After all, it was intended to follow the ActiveRecord pattern for  
simplicity (presumably convention over configuration).  On the other  
hand, seeing as testability is pretty essential for agile  
development, which is one of the big selling points of Rails, it  
sounds like something has to be done.  Might make more sense to take  
this to the Rails list, but I seem to spend all my time here lately.

Has anyone got any experience of Og?  I looked at the source recently  
but haven''t tried to use it.  How does it compare test-wise?

Ashley

On 2/23/07, Ashley Moran <work at ashleymoran.me.uk>
wrote:
> Three replies for the price of one...
>
> I was about to reply to this:
>
> On 22 Feb 2007, at 22:08, Nick Sieger wrote:
> > Joe Walnes, "Don''t mock types you don''t
own"
>
This is a guideline, not an absolute truth.
> Idiot question... doesn''t this preclude stubbing methods like
> ActiveRecord::Base.find?  Or have I misunderstood?
>
It''s ok(ish) to stub/mock at the "surface" of an API. But
once you
start stubbing/mocking methods that the "surface" API calls (i.e. what
AR.find calls) then you''re in trouble land.
>
> Then I saw this, which kinda answers it:
>
> On 22 Feb 2007, at 22:32, David Chelimsky wrote:
>
> > The problem is that in an effort to make life seemingly easier for you
> > and me, AR forces us to couple our business logic with our persistence
> > logic, which puts us in a catch 22 vis a vis pre-Rails thinking about
> > testing and mocking. We can either test these things separately, which
> > requires that we mock code we didn''t write, or we adhere to
only
> > mocking code we don''t own, which requires us to test two
concepts
> > together.
> >
> > Alternatively, we could add concepts to our rails apps to decouple
> > these things - non-AR model classes that delegate to AR backed models
> > - ModelFactories so we don''t call class methods on the model
classes,
> > etc. My instinct is that it would be more complicated that its worth.
> >
> > Has anyone experimented w/ anything like that?
>
> Not tried that, but I''ve been thinking for a while.  Is it time
> ActiveRecord became a data mapper, that defaulted to model -> table
> mapping?  I mean something like,
>
>    class Person
>      persistence_mapper :auto
>
>      # usual stuff
>      validates_presence_of :name
>    end
>
> creates dynamically:
>
>    PersonMapper < ActiveRecord::Base
>    end
>
> and if you want more control,
>
>    class FinanceProposal
>      # make it call ComplexFinanceProposalMapper.new or whatever
>      persistence_mapper :complex_finance_proposal_mapper
>    end
>
>    class FinanceProposalMapper
>      # all the overcomplicated stuff you need to do to save finance
> proposals,
>      # possibly using ActiveRecord::Base subclasses to save to tables
>    end
>
> This way you get separation of validation so you can test it
> independently, without losing any of the out-of-the-box simplicity.
>
>
> On 22 Feb 2007, at 21:33, Steven Baker wrote:
> > The way you describe is the way to do this.  It sucks, but
it''s not
> > our fault, it''s ActiveRecord''s fault.  AcitveRecord
has extra
> > suckage, baked in for free.  You don''t even have to pay
extra.
>
> It seems a bit unfair to throw too much criticism at ActiveRecord.
> After all, it was intended to follow the ActiveRecord pattern for
> simplicity (presumably convention over configuration).  On the other
> hand, seeing as testability is pretty essential for agile
> development, which is one of the big selling points of Rails, it
> sounds like something has to be done.  Might make more sense to take
> this to the Rails list, but I seem to spend all my time here lately.
>
> Has anyone got any experience of Og?  I looked at the source recently
> but haven''t tried to use it.  How does it compare test-wise?
>
> Ashley
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>