rspec users - May 2007 - Mocking, changing method interfaces and integration tests

Suppose we have a method ''foo'' which internally uses another
method
''bar''.

Being good BDDers we mock out the ''bar'' method. After all, we
only
want to spec the ''foo'' method - actually running the
''bar'' method
means slower, less maintainable and brittler specs. That''s why we <3
mocking, right?

We rely on the fact that ''bar'' is being adequately tested
somewhere
else, by whomever wrote it.  They can change their implementation of  
''bar'' and we can change our implementation of
''foo'' and as long as
the existing tests keep passing everyone is happy.

But what happens if ''bar'' needs to be refactored in a way that
changes the interface? rSpec does succeed in informing bar''s author  
that he is breaking someone''s expectations about the
''bar'' function -
however, he really has no idea whose expectations he is breaking.  
(i.e. rSpec doesn''t give us any clues about what exactly has to  
follow our change in interface- it just tells us something may be  
broken).

Changing the interface of bar means going over the code the hard way,  
getting in touch with everyone that uses the ''bar'' function
and
making sure everything still works. Not very agile! It makes changing  
interfaces a very very expensive process. Of course, such changes are  
never really cheap - but because of mocks it becomes really REALLY  
expensive.

So what are our options?

1) Don''t use mocks. Not using mocks would of course catch any such  
problems right away. Yes mocks have benefits, but if we plan on  
refactoring often, the above scenario may just be too high a price to  
pay.

2) Don''t change interfaces (too often) - i.e. specs should be treated  
as immutable. If it''s such an expensive process just make sure it  
doesn''t have to happen that often ;)  I think if you have an all-vet,  
all-star team of programmers this could work out just fine - but even  
then, having change be expensive just isn''t a good thing. Sometimes  
the app grows in a certain way and changing an interface is simply  
The Right Thing To Do? - but it will probably break the app in a hard  
to fix way so Let''s Just Stick With What Works? takes over :p

3) Rely on integration testing to catch these sort of bugs. This too  
can work - although it seems to shift a significant burden on  
integration testing. Doesn''t it sort of imply we need 100% (or close  
to that) integration test coverage? Doesn''t that mean A LOT of  
integration testing, and aren''t integration tests horribly brittle,  
time consuming to write in large numbers and a PITA to maintain?

I''m actually kind of confused about how integration tests fit in with  
the rSpec+Mocks way of doing things. I really don''t hear of teams  
doing regular integration testing in a true "let''s test all our  
coupling"-sense. (and speaking from experience, it really is a pain  
in the ass to do a lot of it - SO BRITTLE!) At the same, it also  
seems kind-of necessary. In addition to the above refactoring  
problem, there''s the question of reliable your external libraries  
really are (which you are going out of your way not to test by using  
mocks). They are often buggy, often famously so. Having 100:1 rSpec  
to code ratio won''t save you from IE6 bugs :p Having a javascript- 
heavy rails app with lots of IE6 users (not exactly an edge case with  
rspec users...) means these bugs are rather important. You can''t rely  
on rSpec to catch problems before pushing out a new version of your  
app. Doesn''t this take away one of the main benefits of automated  
testing?

Just aside from this whole business - I''m wondering how others are  
dealing with this problem? JS unit testing can help, no doubt, but  
there are lot and lots of things that can wrong outside such tests,  
and like I said, comprehensive unit tests are phenomenally hard to  
write and maintain.

So basically all 3 of the above options are pretty crappy.

One idea I had is to automatically translate MyObj#should_receive 
(:method).and_return("value") into a seprate spec for MyObj but...  
that actually just defeats the whole purpose of mock objects in a  
very round-about way :p

Another idea is to have a spec runner option which ignores all mocks  
and stubs - using the real options instead. This run mode would ONLY  
be triggered when someone changes an existing spec, specifically to  
to answer the question of "whose code did i just break". However - I  
think this would radically change the way specs have to be designed  
to really work. :/

I''m curious to hear how I should deal with these problems. Since
i''ve
been kinda rambly, I''ll restate my complaints:

(Note: when I say rSpec, I actually mean rSpec with heavy use of  
mocking, which seems to be the recommended way to go. I am aware that  
rspec != mocking, and that most of these complaints are actually more  
particular to mocking then rspec, but nevertheless, rspec and mocks  
do go hand in hand more often then not:)

1) If you change the interface of a function - all rSpec tells you is  
that something may have broken (because your existing specs for the  
function in question will fail initially). You don''t actually know  
for a fact anything broke, and you definitely don''t know what broke.  
It''s also very easy for someone new to rSpec to not realize that  
changing an existing spec may lead to un-detected failure somewhere  
else.

2) Mocking external libraries and/or access to external applications  
assumes these always work as expected, which as we all know, is a  
dirty lie :p (handy as it may be). rSpec does not protect you against  
bugs in the libraries you''re using in any way.

3) rSpec + Rails leaves a very size-able javascript and browser blind  
spot. No way of dealing with this exists other then integration  
testing (a-la selenium), and there is no accepted way of doing  
integration testing that is both feasible (i.e. doesn''t take for  
ever) and reliable (i.e. actually covers a good part of your code).


Finally, I really should mention that I''m actually quite happy with  
rSpec overall, and yes, I''m aware I''m asking a lot of rSpec
here :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/rspec-users/attachments/20070524/abd12560/attachment.html

Wow, you did a great job of writing up that irc discussion we had.

I just had a thought; what about if there was a way of "running" your
mock against the real thing to see if they (still) match up?


Courtenay


On 5/24/07, Ruy Asan <ruyasan at gmail.com> wrote:
> Suppose we have a method ''foo'' which internally uses
another method ''bar''.
>
> Being good BDDers we mock out the ''bar'' method. After
all, we only want to
> spec the ''foo'' method - actually running the
''bar'' method means slower, less
> maintainable and brittler specs. That''s why we <3 mocking,
right?
>
> We rely on the fact that ''bar'' is being adequately tested
somewhere else, by
> whomever wrote it.  They can change their implementation of
''bar'' and we can
> change our implementation of ''foo'' and as long as the
existing tests keep
> passing everyone is happy.
>
> But what happens if ''bar'' needs to be refactored in a way
that changes the
> interface? rSpec does succeed in informing bar''s author that he is
breaking
> someone''s expectations about the ''bar'' function
- however, he really has no
> idea whose expectations he is breaking. (i.e. rSpec doesn''t give
us any
> clues about what exactly has to follow our change in interface- it just
> tells us something may be broken).
>
> Changing the interface of bar means going over the code the hard way,
> getting in touch with everyone that uses the ''bar''
function and making sure
> everything still works. Not very agile! It makes changing interfaces a very
> very expensive process. Of course, such changes are never really cheap -
but
> because of mocks it becomes really REALLY expensive.
>
> So what are our options?
>
> 1) Don''t use mocks. Not using mocks would of course catch any such
problems
> right away. Yes mocks have benefits, but if we plan on refactoring often,
> the above scenario may just be too high a price to pay.
>
> 2) Don''t change interfaces (too often) - i.e. specs should be
treated as
> immutable. If it''s such an expensive process just make sure it
doesn''t have
> to happen that often ;)  I think if you have an all-vet, all-star team of
> programmers this could work out just fine - but even then, having change be
> expensive just isn''t a good thing. Sometimes the app grows in a
certain way
> and changing an interface is simply The Right Thing To Do? - but it will
> probably break the app in a hard to fix way so Let''s Just Stick
With What
> Works? takes over :p
>
> 3) Rely on integration testing to catch these sort of bugs. This too can
> work - although it seems to shift a significant burden on integration
> testing. Doesn''t it sort of imply we need 100% (or close to that)
> integration test coverage? Doesn''t that mean A LOT of integration
testing,
> and aren''t integration tests horribly brittle, time consuming to
write in
> large numbers and a PITA to maintain?
>
> I''m actually kind of confused about how integration tests fit in
with the
> rSpec+Mocks way of doing things. I really don''t hear of teams
doing regular
> integration testing in a true "let''s test all our
coupling"-sense. (and
> speaking from experience, it really is a pain in the ass to do a lot of it
-
> SO BRITTLE!) At the same, it also seems kind-of necessary. In addition to
> the above refactoring problem, there''s the question of reliable
your
> external libraries really are (which you are going out of your way not to
> test by using mocks). They are often buggy, often famously so. Having 100:1
> rSpec to code ratio won''t save you from IE6 bugs :p Having a
> javascript-heavy rails app with lots of IE6 users (not exactly an edge case
> with rspec users...) means these bugs are rather important. You
can''t rely
> on rSpec to catch problems before pushing out a new version of your app.
> Doesn''t this take away one of the main benefits of automated
testing?
>
> Just aside from this whole business - I''m wondering how others are
dealing
> with this problem? JS unit testing can help, no doubt, but there are lot
and
> lots of things that can wrong outside such tests, and like I
> said, comprehensive unit tests are phenomenally hard to write and maintain.
>
> So basically all 3 of the above options are pretty crappy.
>
> One idea I had is to automatically translate
> MyObj#should_receive(:method).and_return("value") into a
> seprate spec for MyObj but... that actually just defeats the whole purpose
> of mock objects in a very round-about way :p
>
> Another idea is to have a spec runner option which ignores all mocks and
> stubs - using the real options instead. This run mode would ONLY be
> triggered when someone changes an existing spec, specifically to to answer
> the question of "whose code did i just break". However - I think
this would
> radically change the way specs have to be designed to really work. :/
>
> I''m curious to hear how I should deal with these problems. Since
i''ve been
> kinda rambly, I''ll restate my complaints:
>
> (Note: when I say rSpec, I actually mean rSpec with heavy use of
> mocking, which seems to be the recommended way to go. I am aware that rspec
> != mocking, and that most of these complaints are actually more particular
> to mocking then rspec, but nevertheless, rspec and mocks do go hand in hand
> more often then not:)
>
> 1) If you change the interface of a function - all rSpec tells you is that
> something may have broken (because your existing specs for the function in
> question will fail initially). You don''t actually know for a fact
anything
> broke, and you definitely don''t know what broke. It''s
also very easy for
> someone new to rSpec to not realize that changing an existing spec may lead
> to un-detected failure somewhere else.
>
> 2) Mocking external libraries and/or access to external applications
assumes
> these always work as expected, which as we all know, is a dirty lie :p
> (handy as it may be). rSpec does not protect you against bugs in the
> libraries you''re using in any way.
>
> 3) rSpec + Rails leaves a very size-able javascript and browser blind spot.
> No way of dealing with this exists other then integration testing (a-la
> selenium), and there is no accepted way of doing integration testing that
is
> both feasible (i.e. doesn''t take for ever) and reliable (i.e.
actually
> covers a good part of your code).
>
>
> Finally, I really should mention that I''m actually quite happy
with rSpec
> overall, and yes, I''m aware I''m asking a lot of rSpec
here :)
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

On 5/25/07, Courtenay <court3nay at gmail.com>
wrote:
> Wow, you did a great job of writing up that irc discussion we had.
>
> I just had a thought; what about if there was a way of "running"
your
> mock against the real thing to see if they (still) match up?
Aslak had a similar idea a while back:

http://rubyforge.org/tracker/?func=detail&group_id=797&aid=5064&atid=3152

I''d like to see something like this - but controllable from the
command line. So you can run things as you normally do now, or you can
do something like:

$ spec spec --mock_mismatch
Thing#instance_method (not implemented)
- spec/a_spec.rb:37
- spec/b_spec.rb:42

Thing.class_method
- spec/c_spec.rb:13 - (argument mismatch: 2 for 1)

etc. The goal would be to point you to the right places to look to
learn about what needs to change, not to try to decipher the problem
in detail beyond the high level.

I think this could help assuage some of Ruy''s concerns, but not all.

Ruy: you stated that concerns are really about mocking, not about
RSpec. I''d expand that to say they are about testing in general. You
may want to float them on some other lists like
testdrivendevelopment at yahoogroups.com and
extremeprogramming at yahoogroups.com. The mocking questions could also
go to user at jmock.codehaus.org and mocha-developer at rubyforge.org.

Advanced apology if this sounds patronizing - but it sounds like
you''re looking for some sort of silver bullet or "best
practice". In
my experience, this is a road to disaster for two reasons. One, the
"best practice" gets in the way of actually thinking about problems.
Two, the acceptance of something as a "best practice" leads to a false
sense of security. "If I do this, then all will be right with the
world".

What''s worked for me has been a balance of programmer tests, customer
acceptance tests and exploratory testing. RSpec does a good job of
facilitating programmer tests, but can also help with acceptance tests
when coupled with other tools like selenium or watir (assuming you''re
doing webapps).

Keep in mind that TDD evolved on teams that had employed customer
acceptance tests as well as programmer tests. If you''re ONLY doing
programmer tests, you should probably mock a bit less, or divide your
tests up into class-level and integration tests. RSpec''s own tests are
a good example of this. We don''t have a clean separation of these, but
if you look through RSpec''s examples you''ll see some that feel
like
unit tests, some that feel like integration tests and some that feel
like story tests (i.e. customer tests).

Hope that helps.

Cheers,
David
>
>
> Courtenay
>
>
> On 5/24/07, Ruy Asan <ruyasan at gmail.com> wrote:
> > Suppose we have a method ''foo'' which internally uses
another method ''bar''.
> >
> > Being good BDDers we mock out the ''bar'' method.
After all, we only want to
> > spec the ''foo'' method - actually running the
''bar'' method means slower, less
> > maintainable and brittler specs. That''s why we <3 mocking,
right?
> >
> > We rely on the fact that ''bar'' is being adequately
tested somewhere else, by
> > whomever wrote it.  They can change their implementation of
''bar'' and we can
> > change our implementation of ''foo'' and as long as
the existing tests keep
> > passing everyone is happy.
> >
> > But what happens if ''bar'' needs to be refactored in
a way that changes the
> > interface? rSpec does succeed in informing bar''s author that
he is breaking
> > someone''s expectations about the ''bar''
function - however, he really has no
> > idea whose expectations he is breaking. (i.e. rSpec doesn''t
give us any
> > clues about what exactly has to follow our change in interface- it
just
> > tells us something may be broken).
> >
> > Changing the interface of bar means going over the code the hard way,
> > getting in touch with everyone that uses the ''bar''
function and making sure
> > everything still works. Not very agile! It makes changing interfaces a
very
> > very expensive process. Of course, such changes are never really cheap
- but
> > because of mocks it becomes really REALLY expensive.
> >
> > So what are our options?
> >
> > 1) Don''t use mocks. Not using mocks would of course catch any
such problems
> > right away. Yes mocks have benefits, but if we plan on refactoring
often,
> > the above scenario may just be too high a price to pay.
> >
> > 2) Don''t change interfaces (too often) - i.e. specs should be
treated as
> > immutable. If it''s such an expensive process just make sure
it doesn''t have
> > to happen that often ;)  I think if you have an all-vet, all-star team
of
> > programmers this could work out just fine - but even then, having
change be
> > expensive just isn''t a good thing. Sometimes the app grows in
a certain way
> > and changing an interface is simply The Right Thing To Do? - but it
will
> > probably break the app in a hard to fix way so Let''s Just
Stick With What
> > Works? takes over :p
> >
> > 3) Rely on integration testing to catch these sort of bugs. This too
can
> > work - although it seems to shift a significant burden on integration
> > testing. Doesn''t it sort of imply we need 100% (or close to
that)
> > integration test coverage? Doesn''t that mean A LOT of
integration testing,
> > and aren''t integration tests horribly brittle, time consuming
to write in
> > large numbers and a PITA to maintain?
> >
> > I''m actually kind of confused about how integration tests fit
in with the
> > rSpec+Mocks way of doing things. I really don''t hear of teams
doing regular
> > integration testing in a true "let''s test all our
coupling"-sense. (and
> > speaking from experience, it really is a pain in the ass to do a lot
of it -
> > SO BRITTLE!) At the same, it also seems kind-of necessary. In addition
to
> > the above refactoring problem, there''s the question of
reliable your
> > external libraries really are (which you are going out of your way not
to
> > test by using mocks). They are often buggy, often famously so. Having
100:1
> > rSpec to code ratio won''t save you from IE6 bugs :p Having a
> > javascript-heavy rails app with lots of IE6 users (not exactly an edge
case
> > with rspec users...) means these bugs are rather important. You
can''t rely
> > on rSpec to catch problems before pushing out a new version of your
app.
> > Doesn''t this take away one of the main benefits of automated
testing?
> >
> > Just aside from this whole business - I''m wondering how
others are dealing
> > with this problem? JS unit testing can help, no doubt, but there are
lot and
> > lots of things that can wrong outside such tests, and like I
> > said, comprehensive unit tests are phenomenally hard to write and
maintain.
> >
> > So basically all 3 of the above options are pretty crappy.
> >
> > One idea I had is to automatically translate
> > MyObj#should_receive(:method).and_return("value") into a
> > seprate spec for MyObj but... that actually just defeats the whole
purpose
> > of mock objects in a very round-about way :p
> >
> > Another idea is to have a spec runner option which ignores all mocks
and
> > stubs - using the real options instead. This run mode would ONLY be
> > triggered when someone changes an existing spec, specifically to to
answer
> > the question of "whose code did i just break". However - I
think this would
> > radically change the way specs have to be designed to really work. :/
> >
> > I''m curious to hear how I should deal with these problems.
Since i''ve been
> > kinda rambly, I''ll restate my complaints:
> >
> > (Note: when I say rSpec, I actually mean rSpec with heavy use of
> > mocking, which seems to be the recommended way to go. I am aware that
rspec
> > != mocking, and that most of these complaints are actually more
particular
> > to mocking then rspec, but nevertheless, rspec and mocks do go hand in
hand
> > more often then not:)
> >
> > 1) If you change the interface of a function - all rSpec tells you is
that
> > something may have broken (because your existing specs for the
function in
> > question will fail initially). You don''t actually know for a
fact anything
> > broke, and you definitely don''t know what broke.
It''s also very easy for
> > someone new to rSpec to not realize that changing an existing spec may
lead
> > to un-detected failure somewhere else.
> >
> > 2) Mocking external libraries and/or access to external applications
assumes
> > these always work as expected, which as we all know, is a dirty lie :p
> > (handy as it may be). rSpec does not protect you against bugs in the
> > libraries you''re using in any way.
> >
> > 3) rSpec + Rails leaves a very size-able javascript and browser blind
spot.
> > No way of dealing with this exists other then integration testing
(a-la
> > selenium), and there is no accepted way of doing integration testing
that is
> > both feasible (i.e. doesn''t take for ever) and reliable (i.e.
actually
> > covers a good part of your code).
> >
> >
> > Finally, I really should mention that I''m actually quite
happy with rSpec
> > overall, and yes, I''m aware I''m asking a lot of
rSpec here :)
> > _______________________________________________
> > rspec-users mailing list
> > rspec-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/rspec-users
> >
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

This is something I''ve kind of struggled with over the past few months
 [1].  And just yesterday I made the decision to move away from mocks
in most cases.  That decision resulted from me changing a db column
name from ''user'' to ''username'', but one of
my tests mocked a
#find_by_user method, so the test passed even though my code was
broken.

Just to note, I''ve got ~7k lines of Rails spec code that uses mocks,
and tens of thousands that don''t.  So I''ve used both
approaches
extensively in multiple projects, and have formed the following ideas.

I think David has a really good point about mock usage being related
to customer/programmer test coverage.  At this point, we don''t have
any customer tests.  I also don''t like the idea of relying on
heavyweight browser-based tests.  I want to be able to run rake and
know that my code works.

Over the past week, I''ve converted a number of my controller tests to
use the real implementations instead of using mocks.  It was obvious
when I realized that in my controllers I''m concerned with state a lot
more than interaction - a request is made, some state changes, a
response is given.  I want my tests to prove that those state changes
are actually being made.  In this way, my controller specs act as
integration tests.  I do a *tiny* bit of mocking in controller specs,
in cases where I can''t verify state such as making some notification
request to another machine in the system.

I''m still using mocks quite a bit in my model specs, and I''m
not quite
sure if I''ll be cutting back or not.  Just going to take it a day at a
time and try to remove any pain.

You''ve all surely read Martin Fowler''s discussion about
interaction-based vs state-based testing.  One of the points Dave
Astels always hammers on about BDD and interaction-based testing is
that it''s not testing, it''s a design tool.  I''ve
found that with mocks
I tend to arrive at a better design earlier on than I do without
mocks.  I''ve also found that I can''t change the design very
easily.
Mocks are coupled to the design of the code, not the behavior.  Aslak
says that state-based testing ain''t BDD [3], but I disagree.  If I
can''t refactor - changing the design of the code without changing the
behavior - that doesn''t seem very behavior-driven to me.

Mocks are nice as a design tool, but I''ve come to accept the fact that
creating a moderately good design early isn''t as useful as having the
freedom to change my design later on.  I''m just not good enough to
come up with the perfect design the first time around.  My tests need
to give me the confidence to refactor.  And I need to be able to run
those tests quickly from the command line rather than having to start
up a browser, even if it''s automated.  In that regard, mocks just
aren''t for me, at least on my current project.

Sorry for the length.  pat.should be_less_verbose

Pat

[1] http://evang.eli.st/blog/2007/4/9/i-think-i-might-not-get-mocks
[2] http://www.martinfowler.com/articles/mocksArentStubs.html
[3] http://blog.aslakhellesoy.com/2006/12/11/the-bdd-cargo-cult

(for some reason it doesn''t feel right to link to my blog and
MF''s
bliki together :)

On 5/25/07, Pat Maddox <pergesu at gmail.com>
wrote:
> This is something I''ve kind of struggled with over the past few
months
>  [1].  And just yesterday I made the decision to move away from mocks
> in most cases.  That decision resulted from me changing a db column
> name from ''user'' to ''username'', but one
of my tests mocked a
> #find_by_user method, so the test passed even though my code was
> broken.
>
> Just to note, I''ve got ~7k lines of Rails spec code that uses
mocks,
> and tens of thousands that don''t.  So I''ve used both
approaches
> extensively in multiple projects, and have formed the following ideas.
>
> I think David has a really good point about mock usage being related
> to customer/programmer test coverage.  At this point, we don''t
have
> any customer tests.  I also don''t like the idea of relying on
> heavyweight browser-based tests.  I want to be able to run rake and
> know that my code works.
>
> Over the past week, I''ve converted a number of my controller tests
to
> use the real implementations instead of using mocks.  It was obvious
> when I realized that in my controllers I''m concerned with state a
lot
> more than interaction - a request is made, some state changes, a
> response is given.  I want my tests to prove that those state changes
> are actually being made.  In this way, my controller specs act as
> integration tests.  I do a *tiny* bit of mocking in controller specs,
> in cases where I can''t verify state such as making some
notification
> request to another machine in the system.
>
> I''m still using mocks quite a bit in my model specs, and
I''m not quite
> sure if I''ll be cutting back or not.  Just going to take it a day
at a
> time and try to remove any pain.
>
> You''ve all surely read Martin Fowler''s discussion about
> interaction-based vs state-based testing.  One of the points Dave
> Astels always hammers on about BDD and interaction-based testing is
> that it''s not testing, it''s a design tool.  I''ve
found that with mocks
> I tend to arrive at a better design earlier on than I do without
> mocks.  I''ve also found that I can''t change the design
very easily.
> Mocks are coupled to the design of the code, not the behavior.  Aslak
> says that state-based testing ain''t BDD [3], but I disagree.  If I
> can''t refactor - changing the design of the code without changing
the
> behavior - that doesn''t seem very behavior-driven to me.
>
> Mocks are nice as a design tool, but I''ve come to accept the fact
that
> creating a moderately good design early isn''t as useful as having
the
> freedom to change my design later on.  I''m just not good enough to
> come up with the perfect design the first time around.  My tests need
> to give me the confidence to refactor.  And I need to be able to run
> those tests quickly from the command line rather than having to start
> up a browser, even if it''s automated.  In that regard, mocks just
> aren''t for me, at least on my current project.
One thing about TDD and the notion of a "design tool" - it''s
an
ongoing process. If you want to end up w/ real models in your specs,
you can still use mocks to figure out what they should do, then
replace them w/ the real deal later. This is not nearly as expensive
as it sounds, and lets you focus on one thing at a time.

As for integration testing, what I''ve found w/ Rails is that if you do
a couple of happy paths with rails built-in integration tests, you''ll
expose the refactoring problems that are raising such concern in this
thread. For me, that''s a win because you don''t have to do the
expensive in-browser stuff, everything''s in Ruby, you still can use
mocks to keep the isolation level stuff quick, and you run the slow
integration tests before you commit.

As for mocks tying you to implementation - I see a strong relationship
to how well you follow "Tell, Don''t Ask" and the value you
get out of
mocking. If you''re doing a lot telling instead of asking, you have an
implicitly more loosely coupled design. But if you want to make sure
that the object being described does the telling, there''s no better
way to do that then with a mock in my view.

2 more cents.

David
>
> Sorry for the length.  pat.should be_less_verbose
>
> Pat
>
> [1] http://evang.eli.st/blog/2007/4/9/i-think-i-might-not-get-mocks
> [2] http://www.martinfowler.com/articles/mocksArentStubs.html
> [3] http://blog.aslakhellesoy.com/2006/12/11/the-bdd-cargo-cult
>
> (for some reason it doesn''t feel right to link to my blog and
MF''s
> bliki together :)
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

Pat, I use rails'' integration tests to test my whole app.  Usually I
get lazy, and just dump my production data into the test DB and tell
my SpiderTest to just spider every single page and tell me what broke.

Also, I try to hide my database code behind methods, so
''find_by_user''
would forever remain as a method and whatever DB code will always be
hidden there.  To the controller, the actual fields I''m using are
irrelevant.  When starting a new project I''ll try hard to define a
nice uncoupled interface between models and controllers. It''s like,
when you assume that the model''s public methods will never change, you
name them a bit differently.

I''m not much of a theorist, though, and much prefer to let others kick
it out and wait for the dust to settle.



On 5/25/07, Pat Maddox <pergesu at gmail.com>
wrote:
> This is something I''ve kind of struggled with over the past few
months
>  [1].  And just yesterday I made the decision to move away from mocks
> in most cases.  That decision resulted from me changing a db column
> name from ''user'' to ''username'', but one
of my tests mocked a
> #find_by_user method, so the test passed even though my code was
> broken.
>
> Just to note, I''ve got ~7k lines of Rails spec code that uses
mocks,
> and tens of thousands that don''t.  So I''ve used both
approaches
> extensively in multiple projects, and have formed the following ideas.
>
> I think David has a really good point about mock usage being related
> to customer/programmer test coverage.  At this point, we don''t
have
> any customer tests.  I also don''t like the idea of relying on
> heavyweight browser-based tests.  I want to be able to run rake and
> know that my code works.
>
> Over the past week, I''ve converted a number of my controller tests
to
> use the real implementations instead of using mocks.  It was obvious
> when I realized that in my controllers I''m concerned with state a
lot
> more than interaction - a request is made, some state changes, a
> response is given.  I want my tests to prove that those state changes
> are actually being made.  In this way, my controller specs act as
> integration tests.  I do a *tiny* bit of mocking in controller specs,
> in cases where I can''t verify state such as making some
notification
> request to another machine in the system.
>
> I''m still using mocks quite a bit in my model specs, and
I''m not quite
> sure if I''ll be cutting back or not.  Just going to take it a day
at a
> time and try to remove any pain.
>
> You''ve all surely read Martin Fowler''s discussion about
> interaction-based vs state-based testing.  One of the points Dave
> Astels always hammers on about BDD and interaction-based testing is
> that it''s not testing, it''s a design tool.  I''ve
found that with mocks
> I tend to arrive at a better design earlier on than I do without
> mocks.  I''ve also found that I can''t change the design
very easily.
> Mocks are coupled to the design of the code, not the behavior.  Aslak
> says that state-based testing ain''t BDD [3], but I disagree.  If I
> can''t refactor - changing the design of the code without changing
the
> behavior - that doesn''t seem very behavior-driven to me.
>
> Mocks are nice as a design tool, but I''ve come to accept the fact
that
> creating a moderately good design early isn''t as useful as having
the
> freedom to change my design later on.  I''m just not good enough to
> come up with the perfect design the first time around.  My tests need
> to give me the confidence to refactor.  And I need to be able to run
> those tests quickly from the command line rather than having to start
> up a browser, even if it''s automated.  In that regard, mocks just
> aren''t for me, at least on my current project.
>
> Sorry for the length.  pat.should be_less_verbose
>
> Pat
>
> [1] http://evang.eli.st/blog/2007/4/9/i-think-i-might-not-get-mocks
> [2] http://www.martinfowler.com/articles/mocksArentStubs.html
> [3] http://blog.aslakhellesoy.com/2006/12/11/the-bdd-cargo-cult
>
> (for some reason it doesn''t feel right to link to my blog and
MF''s
> bliki together :)
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

On 5/25/07, David Chelimsky <dchelimsky at gmail.com>
wrote:
>
> As for mocks tying you to implementation - I see a strong relationship
> to how well you follow "Tell, Don''t Ask" and the value
you get out of
> mocking. If you''re doing a lot telling instead of asking, you have
an
> implicitly more loosely coupled design. But if you want to make sure
> that the object being described does the telling, there''s no
better
> way to do that then with a mock in my view.
>
David, Pat, Ruy:

I think this is a failure of Rails for letting us talk to the DB from
our controllers. For example

  def show
     @user = User.find(:first, :conditions => { :id => params[:id] },
:include => :images)
  end

This kind of thinking then pervades throughout the entire ''rails
way''
of doing things.

This makes it hard to refactor, and just horrible to mock.  I can
never think up a catchy name for this, (i''ll use find_inclusive for
now) but ...

  def show
    @user = User.find_inclusive( params[:id] )
  end

This works on the idea that, "What if ''find'' were a
private method?"

Now we can mock User.find_inclusive in the controller spec; we can
refactor the entire user model, change the database implementation
(:include something else) and, in an extreme case, you only give an
external coder access to particular models knowing that other code
wouldn''t be touched (yes, I''ve done this in the past. Yuk.)

Now, on a large or distributed team, hell.. any size team.. if someone
goes and changes find_inclusive, and the user-model spec breaks, they
are responsible for ensuring that the failing spec is NOT altered;
rather, that their code respects the old API and the spec passes, in
addition to their fixes.

Finally, higher-level tests (whether selenium or rails'' integrations)
will help you test the whole stack; I still don''t believe that
controllers should_hit the db :)


courtenay

On 5/25/07, David Chelimsky <dchelimsky at gmail.com>
wrote:
> On 5/25/07, Pat Maddox <pergesu at gmail.com> wrote:
> > This is something I''ve kind of struggled with over the past
few months
> >  [1].  And just yesterday I made the decision to move away from mocks
> > in most cases.  That decision resulted from me changing a db column
> > name from ''user'' to ''username'',
but one of my tests mocked a
> > #find_by_user method, so the test passed even though my code was
> > broken.
> >
> > Just to note, I''ve got ~7k lines of Rails spec code that uses
mocks,
> > and tens of thousands that don''t.  So I''ve used both
approaches
> > extensively in multiple projects, and have formed the following ideas.
> >
> > I think David has a really good point about mock usage being related
> > to customer/programmer test coverage.  At this point, we
don''t have
> > any customer tests.  I also don''t like the idea of relying on
> > heavyweight browser-based tests.  I want to be able to run rake and
> > know that my code works.
> >
> > Over the past week, I''ve converted a number of my controller
tests to
> > use the real implementations instead of using mocks.  It was obvious
> > when I realized that in my controllers I''m concerned with
state a lot
> > more than interaction - a request is made, some state changes, a
> > response is given.  I want my tests to prove that those state changes
> > are actually being made.  In this way, my controller specs act as
> > integration tests.  I do a *tiny* bit of mocking in controller specs,
> > in cases where I can''t verify state such as making some
notification
> > request to another machine in the system.
> >
> > I''m still using mocks quite a bit in my model specs, and
I''m not quite
> > sure if I''ll be cutting back or not.  Just going to take it a
day at a
> > time and try to remove any pain.
> >
> > You''ve all surely read Martin Fowler''s discussion
about
> > interaction-based vs state-based testing.  One of the points Dave
> > Astels always hammers on about BDD and interaction-based testing is
> > that it''s not testing, it''s a design tool. 
I''ve found that with mocks
> > I tend to arrive at a better design earlier on than I do without
> > mocks.  I''ve also found that I can''t change the
design very easily.
> > Mocks are coupled to the design of the code, not the behavior.  Aslak
> > says that state-based testing ain''t BDD [3], but I disagree. 
If I
> > can''t refactor - changing the design of the code without
changing the
> > behavior - that doesn''t seem very behavior-driven to me.
> >
> > Mocks are nice as a design tool, but I''ve come to accept the
fact that
> > creating a moderately good design early isn''t as useful as
having the
> > freedom to change my design later on.  I''m just not good
enough to
> > come up with the perfect design the first time around.  My tests need
> > to give me the confidence to refactor.  And I need to be able to run
> > those tests quickly from the command line rather than having to start
> > up a browser, even if it''s automated.  In that regard, mocks
just
> > aren''t for me, at least on my current project.
>
> One thing about TDD and the notion of a "design tool" -
it''s an
> ongoing process. If you want to end up w/ real models in your specs,
> you can still use mocks to figure out what they should do, then
> replace them w/ the real deal later. This is not nearly as expensive
> as it sounds, and lets you focus on one thing at a time.
I''ve considered that as well, and it''s an attractive approach
that I
plan to try soon.
> As for integration testing, what I''ve found w/ Rails is that if
you do
> a couple of happy paths with rails built-in integration tests,
you''ll
> expose the refactoring problems that are raising such concern in this
> thread. For me, that''s a win because you don''t have to do
the
> expensive in-browser stuff, everything''s in Ruby, you still can
use
> mocks to keep the isolation level stuff quick, and you run the slow
> integration tests before you commit.
I''ve yet to get major benefits from the integration tests.  I realize
that it''s nice to codify user stories, but so far it hasn''t
been nice
enough that I care very much.  I''m okay with the "user adds an
item to
car, user checks out" being split into separate controller tests.  If
they both pass, I''m satisfied that my code works.

I don''t think that integration tests are inherently bad, just that
they don''t provide enough value (to me) over standard controller
tests, given the real downside of slowness.
> As for mocks tying you to implementation - I see a strong relationship
> to how well you follow "Tell, Don''t Ask" and the value
you get out of
> mocking. If you''re doing a lot telling instead of asking, you have
an
> implicitly more loosely coupled design. But if you want to make sure
> that the object being described does the telling, there''s no
better
> way to do that then with a mock in my view.
I absolutely agree.  I didn''t really get into this, but my model specs
use mocks quite a bit.  And that''s good, because if I use mocks then
it means I''m far more likely to write better OO code.  After all, if
I''m doing a good job of "tell, don''t ask" then there
shouldn''t be too
much state to verify.

So I guess I can summarize my first post as saying that I prefer to
use controller specs as very lightweight integration tests.  In that
regard, I should be using real implementations where possible and
verifying the modified state of the system.

Pat
> 2 more cents.
>
> David
>
> >
> > Sorry for the length.  pat.should be_less_verbose
> >
> > Pat
> >
> > [1] http://evang.eli.st/blog/2007/4/9/i-think-i-might-not-get-mocks
> > [2] http://www.martinfowler.com/articles/mocksArentStubs.html
> > [3] http://blog.aslakhellesoy.com/2006/12/11/the-bdd-cargo-cult
> >
> > (for some reason it doesn''t feel right to link to my blog and
MF''s
> > bliki together :)
> > _______________________________________________
> > rspec-users mailing list
> > rspec-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/rspec-users
> >
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

On 5/25/07, Courtenay <court3nay at gmail.com>
wrote:
> Pat, I use rails'' integration tests to test my whole app.  Usually
I
> get lazy, and just dump my production data into the test DB and tell
> my SpiderTest to just spider every single page and tell me what broke.
That''s cool.  I discussed what I think of the integration tests in my
response to David.  I''d be really interested to know if I''m
missing
key benefits though.
> Also, I try to hide my database code behind methods, so
''find_by_user''
> would forever remain as a method and whatever DB code will always be
> hidden there.  To the controller, the actual fields I''m using are
> irrelevant.  When starting a new project I''ll try hard to define a
> nice uncoupled interface between models and controllers. It''s
like,
> when you assume that the model''s public methods will never change,
you
> name them a bit differently.
Totally.  I did end up just writing a find_by_user method that
delegated to find_by_username, so I didn''t have to change any of my
application code.  My concern though was that I didn''t know it broke
in the first place because I was using mocks.  But you''re right, had I
just written and tested find_by_user method instead of using the
dynamic finder, I wouldn''t have run into that issue.  I''ll be
sure to
keep that in mind.  It goes along with "don''t mock APIs you
don''t
own," and I certainly don''t own the dynamic finders.

Pat

>
> I''m not much of a theorist, though, and much prefer to let others
kick
> it out and wait for the dust to settle.
>
>
>
> On 5/25/07, Pat Maddox <pergesu at gmail.com> wrote:
> > This is something I''ve kind of struggled with over the past
few months
> >  [1].  And just yesterday I made the decision to move away from mocks
> > in most cases.  That decision resulted from me changing a db column
> > name from ''user'' to ''username'',
but one of my tests mocked a
> > #find_by_user method, so the test passed even though my code was
> > broken.
> >
> > Just to note, I''ve got ~7k lines of Rails spec code that uses
mocks,
> > and tens of thousands that don''t.  So I''ve used both
approaches
> > extensively in multiple projects, and have formed the following ideas.
> >
> > I think David has a really good point about mock usage being related
> > to customer/programmer test coverage.  At this point, we
don''t have
> > any customer tests.  I also don''t like the idea of relying on
> > heavyweight browser-based tests.  I want to be able to run rake and
> > know that my code works.
> >
> > Over the past week, I''ve converted a number of my controller
tests to
> > use the real implementations instead of using mocks.  It was obvious
> > when I realized that in my controllers I''m concerned with
state a lot
> > more than interaction - a request is made, some state changes, a
> > response is given.  I want my tests to prove that those state changes
> > are actually being made.  In this way, my controller specs act as
> > integration tests.  I do a *tiny* bit of mocking in controller specs,
> > in cases where I can''t verify state such as making some
notification
> > request to another machine in the system.
> >
> > I''m still using mocks quite a bit in my model specs, and
I''m not quite
> > sure if I''ll be cutting back or not.  Just going to take it a
day at a
> > time and try to remove any pain.
> >
> > You''ve all surely read Martin Fowler''s discussion
about
> > interaction-based vs state-based testing.  One of the points Dave
> > Astels always hammers on about BDD and interaction-based testing is
> > that it''s not testing, it''s a design tool. 
I''ve found that with mocks
> > I tend to arrive at a better design earlier on than I do without
> > mocks.  I''ve also found that I can''t change the
design very easily.
> > Mocks are coupled to the design of the code, not the behavior.  Aslak
> > says that state-based testing ain''t BDD [3], but I disagree. 
If I
> > can''t refactor - changing the design of the code without
changing the
> > behavior - that doesn''t seem very behavior-driven to me.
> >
> > Mocks are nice as a design tool, but I''ve come to accept the
fact that
> > creating a moderately good design early isn''t as useful as
having the
> > freedom to change my design later on.  I''m just not good
enough to
> > come up with the perfect design the first time around.  My tests need
> > to give me the confidence to refactor.  And I need to be able to run
> > those tests quickly from the command line rather than having to start
> > up a browser, even if it''s automated.  In that regard, mocks
just
> > aren''t for me, at least on my current project.
> >
> > Sorry for the length.  pat.should be_less_verbose
> >
> > Pat
> >
> > [1] http://evang.eli.st/blog/2007/4/9/i-think-i-might-not-get-mocks
> > [2] http://www.martinfowler.com/articles/mocksArentStubs.html
> > [3] http://blog.aslakhellesoy.com/2006/12/11/the-bdd-cargo-cult
> >
> > (for some reason it doesn''t feel right to link to my blog and
MF''s
> > bliki together :)
> > _______________________________________________
> > rspec-users mailing list
> > rspec-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/rspec-users
> >
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

On 5/25/07, Courtenay <court3nay at gmail.com>
wrote:
> On 5/25/07, David Chelimsky <dchelimsky at gmail.com> wrote:
> >
> > As for mocks tying you to implementation - I see a strong relationship
> > to how well you follow "Tell, Don''t Ask" and the
value you get out of
> > mocking. If you''re doing a lot telling instead of asking, you
have an
> > implicitly more loosely coupled design. But if you want to make sure
> > that the object being described does the telling, there''s no
better
> > way to do that then with a mock in my view.
> >
>
> David, Pat, Ruy:
>
> I think this is a failure of Rails for letting us talk to the DB from
> our controllers. For example
>
>   def show
>      @user = User.find(:first, :conditions => { :id => params[:id] },
> :include => :images)
>   end
>
> This kind of thinking then pervades throughout the entire ''rails
way''
> of doing things.
>
> This makes it hard to refactor, and just horrible to mock.
Yeah, I think that sucks.  I think I made a post about it a while back
regarding how to spec it.

it "should find the user, including associated images" do
  User.should receive(:find).with(:conditions => { :id => params[:id]
}, :include => :images).and_return @mock_user
  do_get
end

Ummm..............no
> I can
> never think up a catchy name for this, (i''ll use find_inclusive
for
> now) but ...
>
>   def show
>     @user = User.find_inclusive( params[:id] )
>   end
What about User.find_with_images or User.find_and_include_images
> This works on the idea that, "What if ''find'' were a
private method?"
>
> Now we can mock User.find_inclusive in the controller spec; we can
> refactor the entire user model, change the database implementation
> (:include something else) and, in an extreme case, you only give an
> external coder access to particular models knowing that other code
> wouldn''t be touched (yes, I''ve done this in the past.
Yuk.)
>
> Now, on a large or distributed team, hell.. any size team.. if someone
> goes and changes find_inclusive, and the user-model spec breaks, they
> are responsible for ensuring that the failing spec is NOT altered;
> rather, that their code respects the old API and the spec passes, in
> addition to their fixes.
>
> Finally, higher-level tests (whether selenium or rails''
integrations)
> will help you test the whole stack; I still don''t believe that
> controllers should_hit the db :)
The situation that really burned me, and led to me hitting the db from
controller tests, is one in which I was creating several related
objects.  I''d create a company, and that company had a before_create
method that created a site for it.  A Site would need to create its
own settings.  In the model specs, each level worked fine (I only
tested creating an object and the child object, not all the way down).
 But in production, foreign keys in the child objects just weren''t
being set.  My code didn''t work even though my specs said it did.  Of
course that''s because my specs just weren''t good enough, but
if I had
integration tests (even if they''re done in my controller spec), I
would have known that.

Pat