rspec users - May 2009 - Any way to continue on Assertion Failure

Hi,

I wonder if there is any way to tell the test script to continue even 
when encountering an assertion failure?  The reason I am asking this is 
because I have a test flow that can take a long time to run and it would 
be very useful if I can perform all verification at the end of test flow 
in one shot rather than fail one and exit immediately.  Of course, I 
could break each verification point into separate test case, but then it 
will double or triple the overall execution time.  Does RSpec support this?

Thanks,

--Gary

On Tue, May 26, 2009 at 7:33 PM, Gary Lin <glin at employees.org>
wrote:
> Hi,
>
> I wonder if there is any way to tell the test script to continue even when
> encountering an assertion failure? ?The reason I am asking this is because
I
> have a test flow that can take a long time to run and it would be very
> useful if I can perform all verification at the end of test flow in one
shot
> rather than fail one and exit immediately. ?Of course, I could break each
> verification point into separate test case, but then it will double or
> triple the overall execution time. ?Does RSpec support this?
Nope. RSpec is all about isolated examples of small bits of behaviour.
>
> Thanks,
>
> --Gary
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

On Tue, May 26, 2009 at 7:58 PM, David Chelimsky <dchelimsky at gmail.com>
wrote:
> On Tue, May 26, 2009 at 7:33 PM, Gary Lin <glin at employees.org>
wrote:
>> Hi,
>>
>> I wonder if there is any way to tell the test script to continue even
when
>> encountering an assertion failure? ?The reason I am asking this is
because I
>> have a test flow that can take a long time to run and it would be very
>> useful if I can perform all verification at the end of test flow in one
shot
>> rather than fail one and exit immediately. ?Of course, I could break
each
>> verification point into separate test case, but then it will double or
>> triple the overall execution time. ?Does RSpec support this?
>
> Nope. RSpec is all about isolated examples of small bits of behaviour.
I don''t mean there is no way to do it, btw. Just that RSpec
doesn''t
offer a way out of the box. Each example is, essentially, run in a
begin/rescue block. RSpec expectation matchers raise errors on
failure.

To get what you''re after, you *could* write some sort of wrapper like
this:

def report_later(&block)
  begin
    yield
  rescue Exception => e
    # report the error somewhere but don''t raise
  end
end

describe Something do
  it "should do something" do
    report_later do
      # example code with expectations go here
    end
  end
end

If you do this, though, you''re really working against the intent of
RSpec. The trick about having a series of expectations within one
example is that if one fails early, you can''t really trust the state
of everything that comes after. This is one reason why RSpec is
designed the way it is - to focus on one expectation in each example.

HTH,
David

>
>>
>> Thanks,
>>
>> --Gary

On 27/05/2009, at 9:33 AM, Gary Lin wrote:
> Hi,
>
> I wonder if there is any way to tell the test script to continue  
> even when encountering an assertion failure?  The reason I am asking  
> this is because I have a test flow that can take a long time to run  
> and it would be very useful if I can perform all verification at the  
> end of test flow in one shot rather than fail one and exit  
> immediately.  Of course, I could break each verification point into  
> separate test case, but then it will double or triple the overall  
> execution time.  Does RSpec support this?
>
Isn''t this usual behaviour?

Julian.



----------------------------------------------
Learn: http://sensei.zenunit.com/
Last updated 20-May-09 (Rails, Basic Unix)
Blog: http://random8.zenunit.com/
Twitter: http://twitter.com/random8r

> Thanks,
>
> --Gary
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users

On 27 May 2009, at 03:38, Julian Leviston wrote:
> On 27/05/2009, at 9:33 AM, Gary Lin wrote:
>
>> Hi,
>>
>> I wonder if there is any way to tell the test script to continue  
>> even when encountering an assertion failure?  The reason I am  
>> asking this is because I have a test flow that can take a long time  
>> to run and it would be very useful if I can perform all  
>> verification at the end of test flow in one shot rather than fail  
>> one and exit immediately.  Of course, I could break each  
>> verification point into separate test case, but then it will double  
>> or triple the overall execution time.  Does RSpec support this?
>>
>
> Isn''t this usual behaviour?
>
> Julian.
Yeah I''m a little confused by the question (and David''s
response) so
maybe I''ve misunderstood something. Normal RSpec test runs will catch  
all the test failures and report them at the end of the run. I wonder  
whether the OP is talking about a situation where the interaction with  
the system takes a long time, so that using lots of examples of the  
desired behaviour causes that interaction to be run several times  
making the whole run very slow.

I also wonder whether he''s just thinking in the test/unit mindset,  
which I see a lot, where you have examples like this

<antipattern>
it "should do foo and bar" do
   set_up_state
   do_something_to_system_under_test

   assert_that_foo_was_done
   assert_that_bar_was_done
end
</antipattern>

Here''s my normal RSpec flow:

before(:each) do
   set_up_state
   do_something_to_system_under_test
end

it "should have done foo" do
   assert_that_foo_was_done
end

it "should have done bar" do
   assert_that_bar_was_done
end

if set_up_state and/or do_something_to_system_under_test take ages,  
you could use a before(:all) block instead, but that comes with  
obvious leaky state disadvantages.

Matt Wynne
http://beta.songkick.com
http://blog.mattwynne.net

On Wed, May 27, 2009 at 4:07 AM, Matt Wynne <matt at mattwynne.net>
wrote:
>
> On 27 May 2009, at 03:38, Julian Leviston wrote:
>
>> On 27/05/2009, at 9:33 AM, Gary Lin wrote:
>>
>>> Hi,
>>>
>>> I wonder if there is any way to tell the test script to continue
even
>>> when encountering an assertion failure? ?The reason I am asking
this is
>>> because I have a test flow that can take a long time to run and it
would be
>>> very useful if I can perform all verification at the end of test
flow in one
>>> shot rather than fail one and exit immediately. ?Of course, I could
break
>>> each verification point into separate test case, but then it will
double or
>>> triple the overall execution time. ?Does RSpec support this?
>>>
>>
>> Isn''t this usual behaviour?
>>
>> Julian.
>
> Yeah I''m a little confused by the question (and David''s
response) so maybe
> I''ve misunderstood something. Normal RSpec test runs will catch
all the test
> failures and report them at the end of the run. I wonder whether the OP is
> talking about a situation where the interaction with the system takes a
long
> time, so that using lots of examples of the desired behaviour causes that
> interaction to be run several times making the whole run very slow.
>
> I also wonder whether he''s just thinking in the test/unit mindset,
which I
> see a lot, where you have examples like this
>
> <antipattern>
> it "should do foo and bar" do
> ?set_up_state
> ?do_something_to_system_under_test
>
> ?assert_that_foo_was_done
> ?assert_that_bar_was_done
> end
> </antipattern>
I don''t know that I would call this an antipattern unless set_up_state
is duplicated across examples. I usually don''t factor things out of an
example unless it''s duplicated in another example (then I extract to a
before) or I''m working in a specific context (which I always give a
before since it provides meaning of the context in an explicit way).
>
> Here''s my normal RSpec flow:
>
> before(:each) do
> ?set_up_state
> ?do_something_to_system_under_test
> end
>
> it "should have done foo" do
> ?assert_that_foo_was_done
> end
>
> it "should have done bar" do
> ?assert_that_bar_was_done
> end
Most of the time I would leave the "do_something_to_system_under_test"
inside each example. I find it easier to read, and a more useful
example (since you don''t have to go look else where to see how the
method is called), and it becomes easier to maintain should the object
change in a way that requires a specific pre-condition to be
introduced that aren''t needed by the other examples (you don''t
have to
modify every other example).

another 2 cents to throw in the water fountain,
>
> if set_up_state and/or do_something_to_system_under_test take ages, you
> could use a before(:all) block instead, but that comes with obvious leaky
> state disadvantages.
>
> Matt Wynne
> http://beta.songkick.com
> http://blog.mattwynne.net
>
>
>
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>


-- 
Zach Dennis
http://www.continuousthinking.com (personal)
http://www.mutuallyhuman.com (hire me)
http://ideafoundry.info/behavior-driven-development (first rate BDD training)
@zachdennis (twitter)

Let''s say I''ve got this spec:

describe Team, "add_player" do
  it "should add a player to the team" do
    team = Team.new
    player = Player.new :name => "Pat"
    team.add_player player
    team.should have(1).players
    team.players.first.name.should == "Pat"
  end
end

and for some reason
team.should have(1).players
fails.  Do I get any value from seeing the second expectation fail?
What if it passes?

My suggestion is to break up your long running test flow into bits
that you can test individually, that run quickly and independently of
each other.  Get those tested, and have an integration test which
makes sure that everything works together.

Pat

On Tue, May 26, 2009 at 4:33 PM, Gary Lin <glin at employees.org>
wrote:
> Hi,
>
> I wonder if there is any way to tell the test script to continue even when
> encountering an assertion failure? ?The reason I am asking this is because
I
> have a test flow that can take a long time to run and it would be very
> useful if I can perform all verification at the end of test flow in one
shot
> rather than fail one and exit immediately. ?Of course, I could break each
> verification point into separate test case, but then it will double or
> triple the overall execution time. ?Does RSpec support this?
>
> Thanks,
>
> --Gary
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>