rspec users - Aug 2008 - return code question

We are trying to automate the running of our rspec tests for our Rails app
on a build server using Capistrano.  The problem is that Capistrano seems to
think that the command called to run the model tests failed when in fact I
believe it succeeded. I believe the problem has to do with the return code
from the command I am running.. The command I am running is:

ruby script/spec spec/models -f html:model_test_results.html
 
The output file seems to be complete as it has both the opening and closing
html tags and it has the results of all of our tests. But when I run $? To
get the return code of the last command it says it was "1" which I
believe
signifies that the command was not successful and is what is causing our
Capistrano task to abort.  We are using RSpec 1.1.4 and in case you''re
curious the output from Capistrano looks like this:

 

  * executing "cd /data/nightingalenotes/releases/20080812160703 &&
ruby
script/
spec spec/models -f html:public/rspec_test_results/model_test_results.html"
    servers: ["buildnexus.champ.net"]
    [buildnexus.champ.net] executing command
    command finished
*** [deploy:update_code] rolling back
  * executing "rm -rf /data/nightingalenotes/releases/20080812160703;
true"
    servers: ["buildnexus.champ.net"]
    [buildnexus.champ.net] executing command
    command finished
command "cd /data/nightingalenotes/releases/20080812160703 && ruby
script/spec s
pec/models -f html:public/rspec_test_results/model_test_results.html"
failed
on
buildnexus.champ.net

 

 

Thanks!

 

Ben Fyvie

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/rspec-users/attachments/20080812/990e1610/attachment.html>

On Tue, Aug 12, 2008 at 1:22 PM, Ben Fyvie <ben.fyvie at
champsoftware.com> wrote:
> We are trying to automate the running of our rspec tests for our Rails app
> on a build server using Capistrano.  The problem is that Capistrano seems
to
> think that the command called to run the model tests failed when in fact I
> believe it succeeded. I believe the problem has to do with the return code
> from the command I am running.. The command I am running is:
>
> ruby script/spec spec/models -f html:model_test_results.html
>
>
>
> The output file seems to be complete as it has both the opening and closing
> html tags and it has the results of all of our tests. But when I run $? To
> get the return code of the last command it says it was "1" which
I believe
> signifies that the command was not successful and is what is causing our
> Capistrano task to abort.
I just did this in a Rakefile:

task :models do
  `ruby script/spec spec/models -f html:model_test_results.html`
  puts $?
end

And I get this when I run ''rake models''

$ rake models
(in /Users/david/projects/ruby/mycastory)
0

I''m using the latest from git (naturally).

How are you checking the exit code?
> We are using RSpec 1.1.4 and in case you''re
> curious the output from Capistrano looks like this:
>
>
>
>   * executing "cd /data/nightingalenotes/releases/20080812160703
&& ruby
> script/
>
> spec spec/models -f
html:public/rspec_test_results/model_test_results.html"
>
>     servers: ["buildnexus.champ.net"]
>
>     [buildnexus.champ.net] executing command
>
>     command finished
>
> *** [deploy:update_code] rolling back
>
>   * executing "rm -rf /data/nightingalenotes/releases/20080812160703;
true"
>
>     servers: ["buildnexus.champ.net"]
>
>     [buildnexus.champ.net] executing command
>
>     command finished
>
> command "cd /data/nightingalenotes/releases/20080812160703 &&
ruby
> script/spec s
>
> pec/models -f html:public/rspec_test_results/model_test_results.html"
failed
> on
>
> buildnexus.champ.net
>
>
>
>
>
> Thanks!
>
>
>
> Ben Fyvie
>
>
>
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

The command I was running to get the return code was "echo $?" After
reading
over my email it seems I left out a bit of rather important information. 

You see we expect the command to return a return code of "0" if the
command
was successful in it''s task which was to generate an html file with the
results of running the model tests.  In our case the command is successful
and it generates the desired output and puts it into the HTML file as
expected.  However we find that the return code is changed to "1" if
any
single test fails.  To me this seems incorrect because the result of a test
should not influence the return code as the return code should be associated
with whether or not the command executed successfully or not.  

Our current situation is a great example of why the return code should only
represent the success or failure of the command and not the results of the
command.  Whenever a test fails the return code of "1" causes our
build to
break and therefore nobody even knows a test is failing until we notice that
the build server is no longer updating on regular intervals. 

Ben Fyvie

-----Original Message-----
From: rspec-users-bounces at rubyforge.org
[mailto:rspec-users-bounces at rubyforge.org] On Behalf Of David Chelimsky
Sent: Wednesday, August 13, 2008 7:36 AM
To: rspec-users
Subject: Re: [rspec-users] return code question

On Tue, Aug 12, 2008 at 1:22 PM, Ben Fyvie <ben.fyvie at
champsoftware.com>
wrote:
> We are trying to automate the running of our rspec tests for our Rails app
> on a build server using Capistrano.  The problem is that Capistrano seems
to
> think that the command called to run the model tests failed when in fact I
> believe it succeeded. I believe the problem has to do with the return code
> from the command I am running.. The command I am running is:
>
> ruby script/spec spec/models -f html:model_test_results.html
>
>
>
> The output file seems to be complete as it has both the opening and
closing
> html tags and it has the results of all of our tests. But when I run $? To
> get the return code of the last command it says it was "1" which
I believe
> signifies that the command was not successful and is what is causing our
> Capistrano task to abort.
I just did this in a Rakefile:

task :models do
  `ruby script/spec spec/models -f html:model_test_results.html`
  puts $?
end

And I get this when I run ''rake models''

$ rake models
(in /Users/david/projects/ruby/mycastory)
0

I''m using the latest from git (naturally).

How are you checking the exit code?
> We are using RSpec 1.1.4 and in case you''re
> curious the output from Capistrano looks like this:
>
>
>
>   * executing "cd /data/nightingalenotes/releases/20080812160703
&& ruby
> script/
>
> spec spec/models -f
html:public/rspec_test_results/model_test_results.html"
>
>     servers: ["buildnexus.champ.net"]
>
>     [buildnexus.champ.net] executing command
>
>     command finished
>
> *** [deploy:update_code] rolling back
>
>   * executing "rm -rf /data/nightingalenotes/releases/20080812160703;
true"
>
>     servers: ["buildnexus.champ.net"]
>
>     [buildnexus.champ.net] executing command
>
>     command finished
>
> command "cd /data/nightingalenotes/releases/20080812160703 &&
ruby
> script/spec s
>
> pec/models -f html:public/rspec_test_results/model_test_results.html"
failed
> on
>
> buildnexus.champ.net
>
>
>
>
>
> Thanks!
>
>
>
> Ben Fyvie
>
>
>
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>
_______________________________________________
rspec-users mailing list
rspec-users at rubyforge.org
http://rubyforge.org/mailman/listinfo/rspec-users