rspec users - May 2008 - Failing rspec story not causing a CruiseControl.rb build to fail?

We''ve just added rspec stories to our CruiseControl.rb build.  When
there''s a failing scenario we see the failure in the log output, but it
doesn''t cause the CruiseControl build to fail.  We''ve also
just upgraded to the latest rspec version, 1.1.4, which fixes rspec bug 228, but
the CC.rb build is still passing.

Has anyone got failing rspec stories making CruiseControl.rb fail?  

Thanks,
Sean Miller.

On Tue, May 27, 2008 at 8:41 AM, Sean Miller <sean at seanmiller.ca>
wrote:
> We''ve just added rspec stories to our CruiseControl.rb build. 
When there''s a failing scenario we see the failure in the log output,
but it doesn''t cause the CruiseControl build to fail.  We''ve
also just upgraded to the latest rspec version, 1.1.4, which fixes rspec bug
228, but the CC.rb build is still passing.
>
> Has anyone got failing rspec stories making CruiseControl.rb fail?
>
> Thanks,
> Sean Miller.
>
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>
I wasn''t aware that CC.rb integrated with story runner yet.  Are you
sure it does?

Pat

Pat Maddox wrote:
> On Tue, May 27, 2008 at 8:41 AM, Sean Miller <sean at seanmiller.ca>
wrote:
>   
>> We''ve just added rspec stories to our CruiseControl.rb build. 
When there''s a failing scenario we see the failure in the log output,
but it doesn''t cause the CruiseControl build to fail.  We''ve
also just upgraded to the latest rspec version, 1.1.4, which fixes rspec bug
228, but the CC.rb build is still passing.
>>
>> Has anyone got failing rspec stories making CruiseControl.rb fail?
>>
>> Thanks,
>> Sean Miller.
>>
>> _______________________________________________
>> rspec-users mailing list
>> rspec-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/rspec-users
>>
>>     
>
> I wasn''t aware that CC.rb integrated with story runner yet.  Are
you
> sure it does?
>
> Pat
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>   
To integrate with CC.rb all is that required is that your build 
task/process exits with the appropriate code (0 or non-zero for 
failing.)   I originally made the patch so that we could integrate it 
with CC.rb...

Sean, if you run the stories from the command line and it fails what is 
the exit code?  (run the stories and type ''echo $?'' to find
out.)  If
you can verify that the exit code is correct then the problem might be 
with your cc.rb rake task.  Maybe you could post it?  Also, what version 
of ruby are you using?  I have heard that a recent version of ruby had 
an at_exit bug, so try it on different versions of ruby to see if it is 
environment specific.

-Ben

Thanks!  The problem seems to be that the rake spec task is messing up the exit
code of the story spec, but I still can''t see how.

When I run the failing story spec outside of a rake task (ruby stories/all.rb),
I get the correct exit code of 1.

When I run it in rake spec:stories, I get the incorrect exit code of 0.

namespace :spec do

  desc "run spec user stories"
  Spec::Rake::SpecTask.new(''stories'') do |t|
    t.spec_files = FileList[''stories/**/*_stories.rb'']
    t.rcov = true
    output_root = ENV["CC_BUILD_ARTIFACTS"] || "spec"
    t.rcov_dir = "#{output_root}/user_stories_coverage"
    t.rcov_opts = RCOV_OPTIONS
  end  

But when I make one of the non-story specs fail, I get the exit code of 1 both
running the spec outside of rake and as part of "rake spec:coverage",
and the rake SpecTask is defined the same way:

  
  desc "Run all examples with RCov"
  Spec::Rake::SpecTask.new(''coverage'') do |t|
    t.spec_files = full_spec_file_list
    t.rcov = true
    output_root = ENV["CC_BUILD_ARTIFACTS"] || "spec"
    t.rcov_dir = "#{output_root}/rspec_coverage"
    t.rcov_opts = RCOV_OPTIONS
  end

Any ideas?

Thanks,
Sean.

On Tue, 27 May 2008 11:07:23 -0600, Ben Mabey <ben at benmabey.com>
wrote:
> Pat Maddox wrote:
>> On Tue, May 27, 2008 at 8:41 AM, Sean Miller <sean at
seanmiller.ca> wrote:
>>
>>> We''ve just added rspec stories to our CruiseControl.rb
build.  When
> there''s a failing scenario we see the failure in the log output,
but it
> doesn''t cause the CruiseControl build to fail.  We''ve
also just upgraded
> to the latest rspec version, 1.1.4, which fixes rspec bug 228, but the
> CC.rb build is still passing.
>>>
>>> Has anyone got failing rspec stories making CruiseControl.rb fail?
>>>
>>> Thanks,
>>> Sean Miller.
>>>
>>> _______________________________________________
>>> rspec-users mailing list
>>> rspec-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/rspec-users
>>>
>>>
>>
>> I wasn''t aware that CC.rb integrated with story runner yet. 
Are you
>> sure it does?
>>
>> Pat
>> _______________________________________________
>> rspec-users mailing list
>> rspec-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/rspec-users
>>
> 
> To integrate with CC.rb all is that required is that your build
> task/process exits with the appropriate code (0 or non-zero for
> failing.)   I originally made the patch so that we could integrate it
> with CC.rb...
> 
> Sean, if you run the stories from the command line and it fails what is
> the exit code?  (run the stories and type ''echo $?'' to
find out.)  If
> you can verify that the exit code is correct then the problem might be
> with your cc.rb rake task.  Maybe you could post it?  Also, what version
> of ruby are you using?  I have heard that a recent version of ruby had
> an at_exit bug, so try it on different versions of ruby to see if it is
> environment specific.
> 
> -Ben
> 
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users

Sean,
This is what I have on a project that is running CC.rb with stories:

desc "Task for cruise Control"
task :cruise do
  RAILS_ENV = ENV[''RAILS_ENV''] = ''test'' #
Without this, it will drop
your production database.
  CruiseControl::invoke_rake_task ''db:reset_all''
  CruiseControl::invoke_rake_task ''cruise_coverage''
  CruiseControl::invoke_rake_task ''stories:all''
  sh "rake kablame:git:specs > 
#{ENV[''CC_BUILD_ARTIFACTS'']}/hall_of_shame.txt"
end

namespace :stories do
  desc "Run all stories (that are not pending)"
  task :all do
    ruby "stories/all.rb --format plain"
  end
end

I think your problem is that you are using a SpecTask to run the 
stories... I don''t know if that is supported.  I just open up a new
ruby
process to run the stories.
Hope that helps,
Ben

Sean Miller wrote:
> Thanks!  The problem seems to be that the rake spec task is messing up the
exit code of the story spec, but I still can''t see how.
>
> When I run the failing story spec outside of a rake task (ruby
stories/all.rb), I get the correct exit code of 1.
>
> When I run it in rake spec:stories, I get the incorrect exit code of 0.
>
> namespace :spec do
>
>   desc "run spec user stories"
>   Spec::Rake::SpecTask.new(''stories'') do |t|
>     t.spec_files = FileList[''stories/**/*_stories.rb'']
>     t.rcov = true
>     output_root = ENV["CC_BUILD_ARTIFACTS"] || "spec"
>     t.rcov_dir = "#{output_root}/user_stories_coverage"
>     t.rcov_opts = RCOV_OPTIONS
>   end  
>
> But when I make one of the non-story specs fail, I get the exit code of 1
both running the spec outside of rake and as part of "rake
spec:coverage", and the rake SpecTask is defined the same way:
>
>   
>   desc "Run all examples with RCov"
>   Spec::Rake::SpecTask.new(''coverage'') do |t|
>     t.spec_files = full_spec_file_list
>     t.rcov = true
>     output_root = ENV["CC_BUILD_ARTIFACTS"] || "spec"
>     t.rcov_dir = "#{output_root}/rspec_coverage"
>     t.rcov_opts = RCOV_OPTIONS
>   end
>
> Any ideas?
>
> Thanks,
> Sean.
>
> On Tue, 27 May 2008 11:07:23 -0600, Ben Mabey <ben at benmabey.com>
wrote:
>   
>> Pat Maddox wrote:
>>     
>>> On Tue, May 27, 2008 at 8:41 AM, Sean Miller <sean at
seanmiller.ca> wrote:
>>>
>>>       
>>>> We''ve just added rspec stories to our CruiseControl.rb
build.  When
>>>>         
>> there''s a failing scenario we see the failure in the log
output, but it
>> doesn''t cause the CruiseControl build to fail.  We''ve
also just upgraded
>> to the latest rspec version, 1.1.4, which fixes rspec bug 228, but the
>> CC.rb build is still passing.
>>     
>>>> Has anyone got failing rspec stories making CruiseControl.rb
fail?
>>>>
>>>> Thanks,
>>>> Sean Miller.
>>>>
>>>> _______________________________________________
>>>> rspec-users mailing list
>>>> rspec-users at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/rspec-users
>>>>
>>>>
>>>>         
>>> I wasn''t aware that CC.rb integrated with story runner
yet.  Are you
>>> sure it does?
>>>
>>> Pat
>>> _______________________________________________
>>> rspec-users mailing list
>>> rspec-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/rspec-users
>>>
>>>       
>> To integrate with CC.rb all is that required is that your build
>> task/process exits with the appropriate code (0 or non-zero for
>> failing.)   I originally made the patch so that we could integrate it
>> with CC.rb...
>>
>> Sean, if you run the stories from the command line and it fails what is
>> the exit code?  (run the stories and type ''echo $?''
to find out.)  If
>> you can verify that the exit code is correct then the problem might be
>> with your cc.rb rake task.  Maybe you could post it?  Also, what
version
>> of ruby are you using?  I have heard that a recent version of ruby had
>> an at_exit bug, so try it on different versions of ruby to see if it is
>> environment specific.
>>
>> -Ben
>>
>> _______________________________________________
>> rspec-users mailing list
>> rspec-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/rspec-users
>>     
>
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>