Cruisecontrolrb users - Feb 2008 - Failing unit tests, but not failing Cruise?

Hi all,

I intentionally committed a dud build to Cruise Control just to test  
that everything was working, but although Cruise picks up on the fact  
that there were test failures in the build, it still says the build  
was completed successfully. Is this expected behaviour? I''m using  
Cruise 1.2.1, Rails 2.0.2 and Shoulda for my unit tests, and I have  
the growl plugin installed.

Below is the output of the build in the Cruise Control dashboard  
(anonymised a bit). Any light anyone could shed on this would be  
fantastic.

Thanks,
Matt

---

MyProject build 17.3

finished at 10:50 AM on 07 February 2008

Build Changeset
Revision 17 committed by matt on 2008-02-06 21:36:08
Testing CruiseControl
   M /trunk/test/unit/test_element_test.rb
Test Failures and Errors
Name: test:
Type: Failure
Message: <0.5> and
<0.107> expected to be within
<0.05> of each other.

./test/unit/split_test_element_test.rb:27:in `__bind_1202334653_906151''
      /usr/local/src/cruisecontrolrb-1.2.1/projects/MyProject/work/ 
vendor/plugins/shoulda/lib/shoulda/gem/shoulda.rb:102:in `call''
      /usr/local/src/cruisecontrolrb-1.2.1/projects/MyProject/work/ 
vendor/plugins/shoulda/lib/shoulda/gem/shoulda.rb:102:in `test: A  
Test should be correctly distributed when its variants are unevenly  
weighted. ''
      /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/ 
active_support/testing/default.rb:7:in `run''

Build Log
/usr/local/src/cruisecontrolrb-1.2.1/projects/MyProject/work matt$  
ruby -e require ''rubygems'' rescue nil; require
''rake''; load ''/usr/
local/src/cruisecontrolrb-1.2.1/tasks/cc_build.rake''; ARGV <<
''--
nosearch'' << ''cc:build''; Rake.application.run
/usr/local/bin/ruby -Ilib:test "/usr/local/lib/ruby/gems/1.8/gems/ 
rake-0.7.3/lib/rake/rake_test_loader.rb"
"test/unit/campaign_test.rb"
"test/unit/client_test.rb"
"test/unit/dynamic_choice_test.rb" "test/
unit/dynamic_element_test.rb" "test/unit/dynamic_variant_test.rb"
"test/unit/form_field_test.rb"
"test/unit/landing_experience_test.rb"
"test/unit/landing_page_test.rb" "test/unit/site_test.rb"
"test/unit/
test_element_test.rb" "test/unit/test_test.rb" "test/unit/ 
visitor_test.rb"
(in /usr/local/src/cruisecontrolrb-1.2.1/projects/MyProject/work)
[CruiseControl] Invoking Rake task "test:units"
Loaded suite /usr/local/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/ 
rake_test_loader
Started
................................................F.....
Finished in 3.260502 seconds.

   1) Failure:
test: A SplitTest should be correctly distributed when its variants  
are unevenly weighted. (SplitTestElementTest)
     [./test/unit/test_element_test.rb:27:in `__bind_1202334653_906151''
      /usr/local/src/cruisecontrolrb-1.2.1/projects/MyProject/work/ 
vendor/plugins/shoulda/lib/shoulda/gem/shoulda.rb:102:in `call''
      /usr/local/src/cruisecontrolrb-1.2.1/projects/MyProject/work/ 
vendor/plugins/shoulda/lib/shoulda/gem/shoulda.rb:102:in `test: A  
Test should be correctly distributed when its variants are unevenly  
weighted. ''
      /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/ 
active_support/testing/default.rb:7:in `run'']:
<0.5> and
<0.107> expected to be within
<0.05> of each other.

54 tests, 202 assertions, 1 failures, 0 errors

On Feb 6, 2008 10:00 PM, Matt Powell, LeftClick Ltd <matt at
leftclick.com> wrote:
>test failures in the build, it still says the build was completed
successfully.
this is a known bug in Ruby 1.8.6.
Fixed in the most recent patch release (1.8.6 p111).

-- 
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

>> test failures in the build, it still says the build was completed  
>> successfully.
>
> this is a known bug in Ruby 1.8.6.
> Fixed in the most recent patch release (1.8.6 p111).
Ah, brilliant, thanks :)

Might I suggest adding this to the FAQ or documentation somewhere?

Thanks,
Matt