rspec users - Jul 2008 - Failing on rake but not on textmate (rails)

I have a spec for a model that passes all tests if I run it using
textmate but I run rake some of the otherwise passed tests fail.

I would like to use autotest but I can''t trust the results.

Any ideas?
-- 
Posted via http://www.ruby-forum.com/.

On Mon, Jul 21, 2008 at 4:33 PM, Macario Ortega <lists at ruby-forum.com>
wrote:
>
> I have a spec for a model that passes all tests if I run it using
> textmate but I run rake some of the otherwise passed tests fail.
>
AFAIK, the main difference is that rake copies the development db structure
to the test db, whereas TextMate does not (and runs faster as a result). I
don''t know if that explains the behavior you''re seeing.

> I would like to use autotest but I can''t trust the results.
>
autotest also does not copy the database structure. I''ve been using it
for
six months and I''ve never noticed a problem with it. Why don''t
you trust it?

///ark
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/rspec-users/attachments/20080721/5197b393/attachment.html>

On Mon, Jul 21, 2008 at 8:34 PM, Mark Wilden <mark.wilden at gmail.com>
wrote:
> On Mon, Jul 21, 2008 at 4:33 PM, Macario Ortega <lists at
ruby-forum.com>
> wrote:
>
>>
>> I have a spec for a model that passes all tests if I run it using
>> textmate but I run rake some of the otherwise passed tests fail.
>>
>
> AFAIK, the main difference is that rake copies the development db structure
> to the test db, whereas TextMate does not (and runs faster as a result). I
> don''t know if that explains the behavior you''re seeing.
>
>
>> I would like to use autotest but I can''t trust the results.
>>
>
> autotest also does not copy the database structure. I''ve been
using it for
> six months and I''ve never noticed a problem with it. Why
don''t you trust it?
>
> I doubt that it''s a problem with not copy the database structure
from dev
to test. That sync''s the db schema and would only be needed after a
schema
change (i.e. one or more migrations have been run), and it would be much
more likely to cause a failure in Textmate when the code under test
doesn''t
see the expected db schema.

It''s more likely that there are undeclared fixtures affecting the spec,
a
spec which needs particular state in the database might succeed or not
depending on whether or not previous specs either left data behind, or
deleted data needed by the spec in question.

When I''ve seen cases where test/specs behaved differently when run
separately vs. being run in batch (e.g. by Rake) it''s almost always
because
I left out a fixture declaration.

-- 
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/rspec-users/attachments/20080722/aa4c3955/attachment.html>

Mark Wilden wrote:
> On Mon, Jul 21, 2008 at 4:33 PM, Macario Ortega <lists at
ruby-forum.com>
> wrote:
> 
>>
>> I have a spec for a model that passes all tests if I run it using
>> textmate but I run rake some of the otherwise passed tests fail.
>>
> 
> AFAIK, the main difference is that rake copies the development db 
> structure
> to the test db, whereas TextMate does not (and runs faster as a result). 
> I
> don''t know if that explains the behavior you''re seeing.
> 
> 
>> I would like to use autotest but I can''t trust the results.
>>
> 
> autotest also does not copy the database structure. I''ve been
using it
> for
> six months and I''ve never noticed a problem with it. Why
don''t you trust
> it?
> 
> ///ark
Well I don''t trust autotest because the same tests pass when i run them
from textmate and they fail when I run autotest.

The dubbious specs are all for a specific model but some of them are 
very basic such as testing validates_presence_of in which the model code 
is obviously good and yet the spec fails on rake or autotest.

So I''ve been running my specs just on textmate but I would like to use 
autotest too.


-- 
Posted via http://www.ruby-forum.com/.

On Tue, Jul 22, 2008 at 1:56 PM, Macario Ortega <lists at ruby-forum.com>
wrote:
>
> Well I don''t trust autotest because the same tests pass when i run
them
> from textmate and they fail when I run autotest.
>
> The dubbious specs are all for a specific model but some of them are
> very basic such as testing validates_presence_of in which the model code
> is obviously good and yet the spec fails on rake or autotest.
I''ve had this happen before, and it turned out to be a state problem.
Things I did in one spec file were altering the state of the
application, so that spec files run after it would fail in certain
ways.  Thus, passing or failing was entirely dependent on the *order*
the files were run in.  Rake runs things in a different order than
autotest, so different tests were failing; and running a single file
by itself always passed.

In my case the state change was stupid and totally unnecessary.  At
the time I didn''t really understand mocking and stubbing, so I was
trying to bypass my authentication code in my specs by reopening the
authentication modules in lib/ and overriding the live logged_in?
method to return ''true''.  This worked, but I didn''t
realize the
overrides would continue to be in place for all future specs.  And
quite naturally, some of the specs for restful_authentication were
failing.

I wasted hours figuring that one out.  The moral I learned was, don''t
EVER try to screw with your actual application code in a spec, and
make sure that nothing you do in a spec leaves a permanent change in
memory.  I can''t know if that''s your problem, of course, but
it''s
something to think about.


-- 
Have Fun,
Steve Eley
Deep Salt Team

Rick Denatale wrote:
> On Mon, Jul 21, 2008 at 8:34 PM, Mark Wilden <mark.wilden at
gmail.com>
> wrote:
> 
>> don''t know if that explains the behavior you''re
seeing.
>>
>>
>>> I would like to use autotest but I can''t trust the
results.
>>>
>>
>> autotest also does not copy the database structure. I''ve been
using it for
>> six months and I''ve never noticed a problem with it. Why
don''t you trust it?
>>
>> I doubt that it''s a problem with not copy the database
structure from dev
> to test. That sync''s the db schema and would only be needed after
a
> schema
> change (i.e. one or more migrations have been run), and it would be much
> more likely to cause a failure in Textmate when the code under test 
> doesn''t
> see the expected db schema.
> 
> It''s more likely that there are undeclared fixtures affecting the
spec,
> a
> spec which needs particular state in the database might succeed or not
> depending on whether or not previous specs either left data behind, or
> deleted data needed by the spec in question.
> 
> When I''ve seen cases where test/specs behaved differently when run
> separately vs. being run in batch (e.g. by Rake) it''s almost
always
> because
> I left out a fixture declaration.
> 
> --
> Rick DeNatale
> 
> My blog on Ruby
> http://talklikeaduck.denhaven2.com/

Hi, thanks for the answer today I ran rake several times and all my test 
passed (and that made me very happy) but suddenly the tests for this 
model started failing again.

Prior to each test I delete all records for the model and I have no 
fixture for that particular model.

I am just starting with sdd and I find fixtures to give me more trouble 
than they solve so when I need to populate the database I create and 
save the models I need thus I have all the dependencies.

What do you mean by undeclared fixture? a fixture with no data?

What is strange is that some validation specs (''it should require
name'')
that don''t depend on database also fail.


-- 
Posted via http://www.ruby-forum.com/.

On Thu, Jul 24, 2008 at 12:27 PM, Macario Ortega <lists at ruby-forum.com>
wrote:
> Rick Denatale wrote:
>> On Mon, Jul 21, 2008 at 8:34 PM, Mark Wilden <mark.wilden at
gmail.com>
>> wrote:
>>
>>> don''t know if that explains the behavior you''re
seeing.
>>>
>>>
>>>> I would like to use autotest but I can''t trust the
results.
>>>>
>>>
>>> autotest also does not copy the database structure. I''ve
been using it for
>>> six months and I''ve never noticed a problem with it. Why
don''t you trust it?
>>>
>>> I doubt that it''s a problem with not copy the database
structure from dev
>> to test. That sync''s the db schema and would only be needed
after a
>> schema
>> change (i.e. one or more migrations have been run), and it would be
much
>> more likely to cause a failure in Textmate when the code under test
>> doesn''t
>> see the expected db schema.
>>
>> It''s more likely that there are undeclared fixtures affecting
the spec,
>> a
>> spec which needs particular state in the database might succeed or not
>> depending on whether or not previous specs either left data behind, or
>> deleted data needed by the spec in question.
>>
>> When I''ve seen cases where test/specs behaved differently when
run
>> separately vs. being run in batch (e.g. by Rake) it''s almost
always
>> because
>> I left out a fixture declaration.
>>
>> --
>> Rick DeNatale
>>
>> My blog on Ruby
>> http://talklikeaduck.denhaven2.com/
>
>
> Hi, thanks for the answer today I ran rake several times and all my test
> passed (and that made me very happy) but suddenly the tests for this
> model started failing again.
We get failures at times when running specs with rake spec that we
can''t reproduce when running in Textmate.  We came to the conclusion
that it may have something to do with transactions not rolling back
before the start of the next test.  So even though the models we
create should be unique (we create them with some randomness built
in), we will get validation failures when saving a new record to the
DB.

You may want to try turning off transactions with:
  config.use_transactional_fixtures = false
and see if that makes a difference.

We are using Rspec 1.1.4 and PostgreSQL as our db.  Let me know if that helps.

dan

> Prior to each test I delete all records for the model and I have no
> fixture for that particular model.
>
> I am just starting with sdd and I find fixtures to give me more trouble
> than they solve so when I need to populate the database I create and
> save the models I need thus I have all the dependencies.
>
> What do you mean by undeclared fixture? a fixture with no data?
>
> What is strange is that some validation specs (''it should require
name'')
> that don''t depend on database also fail.

Dan Herrera wrote:
> We get failures at times when running specs with rake spec that we
> can''t reproduce when running in Textmate.  We came to the
conclusion
> that it may have something to do with transactions not rolling back
> before the start of the next test.  So even though the models we
> create should be unique (we create them with some randomness built
> in), we will get validation failures when saving a new record to the
> DB.
> 
> You may want to try turning off transactions with:
>   config.use_transactional_fixtures = false
> and see if that makes a difference.
> 
> We are using Rspec 1.1.4 and PostgreSQL as our db.  Let me know if that 
> helps.
> 
> dan
Thanks, I''ve tried that and I got four additional failures concerning
Attachment Fu not using the expected path to save the images.

I guess the whole rspec system is very complex.

-- 
Posted via http://www.ruby-forum.com/.

Btw, I am with sqlite3.

-- 
Posted via http://www.ruby-forum.com/.

Macario Ortega wrote:
> 
> Btw, I am with sqlite3.
Hi, I think I solved the issue, I had mocha in my plugins but I was 
using it in just a couple of tests and this line was commented in the 
spec_helper.rb

# config.mock_with :mocha

I removed mocha from my plugins folder ran rake and this two tests 
failed, I fixed them and now I can see about 600 green dots and no F''s 
when I run rake :)

-- 
Posted via http://www.ruby-forum.com/.