Rails core - Nov 2006 - Many Rails tests broken when run individually; tests depend on each other

Ok, I think I understand why so many tests I run individually appear to
be broken under Ruby (and subsequently, under JRuby).

The problem is that many of the Rails libraries make irreversible
changes to core classes and libraries. They require in new classes,
modify core classes, and so on. When running all tests together, this
has a negative effect on the individuality of each test: previous tests
will have modified the runtime in ways that taint the results of
subsequent tests.

For example, if an earlier test causes extensions to be added to the
String class, later tests will see those changes and not make them on
their own. This results in those tests requiring previous test runs to
run successfully themselves; those tests can''t be run individually.

I''m no testing maven, but it seems to be a fairly significant flaw that
tests can''t be run individually, or that tests depend on each other for
success. Can something be done to correct this?

I can provide my growing list of tests that can''t be run individually,
if it would help.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

> I''m no testing maven, but it seems to be a fairly significant flaw
that
> tests can''t be run individually, or that tests depend on each
other for
> success. Can something be done to correct this?
That''s bound to happen in any system that has any kind of dependency.

Say we add String#first and have a test for that. Now we have another
feature, which uses String#first. Of course that''ll fail too if
String#first is broken.

That''s a completely natural state of affairs in anything but the
simplest of systems.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

DHH wrote:
> > I''m no testing maven, but it seems to be a fairly significant
flaw that
> > tests can''t be run individually, or that tests depend on each
other for
> > success. Can something be done to correct this?
>
> That''s bound to happen in any system that has any kind of
dependency.
>
> Say we add String#first and have a test for that. Now we have another
> feature, which uses String#first. Of course that''ll fail too if
> String#first is broken.
>
> That''s a completely natural state of affairs in anything but the
> simplest of systems.
So it''s expected behavior that Rails tests can''t be run
individually? I
think just about anyone familiar with unit testing would say that''s a
little odd. In this case, I also believe that the breakages are
entirely due to the ordering of tests and the side-effects of previous
tests being available to subsequent tests. If a test is contained
within a single class, or at the very least if it''s within its own
file, it should not require that other tests be run for it to execute
successfully.

Now I recognize that some extra-test setup is sometimes
necessary--especially something like database setup, where it''s much
more complicated to setup/teardown for every test in that case. But
situations where test B doesn''t require a String extension simply
because when run in batch mode test A already has done so...well,
that''s a broken test. Or at the very least, it shouldn''t be
runnable
without test A running as well, since that''s an undocumented
prerequisite of test B. I think the preferred logic would be to do the
require everywhere it''s needed; that would fix the bulk of these.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

>I think the preferred logic would be to do the
> require everywhere it''s needed; that would fix the bulk of these.
If that''s the case, then I think it''s definitely reasonable to
apply
the patches.

-- 
Cheers

Koz

--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

Michael Koziarski wrote:
> >I think the preferred logic would be to do the
> > require everywhere it''s needed; that would fix the bulk of
these.
>
> If that''s the case, then I think it''s definitely
reasonable to apply
> the patches.
If y''all are on board, I can certainly report this as a bug and try to
come up with patches. I wouldn''t imagine that would help me with
compability efforts for JRuby and Rails 1.1.6 (or 1.2 soon) though.

This may also be worth bringing to the attention of Rake folks, since
they''ve got the standard test target logic set up to run all in the
same process; if they had always been making it run individual tests in
separate runtimes, you all wouldn''t have accidentally introduced
cross-test dependencies.

- Charlie


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

> So it''s expected behavior that Rails tests can''t be run
individually?
I think we''re taking past each other. Each test should be able to run
predictably in isolation, but many tests will depend on the same kind
of functionality, so String#first breaking will break many tests.

Please do post tickets to fix any tests that rely on setup work of
other tests.


--~--~---------~--~----~------------~-------~--~----~
 You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---