Rails - Sep 2009 - How can I have the tests for my Rails app to be executed in a random order?

Hi
How can I have the tests for my Rails app to be executed in a random
order? Is there a simple solution using rake?

I already posted my question to stackoverflow (http://
stackoverflow.com/questions/1376267/ruby-executing-tests-in-a-random-
order-with-rake) but the few responses I got were not able to solve
the problem.

Thanks a lot!

Pietro Di Bello

Piero Di Bello wrote:
> Hi
> How can I have the tests for my Rails app to be executed in a random
> order? 
At least with RSpec, specs are executed in unpredictable order.  I think 
it''s the same each time, but it''s not the order in which they
appear in
the spec file.

But why do you need this anyway? Each test should start from a clean 
slate, so the order should be completely immaterial.

Best,
--
Marnen Laibow-Koser
http://www.marnen.org
marnen-sbuyVjPbboAdnm+yROfE0A@public.gmane.org
-- 
Posted via http://www.ruby-forum.com/.

On 8 Sep 2009, at 14:01, Marnen Laibow-Koser wrote:
>
> Piero Di Bello wrote:
>> Hi
>> How can I have the tests for my Rails app to be executed in a random
>> order?
>
> At least with RSpec, specs are executed in unpredictable order.  I  
> think
> it''s the same each time, but it''s not the order in which
they appear
> in
> the spec file.
>
> But why do you need this anyway? Each test should start from a clean
> slate, so the order should be completely immaterial.
Should, but it is possible for this not to be true. A random order  
would be hard to work with though - Run your suite and a test fails,  
run it again and it doesn''t any more.

Fred

Marnen Laibow-Koser wrote:
> Piero Di Bello wrote:
>> Hi
>> How can I have the tests for my Rails app to be executed in a random
>> order? 
> 
> At least with RSpec, specs are executed in unpredictable order.  I think 
> it''s the same each time, but it''s not the order in which
they appear in
> the spec file.
> 
> But why do you need this anyway? Each test should start from a clean 
> slate, so the order should be completely immaterial.
First I''ll say I completely agree with this. If tests/specs fail or
pass
depending on the order of execution then there must be serious problems 
with your test suite.

Now to address the OP. I know of nothing "built-in" any of the testing
frameworks for this. What I suppose you would need to do is write your 
own script that searches though your project for all test classes then 
executes them in a random order.

This, however, would only randomize the testing classes, each test 
within each class would still execute in whatever order the test 
framework decided to run them. Which, would mean that you would probably 
also have to replace the test runner with your own that could randomize 
each method of each test class.

In conclusion if this is a solid requirement of your project, I''m
really
glad it''s your project and not mine. :)

Good luck.
-- 
Posted via http://www.ruby-forum.com/.

On Sep 8, 3:01 pm, Marnen Laibow-Koser <rails-mailing-l...@andreas-
s.net> wrote:
> Piero Di Bello wrote:
> > Hi
> > How can I have the tests for my Rails app to be executed in a random
> > order?
>
> At least with RSpec, specs are executed in unpredictable order.  I think
> it''s the same each time, but it''s not the order in which
they appear in
> the spec file.
>
Unfortunately we don''t use Rspec on our project, although I think we
may give it a try
> But why do you need this anyway? Each test should start from a clean
> slate, so the order should be completely immaterial.
Well, actually, the answer is in your words :-)
Each test *should* start from a clean state, but what if it doesn''t?
For us it''s a great value to have a suite of test executed every time
in a random order (better, every single test method in every test
class should be "shuffled").
Dependency between tests is a really bad anti-pattern, and we want to
know if this happens as soon as possible.

anyway, thanks for your response,
Piero
>
> Best,
> --
> Marnen Laibow-Koserhttp://www.marnen.org
> mar...-sbuyVjPbboAdnm+yROfE0A@public.gmane.org
> --
> Posted viahttp://www.ruby-forum.com/.

On Sep 8, 9:27 pm, Robert Walker
<rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org>
wrote:
> Marnen Laibow-Koser wrote:
> > Piero Di Bello wrote:
> >> Hi
> >> How can I have the tests for my Rails app to be executed in a
random
> >> order?
>
> > At least with RSpec, specs are executed in unpredictable order.  I
think
> > it''s the same each time, but it''s not the order in
which they appear in
> > the spec file.
>
> > But why do you need this anyway? Each test should start from a clean
> > slate, so the order should be completely immaterial.
>
> First I''ll say I completely agree with this. If tests/specs fail
or pass
> depending on the order of execution then there must be serious problems
> with your test suite.
>
> Now to address the OP. I know of nothing "built-in" any of the
testing
> frameworks for this. What I suppose you would need to do is write your
> own script that searches though your project for all test classes then
> executes them in a random order.
>
> This, however, would only randomize the testing classes, each test
> within each class would still execute in whatever order the test
> framework decided to run them. Which, would mean that you would probably
> also have to replace the test runner with your own that could randomize
> each method of each test class.
>
> In conclusion if this is a solid requirement of your project, I''m
really
> glad it''s your project and not mine. :)
Thanks for your ideas. I''ll definitely explore some of them.
No, it''s not a solid requirement, but as I already said in a previous
answer, we find it *very* useful to discover dependencies between
tests as soon as possible (and probably the Rspec authors'' find this
important too, given that Rspec execute specs in a random order...).

Thanks
Piero
>
> Good luck.
> --
> Posted viahttp://www.ruby-forum.com/.

On Sep 8, 3:44 pm, Frederick Cheung
<frederick.che...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> On 8 Sep 2009, at 14:01, Marnen Laibow-Koser wrote:
>
>
>
> > Piero Di Bello wrote:
> >> Hi
> >> How can I have the tests for my Rails app to be executed in a
random
> >> order?
>
> > At least with RSpec, specs are executed in unpredictable order.  I  
> > think
> > it''s the same each time, but it''s not the order in
which they appear  
> > in
> > the spec file.
>
> > But why do you need this anyway? Each test should start from a clean
> > slate, so the order should be completely immaterial.
>
> Should, but it is possible for this not to be true. A random order  
> would be hard to work with though - Run your suite and a test fails,  
> run it again and it doesn''t any more.
The whole point of randomizing test execution is to have a better
chance to discover that the tests are not independent.  Once you find
a test that fails because of the order of execution, you fix it so
that it does not happen anymore.

Matteo

On Sep 9, 7:18 am, Matteo Vaccari
<matteo.vacc...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
wrote:
> On Sep 8, 3:44 pm, Frederick Cheung
<frederick.che...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> wrote:
> > Should, but it is possible for this not to be true. A random order  
> > would be hard to work with though - Run your suite and a test fails,  
> > run it again and it doesn''t any more.
>
> The whole point of randomizing test execution is to have a better
> chance to discover that the tests are not independent.  Once you find
> a test that fails because of the order of execution, you fix it so
> that it does not happen anymore.
>
I get that, but unless you could say ''run the tests again but in the
same random order as before'' how are you going to be sure that
you''ve
actually fixed your code/tests ?

Fred
> Matteo