rspec users - Sep 2008 - Running cucumber distributed across hardware

At my job, our story suite contains over 500 scenarios. While we''re
very happy with the regressions we''ve caught by running the suite, the
long time it takes (~ 10+ minutes) is starting to lead to developers
checking in without running it. This is leading to broken builds, and
the problems cascade from there (not being able to integrate, etc.)

What I''m wondering is if anyone has had success running Cucumber in a
parallel/distributed mode, like I know some people have done with
RSpec proper. I''m optimistic that if we can get the build time scaling
linearly with hardware, we can get the story run down to around one or
two minutes and keep it there regardless of how many scenarios we
introduce.

Thoughts?

Cheers,

-Bryan

-- 
Bryan Helmkamp
http://brynary.com -- My blog

On Mon, Sep 22, 2008 at 5:11 PM, Bryan Helmkamp <lists at brynary.com>
wrote:
> At my job, our story suite contains over 500 scenarios. While
we''re
> very happy with the regressions we''ve caught by running the suite,
the
> long time it takes (~ 10+ minutes) is starting to lead to developers
> checking in without running it. This is leading to broken builds, and
> the problems cascade from there (not being able to integrate, etc.)
>
I''m curious: could your unit tests be improved to minimize these
problems?
In other words, when your build breaks, does that indicate a missing unit
test?

///ark
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/rspec-users/attachments/20080922/0b2cc2ab/attachment-0001.html>

On Mon, Sep 22, 2008 at 8:32 PM, Mark Wilden <mark at mwilden.com>
wrote:
> I''m curious: could your unit tests be improved to minimize these
problems?
> In other words, when your build breaks, does that indicate a missing unit
> test?
That''s a very good point. Sometimes, yes. If the stories fail because
an object or layer of the stack is incorrect, that''s definitely a unit
test that is lacking which needs to be added. I think at those points
it''s also worth examining how the code became incorrect without a
failing unit test to identify root causes.

Often, however, the Story suite exposes an integration failure that
(isolated/fast) unit tests would not catch.

-- 
Bryan Helmkamp
http://brynary.com -- My blog

On Sep 22, 2008, at 8:11 PM, Bryan Helmkamp wrote:
> At my job, our story suite contains over 500 scenarios. While
we''re
> very happy with the regressions we''ve caught by running the suite,
the
> long time it takes (~ 10+ minutes) is starting to lead to developers
> checking in without running it. This is leading to broken builds, and
> the problems cascade from there (not being able to integrate, etc.)
>
Hey Bryan,

You know my thoughts on this -I think this problem will eventually be  
solved by Guillotine (or something like it).  Hopefully I should be  
getting a beta-version running up soon.  I''ll let you and the list  
know when I do.  I''ll also get some benchmarks running against FP or  
some other large test suite.
> What I''m wondering is if anyone has had success running Cucumber
in a
> parallel/distributed mode, like I know some people have done with
> RSpec proper. I''m optimistic that if we can get the build time
scaling
>
Regarding drb, as you know, there was a project named spec_distributed  
a while back which attempted it.  It''s major failing was that it could
only be run *after* check in. If you want something developers can run  
*before* check-in, you''ll not only need to distribute examples  
remotely, but also the code as well (I''m sure that''s why  
spec_distributed made the decision to run *after* check-in).

There was also a move to get Deeptest to work multi-core, and I''m not  
sure if that ever got done - it was buggy the last time I tried it.   
That might be a short temporarily solution, but I don''t think it will  
scale up.

Seriously, though, no one has done this well yet, even for rspec proper.
> linearly with hardware, we can get the story run down to around one or
> two minutes and keep it there regardless of how many scenarios we
> introduce.
AFAIK, There doesn''t seem to be an a priori reason why distributing  
workload via rinda *wouldn''t* work (you could, for example, distribute
feature files individually).

Let me know if you come up with something - I''d be interested in  
contributing to such a project.

Scott

On Mon, 22 Sep 2008 21:57:20 -0400, "Scott Taylor"
<mailing_lists at railsnewbie.com> said:
> 
> On Sep 22, 2008, at 8:11 PM, Bryan Helmkamp wrote:
> 
> > At my job, our story suite contains over 500 scenarios. While
we''re
> > very happy with the regressions we''ve caught by running the
suite, the
> > long time it takes (~ 10+ minutes) is starting to lead to developers
> > checking in without running it. This is leading to broken builds, and
> > the problems cascade from there (not being able to integrate, etc.)
> >
I''m obviously not as far down the path as ou guys in thinking about
this, but I can already see us having similar issues. Once idea I had
was just to create separate rails environments for features and specs,
thereby letting me run features at the same time as autotest is running.

I also wonder what we can do with in-memory databases, since our
integration tests don''t (so far) work on the scale of more than a
handful of records.

Anyone had any luck with either of these tricks?

cheers,
Matt

On Mon, Sep 22, 2008 at 9:57 PM, Scott Taylor
<mailing_lists at railsnewbie.com> wrote:
> Let me know if you come up with something - I''d be interested in
> contributing to such a project.
I started working on a project I''m calling Testjour to solve this at
http://github.com/brynary/testjour/tree/master .

It''s not at all usable for general consumption, but there''s a
base to
hack on. Here''s what I have working:

* Using Bonjour autodiscovery of available test running slaves
* Sending feature files to be run and results back and forth over DRb
* Setting up and tearing down new MySQL databases for each test run to
avoid conflicts

There''s plenty that needs to be done before it''s really
usable, specifically:

* Adding RSync support to transfer code to worker servers
* Daemonizing/detaching the slave process
* Adding logging and PID tracking
* And more in the TODO.txt

If I can get this running well, we''re going to purchase Mac Mini build
nodes for the office. My (ambitious) goal is to cut the story run time
on a developer machine from 12 minutes down to two.

Example from two Terminal windows:

$ testjour_slave
Broadcasting: Ready to run tests under name ''bhelmkamp'' on
port 51814...

Running tests from queue druby://Fozzie.local:51818 on PID 16482

----------------------

$ testjour_run features/authentication/authenticaton.feature
....F_...

1)
BOOM
/Users/bhelmkamp/p/weplay/features/steps/authentication.rb:10:in `When
"I login"''
/Users/bhelmkamp/p/weplay/features/authentication/authenticaton.feature:13:in
`When I login''


-- 
Bryan Helmkamp
http://brynary.com -- My blog

On Wed, Sep 24, 2008 at 7:31 AM, Bryan Helmkamp <lists at brynary.com>
wrote:
> On Mon, Sep 22, 2008 at 9:57 PM, Scott Taylor
> <mailing_lists at railsnewbie.com> wrote:
>> Let me know if you come up with something - I''d be interested
in
>> contributing to such a project.
>
> I started working on a project I''m calling Testjour to solve this
at
> http://github.com/brynary/testjour/tree/master .
>
That looks promising. You may also want to look at the now defunct
rspec-distributed, where we have some fancy Rinda mojo. Bob Cotton and
I did a lot on that 1-2 years back...

http://github.com/aslakhellesoy/rspec-distributed/tree/master

Aslak
> It''s not at all usable for general consumption, but
there''s a base to
> hack on. Here''s what I have working:
>
> * Using Bonjour autodiscovery of available test running slaves
> * Sending feature files to be run and results back and forth over DRb
> * Setting up and tearing down new MySQL databases for each test run to
> avoid conflicts
>
> There''s plenty that needs to be done before it''s really
usable, specifically:
>
> * Adding RSync support to transfer code to worker servers
> * Daemonizing/detaching the slave process
> * Adding logging and PID tracking
> * And more in the TODO.txt
>
> If I can get this running well, we''re going to purchase Mac Mini
build
> nodes for the office. My (ambitious) goal is to cut the story run time
> on a developer machine from 12 minutes down to two.
>
> Example from two Terminal windows:
>
> $ testjour_slave
> Broadcasting: Ready to run tests under name ''bhelmkamp''
on port 51814...
>
> Running tests from queue druby://Fozzie.local:51818 on PID 16482
>
> ----------------------
>
> $ testjour_run features/authentication/authenticaton.feature
> ....F_...
>
> 1)
> BOOM
> /Users/bhelmkamp/p/weplay/features/steps/authentication.rb:10:in `When
> "I login"''
>
/Users/bhelmkamp/p/weplay/features/authentication/authenticaton.feature:13:in
> `When I login''
>
>
> --
> Bryan Helmkamp
> http://brynary.com -- My blog
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>

On Sep 24, 2008, at 1:31 AM, Bryan Helmkamp wrote:
> On Mon, Sep 22, 2008 at 9:57 PM, Scott Taylor
> <mailing_lists at railsnewbie.com> wrote:
>> Let me know if you come up with something - I''d be interested
in
>> contributing to such a project.
>
> I started working on a project I''m calling Testjour to solve this
at
> http://github.com/brynary/testjour/tree/master .
>
> It''s not at all usable for general consumption, but
there''s a base to
> hack on. Here''s what I have working:
>
> * Using Bonjour autodiscovery of available test running slaves
> * Sending feature files to be run and results back and forth over DRb
> * Setting up and tearing down new MySQL databases for each test run to
> avoid conflicts
>
Sweet!  These are the bare-bones already.
> There''s plenty that needs to be done before it''s really
usable,
> specifically:
>
> * Adding RSync support to transfer code to worker servers
> * Daemonizing/detaching the slave process
> * Adding logging and PID tracking
> * And more in the TODO.txt
> \
The hard part is always dealing with concurrency - it''d be nice if you
could you other computers in the office which currently running specs  
to utilize their spare processes.  And it would be nice to get  
reporting - if a slave goes down, can you redistribute the steps to a  
new slave dynamically?

Good to hear about this.  Will look into it today.

Scott

Bryan Helmkamp wrote:
> On Mon, Sep 22, 2008 at 9:57 PM, Scott Taylor
> <mailing_lists at railsnewbie.com> wrote:
>> Let me know if you come up with something - I''d be interested
in
>> contributing to such a project.
> 
> I started working on a project I''m calling Testjour to solve this
at
> http://github.com/brynary/testjour/tree/master .
> 
> It''s not at all usable for general consumption, but
there''s a base to
> hack on. Here''s what I have working:
> 
> * Using Bonjour autodiscovery of available test running slaves
> * Sending feature files to be run and results back and forth over DRb
> * Setting up and tearing down new MySQL databases for each test run to
> avoid conflicts
> 
> There''s plenty that needs to be done before it''s really
usable,
> specifically:
> 
> * Adding RSync support to transfer code to worker servers
> * Daemonizing/detaching the slave process
> * Adding logging and PID tracking
> * And more in the TODO.txt
> 
> If I can get this running well, we''re going to purchase Mac Mini
build
> nodes for the office. My (ambitious) goal is to cut the story run time
> on a developer machine from 12 minutes down to two.
> 
> Example from two Terminal windows:
> 
> $ testjour_slave
> Broadcasting: Ready to run tests under name ''bhelmkamp''
on port 51814...
> 
> Running tests from queue druby://Fozzie.local:51818 on PID 16482
> 
> ----------------------
> 
> $ testjour_run features/authentication/authenticaton.feature
> ....F_...
> 
> 1)
> BOOM
> /Users/bhelmkamp/p/weplay/features/steps/authentication.rb:10:in `When
> "I login"''
>
/Users/bhelmkamp/p/weplay/features/authentication/authenticaton.feature:13:in
> `When I login''
I''m resurrecting this thread. This is becoming a major issue for us,
and
we have some allocated time coming into play soon to work on a solution.

What''s the current state of the art for distributed test runs? Is 
anybody doing it for real? As part of the build or before check-in? 
Where would be the best place to contribute effort? Testjour looks the 
most likely... Any others out there?

cheers,
Matt
-- 
Posted via http://www.ruby-forum.com/.

Matt Wynne wrote:
> Bryan Helmkamp wrote:
>   
>> On Mon, Sep 22, 2008 at 9:57 PM, Scott Taylor
>> <mailing_lists at railsnewbie.com> wrote:
>>     
>>> Let me know if you come up with something - I''d be
interested in
>>> contributing to such a project.
>>>       
>> I started working on a project I''m calling Testjour to solve
this at
>> http://github.com/brynary/testjour/tree/master .
>>
>> It''s not at all usable for general consumption, but
there''s a base to
>> hack on. Here''s what I have working:
>>
>> * Using Bonjour autodiscovery of available test running slaves
>> * Sending feature files to be run and results back and forth over DRb
>> * Setting up and tearing down new MySQL databases for each test run to
>> avoid conflicts
>>
>> There''s plenty that needs to be done before it''s
really usable,
>> specifically:
>>
>> * Adding RSync support to transfer code to worker servers
>> * Daemonizing/detaching the slave process
>> * Adding logging and PID tracking
>> * And more in the TODO.txt
>>
>> If I can get this running well, we''re going to purchase Mac
Mini build
>> nodes for the office. My (ambitious) goal is to cut the story run time
>> on a developer machine from 12 minutes down to two.
>>
>> Example from two Terminal windows:
>>
>> $ testjour_slave
>> Broadcasting: Ready to run tests under name
''bhelmkamp'' on port 51814...
>>
>> Running tests from queue druby://Fozzie.local:51818 on PID 16482
>>
>> ----------------------
>>
>> $ testjour_run features/authentication/authenticaton.feature
>> ....F_...
>>
>> 1)
>> BOOM
>> /Users/bhelmkamp/p/weplay/features/steps/authentication.rb:10:in `When
>> "I login"''
>>
/Users/bhelmkamp/p/weplay/features/authentication/authenticaton.feature:13:in
>> `When I login''
>>     
>
> I''m resurrecting this thread. This is becoming a major issue for
us, and
> we have some allocated time coming into play soon to work on a solution.
>
> What''s the current state of the art for distributed test runs? Is 
> anybody doing it for real? As part of the build or before check-in? 
> Where would be the best place to contribute effort? Testjour looks the 
> most likely... 
If you are thinking Selenium then there is the Selenium grid direction.
> Any others out there?
>
> cheers,
> Matt
>   

-- 
Joseph Wilk
http://blog.josephwilk.net

On 2 Apr 2009, at 21:21, Joseph Wilk wrote:
> If you are thinking Selenium then there is the Selenium grid  
> direction.
Thanks. Right now we''re just running against the rails integration  
session using Webrat, but that does definitely look useful for when we  
get more Ajax heavy.

Matt Wynne
http://beta.songkick.com
http://blog.mattwynne.net

On Thu, Apr 2, 2009 at 4:14 PM, Matt Wynne <lists at ruby-forum.com>
wrote:
> What''s the current state of the art for distributed test runs? Is
> anybody doing it for real? As part of the build or before check-in?
> Where would be the best place to contribute effort? Testjour looks the
> most likely... Any others out there?
We (weplay) use Testjour on a daily basis.  That said, its a bit
temperamental and requires a bit of effort to get it up and running.
I''d love to see more adoption, and am happy to work with whomever is
interested on making it easier to get up and running.

-- 
Josh Knowles
phone: 509-979-1593
email:  joshknowles at gmail.com
web:    http://joshknowles.com

I''m thinking about trying to get TestJour in place to run on a single
system, but one with a crapload of processors, since that''s about the
only
way to make anything run reasonably on a Sparc ...

I think I remember seeing that there was a Bonjour protocol client for
Solaris, but I''m not sure where.  Of course, if there was some way I
could
specify a number of processes for cucumber to spawn, that would work just as
well for my particular case ....

Desparately need to do _something_, because right now, our Cucumber tests
take about 30 minutes to run ...  Ruby on Solaris/Sparc is *painfully* slow,
but that''s the architecture I''m stuck on ...


On Fri, Apr 3, 2009 at 3:54 PM, Josh Knowles <joshknowles at gmail.com>
wrote:
> On Thu, Apr 2, 2009 at 4:14 PM, Matt Wynne <lists at ruby-forum.com>
wrote:
>
> > What''s the current state of the art for distributed test
runs? Is
> > anybody doing it for real? As part of the build or before check-in?
> > Where would be the best place to contribute effort? Testjour looks the
> > most likely... Any others out there?
>
> We (weplay) use Testjour on a daily basis.  That said, its a bit
> temperamental and requires a bit of effort to get it up and running.
> I''d love to see more adoption, and am happy to work with whomever
is
> interested on making it easier to get up and running.
>
> --
> Josh Knowles
> phone: 509-979-1593
> email:  joshknowles at gmail.com
> web:    http://joshknowles.com
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>


-- 
// anything worth taking seriously is worth making fun of
// http://blog.devcaffeine.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/rspec-users/attachments/20090403/dd2e05d3/attachment-0001.html>