Rails core - Mar 2009 - Improving Testing in rails and ruby as a probable GSOC project

hi all,
   a very rough idea has been developing in my mind as to improving
testing in general in ruby and rails to be a probable project.
        One of the ideas implemented to a certain extent and suggested
is by brynary to parallelize testing and using bonjour for auto
discovery of test servers.
  But i had a chat with him on irc and that in itself is not something
he believes requires work to last the duration of GSOC.  So just
wanted some pointers from people about whether there has been requests
or a need from people to improve the test architecture that exists in
rails and ruby.
    If someone can give me direction to discussions people have had
regarding this it would be great help in expanding this idea.  And
also do people think that there is a need to touch the testing modules
right now or is the way things work right now satisfactory for people
to give very little priority to this?

Abhas Saroha

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

On Sun, Mar 29, 2009 at 12:13 PM, kaluza_klein <abhassaroha@gmail.com>
wrote:
>        One of the ideas implemented to a certain extent and suggested
> is by brynary to parallelize testing and using bonjour for auto
> discovery of test servers.
Shared test server farms are a lot of work and effort to get right.  I
was just discussing this with someone yesterday.  This is even more so
in the ruby world, where the environment can be really touchy (a wrong
gem version and everything can fall apart, unless you vendor your
entire world or lock down every dependency version, neither of which
are very practical - and this isn''t even considering native dependency
versions such as imagemagick, browser versions, etc, etc).

Before going down this road, you should carefully weigh the costs of
dedicated test hardware (which is very fixed, especially with cloud
computing) against the engineering labor to maintain reliable shared
test servers, and all of the associated environment-management issues.
 Are you really going to save money, effort, complexity, and time in
the long run?

-- Chad

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

>
> Before going down this road, you should carefully weigh the costs of
> dedicated test hardware (which is very fixed, especially with cloud
> computing) against the engineering labor to maintain reliable shared
> test servers, and all of the associated environment-management issues.
>  Are you really going to save money, effort, complexity, and time in
> the long run?
>

the way i see it is that if a parallel testing implementation exists then
implementing it for lets say a designated server farm or a cloud should be a
similar thing. If you have multiple instances running on a cloud and they
are able to identify themselves to each other through bonjour then
isn''t it
the same situation as local hardware of an organization being able to
communicate and distribute a task? please correct me if i am oversimplifying
things...
   And if so wouldn''t it be good if an implementation exists in rails
itself
rather than each organization having to do it for its own needs?

Abhas Saroha

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

kaluza_klein wrote:
>   But i had a chat with him on irc and that in itself is not something
> he believes requires work to last the duration of GSOC.  So just
> wanted some pointers from people about whether there has been requests
> or a need from people to improve the test architecture that exists in
> rails and ruby.
To clarify, I said adding autodiscovery support to Testjour (which
currently lacks it since I rewrote it a couple months ago) was a few
day project, but there is plenty of work to be done on Testjour as a
distributed testing tool including:

* Fault tolerance
* Locking servers
* Supporting the cloud
* Supporting Test::Unit or RSpec (it''s Cucumber only currently)

...and probably a lot more.

Cheers,

-Bryan

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

sorry for misunderstanding and misquoting you bryan.
Well this list does give me a lot to ponder over then.

Abhas Saroha.
On Mon, Mar 30, 2009 at 6:34 AM, Bryan Helmkamp <brynaryh@gmail.com>
wrote:
>
> kaluza_klein wrote:
> >   But i had a chat with him on irc and that in itself is not something
> > he believes requires work to last the duration of GSOC.  So just
> > wanted some pointers from people about whether there has been requests
> > or a need from people to improve the test architecture that exists in
> > rails and ruby.
>
> To clarify, I said adding autodiscovery support to Testjour (which
> currently lacks it since I rewrote it a couple months ago) was a few
> day project, but there is plenty of work to be done on Testjour as a
> distributed testing tool including:
>
> * Fault tolerance
> * Locking servers
> * Supporting the cloud
> * Supporting Test::Unit or RSpec (it''s Cucumber only currently)
>
> ...and probably a lot more.
>
> Cheers,
>
> -Bryan
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

Abhas,

I just wanted to chime in on some of the interesting problems you
could potentially tackle in this space. We''ve been thinking about this
stuff quite a bit here at Devver, since we''re building a commercial
solution to the same problem (we currently distribute Test::Unit tests
and are working on supporting RSpec). Although our solution is
currently
closed-source, we''re excited about seeing an open-source alternative
since this is a big, challenging space and both solutions could really
help out the Ruby community.

Bryan - I haven''t looked too closely at Testjour yet, so I''m
frankly
not sure if you''ve already tackled any of these problems. If so,
please ignore them.

Test Detection - It''s certainly possible to break up tests on a
file-by-file basis, but some performance improvements and interesting
features (see Test Ordering below) can be done if the tests are broken
up on a per-test level. However, this can be tricky to do. We''re
currently experimenting with both approaches. Our per-test detection
can take a fair amount of time since it actually loads up the project
(which ultimately you have to do, since Ruby can dynamically generate
test methods). However, I suspect you could speed up the majority of
cases by using a parser and falling back to loading code (or executing
the entire file) when methods are dynamically generated.

Test Ordering - We''re finding that for long-running test suites, just
distributing the tests can shorten the run time by 66-75%. However, it
seems that you could do better by figuring out the the optimal order
of running tests (e.g. run the longest-running tests first) and
determining which files are worth breaking up into individual tests
and which should be run as a file. Furthermore, you could make runs
seem faster by giving users the most useful information early. For
example, you could first run the tests that relate the code being
changed (like Autotest, but potentially more accurate), then tests
that tend to break often, then everything else.

Test Dependency Detection - In a perfect world, the tests in any test
suite should be able to run in any order, because they would all have
no side effects. Unfortunately, that tends to not be true for some
projects. It would be useful to have the system be able to
automatically detect which tests need to run before/after other tests
and keep this information around. This information would not only help
people more easily move their project to Testjour, it would also be
useful information about a way to improve their test suite.

Those are a few ideas that come to mind. I''d say that the features
Bryan mentioned are probably more important in the short term, but if
you''re looking for difficult features to work on, there''s
plenty to
pick from :)

Let me know if you have any questions - I''m more than happy to help.

Ben

On Mar 29, 7:16 pm, abhas saroha <abhassar...@gmail.com>
wrote:
> sorry for misunderstanding and misquoting you bryan.
> Well this list does give me a lot to ponder over then.
>
> Abhas Saroha.
>
> On Mon, Mar 30, 2009 at 6:34 AM, Bryan Helmkamp <bryna...@gmail.com>
wrote:
>
> > kaluza_klein wrote:
> > >   But i had a chat with him on irc and that in itself is not
something
> > > he believes requires work to last the duration of GSOC.  So just
> > > wanted some pointers from people about whether there has been
requests
> > > or a need from people to improve the test architecture that
exists in
> > > rails and ruby.
>
> > To clarify, I said adding autodiscovery support to Testjour (which
> > currently lacks it since I rewrote it a couple months ago) was a few
> > day project, but there is plenty of work to be done on Testjour as a
> > distributed testing tool including:
>
> > * Fault tolerance
> > * Locking servers
> > * Supporting the cloud
> > * Supporting Test::Unit or RSpec (it''s Cucumber only
currently)
>
> > ...and probably a lot more.
>
> > Cheers,
>
> > -Bryan
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---

Thank you Ben for pointing out these points. I think these points  can be
taken account during the actual integration of testing modules with parallel
architecture.
       I will consult bryan about the direction of this project and see if
these points can be included.
I have submitted a draft of my proposal and you are welcome to comment on
it.
Abhas Saroha




On Tue, Mar 31, 2009 at 6:26 AM, Ben Brinckerhoff <ben@devver.net> wrote:
>
> Abhas,
>
> I just wanted to chime in on some of the interesting problems you
> could potentially tackle in this space. We''ve been thinking about
this
> stuff quite a bit here at Devver, since we''re building a
commercial
> solution to the same problem (we currently distribute Test::Unit tests
> and are working on supporting RSpec). Although our solution is
> currently
> closed-source, we''re excited about seeing an open-source
alternative
> since this is a big, challenging space and both solutions could really
> help out the Ruby community.
>
> Bryan - I haven''t looked too closely at Testjour yet, so
I''m frankly
> not sure if you''ve already tackled any of these problems. If so,
> please ignore them.
>
> Test Detection - It''s certainly possible to break up tests on a
> file-by-file basis, but some performance improvements and interesting
> features (see Test Ordering below) can be done if the tests are broken
> up on a per-test level. However, this can be tricky to do. We''re
> currently experimenting with both approaches. Our per-test detection
> can take a fair amount of time since it actually loads up the project
> (which ultimately you have to do, since Ruby can dynamically generate
> test methods). However, I suspect you could speed up the majority of
> cases by using a parser and falling back to loading code (or executing
> the entire file) when methods are dynamically generated.
>
> Test Ordering - We''re finding that for long-running test suites,
just
> distributing the tests can shorten the run time by 66-75%. However, it
> seems that you could do better by figuring out the the optimal order
> of running tests (e.g. run the longest-running tests first) and
> determining which files are worth breaking up into individual tests
> and which should be run as a file. Furthermore, you could make runs
> seem faster by giving users the most useful information early. For
> example, you could first run the tests that relate the code being
> changed (like Autotest, but potentially more accurate), then tests
> that tend to break often, then everything else.
>
> Test Dependency Detection - In a perfect world, the tests in any test
> suite should be able to run in any order, because they would all have
> no side effects. Unfortunately, that tends to not be true for some
> projects. It would be useful to have the system be able to
> automatically detect which tests need to run before/after other tests
> and keep this information around. This information would not only help
> people more easily move their project to Testjour, it would also be
> useful information about a way to improve their test suite.
>
> Those are a few ideas that come to mind. I''d say that the features
> Bryan mentioned are probably more important in the short term, but if
> you''re looking for difficult features to work on, there''s
plenty to
> pick from :)
>
> Let me know if you have any questions - I''m more than happy to
help.
>
> Ben
>
> On Mar 29, 7:16 pm, abhas saroha <abhassar...@gmail.com> wrote:
> > sorry for misunderstanding and misquoting you bryan.
> > Well this list does give me a lot to ponder over then.
> >
> > Abhas Saroha.
> >
> > On Mon, Mar 30, 2009 at 6:34 AM, Bryan Helmkamp
<bryna...@gmail.com>
> wrote:
> >
> > > kaluza_klein wrote:
> > > >   But i had a chat with him on irc and that in itself is not
> something
> > > > he believes requires work to last the duration of GSOC.  So
just
> > > > wanted some pointers from people about whether there has
been
> requests
> > > > or a need from people to improve the test architecture that
exists in
> > > > rails and ruby.
> >
> > > To clarify, I said adding autodiscovery support to Testjour
(which
> > > currently lacks it since I rewrote it a couple months ago) was a
few
> > > day project, but there is plenty of work to be done on Testjour
as a
> > > distributed testing tool including:
> >
> > > * Fault tolerance
> > > * Locking servers
> > > * Supporting the cloud
> > > * Supporting Test::Unit or RSpec (it''s Cucumber only
currently)
> >
> > > ...and probably a lot more.
> >
> > > Cheers,
> >
> > > -Bryan
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Core" group.
To post to this group, send email to rubyonrails-core@googlegroups.com
To unsubscribe from this group, send email to
rubyonrails-core+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/rubyonrails-core?hl=en
-~----------~----~----~----~------~----~------~--~---