I am looking for some advice on the best way to parallelize a large set of browser-based regression tests written in rspec. Just as a note; we are running these specs with RSpec 2.5 on JRuby 1.6.1. Our current set of specs takes about 4 hours to run when it is not parallelized. We have implemented a simple "bucket" parallelization scheme that basically takes each spec file and divides them evenly across a specified number of forked buckets. This simple solution has problems: * Some forked buckets finish early and exit. They don''t get the chance to contribute to finishing the remaining work. * It is somewhat difficult to aggregate all the results into one spot (Not really, but annoying). So... What do I want? 1. Is RSpec the right tool? If no, what would you suggest? 2. It seems like having a queue of specs and a thread pool would address my two points above. But... I don''t think RSpec is thread safe (Specifically RSpec::Core.world and RSpec::Core.configuration). Thoughts? 3. Other ideas? Thanks for any help you can give. AE
Have you already tried Parallel Tests? https://github.com/grosser/parallel_tests It helped a lot for me... Hope it helps, Rodrigo. Em 28-04-2011 16:54, Adam Esterline escreveu:> I am looking for some advice on the best way to parallelize a large > set of browser-based regression tests written in rspec. Just as a > note; we are running these specs with RSpec 2.5 on JRuby 1.6.1. > > Our current set of specs takes about 4 hours to run when it is not > parallelized. We have implemented a simple "bucket" parallelization > scheme that basically takes each spec file and divides them evenly > across a specified number of forked buckets. This simple solution > has problems: > > * Some forked buckets finish early and exit. They don''t get the > chance to contribute to finishing the remaining work. > * It is somewhat difficult to aggregate all the results into one > spot (Not really, but annoying). > > So... What do I want? > > 1. Is RSpec the right tool? If no, what would you suggest? > 2. It seems like having a queue of specs and a thread pool would > address my two points above. But... I don''t think RSpec is thread > safe (Specifically RSpec::Core.world and RSpec::Core.configuration). > Thoughts? > 3. Other ideas? > > Thanks for any help you can give. > > AE
You can also take a look at https://github.com/test-load-balancer Best, Sidu. http://c42.in http://about.me/ponnappa On 29 April 2011 01:24, Adam Esterline <adam at esterlines.com> wrote:> I am looking for some advice on the best way to parallelize a large > set of browser-based regression tests written in rspec. ? ?Just as a > note; we are running these specs with RSpec 2.5 on JRuby 1.6.1. > > Our current set of specs takes about 4 hours to run when it is not > parallelized. ? ?We have implemented a simple "bucket" parallelization > scheme that basically takes each spec file and divides them evenly > across a specified number of forked buckets. ? ?This simple solution > has problems: > > * ? Some forked buckets finish early and exit. ? ?They don''t get the > chance to contribute to finishing the remaining work. > * ? It is somewhat difficult to aggregate all the results into one > spot (Not really, but annoying). > > So... ?What do I want? > > 1. ? ?Is RSpec the right tool? ? ?If no, what would you suggest? > 2. ? ?It seems like having a queue of specs and a thread pool would > address my two points above. ? ?But... I don''t think RSpec is thread > safe (Specifically RSpec::Core.world and RSpec::Core.configuration). > ?Thoughts? > 3. ? ?Other ideas? > > Thanks for any help you can give. > > AE > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >
Both https://github.com/test-load-balancer and https://github.com/grosser/parallel_tests seem more complicated than needed. They both require other moving parts (database; other server). It seems now with ruby 1.9 and jruby a simpler solution (maybe harder to code) would be to use a queue and native threads. Thoughts? AE On Thu, Apr 28, 2011 at 5:53 PM, Sidu Ponnappa <ckponnappa at gmail.com> wrote:> You can also take a look at https://github.com/test-load-balancer > > Best, > Sidu. > http://c42.in > http://about.me/ponnappa > > On 29 April 2011 01:24, Adam Esterline <adam at esterlines.com> wrote: >> I am looking for some advice on the best way to parallelize a large >> set of browser-based regression tests written in rspec. ? ?Just as a >> note; we are running these specs with RSpec 2.5 on JRuby 1.6.1. >> >> Our current set of specs takes about 4 hours to run when it is not >> parallelized. ? ?We have implemented a simple "bucket" parallelization >> scheme that basically takes each spec file and divides them evenly >> across a specified number of forked buckets. ? ?This simple solution >> has problems: >> >> * ? Some forked buckets finish early and exit. ? ?They don''t get the >> chance to contribute to finishing the remaining work. >> * ? It is somewhat difficult to aggregate all the results into one >> spot (Not really, but annoying). >> >> So... ?What do I want? >> >> 1. ? ?Is RSpec the right tool? ? ?If no, what would you suggest? >> 2. ? ?It seems like having a queue of specs and a thread pool would >> address my two points above. ? ?But... I don''t think RSpec is thread >> safe (Specifically RSpec::Core.world and RSpec::Core.configuration). >> ?Thoughts? >> 3. ? ?Other ideas? >> >> Thanks for any help you can give. >> >> AE >> _______________________________________________ >> rspec-users mailing list >> rspec-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/rspec-users >> > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users
Wouldn''t using threads require that the code being tested also be threadsafe? Best, Sidu. http://c42.in http://about.me/ponnappa On 29 April 2011 08:49, Adam Esterline <adam at esterlines.com> wrote:> Both https://github.com/test-load-balancer and > https://github.com/grosser/parallel_tests seem more complicated than > needed. ? ?They both require other moving parts (database; other > server). ? ?It seems now with ruby 1.9 and jruby a simpler solution > (maybe harder to code) would be to use a queue and native threads. > > Thoughts? > > AE > > > > On Thu, Apr 28, 2011 at 5:53 PM, Sidu Ponnappa <ckponnappa at gmail.com> wrote: >> You can also take a look at https://github.com/test-load-balancer >> >> Best, >> Sidu. >> http://c42.in >> http://about.me/ponnappa >> >> On 29 April 2011 01:24, Adam Esterline <adam at esterlines.com> wrote: >>> I am looking for some advice on the best way to parallelize a large >>> set of browser-based regression tests written in rspec. ? ?Just as a >>> note; we are running these specs with RSpec 2.5 on JRuby 1.6.1. >>> >>> Our current set of specs takes about 4 hours to run when it is not >>> parallelized. ? ?We have implemented a simple "bucket" parallelization >>> scheme that basically takes each spec file and divides them evenly >>> across a specified number of forked buckets. ? ?This simple solution >>> has problems: >>> >>> * ? Some forked buckets finish early and exit. ? ?They don''t get the >>> chance to contribute to finishing the remaining work. >>> * ? It is somewhat difficult to aggregate all the results into one >>> spot (Not really, but annoying). >>> >>> So... ?What do I want? >>> >>> 1. ? ?Is RSpec the right tool? ? ?If no, what would you suggest? >>> 2. ? ?It seems like having a queue of specs and a thread pool would >>> address my two points above. ? ?But... I don''t think RSpec is thread >>> safe (Specifically RSpec::Core.world and RSpec::Core.configuration). >>> ?Thoughts? >>> 3. ? ?Other ideas? >>> >>> Thanks for any help you can give. >>> >>> AE >>> _______________________________________________ >>> rspec-users mailing list >>> rspec-users at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/rspec-users >>> >> _______________________________________________ >> rspec-users mailing list >> rspec-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/rspec-users > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >
Em 29-04-2011 00:19, Adam Esterline escreveu:> Both https://github.com/test-load-balancer and > https://github.com/grosser/parallel_tests seem more complicated than > needed. They both require other moving parts (database; other > server). It seems now with ruby 1.9 and jruby a simpler solution > (maybe harder to code) would be to use a queue and native threads. > > Thoughts?The multiple databases will be necessary even if you do have multiple threads if you are accessing your database in your tests, unless either: - You run all your examples that hit the database in a single thread, or; - You will have to careful choose how to write your tests in a thread-safe way, which will is very likely error prone and you should make your tests as simple as possible. On the other hand, having as many databases as many threads/processes you have is a much simpler solution and it is not that complicated since Rails can generate them for you: "rake parallel:create_databases" or something like that. Cheers, Rodrigo.> > On Thu, Apr 28, 2011 at 5:53 PM, Sidu Ponnappa<ckponnappa at gmail.com> wrote: >> You can also take a look at https://github.com/test-load-balancer >> >> Best, >> Sidu. >> http://c42.in >> http://about.me/ponnappa >> >> On 29 April 2011 01:24, Adam Esterline<adam at esterlines.com> wrote: >>> I am looking for some advice on the best way to parallelize a large >>> set of browser-based regression tests written in rspec. Just as a >>> note; we are running these specs with RSpec 2.5 on JRuby 1.6.1. >>> >>> Our current set of specs takes about 4 hours to run when it is not >>> parallelized. We have implemented a simple "bucket" parallelization >>> scheme that basically takes each spec file and divides them evenly >>> across a specified number of forked buckets. This simple solution >>> has problems: >>> >>> * Some forked buckets finish early and exit. They don''t get the >>> chance to contribute to finishing the remaining work. >>> * It is somewhat difficult to aggregate all the results into one >>> spot (Not really, but annoying). >>> >>> So... What do I want? >>> >>> 1. Is RSpec the right tool? If no, what would you suggest? >>> 2. It seems like having a queue of specs and a thread pool would >>> address my two points above. But... I don''t think RSpec is thread >>> safe (Specifically RSpec::Core.world and RSpec::Core.configuration). >>> Thoughts? >>> 3. Other ideas? >>> >>> Thanks for any help you can give. >>> >>> AE >>> __
On Fri, Apr 29, 2011 at 1:19 PM, Adam Esterline <adam at esterlines.com> wrote:> Both https://github.com/test-load-balancer and > https://github.com/grosser/parallel_tests seem more complicated than > needed. ? ?They both require other moving parts (database; other > server). ? ?It seems now with ruby 1.9 and jruby a simpler solution > (maybe harder to code) would be to use a queue and native threads. >Maybe dripdrop can make it easy to write some simple performant queues? The nice thing is that if you do need to grow beyond on machine zeromq makes that mauch less painful than it was before. There was a thread.new in DripDrop the last time I looked, and I haven''t had the time to see if making it optionnal (vs some observable) is possible/easy. HTH?> Thoughts? > > AE > > > > On Thu, Apr 28, 2011 at 5:53 PM, Sidu Ponnappa <ckponnappa at gmail.com> wrote: >> You can also take a look at https://github.com/test-load-balancer >> >> Best, >> Sidu. >> http://c42.in >> http://about.me/ponnappa >> >> On 29 April 2011 01:24, Adam Esterline <adam at esterlines.com> wrote: >>> I am looking for some advice on the best way to parallelize a large >>> set of browser-based regression tests written in rspec. ? ?Just as a >>> note; we are running these specs with RSpec 2.5 on JRuby 1.6.1. >>> >>> Our current set of specs takes about 4 hours to run when it is not >>> parallelized. ? ?We have implemented a simple "bucket" parallelization >>> scheme that basically takes each spec file and divides them evenly >>> across a specified number of forked buckets. ? ?This simple solution >>> has problems: >>> >>> * ? Some forked buckets finish early and exit. ? ?They don''t get the >>> chance to contribute to finishing the remaining work. >>> * ? It is somewhat difficult to aggregate all the results into one >>> spot (Not really, but annoying). >>> >>> So... ?What do I want? >>> >>> 1. ? ?Is RSpec the right tool? ? ?If no, what would you suggest? >>> 2. ? ?It seems like having a queue of specs and a thread pool would >>> address my two points above. ? ?But... I don''t think RSpec is thread >>> safe (Specifically RSpec::Core.world and RSpec::Core.configuration). >>> ?Thoughts? >>> 3. ? ?Other ideas? >>> >>> Thanks for any help you can give. >>> >>> AE >>> _______________________________________________ >>> rspec-users mailing list >>> rspec-users at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/rspec-users >>> >> _______________________________________________ >> rspec-users mailing list >> rspec-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/rspec-users > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users >-- ????'' ??? ??????, ???'' ?????? ?? ???? [The fox knows many things, but the hedgehog knows one big thing.] ? Archilochus, Greek poet (c. 680 BC ? c. 645 BC) http://wiki.hedgehogshiatus.com
On Fri, Apr 29, 2011 at 3:52 AM, Sidu Ponnappa <ckponnappa at gmail.com> wrote:> Wouldn''t using threads require that the code being tested also be threadsafe?Yes. In my case, we are testing a web application through the browser (regression tests). If the web application is not thread safe, we want to know about it :)> > Best, > Sidu. > http://c42.in > http://about.me/ponnappa > > On 29 April 2011 08:49, Adam Esterline <adam at esterlines.com> wrote: >> Both https://github.com/test-load-balancer and >> https://github.com/grosser/parallel_tests seem more complicated than >> needed. ? ?They both require other moving parts (database; other >> server). ? ?It seems now with ruby 1.9 and jruby a simpler solution >> (maybe harder to code) would be to use a queue and native threads. >> >> Thoughts? >> >> AE >> >> >> >> On Thu, Apr 28, 2011 at 5:53 PM, Sidu Ponnappa <ckponnappa at gmail.com> wrote: >>> You can also take a look at https://github.com/test-load-balancer >>> >>> Best, >>> Sidu. >>> http://c42.in >>> http://about.me/ponnappa >>> >>> On 29 April 2011 01:24, Adam Esterline <adam at esterlines.com> wrote: >>>> I am looking for some advice on the best way to parallelize a large >>>> set of browser-based regression tests written in rspec. ? ?Just as a >>>> note; we are running these specs with RSpec 2.5 on JRuby 1.6.1. >>>> >>>> Our current set of specs takes about 4 hours to run when it is not >>>> parallelized. ? ?We have implemented a simple "bucket" parallelization >>>> scheme that basically takes each spec file and divides them evenly >>>> across a specified number of forked buckets. ? ?This simple solution >>>> has problems: >>>> >>>> * ? Some forked buckets finish early and exit. ? ?They don''t get the >>>> chance to contribute to finishing the remaining work. >>>> * ? It is somewhat difficult to aggregate all the results into one >>>> spot (Not really, but annoying). >>>> >>>> So... ?What do I want? >>>> >>>> 1. ? ?Is RSpec the right tool? ? ?If no, what would you suggest? >>>> 2. ? ?It seems like having a queue of specs and a thread pool would >>>> address my two points above. ? ?But... I don''t think RSpec is thread >>>> safe (Specifically RSpec::Core.world and RSpec::Core.configuration). >>>> ?Thoughts? >>>> 3. ? ?Other ideas? >>>> >>>> Thanks for any help you can give. >>>> >>>> AE >>>> _______________________________________________ >>>> rspec-users mailing list >>>> rspec-users at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/rspec-users >>>> >>> _______________________________________________ >>> rspec-users mailing list >>> rspec-users at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/rspec-users >> _______________________________________________ >> rspec-users mailing list >> rspec-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/rspec-users >> > _______________________________________________ > rspec-users mailing list > rspec-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users