Backgroundrb devel - Nov 2006 - New Backgroundrb Speed Issues?

The new backgroundrb seemed to me to be markedly slower than the old  
one.  So, I ran a "side-by-side" comparison test using a worker that  
I had written which parses a CSV file and loads the results into a  
database.  The results?  The latest version appears to take twice as  
long to run the same code as the previous version.  Has anyone else  
experienced speed issues with the new version?

Regards,
Jason

Well, it forks a new process for every worker, instead of just spawning 
a thread. That''s not free, and the overall impact depends on how long 
your workers run.
On the positive side, you can launch a "persistent" worker, and then 
start threads inside that worker from MiddleMan. So for short workers 
you can emulate the old behavior, and presumably performance. (Well, 
there''s an additional IPC there.)
Thorsten

Jason Fox wrote:
> The new backgroundrb seemed to me to be markedly slower than the old  
> one.  So, I ran a "side-by-side" comparison test using a worker
that
> I had written which parses a CSV file and loads the results into a  
> database.  The results?  The latest version appears to take twice as  
> long to run the same code as the previous version.  Has anyone else  
> experienced speed issues with the new version?
>
> Regards,
> Jason
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
>

* Jason Fox (jfoxny at gmail.com) [061107 19:54]:
> The new backgroundrb seemed to me to be markedly slower than the old  
> one.  So, I ran a "side-by-side" comparison test using a worker
that
> I had written which parses a CSV file and loads the results into a  
> database.  The results?  The latest version appears to take twice as  
> long to run the same code as the previous version.  Has anyone else  
> experienced speed issues with the new version?
there is more setup when you are creating a new worker now, but
performance when that is done, shouldn''t be any worse. What kind of
difference are you seeing?

/skaar

-- 
----------------------------------------------------------------------
|\|\             where in the       |          s_u_b_s_t_r_u_c_t_i_o_n 
| | >===========  W.A.S.T.E.        |                  genarratologies 
|/|/    (_)     is the wisdom       |                  skaar at waste.org
----------------------------------------------------------------------

The working running under the old backgroundrb took approximately 14  
seconds to complete its task.  With the new worker it is taking 28  
seconds to perform the same amount of work under the same conditions  
(i.e., database state, system load, etc.).  I could attempt the test  
again with a larger file and see if the ratio holds.  Thoughts?

Regards,
Jason

On Nov 7, 2006, at 9:20 PM, skaar wrote:

> * Jason Fox (jfoxny at gmail.com) [061107 19:54]:
>
>> The new backgroundrb seemed to me to be markedly slower than the old
>> one.  So, I ran a "side-by-side" comparison test using a
worker that
>> I had written which parses a CSV file and loads the results into a
>> database.  The results?  The latest version appears to take twice as
>> long to run the same code as the previous version.  Has anyone else
>> experienced speed issues with the new version?
>>
>
> there is more setup when you are creating a new worker now, but
> performance when that is done, shouldn''t be any worse. What kind
of
> difference are you seeing?
>
> /skaar
>
> -- 
> ----------------------------------------------------------------------
> |\|\             where in the       |          s_u_b_s_t_r_u_c_t_i_o_n
> | | >===========  W.A.S.T.E.        |                  genarratologies
> |/|/    (_)     is the wisdom       |                  skaar at waste.org
> ----------------------------------------------------------------------
>

Jason are you starting a new worker each time you try this? Or do you  
have a long running worker you can call multiple times to do the same  
calculations? The new system does have more overhead for starting new  
workers. So it is better to design your system with a few core  
workers that always run and then you continue to call methods on  
these long workers.

The old system may be faster in a few cases and thats fine if it  
still works for you. This new version is much more scalable. I need  
it to be able to handle a significant number of things that each  
require the power of their own ruby interpreter. You can easily run  
into issues with ruby''s green threads in the old code.

That said the new code is still teething. Like we said we will have a  
rock solid production version by 0.3.0. We appreciate people testing  
the new version and please file bug reports on the trac:

	http://backgroundrb.devjavu.com/

Thanks
-Ezra


On Nov 7, 2006, at 6:29 PM, Jason Fox wrote:
> The working running under the old backgroundrb took approximately 14
> seconds to complete its task.  With the new worker it is taking 28
> seconds to perform the same amount of work under the same conditions
> (i.e., database state, system load, etc.).  I could attempt the test
> again with a larger file and see if the ratio holds.  Thoughts?
>
> Regards,
> Jason
>
> On Nov 7, 2006, at 9:20 PM, skaar wrote:
>
>
>> * Jason Fox (jfoxny at gmail.com) [061107 19:54]:
>>
>>> The new backgroundrb seemed to me to be markedly slower than the
old
>>> one.  So, I ran a "side-by-side" comparison test using a
worker that
>>> I had written which parses a CSV file and loads the results into a
>>> database.  The results?  The latest version appears to take twice
as
>>> long to run the same code as the previous version.  Has anyone else
>>> experienced speed issues with the new version?
>>>
>>
>> there is more setup when you are creating a new worker now, but
>> performance when that is done, shouldn''t be any worse. What
kind of
>> difference are you seeing?
>>
>> /skaar
>>
>> -- 
>> --------------------------------------------------------------------- 
>> -
>> |\|\             where in the       |           
>> s_u_b_s_t_r_u_c_t_i_o_n
>> | | >===========  W.A.S.T.E.        |                   
>> genarratologies
>> |/|/    (_)     is the wisdom       |                   
>> skaar at waste.org
>> --------------------------------------------------------------------- 
>> -
>>
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

Here''s what I did:

- Started the old backgroundrb
- Called the worker from my controller
- Recorded the time (~14 seconds)
- Shutdown the old backgroundrb
- Started the new backgroundrb
- Called the worker from my controller
- Recorded the time (~28 seconds)

So, basically, one call to each version right after starting them up.

Regards,
Jason

On Nov 7, 2006, at 9:37 PM, Ezra Zygmuntowicz wrote:
> Jason are you starting a new worker each time you try this? Or do  
> you have a long running worker you can call multiple times to do  
> the same calculations? The new system does have more overhead for  
> starting new workers. So it is better to design your system with a  
> few core workers that always run and then you continue to call  
> methods on these long workers.
>
> The old system may be faster in a few cases and thats fine if it  
> still works for you. This new version is much more scalable. I need  
> it to be able to handle a significant number of things that each  
> require the power of their own ruby interpreter. You can easily run  
> into issues with ruby''s green threads in the old code.
>
> That said the new code is still teething. Like we said we will have  
> a rock solid production version by 0.3.0. We appreciate people  
> testing the new version and please file bug reports on the trac:
>
> 	http://backgroundrb.devjavu.com/
>
> Thanks
> -Ezra
>
>
> On Nov 7, 2006, at 6:29 PM, Jason Fox wrote:
>
>> The working running under the old backgroundrb took approximately 14
>> seconds to complete its task.  With the new worker it is taking 28
>> seconds to perform the same amount of work under the same conditions
>> (i.e., database state, system load, etc.).  I could attempt the test
>> again with a larger file and see if the ratio holds.  Thoughts?
>>
>> Regards,
>> Jason
>>
>> On Nov 7, 2006, at 9:20 PM, skaar wrote:
>>
>>
>>> * Jason Fox (jfoxny at gmail.com) [061107 19:54]:
>>>
>>>> The new backgroundrb seemed to me to be markedly slower than
the
>>>> old
>>>> one.  So, I ran a "side-by-side" comparison test
using a worker
>>>> that
>>>> I had written which parses a CSV file and loads the results
into a
>>>> database.  The results?  The latest version appears to take  
>>>> twice as
>>>> long to run the same code as the previous version.  Has anyone
else
>>>> experienced speed issues with the new version?
>>>>
>>>
>>> there is more setup when you are creating a new worker now, but
>>> performance when that is done, shouldn''t be any worse.
What kind of
>>> difference are you seeing?
>>>
>>> /skaar
>>>
>>> -- 
>>>
--------------------------------------------------------------------
>>> --
>>> |\|\             where in the       |           
>>> s_u_b_s_t_r_u_c_t_i_o_n
>>> | | >===========  W.A.S.T.E.        |                   
>>> genarratologies
>>> |/|/    (_)     is the wisdom       |                   
>>> skaar at waste.org
>>>
--------------------------------------------------------------------
>>> --
>>>
>>
>>
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
> -- Ezra Zygmuntowicz-- Lead Rails Evangelist
> -- ez at engineyard.com
> -- Engine Yard, Serious Rails Hosting
> -- (866) 518-YARD (9273)
>
>