Backgroundrb devel - Nov 2006 - how many processes?

Hello,

I''m currently using backgroundrb to fetch feeds for an RSS aggregator.
So as
you can imagine, backgroundrb is a very critical piece of the app. I was
wondering what a recommended pool size would be for this kind of
application. Ideally, I''d like to have as many processes in the pool as
possible, but what would be too many?

Thank you,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061111/f68ac5ac/attachment.html

On Nov 11, 2006, at 12:48 PM, David Lee wrote:
> Hello,
>
> I''m currently using backgroundrb to fetch feeds for an RSS  
> aggregator. So as you can imagine, backgroundrb is a very critical  
> piece of the app. I was wondering what a recommended pool size  
> would be for this kind of application. Ideally, I''d like to have
as
> many processes in the pool as possible, but what would be too many?
>
> Thank you,
> David
>
Hey David-

	That really depends on the hardware and memory you have to use as  
well as the kind of traffic and throughput you need to handle. Are  
you using ActiveRecord models in the workers? If so each worker  
requires quite a bit of memory. Without using AR its only 1 or 2 MB  
per process. If you drag rails into the mix each rails base workers  
is 20Mb or so.

	The new version also has a slower startup time for spawning workers.  
So the best way to approach development with the new code is to run a  
number of named immortal workers that boot when you start  
backgroundrb. Then these workers can always be working or can be  
repeatedly called to do the same job over and over. So if you can do  
without ActiveRecord then you can run many processes. If each worker  
has to laod rails then it adds up quick and you should use fewer  
workers.

	I know AR is great convenience but loading all of rails to accces  
the database is a lot of overhead. It may be best to use the mysql  
ruby bindings directly to query or insert to the db. Then you still  
get db access but you can run a lot more workers since you don''t have  
to load rails.

	But all in all I think somewhere around 10-20 workers that don''t use  
AR can do a *ton* of work.

Cheers-

-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)