Backgroundrb devel - Aug 2008 - Multiple worker threads per type per server?

Hello,

I''ve recently integrated backgroundrb into my rails application and I
am very happy with how it is performing. We strictly use the
''enq'' job
queuing functionality and it works well for us. We have 2 clustered
web/app servers, both of which backgroundrb is running on. When
starting up backgroundrb, it looks for all the workers classes in the
default location (lib/workers), and starts up a packet worker for
each.

However, I am not clear on how to spin up multiple instances of a
worker on the same machine. One type of worker is very taxed and I''d
like to create several of him to facilitate processing queue contents
quicker. The documents seem to suggest using MiddleMan.new_worker, but
this has not worked well for me. I''m able to get the middleman to
create a new worker instance, but it doesn''t do any work to pull tasks
out of the queue.

How do you recommend I spin up multiples of one type of worker on one
machine for parallel processing of a job queue?

Thanks in advance,
Benjamin Smith

On Fri, Aug 22, 2008 at 5:06 AM, benjamin smith
<smith.benjamin at gmail.com> wrote:
> Hello,
>
> I''ve recently integrated backgroundrb into my rails application
and I
> am very happy with how it is performing. We strictly use the
''enq'' job
> queuing functionality and it works well for us. We have 2 clustered
> web/app servers, both of which backgroundrb is running on. When
> starting up backgroundrb, it looks for all the workers classes in the
> default location (lib/workers), and starts up a packet worker for
> each.
>
> However, I am not clear on how to spin up multiple instances of a
> worker on the same machine. One type of worker is very taxed and
I''d
> like to create several of him to facilitate processing queue contents
> quicker. The documents seem to suggest using MiddleMan.new_worker, but
> this has not worked well for me. I''m able to get the middleman to
> create a new worker instance, but it doesn''t do any work to pull
tasks
> out of the queue.
>
> How do you recommend I spin up multiples of one type of worker on one
> machine for parallel processing of a job queue?
Thats correct, since each enqueued task belongs to a worker with a
name and worker_key, it won''t be pulled out by workers, other than for
what it was created. Now, since each newly started worker, will have
its own new key, the arrangement won''t work.

You can perhaps, patch up bdrb_job_queue.rb and remove the worker_key
match restriction, make it configurable and submit a patch. ;)

Oooo, i think i get it now. I wasn''t explicitly setting a worker_key
when enqueuing jobs, so an empty string was being placed into the
bdrb_job_queues.worker_key field. When starting up the default worker
threads, they get started with a worker_key of "" (empty string), thus
there is a match. I see!

Now understanding that, after a bit of experimentation, it seems that
it is possible to start new workers with duplicate worker_keys. So, I
just used the console to start up another worker of the type I wanted
with a worker_key of empty string, and voila!

MiddleMan.new_worker(:worker=>:foo_worker, :worker_key=>"")

Now, I just need to figure out how to start multiples by default when
BackgroundRb spins up! Or, perhaps it makes better sense to have a
monitoring function set on a schedule that will spin up another worker
instances once the queue grows to a certain length...

Regardless, thanks for triggering the synapses needed for me to figure this out!

Ben