Backgroundrb devel - Apr 2008 - how to do and how not to do (in the background)

Hi everyone,

I recently upgraded to backgroundrb 1.0.3 (which is much more stable
than the old drb version). However, I am not sure whether backgroundrb
is the right solution nor whether I am using it the right way. For a
while now, I have followed various discussions out there about
background-fu, spawn, workling, startling, ap4r, sqs etc. Seems the
world of background processing is quite undecided.

I think the best way to start is to describe the three main use cases
(I think they are pretty universal). Feedback is highly appreciated.

Periodic Tasks - automatic tasks that execute at specific intervals or
certain times of day (think cron as in * */4 * * *). Example is
charging monthly subscriptions or sending nightly digest emails.

Background Tasks with status - user initiated tasks that need to
execute outside of the request / response for which the client
periodically check their status. Example is charging credit card at
checkout or generating a report.

Background Tasks without status - the spawn and forget tasks. Example
is solr commit.

In our world, we have multiple mongrels spread over more than one
physical machine for hardware redudancy. In addition, we have
monitoring (in my case monit) that watches processes and restarts them
if they crash or exceed limits.

Having all of this in mind, I currently have the following:

I have backgroundrb running on multiple machines. I don''t use
MiddleMan in RoR because that doesn''t deal with a backgroundrb or a
worker being down. Instead, I use the database to communicate between
RoR and the workers (I suppose home grown queue). RoR inserts a job to
be done. Workers polls the database, lock the job to ensure only one
worker executes the job, execute the jobs, and update the job progress
/ status. This essentially handles background tasks.

For periodic tasks, I have scheduled backgroundrb workers. Some of
them just perform an action like delete temporary files. Other ones
query the database to see what to do (i.e. who to charge monthly
subscription or who should receive the digest email). Depending on the
task, there is some status that tells who was already charged or who
already received the digest email to avoid doing something twice.

Does that sound reasonable?

Couple additional things I am not sure about ... I mentioned I use
monit to stop/stop/monitor backgroundrb server. But then the
backgroundrb server spawns off workers. At this point, I don''t see how
to monitor a worker and restart it were it to die. One workaround I
can think off is to have a periodic worker that is reloaded by
backgroundrb server that starts the other workers
(MiddleMan.start_worker will not create a new worker if one with same
task name and job key already exists). Can this same worker tell me
another worker has died (i.e. MiddleMan.all_worker_info and ensure all
workers are :running)?

Paul
ps. I always see people talking trash about backgroundrb because of
its previous implementation ... and there is hemant always trying to
fend off the assaults :). Good job!