Backgroundrb devel - Nov 2007 - Backgroundrb with Load Balancing Rails engines

Hi there,

We run several (3-4) Rails servers behind the reverse proxy / load
balancing web server serving the  single application. All Rails
instances share the same database and use the database as a session
storage. Memcached is also part of the picture.

I''m looking into using Backgroundrb for some large uploads / parsing
task that provide progress status updates (via ajax calls). Bdrb would
run on each Rails server using drbunix (domain sockets) and not on the
dedicated / shared Bdrb server.

In this scenario it is not known in which Rails engine will the status
update request land. It may or may not hit the engine where the Bdrb
worker has been created( we don''t plan for session affinity nor
anything similar). As mentioned earlier the only shared component is
the database, and I was planning to introduce a
''background_jobs''
table - keyed with the worker key and storing the key in the session.
In addition to the worker key the table would have a blob data where
the updates are stored. Then the MiddleMan is basically used only for
starting the job and  the results could be read by any engine.

This looks to me as a fairly ''generic'' requirement and I was
wandering
whether someone had a similar requirement and what approach was taken,
whether there is something that is part of the framework already. Also
any comments / suggestions are much appreciated.

cheers,
emil

On Nov 16, 2007 6:16 PM, Emil Marceta <emarceta at gmail.com>
wrote:
> Hi there,
>
> We run several (3-4) Rails servers behind the reverse proxy / load
> balancing web server serving the  single application. All Rails
> instances share the same database and use the database as a session
> storage. Memcached is also part of the picture.
>
> I''m looking into using Backgroundrb for some large uploads /
parsing
> task that provide progress status updates (via ajax calls). Bdrb would
> run on each Rails server using drbunix (domain sockets) and not on the
> dedicated / shared Bdrb server.
Any reasons, to not run bdrb server centrally? Guaranteed, you get the
failover measures and stuff,
but any other reasons?

>
> In this scenario it is not known in which Rails engine will the status
> update request land. It may or may not hit the engine where the Bdrb
> worker has been created( we don''t plan for session affinity nor
> anything similar). As mentioned earlier the only shared component is
> the database, and I was planning to introduce a
''background_jobs''
> table - keyed with the worker key and storing the key in the session.
> In addition to the worker key the table would have a blob data where
> the updates are stored. Then the MiddleMan is basically used only for
> starting the job and  the results could be read by any engine.
>
> This looks to me as a fairly ''generic'' requirement and I
was wandering
> whether someone had a similar requirement and what approach was taken,
> whether there is something that is part of the framework already. Also
> any comments / suggestions are much appreciated.
Ok, Are you aware of the new version of bdrb? Its a complete rewrite.
I would advise, you to checkout new version and implement your project on
top of it.

In  new bdrb, master process is nothing but a tcp server and hence you can
connect from any of the machines in your cluster.

checkout the code from here:

http://svn.devjavu.com/backgroundrb/branches/version10/

go through the read file, it should be suffice to get started.




-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org