Hi, Can somebody point me in the right direction about what approach would be best to load-balance between two or more physical servers running backgroundrb while using BackgrounDRb::Worker::RailsBase with mongrel? Thanks, Robert Bjarnason
We chose to bypass MiddleMan and place our requests for workers into a queue. This way we can create an arbitrary number of workers that loop and ask the queue for work. The Mongrels have no need to know the location or number of workers. You will of course need another strategy to get progress/results from the workers. On Jan 11, 2007, at 4:41 PM, Robert Bjarnason wrote:> Hi, > > Can somebody point me in the right direction about what approach would > be best to load-balance between two or more physical servers running > backgroundrb while using BackgrounDRb::Worker::RailsBase with > mongrel? > > Thanks, > Robert Bjarnason > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
Hi Erik, Thanks for the pointer, I have used JavaSpaces in the past and really like this pattern. However, my workers are user initiated and trigger x264 and mp4box video encoding/editing processes and I''m planning to have a limit of 1-2 simulations backgroundrb processes running on each physical server. I have an entry in the database for each user initiated job, maybe I can add the IP of each backgroundrb server to that entry and do the load-balancing in Rails using the database? The question then is how do I dynamically change the backgroundrb server IP for each instantiated MiddleMan object? Warm regards, Robert Bjarnason ps. here are links to the sites that are currently using backgroundrb-2.1 more or less successfully in production mode: http://store.longwayround.com/ http://store.missingface.com/ http://store.racetodakar.com/ http://store.murderormutiny.com/ Erik Morton wrote:> We chose to bypass MiddleMan and place our requests for workers into a > queue. This way we can create an arbitrary number of workers that loop > and ask the queue for work. The Mongrels have no need to know the > location or number of workers. You will of course need another > strategy to get progress/results from the workers. > > On Jan 11, 2007, at 4:41 PM, Robert Bjarnason wrote: > >> Hi, >> >> Can somebody point me in the right direction about what approach would >> be best to load-balance between two or more physical servers running >> backgroundrb while using BackgrounDRb::Worker::RailsBase with mongrel? >> >> Thanks, >> Robert Bjarnason >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > >
* Robert Bjarnason (robert.bjarnason at gmail.com) [070111 16:56]:> Hi Erik, > > Thanks for the pointer, I have used JavaSpaces in the past and really > like this pattern. > > However, my workers are user initiated and trigger x264 and mp4box video > encoding/editing processes and I''m planning to have a limit of 1-2 > simulations backgroundrb processes running on each physical server. > > I have an entry in the database for each user initiated job, maybe I can > add the IP of each backgroundrb server to that entry and do the > load-balancing in Rails using the database? The question then is how do > I dynamically change the backgroundrb server IP for each instantiated > MiddleMan object?you can use BackgrounDRb::MiddleManDRbObject and instead of calling the MiddleMan constant, use a regular object - actually it''s a class method which takes a DRb uri as it''s argument - behaves just like the MiddleMan. /skaar> > Warm regards, > Robert Bjarnason > > ps. here are links to the sites that are currently using > backgroundrb-2.1 more or less successfully in production mode: > http://store.longwayround.com/ > http://store.missingface.com/ > http://store.racetodakar.com/ > http://store.murderormutiny.com/ > > Erik Morton wrote: > > We chose to bypass MiddleMan and place our requests for workers into a > > queue. This way we can create an arbitrary number of workers that loop > > and ask the queue for work. The Mongrels have no need to know the > > location or number of workers. You will of course need another > > strategy to get progress/results from the workers. > > > > On Jan 11, 2007, at 4:41 PM, Robert Bjarnason wrote: > > > >> Hi, > >> > >> Can somebody point me in the right direction about what approach would > >> be best to load-balance between two or more physical servers running > >> backgroundrb while using BackgrounDRb::Worker::RailsBase with mongrel? > >> > >> Thanks, > >> Robert Bjarnason > >> > >> _______________________________________________ > >> Backgroundrb-devel mailing list > >> Backgroundrb-devel at rubyforge.org > >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel-- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ----------------------------------------------------------------------
Hi,
FYI: Here is how I ended up implemented the load-balancing for our sites:
1. Create an array of available backgroundrb servers that includes how
many workers can be active simultaneously:
BACKGROUNDRB_SERVERS = [ ["druby://video1:2223", 3],
["druby://video2:2223", 1] , ... ]
2. When a middleman is needed I iterate over the array, connect to each
server until I find a backgroundrb server that has a free slot.
3. Use BackgrounDRb::MiddleManDRbObject.init(:uri =>
"#{server[0]}") to
open the connection to the selected server
4. If no server is free then the client request is placed in a queue and
the user is informed.
5. Use self.delete in workers running on the servers, taking that
responsibility away from client initiated actions.
6. Use the database to store the progress to get around that results bug.
Note: I tried the setup with ruby-1.8.5-p12 and it has a silent crash
that was not detected by begin/rescue when running more than one
process, everything still works fine with ruby-1.8.4.
The only thing I haven''t figured out yet is how I close the connection
from the client, self.delete doesn''t close the middleman connection
from
the server side. The symptom is an ever growing list of established
connections between clients and servers. I couldn''t find an API in the
middleman to close the connection explicitly - I think it must be the
opposite to "DRb.start_service" but can''t find anything in
the
backgroundrb code that would do this. Any ideas?
Thanks,
Robert Bjarnason
skaar wrote:> * Robert Bjarnason (robert.bjarnason at gmail.com) [070111 16:56]:
>
>> Hi Erik,
>>
>> Thanks for the pointer, I have used JavaSpaces in the past and really
>> like this pattern.
>>
>> However, my workers are user initiated and trigger x264 and mp4box
video
>> encoding/editing processes and I''m planning to have a limit of
1-2
>> simulations backgroundrb processes running on each physical server.
>>
>> I have an entry in the database for each user initiated job, maybe I
can
>> add the IP of each backgroundrb server to that entry and do the
>> load-balancing in Rails using the database? The question then is how
do
>> I dynamically change the backgroundrb server IP for each instantiated
>> MiddleMan object?
>>
>
> you can use BackgrounDRb::MiddleManDRbObject and instead of calling the
> MiddleMan constant, use a regular object - actually it''s a class
method
> which takes a DRb uri as it''s argument - behaves just like the
> MiddleMan.
>
> /skaar
>
>
>
>> Warm regards,
>> Robert Bjarnason
>>
>> ps. here are links to the sites that are currently using
>> backgroundrb-2.1 more or less successfully in production mode:
>> http://store.longwayround.com/
>> http://store.missingface.com/
>> http://store.racetodakar.com/
>> http://store.murderormutiny.com/
>>
>> Erik Morton wrote:
>>
>>> We chose to bypass MiddleMan and place our requests for workers
into a
>>> queue. This way we can create an arbitrary number of workers that
loop
>>> and ask the queue for work. The Mongrels have no need to know the
>>> location or number of workers. You will of course need another
>>> strategy to get progress/results from the workers.
>>>
>>> On Jan 11, 2007, at 4:41 PM, Robert Bjarnason wrote:
>>>
>>>
>>>> Hi,
>>>>
>>>> Can somebody point me in the right direction about what
approach would
>>>> be best to load-balance between two or more physical servers
running
>>>> backgroundrb while using BackgrounDRb::Worker::RailsBase with
mongrel?
>>>>
>>>> Thanks,
>>>> Robert Bjarnason
>>>>
>>>> _______________________________________________
>>>> Backgroundrb-devel mailing list
>>>> Backgroundrb-devel at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>>
>>>
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>
>
>