Backgroundrb devel - Mar 2008 - extending bdrb / running multiple servers

Hi,

Thanks for the help getting the new backgroundrb working, things are  
working fabulously now.

I have two questions:

1)  I have a few basic methods I need available on all my workers in  
order to communicate properly with my application... it seems to make  
the most sense right now to subclass BackgrounDRb::MetaWorker, or make  
a mixin module, because I''m actually extending some of the base  
methods as well (like the exit method, which I have extended to update  
status information in a central queue monitor).  Is there a proper  
place to put in custom code that extends BackgrounDrb, so that it is  
automatically loaded by the server before the workers are instantiated  
(sort of like an environment.rb)?

2)  I have one application server that runs Rails and does all the  
user interface, but several (10+) servers running backgroundrb to do  
all the heavy lifting (we do a lot of hi-res image processing,  
requires many CPU''s).  I have a central monitor that monitors what  
servers are available to run which tasks, and load balances the tasks  
across all the available backgroundrb servers, instantiating a  
MiddleMan for the correct server and then sending the job request to  
that server.  Needless to say, this process is going a lot easier with  
the new version, except for one catch.  With the old backgroundrb, I  
was able to connect to an arbitrary server like this:

thisMiddleMan = DRbObject.new(nil, "druby://#{server_name}:22223");

Now this doesn''t work.  So, I took a look at  
BackgrounDRb::WorkerProxy, and it seems like it can only connect to  
the server listed in the config file.  Is there supposed to be a way  
to manage multiple backgroundrb servers?  I''ve read the manual,
it''s
not in there.  My solution so far is to subclass WorkerProxy, with a  
new init method that takes the host and port as parameters:

class BackgrounDRb::CustomProxy < BackgrounDRb::WorkerProxy
   def self.init(ip, port)
     @@config = BackgrounDRb::Config.read_config("#{BACKGROUNDRB_ROOT}/ 
config/backgroundrb.yml")
     @@server_ip = ip
     @@server_port = port
     new
   end
end

But this doesn''t seem very elegant to me, I worry about future  
software updates...  Is there a clean method for doing this or is this  
just not supported (yet)?

Thanks for any insight, really appreciate it!

--
Steve

On Tue, Mar 25, 2008 at 10:47 PM, Stephen Weiss <sweiss at stylesight.com>
wrote:
> Hi,
>
>  Thanks for the help getting the new backgroundrb working, things are
>  working fabulously now.
>
>  I have two questions:
>
>  1)  I have a few basic methods I need available on all my workers in
>  order to communicate properly with my application... it seems to make
>  the most sense right now to subclass BackgrounDRb::MetaWorker, or make
>  a mixin module, because I''m actually extending some of the base
>  methods as well (like the exit method, which I have extended to update
>  status information in a central queue monitor).  Is there a proper
>  place to put in custom code that extends BackgrounDrb, so that it is
>  automatically loaded by the server before the workers are instantiated
>  (sort of like an environment.rb)?
For the time being, you can put that file in lib folder of your rails
application and require that file in environment.rb.
>
>  2)  I have one application server that runs Rails and does all the
>  user interface, but several (10+) servers running backgroundrb to do
>  all the heavy lifting (we do a lot of hi-res image processing,
>  requires many CPU''s).  I have a central monitor that monitors
what
>  servers are available to run which tasks, and load balances the tasks
>  across all the available backgroundrb servers, instantiating a
>  MiddleMan for the correct server and then sending the job request to
>  that server.  Needless to say, this process is going a lot easier with
>  the new version, except for one catch.  With the old backgroundrb, I
>  was able to connect to an arbitrary server like this:
>
>  thisMiddleMan = DRbObject.new(nil,
"druby://#{server_name}:22223");
>
>  Now this doesn''t work.  So, I took a look at
>  BackgrounDRb::WorkerProxy, and it seems like it can only connect to
>  the server listed in the config file.  Is there supposed to be a way
>  to manage multiple backgroundrb servers?  I''ve read the manual,
it''s
>  not in there.  My solution so far is to subclass WorkerProxy, with a
>  new init method that takes the host and port as parameters:
>
>  class BackgrounDRb::CustomProxy < BackgrounDRb::WorkerProxy
>    def self.init(ip, port)
>      @@config =
BackgrounDRb::Config.read_config("#{BACKGROUNDRB_ROOT}/
>  config/backgroundrb.yml")
>      @@server_ip = ip
>      @@server_port = port
>      new
>    end
>  end
>
>  But this doesn''t seem very elegant to me, I worry about future
>  software updates...  Is there a clean method for doing this or is this
>  just not supported (yet)?
>
Since, each connection to BackgrounDRb server is nothing but a plain
tcp socket connection. Now, before composing a reply to your message,
I actually committed a little code change, which allows you do do:

Loading development environment (Rails 2.0.2)
server ip: 0.0.0.0 11006
>> foo =
BackgrounDRb::WorkerProxy.custom_connection("localhost",11006)
server ip: localhost 11006
=> #<BackgrounDRb::WorkerProxy:0xb7477e14
@mutex=#<Mutex:0xb7477dec>,
@connection_status=true,
@connection=#<TCPSocket:0xb7477ce8>>
>> foo
=> #<BackgrounDRb::WorkerProxy:0xb7477e14
@mutex=#<Mutex:0xb7477dec>,
@connection_status=true,
@connection=#<TCPSocket:0xb7477ce8>>
>> foo.all_worker_info
=> [{:job_key=>"", :status=>:running,
:worker=>:log_worker}]
>> foo.query_all_worker_info
NoMethodError: undefined method `query_all_worker_info'' for
#<BackgrounDRb::WorkerProxy:0xb7477e14>
        from (irb):4
>> foo.query_all_workers
=> {:log_worker=>nil}

So, update the code from git and use it.
I hope this helps.