We have a fairly expensive task that we''d rather not schedule too many
instances of in parallel.
Couldn''t get pool_size to limit the amount of workers, and I figure
that''s hardly optimal anyway, so would like to have the work queued up
for a single named worker instead, running as soon as possible.
Using MiddleMan.schedule_worker() with :job_key does start off a
worker, but I can''t manage to add another task to the same worker and
have it queue up. It''s not just a matter of adding :worker_method =>
:do_work/:worker_method_args => whatever to schedule_worker()..? (see
example below)
Ideally using the same interface for running the first and any subsequent tasks.
Here''s a debug run, which may or may not make any sense.
Task a, b and c were started in 10-20 sec total, d was started after
a and b finished.
As you can see, b isn''t queued up, and c/d never start. I''ve
also had
''b'' start but never finish. (May have been due to do_work
deleting the
worker as ''a'' finished in that case.. figured delete was
neccessary to
have the other jobs start, but no..)
lib/workers/my_worker:
class MyWorker < BackgrounDRb::Worker::RailsBase
def do_work(args)
logger.info("MyWorker#do_work. args: #{args}")
sleep(60)
logger.info("MyWorker#work done, deleting self. args: #{args}")
# probably shouldn''t call self.delete here, as we reuse the
worker..?
end
end
MyWorker.register
script/console:>> MiddleMan.schedule_worker(:class => :my_worker, :job_key => :foo,
:worker_method => :do_work, :worker_method_args => "a", :args
=>
"_a_", :trigger_args => { :start => Time.now })
=> 0>> MiddleMan.schedule_worker(:class => :my_worker, :job_key => :foo,
:worker_method => :do_work, :worker_method_args => "b", :args
=>
"_b_", :trigger_args => { :start => Time.now })
=> 1>> MiddleMan.schedule_worker(:class => :my_worker, :job_key => :foo,
:worker_method => :do_work, :worker_method_args => "c", :args
=>
"_c_", :trigger_args => { :start => Time.now })
=> 2>> MiddleMan.schedule_worker(:class => :my_worker, :job_key => :foo,
:worker_method => :do_work, :worker_method_args => "d", :args
=>
"_d_", :trigger_args => { :start => Time.now })
=> 3>>
bdrb log #1:
20070515-13:55:27 (10865) MyWorker#do_work. args: _a_
20070515-13:55:32 (10865) MyWorker#do_work. args: b
20070515-13:56:27 (10865) MyWorker#work done, deleting self. args: _a_
20070515-13:56:32 (10865) MyWorker#work done, deleting self. args: b
bdrb log #2:
20070515-13:55:22 (10864) Loading Sechedule:
job_keyfooargs_a_classmy_worker
trigger_typetriggertrigger_argsstartTue May 15 13:55:22 CEST
2007worker_methoddo_workworker_method_argsajob_keyfooclassmy_workerargs_a_
#<BackgrounDRb::Trigger:0xb7b60054>
20070515-13:55:23 (10864) Schedule triggered: #<struct
#<Class:0xb7ddb4f0>
job=#<Proc:0xb7cfb7d8@/var/www/html/apps/xxx/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:319>,
trigger=#<BackgrounDRb::Trigger:0xb7b60054 @start_time=Tue May 15
13:55:22 CEST 2007, @repeat_interval=1, @end_time=Tue May 15 13:55:23
CEST 2007>, earliest=Tue May 15 13:55:23 CEST 2007, last=Tue May 15
13:55:23 CEST 2007>
20070515-13:55:27 (10864) Starting worker: my_worker foo (my_worker_foo) (_a_)
20070515-13:55:31 (10864) Loading Sechedule:
job_keyfooargs_b_classmy_worker
trigger_typetriggertrigger_argsstartTue May 15 13:55:31 CEST
2007worker_methoddo_workworker_method_argsbjob_keyfooclassmy_workerargs_b_
#<BackgrounDRb::Trigger:0xb7874ae8>
20070515-13:55:32 (10864) Schedule triggered: #<struct
#<Class:0xb7ddb4f0>
job=#<Proc:0xb7cfb7d8@/var/www/html/apps/xxx/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:319>,
trigger=#<BackgrounDRb::Trigger:0xb7874ae8 @start_time=Tue May 15
13:55:31 CEST 2007, @repeat_interval=1, @end_time=Tue May 15 13:55:32
CEST 2007>, earliest=Tue May 15 13:55:32 CEST 2007, last=Tue May 15
13:55:32 CEST 2007>
20070515-13:55:40 (10864) Loading Sechedule:
job_keyfooargs_c_classmy_worker
trigger_typetriggertrigger_argsstartTue May 15 13:55:40 CEST
2007worker_methoddo_workworker_method_argscjob_keyfooclassmy_workerargs_c_
#<BackgrounDRb::Trigger:0xb786eb48>
20070515-13:56:03 (10864) Loading Sechedule:
job_keyfooargs_d_classmy_worker
trigger_typetriggertrigger_argsstartTue May 15 13:56:03 CEST
2007worker_methoddo_workworker_method_argsdjob_keyfooclassmy_workerargs_d_
#<BackgrounDRb::Trigger:0xb786c44c>
On 5/15/07, Michael Siebert <siebertm85 at googlemail.com> wrote:> for queueing up work, here is what i use, it runs in its own named worker > and is very stable. currently, there is only one job at a time, but you > could also generate two workers so you could process two jobs at a time. >*snipped code where jobs are queued up in the worker* Interesting approach. I was depending on bdrb itself to take care of the queueing, but will give this a shot if nothing is changed by the new release.> does the list have any improvements or comments? >Hmm.. I''m thinking some kind of synchronization may be neccessary on the worker''s job queue. Btw.. How does this stuff really work? The worker instance lives in the bdrb process, while middleman creates a remote-call-somehow-proxy to that instance for your rails app to use? Any limitation on argument type(s) for public methods in my worker? Being serializable is probably a must? Regards, Isak
On 5/16/07, Michael Siebert <siebertm85 at googlemail.com> wrote:> 2007/5/16, Isak Hansen <isak.hansen at gmail.com>: > > Hmm.. I''m thinking some kind of synchronization may be neccessary on > > the worker''s job queue. > > help me understand what you mean... :) there is only one instance of the > worker so i dont see any place that should need synchronization. >There''s the thread running do_work, and I''d think there are additional thread(s) adding jobs to the worker?> > Btw.. How does this stuff really work? The worker instance lives in > > the bdrb process, while middleman creates a remote-call-somehow-proxy > > to that instance for your rails app to use? > > I use this thing in productio and it just works. there is no other magic > behind that (besides what i have shown you >Sorry, I meant bdrb in general. Your example was great, and easy to follow. :)> > Any limitation on argument type(s) for public methods in my worker? > > Being serializable is probably a must? > > they must be serializable since they are sent via drb. its the same like > when passing arguments to do_work. >That''s one piece of the puzzle down, thanks. Regards, Isak> kind regards, > micha > > > > -- > Siebert Webdesign > Michael Siebert
Michael Siebert
2007-May-16 09:23 UTC
[Backgroundrb-devel] Need help with singleton worker
2007/5/16, Isak Hansen <isak.hansen at gmail.com>:> > There''s the thread running do_work, and I''d think there are additional > thread(s) adding jobs to the worker?no there is only one process running. for the exact things going on, ask ezra or skaar, for me, it just works great Sorry, I meant bdrb in general. as i already said: its like rails and ubuntu: it just works. Your example was great, and easy to follow. :) thank you. kind regards -- Siebert Webdesign Michael Siebert -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070516/df88ed15/attachment.html
On 5/16/07, Michael Siebert <siebertm85 at googlemail.com> wrote:> 2007/5/16, Isak Hansen <isak.hansen at gmail.com>: > > There''s the thread running do_work, and I''d think there are additional > > thread(s) adding jobs to the worker? > > no there is only one process running. for the exact things going on, ask > ezra or skaar, for me, it just works great >One process yes, but I believe add_job() is called from a different thread than do_work, which is stuck in an infinite loop. I''m not sure Array isn''t thread safe (been getting conflicting feedback about that), but figure it can''t hurt to use a mutex around code accessing the job queue. How about cleanup? Say we eventually get 5-6 different named workers, that''s quite a bit of memory. Should we just trust the OS to swap those processes out, or would you guys recommend having the worker deleting itself and terminate after for instance X minutes of idle time? Regards, Isak