Backgroundrb devel - May 2007 - Recommendations for eternally-running backgroundrb workers?

I''ve got some workers that I want to have running all the time.  Right
now I''m just launching them manually, by requesting a special page in
my rails app that includes lines like:

      MiddleMan.new_worker(:class=> :receiver, :job_key=>:r,
:args=>{:sleep_time=>10})

This strikes me as a really weak way to fire up my workers.  I
basically want to automate things so that these will be started by a
script in /etc/rc?.d.  It seems like using the backgroundrb scheduler
is not a possibility, since do_work never returns from my workers
(each of them runs in an infinite loop, polling the database
periodically to look for work).  So it looks like I need another
startup script, running after the rest of the system is up, that does
one of these things:

- use curl or wget to hit my special worker-starting page (how lame is that?)
- pipe my MiddleMan.new_worker calls into a "script/backgroundrb
console" or just "script/console" session (is that any better?)

As you can see, neither of these options seems very appealing to me.
I can''t be the first person to do this...  How are others handling
this?

-- 
// jack
// http://www.nuthole.com

Can''t you remove the periodic checker code that wraps the actual
worker and let backgroundrb scheduler do the periodic thing for you?
In other words, why do you have do_work run "in an infinite loop, ...
periodically to look for work" when essentially that''s what
backgroundrb does for you?

On 5/22/07, Jack Nutting <jnutting at gmail.com>
wrote:
> I''ve got some workers that I want to have running all the time. 
Right
> now I''m just launching them manually, by requesting a special page
in
> my rails app that includes lines like:
>
>       MiddleMan.new_worker(:class=> :receiver, :job_key=>:r,
> :args=>{:sleep_time=>10})
>
> This strikes me as a really weak way to fire up my workers.  I
> basically want to automate things so that these will be started by a
> script in /etc/rc?.d.  It seems like using the backgroundrb scheduler
> is not a possibility, since do_work never returns from my workers
> (each of them runs in an infinite loop, polling the database
> periodically to look for work).  So it looks like I need another
> startup script, running after the rest of the system is up, that does
> one of these things:
>
> - use curl or wget to hit my special worker-starting page (how lame is
that?)
> - pipe my MiddleMan.new_worker calls into a "script/backgroundrb
> console" or just "script/console" session (is that any
better?)
>
> As you can see, neither of these options seems very appealing to me.
> I can''t be the first person to do this...  How are others handling
> this?
>
> --
> // jack
> // http://www.nuthole.com
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

Hmmm, I meant to reply all on this...

---------- Forwarded message ----------
From: Jack Nutting <jnutting at gmail.com>
Date: May 22, 2007 1:39 PM
Subject: Re: [Backgroundrb-devel] Recommendations for
eternally-running backgroundrb workers?
To: Eden Li <eden at mojiti.com>


On 5/22/07, Eden Li <eden at mojiti.com> wrote:
> Can''t you remove the periodic checker code that wraps the actual
> worker and let backgroundrb scheduler do the periodic thing for you?
> In other words, why do you have do_work run "in an infinite loop, ...
> periodically to look for work" when essentially that''s what
> backgroundrb does for you?
Well, that''s the first thing I tried.  What I found is that if one
worker was tied up doing something that took a long time, the other
workers wouldn''t be triggered to run at all.  So, I''ve got 3
workers
(let''s call them A, B, and C), each set to pause about a minute
between looking for work.  At some point, worker A finds it has some
tasks waiting for it, and starts doing them.  If this ends up taking
15 minutes, workers B and C are completely idle for 15 minutes.  I''ve
never figured out if this is happening on accident or by design, but
it seems that the scheduler runs in a single thread and waits for each
worker call to finish before proceeding.

I suppose I could get around that by breaking up the work, so that
each call to A.real_work only handles a chunk of its work before
returning and allowing the others to run.  But even then, the effect
is that I''d still be stuck serializing (though in smaller chunks)
something that I want to run in parallel.  In particular, both A and B
spend a large amount of their working time waiting for results from
http requests (which are independent of each other, coming from
different hosts), so these are good candidates for running in parallel
instead of saying, "OK, now A can make some requests.... OK, now
it''s
B''s turn...." etc.

-- 
// jack
// http://www.nuthole.com

Jack -- you can use the scheduler to fire up long-running processes.

Just don''t specify a repeat interval.

You backgroundrb_schedules.yml file should look something like:

my_worker:
   :class: :polling_worker
   :job_key: :my_worker
   :worker_method: :do_work
   :worker_method_args:
placeholder
   :trigger_args:
     :start: <%= Time.now + 1.seconds %>

On 5/22/07, Jack Nutting <jnutting at gmail.com>
wrote:
>
> I''ve got some workers that I want to have running all the time. 
Right
> now I''m just launching them manually, by requesting a special page
in
> my rails app that includes lines like:
>
>       MiddleMan.new_worker(:class=> :receiver, :job_key=>:r,
> :args=>{:sleep_time=>10})
>
> This strikes me as a really weak way to fire up my workers.  I
> basically want to automate things so that these will be started by a
> script in /etc/rc?.d.  It seems like using the backgroundrb scheduler
> is not a possibility, since do_work never returns from my workers
> (each of them runs in an infinite loop, polling the database
> periodically to look for work).  So it looks like I need another
> startup script, running after the rest of the system is up, that does
> one of these things:
>
> - use curl or wget to hit my special worker-starting page (how lame is
> that?)
> - pipe my MiddleMan.new_worker calls into a "script/backgroundrb
> console" or just "script/console" session (is that any
better?)
>
> As you can see, neither of these options seems very appealing to me.
> I can''t be the first person to do this...  How are others handling
> this?
>
> --
> // jack
> // http://www.nuthole.com
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070523/60587e8f/attachment.html

hit send prematurely...

my_worker:
   :class: :polling_worker
   :job_key: :my_worker
   :worker_method: :do_work
   :worker_method_args:
      :arg1: placeholder
   :trigger_args:
     :start: <%= Time.now + 1.seconds %>

Note there is not :repeate_interval: argument under :trigger_args:

In your worker class it could look like:


  def do_work(args)
    logger.debug("#{self.jobkey}: do_work called")
    loop do # loop forever
        check_for_work # this is the method that really does the work
        sleep 60
      end
    end
  end

You should see the "do_work called" message once.

The fact that do_work does not return should not matter, the other scheduled
jobs will still run.

I use the method above in production with a high level of reliability.

Mason


On 5/23/07, Mason Hale <masonhale at gmail.com>
wrote:
>
> Jack -- you can use the scheduler to fire up long-running processes.
>
> Just don''t specify a repeat interval.
>
> You backgroundrb_schedules.yml file should look something like:
>
> my_worker:
>    :class: :polling_worker
>    :job_key: :my_worker
>    :worker_method: :do_work
>    :worker_method_args:
> placeholder
>    :trigger_args:
>      :start: <%= Time.now + 1.seconds %>
>
> On 5/22/07, Jack Nutting <jnutting at gmail.com> wrote:
> >
> > I''ve got some workers that I want to have running all the
time.  Right
> > now I''m just launching them manually, by requesting a special
page in
> > my rails app that includes lines like:
> >
> >       MiddleMan.new_worker (:class=> :receiver, :job_key=>:r,
> > :args=>{:sleep_time=>10})
> >
> > This strikes me as a really weak way to fire up my workers.  I
> > basically want to automate things so that these will be started by a
> > script in /etc/rc?.d.  It seems like using the backgroundrb scheduler
> > is not a possibility, since do_work never returns from my workers
> > (each of them runs in an infinite loop, polling the database
> > periodically to look for work).  So it looks like I need another
> > startup script, running after the rest of the system is up, that does
> > one of these things:
> >
> > - use curl or wget to hit my special worker-starting page (how lame is
> > that?)
> > - pipe my MiddleMan.new_worker calls into a "script/backgroundrb
> > console" or just "script/console" session (is that any
better?)
> >
> > As you can see, neither of these options seems very appealing to me.
> > I can''t be the first person to do this...  How are others
handling
> > this?
> >
> > --
> > // jack
> > // http://www.nuthole.com
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070523/960cad0f/attachment.html