I''ve been using backgroundrb since back in March 2007 or so.
It''s
been working mostly OK for me, but since the old version doesn''t seem
to work out of the box with rails 2.0 for me, I decided to test out
the latest version.
So, I''m looking at the scheduling, and trying to figure out the best
option for what I want to do. What I have is a set of workers that
each poll the database occasionally. I''d like to set it up so that
each of them can poll for work, do the work (however long that may
take), then pause for a "sleep time" and start over. With the old
version of backgroundrb, I was never able to get any of the scheduling
options to quite do what I wanted to do, so I just ran without
scheduling. I manually started all my workers by calling
MiddleMan.new_worker in a script, passing in the "sleep time", and
then implemented each worker something like this:
def do_work(args)
@args = args
@sleep_time = @args[:sleep_time]
logger.info "CampaignStarter #{jobkey} started"
while(true)
# do the actual work in another method
main_work
sleep @sleep_time
end
end
That seemed to work pretty well for the most part.
So now, looking at the current version. I''m guessing that
it''s
probably not OK for me to use create() the way I was using do_work()
in the past; Presumably create() is supposed to return at some point.
Besides which, one of my workers needs to have multiple instances, so
I have to pass in a job_key somewhere. I''m guessing that I should
call MiddleMan.new_worker(), telling it to run my main_work() method
with some scheduling option, but I''m not sure which scheduling option
to use. I don''t want to schedule it to run every 10 seconds, since
each call to main_work could take anything up to several minutes, and
I don''t want the invocations piling up. The docs also seem to suggest
that I could do things my old way by calling MiddleMan.ask_work() and
letting it run forever, but it''s not clear whether I can pass in a
job_key there. I''d love any suggestions here...
Next, is there any way from inside a worker to see what its job_key
is? With the old version I''ve been using the jobkey as a way to split
up some of the work requests, so each instance of my worker class
needs to know its own key in order to find what work is assigned to
it.
Finally, it''s not at all clear to me how the current version handles
multiple instances and threads. For example, if I call
MiddleMan.new_worker(), does it actually create a new instance of my
worker class (as the old version did), and if so, is that instance in
a new thread of its own? How about MiddleMan.ask_work() and
MiddleMan.send_request()?
--
// jack
// http://www.nuthole.com
Hi Jack, On Jan 17, 2008 10:30 PM, Jack Nutting <jnutting at gmail.com> wrote:> I''ve been using backgroundrb since back in March 2007 or so. It''s > been working mostly OK for me, but since the old version doesn''t seem > to work out of the box with rails 2.0 for me, I decided to test out > the latest version. > > So, I''m looking at the scheduling, and trying to figure out the best > option for what I want to do. What I have is a set of workers that > each poll the database occasionally. I''d like to set it up so that > each of them can poll for work, do the work (however long that may > take), then pause for a "sleep time" and start over. With the old > version of backgroundrb, I was never able to get any of the scheduling > options to quite do what I wanted to do, so I just ran without > scheduling. I manually started all my workers by calling > MiddleMan.new_worker in a script, passing in the "sleep time", and > then implemented each worker something like this: > > def do_work(args) > @args = args > @sleep_time = @args[:sleep_time] > logger.info "CampaignStarter #{jobkey} started" > while(true) > # do the actual work in another method > main_work > sleep @sleep_time > end > end > > That seemed to work pretty well for the most part.Yuck. Never use sleep() in new backgroundrb workers. :)> > So now, looking at the current version. I''m guessing that it''s > probably not OK for me to use create() the way I was using do_work() > in the past; Presumably create() is supposed to return at some point. > Besides which, one of my workers needs to have multiple instances, so > I have to pass in a job_key somewhere. I''m guessing that I should > call MiddleMan.new_worker(), telling it to run my main_work() method > with some scheduling option, but I''m not sure which scheduling option > to use. I don''t want to schedule it to run every 10 seconds, since > each call to main_work could take anything up to several minutes, and > I don''t want the invocations piling up. The docs also seem to suggest > that I could do things my old way by calling MiddleMan.ask_work() and > letting it run forever, but it''s not clear whether I can pass in a > job_key there. I''d love any suggestions here...I am still unclear about whats your purpose of using bdrb? If you just want to use it to schedule tasks, then you can specify your schedule in config file, using either Cron on UNIX syntax and it will work. If you are scheduling at in interval of 10 seconds and your actual tasks takes longer than that, then also there is nothing to worry and bdrb will schedule tasks as they are ready to run and worker is free. So your tasks won''t be piling up. create() does return at some point, but at each iteration of reactor loop, tasks ready to run are executed. Currently if you need more than one instance of same worker running at the same time, its not supported via backgroundrb.yml file, but you can create as many instances of workers as you want using different job_keys using MiddleMan.new_worker() method from rails. Hell, I think you can invoke MiddleMan.new_worker() from one of your workers to start workers on demand and supply job_keys. ask_work() is for doing one shot task execution. It can be only used in a already runner worker. If you started a worker with a job_key using new_worker() method, then you will have to use that job_key along with worker name in ask_work() or ask_status() invocation.> > Next, is there any way from inside a worker to see what its job_key > is? With the old version I''ve been using the jobkey as a way to split > up some of the work requests, so each instance of my worker class > needs to know its own key in order to find what work is assigned to > it.Sad truth of life is that, you can''t. Not from inside of a worker. But since inside any worker you have full rails environment and hence MiddleMan proxy object, you can query job_key using: MiddleMan.worker_info(:worker => worker_name.to_sym) Haven''t tried, but it should work.> > Finally, it''s not at all clear to me how the current version handles > multiple instances and threads. For example, if I call > MiddleMan.new_worker(), does it actually create a new instance of my > worker class (as the old version did), and if so, is that instance in > a new thread of its own? How about MiddleMan.ask_work() and > MiddleMan.send_request()?new_worker() creates a full forked process. ask_work(), send_request() can only operate on workers which are already running. And lastly, sorry for documentation, if you found it a bit out of touch. *vows to improve it* -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org
On Jan 17, 2008 6:41 PM, hemant <gethemant at gmail.com> wrote:> Yuck. Never use sleep() in new backgroundrb workers. :)Yeah, I wasn''t really 100% happy with that solution either... I will change my ways!> I am still unclear about whats your purpose of using bdrb? If you just > want to use it to schedule tasks, then you can specify your schedule > in config file, using either Cron on UNIX syntax and it will work.Good question! At an earlier stage in my application''s design, there was a place for more standard backgroundrb usage, with some processing that was triggered by user actions. I liked the way it worked, and was enough of a rails noob that I frankly didn''t know any other way to do background tasks that accessed activerecord, my model classes, etc, so I started using backgroundrb for perpetually-running processes as well; eventually these also incorporated the functions that were previously performed by user actions as well, and there I am today. Maybe I should just stop using backgroundrb for now, until I have a use that better matches its intended purpose.> If > you are scheduling at in interval of 10 seconds and your actual tasks > takes longer than that, then also there is nothing to worry and bdrb > will schedule tasks as they are ready to run and worker is free. So > your tasks won''t be piling up.Ah, OK. I picked up a different meaning from this text in the README, the Cron Scheduling section: The method specified in the configuration file would be called periodically. You should accommodate for the fact that the time gap between periodic invocation of a method should be more than the time that is actually required to execute the method. If a method takes longer time than the time window specified, your method invocations will lag perpetually. Doesn''t that seem to be saying that requests will be piling up?> ask_work() is for doing one shot task execution. It can be only used > in a already runner worker. If you started a worker with a job_key > using new_worker() method, then you will have to use that job_key > along with worker name in ask_work() or ask_status() invocation.OK, I didn''t realize you could use job_key with ask_work(). Now it makes sense to me.> > Next, is there any way from inside a worker to see what its job_key > > is? > > Sad truth of life is that, you can''t. Not from inside of a worker. But > since inside any worker you have full rails environment and hence > MiddleMan proxy object, you can query job_key using: > > MiddleMan.worker_info(:worker => worker_name.to_sym)It also occurred to me that I could pass the job_key in with the data parameter when I create the worker.> new_worker() creates a full forked process. ask_work(), send_request() > can only operate on workers which are already running.Thanks for the clarification.> And lastly, sorry for documentation, if you found it a bit out of > touch. *vows to improve it*No problem, I appreciate that the documentation is there at all, and most of all that you''ve picked up the backgroundrb ball and run with it! Keep up the good work! -- // jack // http://www.nuthole.com
Hi Jack, On Jan 18, 2008 1:56 AM, Jack Nutting <jnutting at gmail.com> wrote:> Good question! At an earlier stage in my application''s design, there > was a place for more standard backgroundrb usage, with some processing > that was triggered by user actions. I liked the way it worked, and > was enough of a rails noob that I frankly didn''t know any other way to > do background tasks that accessed activerecord, my model classes, etc, > so I started using backgroundrb for perpetually-running processes as > well; eventually these also incorporated the functions that were > previously performed by user actions as well, and there I am today. > Maybe I should just stop using backgroundrb for now, until I have a > use that better matches its intended purpose.What I meant was, if you are using backgroundrb for periodic scheduling of tasks, then you use cron or unix scheduling of tasks provided by backgroundrb. I wouldn''t ask you to go to crontab. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org