Hi All, One can easily code a backgroundrb worker so as it can act as a worker queue, having reached a stable status, perhaps its time we should add this feature. Why? Because if you are already running your application using BackgrounDRb, perhaps you don''t want to manage other solutions when capability to do that is right there with you. Interface? This where i need feedback. You can already dynamically start a worker and submit a job to it ( using MiddleMan.new_worker) and it would be done in a new process. But its not a queue, it starts executing your task immediately. Sometimes I doubt if its seriously required, because you can easily queue your tasks within a worker using inbuilt thread pool ( if you don''t want to spawn new worker ). What then? Perhaps what we CAN do is, have a specialized worker that can act on a priority queue. This priority queue is nothing but a database table and depending upon the number of tasks in the queue, master process automatically starts new workers or deletes them as need be.Our specialized worker, when picks a task from db, it marks that task as taken and starts processing them. The control to start new workers depending upon size of queue can be left either to master or the "specialized worker" itself. I was thinking on lines of having a ratio as in lets say per 10 queued tasks you want one worker, then you can specify that in configuration file. If the number of tasks goes higher either "specialized worker" or master starts a new instance of "specialized worker". There could be a upper limit on number of parallel workers running and if on one machine this limit is reached backgroundrb can request master to start workers on another machine. How does this sound? Feedbacks? -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org
Jim Scott
2007-Dec-22 16:10 UTC
[Backgroundrb-devel] Implementation details for worker Queue
Hemant - Strangely enough, you have exactly described how we use backgroundrb. We queue jobs to a table, and BackgroundRb processes them. We have specialized workers for each row type. When we have higher volumes of work, we use Xen virtualization to stand up new utility servers that process queued jobs. We can scale this solution horizontally, much in the same we that we scale application servers. I for one vote YES. Jim Scott Plan Source -----Original Message----- From: backgroundrb-devel-bounces at rubyforge.org [mailto:backgroundrb-devel-bounces at rubyforge.org] On Behalf Of hemant Sent: Saturday, December 22, 2007 6:31 AM To: backgroundrb-devel Subject: [Backgroundrb-devel] Implementation details for worker Queue Hi All, One can easily code a backgroundrb worker so as it can act as a worker queue, having reached a stable status, perhaps its time we should add this feature. Why? Because if you are already running your application using BackgrounDRb, perhaps you don''t want to manage other solutions when capability to do that is right there with you. Interface? This where i need feedback. You can already dynamically start a worker and submit a job to it ( using MiddleMan.new_worker) and it would be done in a new process. But its not a queue, it starts executing your task immediately. Sometimes I doubt if its seriously required, because you can easily queue your tasks within a worker using inbuilt thread pool ( if you don''t want to spawn new worker ). What then? Perhaps what we CAN do is, have a specialized worker that can act on a priority queue. This priority queue is nothing but a database table and depending upon the number of tasks in the queue, master process automatically starts new workers or deletes them as need be.Our specialized worker, when picks a task from db, it marks that task as taken and starts processing them. The control to start new workers depending upon size of queue can be left either to master or the "specialized worker" itself. I was thinking on lines of having a ratio as in lets say per 10 queued tasks you want one worker, then you can specify that in configuration file. If the number of tasks goes higher either "specialized worker" or master starts a new instance of "specialized worker". There could be a upper limit on number of parallel workers running and if on one machine this limit is reached backgroundrb can request master to start workers on another machine. How does this sound? Feedbacks? -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org _______________________________________________ Backgroundrb-devel mailing list Backgroundrb-devel at rubyforge.org http://rubyforge.org/mailman/listinfo/backgroundrb-devel No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.17.6/1192 - Release Date: 12/21/2007 1:17 PM No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.17.6/1192 - Release Date: 12/21/2007 1:17 PM
Joe Van Dyk
2007-Dec-23 01:59 UTC
[Backgroundrb-devel] Implementation details for worker Queue
On Dec 22, 2007 5:30 AM, hemant <gethemant at gmail.com> wrote:> Hi All, > > One can easily code a backgroundrb worker so as it can act as a worker > queue, having reached a stable status, perhaps its time we should add > this feature. > > Why? > Because if you are already running your application using > BackgrounDRb, perhaps you don''t want to manage other solutions when > capability to do that is right there with you.Sort of related question: If I submit multiple jobs to a single worker (i.e. from multiple Rails processes), does the worker do one of those jobs at a time? Or does it two both of them at the same time? Joe