Hello everyone. I am making several asynchronous calls to one particular method in my worker like this: MiddleMan.worker(:reindex_worker).async_reindex(:arg => options) If I call this several times in a row, all subsequent calls are put in a queue and I would like to be able to keep track of this. The behavior is perfect because I only ever want reindex() running once at any one time but I would like to be able to show my users where in the queue their request is. Is there any way to view what async calls are lined up? The worker_info stuff doesn''t seem to give away any details like this. I also thought about using "enq_" to schedule but is there a way to specify that the trigger is the completion of the previous scheduled instance and so on? Or is the scheduling purely time based? Kind regards, Chad.
That data is in write Queue of master worker. It can''t be meaningfully extracted to provide how many tasks are in the queue. Put it other way, its not even in a Queue in traditional sense. Data is just there in a buffer so that master will write it to the socket when socket is ready for write. If you want Queue. You will have either use enq_xxx stuff or provide a sugar around asyc_index() method so as actual method is not directly called but is added to the thread pool (thread_pool.defer), this way you can query how many tasks are there in thread pool queue. However word "thread" comes with its own meaning and associated crapiness in ruby, so beware. On Fri, Jan 23, 2009 at 6:35 PM, Chad Thatcher <chad.thatcher at gmail.com> wrote:> Hello everyone. > > I am making several asynchronous calls to one particular method in my worker > like this: > > MiddleMan.worker(:reindex_worker).async_reindex(:arg => options) > > If I call this several times in a row, all subsequent calls are put in a > queue and I would like to be able to keep track of this. The behavior is > perfect because I only ever want reindex() running once at any one time but > I would like to be able to show my users where in the queue their request > is. Is there any way to view what async calls are lined up? The > worker_info stuff doesn''t seem to give away any details like this. > > I also thought about using "enq_" to schedule but is there a way to specify > that the trigger is the completion of the previous scheduled instance and so > on? Or is the scheduling purely time based? > > Kind regards, > > Chad. > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel >-- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org
Thanks Hemant, Yes I was specifically trying to avoid threads. However, I am guessing I could configure it so that only one thread ran at one time right? That would cut down on the complexity. Is the enq_xxx stuff only time based? If so I wonder if it would be difficult to add a feature that allows you to add a whole bunch of tasks to the queue and then they just run one after the other in the order in which they were added. Maybe I could intercept persistent_job#finished! to find the next (grouped) task in the queue and update it to start a few seconds in the future so the bgrb picks it up and runs it etc. But that seems evil somehow :). I realise this might not be the target usage for backgroundrb but I look forward to hearing your thoughts because I''m hoping it won''t be too difficult for me to do. The ultimate fall-back, of course, is to roll my own task queue that uses a table similar table to bdrb_job_queues but that seems wasteful when there is so much there for it already. All the best, Chad. On 23 Jan 2009, at 18:57, hemant wrote:> That data is in write Queue of master worker. It can''t be meaningfully > extracted to provide how many tasks are in the queue. Put it other > way, its not even in a Queue in traditional sense. Data is just there > in a buffer so that master will write it to the socket when socket is > ready for write. > > If you want Queue. You will have either use enq_xxx stuff or provide a > sugar around asyc_index() method so as actual method is not directly > called but is added to the thread pool (thread_pool.defer), this way > you can query how many tasks are there in thread pool queue. > > However word "thread" comes with its own meaning and associated > crapiness in ruby, so beware. > > > On Fri, Jan 23, 2009 at 6:35 PM, Chad Thatcher <chad.thatcher at gmail.com > > wrote: >> Hello everyone. >> >> I am making several asynchronous calls to one particular method in >> my worker >> like this: >> >> MiddleMan.worker(:reindex_worker).async_reindex(:arg => options) >> >> If I call this several times in a row, all subsequent calls are put >> in a >> queue and I would like to be able to keep track of this. The >> behavior is >> perfect because I only ever want reindex() running once at any one >> time but >> I would like to be able to show my users where in the queue their >> request >> is. Is there any way to view what async calls are lined up? The >> worker_info stuff doesn''t seem to give away any details like this. >> >> I also thought about using "enq_" to schedule but is there a way to >> specify >> that the trigger is the completion of the previous scheduled >> instance and so >> on? Or is the scheduling purely time based? >> >> Kind regards, >> >> Chad. >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> > > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org
On Sat, Jan 24, 2009 at 1:46 AM, Chad Thatcher <chad.thatcher at gmail.com> wrote:> Thanks Hemant, > > Yes I was specifically trying to avoid threads. However, I am guessing I > could configure it so that only one thread ran at one time right? That > would cut down on the complexity.Yes, changing thread pool size to 1 usually helps. Libraries like mechanize (even Net/HTTP) crap out under concurrent threads but if your pool size is 1, you won''t have these problems, thread pool will be just a queue. But if you don''t want any concurrency, why not just do something like this: class FooWorker < BackgrounDRb::MetaWorker def create(args = nil) @task_array = [] # check task array for pending tasks and run them add_periodic_timer(2) { run_from_task_array() } end def accept_indexing(args = nil) klass = Struct.new(:cmd_args) # add task to the task array @task_array << klass.new(args) end def run_from_task_array task = @task_array.shift # run the task end def check_count return @task_array.count end end if you don''t want threads and above code will work too. Specially since thread pool uses Queue data structure from ruby stdlib which is quite heavyweight and thread safe, but you don''t want any thread safety here, so why pay additional penalty for that? All your tasks run in same main thread.> > Is the enq_xxx stuff only time based? If so I wonder if it would be > difficult to add a feature that allows you to add a whole bunch of tasks to > the queue and then they just run one after the other in the order in which > they were added. Maybe I could intercept persistent_job#finished! to find > the next (grouped) task in the queue and update it to start a few seconds in > the future so the bgrb picks it up and runs it etc. But that seems evil > somehow :).You can totally do that! Check add_timer or add_periodic_timer> > I realise this might not be the target usage for backgroundrb but I look > forward to hearing your thoughts because I''m hoping it won''t be too > difficult for me to do. The ultimate fall-back, of course, is to roll my > own task queue that uses a table similar table to bdrb_job_queues but that > seems wasteful when there is so much there for it already. >Right.> > On 23 Jan 2009, at 18:57, hemant wrote: > >> That data is in write Queue of master worker. It can''t be meaningfully >> extracted to provide how many tasks are in the queue. Put it other >> way, its not even in a Queue in traditional sense. Data is just there >> in a buffer so that master will write it to the socket when socket is >> ready for write. >> >> If you want Queue. You will have either use enq_xxx stuff or provide a >> sugar around asyc_index() method so as actual method is not directly >> called but is added to the thread pool (thread_pool.defer), this way >> you can query how many tasks are there in thread pool queue. >> >> However word "thread" comes with its own meaning and associated >> crapiness in ruby, so beware. >> >> >> On Fri, Jan 23, 2009 at 6:35 PM, Chad Thatcher <chad.thatcher at gmail.com> >> wrote: >>> >>> Hello everyone. >>> >>> I am making several asynchronous calls to one particular method in my >>> worker >>> like this: >>> >>> MiddleMan.worker(:reindex_worker).async_reindex(:arg => options) >>> >>> If I call this several times in a row, all subsequent calls are put in a >>> queue and I would like to be able to keep track of this. The behavior is >>> perfect because I only ever want reindex() running once at any one time >>> but >>> I would like to be able to show my users where in the queue their request >>> is. Is there any way to view what async calls are lined up? The >>> worker_info stuff doesn''t seem to give away any details like this. >>> >>> I also thought about using "enq_" to schedule but is there a way to >>> specify >>> that the trigger is the completion of the previous scheduled instance and >>> so >>> on? Or is the scheduling purely time based? >>> >>> Kind regards, >>> >>> Chad. >>> >>> _______________________________________________ >>> Backgroundrb-devel mailing list >>> Backgroundrb-devel at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>> >> >> >> >> -- >> Let them talk of their oriental summer climes of everlasting >> conservatories; give me the privilege of making my own summer with my >> own coals. >> >> http://gnufied.org > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel >-- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org
Thanks Hemant, you are absolutely correct about threads being too much overhead for what I need so this looks like a perfect solution! Cheers, Chad. On 24 Jan 2009, at 01:45, hemant wrote:> On Sat, Jan 24, 2009 at 1:46 AM, Chad Thatcher <chad.thatcher at gmail.com > > wrote: >> Thanks Hemant, >> >> Yes I was specifically trying to avoid threads. However, I am >> guessing I >> could configure it so that only one thread ran at one time right? >> That >> would cut down on the complexity. > > Yes, changing thread pool size to 1 usually helps. Libraries like > mechanize (even Net/HTTP) crap out under concurrent threads but if > your pool size is 1, you won''t have these problems, thread pool will > be just a queue. But if you don''t want any concurrency, why not just > do something like this: > > class FooWorker < BackgrounDRb::MetaWorker > def create(args = nil) > @task_array = [] > # check task array for pending tasks and run them > add_periodic_timer(2) { run_from_task_array() } > end > > def accept_indexing(args = nil) > klass = Struct.new(:cmd_args) > # add task to the task array > @task_array << klass.new(args) > end > > def run_from_task_array > task = @task_array.shift > # run the task > end > > def check_count > return @task_array.count > end > end > > if you don''t want threads and above code will work too. Specially > since thread pool uses Queue data structure from ruby stdlib which is > quite heavyweight and thread safe, but you don''t want any thread > safety here, so why pay additional penalty for that? All your tasks > run in same main thread. > > > >> >> Is the enq_xxx stuff only time based? If so I wonder if it would be >> difficult to add a feature that allows you to add a whole bunch of >> tasks to >> the queue and then they just run one after the other in the order >> in which >> they were added. Maybe I could intercept persistent_job#finished! >> to find >> the next (grouped) task in the queue and update it to start a few >> seconds in >> the future so the bgrb picks it up and runs it etc. But that seems >> evil >> somehow :). > > You can totally do that! Check add_timer or add_periodic_timer > >> >> I realise this might not be the target usage for backgroundrb but I >> look >> forward to hearing your thoughts because I''m hoping it won''t be too >> difficult for me to do. The ultimate fall-back, of course, is to >> roll my >> own task queue that uses a table similar table to bdrb_job_queues >> but that >> seems wasteful when there is so much there for it already. >> > > Right. > >> >> On 23 Jan 2009, at 18:57, hemant wrote: >> >>> That data is in write Queue of master worker. It can''t be >>> meaningfully >>> extracted to provide how many tasks are in the queue. Put it other >>> way, its not even in a Queue in traditional sense. Data is just >>> there >>> in a buffer so that master will write it to the socket when socket >>> is >>> ready for write. >>> >>> If you want Queue. You will have either use enq_xxx stuff or >>> provide a >>> sugar around asyc_index() method so as actual method is not directly >>> called but is added to the thread pool (thread_pool.defer), this way >>> you can query how many tasks are there in thread pool queue. >>> >>> However word "thread" comes with its own meaning and associated >>> crapiness in ruby, so beware. >>> >>> >>> On Fri, Jan 23, 2009 at 6:35 PM, Chad Thatcher <chad.thatcher at gmail.com >>> > >>> wrote: >>>> >>>> Hello everyone. >>>> >>>> I am making several asynchronous calls to one particular method >>>> in my >>>> worker >>>> like this: >>>> >>>> MiddleMan.worker(:reindex_worker).async_reindex(:arg => options) >>>> >>>> If I call this several times in a row, all subsequent calls are >>>> put in a >>>> queue and I would like to be able to keep track of this. The >>>> behavior is >>>> perfect because I only ever want reindex() running once at any >>>> one time >>>> but >>>> I would like to be able to show my users where in the queue their >>>> request >>>> is. Is there any way to view what async calls are lined up? The >>>> worker_info stuff doesn''t seem to give away any details like this. >>>> >>>> I also thought about using "enq_" to schedule but is there a way to >>>> specify >>>> that the trigger is the completion of the previous scheduled >>>> instance and >>>> so >>>> on? Or is the scheduling purely time based? >>>> >>>> Kind regards, >>>> >>>> Chad. >>>> >>>> _______________________________________________ >>>> Backgroundrb-devel mailing list >>>> Backgroundrb-devel at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>>> >>> >>> >>> >>> -- >>> Let them talk of their oriental summer climes of everlasting >>> conservatories; give me the privilege of making my own summer with >>> my >>> own coals. >>> >>> http://gnufied.org >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> > > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org