Raghu Srinivasan
2008-May-30 04:05 UTC
[Backgroundrb-devel] Upper limit to number of jobs queued by BDRb?
I use BDRb to process RSS feeds for users on my site (http://feedflix.com). I have a batch job that queries the DB for records that haven''t been updated in the last so many hours and kicks off a background job for each of them. If N records are returned by the DB, N background jobs gets queued and get done serially. As long as N is 255 or under, everything works like a charm. I''ve noticed that whenever N is >= 256 (2 power 8), then at the 257th job BDRb stops processing any more users. I can get around it by limiting the DB query to return no more than 255 records and then all is fine. No problems at all. But over that, I see this issue. Repeatedly. Here''s what I found in my log/backgroundrb_11007_debug.log file. ========/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in `start_reactor'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in `run'' /home/raghus/public_html/ feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in`initialize'' ./script/backgroundrb:42:in `new'' ./script/backgroundrb:42 {:type=>:do_work, :worker=>:batch_worker, :data=>{:body=>"All Done!!!", :subject=>"Completed processing for : 275 users"}, :worker_method=>:email_admin} You have a nil object when you didn''t expect it! The error occurred while evaluating nil.instance /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:145:in `schedule_write'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:57:in `write_and_schedule'' /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/whiny_nil.rb:35:in `each_with_index'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in `each'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in `each_with_index'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in `write_and_schedule'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:72:in `dump_object'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_pimp.rb:23:in `send_data'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:76:in `send_request'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:44:in `ask_worker'' /home/raghus/public_html/ feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:104:in`process_work'' /home/raghus/public_html/ feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:35:in`receive_data'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:29:in `call'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:29:in `extract'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:42:in `extract'' /home/raghus/public_html/ feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:31:in`receive_data'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:212:in `read_external_socket'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:204:in `handle_external_messages'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:178:in `handle_read_event'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in `each'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in `handle_read_event'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in `start_reactor'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in `loop'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in `start_reactor'' /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in `run'' /home/raghus/public_html/ feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in`initialize'' ./script/backgroundrb:42:in `new'' ./script/backgroundrb:42 ========My log/backgroundrb_server_11007.log is clean - there are no error messages at all there. Is anyone else seeing this? Am I missing something obvious? Any solutions? Thanks, Raghu -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080529/dced6a98/attachment.html>
hemant
2008-May-30 16:17 UTC
[Backgroundrb-devel] Upper limit to number of jobs queued by BDRb?
On Fri, May 30, 2008 at 9:35 AM, Raghu Srinivasan <raghu.srinivasan at gmail.com> wrote:> I use BDRb to process RSS feeds for users on my site (http://feedflix.com). > > I have a batch job that queries the DB for records that haven''t been updated > in the last so many hours and kicks off a background job for each of them. > If N records are returned by the DB, N background jobs gets queued and get > done serially. As long as N is 255 or under, everything works like a charm. > I''ve noticed that whenever N is >= 256 (2 power 8), then at the 257th job > BDRb stops processing any more users. I can get around it by limiting the DB > query to return no more than 255 records and then all is fine. No problems > at all. But over that, I see this issue. Repeatedly. >How are you queuing the jobs? Are you using thread_pool? I am afraid, it could be because of restriction in number of open file descriptors open to 1024.
Raghu Srinivasan
2008-May-30 17:25 UTC
[Backgroundrb-devel] Upper limit to number of jobs queued by BDRb?
Hi Hemant - no, I am not using thread_pool right now. I do have two separate workers like you and Ryan Leavengood suggested - one for the batch process and the other for the live/web user initiated process - which by the way works out great, thanks!. How are other folks handling 1000s of RSS refreshes? Via BDRb - or something else? Is BDRb really the best tool for what I am trying to do? I''d really appreciate if others could share their experiences. Thanks in advance On Fri, May 30, 2008 at 9:17 AM, hemant <gethemant at gmail.com> wrote:> On Fri, May 30, 2008 at 9:35 AM, Raghu Srinivasan > <raghu.srinivasan at gmail.com> wrote: > > I use BDRb to process RSS feeds for users on my site ( > http://feedflix.com). > > > > I have a batch job that queries the DB for records that haven''t been > updated > > in the last so many hours and kicks off a background job for each of > them. > > If N records are returned by the DB, N background jobs gets queued and > get > > done serially. As long as N is 255 or under, everything works like a > charm. > > I''ve noticed that whenever N is >= 256 (2 power 8), then at the 257th job > > BDRb stops processing any more users. I can get around it by limiting the > DB > > query to return no more than 255 records and then all is fine. No > problems > > at all. But over that, I see this issue. Repeatedly. > > > > How are you queuing the jobs? Are you using thread_pool? I am afraid, > it could be because of restriction in number of open file descriptors > open to 1024. >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080530/3c0ccdad/attachment.html>
hemant kumar
2008-May-30 19:01 UTC
[Backgroundrb-devel] Upper limit to number of jobs queued by BDRb?
On Fri, 2008-05-30 at 10:25 -0700, Raghu Srinivasan wrote:> Hi Hemant - no, I am not using thread_pool right now. I do have two > separate workers like you and Ryan Leavengood suggested - one for the > batch process and the other for the live/web user initiated process - > which by the way works out great, thanks!. > > How are other folks handling 1000s of RSS refreshes? Via BDRb - or > something else? Is BDRb really the best tool for what I am trying to > do? I''d really appreciate if others could share their experiences. >okay, then it shouldn''t be a problem. Can you try git version of backgroundrb as suggest in following link and report us back? http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
Raghu Srinivasan
2008-May-30 19:38 UTC
[Backgroundrb-devel] Upper limit to number of jobs queued by BDRb?
On Fri, May 30, 2008 at 12:01 PM, hemant kumar <gethemant at gmail.com> wrote:> On Fri, 2008-05-30 at 10:25 -0700, Raghu Srinivasan wrote: > > Hi Hemant - no, I am not using thread_pool right now. I do have two > > separate workers like you and Ryan Leavengood suggested - one for the > > batch process and the other for the live/web user initiated process - > > which by the way works out great, thanks!. > > > > How are other folks handling 1000s of RSS refreshes? Via BDRb - or > > something else? Is BDRb really the best tool for what I am trying to > > do? I''d really appreciate if others could share their experiences. > > > > okay, then it shouldn''t be a problem. Can you try git version of > backgroundrb as suggest in following link and report us back? > > > http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/ > > > > Hemant - Thanks for your help. I''ll try this first on development and thenreport back. With this, will I be able to queue up 256+ or 1024+ b/g jobs? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080530/9b7f12cc/attachment.html>
hemant
2008-May-31 12:17 UTC
[Backgroundrb-devel] Upper limit to number of jobs queued by BDRb?
On Sat, May 31, 2008 at 1:08 AM, Raghu Srinivasan <raghu.srinivasan at gmail.com> wrote:> > > On Fri, May 30, 2008 at 12:01 PM, hemant kumar <gethemant at gmail.com> wrote: >> >> On Fri, 2008-05-30 at 10:25 -0700, Raghu Srinivasan wrote: >> > Hi Hemant - no, I am not using thread_pool right now. I do have two >> > separate workers like you and Ryan Leavengood suggested - one for the >> > batch process and the other for the live/web user initiated process - >> > which by the way works out great, thanks!. >> > >> > How are other folks handling 1000s of RSS refreshes? Via BDRb - or >> > something else? Is BDRb really the best tool for what I am trying to >> > do? I''d really appreciate if others could share their experiences. >> > >> >> okay, then it shouldn''t be a problem. Can you try git version of >> backgroundrb as suggest in following link and report us back? >> >> >> http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/ >> >> >> > Hemant - Thanks for your help. I''ll try this first on development and then > report back. With this, will I be able to queue up 256+ or 1024+ b/g jobs? >Why, the newer version will fix many of the problem is because, newer version makes use of fork and exec, rather than just fork(). What it means is, each of worker process is totally independent of each other. So, even if you are opening lots of files and network resources in your workers, they stay completely independent of each other. Now, Ruby runtime is not copy on write friendly and hence with plain fork it copies all the resources in forked processes and hence causing some memory issues. -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org