Backgroundrb devel - May 2008 - Upper limit to number of jobs queued by BDRb?

I use BDRb to process RSS feeds for users on my site (http://feedflix.com).

I have a batch job that queries the DB for records that haven''t been
updated
in the last so many hours and kicks off a background job for each of them.
If N records are returned by the DB, N background jobs gets queued and get
done serially. As long as N is 255 or under, everything works like a charm.
I''ve noticed that whenever N is >= 256 (2 power 8), then at the
257th job
BDRb stops processing any more users. I can get around it by limiting the DB
query to return no more than 255 records and then all is fine. No problems
at all. But over that, I see this issue. Repeatedly.

Here''s what I found in my log/backgroundrb_11007_debug.log file.
========/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
`run''
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in`initialize''
./script/backgroundrb:42:in `new''
./script/backgroundrb:42
{:type=>:do_work, :worker=>:batch_worker, :data=>{:body=>"All
Done!!!",
:subject=>"Completed processing for : 275 users"},
:worker_method=>:email_admin}
You have a nil object when you didn''t expect it!
The error occurred while evaluating nil.instance
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:145:in
`schedule_write''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:57:in
`write_and_schedule''
/usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/whiny_nil.rb:35:in
`each_with_index''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`each''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`each_with_index''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`write_and_schedule''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:72:in
`dump_object''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_pimp.rb:23:in
`send_data''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:76:in
`send_request''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:44:in
`ask_worker''
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:104:in`process_work''
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:35:in`receive_data''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:29:in
`call''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:29:in
`extract''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:42:in
`extract''
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:31:in`receive_data''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:212:in
`read_external_socket''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:204:in
`handle_external_messages''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:178:in
`handle_read_event''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`each''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`handle_read_event''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
`start_reactor''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`loop''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor''
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
`run''
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in`initialize''
./script/backgroundrb:42:in `new''
./script/backgroundrb:42
========My log/backgroundrb_server_11007.log is clean - there are no error
messages
at all there.
Is anyone else seeing this? Am I missing something obvious? Any solutions?
Thanks,
Raghu
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080529/dced6a98/attachment.html>

On Fri, May 30, 2008 at 9:35 AM, Raghu Srinivasan
<raghu.srinivasan at gmail.com> wrote:
> I use BDRb to process RSS feeds for users on my site (http://feedflix.com).
>
> I have a batch job that queries the DB for records that haven''t
been updated
> in the last so many hours and kicks off a background job for each of them.
> If N records are returned by the DB, N background jobs gets queued and get
> done serially. As long as N is 255 or under, everything works like a charm.
> I''ve noticed that whenever N is >= 256 (2 power 8), then at the
257th job
> BDRb stops processing any more users. I can get around it by limiting the
DB
> query to return no more than 255 records and then all is fine. No problems
> at all. But over that, I see this issue. Repeatedly.
>
How are you queuing the jobs? Are you using thread_pool? I am afraid,
it could be because of restriction in number of open file descriptors
open to 1024.

Hi Hemant - no, I am not using thread_pool right now. I do have two separate
workers like you and Ryan Leavengood suggested - one for the batch process
and the other for the live/web user initiated process - which by the way
works out great, thanks!.

How are other folks handling 1000s of RSS refreshes? Via BDRb - or something
else? Is BDRb really the best tool for what I am trying to do? I''d
really
appreciate if others could share their experiences.

Thanks in advance

On Fri, May 30, 2008 at 9:17 AM, hemant <gethemant at gmail.com> wrote:
> On Fri, May 30, 2008 at 9:35 AM, Raghu Srinivasan
> <raghu.srinivasan at gmail.com> wrote:
> > I use BDRb to process RSS feeds for users on my site (
> http://feedflix.com).
> >
> > I have a batch job that queries the DB for records that
haven''t been
> updated
> > in the last so many hours and kicks off a background job for each of
> them.
> > If N records are returned by the DB, N background jobs gets queued and
> get
> > done serially. As long as N is 255 or under, everything works like a
> charm.
> > I''ve noticed that whenever N is >= 256 (2 power 8), then
at the 257th job
> > BDRb stops processing any more users. I can get around it by limiting
the
> DB
> > query to return no more than 255 records and then all is fine. No
> problems
> > at all. But over that, I see this issue. Repeatedly.
> >
>
> How are you queuing the jobs? Are you using thread_pool? I am afraid,
> it could be because of restriction in number of open file descriptors
> open to 1024.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080530/3c0ccdad/attachment.html>

On Fri, 2008-05-30 at 10:25 -0700, Raghu Srinivasan
wrote:
> Hi Hemant - no, I am not using thread_pool right now. I do have two
> separate workers like you and Ryan Leavengood suggested - one for the
> batch process and the other for the live/web user initiated process -
> which by the way works out great, thanks!. 
> 
> How are other folks handling 1000s of RSS refreshes? Via BDRb - or
> something else? Is BDRb really the best tool for what I am trying to
> do? I''d really appreciate if others could share their experiences.
> 
okay, then it shouldn''t be a problem. Can you try git version of
backgroundrb as suggest in following link and report us back?

http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/

On Fri, May 30, 2008 at 12:01 PM, hemant kumar <gethemant at gmail.com>
wrote:
> On Fri, 2008-05-30 at 10:25 -0700, Raghu Srinivasan wrote:
> > Hi Hemant - no, I am not using thread_pool right now. I do have two
> > separate workers like you and Ryan Leavengood suggested - one for the
> > batch process and the other for the live/web user initiated process -
> > which by the way works out great, thanks!.
> >
> > How are other folks handling 1000s of RSS refreshes? Via BDRb - or
> > something else? Is BDRb really the best tool for what I am trying to
> > do? I''d really appreciate if others could share their
experiences.
> >
>
> okay, then it shouldn''t be a problem. Can you try git version of
> backgroundrb as suggest in following link and report us back?
>
>
>
http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
>
>
>
> Hemant - Thanks for your help. I''ll try this first on development
and then
report back. With this, will I be able to queue up 256+ or 1024+ b/g jobs?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080530/9b7f12cc/attachment.html>

On Sat, May 31, 2008 at 1:08 AM, Raghu Srinivasan
<raghu.srinivasan at gmail.com> wrote:
>
>
> On Fri, May 30, 2008 at 12:01 PM, hemant kumar <gethemant at
gmail.com> wrote:
>>
>> On Fri, 2008-05-30 at 10:25 -0700, Raghu Srinivasan wrote:
>> > Hi Hemant - no, I am not using thread_pool right now. I do have
two
>> > separate workers like you and Ryan Leavengood suggested - one for
the
>> > batch process and the other for the live/web user initiated
process -
>> > which by the way works out great, thanks!.
>> >
>> > How are other folks handling 1000s of RSS refreshes? Via BDRb - or
>> > something else? Is BDRb really the best tool for what I am trying
to
>> > do? I''d really appreciate if others could share their
experiences.
>> >
>>
>> okay, then it shouldn''t be a problem. Can you try git version
of
>> backgroundrb as suggest in following link and report us back?
>>
>>
>>
http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
>>
>>
>>
> Hemant - Thanks for your help. I''ll try this first on development
and then
> report back. With this, will I be able to queue up 256+ or 1024+ b/g jobs?
>
Why, the newer version will fix many of the problem is because, newer
version makes use of fork and exec, rather than just fork(). What it
means is, each of worker process is totally independent of each other.
So, even if you are opening lots of files and network resources in
your workers, they stay completely independent of each other. Now,
Ruby runtime is not copy on write friendly and hence with plain fork
it copies all the resources in forked processes  and hence causing
some memory issues.

-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org