Backgroundrb devel - Sep 2008 - Best way to run hundreds of concurrent tasks?

Hi all,

I''m trying to figure out to run some asynchronous tasks in a Facebook
app.
I''ve got things working, but BRB crashes after a little while, and
I''m not
sure if my setup is ideal.

Here''s the scenario:
- whenever a user visits my app, I need to fire off a bunch of API calls.
- these calls need to start right away, because the user sees a
''loading''
screen until they return.
- the job is not calculation or memory-intensive on my end, but it does take
15-20 seconds to get all the responses from Facebook.

I''m on a fairly small server, so it''s important to keep memory
use low (and
stability high), while still being able to handle thousands of simultaneous
visitors. Right now I''m using a single worker, and each visitor
generates a
new thread in the thread pool. After a little while tasks only get a few
calls in before failing with a "Connection reset by peer" error.
Eventually
my worker just disappears altogether.

What''s going on? What kind of setup should I be running?

Thanks a lot for a great plugin...
Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080914/638c20a6/attachment.html>

On Sun, 2008-09-14 at 19:34 -0700, Robert Matei wrote:
> Hi all, 
> 
> I''m trying to figure out to run some asynchronous tasks in a
Facebook
> app. I''ve got things working, but BRB crashes after a little
while,
> and I''m not sure if my setup is ideal. 
> 
> Here''s the scenario:
> - whenever a user visits my app, I need to fire off a bunch of API
> calls. 
> - these calls need to start right away, because the user sees a
> ''loading'' screen until they return. 
> - the job is not calculation or memory-intensive on my end, but it
> does take 15-20 seconds to get all the responses from Facebook. 
> 
> I''m on a fairly small server, so it''s important to keep
memory use low
> (and stability high), while still being able to handle thousands of
> simultaneous visitors. Right now I''m using a single worker, and
each
> visitor generates a new thread in the thread pool. After a little
> while tasks only get a few calls in before failing with a "Connection
> reset by peer" error. Eventually my worker just disappears
> altogether. 
> 
> What''s going on? What kind of setup should I be running?
What version of BDRB? What version of packet gem? Which OS?

Are you using mechanize or something for interacting with facebook? or
just normal RFacebook or something?

BDRB is the latest version from the SVN repository (supposed to mirror
stable git releases)
Packet 0.1.13
using Facebooker to interact with Facebook.


On Sun, Sep 14, 2008 at 9:30 PM, hemant kumar <gethemant at gmail.com>
wrote:
> On Sun, 2008-09-14 at 19:34 -0700, Robert Matei wrote:
> > Hi all,
> >
> > I''m trying to figure out to run some asynchronous tasks in a
Facebook
> > app. I''ve got things working, but BRB crashes after a little
while,
> > and I''m not sure if my setup is ideal.
> >
> > Here''s the scenario:
> > - whenever a user visits my app, I need to fire off a bunch of API
> > calls.
> > - these calls need to start right away, because the user sees a
> > ''loading'' screen until they return.
> > - the job is not calculation or memory-intensive on my end, but it
> > does take 15-20 seconds to get all the responses from Facebook.
> >
> > I''m on a fairly small server, so it''s important to
keep memory use low
> > (and stability high), while still being able to handle thousands of
> > simultaneous visitors. Right now I''m using a single worker,
and each
> > visitor generates a new thread in the thread pool. After a little
> > while tasks only get a few calls in before failing with a
"Connection
> > reset by peer" error. Eventually my worker just disappears
> > altogether.
> >
> > What''s going on? What kind of setup should I be running?
>
> What version of BDRB? What version of packet gem? Which OS?
>
> Are you using mechanize or something for interacting with facebook? or
> just normal RFacebook or something?
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080914/04311bd2/attachment.html>

Can you paste the worker code? I am not sure, whats causing this.
Which OS btw?



On Sun, 2008-09-14 at 21:58 -0700, Robert Matei wrote:
> BDRB is the latest version from the SVN repository (supposed to mirror
> stable git releases)
> Packet 0.1.13
> using Facebooker to interact with Facebook. 
> 
> 
> On Sun, Sep 14, 2008 at 9:30 PM, hemant kumar <gethemant at
gmail.com>
> wrote:
>         
>         On Sun, 2008-09-14 at 19:34 -0700, Robert Matei wrote:
>         > Hi all,
>         >
>         > I''m trying to figure out to run some asynchronous
tasks in a
>         Facebook
>         > app. I''ve got things working, but BRB crashes after a
little
>         while,
>         > and I''m not sure if my setup is ideal.
>         >
>         > Here''s the scenario:
>         > - whenever a user visits my app, I need to fire off a bunch
>         of API
>         > calls.
>         > - these calls need to start right away, because the user
>         sees a
>         > ''loading'' screen until they return.
>         > - the job is not calculation or memory-intensive on my end,
>         but it
>         > does take 15-20 seconds to get all the responses from
>         Facebook.
>         >
>         > I''m on a fairly small server, so it''s
important to keep
>         memory use low
>         > (and stability high), while still being able to handle
>         thousands of
>         > simultaneous visitors. Right now I''m using a single
worker,
>         and each
>         > visitor generates a new thread in the thread pool. After a
>         little
>         > while tasks only get a few calls in before failing with a
>         "Connection
>         > reset by peer" error. Eventually my worker just
disappears
>         > altogether.
>         >
>         > What''s going on? What kind of setup should I be
running?
>         
>         
>         What version of BDRB? What version of packet gem? Which OS?
>         
>         Are you using mechanize or something for interacting with
>         facebook? or
>         just normal RFacebook or something?
>         
>         
>         
>         
> 
>