Mongrel users - Dec 2007 - Long-running pages cause mongrel time-outs

Hi all,

We use mongrel behind apache (using mod_proxy_balancer) to run an internal
app for our business. Lately we''ve been seeing frequent errors from
some of
our mongrel processes.

[Sun Dec 16 08:48:47 2007] [error] [client <CLIENT1>] (70007)The timeout
specified has expired: proxy: error reading status line from remote server
127.0.0.1, referer: http://<long-running-request>
[Sun Dec 16 08:48:47 2007] [error] [client <CLIENT1>] proxy: Error reading
from remote server returned by /<long-running-request>, referer:
http://<long-running-request>
[Sun Dec 16 08:49:08 2007] [error] [client <CLIENT2>] (70007)The timeout
specified has expired: proxy: error reading status line from remote server
127.0.0.1
[Sun Dec 16 08:49:08 2007] [error] [client <CLIENT2>] proxy: Error reading
from remote server returned by /<not-so-long-running-request>

The problem seems to be that the long-running request hogs a mongrel
process, and when apache sends the next request to it (after a round-robin
tour through the other 9 mongrels), that request times out too. The original
action eventually comes back, but sometimes not for several minutes, so we
can timeout on many requests in a row.

I''ve seen problems like this mentioned in the context of file uploads
(which
explain some of our long-running requests), but I''ve never seen a fix
posted. Maybe I haven''t looked hard enough? Does anyone know of a
config fix
that will skip over these blocked processes?

Thanks!
Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20071217/ad763669/attachment.html

On Dec 17, 2007, at 4:10 PM, Michael Van Wie wrote:
> Hi all,
>
> We use mongrel behind apache (using mod_proxy_balancer) to run an  
> internal app for our business. Lately we''ve been seeing frequent  
> errors from some of our mongrel processes.
>
> [Sun Dec 16 08:48:47 2007] [error] [client <CLIENT1>] (70007)The  
> timeout specified has expired: proxy: error reading status line from  
> remote server 127.0.0.1, referer: http://<long-running-request>
> [Sun Dec 16 08:48:47 2007] [error] [client <CLIENT1>] proxy: Error  
> reading from remote server returned by /<long-running-request>,  
> referer: http://<long-running-request>
> [Sun Dec 16 08:49:08 2007] [error] [client <CLIENT2>] (70007)The  
> timeout specified has expired: proxy: error reading status line from  
> remote server 127.0.0.1
> [Sun Dec 16 08:49:08 2007] [error] [client <CLIENT2>] proxy: Error  
> reading from remote server returned by /<not-so-long-running-request>
>
> The problem seems to be that the long-running request hogs a mongrel  
> process, and when apache sends the next request to it (after a round- 
> robin tour through the other 9 mongrels), that request times out  
> too. The original action eventually comes back, but sometimes not  
> for several minutes, so we can timeout on many requests in a row.
>
> I''ve seen problems like this mentioned in the context of file  
> uploads (which explain some of our long-running requests), but
I''ve
> never seen a fix posted. Maybe I haven''t looked hard enough? Does
> anyone know of a config fix that will skip over these blocked  
> processes?
>
> Thanks!
> Mike
>
	The best thing to do is to move any of your long running tasks off  
into a work queue and out of the mongrel/rails request response cycle.  
Something like Bj woudl help:

http://codeforpeople.rubyforge.org/svn/bj/trunk/README

	But I think there has been apache mod_proxy_balancer configs posted  
to this list in the past few months that will only send one request to  
a mongrel at a time, search the archives.

	or if you can use nginx instead there is a new fair proxy balancer  
for nginx that works well.

http://brainspl.at/articles/2007/11/09/a-fair-proxy-balancer-for-nginx-and-mongrel

Cheers-
- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra at engineyard.com
-- EngineYard.com

Hi Ezra --

I found it interesting that you recommended Bj (backgroundjobs) and not
BackgrounDrb. I hadn''t heard of Bj before, and I''ll check it
out. Thanks for
the link. Given that you initiated the development BackgrounDrb, I''m
curious
when and where you would recommend Bj over BackgrounDrb.

Mason


On Dec 17, 2007 6:18 PM, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
>
>        The best thing to do is to move any of your long running tasks off
> into a work queue and out of the mongrel/rails request response cycle.
> Something like Bj woudl help:
>
> http://codeforpeople.rubyforge.org/svn/bj/trunk/README
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20071218/735991e7/attachment.html

There''s a library called Spawn, too, which is nice for doing
asynchronous but still request-triggered background processing. It
might get less nice once 1.9 comes around, though, but... bridges when
we come to them.

Evan

On Dec 18, 2007 1:34 PM, Mason Hale <masonhale at gmail.com>
wrote:
> Hi Ezra --
>
> I found it interesting that you recommended Bj (backgroundjobs) and not
> BackgrounDrb. I hadn''t heard of Bj before, and I''ll check
it out. Thanks for
> the link. Given that you initiated the development BackgrounDrb,
I''m curious
> when and where you would recommend Bj over BackgrounDrb.
>
> Mason
>
>
>
>
> On Dec 17, 2007 6:18 PM, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
> >
> >
> >
> >
> >        The best thing to do is to move any of your long running tasks
off
> > into a work queue and out of the mongrel/rails request response cycle.
> > Something like Bj woudl help:
> >
> > http://codeforpeople.rubyforge.org/svn/bj/trunk/README
> >
> >
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>


-- 
Evan Weaver
Cloudburst, LLC

Evan Weaver wrote:
> There''s a library called Spawn, too, which is nice for doing
> asynchronous but still request-triggered background processing. It
> might get less nice once 1.9 comes around, though, but... bridges when
> we come to them.
My Google-fu is eluding me. Can someone point me to somewhere I can 
learn more about Spawn?

-- 
Cheers,
- Jacob Atzen

On Dec 18, 2007, at 1:38 PM, Jacob Atzen wrote:
> Evan Weaver wrote:
>> There''s a library called Spawn, too, which is nice for doing
>> asynchronous but still request-triggered background processing. It
>> might get less nice once 1.9 comes around, though, but... bridges  
>> when
>> we come to them.
>
> My Google-fu is eluding me. Can someone point me to somewhere I can
> learn more about Spawn?
>
> -- 
> Cheers,
> - Jacob Atzen
http://rubyforge.org/projects/spawn/

- Jason

http://rubyforge.org/projects/spawn/

On Dec 18, 2007 4:38 PM, Jacob Atzen <jacob at jacobatzen.dk>
wrote:
> Evan Weaver wrote:
> > There''s a library called Spawn, too, which is nice for doing
> > asynchronous but still request-triggered background processing. It
> > might get less nice once 1.9 comes around, though, but... bridges when
> > we come to them.
>
> My Google-fu is eluding me. Can someone point me to somewhere I can
> learn more about Spawn?
>
> --
> Cheers,
> - Jacob Atzen
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>


-- 
Evan Weaver
Cloudburst, LLC

At 02:42 AM 12/19/2007, mongrel-users-request at rubyforge.org
wrote:
>Message: 6
>Date: Tue, 18 Dec 2007 12:34:54 -0600
>From: "Mason Hale" <masonhale at gmail.com>
>Subject: Re: [Mongrel] Long-running pages cause mongrel time-outs
>To: mongrel-users at rubyforge.org
>Message-ID:
>         <8bca3aa10712181034m79968dd4he6fa91ea44e3d2d6 at
mail.gmail.com>
>Content-Type: text/plain; charset="iso-8859-1"
>
>Hi Ezra --
>
>I found it interesting that you recommended Bj (backgroundjobs) and 
>not
>BackgrounDrb. I hadn''t heard of Bj before, and I''ll check
it out.
>Thanks for
>the link. Given that you initiated the development BackgrounDrb,
I''m
>curious
>when and where you would recommend Bj over BackgrounDrb.
>
>Mason
Hi Mason,

I found this link which clearly illuminates your question.

http://groups.google.com/group/ruby-talk-google/browse_frm/thread/144e9b3644ffe88e#38b99ec30098b956

In short: BackgroundRb relies on slave.rb (maintained by Ara T Howard, 
the author of Bj), and Ara claims only one person (Eric Hodel) 
understands how to use slave.rb properly: so BackgroundRb is just too 
complex to maintain (too much flexibility). Bj is specifically designed 
to address this issue.

Erza''s company EngineYard is one of the sponsors of Bj - so
it''s
designed to meet some business needs of theirs (I''m a customer so that 
includes me!).

 From my analysis, Bj looks good b/c it puts the jobs into a database 
table so things are both persistent and scalable with very little 
additional infrastructure.

Steve

On Dec 19, 2007 10:31 PM, Steve Midgley <public at misuse.org>
wrote:
> At 02:42 AM 12/19/2007, mongrel-users-request at rubyforge.org wrote:
> >Message: 6
> >Date: Tue, 18 Dec 2007 12:34:54 -0600
> >From: "Mason Hale" <masonhale at gmail.com>
> >Subject: Re: [Mongrel] Long-running pages cause mongrel time-outs
> >To: mongrel-users at rubyforge.org
> >Message-ID:
> >         <8bca3aa10712181034m79968dd4he6fa91ea44e3d2d6 at
mail.gmail.com>
> >Content-Type: text/plain; charset="iso-8859-1"
> >
> >Hi Ezra --
> >
> >I found it interesting that you recommended Bj (backgroundjobs) and
> >not
> >BackgrounDrb. I hadn''t heard of Bj before, and I''ll
check it out.
> >Thanks for
> >the link. Given that you initiated the development BackgrounDrb,
I''m
> >curious
> >when and where you would recommend Bj over BackgrounDrb.
> >
> >Mason
>
> Hi Mason,
>
> I found this link which clearly illuminates your question.
>
>
http://groups.google.com/group/ruby-talk-google/browse_frm/thread/144e9b3644ffe88e#38b99ec30098b956
>
> In short: BackgroundRb relies on slave.rb (maintained by Ara T Howard,
> the author of Bj), and Ara claims only one person (Eric Hodel)
> understands how to use slave.rb properly: so BackgroundRb is just too
> complex to maintain (too much flexibility). Bj is specifically designed
> to address this issue.
Thats just one side of the story. BackgrounDRb no longer depends
slave.rb. Its stable as hell. There is not a single bug in
BackgrounDRb that has been reported and not solved. You can checkout
activity on BackgrounDRb mailing list.

BackgrounDRb is not Bj however, It can be if you want to use it like
that. Please check your facts before making any such claims.
>
> Erza''s company EngineYard is one of the sponsors of Bj - so
it''s
> designed to meet some business needs of theirs (I''m a customer so
that
> includes me!).
>
>  From my analysis, Bj looks good b/c it puts the jobs into a database
> table so things are both persistent and scalable with very little
> additional infrastructure.
>
> Steve
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

On Dec 19, 2007, at 9:01 AM, Steve Midgley wrote:
> At 02:42 AM 12/19/2007, mongrel-users-request at rubyforge.org wrote:
>> Message: 6
>> Date: Tue, 18 Dec 2007 12:34:54 -0600
>> From: "Mason Hale" <masonhale at gmail.com>
>> Subject: Re: [Mongrel] Long-running pages cause mongrel time-outs
>> To: mongrel-users at rubyforge.org
>> Message-ID:
>>        <8bca3aa10712181034m79968dd4he6fa91ea44e3d2d6 at
mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Hi Ezra --
>>
>> I found it interesting that you recommended Bj (backgroundjobs) and
>> not
>> BackgrounDrb. I hadn''t heard of Bj before, and I''ll
check it out.
>> Thanks for
>> the link. Given that you initiated the development BackgrounDrb,
I''m
>> curious
>> when and where you would recommend Bj over BackgrounDrb.
>>
>> Mason


	Backgroundrb and Bj serve different purposes. The new backgroundrb is  
a complete rewrite by Hemant that throws out drb and uses an event  
driven styel of programming. It really is very stable now and it''s  
main focus is on doing jobs that require reporting status in your  
rails app. Like if you want to fetch some news feeds and report a  
progress bar via ajax polling in your app. Bdrb works very well for  
this and other more real time background jobs.


	Bj is a work queue. You just shove jobs into the queue which is a  
database table and the Bj daemon will run through them and call each  
job one at a time. But you can also have Bj daemons on each node in a  
cluster of servers and have them all pulling from the same queue. Bj  
jobs are just shell commands. So your actual jobs can be ruby scripts,  
shell scripts, or script/runner calls.  The main idea behind Bj is  
that you could throw 100K jobs at it and it will just do them as fast  
as it can one at a time without overwhelming the system all at once.


	So both of these tools are very solid now, they just serve different  
purposes. Backgroiundrb is more for real time jobs and status reports  
as well as having a very nice cron like scheduler for jobs.  Bj is  
more of a work queue that is persistent.


	Hope that clears things up, Both of these tools are great for  
different circumstances.

Cheers-
- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra at engineyard.com
-- EngineYard.com