Mongrel users - Jan 2007 - Rails App on Mongrel+Apache Stops Listening

Howdy,

I''m trying to debug a situation where a Rails app running on Mongrel 
+Apache stops listening to requests. Restarting Mongrel and Apache  
brings it back to life.

First off, here''s some info on versions, etc...:

Red Hat Enterprise Linux ES release 3 (Taroon Update 8)
ruby 1.8.5 (2006-12-04 patchlevel 2) [x86_64-linux]
Mongrel 1.0.1
Rails 1.2.1
fastthread (0.6.2)

Basically, after about a day or so, the Web app stops responding.  
Load on the machine is about 0.05-0.10 during this time - low. Memory  
usage is solid but there is still some physical RAM available - it''s  
not out of RAM.

Here''s some log info:

Apache Log:

[Tue Jan 23 03:00:03 2007] [error] (111)Connection refused: proxy:  
HTTP: attempt to connect to 127.0.0.1:8003 (127.0.0.1) failed
[Tue Jan 23 03:00:03 2007] [error] ap_proxy_connect_backend disabling  
worker for (127.0.0.1)
[Tue Jan 23 03:00:03 2007] [error] (111)Connection refused: proxy:  
HTTP: attempt to connect to 127.0.0.1:8000 (127.0.0.1) failed
[Tue Jan 23 03:00:03 2007] [error] ap_proxy_connect_backend disabling  
worker for (127.0.0.1)
[Tue Jan 23 03:00:03 2007] [error] (111)Connection refused: proxy:  
HTTP: attempt to connect to 127.0.0.1:8001 (127.0.0.1) failed
[Tue Jan 23 03:00:03 2007] [error] ap_proxy_connect_backend disabling  
worker for (127.0.0.1)
[Tue Jan 23 03:00:03 2007] [error] (111)Connection refused: proxy:  
HTTP: attempt to connect to 127.0.0.1:8002 (127.0.0.1) failed
[Tue Jan 23 03:00:03 2007] [error] ap_proxy_connect_backend disabling  
worker for (127.0.0.1)
[Tue Jan 23 03:00:03 2007] [error] proxy: BALANCER: (balancer:// 
fatwreck). All workers are in error state
[Tue Jan 23 03:00:08 2007] [error] proxy: BALANCER: (balancer:// 
fatwreck). All workers are in error state
[Tue Jan 23 03:00:08 2007] [error] proxy: BALANCER: (balancer:// 
fatwreck). All workers are in error state
[Tue Jan 23 03:00:09 2007] [error] proxy: BALANCER: (balancer:// 
fatwreck). All workers are in error state
[Tue Jan 23 03:00:09 2007] [error] proxy: BALANCER: (balancer:// 
fatwreck). All workers are in error state
[Tue Jan 23 03:00:11 2007] [error] proxy: BALANCER: (balancer:// 
fatwreck). All workers are in error state
[Tue Jan 23 03:00:12 2007] [error] proxy: BALANCER: (balancer:// 
fatwreck). All workers are in error state
[Tue Jan 23 03:00:14 2007] [error] proxy: BALANCER: (balancer:// 
fatwreck). All workers are in error state

The Mongrel log doesn''t have anything in it of note. Just a USR2  
signal from when I restarted it.

I''ve got four Mongrels running and except for when it is in this  
error state, it works great.

I''m interested to hear any suggestions on what to check. FYI, this  
has been happening LESS often since Mongrel 1.0RC1.

Thanks,
Hunter

On 1/23/07, Hunter Hillegas <lists at lastonepicked.com>
wrote:
> Howdy,
>
> I''m trying to debug a situation where a Rails app running on
Mongrel
> +Apache stops listening to requests. Restarting Mongrel and Apache
> brings it back to life.
>
> First off, here''s some info on versions, etc...:
>
> Red Hat Enterprise Linux ES release 3 (Taroon Update 8)
Are you running stock apache?  If so are you running something between
httpd and the mongrel cluster?

Pat

Hey Zed,

First off, congrats on 1.0 and thanks for all your hard work.

On your list of items, I clearly left out some info that I should  
have included:

1. I''m using PostgreSQL and the C bindings.
2. I''m using the ActiveRecord store.
3. DNS... Okay, I''ll take a look at this. I think we''re okay
here but
I will double check.
4. Not using memcached.
5. Interesting. I''ll check this one too.

I''ll check out your list of debugging suggestions. I may just skip to  
#4 as that''s worked for me in the past.

Cheers,
Hunter

On Jan 24, 2007, at 1:07 AM, Zed A. Shaw wrote:
> On Tue, 23 Jan 2007 21:53:39 -0800
> Hunter Hillegas <lists at lastonepicked.com> wrote:
>
>> Howdy,
>>
>> I''m trying to debug a situation where a Rails app running on
Mongrel
>> +Apache stops listening to requests. Restarting Mongrel and Apache
>> brings it back to life.
>>
>> First off, here''s some info on versions, etc...:
>
> Here''s the shortlist of stuff so far that can cause this:
>
> 1) Using the stock mysql.rb that comes with rails rather than using  
> the compiled mysql gem.
> 2) Use a pstore for sessions, or actually trying to negotiate file  
> locking in general.
> 3) Trying to resolve dns queries against a slow DNS server or  
> broken one.
> 4) Using memcached and putting keys in it with a space or \0 char  
> in them (memcached don''t like that and will block the world).
> 5) An interesting new one is having Monit check the mongrel port on  
> sites with long running requests.  What happens is since rails  
> blocks further requests during these requests, and monit is  
> typically told to restart when they''re slow, you''ll get
frequent
> "slow" requests so monit will frequently restart mongrels for no
> apparent reason.
>
> Ways to debug this include:
>
> 1) Using lsof -i -P to see what''s open.
> 2) Attaching to the stuck process with gdb and using some of the  
> Ruby gdb scripts out there to force an exception to be thrown.   
> Then look in the mongrel.log to see what is logged.
> 3) Running strace or truss on the stuck process to see what system  
> call it''s doing.  If it''s repeatedly running select()
then that
> means it is most likely trying to write to a socket but getting  
> nothing back.  This happens in the memcached case especially.
> 4) Praying to some ancient god for redemption.
>
> I hear #4 works really well if you''re bathed in the blood of 7  
> orphaned virgins, but I''m not that hardcore.
>
> -- 
> Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
> http://www.zedshaw.com/
> http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

I''m running Apache 2.2.4, compiled myself from the source.

Both httpd and Mongrel are running on the same box without any  
software or firewalls in between.

On Jan 23, 2007, at 10:04 PM, Patrick Berry wrote:
> On 1/23/07, Hunter Hillegas <lists at lastonepicked.com> wrote:
>> Howdy,
>>
>> I''m trying to debug a situation where a Rails app running on
Mongrel
>> +Apache stops listening to requests. Restarting Mongrel and Apache
>> brings it back to life.
>>
>> First off, here''s some info on versions, etc...:
>>
>> Red Hat Enterprise Linux ES release 3 (Taroon Update 8)
>
> Are you running stock apache?  If so are you running something between
> httpd and the mongrel cluster?
>
> Pat
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

On Tue, 23 Jan 2007 21:53:39 -0800
Hunter Hillegas <lists at lastonepicked.com> wrote:
> Howdy,
> 
> I''m trying to debug a situation where a Rails app running on
Mongrel
> +Apache stops listening to requests. Restarting Mongrel and Apache  
> brings it back to life.
> 
> First off, here''s some info on versions, etc...:
Here''s the shortlist of stuff so far that can cause this:

1) Using the stock mysql.rb that comes with rails rather than using the compiled
mysql gem.
2) Use a pstore for sessions, or actually trying to negotiate file locking in
general.
3) Trying to resolve dns queries against a slow DNS server or broken one.
4) Using memcached and putting keys in it with a space or \0 char in them
(memcached don''t like that and will block the world).
5) An interesting new one is having Monit check the mongrel port on sites with
long running requests.  What happens is since rails blocks further requests
during these requests, and monit is typically told to restart when
they''re slow, you''ll get frequent "slow" requests so
monit will frequently restart mongrels for no apparent reason.

Ways to debug this include:

1) Using lsof -i -P to see what''s open.
2) Attaching to the stuck process with gdb and using some of the Ruby gdb
scripts out there to force an exception to be thrown.  Then look in the
mongrel.log to see what is logged.
3) Running strace or truss on the stuck process to see what system call
it''s doing.  If it''s repeatedly running select() then that
means it is most likely trying to write to a socket but getting nothing back. 
This happens in the memcached case especially.
4) Praying to some ancient god for redemption.

I hear #4 works really well if you''re bathed in the blood of 7 orphaned
virgins, but I''m not that hardcore.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.