Mongrel users - Aug 2008 - Unresponsive Mongrel Windows 2003 Timeout Shutdown

I have a very strange issue going on that just started happening last
week.  I am running an apache/mongrel stack on a windows 2003 box
connecting to a sql server 2005.

Apache 2.2.3
Mongrel 1.1.5

Since last Thursday I have had 5 crashes where the mongrel service
becomes unresponsive.   If I restart the service all is fine.  Nothing
is being reported in the rails logs... But I did notice something in the
mongrel logs.

Each time it look like I get the following log entry:

** INT signal received.
Wed Aug 13 15:01:20 -0500 2008: Reaping 103 threads for slow workers
because of ''shutdown''
Thread #<Thread:0x8553270 sleep> is too old, killing.
Wed Aug 13 15:01:20 -0500 2008: Error calling Dispatcher.dispatch
#<Mongrel::TimeoutError: Mongrel timed out this thread: shutdown>
c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.5-x86-mswin32-60/bin/../lib/mongrel.rb:221:in
`process''
c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.5-x86-mswin32-60/bin/../lib/mongrel.rb:159:in
`process_client''
c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.5-x86-mswin32-60/bin/../lib/mongrel.rb:158:in
`each''

I have seen this pattern repeated for each of the crashes.   I have 5
instances running and about 50 concurrent users who are banging away at
this web application.

Any advice or thoughts would be much appreciated.

Thanks in advance
-- 
Posted via http://www.ruby-forum.com/.

On Wed, Aug 13, 2008 at 11:45 PM, Chris Schmitt <lists at ruby-forum.com>
wrote:
> I have a very strange issue going on that just started happening last
> week.  I am running an apache/mongrel stack on a windows 2003 box
> connecting to a sql server 2005.
>
> Apache 2.2.3
> Mongrel 1.1.5
>
> Since last Thursday I have had 5 crashes where the mongrel service
> becomes unresponsive.   If I restart the service all is fine.  Nothing
> is being reported in the rails logs... But I did notice something in the
> mongrel logs.
>
> Each time it look like I get the following log entry:
>
> ** INT signal received.
> Wed Aug 13 15:01:20 -0500 2008: Reaping 103 threads for slow workers
> because of ''shutdown''
> Thread #<Thread:0x8553270 sleep> is too old, killing.
> Wed Aug 13 15:01:20 -0500 2008: Error calling Dispatcher.dispatch
> #<Mongrel::TimeoutError: Mongrel timed out this thread: shutdown>
>
c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.5-x86-mswin32-60/bin/../lib/mongrel.rb:221:in
> `process''
>
c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.5-x86-mswin32-60/bin/../lib/mongrel.rb:159:in
> `process_client''
>
c:/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.5-x86-mswin32-60/bin/../lib/mongrel.rb:158:in
> `each''
>
> I have seen this pattern repeated for each of the crashes.   I have 5
> instances running and about 50 concurrent users who are banging away at
> this web application.
>
> Any advice or thoughts would be much appreciated.
>
> Thanks in advance
Hey Chris,

I have seen this, but with mysql.

When you try to stop the mongrel_service, mongrel is trying to stop
all the threads that are pending, so that is what you see is being
logged in the mongrel log.

There are several ways to end having "stalled" threads, a few of them
involves timeouts and disconnections from the database.

You can check the docs about that here:
http://mongrel.rubyforge.org/wiki/FAQ#WhydoIgetMySQLlostconnectiontodatabaseerrors

The other issue can be related to code that is being doing something
wrong in some request and just never finish those.
This FAQ section describe what to do on a *nix environment:

http://mongrel.rubyforge.org/wiki/FAQ#WhydoesMongrelkeepdyingonme

But since you''re not there, you can:

1) remove and reinstall the mongrel_service adding the debugging
option (-B or --debug)
2) Start the service
3) when you see it is staled, stop it and collect the log information
to pinpoint which thread and which library break it..

Hope this help you, but without more information about your Rails
version, Ruby version (and patchlevel) and some lirbaries involved I
cannot comment further than that :-D

Regards,
-- 
Luis Lavena
AREA 17
-
Human beings, who are almost unique in having the ability to learn from
the experience of others, are also remarkable for their apparent
disinclination to do so.
Douglas Adams

Thanks Luis..  I will check those links out.

-- 
Posted via http://www.ruby-forum.com/.