Mongrel users - Jun 2009 - Mongrel Hanging (Apache, mod_proxy, Mongrel Cluster, Rails)

We have been having a problem with our mongrel servers hanging at
seemingly random times.

The problem can be most often reproduced by accessing a page, leaving
the browser open for some time (60+ min) then making another request.
The browser will receive a "Proxy Error" from Apache, If we look at
mongrel via "ps aux" it says

"mongrel_rails [8000/1/253]: handling 127.0.0.1: GET /"

Using strace we pulled this:

14:04:15 poll([{fd=12, events=POLLIN|POLLPRI}], 1, 0) = 0
14:04:15 write(12, "\1\0\0\0\t", 5)     = 5
14:04:15 read(12,

SEVER IS LOCKED HERE...

Then 15 min later...

14:04:15 read(12, 0x8eab758, 16384)     = -1 ETIMEDOUT (Connection timed out)
14:19:40 shutdown(12, 2 /* send and receive */) = -1 ENOTCONN
(Transport endpoint is not connected)
14:19:40 close(12)                      = 0
14:19:40 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
14:19:40 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 12
14:19:40 fcntl64(12, F_SETFL, O_RDONLY) = 0
14:19:40 fcntl64(12, F_GETFL)           = 0x2 (flags O_RDWR)
14:19:40 connect(12, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr("XXX.XXX.XXX.XXX")}, 16) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_RCVTIMEO,
"\2003\341\1\0\0\0\0", 8) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_SNDTIMEO,
"\2003\341\1\0\0\0\0", 8) = 0
14:19:40 setsockopt(12, SOL_IP, IP_TOS, [8], 4) = 0
14:19:40 setsockopt(12, SOL_TCP, TCP_NODELAY, [1], 4) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
14:19:40 read(12, "F\0\0\0\n5.0.60sp1-enterprise-gpl\0\3679"...,
16384) = 74

We have tried all the results we can find on the internet regarding
"Mongrel Hanging"

For Example:

We are NOT using ruby log rolling
We ARE using the c compiled mysql Adapter
We have tried setting "ActiveRecord::Base.verification_timeout =
14400"
The timeout setting on MySQL are default (interactive_timout,
wait_timeout) = 28800

We have tried the TCP keepalive setting found in here:
http://wooga.drbacchus.com/rails-and-mysql-timeouts

We have tried setting an apache setting:
#Fix for Apache bug 39499
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

We have even tried many mysql_adapter.rb patches that "correctly"
reconnect the connection on an error.

None of these have made any difference.

We are currently using:
MySQL 5.0.60sp1
Rails 2.0.2
Mongrel 1.1.5

Thanks you very much for any help!

Justin

Hi, Justin.  Were you ever able to get this resolved?  I''m currently 
experiencing the same behavior you describe in your post.  Any help is 
much appreciated.
-- 
Posted via http://www.ruby-forum.com/.

Hi Daniel,

I had the same issue recently and detailled it here :
http://groups.google.ca/group/rubyonrails-talk/browse_thread/thread/a53e75680a98f014/81dfbf8cbb0e7c69?lnk=raot

In my case, as in Justin''s IIRC , the firewall between the mongrel and
the
database caused the connection to drop after some time.

Hope this was helpful

Damien



2010/2/8 Daniel Spring <lists at ruby-forum.com>
> Hi, Justin.  Were you ever able to get this resolved?  I''m
currently
> experiencing the same behavior you describe in your post.  Any help is
> much appreciated.
> --
> Posted via http://www.ruby-forum.com/.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/mongrel-users/attachments/20100208/71235e7b/attachment.html>