Hello, I''m experiencing a strange, but very bad behavior with Mongrel 0.3.13.4 and Pound 1.8. Every 6 hours or so one of our nine (not consistent on which one) application servers ( each one running several mongrel processes ) will start leaving lots of socket connections with pound open. This leads to "Too many open files" errors. I''ve set pound to close connections after 60 seconds ( after doing analysis of our Rails logs and finding that all requests end under 60 seconds, and only 0.12% take over 1 second ). Pound closes these, but mongrel apparently isn''t getting the message and I end up with rapidly increasing socket connections on the app server all left in CLOSE_WAIT state ( which is when the remote end, pound, closes the connection, and the OS is waiting for the app, mongrel, to close it on its end ). Before setting Pound to kill these, one app server could effectively take out our web server, because of "Too many open files". The plagued app server still responds, and has low load, CPU usage, and memory usage. I can even access the mongrels on their individual ports. Nothing ( I could see ) shows up in the mongrel log and the Rails logs all show it completing requests in a very timely manner. Therefore I don''t think its a code issue or resource issue. But it''s very odd and disturbing. One other person ( http://poocs.net/articles/2006/03/27/the-adventures-of-scaling-stage-3 ) had a similar problem with lighttpd and FastCGI, so I''m not sure if the problem is Mongrel or maybe just Ruby. That person just set up something to restart the plagued FastCGI processes, and that''s what I''m doing for Mongrel right now. Has anyone else had this problem? Is there something I can do to fix it? Thanks for any insight. Thanks so much! -John Butler
On Wed, 2006-08-23 at 11:47 -0700, John Butler wrote:> Hello, > > I''m experiencing a strange, but very bad behavior with Mongrel > 0.3.13.4 and Pound 1.8. Every 6 hours or so one of our nine (not > consistent on which one) application servers ( each one running > several mongrel processes ) will start leaving lots of socket > connections with pound open. This leads to "Too many open files" > errors. I''ve set pound to close connections after 60 seconds ( after > doing analysis of our Rails logs and finding that all requests end > under 60 seconds, and only 0.12% take over 1 second ). Pound closes > these, but mongrel apparently isn''t getting the message and I end up > with rapidly increasing socket connections on the app server all left > in CLOSE_WAIT state ( which is when the remote end, pound, closes the > connection, and the OS is waiting for the app, mongrel, to close it on > its end ). Before setting Pound to kill these, one app server could > effectively take out our web server, because of "Too many open files". > The plagued app server still responds, and has low load, CPU usage, > and memory usage. I can even access the mongrels on their individual > ports. Nothing ( I could see ) shows up in the mongrel log and the > Rails logs all show it completing requests in a very timely manner. > Therefore I don''t think its a code issue or resource issue. But it''s > very odd and disturbing.Hey Jon, is it possible for you to come trouble shoot this with me in the unofficial mongrel support room: http://www.lingr.com/room/3yXhqKbfPy8 It''ll be easier for me to have you run through some tests that way. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
we''re running 0.3.13 and had this pop up just the other day. we''re running 4 instances for our app behind apache/mod_proxy_balancer and this is the first time it has happened (we''ve been running for a month or two). the app was not under heavy load at the time ( < 10 req/s). Linux XXX 2.6.16.13-4-smp #1 SMP Wed May 3 04:53:23 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux Apache/2.2.0 mongrel (0.3.13) rails (1.1.6) ruby 1.8.4 (2005-12-24) [x86_64-linux] jeff On 8/23/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> > On Wed, 2006-08-23 at 11:47 -0700, John Butler wrote: > > Hello, > > > > I''m experiencing a strange, but very bad behavior with Mongrel > > 0.3.13.4 and Pound 1.8. Every 6 hours or so one of our nine (not > > consistent on which one) application servers ( each one running > > several mongrel processes ) will start leaving lots of socket > > connections with pound open. This leads to "Too many open files" > > errors. I''ve set pound to close connections after 60 seconds ( after > > doing analysis of our Rails logs and finding that all requests end > > under 60 seconds, and only 0.12% take over 1 second ). Pound closes > > these, but mongrel apparently isn''t getting the message and I end up > > with rapidly increasing socket connections on the app server all left > > in CLOSE_WAIT state ( which is when the remote end, pound, closes the > > connection, and the OS is waiting for the app, mongrel, to close it on > > its end ). Before setting Pound to kill these, one app server could > > effectively take out our web server, because of "Too many open files". > > The plagued app server still responds, and has low load, CPU usage, > > and memory usage. I can even access the mongrels on their individual > > ports. Nothing ( I could see ) shows up in the mongrel log and the > > Rails logs all show it completing requests in a very timely manner. > > Therefore I don''t think its a code issue or resource issue. But it''s > > very odd and disturbing. > > Hey Jon, is it possible for you to come trouble shoot this with me in > the unofficial mongrel support room: > > http://www.lingr.com/room/3yXhqKbfPy8 > > It''ll be easier for me to have you run through some tests that way. > > > -- > Zed A. Shaw > http://www.zedshaw.com/ > http://mongrel.rubyforge.org/ > http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20060823/0c4445bd/attachment-0001.html