Greetings, I need some help... please be gentle :-) MY SETUP: Solaris-X86, ruby-1.8.4, Apache-2.2.0, Mongrel-0.3.13.4, MySql-4.1.10a I am using the typical setup with apache proxying request to mongrel with mod_proxy_balancer. Currently, I am only running a single instance of mongrel behind the proxy. I am also using the following gems with this app: capistrano (1.2.0) daemons (1.0.2) gem_plugin (0.2.1) mongrel (0.3.13.4, 0.3.13.3) mysql (2.7) needle (1.3.0) rails (1.1.6) ruby-net-ldap (0.0.4) NOTE: My MySql server is running on a different machine than Mongrel. THE PROBLEM The problem, is that after a period of inactivity Mongrel just stops responding. I believe the problem has to do with MySql as mentioned in the FAQ. I''ve confirmed this--I think--by creating a dummy app that doesn''t use MySql for any of the page requests. In this instance, after long periods of inactivity Mongrel works without any problems. WHAT I''VE TRIED: 1) After 20 minutes of inactivity, I try to load the page in the browser. The browser just hangs and eventually times out. I then ssh''d to the machine and did a (curl localhost:50042), mongrel does not respond. I can see that mongrel is still running if I to a (ps -A). Running (top) shows that mongrel is consuming around 34M of memory and using no .01% CPU. I''m running mongrel in development mode, yet no entries are making it to (deveopment.log). I''m also running mongrel in debug_mode and no entries are being added to any of the log files in the mongrel_debug directory. BTW, I am running mongrel on port 50042 because this is the port I''ve been given permission to run Mongrel on. 2) After hitting mongrel with a bunch of (curl locahost:50042) calls, I did (lsof -i -P | grep CLOSE_WAIT) and here is the output: mongrel_r 5522 agile 6u IPv4 0xffffffffa02b8e00 0t617 TCP localhost:50042->localhost:51299 (CLOSE_WAIT) curl 20784 agile 4u IPv4 0xffffffffb1083200 0t0 TCP webfarm-dev.Berkeley.EDU:51330->webfarm-dev.Berkeley.EDU:80 (CLOSE_WAIT) curl 20785 agile 4u IPv4 0xffffffffb9477a00 0t0 TCP webfarm-dev.Berkeley.EDU:51333->webfarm-dev.Berkeley.EDU:80 (CLOSE_WAIT) curl 20788 agile 4u IPv4 0xffffffffb1025c00 0t0 TCP webfarm-dev.Berkeley.EDU:51339->webfarm-dev.Berkeley.EDU:80 (CLOSE_WAIT) curl 20798 agile 4u IPv4 0xffffffffa02b9a00 0t0 TCP webfarm-dev.Berkeley.EDU:51345->webfarm-dev.Berkeley.EDU:80 (CLOSE_WAIT) curl 20799 agile 4u IPv4 0xffffffffb945be00 0t0 TCP webfarm-dev.Berkeley.EDU:51348->webfarm-dev.Berkeley.EDU:80 (CLOSE_WAIT) curl 20805 agile 4u IPv4 0xffffffffb0d7ae00 0t0 TCP webfarm-dev.Berkeley.EDU:51357->webfarm-dev.Berkeley.EDU:80 (CLOSE_WAIT) NOTE: I do not have access to the root account, my user name is "agile". I''m not even sure what (lsof) is telling me here, so if someone could give me a hint, or point me to a url with a good overview, that would be killer. 3) After checking the logs (which show nothing) and running (lsof) I tried to shut mongrel down with (mongrel_rails mongrel::stop -p pids/mongrel.pid I get the response "Sending TERM to Mongrel at PID 5522...Done." However, when I check the log directory the mongrel.pid file is still there and (ps -A) verifies that mongrel is still running. I called the shutdown method 4 more times and on the 4th try mongrel finally stops. I then checked the monrel.log file and found the following: "Thu Nov 16 09:21:00 PST 2006: Reaping 1 threads for slow workers because of ''shutdown'' Thread #<Thread:0x8d54f50 run> is too old, killing. Waiting for 1 requests to finish, could take 60 seconds." 4) I tried adding "ActiveRecord::Base.verification_timeout = "14400" to my rails environment.rb file, as per the FAQ, this had no effect. MOVING FORWARD I''m not really sure where to go from here. Like I said, I have a strong suspicion that MySql is the culprit, but I''m not sure how to fix the problem. If anyone could give me some advice on what to try next, I would really appreciate it. Regards, Steven
On 11/16/06, Steven Hansen <runner at berkeley.edu> wrote:> > Greetings, > > I need some help... please be gentle :-)Hello Steven, and thanks for the good collection of info ;)> NOTE: My MySql server is running on a different machine than Mongrel.Do you have access to any of the MySql server''s configuration data? Which settings (if any) have been altered from the defaults? I ask due to this bit:> 4) I tried adding "ActiveRecord::Base.verification_timeout = "14400" to > my rails > environment.rb file, as per the FAQ, this had no effect.If the settings have been altered per the "Decrease the value of connect_timeout" section of the following document: http://mysqldatabaseadministration.blogspot.com/2005/11/mysql-5-optimization-and-tuning-guide.html I know its suggestions are for MySql 5 (vs. your version), but if changes similar to these have been made on your MySql server, you would need to change the "14400" value in your Rails environment to a lower value than the one set for interactive_timeout on the MySql end of things (assuming the Mongrel FAQ is accurate and current in that regard; I''m new to Mongrel, myself).> MOVING FORWARD > I''m not really sure where to go from here. Like I said, I have a strong > suspicion that MySql > is the culprit, but I''m not sure how to fix the problem. If anyone > could give me some advice > on what to try next, I would really appreciate it.If you could check (with someone, or otherwise) the MySql server-in-question''s configuration to make sure Rails is using a low enough verification_timeout, that would be another positive step, IMO. Maybe it''ll turn up something, as it sounds like the ''timeout'' is happening at approximately 20 minutes for you, vs. documentation I have seen suggests that the default is 480 minutes. Good Luck.> Regards, > StevenSincerely, -Parker
On 11/16/06, Steven Hansen <runner at berkeley.edu> wrote: > > Greetings, > > I need some help... please be gentle :-) Hello Steven, and thanks for the good collection of info ;) > NOTE: My MySql server is running on a different machine than Mongrel. Do you have access to any of the MySql server''s configuration data? Which settings (if any) have been altered from the defaults? I ask due to this bit: > 4) I tried adding "ActiveRecord::Base.verification_timeout = "14400" to > my rails > environment.rb file, as per the FAQ, this had no effect. If the settings have been altered per the "Decrease the value of connect_timeout" section of the following document: http://mysqldatabaseadministration.blogspot.com/2005/11/mysql-5-optimization-and-tuning-guide.html I know its suggestions are for MySql 5 (vs. your version), but if changes similar to these have been made on your MySql server, you would need to change the "14400" value in your Rails environment to a lower value than the one set for interactive_timeout on the MySql end of things (assuming the Mongrel FAQ is accurate and current in that regard; I''m new to Mongrel, myself). > MOVING FORWARD > I''m not really sure where to go from here. Like I said, I have a strong > suspicion that MySql > is the culprit, but I''m not sure how to fix the problem. If anyone > could give me some advice > on what to try next, I would really appreciate it. If you could check (with someone, or otherwise) the MySql server-in-question''s configuration to make sure Rails is using a low enough verification_timeout, that would be another positive step, IMO. Maybe it''ll turn up something, as it sounds like the ''timeout'' is happening at approximately 20 minutes for you, vs. documentation I have seen suggests that the default is 480 minutes. Good Luck. > Regards, > Steven Sincerely, -Parker Hi Parker, Thanks for your suggestions. Sorry for the late reply, it looks like my over zealous spam filter snagged your reply, I stumbled on your email while browsing the mailing list archives :-P I am--for now--administering the machine that the database is running on and I will try your suggestions. FWIW, I fired up good old webrick to see if the same problem would occur. Sure enough, after 20 minutes of inactivity the app would no longer respond through the browser or by doing (curl localhost:50042) while logged on to the machine. Regards, Steven
Steven Hansen wrote:> > > On 11/16/06, Steven Hansen <runner at berkeley.edu> wrote: > > > > Greetings, > > > > I need some help... please be gentle :-) > > Hello Steven, and thanks for the good collection of info ;) > > > NOTE: My MySql server is running on a different machine than Mongrel. > > Do you have access to any of the MySql server''s configuration data? > Which settings (if any) have been altered from the defaults? I ask > due to this bit: > > > 4) I tried adding "ActiveRecord::Base.verification_timeout = "14400" to > > my rails > > environment.rb file, as per the FAQ, this had no effect. > > If the settings have been altered per the "Decrease the value of > connect_timeout" section of the following document: > > http://mysqldatabaseadministration.blogspot.com/2005/11/mysql-5-optimization-and-tuning-guide.html > > > I know its suggestions are for MySql 5 (vs. your version), but if > changes similar to these have been made on your MySql server, you > would need to change the "14400" value in your Rails environment to a > lower value than the one set for interactive_timeout on the MySql end > of things (assuming the Mongrel FAQ is accurate and current in that > regard; I''m new to Mongrel, myself). > > > MOVING FORWARD > > I''m not really sure where to go from here. Like I said, I have a > strong > > suspicion that MySql > > is the culprit, but I''m not sure how to fix the problem. If anyone > > could give me some advice > > on what to try next, I would really appreciate it. > > If you could check (with someone, or otherwise) the MySql > server-in-question''s configuration to make sure Rails is using a low > enough verification_timeout, that would be another positive step, IMO. > Maybe it''ll turn up something, as it sounds like the ''timeout'' is > happening at approximately 20 minutes for you, vs. documentation I > have seen suggests that the default is 480 minutes. Good Luck. > > > Regards, > > Steven > > Sincerely, > -Parker > > > > Hi Parker, > > Thanks for your suggestions. Sorry for the late reply, it looks like > my over zealous spam > filter snagged your reply, I stumbled on your email while browsing the > mailing list > archives :-P > > I am--for now--administering the machine that the database is running > on and I will > try your suggestions. > > FWIW, I fired up good old webrick to see if the same problem would > occur. Sure enough, > after 20 minutes of inactivity the app would no longer respond through > the browser or by > doing (curl localhost:50042) while logged on to the machine. > > Regards, > Steven > > >Well it looks like I''ve resolved the problem. Hooray! My mysql server''s (mysqld) wait_timeout was set as follows (wait_timeout=28800). Which as Parker mentioned is 480 minutes or 8hrs! Here is a link to the mysql docs that has a description of what the wait_timeout var controls, as well as many other mysqld variables: http://dev.mysql.com/doc/refman/4.1/en/server-system-variables.html And for anyone who stumbles on an archive of this message, here is the mysql docs description of "wait_timeout" "The number of seconds the server waits for activity on a non-interactive connection before closing it. This timeout applies only to TCP/IP connections, not to connections made via Unix socket files, named pipes, or shared memory. On thread startup, the session |wait_timeout| value is initialized from the global |wait_timeout| value or from the global |interactive_timeout| value, depending on the type of client (as defined by the |CLIENT_INTERACTIVE| connect option to |mysql_real_connect()|). See also |interactive_timeout|." Interestingly enough, the mongrel FAQ says that all you need to do is to set (ActiveRecord::Base.verification_timeout = 14400) or a value lower than the server''s wait_timeout value. Before altering my server''s wait_timeout setting, I tried setting ActiveRecord::Base.verification_timeout to 60, 30 and 5. None of these settings made any difference as far as preventing Mongrel from locking up. Perhaps I added the setting in the wrong location in my rails environment? The FAQ doesn''t mention where this setting should be added, I just assumed it should be added to config/enviroment.rb if this is wrong, let me know. -Steven