Hi,
I have Apache (2.2) setup to proxy requests through to Mongrel (0.3.12.4)
for a Rails (1.1.2) application (on ruby 1.8.4). The setup is great, but
I''m experiencing some nightly failures. I thought I''d post
here before
digging in too deeply, but it looks like Rails is going into lala land and
Mongrel can''t recover...
Seems like everything gets all backed up in mongrel/rails.rb:64 at:
@guard.synchronize do
# Rails is not thread safe so must be run entirely within
synchronize
Dispatcher.dispatch(cgi,
ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS, response.body)
end
Prior to doing a "mongrel_rails restart" the mongrel log is normal and
there''s nothing unusual in my rails log. Once I do a
"mongrel_rails
restart," I get a bunch of the following dumped into the mongrel.log (I
assume one for each thread that''s chillin'' at
@guard.synchronize)...
Error calling Dispatcher.dispatch #<Mongrel::StopServer: Timed out
thread.>
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:472:in`lock''
/usr/local/lib/ruby/1.8/thread.rb:133:in `synchronize''
/usr/local/lib/ruby/gems/1.8/gems/mongrel-
0.3.12.4/lib/mongrel/rails.rb:64:in `process''
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:425:in`process_client''
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:424:in`process_client''
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:495:in`run''
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:494:in`run''
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:483:in`run''
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:774:in`run''
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:772:in`run''
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/bin/mongrel_rails:97:in`run''
/usr/local/lib/ruby/gems/1.8/gems/mongrel-
0.3.12.4/lib/mongrel/command.rb:163:in `run''
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/bin/mongrel_rails:194
/usr/local/bin/mongrel_rails:18
If anyone has any ideas please let me know... I promise I''ll jump up
and
down!
Thanks,
Jeremy
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20060428/a844270e/attachment.htm
Usually this happens if you have a Rails handler that gets to a point and then stops for some reason. Since Mongrel has a lock around the dispatcher, and you?ve got an action that isn?t exiting, then all subsequent requests will become jammed. Typical things that can do this are not closing files, forking or spawning processes and not waiting for them, connecting to remote servers that don?t answer right away, and even opening too many files. Look for any actions which could possibly cause something like this, especially if you?re using an external program to do some processing. I?ve added a ?B option and also created a small set of debugging handlers which dump lots of information for you. It slows your application down, so the best way to do it is this: 1. Run mongrel with ?B so it dumps thread, memory, etc. information to log/mongrel_debug/*.log files. 2. Write a Watir or Selenium test that goes through your application like a user. 3. Run the test and watch the mongrel_debug logs, paying attention to the ones reporting files and threads. 4. If you see an action that seems to have lots of objects created, lots of threads left open, or dangling files then that?s probably the one. 5. Go in and fix it. If you?re able to let me view your code, I?d like to work through this with you. It?s a very common problem but I?d like to go through helping someone fix it. Can you get together with me at some point? Zed On 4/28/06 10:48 AM, "Jeremy Hopple" <jeremy at jthopple.com> wrote:> Hi, > > I have Apache (2.2) setup to proxy requests through to Mongrel (0.3.12.4 > <http://0.3.12.4> ) for a Rails (1.1.2) application (on ruby 1.8.4). The > setup is great, but I''m experiencing some nightly failures. I thought I''d > post here before digging in too deeply, but it looks like Rails is going into > lala land and Mongrel can''t recover... > > Seems like everything gets all backed up in mongrel/rails.rb:64 at: > > @guard.synchronize do > # Rails is not thread safe so must be run entirely within > synchronize > Dispatcher.dispatch (cgi, > ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS, response.body) > end > > Prior to doing a "mongrel_rails restart" the mongrel log is normal and there''s > nothing unusual in my rails log. Once I do a "mongrel_rails restart," I get a > bunch of the following dumped into the mongrel.log (I assume one for each > thread that''s chillin'' at @guard.synchronize)... > > Error calling Dispatcher.dispatch #<Mongrel::StopServer: Timed out thread.> > /usr/local/lib/ruby/gems/1.8/gems/mongrel- 0.3.12.4/lib/mongrel.rb:472:in > <http://0.3.12.4/lib/mongrel.rb:472:in> `lock'' > /usr/local/lib/ruby/1.8/thread.rb:133:in `synchronize'' > /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel/rails.rb:64:in > <http://0.3.12.4/lib/mongrel/rails.rb:64:in> `process'' > /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:425:in > <http://0.3.12.4/lib/mongrel.rb:425:in> `process_client'' > /usr/local/lib/ruby/gems/1.8/gems/mongrel- 0.3.12.4/lib/mongrel.rb:424:in > <http://0.3.12.4/lib/mongrel.rb:424:in> `process_client'' > /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:495:in > <http://0.3.12.4/lib/mongrel.rb:495:in> `run'' > /usr/local/lib/ruby/gems/1.8/gems/mongrel- 0.3.12.4/lib/mongrel.rb:494:in > <http://0.3.12.4/lib/mongrel.rb:494:in> `run'' > /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:483:in > <http://0.3.12.4/lib/mongrel.rb:483:in> `run'' > /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/lib/mongrel.rb:774:in > <http://0.3.12.4/lib/mongrel.rb:774:in> `run'' > /usr/local/lib/ruby/gems/1.8/gems/mongrel- 0.3.12.4/lib/mongrel.rb:772:in > <http://0.3.12.4/lib/mongrel.rb:772:in> `run'' > /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/bin/mongrel_rails:97:in > <http://0.3.12.4/bin/mongrel_rails:97:in> `run'' > /usr/local/lib/ruby/gems/1.8/gems/mongrel- > 0.3.12.4/lib/mongrel/command.rb:163:in > <http://0.3.12.4/lib/mongrel/command.rb:163:in> `run'' > /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.12.4/bin/mongrel_rails:194 > <http://0.3.12.4/bin/mongrel_rails:194> > /usr/local/bin/mongrel_rails:18 > > If anyone has any ideas please let me know... I promise I''ll jump up and down! > > Thanks, > Jeremy > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20060429/dded364a/attachment.htm
Zed, Thanks for getting back to me and I really appreciate all the hard work you''ve put into Mongrel. A little bit more background. I''m in the process of setting up a new, dedicated FC4 machine to host 3 applications that I''m getting ready to release. None of them are live yet, but I have two of them in a "staging" environment on the new server for testing. Both of the applications I have running on the new server have exhibited the same behavior multiple times. Also, they both have done this upon the first request after a long period of inactivity (i.e. while we''re sleeping). Unless one of the members of my team is testing the application or it''s being load tested, the apps are idle. This problem has never happened we were actively using the application, or had the application under load... only when we send a request to it for the first time in the morning. The one thing I did realize this morning is that I had not installed the latest mysql bindings on the server, so I gem installed mysql (2.7). To tell you the truth, I''m not sure which bindings were installed prior to that (ruby and mysql were both installed from the fedora yum repositories). I''m wondering if something was dying in Rails when it tries to connect to mysql after a long period of inactivity... The simpler of the two applications that this is happening with is a very basic event system we put together to publish arts and entertainment happenings in our local community. If you''d like to check it out, we''re currently testing it at http://staging.the-j-list.com. This application is very simple. It uses a mysql (5.0.18) and sends a few emails with ActionMailer (using a remote smtp server). Prior to launching mongrel in debug mode, the only file it opens is the production.log. Assuming nothing above rings any bells, I''d be thrilled to get together with you and review the code. I live only about 2 blocks from my office, so I can be here pretty much whenever and we''re in Mountain Daylight Time. Also, I can get you access to the svn repository. Thanks again, and let me know what works for you. Jeremy On 4/29/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> > Usually this happens if you have a Rails handler that gets to a point and > then stops for some reason. Since Mongrel has a lock around the dispatcher, > and you''ve got an action that isn''t exiting, then all subsequent requests > will become jammed. > > Typical things that can do this are not closing files, forking or spawning > processes and not waiting for them, connecting to remote servers that don''t > answer right away, and even opening too many files. Look for any actions > which could possibly cause something like this, especially if you''re using > an external program to do some processing. > > I''ve added a ?B option and also created a small set of debugging handlers > which dump lots of information for you. It slows your application down, so > the best way to do it is this: > > > 1. Run mongrel with ?B so it dumps thread, memory, etc. information > to log/mongrel_debug/*.log files. > 2. Write a Watir or Selenium test that goes through your application > like a user. > 3. Run the test and watch the mongrel_debug logs, paying attention > to the ones reporting files and threads. > 4. If you see an action that seems to have lots of objects created, > lots of threads left open, or dangling files then that''s probably the one. > 5. Go in and fix it. > > > If you''re able to let me view your code, I''d like to work through this > with you. It''s a *very* common problem but I''d like to go through helping > someone fix it. Can you get together with me at some point? > > Zed > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20060429/90097d03/attachment-0001.htm
Hey Jeremy, Comments below... On 4/29/06 6:44 PM, "Jeremy Hopple" <jeremy at jthopple.com> wrote:> > Zed, > > Thanks for getting back to me and I really appreciate all the hard work you''ve > put into Mongrel. > > A little bit more background. I''m in the process of setting up a new, > dedicated FC4 machine to host 3 applications that I''m getting ready to > release. None of them are live yet, but I have two of them in a "staging" > environment on the new server for testing. > > Both of the applications I have running on the new server have exhibited the > same behavior multiple times. Also, they both have done this upon the first > request after a long period of inactivity (i.e. while we''re sleeping). Unless > one of the members of my team is testing the application or it''s being load > tested, the apps are idle. >Ah, I''m thinking this is mysql''s lame ass connection crap. Rails opens connections to the database and just leaves them there. Since TCP has a timeout, and mysql doesn''t have a heartbeat element to the protocol, your connections will time out and things go bad.> The one thing I did realize this morning is that I had not installed the > latest mysql bindings on the server, so I gem installed mysql ( 2.7). To tell > you the truth, I''m not sure which bindings were installed prior to that (ruby > and mysql were both installed from the fedora yum repositories). I''m wondering > if something was dying in Rails when it tries to connect to mysql after a long > period of inactivity... >Yes, I sent another e-mail off-list, but the default mysql.rb that rails has is total junk. Always install the mysql native compiled gem, this is also a performance boost.> The simpler of the two applications that this is happening with is a very > basic event system we put together to publish arts and entertainment > happenings in our local community. If you''d like to check it out, we''re > currently testing it at http://staging.the-j-list.com. This application is > very simple. It uses a mysql (5.0.18) and sends a few emails with > ActionMailer (using a remote smtp server). Prior to launching mongrel in > debug mode, the only file it opens is the production.log. >It seems like the mysql tcp connection is dying during the night. I''d be curious to know if this same problem happens under webrick. In theory it should since this would be a rails driver problem. Another thing is, why mysql? Maybe another database like postgresql doesn''t have these problems. I''ve been using postgresql in my apps and I don''t any of these problems.> Assuming nothing above rings any bells, I''d be thrilled to get together with > you and review the code. I live only about 2 blocks from my office, so I can > be here pretty much whenever and we''re in Mountain Daylight Time. Also, I can > get you access to the svn repository. >Yeah, I''ve got a mongrel version going out tonight (hopefully), so I''ll try to catch you when I can either tonight or tomorrow. Zed
> It seems like the mysql tcp connection is dying during the night. > I''d beinteractive_timeout in mysql defaults to 28800 seconds (480 minutes or 8 hours). So what you''re experiencing is that you probably stop touching the app at about 4pm, go home for the day and then mysql closes those connections at about midnight. If you set interactive_timeout = 86400 in a my.cnf file then what you''re experiencing would go away. But it should be clear why lower values are a good thing in a production application. You can read about interactive_timeout and wait_timeout at http:// dev.mysql.com/doc/refman/5.0/en/server-system-variables.html - J
Jason Hoffman wrote:>>It seems like the mysql tcp connection is dying during the night. >>I''d be >> >> > >interactive_timeout in mysql defaults to 28800 seconds (480 minutes >or 8 hours). > >Just raising the interactive_timeout doesn''t really solve the problem (while it may appear to.) I would suggest setting the verification_timeout for ActiveRecord. Set it lower than the interactive_timeout in MySQL. ActiveRecord::Base.verification_timeout = 14400 This will force Rails to reconnect to the database before MySQL can get a chance to timeout. _why
Thanks for the help! Everything has been stable for a couple nights now. I think a combination of getting the native ruby-mysql drivers compiled and setting the verification_timeout for ActiveRecord did the trick. On 5/1/06, why the lucky stiff <why at ruby-lang.org> wrote:> > Jason Hoffman wrote: > > >>It seems like the mysql tcp connection is dying during the night. > >>I''d be > >> > >> > > > >interactive_timeout in mysql defaults to 28800 seconds (480 minutes > >or 8 hours). > > > > > Just raising the interactive_timeout doesn''t really solve the problem > (while it may appear to.) I would suggest setting the > verification_timeout for ActiveRecord. Set it lower than the > interactive_timeout in MySQL. > > ActiveRecord::Base.verification_timeout = 14400 > > This will force Rails to reconnect to the database before MySQL can get > a chance to timeout. > > _why > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20060502/60b38160/attachment.htm