Hello, I''m running mongrel behind apache via proxypass. Every now and then, a spammer tries to proxy traffic through apache, and for some reason, apache forwards the request to mongrel, even though it''s not a site being hosted by me. At times this seems to cause mongrel to become unresponsive, and all requests to the site then fail with a proxy error until mongrel is restarted. I''m not sure of how to troubleshoot this issue. Help appreciated. Mike -- Michael P. Soulier <msoulier at digitaltorque.ca> "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070505/b7a71f69/attachment.bin
Hi Mike, My mongrels are dying too, but running behind Lighttpd. Are you seeing anything in the logs? I see absolutely nothing. Just the last request that went through... but it''s pretty old. At first, I thought it might be my version of Ruby or Rails. But now I''ve tried all sort of combinations, and rebuilt all the gems each time (based on advice on this forum). My solution for now is to run a few mongrels, and have Monit restart them when they become unresponsive. I have a few die each day. Zed, is there anything we can do to debug this further? Is there a good place we can put some logs in the Mongrel code? Or something we can try the next time they go down? Thanks, Pete On May 5, 2007, at 6:04 PM, Michael P. Soulier wrote:> Hello, > > I''m running mongrel behind apache via proxypass. Every now and > then, a spammer > tries to proxy traffic through apache, and for some reason, apache > forwards > the request to mongrel, even though it''s not a site being hosted by > me. > > At times this seems to cause mongrel to become unresponsive, and > all requests > to the site then fail with a proxy error until mongrel is restarted. > > I''m not sure of how to troubleshoot this issue. Help appreciated. > > Mike > -- > Michael P. Soulier <msoulier at digitaltorque.ca> > "Any intelligent fool can make things bigger and more complex... It > takes a touch of genius - and a lot of courage to move in the opposite > direction." --Albert Einstein > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
* Michael P. Soulier (msoulier at digitaltorque.ca) wrote:> At times this seems to cause mongrel to become unresponsive, and all > requests to the site then fail with a proxy error until mongrel is > restarted. > > I''m not sure of how to troubleshoot this issue. Help appreciated.Try using gdb to get a backtrace: % gdb `which ruby` <pid of mongrel> Then "bt" to get a list of the functions it''s calling. You may need to recompile Ruby with debugging symbols if it''s mostly "???". On FreeBSD this is as simple as "portupgrade -fm ''-DWITH_DEBUG'' ruby" http://eigenclass.org/hiki/ruby+live+process+introspection may be of use. Other options are ktrace, strace and truss (depending on what system you''re on) to get an idea of the syscalls it''s performing, but this may not be very useful depending on how it''s hung. -- Thomas ''Freaky'' Hurst http://hur.st/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070506/588fdd4b/attachment.bin
On 05/05/07 Pete DeLaurentis said:> Hi Mike, > > My mongrels are dying too, but running behind Lighttpd. Are you > seeing anything in the logs? I see absolutely nothing. Just the > last request that went through... but it''s pretty old.Nothing in the logs to indicate what the problem might be. Seems to happen when it processes a 404 error sometimes. ActionController::RoutingError (no route found to match "/2006/04/15/employee-of-the-century-dead-at-100/trackback/" with {:method=>:post}): /usr/local/ruby/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/routing.rb:1292:in `recognize_path'' /usr/local/ruby/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/routing.rb:1282:in `recognize'' /usr/local/ruby/lib/ruby/gems/1.8/gems/rails-1.2.2/lib/dispatcher.rb:40:in `dispatch'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/rails.rb:78:in `process'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/rails.rb:76:in `process'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:618:in `process_client'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:617:in `process_client'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:736:in `run'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:736:in `run'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel.rb:720:in `run'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/configurator.rb:271:in `run'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/configurator.rb:270:in `run'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/bin/mongrel_rails:127:in `run'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/mongrel/command.rb:211:in `run'' /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/bin/mongrel_rails:243 /usr/local/ruby/bin/mongrel_rails:18 rescue_action_in_public executed Rendering status404 Not Foundfile/var/www/papproach/config/../public/404.htmllayoutfalse within layouts//application Rendering /var/www/papproach/config/../public/404.html (404 Not Found)> My solution for now is to run a few mongrels, and have Monit restart > them when they become unresponsive. I have a few die each day.I have one instance supervised with runit, but supervision doesn''t help since the mongrel doesn''t crash, it just hangs. I now have a 10 minute cron that checks on it and restarts it if it''s unresponsive, and it''s doing so a few times an hour now, which is disturbing.> Zed, is there anything we can do to debug this further? Is there a > good place we can put some logs in the Mongrel code? Or something we > can try the next time they go down?It''s happening a lot to me right now. No --debug option to mongrel? Mike -- Michael P. Soulier <msoulier at digitaltorque.ca> "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070505/453d1cdb/attachment.bin
Hi Mike, You''re definitely hitting this more often then I am (about once per day). With monit, you can do an HTTP request and test the response, which is how I determined if it''s hanged. Sounds pretty similar to what you''re doing with cron. Which OS are you running + version of ruby? I''m on OpenSUSE 10.3, Ruby 1.8.5, Rails 1.2, and Mongrel 1.0.1. I also run memcached, lighttpd, and pound. I''m running 6 mongrels using mongrel_cluster (the latest version). Thanks, Pete On May 5, 2007, at 7:05 PM, Michael P. Soulier wrote:> On 05/05/07 Pete DeLaurentis said: > >> Hi Mike, >> >> My mongrels are dying too, but running behind Lighttpd. Are you >> seeing anything in the logs? I see absolutely nothing. Just the >> last request that went through... but it''s pretty old. > > Nothing in the logs to indicate what the problem might be. Seems to > happen > when it processes a 404 error sometimes. > > ActionController::RoutingError (no route found to match > "/2006/04/15/employee-of-the-century-dead-at-100/trackback/" with > {:method=>:post}): > /usr/local/ruby/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/ > action_controller/routing.rb:1292:in > `recognize_path'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/ > action_controller/routing.rb:1282:in > `recognize'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/rails-1.2.2/lib/ > dispatcher.rb:40:in > `dispatch'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/ > mongrel/rails.rb:78:in > `process'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/ > mongrel/rails.rb:76:in > `process'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/ > mongrel.rb:618:in > `process_client'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/ > mongrel.rb:617:in > `process_client'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/ > mongrel.rb:736:in > `run'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/ > mongrel.rb:736:in > `run'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/ > mongrel.rb:720:in > `run'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/ > mongrel/configurator.rb:271:in > `run'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/ > mongrel/configurator.rb:270:in > `run'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/bin/ > mongrel_rails:127:in > `run'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/lib/ > mongrel/command.rb:211:in > `run'' > /usr/local/ruby/lib/ruby/gems/1.8/gems/mongrel-1.0.1/bin/ > mongrel_rails:243 > /usr/local/ruby/bin/mongrel_rails:18 > > > rescue_action_in_public executed > Rendering status404 Not > Foundfile/var/www/papproach/config/../public/404.htmllayoutfalse > within > layouts//application > Rendering /var/www/papproach/config/../public/404.html (404 Not Found) > >> My solution for now is to run a few mongrels, and have Monit restart >> them when they become unresponsive. I have a few die each day. > > I have one instance supervised with runit, but supervision doesn''t > help since > the mongrel doesn''t crash, it just hangs. I now have a 10 minute > cron that > checks on it and restarts it if it''s unresponsive, and it''s doing > so a few > times an hour now, which is disturbing. > >> Zed, is there anything we can do to debug this further? Is there a >> good place we can put some logs in the Mongrel code? Or something we >> can try the next time they go down? > > It''s happening a lot to me right now. No --debug option to mongrel? > > Mike > -- > Michael P. Soulier <msoulier at digitaltorque.ca> > "Any intelligent fool can make things bigger and more complex... It > takes a touch of genius - and a lot of courage to move in the opposite > direction." --Albert Einstein > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On 06/05/07 Thomas Hurst said:> Try using gdb to get a backtrace: > > % gdb `which ruby` <pid of mongrel>Unfortunately it''s on a VPS where I don''t get gdb. I don''t have full control over the box.> Then "bt" to get a list of the functions it''s calling. You may need to > recompile Ruby with debugging symbols if it''s mostly "???". On FreeBSD > this is as simple as "portupgrade -fm ''-DWITH_DEBUG'' ruby" > > http://eigenclass.org/hiki/ruby+live+process+introspection may be of > use. > > Other options are ktrace, strace and truss (depending on what system > you''re on) to get an idea of the syscalls it''s performing, but this may > not be very useful depending on how it''s hung.Mike -- Michael P. Soulier <msoulier at digitaltorque.ca> "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070505/6604fa81/attachment.bin
On 05/05/07 Pete DeLaurentis said:> Hi Mike, > > You''re definitely hitting this more often then I am (about once per > day). > > With monit, you can do an HTTP request and test the response, which > is how I determined if it''s hanged. Sounds pretty similar to what > you''re doing with cron. > > Which OS are you running + version of ruby? I''m on OpenSUSE 10.3, > Ruby 1.8.5, Rails 1.2, and Mongrel 1.0.1. I also run memcached, > lighttpd, and pound. I''m running 6 mongrels using mongrel_cluster > (the latest version).I''m running a Linux VPS, which I think is RedHat based. ruby 1.8.4 and mongrel 1.0.1, with Rails 1.2.2. Mike -- Michael P. Soulier <msoulier at digitaltorque.ca> "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070505/09201a6f/attachment.bin
Try upgrading to Ruby 1.8.6, my mongrels were hanging too until I upgraded my ruby, gem and rails to the latest versions and it''s all working better now. On 5/5/07, Michael P. Soulier <msoulier at digitaltorque.ca> wrote:> > On 05/05/07 Pete DeLaurentis said: > > > Hi Mike, > > > > You''re definitely hitting this more often then I am (about once per > > day). > > > > With monit, you can do an HTTP request and test the response, which > > is how I determined if it''s hanged. Sounds pretty similar to what > > you''re doing with cron. > > > > Which OS are you running + version of ruby? I''m on OpenSUSE 10.3, > > Ruby 1.8.5, Rails 1.2, and Mongrel 1.0.1. I also run memcached, > > lighttpd, and pound. I''m running 6 mongrels using mongrel_cluster > > (the latest version). > > I''m running a Linux VPS, which I think is RedHat based. ruby 1.8.4 and > mongrel 1.0.1, with Rails 1.2.2. > > Mike > -- > Michael P. Soulier <msoulier at digitaltorque.ca> > "Any intelligent fool can make things bigger and more complex... It > takes a touch of genius - and a lot of courage to move in the opposite > direction." --Albert Einstein > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070506/c85bff04/attachment.html
I wouldn''t upgrade to Ruby 1.8.6 unless my app was written for Rails 1.2 as Ruby 1.8.6 requires Rails 1.2.3/ On 5/7/07, Scott Hodson <shodson at gmail.com> wrote:> > Try upgrading to Ruby 1.8.6, my mongrels were hanging too until I upgraded > my ruby, gem and rails to the latest versions and it''s all working better > now. > > On 5/5/07, Michael P. Soulier <msoulier at digitaltorque.ca> wrote: > > > > On 05/05/07 Pete DeLaurentis said: > > > > > Hi Mike, > > > > > > You''re definitely hitting this more often then I am (about once per > > > day). > > > > > > With monit, you can do an HTTP request and test the response, which > > > is how I determined if it''s hanged. Sounds pretty similar to what > > > you''re doing with cron. > > > > > > Which OS are you running + version of ruby? I''m on OpenSUSE 10.3, > > > Ruby 1.8.5 , Rails 1.2, and Mongrel 1.0.1. I also run memcached, > > > lighttpd, and pound. I''m running 6 mongrels using mongrel_cluster > > > (the latest version). > > > > I''m running a Linux VPS, which I think is RedHat based. ruby 1.8.4 and > > mongrel 1.0.1, with Rails 1.2.2. > > > > Mike > > -- > > Michael P. Soulier <msoulier at digitaltorque.ca> > > "Any intelligent fool can make things bigger and more complex... It > > takes a touch of genius - and a lot of courage to move in the opposite > > direction." --Albert Einstein > > > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070507/d3337094/attachment.html
That''s interesting Scott. I was running Ruby 1.8.4, and first noticed the problem when I upgraded to 1.8.6. I did some other things at that same time though... I also added memcached (which I connect to in environment.rb). I recently tried having monit call a simple test controller / action every 2 minutes on each mongrel. The controller doesn''t access the DB or memcache... it just renders "hello world" to the HTML. Shouldn''t help... but it does. So far so good, the mongrels have stayed responsive for the last couple days. Based on the fact that this fixed things, I''m thinking it''s some kind of timeout issue within mongrel. -Pete On May 7, 2007, at 7:07 AM, Michael D''Auria wrote:> I wouldn''t upgrade to Ruby 1.8.6 unless my app was written for > Rails 1.2 as Ruby 1.8.6 requires Rails 1.2.3/ > > On 5/7/07, Scott Hodson <shodson at gmail.com> wrote: > Try upgrading to Ruby 1.8.6, my mongrels were hanging too until I > upgraded my ruby, gem and rails to the latest versions and it''s all > working better now. > > On 5/5/07, Michael P. Soulier <msoulier at digitaltorque.ca > wrote: > On 05/05/07 Pete DeLaurentis said: > > > Hi Mike, > > > > You''re definitely hitting this more often then I am (about once per > > day). > > > > With monit, you can do an HTTP request and test the response, which > > is how I determined if it''s hanged. Sounds pretty similar to what > > you''re doing with cron. > > > > Which OS are you running + version of ruby? I''m on OpenSUSE 10.3, > > Ruby 1.8.5 , Rails 1.2, and Mongrel 1.0.1. I also run memcached, > > lighttpd, and pound. I''m running 6 mongrels using mongrel_cluster > > (the latest version). > > I''m running a Linux VPS, which I think is RedHat based. ruby 1.8.4 and > mongrel 1.0.1, with Rails 1.2.2. > > Mike > -- > Michael P. Soulier < msoulier at digitaltorque.ca> > "Any intelligent fool can make things bigger and more complex... It > takes a touch of genius - and a lot of courage to move in the opposite > direction." --Albert Einstein > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070507/ca516622/attachment-0001.html
On 5/7/07, Pete DeLaurentis <pete at nextengine.com> wrote:> That''s interesting Scott. I was running Ruby 1.8.4, and first noticed the > problem when I upgraded to 1.8.6. I did some other things at that same time > though... I also added memcached (which I connect to in environment.rb).Just another useless data point, but I never have had a problem with Mongrel on Ruby 1.8.5. Kirk Haines
We''ve had this here too, to the point where it was becoming a bit of a crisis. I''ve seen several suggested remedies: 1. make sure your log rotation is done outside mongrel, not using ruby''s log rotation features 2. make sure your rails app doesn''t ever try to write to stdout or stderr 3. avoid pstore-based sessions 4. disable http keepalive on the apache frontend We tried all of these and it looks like #4 might have finally done the trick. -- miles "If money played an instrument it would be the saxophone." - Achewood
Thanks for the tips Miles. One of them really hit home: I love writing to stdout, and it''s all over the place in my app. It''s great because you can run your scripts / apps as foreground and get a live trace, or daemonized and it goes to the log file. Your warning on this one sounds like the voice of experience, and I believe it. But I''d like to understand the details. Any ideas why this is a dangerous thing to do? Thanks, Pete BTW. I don''t know if lighty has a keepalive feature, but I''ll look into this. Sounds like it was the main issue for you, and it could be for me too. On May 7, 2007, at 8:26 AM, Miles Egan wrote:> We''ve had this here too, to the point where it was becoming a bit of > a crisis. I''ve seen several suggested remedies: > > 1. make sure your log rotation is done outside mongrel, not using > ruby''s log rotation features > 2. make sure your rails app doesn''t ever try to write to stdout or > stderr > 3. avoid pstore-based sessions > 4. disable http keepalive on the apache frontend > > We tried all of these and it looks like #4 might have finally done > the trick. > > -- > miles > > "If money played an instrument it would be the saxophone." - Achewood > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On May 7, 2007, at 9:06 AM, Pete DeLaurentis wrote:> Thanks for the tips Miles. One of them really hit home: > > I love writing to stdout, and it''s all over the place in my app. > It''s great because you can run your scripts / apps as foreground and > get a live trace, or daemonized and it goes to the log file. Your > warning on this one sounds like the voice of experience, and I > believe it. But I''d like to understand the details. > > Any ideas why this is a dangerous thing to do?I gather from browsing this list that it interferes with Ruby''s threads and non-blocking i/o but I''m sure somebody on the list can give you a better explanation. I''d recommend using the logging features and just keeping an active tail -f on the log. -- miles "If money played an instrument it would be the saxophone." - Achewood
On 5/7/07, Miles Egan <miles at pixar.com> wrote:> > I love writing to stdout, and it''s all over the place in my app. > > I gather from browsing this list that it interferes with Ruby''s > threads and non-blocking i/o but I''m sure somebody on the list can > give you a better explanation. I''d recommend using the logging > features and just keeping an active tail -f on the log.It''s a shameless plug, but you might look at http://analogger.swiftcore.org to help with logging. It should be simple to use it from Rails, though nobody has given me any specific feedback on that, yet. Kirk Haines
Just out of curiosity, why does mongrel_rails by default put its .pid files in the log directory rather than in tmp or tmp/pids? Putting them in the log directory seems to me like a strange decision, since they''re not logs -- they''re temporary state files. This e-mail message is privileged, confidential and subject to copyright. Any unauthorized use or disclosure is prohibited. Le contenu du pr''esent courriel est privil''egi''e, confidentiel et soumis `a des droits d''auteur. Il est interdit de l''utiliser ou de le divulguer sans autorisation.
I have thought this as well -- but it''s not just Mongrel that does this. Ferret''s DRb server also puts the .pid file into the logs directory. I''d prefer a dedicated location myself. Thanks, Doug On 5/7/07, Matt Zukowski <mzukowski at urbacon.net> wrote:> Just out of curiosity, why does mongrel_rails by default put its .pid > files in the log directory rather than in tmp or tmp/pids? Putting them > in the log directory seems to me like a strange decision, since they''re > not logs -- they''re temporary state files. > > > This e-mail message is privileged, confidential and subject to copyright. Any unauthorized use or disclosure is prohibited. > Le contenu du pr''esent courriel est privil''egi''e, confidentiel et soumis `a des droits d''auteur. Il est interdit de l''utiliser ou de le divulguer sans autorisation. > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
i believe the answer is the decision was made pre-rails 1.2 which added tmp/* to the rails directory tree. plus maybe other frameworks dont have tmp/pids and log was a safe default. don On May 7, 2007, at 11:39 AM, Doug Smith wrote:> I have thought this as well -- but it''s not just Mongrel that does > this. Ferret''s DRb server also puts the .pid file into the logs > directory. I''d prefer a dedicated location myself. > > Thanks, > > Doug > > On 5/7/07, Matt Zukowski <mzukowski at urbacon.net> wrote: >> Just out of curiosity, why does mongrel_rails by default put its .pid >> files in the log directory rather than in tmp or tmp/pids? Putting >> them >> in the log directory seems to me like a strange decision, since >> they''re >> not logs -- they''re temporary state files. >> >> >> This e-mail message is privileged, confidential and subject to >> copyright. Any unauthorized use or disclosure is prohibited. >> Le contenu du pr''esent courriel est privil''egi''e, confidentiel et >> soumis `a des droits d''auteur. Il est interdit de l''utiliser ou de >> le divulguer sans autorisation. >> >> >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
So just change your config files. Easy peasy. Best practice, if using mongrel_cluster, is to throw it into a directory like what you''re suggesting. In the announcement thread for mongrel_cluster 1.0.1.1 it''s recommended to config for using the /var/run directory. But you can choose what you want. FYI matte - matte at ruckuswireless.com Doug Smith wrote:> I have thought this as well -- but it''s not just Mongrel that does > this. Ferret''s DRb server also puts the .pid file into the logs > directory. I''d prefer a dedicated location myself. > > Thanks, > > Doug > > On 5/7/07, Matt Zukowski <mzukowski at urbacon.net> wrote: > >> Just out of curiosity, why does mongrel_rails by default put its .pid >> files in the log directory rather than in tmp or tmp/pids? Putting them >> in the log directory seems to me like a strange decision, since they''re >> not logs -- they''re temporary state files. >> >>
I am also having an issue with mongrel becoming unresponsive.
Here is it''s environment:
Red Hat Enterprise Linux ES release 4
ruby 1.8.5 (2007-03-13 patchlevel 35) [x86_64-linux]
Mongrel Web Server 1.0.1
In front of these mongrels, is Apache and the haproxy load balancer
I am running several different ROR sites on the same server.
after a few hours the rails application will become unresponsive.
My log file sings this tune:
Server overloaded with 5 processors (5 max). Dropping connection.
Mon May 07 11:53:04 -0500 2007: Reaping 4 threads for slow workers
because of ''max processors''
Thread #<Thread:0x2a98447ab0 sleep> is too old, killing.
Mon May 07 11:53:04 -0500 2007: ERROR: Mongrel timed out this thread:
max processorsThread #<Thread:0x2a98451740 sleep> is too old, killing.
Mon May 07 11:53:04 -0500 2007: ERROR: Mongrel timed out this thread:
max processorsThread #<Thread:0x2a98459990 sleep> is too old, killing.
Mon May 07 11:53:04 -0500 2007: ERROR: Mongrel timed out this thread:
max processorsThread #<Thread:0x2a984d9168 sleep> is too old, killing.
Server overloaded with 5 processors (5 max). Dropping connection.
Mon May 07 11:53:06 -0500 2007: Reaping 5 threads for slow workers
because of ''max processors''
Mon May 07 13:29:29 -0500 2007: Reaping 3 threads for slow workers
because of ''shutdown''
Thread #<Thread:0x2a98472468 sleep> is too old, killing.
Mon May 07 13:29:29 -0500 2007: Error calling
Dispatcher.dispatch#<Mongrel::TimeoutError: Mongrel timed out this
thread: shutdown>Thread
#<Thread:0x2a98597b68 sleep> is too old, killing.
A restart of the mongrel, fixes it, but it doesn''t last.
I have been trying to use the gdb to to gain some insight as the the root
cause.
Using the examples I found on the web,, the following was suggested
(gdb) set $ary = (int)backtrace(-1)
(gdb) set $count = *($ary+8)
(gdb) set $index = 0
(gdb) while $index < $count
> x/1s *((int)rb_ary_entry($ary, $index)+12)
> set $index = $index + 1
>end
However, I get an error on the second line
"(gdb) set $count = *($ary+8)
Cannot access memory at address 0xffffffff985779d8"
Has anyone seen this before?
Cheers,
Nathan C.
--
Together we can solve any issue.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070507/87c2ab8b/attachment.html
Nathan- This line is very telling: Server overloaded with 5 processors (5 max). Dropping connection Mongrel has a numprocs setting that defaults to 900 i think. It looks like somehow it is being set to 5 in your mongrels so they start timing out threads when there are more then 5 connections in mongrel being processed. I''d say up your numb procs to 50 or so and if you are using haproxy make it limit connections to mongrel to one at atime. That way the requests queue in haproxy and only go to mongrels that aren''t busy. Cheers- -Ezra On May 7, 2007, at 3:55 PM, Nathan Clark wrote:> I am also having an issue with mongrel becoming unresponsive. > Here is it''s environment: > Red Hat Enterprise Linux ES release 4 > ruby 1.8.5 (2007-03-13 patchlevel 35) [x86_64-linux] > Mongrel Web Server 1.0.1 > > In front of these mongrels, is Apache and the haproxy load balancer > > I am running several different ROR sites on the same server. > after a few hours the rails application will become unresponsive. > > My log file sings this tune: > > Server overloaded with 5 processors (5 max). Dropping connection. > Mon May 07 11:53:04 -0500 2007: Reaping 4 threads for slow > workers because of ''max processors'' > Thread #<Thread:0x2a98447ab0 sleep> is too old, killing. > Mon May 07 11:53:04 -0500 2007: ERROR: Mongrel timed out this > thread: max processorsThread #<Thread:0x2a98451740 sleep> is too > old, killing. > Mon May 07 11:53:04 -0500 2007: ERROR: Mongrel timed out this > thread: max processorsThread #<Thread:0x2a98459990 sleep> is too > old, killing. > Mon May 07 11:53:04 -0500 2007: ERROR: Mongrel timed out this > thread: max processorsThread #<Thread:0x2a984d9168 sleep> is too > old, killing. > > Server overloaded with 5 processors (5 max). Dropping connection. > Mon May 07 11:53:06 -0500 2007: Reaping 5 threads for slow > workers because of ''max processors'' > > Mon May 07 13:29:29 -0500 2007: Reaping 3 threads for slow > workers because of ''shutdown'' > Thread #<Thread:0x2a98472468 sleep> is too old, killing. > Mon May 07 13:29:29 -0500 2007: Error calling > Dispatcher.dispatch #<Mongrel::TimeoutError: Mongrel timed out this > thread: shutdown>Thread #<Thread:0x2a98597b68 sleep> is too old, > killing. > > A restart of the mongrel, fixes it, but it doesn''t last. > I have been trying to use the gdb to to gain some insight as the > the root cause. > Using the examples I found on the web,, the following was suggested > > (gdb) set $ary = (int)backtrace(-1) > (gdb) set $count = *($ary+8) > (gdb) set $index = 0 > (gdb) while $index < $count > > x/1s *((int)rb_ary_entry($ary, $index)+12) > > set $index = $index + 1 > >end > > However, I get an error on the second line > "(gdb) set $count = *($ary+8) > Cannot access memory at address 0xffffffff985779d8" > > Has anyone seen this before? > > Cheers, > Nathan C. > > > -- > Together we can solve any issue. > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users-- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273)
One more reason -- in capistrano deployed setups, the log directory is shared among all releases, but the rails tmp directory is not. The default mongrel restart recipes will complain that they can''t find pid files because the xxx/current/tmp will have been moved to a freshly checked-out version of your which doesn''t have the old tmp directory from your previous deploy. You''d probably have to mess with your recipes if you''re using capistrano, or make the tmp directory shared as well... On 5/8/07, Matte Edens <matte at ruckuswireless.com> wrote:> So just change your config files. Easy peasy. > > Best practice, if using mongrel_cluster, is to throw it into a directory > like what you''re suggesting. In the announcement thread for > mongrel_cluster 1.0.1.1 it''s recommended to config for using the > /var/run directory. But you can choose what you want. > > FYI > > matte - matte at ruckuswireless.com > > Doug Smith wrote: > > I have thought this as well -- but it''s not just Mongrel that does > > this. Ferret''s DRb server also puts the .pid file into the logs > > directory. I''d prefer a dedicated location myself. > > > > Thanks, > > > > Doug > > > > On 5/7/07, Matt Zukowski <mzukowski at urbacon.net> wrote: > > > >> Just out of curiosity, why does mongrel_rails by default put its .pid > >> files in the log directory rather than in tmp or tmp/pids? Putting them > >> in the log directory seems to me like a strange decision, since they''re > >> not logs -- they''re temporary state files. > >> > >> > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
On 5/7/07, Pete DeLaurentis <pete at nextengine.com> wrote:> Thanks for the tips Miles. One of them really hit home: > > I love writing to stdout, and it''s all over the place in my app. > It''s great because you can run your scripts / apps as foreground and > get a live trace, or daemonized and it goes to the log file. Your > warning on this one sounds like the voice of experience, and I > believe it. But I''d like to understand the details. > > Any ideas why this is a dangerous thing to do? >Pete, puts or anything that prints to stdout/stderr interfere with CGI (Servlet dispatcher used by Rails). Also, puts not being threadsafe (you cannot garantee "this line #1" will get print fully form one thread and other text line from another will without interfere eachother). use RAILS_DEFAULT_LOGGER (or just ''logger'' in your controllers, views or models). logger.debug (for debug level) is very helpful during development. -- Luis Lavena Multimedia systems - Leaders are made, they are not born. They are made by hard effort, which is the price which all of us must pay to achieve any goal that is worthwhile. Vince Lombardi
I beileve that pid files are release specific, as are the running mongrels. the problem you describe is when the capistrano recipe does an svn update, shutdown, startup. the svn update replaces the directory tree under the running processes. i imagine the reasoning was to minimize downtime if the svn up is a time-consuming process. a better order is to shutdown, svn up, startup. there is an association between the running mongrels and the tmp/pids of the release directory in which they started. let the running processes shutdown in their own release directory, then replace the release directory. don On May 7, 2007, at 11:55 PM, Eden Li wrote:> One more reason -- in capistrano deployed setups, the log directory is > shared among all releases, but the rails tmp directory is not. The > default mongrel restart recipes will complain that they can''t find pid > files because the xxx/current/tmp will have been moved to a freshly > checked-out version of your which doesn''t have the old tmp directory > from your previous deploy. > > You''d probably have to mess with your recipes if you''re using > capistrano, or make the tmp directory shared as well... > > On 5/8/07, Matte Edens <matte at ruckuswireless.com> wrote: >> So just change your config files. Easy peasy. >> >> Best practice, if using mongrel_cluster, is to throw it into a >> directory >> like what you''re suggesting. In the announcement thread for >> mongrel_cluster 1.0.1.1 it''s recommended to config for using the >> /var/run directory. But you can choose what you want. >> >> FYI >> >> matte - matte at ruckuswireless.com >> >> Doug Smith wrote: >>> I have thought this as well -- but it''s not just Mongrel that does >>> this. Ferret''s DRb server also puts the .pid file into the logs >>> directory. I''d prefer a dedicated location myself. >>> >>> Thanks, >>> >>> Doug >>> >>> On 5/7/07, Matt Zukowski <mzukowski at urbacon.net> wrote: >>> >>>> Just out of curiosity, why does mongrel_rails by default put >>>> its .pid >>>> files in the log directory rather than in tmp or tmp/pids? >>>> Putting them >>>> in the log directory seems to me like a strange decision, since >>>> they''re >>>> not logs -- they''re temporary state files. >>>> >>>> >> >> >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
Thanks for the explanation Luis. This makes good sense. It seems like Mongrel is intercepting the stdout + sterr (puts it all into mongrel.log), but it''s not good to work with something that isn''t threadsafe. Makes me wonder if the rails logs are safe if you write to them from different mongrels in a cluster. Any experience with this? Thanks, Pete> Pete, > > puts or anything that prints to stdout/stderr interfere with CGI > (Servlet dispatcher used by Rails). > > Also, puts not being threadsafe (you cannot garantee "this line #1" > will get print fully form one thread and other text line from another > will without interfere eachother). > > > use RAILS_DEFAULT_LOGGER (or just ''logger'' in your controllers, views > or models). > > logger.debug (for debug level) is very helpful during development. > > > -- > Luis Lavena > Multimedia systems > - > Leaders are made, they are not born. They are made by hard effort, > which is the price which all of us must pay to achieve any goal that > is worthwhile. > Vince Lombardi > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On 5/8/07, Pete DeLaurentis <pete at nextengine.com> wrote:> Thanks for the explanation Luis. This makes good sense. It seems > like Mongrel is intercepting the stdout + sterr (puts it all into > mongrel.log), but it''s not good to work with something that isn''t > threadsafe. > > Makes me wonder if the rails logs are safe if you write to them from > different mongrels in a cluster. Any experience with this? >Are safe only under the following conditions: 2 Mongrels in the cluster don''t get hit at the same time, with the same request that takes the same time to parse and process... That means: no :-) (Even under labs conditions that couldn''t be granted). Page sharing of files opened between process is something difficult to achieve, I''ll suggest 1 log file per process on each port, which will reduce chances of logfiles get corrupted.> Thanks, > Pete >You''re welcome. Good week, -- Luis Lavena Multimedia systems - Leaders are made, they are not born. They are made by hard effort, which is the price which all of us must pay to achieve any goal that is worthwhile. Vince Lombardi
On 5/8/07, Pete DeLaurentis <pete at nextengine.com> wrote:> Thanks for the explanation Luis. This makes good sense. It seems > like Mongrel is intercepting the stdout + sterr (puts it all into > mongrel.log), but it''s not good to work with something that isn''t > threadsafe.This probably begs some elaboration on just what puts() does. It writes your string in one write operation, and does the newline in a second operation, which opens the possibility that Ruby can make a context switch between them. If the newline is already in the string, though, it is just a single write operation. You can also use write() instead of puts() to explicitly make sure that everything is done in a single write call. Take a look at the code in io.c for the details.> Makes me wonder if the rails logs are safe if you write to them from > different mongrels in a cluster. Any experience with this?Even if it can be made "safe", contention from multiple processes trying to write to one file can be a non-trivial performance hit. This was one of my motivations for writing an async logger. I want all of my app logs in the same place, even if I have multiple backend processes, without worrying about resource contention. It''s that, or I have N log files, one for each backend process for my app. Kirk Haines
Thanks for the info Kirk, and for letting us know about your logger plugin. It sounds like exactly what I need. I think I''ll override Kernel.puts to use your plugin. -Pete On May 8, 2007, at 8:47 AM, Kirk Haines wrote:> On 5/8/07, Pete DeLaurentis <pete at nextengine.com> wrote: >> Thanks for the explanation Luis. This makes good sense. It seems >> like Mongrel is intercepting the stdout + sterr (puts it all into >> mongrel.log), but it''s not good to work with something that isn''t >> threadsafe. > > This probably begs some elaboration on just what puts() does. It > writes your string in one write operation, and does the newline in a > second operation, which opens the possibility that Ruby can make a > context switch between them. If the newline is already in the string, > though, it is just a single write operation. You can also use write() > instead of puts() to explicitly make sure that everything is done in a > single write call. > > Take a look at the code in io.c for the details. > >> Makes me wonder if the rails logs are safe if you write to them from >> different mongrels in a cluster. Any experience with this? > > Even if it can be made "safe", contention from multiple processes > trying to write to one file can be a non-trivial performance hit. > This was one of my motivations for writing an async logger. I want > all of my app logs in the same place, even if I have multiple backend > processes, without worrying about resource contention. It''s that, or > I have N log files, one for each backend process for my app. > > > Kirk Haines > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On 5/8/07, Pete DeLaurentis <pete at nextengine.com> wrote:> Thanks for the info Kirk, and for letting us know about your logger > plugin. It sounds like exactly what I need. > > I think I''ll override Kernel.puts to use your plugin.Let me know if there''s anything I can bundle in the next release to make Rails or Mongrel logging integration simpler. Kirk Haines