Hi guys, I''ve been running mongrel for a while now with Ruby 1.8.4, and last week upgraded to 1.8.6. Since upgrading, each morning when I wake up there''s a big problem: 1. Accessing the site returns a "500 Internal Server Error" 2. All the mongrel_rails processes are still running, but none of them are active (when I run top) 3. Lighttpd and pound are still running nicely 4. There''s plenty of free RAM 5. There''s plenty of free HD 6. When I restart mongrel, it all starts working again I haven''t seen anything like this before. Does this sound familiar to anyone? Thanks for the help, Pete
Standard questions come to mind: - What do your logs say? - Are you running anything semi-dangerous like rmagick or older versions of acts_as_ferret? - Are you using Ruby based log rotation? BTW, maybe we could put up solutions to common issues into a page on mongrel.rubyforge.org? I don''t see one on http://mongrel.rubyforge.org/docs/index.html... On 4/4/07, Pete DeLaurentis <pete at nextengine.com> wrote:> Hi guys, > > I''ve been running mongrel for a while now with Ruby 1.8.4, and last > week upgraded to 1.8.6. > > Since upgrading, each morning when I wake up there''s a big problem: > > 1. Accessing the site returns a "500 Internal Server Error" > 2. All the mongrel_rails processes are still running, but none of > them are active (when I run top) > 3. Lighttpd and pound are still running nicely > 4. There''s plenty of free RAM > 5. There''s plenty of free HD > 6. When I restart mongrel, it all starts working again > > I haven''t seen anything like this before. Does this sound familiar > to anyone? > > Thanks for the help, > Pete > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
Hi Eden, I agree: a common issues page is a great idea. Unfortunately, this issue doesn''t seem to be one of the usual suspects. 1. The logs just have normal chatter for my app ... nothing unusual 2. No rmagick or ferret, but I believe it is related to Ruby 1.8.6, since that''s when ti started dying. 3. I''m not using log rotation, but the files aren''t that big after a day.. and there''s plenty of free space on disk Here''s a question: Is anyone running ruby1.8.6 succesfully with mongrel? Thanks, Pete On Apr 4, 2007, at 6:01 AM, Eden Li wrote:> Standard questions come to mind: > - What do your logs say? > - Are you running anything semi-dangerous like rmagick or older > versions of acts_as_ferret? > - Are you using Ruby based log rotation? > > BTW, maybe we could put up solutions to common issues into a page on > mongrel.rubyforge.org? I don''t see one on > http://mongrel.rubyforge.org/docs/index.html... > > On 4/4/07, Pete DeLaurentis <pete at nextengine.com> wrote: >> Hi guys, >> >> I''ve been running mongrel for a while now with Ruby 1.8.4, and last >> week upgraded to 1.8.6. >> >> Since upgrading, each morning when I wake up there''s a big problem: >> >> 1. Accessing the site returns a "500 Internal Server Error" >> 2. All the mongrel_rails processes are still running, but none of >> them are active (when I run top) >> 3. Lighttpd and pound are still running nicely >> 4. There''s plenty of free RAM >> 5. There''s plenty of free HD >> 6. When I restart mongrel, it all starts working again >> >> I haven''t seen anything like this before. Does this sound familiar >> to anyone? >> >> Thanks for the help, >> Pete >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
Pete DeLaurentis wrote:> > Here''s a question: Is anyone running ruby1.8.6 succesfully with mongrel? >Yes. In fact, my problems with mongrels dying went away when I went to ruby 1.8.6. Of course, I changed a number of other things at the same time so I can''t say conclusively that the ruby upgrade was the solution. But I haven''t had any mongrels die since I deployed my new setup last 5 days ago, and prior to that I was having a mongrel die (as in crash not just stop responding) every few hours or so. -- Jack Christensen jackc at hylesanderson.edu
Which version of rails are you using? Version prior to 1.2.3 have a few issues... Also, what version of mongrel are you running? On 4/4/07, Pete DeLaurentis <pete at nextengine.com> wrote:> Hi Eden, > > I agree: a common issues page is a great idea. Unfortunately, this > issue doesn''t seem to be one of the usual suspects. > > 1. The logs just have normal chatter for my app ... nothing unusual > 2. No rmagick or ferret, but I believe it is related to Ruby 1.8.6, > since that''s when ti started dying. > 3. I''m not using log rotation, but the files aren''t that big after a > day.. and there''s plenty of free space on disk > > Here''s a question: Is anyone running ruby1.8.6 succesfully with mongrel? > > Thanks, > Pete > > > > On Apr 4, 2007, at 6:01 AM, Eden Li wrote: > > > Standard questions come to mind: > > - What do your logs say? > > - Are you running anything semi-dangerous like rmagick or older > > versions of acts_as_ferret? > > - Are you using Ruby based log rotation? > > > > BTW, maybe we could put up solutions to common issues into a page on > > mongrel.rubyforge.org? I don''t see one on > > http://mongrel.rubyforge.org/docs/index.html... > > > > On 4/4/07, Pete DeLaurentis <pete at nextengine.com> wrote: > >> Hi guys, > >> > >> I''ve been running mongrel for a while now with Ruby 1.8.4, and last > >> week upgraded to 1.8.6. > >> > >> Since upgrading, each morning when I wake up there''s a big problem: > >> > >> 1. Accessing the site returns a "500 Internal Server Error" > >> 2. All the mongrel_rails processes are still running, but none of > >> them are active (when I run top) > >> 3. Lighttpd and pound are still running nicely > >> 4. There''s plenty of free RAM > >> 5. There''s plenty of free HD > >> 6. When I restart mongrel, it all starts working again > >> > >> I haven''t seen anything like this before. Does this sound familiar > >> to anyone? > >> > >> Thanks for the help, > >> Pete > >> _______________________________________________ > >> Mongrel-users mailing list > >> Mongrel-users at rubyforge.org > >> http://rubyforge.org/mailman/listinfo/mongrel-users > >> > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
Hi Eden, I''m running mongrel1.0.1, mongrel cluster 0.2.1, rails 1.2.3, and fastthread 1.0. I just cleaned up all old gems (since I had some back versions) and re-updated all of these gems just to be sure. Tomorrow morning, I''ll get to see if this helped :-) -Pete On Apr 4, 2007, at 7:11 AM, Eden Li wrote:> Which version of rails are you using? Version prior to 1.2.3 have a > few issues... Also, what version of mongrel are you running? > > On 4/4/07, Pete DeLaurentis <pete at nextengine.com> wrote: >> Hi Eden, >> >> I agree: a common issues page is a great idea. Unfortunately, this >> issue doesn''t seem to be one of the usual suspects. >> >> 1. The logs just have normal chatter for my app ... nothing unusual >> 2. No rmagick or ferret, but I believe it is related to Ruby 1.8.6, >> since that''s when ti started dying. >> 3. I''m not using log rotation, but the files aren''t that big after a >> day.. and there''s plenty of free space on disk >> >> Here''s a question: Is anyone running ruby1.8.6 succesfully with >> mongrel? >> >> Thanks, >> Pete >> >> >> >> On Apr 4, 2007, at 6:01 AM, Eden Li wrote: >> >>> Standard questions come to mind: >>> - What do your logs say? >>> - Are you running anything semi-dangerous like rmagick or older >>> versions of acts_as_ferret? >>> - Are you using Ruby based log rotation? >>> >>> BTW, maybe we could put up solutions to common issues into a page on >>> mongrel.rubyforge.org? I don''t see one on >>> http://mongrel.rubyforge.org/docs/index.html... >>> >>> On 4/4/07, Pete DeLaurentis <pete at nextengine.com> wrote: >>>> Hi guys, >>>> >>>> I''ve been running mongrel for a while now with Ruby 1.8.4, and last >>>> week upgraded to 1.8.6. >>>> >>>> Since upgrading, each morning when I wake up there''s a big problem: >>>> >>>> 1. Accessing the site returns a "500 Internal Server Error" >>>> 2. All the mongrel_rails processes are still running, but none of >>>> them are active (when I run top) >>>> 3. Lighttpd and pound are still running nicely >>>> 4. There''s plenty of free RAM >>>> 5. There''s plenty of free HD >>>> 6. When I restart mongrel, it all starts working again >>>> >>>> I haven''t seen anything like this before. Does this sound familiar >>>> to anyone? >>>> >>>> Thanks for the help, >>>> Pete >>>> _______________________________________________ >>>> Mongrel-users mailing list >>>> Mongrel-users at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/mongrel-users >>>> >>> _______________________________________________ >>> Mongrel-users mailing list >>> Mongrel-users at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/mongrel-users >> >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On Wed, 4 Apr 2007 05:43:54 -0700 Pete DeLaurentis <pete at nextengine.com> wrote:> Hi guys, > > I''ve been running mongrel for a while now with Ruby 1.8.4, and last > week upgraded to 1.8.6.First, NEVER upgrade a production server to new versions before you test the living hell out of it. This always leads to disaster. Second, I haven''t tested Mongrel on 1.8.6 and there''s apparently a small bug in the new thread implementation that''s included. You''ll want to try out fastthread very latest even if you''re running 1.8.6. Third, whenever you do an upgrade you have to recompile ALL of your gems. Every last one. No matter what. Best way to do this is uninstall all of your gems then reinstall the ones you need. It''s good spring cleaning, but it also makes sure that the compiled extensions work with the new ruby interpreter. Lastly, I only deploy 1.8.5 with fastthread, not 1.8.6. I''m sure there''s bugs lurking in 1.8.6 that haven''t been tested out on both Rails and Mongrel. Let me know what you find. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/
Are you sure this isn''t being caused by something unrelated to mongrel or ruby 1.8.6? We were having a similar problem a while ago and traced it down mysql closing its connection after 8 hours of inactivity (ActiveRecord, somewhat stupidly, does not attempt to reconnect the closed connection). Pete DeLaurentis wrote:> Hi guys, > > I''ve been running mongrel for a while now with Ruby 1.8.4, and last > week upgraded to 1.8.6. > > Since upgrading, each morning when I wake up there''s a big problem: > > 1. Accessing the site returns a "500 Internal Server Error" > 2. All the mongrel_rails processes are still running, but none of > them are active (when I run top) > 3. Lighttpd and pound are still running nicely > 4. There''s plenty of free RAM > 5. There''s plenty of free HD > 6. When I restart mongrel, it all starts working again > > I haven''t seen anything like this before. Does this sound familiar > to anyone? > > Thanks for the help, > Pete > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-usersThis e-mail message is privileged, confidential and subject to copyright. Any unauthorized use or disclosure is prohibited. Le contenu du pr''esent courriel est privil''egi''e, confidentiel et soumis `a des droits d''auteur. Il est interdit de l''utiliser ou de le divulguer sans autorisation.
Thanks for the tips Zed, I will uninstall + reinstall all of the gems. There is also a new version of fastthread, which I just installed. In the words of the author: "Well, just when I thought I was done with fastthread, it turns out that the version of fastthread merged into Ruby 1.8.6?s thread library had a couple serious bugs. So, here?s a new version of fastthread which can serve as a hotfix until the next Ruby release." - MenTaLguY, http://moonbase.rydia.net/ We made the switch to 1.8.6 for the reputed 10% speed boost. We''re using it on our in-house server that runs our beta app, and we''re not upgrading the customer servers until this is stable. Thanks for the help, Pete On Apr 4, 2007, at 7:37 AM, Zed A. Shaw wrote:> On Wed, 4 Apr 2007 05:43:54 -0700 > Pete DeLaurentis <pete at nextengine.com> wrote: > >> Hi guys, >> >> I''ve been running mongrel for a while now with Ruby 1.8.4, and last >> week upgraded to 1.8.6. > > First, NEVER upgrade a production server to new versions before you > test the living hell out of it. This always leads to disaster. > > Second, I haven''t tested Mongrel on 1.8.6 and there''s apparently a > small bug in the new thread implementation that''s included. You''ll > want to try out fastthread very latest even if you''re running 1.8.6. > > Third, whenever you do an upgrade you have to recompile ALL of your > gems. Every last one. No matter what. Best way to do this is > uninstall all of your gems then reinstall the ones you need. It''s > good > spring cleaning, but it also makes sure that the compiled extensions > work with the new ruby interpreter. > > Lastly, I only deploy 1.8.5 with fastthread, not 1.8.6. I''m sure > there''s bugs lurking in 1.8.6 that haven''t been tested out on both > Rails and Mongrel. > > Let me know what you find. > > -- > Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu > http://www.zedshaw.com/ > http://www.awprofessional.com/title/0321483502 -- The Mongrel Book > http://mongrel.rubyforge.org/ > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070404/6105b407/attachment-0001.html
Pete DeLaurentis wrote:> Here''s a question: Is anyone running ruby1.8.6 succesfully with mongrel?On Windows Vista I''m running Mongrel with Ruby 1.8.6, rails 1.2.3 as a service. I have no problems with the service shutting down or becoming unresponsive. I run this on a laptop, (with hibernation) uptime upto 50 hrs hasn''t been a problem. The "500 Internal Server Error" rings a bell, but I can''t remember what was causing it and the issue has been resolved for me. -Kaushik
On 4/4/07, Zed A. Shaw <zedshaw at zedshaw.com> wrote:> > Third, whenever you do an upgrade you have to recompile ALL of your > gems. Every last one. No matter what. Best way to do this is > uninstall all of your gems then reinstall the ones you need. It''s good > spring cleaning, but it also makes sure that the compiled extensions > work with the new ruby interpreter.You are referring here to *only* when you do an upgrade of the ruby intrepreter, right? - Rob
Yeah, I''ve been bitten by this one before too... in one of the background processes I run. Now that background process bites ActiveRecord every hour to keep it alive. I''ve never had any trouble with activerecord losing it''s connection in the mongrel processes, but it sounds like it''s possible. Maybe I should have that background process contact each of the mongrels. Or better yet, maybe I should look into how hard it would be to get ActiveRecord to restore the connection. Thanks, Pete On Apr 4, 2007, at 7:54 AM, Matt Zukowski wrote:> Are you sure this isn''t being caused by something unrelated to mongrel > or ruby 1.8.6? We were having a similar problem a while ago and traced > it down mysql closing its connection after 8 hours of inactivity > (ActiveRecord, somewhat stupidly, does not attempt to reconnect the > closed connection). > > Pete DeLaurentis wrote: >> Hi guys, >> >> I''ve been running mongrel for a while now with Ruby 1.8.4, and last >> week upgraded to 1.8.6. >> >> Since upgrading, each morning when I wake up there''s a big problem: >> >> 1. Accessing the site returns a "500 Internal Server Error" >> 2. All the mongrel_rails processes are still running, but none of >> them are active (when I run top) >> 3. Lighttpd and pound are still running nicely >> 4. There''s plenty of free RAM >> 5. There''s plenty of free HD >> 6. When I restart mongrel, it all starts working again >> >> I haven''t seen anything like this before. Does this sound familiar >> to anyone? >> >> Thanks for the help, >> Pete >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users > > > > This e-mail message is privileged, confidential and subject to > copyright. Any unauthorized use or disclosure is prohibited. > Le contenu du pr''esent courriel est privil''egi''e, confidentiel et > soumis `a des droits d''auteur. Il est interdit de l''utiliser ou de > le divulguer sans autorisation. > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On 4/4/07, kaushik.ghose <kaushik.ghose at gmail.com> wrote:> Pete DeLaurentis wrote: > > > Here''s a question: Is anyone running ruby1.8.6 succesfully with mongrel? > On Windows Vista I''m running Mongrel with Ruby 1.8.6, rails 1.2.3 as a > service. I have no problems with the service shutting down or becoming > unresponsive. I run this on a laptop, (with hibernation) uptime upto 50 > hrs hasn''t been a problem. >Hehehe, is good to know that you are up and running Kaus ;-)> The "500 Internal Server Error" rings a bell, but I can''t remember what > was causing it and the issue has been resolved for me. >Maybe MySql timeout? from mongrel faq [1] Q. Mongrel stops working if it''s left alone for a long time. If you find that Mongrel stops working after a long idle time and you''re using MySQL then you''re hitting a bug in the MySQL driver that doesn''t properly timeout connections. What happens is the MySQL server side of the connection times out and closes, but the MySQL client doesn''t detect this and just sits there. What you have to do is set: ActiveRecord::Base.verification_timeout = 14400 Or to any value that is lower than the MySQL server''s interactive_timeout setting. This will make sure that ActiveRecord checks the connection often enough to reset the connection. [1] http://mongrel.rubyforge.org/faq.html -- Luis Lavena Multimedia systems - Leaders are made, they are not born. They are made by hard effort, which is the price which all of us must pay to achieve any goal that is worthwhile. Vince Lombardi
> What you have to do is set: > > ActiveRecord::Base.verification_timeout = 14400 > > Or to any value that is lower than the MySQL server''s > interactive_timeout setting. This will make sure that ActiveRecord > checks the connection often enough to reset the connection.Sweet. Thanks Luis. This is exactly what I''m looking for. -Pete On Apr 4, 2007, at 9:38 AM, Luis Lavena wrote:> On 4/4/07, kaushik.ghose <kaushik.ghose at gmail.com> wrote: >> Pete DeLaurentis wrote: >> >>> Here''s a question: Is anyone running ruby1.8.6 succesfully with >>> mongrel? >> On Windows Vista I''m running Mongrel with Ruby 1.8.6, rails 1.2.3 >> as a >> service. I have no problems with the service shutting down or >> becoming >> unresponsive. I run this on a laptop, (with hibernation) uptime >> upto 50 >> hrs hasn''t been a problem. >> > > Hehehe, is good to know that you are up and running Kaus ;-) > >> The "500 Internal Server Error" rings a bell, but I can''t remember >> what >> was causing it and the issue has been resolved for me. >> > > Maybe MySql timeout? from mongrel faq [1] > > Q. Mongrel stops working if it''s left alone for a long time. > > If you find that Mongrel stops working after a long idle time and > you''re using MySQL then you''re hitting a bug in the MySQL driver that > doesn''t properly timeout connections. What happens is the MySQL server > side of the connection times out and closes, but the MySQL client > doesn''t detect this and just sits there. > > What you have to do is set: > > ActiveRecord::Base.verification_timeout = 14400 > > Or to any value that is lower than the MySQL server''s > interactive_timeout setting. This will make sure that ActiveRecord > checks the connection often enough to reset the connection. > > > [1] http://mongrel.rubyforge.org/faq.html > > -- > Luis Lavena > Multimedia systems > - > Leaders are made, they are not born. They are made by hard effort, > which is the price which all of us must pay to achieve any goal that > is worthwhile. > Vince Lombardi > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users