Matthew Whittaker
2006-Dec-15 15:01 UTC
[Mongrel] Early morning strange error saying: Status: 500 Internal Server Error
So I have updated to the latest mongrel and fastthread and my environment is as follows: Debian 3.1 Rails 1.1.6 Ruby 1.8.5 Mongrel 0.3.18 Fastthread 0.4 Mongrel Cluster 0.2.1 Apache 2.2.3 I again woke up this morning to my disappointment of seeing the strange Status: 500 Internal Server error. It happens the same every morning, I have three mongrels running, two give the error, one processes the request successfully. This error is not being reported in ANY logs. Apache, Mongrel debug, rails, nothing. The only thing I notice while tailing the logs is that the request reaches mongrel and is reported in both rails.log and threads.log, however the request is NOT being reported in ruby / rails production.log. This appears to be craping out in mongrel. Again this only happens overnight when there is no activity with the site. PLEASE HELP. Is there any way to take a core dump of mongrel? How can I get more logs to see what is happening. Regards, Matt Whittaker
Erik Morton
2006-Dec-15 15:17 UTC
[Mongrel] Early morning strange error saying: Status: 500 Internal Server Error
Have you ruled out the MySQL connection timeout issue? Erik On Dec 15, 2006, at 10:01 AM, Matthew Whittaker wrote:> So I have updated to the latest mongrel and fastthread and my > environment is as follows: > > Debian 3.1 > Rails 1.1.6 > Ruby 1.8.5 > Mongrel 0.3.18 > Fastthread 0.4 > Mongrel Cluster 0.2.1 > Apache 2.2.3 > > I again woke up this morning to my disappointment of seeing the > strange Status: 500 Internal Server error. It happens the same every > morning, I have three mongrels running, two give the error, one > processes the request successfully. > > This error is not being reported in ANY logs. Apache, Mongrel debug, > rails, nothing. The only thing I notice while tailing the logs is > that the request reaches mongrel and is reported in both rails.log > and threads.log, however the request is NOT being reported in ruby / > rails production.log. This appears to be craping out in mongrel. > Again this only happens overnight when there is no activity with the > site. PLEASE HELP. Is there any way to take a core dump of mongrel? > How can I get more logs to see what is happening. > > Regards, > > Matt Whittaker > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
Steven Hansen
2006-Dec-15 15:36 UTC
[Mongrel] Early morning strange error saying: Status: 500 Internal Server Error
I second this. These symptoms are very similar to something that I experienced and the culprit ended up being the MySql timeout issue. I talk more about my solution here: http://rubyforge.org/pipermail/mongrel-users/2006-November/002178.html Cheers, Steven Erik Morton wrote:> Have you ruled out the MySQL connection timeout issue? > > Erik > On Dec 15, 2006, at 10:01 AM, Matthew Whittaker wrote: > > >> So I have updated to the latest mongrel and fastthread and my >> environment is as follows: >> >> Debian 3.1 >> Rails 1.1.6 >> Ruby 1.8.5 >> Mongrel 0.3.18 >> Fastthread 0.4 >> Mongrel Cluster 0.2.1 >> Apache 2.2.3 >> >> I again woke up this morning to my disappointment of seeing the >> strange Status: 500 Internal Server error. It happens the same every >> morning, I have three mongrels running, two give the error, one >> processes the request successfully. >> >> This error is not being reported in ANY logs. Apache, Mongrel debug, >> rails, nothing. The only thing I notice while tailing the logs is >> that the request reaches mongrel and is reported in both rails.log >> and threads.log, however the request is NOT being reported in ruby / >> rails production.log. This appears to be craping out in mongrel. >> Again this only happens overnight when there is no activity with the >> site. PLEASE HELP. Is there any way to take a core dump of mongrel? >> How can I get more logs to see what is happening. >> >> Regards, >> >> Matt Whittaker >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
Matthew Whittaker
2006-Dec-15 15:54 UTC
[Mongrel] Early morning strange error saying: Status: 500 Internal Server Error
Thanks, I will look into it. Cheers, Matt On Dec 15, 2006, at 10:36 AM, Steven Hansen wrote:> > I second this. These symptoms are very similar to something that I > experienced and the culprit ended up being the MySql timeout issue. > > I talk more about my solution here: > http://rubyforge.org/pipermail/mongrel-users/2006-November/002178.html > > > Cheers, > Steven > > > Erik Morton wrote: >> Have you ruled out the MySQL connection timeout issue? >> >> Erik >> On Dec 15, 2006, at 10:01 AM, Matthew Whittaker wrote: >> >> >>> So I have updated to the latest mongrel and fastthread and my >>> environment is as follows: >>> >>> Debian 3.1 >>> Rails 1.1.6 >>> Ruby 1.8.5 >>> Mongrel 0.3.18 >>> Fastthread 0.4 >>> Mongrel Cluster 0.2.1 >>> Apache 2.2.3 >>> >>> I again woke up this morning to my disappointment of seeing the >>> strange Status: 500 Internal Server error. It happens the same every >>> morning, I have three mongrels running, two give the error, one >>> processes the request successfully. >>> >>> This error is not being reported in ANY logs. Apache, Mongrel >>> debug, >>> rails, nothing. The only thing I notice while tailing the logs is >>> that the request reaches mongrel and is reported in both rails.log >>> and threads.log, however the request is NOT being reported in ruby / >>> rails production.log. This appears to be craping out in mongrel. >>> Again this only happens overnight when there is no activity with the >>> site. PLEASE HELP. Is there any way to take a core dump of >>> mongrel? >>> How can I get more logs to see what is happening. >>> >>> Regards, >>> >>> Matt Whittaker >>> _______________________________________________ >>> Mongrel-users mailing list >>> Mongrel-users at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/mongrel-users >>> >> >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
Robert Vogel
2006-Dec-15 17:15 UTC
[Mongrel] Early morning strange error saying: Status: 500 Internal Server Error
I, too had a similar problem - which persisted even after I inserted this: ActiveRecord::Base.verification_timeout = 14400 Into my config/production.rb On close inspection of my apache logs I discovered that a robot was hitting my application with a (near) simultaneous pair of get requests with identical session_id''s. The session_id was new - and so both rails processes attempted to insert a new record into the session table, causing the database to report a duplicate key violation on the second insert. Apparently the second rails process determined that the session_id was new before the first rails process had finished its insert. Hope this is helpful. Robert
Eden Li
2006-Dec-16 11:39 UTC
[Mongrel] Early morning strange error saying: Status: 500 Internal Server Error
How are you handling log rotation? I was encountering really strange, non-logged 500 errors every week because I used some bad log rotation advice from the Rails wiki. I don''t have the link handy since it seems the wiki is down right now, but I found this thread: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/218107 The jist: Don''t let Logger rotate the log. Instead, let Rails log as much as it wants to into production.log and rely on some out-of-process log rotation daemon (eg, logrotate). Or, switch to some ruby-based logging solution that works when more than one process tries to rotate the same log file. I personally use the former. HTH, eden On 12/15/06, Matthew Whittaker <mwhitt89 at gmail.com> wrote:> So I have updated to the latest mongrel and fastthread and my > environment is as follows: > > Debian 3.1 > Rails 1.1.6 > Ruby 1.8.5 > Mongrel 0.3.18 > Fastthread 0.4 > Mongrel Cluster 0.2.1 > Apache 2.2.3 > > I again woke up this morning to my disappointment of seeing the > strange Status: 500 Internal Server error. It happens the same every > morning, I have three mongrels running, two give the error, one > processes the request successfully. > > This error is not being reported in ANY logs. Apache, Mongrel debug, > rails, nothing. The only thing I notice while tailing the logs is > that the request reaches mongrel and is reported in both rails.log > and threads.log, however the request is NOT being reported in ruby / > rails production.log. This appears to be craping out in mongrel. > Again this only happens overnight when there is no activity with the > site. PLEASE HELP. Is there any way to take a core dump of mongrel? > How can I get more logs to see what is happening. > > Regards, > > Matt Whittaker > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
Matthew Whittaker
2006-Dec-16 17:00 UTC
[Mongrel] Early morning strange error saying: Status: 500 Internal Server Error
THANK YOU EDEN! I have my production log on rotate! I am going to go change this now. I will post the results. Cheers, matt On Dec 16, 2006, at 6:39 AM, Eden Li wrote:> How are you handling log rotation? I was encountering really strange, > non-logged 500 errors every week because I used some bad log rotation > advice from the Rails wiki. I don''t have the link handy since it > seems the wiki is down right now, but I found this thread: > > http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/218107 > > The jist: Don''t let Logger rotate the log. > > Instead, let Rails log as much as it wants to into production.log and > rely on some out-of-process log rotation daemon (eg, logrotate). Or, > switch to some ruby-based logging solution that works when more than > one process tries to rotate the same log file. I personally use the > former. > > HTH, > eden > > On 12/15/06, Matthew Whittaker <mwhitt89 at gmail.com> wrote: >> So I have updated to the latest mongrel and fastthread and my >> environment is as follows: >> >> Debian 3.1 >> Rails 1.1.6 >> Ruby 1.8.5 >> Mongrel 0.3.18 >> Fastthread 0.4 >> Mongrel Cluster 0.2.1 >> Apache 2.2.3 >> >> I again woke up this morning to my disappointment of seeing the >> strange Status: 500 Internal Server error. It happens the same every >> morning, I have three mongrels running, two give the error, one >> processes the request successfully. >> >> This error is not being reported in ANY logs. Apache, Mongrel debug, >> rails, nothing. The only thing I notice while tailing the logs is >> that the request reaches mongrel and is reported in both rails.log >> and threads.log, however the request is NOT being reported in ruby / >> rails production.log. This appears to be craping out in mongrel. >> Again this only happens overnight when there is no activity with the >> site. PLEASE HELP. Is there any way to take a core dump of mongrel? >> How can I get more logs to see what is happening. >> >> Regards, >> >> Matt Whittaker >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
Zed A. Shaw
2006-Dec-16 23:40 UTC
[Mongrel] Early morning strange error saying: Status: 500 Internal Server Error
On Sat, 16 Dec 2006 12:00:28 -0500 Matthew Whittaker <mwhitt89 at gmail.com> wrote:> THANK YOU EDEN! > > I have my production log on rotate! I am going to go change this now. > > I will post the results.Hmmm, I may have to put this in the FAQ. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Matthew Whittaker
2006-Dec-17 15:29 UTC
[Mongrel] Early morning strange error saying: Status: 500 Internal Server Error
I changed the ruby logger to not rotate anymore and the problem is fixed. Zed - This would be a good one to put on the FAQ. DO NOT have ruby logger rotate itself in a mongrel clustered environment. Thanks again Eden! On Dec 16, 2006, at 6:40 PM, Zed A. Shaw wrote:> On Sat, 16 Dec 2006 12:00:28 -0500 > Matthew Whittaker <mwhitt89 at gmail.com> wrote: > >> THANK YOU EDEN! >> >> I have my production log on rotate! I am going to go change this >> now. >> >> I will post the results. > > Hmmm, I may have to put this in the FAQ. > > -- > Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu > http://www.zedshaw.com/ > http://www.awprofessional.com/title/0321483502 -- The Mongrel Book > http://mongrel.rubyforge.org/ > http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users