Mongrel users - Dec 2006 - Early morning strange error saying: Status: 500 Internal Server Error

So I have updated to the latest mongrel and fastthread and my  
environment is as follows:

Debian 3.1
Rails 1.1.6
Ruby 1.8.5
Mongrel 0.3.18
Fastthread 0.4
Mongrel Cluster 0.2.1
Apache 2.2.3

I again woke up this morning to my disappointment of seeing the  
strange Status: 500 Internal Server error. It happens the same every  
morning, I have three mongrels running, two give the error, one  
processes the request successfully.

This error is not being reported in ANY logs.  Apache, Mongrel debug,  
rails, nothing.  The only thing I notice while tailing the logs is  
that the request reaches mongrel and is reported in both rails.log  
and threads.log, however the request is NOT being reported in ruby /  
rails production.log.  This appears to be craping out in mongrel.   
Again this only happens overnight when there is no activity with the  
site.  PLEASE HELP.  Is there any way to take a core dump of mongrel?  
How can I get more logs to see what is happening.

Regards,

Matt Whittaker

Have you ruled out the MySQL connection timeout issue?

Erik
On Dec 15, 2006, at 10:01 AM, Matthew Whittaker wrote:
> So I have updated to the latest mongrel and fastthread and my
> environment is as follows:
>
> Debian 3.1
> Rails 1.1.6
> Ruby 1.8.5
> Mongrel 0.3.18
> Fastthread 0.4
> Mongrel Cluster 0.2.1
> Apache 2.2.3
>
> I again woke up this morning to my disappointment of seeing the
> strange Status: 500 Internal Server error. It happens the same every
> morning, I have three mongrels running, two give the error, one
> processes the request successfully.
>
> This error is not being reported in ANY logs.  Apache, Mongrel debug,
> rails, nothing.  The only thing I notice while tailing the logs is
> that the request reaches mongrel and is reported in both rails.log
> and threads.log, however the request is NOT being reported in ruby /
> rails production.log.  This appears to be craping out in mongrel.
> Again this only happens overnight when there is no activity with the
> site.  PLEASE HELP.  Is there any way to take a core dump of mongrel?
> How can I get more logs to see what is happening.
>
> Regards,
>
> Matt Whittaker
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

I second this.  These symptoms are very similar to something that I 
experienced and the culprit ended up being the MySql timeout issue.

I talk more about my solution here: 
http://rubyforge.org/pipermail/mongrel-users/2006-November/002178.html


Cheers,
Steven


Erik Morton wrote:
> Have you ruled out the MySQL connection timeout issue?
>
> Erik
> On Dec 15, 2006, at 10:01 AM, Matthew Whittaker wrote:
>
>   
>> So I have updated to the latest mongrel and fastthread and my
>> environment is as follows:
>>
>> Debian 3.1
>> Rails 1.1.6
>> Ruby 1.8.5
>> Mongrel 0.3.18
>> Fastthread 0.4
>> Mongrel Cluster 0.2.1
>> Apache 2.2.3
>>
>> I again woke up this morning to my disappointment of seeing the
>> strange Status: 500 Internal Server error. It happens the same every
>> morning, I have three mongrels running, two give the error, one
>> processes the request successfully.
>>
>> This error is not being reported in ANY logs.  Apache, Mongrel debug,
>> rails, nothing.  The only thing I notice while tailing the logs is
>> that the request reaches mongrel and is reported in both rails.log
>> and threads.log, however the request is NOT being reported in ruby /
>> rails production.log.  This appears to be craping out in mongrel.
>> Again this only happens overnight when there is no activity with the
>> site.  PLEASE HELP.  Is there any way to take a core dump of mongrel?
>> How can I get more logs to see what is happening.
>>
>> Regards,
>>
>> Matt Whittaker
>> _______________________________________________
>> Mongrel-users mailing list
>> Mongrel-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>     
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

Thanks, I will look into it.

Cheers,
Matt


On Dec 15, 2006, at 10:36 AM, Steven Hansen wrote:
>
> I second this.  These symptoms are very similar to something that I
> experienced and the culprit ended up being the MySql timeout issue.
>
> I talk more about my solution here:
> http://rubyforge.org/pipermail/mongrel-users/2006-November/002178.html
>
>
> Cheers,
> Steven
>
>
> Erik Morton wrote:
>> Have you ruled out the MySQL connection timeout issue?
>>
>> Erik
>> On Dec 15, 2006, at 10:01 AM, Matthew Whittaker wrote:
>>
>>
>>> So I have updated to the latest mongrel and fastthread and my
>>> environment is as follows:
>>>
>>> Debian 3.1
>>> Rails 1.1.6
>>> Ruby 1.8.5
>>> Mongrel 0.3.18
>>> Fastthread 0.4
>>> Mongrel Cluster 0.2.1
>>> Apache 2.2.3
>>>
>>> I again woke up this morning to my disappointment of seeing the
>>> strange Status: 500 Internal Server error. It happens the same
every
>>> morning, I have three mongrels running, two give the error, one
>>> processes the request successfully.
>>>
>>> This error is not being reported in ANY logs.  Apache, Mongrel  
>>> debug,
>>> rails, nothing.  The only thing I notice while tailing the logs is
>>> that the request reaches mongrel and is reported in both rails.log
>>> and threads.log, however the request is NOT being reported in ruby
/
>>> rails production.log.  This appears to be craping out in mongrel.
>>> Again this only happens overnight when there is no activity with
the
>>> site.  PLEASE HELP.  Is there any way to take a core dump of  
>>> mongrel?
>>> How can I get more logs to see what is happening.
>>>
>>> Regards,
>>>
>>> Matt Whittaker
>>> _______________________________________________
>>> Mongrel-users mailing list
>>> Mongrel-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>>
>>
>> _______________________________________________
>> Mongrel-users mailing list
>> Mongrel-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

I, too had a similar problem - which persisted even after I inserted
this:

ActiveRecord::Base.verification_timeout = 14400

Into my config/production.rb

On close inspection of my apache logs I discovered that a robot was
hitting my application with a (near) simultaneous pair of get requests
with identical session_id''s.  The session_id was new - and so both
rails processes attempted to insert a new record into the session
table, causing the database to report a duplicate key violation on the
second insert.

Apparently the second rails process determined that the session_id was
new before the first rails process had finished its insert.

Hope this is helpful.

Robert

How are you handling log rotation?  I was encountering really strange,
non-logged 500 errors every week because I used some bad log rotation
advice from the Rails wiki.  I don''t have the link handy since it
seems the wiki is down right now, but I found this thread:

  http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/218107

The jist: Don''t let Logger rotate the log.

Instead, let Rails log as much as it wants to into production.log and
rely on some out-of-process log rotation daemon (eg, logrotate).   Or,
switch to some ruby-based logging solution that works when more than
one process tries to rotate the same log file. I personally use the
former.

HTH,
eden

On 12/15/06, Matthew Whittaker <mwhitt89 at gmail.com>
wrote:
> So I have updated to the latest mongrel and fastthread and my
> environment is as follows:
>
> Debian 3.1
> Rails 1.1.6
> Ruby 1.8.5
> Mongrel 0.3.18
> Fastthread 0.4
> Mongrel Cluster 0.2.1
> Apache 2.2.3
>
> I again woke up this morning to my disappointment of seeing the
> strange Status: 500 Internal Server error. It happens the same every
> morning, I have three mongrels running, two give the error, one
> processes the request successfully.
>
> This error is not being reported in ANY logs.  Apache, Mongrel debug,
> rails, nothing.  The only thing I notice while tailing the logs is
> that the request reaches mongrel and is reported in both rails.log
> and threads.log, however the request is NOT being reported in ruby /
> rails production.log.  This appears to be craping out in mongrel.
> Again this only happens overnight when there is no activity with the
> site.  PLEASE HELP.  Is there any way to take a core dump of mongrel?
> How can I get more logs to see what is happening.
>
> Regards,
>
> Matt Whittaker
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

THANK YOU EDEN!

I have my production log on rotate!  I am going to go change this now.

I will post the results.

Cheers,
matt


On Dec 16, 2006, at 6:39 AM, Eden Li wrote:
> How are you handling log rotation?  I was encountering really strange,
> non-logged 500 errors every week because I used some bad log rotation
> advice from the Rails wiki.  I don''t have the link handy since it
> seems the wiki is down right now, but I found this thread:
>
>   http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/218107
>
> The jist: Don''t let Logger rotate the log.
>
> Instead, let Rails log as much as it wants to into production.log and
> rely on some out-of-process log rotation daemon (eg, logrotate).   Or,
> switch to some ruby-based logging solution that works when more than
> one process tries to rotate the same log file. I personally use the
> former.
>
> HTH,
> eden
>
> On 12/15/06, Matthew Whittaker <mwhitt89 at gmail.com> wrote:
>> So I have updated to the latest mongrel and fastthread and my
>> environment is as follows:
>>
>> Debian 3.1
>> Rails 1.1.6
>> Ruby 1.8.5
>> Mongrel 0.3.18
>> Fastthread 0.4
>> Mongrel Cluster 0.2.1
>> Apache 2.2.3
>>
>> I again woke up this morning to my disappointment of seeing the
>> strange Status: 500 Internal Server error. It happens the same every
>> morning, I have three mongrels running, two give the error, one
>> processes the request successfully.
>>
>> This error is not being reported in ANY logs.  Apache, Mongrel debug,
>> rails, nothing.  The only thing I notice while tailing the logs is
>> that the request reaches mongrel and is reported in both rails.log
>> and threads.log, however the request is NOT being reported in ruby /
>> rails production.log.  This appears to be craping out in mongrel.
>> Again this only happens overnight when there is no activity with the
>> site.  PLEASE HELP.  Is there any way to take a core dump of mongrel?
>> How can I get more logs to see what is happening.
>>
>> Regards,
>>
>> Matt Whittaker
>> _______________________________________________
>> Mongrel-users mailing list
>> Mongrel-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

On Sat, 16 Dec 2006 12:00:28 -0500
Matthew Whittaker <mwhitt89 at gmail.com> wrote:
> THANK YOU EDEN!
> 
> I have my production log on rotate!  I am going to go change this now.
> 
> I will post the results.
Hmmm, I may have to put this in the FAQ.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

I changed the ruby logger to not rotate anymore and the problem is  
fixed.

Zed - This would be a good one to put on the FAQ.

DO NOT have ruby logger rotate itself in a mongrel clustered  
environment.

Thanks again Eden!


On Dec 16, 2006, at 6:40 PM, Zed A. Shaw wrote:
> On Sat, 16 Dec 2006 12:00:28 -0500
> Matthew Whittaker <mwhitt89 at gmail.com> wrote:
>
>> THANK YOU EDEN!
>>
>> I have my production log on rotate!  I am going to go change this  
>> now.
>>
>> I will post the results.
>
> Hmmm, I may have to put this in the FAQ.
>
> -- 
> Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
> http://www.zedshaw.com/
> http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users