Mongrel users - Jul 2007 - Mongrel stops responding after period of inactivity

Hi
I posted this to the Ruby on Rails Talk group but I thought I''d post it
here
too as it''s probably a more appropriate group.  Hope this is ok.

I''m running a 2-instance Mongrel cluster behind Apache 2.2.4 with Rails
1.2.3.  If no requests are received by the application for several hours
(this usually happens overnight) then Mongrel stops
responding and no requests are detected by Rails (at least nothing is in the
Rails log).  Nothing untoward is in the Mongrel log.

If you try and visit the application, the request times out with a 502 Proxy
Error.  Apache is still up and running and serving static files without a
problem, but the following is present in the Apache error log (from trying
to hit the SessionController, identified by / sessions) which may provide a
clue:

    [Sun Jul 29 08:37:36 2007] [error] proxy: error reading status line from
remote server 127.0.0.1
    [Sun Jul 29 08:37:36 2007] [error] proxy: Error reading from remote
server returned by /sessions

Restarting the Mongrel cluster resolves the problem until the next time it
happens.  I have done a lot of reading about this issue online and a number
of sources -- including the Mongrel FAQ -- point to being able to fix a
''hanging mongrel'' situation by setting this value in
environment.rb:

    ActiveRecord::Base.verification_timeout = 14400

This make the ActiveRecord timeout value less than the MySQL default of
28800.  I have made this change and it doesn''t appear to make a
difference.

Can anyone advise me on what to try next to diagnose this issue?
 I''m quickly running out of ideas and I''d appreciate a bit of
help!

Here are the vitals from our Ubuntu 6.0.6 server:

    MySQL
        Ver 14.12 Distrib 5.0.22, for pc-linux-gnu (x86_64) using readline
5.1
        connect_timeout                 5
        interactive_timeout             28800
        max_connect_errors           10
        max_connections               100
        max_user_connections       0
        wait_timeout                      28800

    ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]
        cgi_multipart_eof_fix (2.2)
        fastthread (1.0)
        mongrel (1.0.1)
        mongrel_cluster (1.0.2)

    Apache/2.2.4 (Unix)

    Mongrel Cluster Config:
        port: "8000"
        environment: production
        address: 127.0.0.1
        pid_file: log/mongrel.pid
        servers: 2
        user: [the user]
        group: [the group]

    Apache Load Balancer setup:
        <Proxy balancer://appname_cluster>
        BalancerMember http://127.0.0.1:8000
        BalancerMember http://127.0.0.1:8001
        </Proxy>

    Apache Loaded Modules:
        core_module (static)
        authn_file_module (static)
        authn_default_module (static)
        authz_host_module (static)
        authz_groupfile_module (static)
        authz_user_module (static)
        authz_default_module (static)
        auth_basic_module (static)
        cache_module (static)
        include_module (static)
        filter_module (static)
        deflate_module (static)
        log_config_module (static)
        env_module (static)
        headers_module (static)
        setenvif_module (static)
        proxy_module (static)
        proxy_connect_module (static)
        proxy_ftp_module (static)
        proxy_http_module (static)
        proxy_ajp_module (static)
        proxy_balancer_module (static)
        ssl_module (static)
        mpm_prefork_module (static)
        http_module (static)
        mime_module (static)
        status_module (static)
        autoindex_module (static)
        asis_module (static)
        cgi_module (static)
        negotiation_module (static)
        dir_module (static)
        actions_module (static)
        userdir_module (static)
        alias_module (static)
        rewrite_module (static)
        so_module (static)
        php5_module (shared)
        info_module (shared)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070729/44d69379/attachment.html

Looks like you hit the venerable Apache bug 39499 (and/or 37770), maybe.

Try adding

  SetEnv force-proxy-request-1.0 1
  SetEnv proxy-nokeepalive 1

to your VirtualHost block for the Rails app.

Evan

On 7/29/07, Olly Lylo <list at lylo.co.uk> wrote:
> Hi
>
>
> I posted this to the Ruby on Rails Talk group but I thought I''d
post it here
> too as it''s probably a more appropriate group.  Hope this is ok.
>
>
> I''m running a 2-instance Mongrel cluster behind Apache 2.2.4 with
Rails
> 1.2.3.  If no requests are received by the application for several hours
> (this usually happens overnight) then Mongrel stops
> responding and no requests are detected by Rails (at least nothing is in
the
> Rails log).  Nothing untoward is in the Mongrel log.
>
>
> If you try and visit the application, the request times out with a 502
Proxy
> Error.  Apache is still up and running and serving static files without a
> problem, but the following is present in the Apache error log (from trying
> to hit the SessionController, identified by / sessions) which may provide a
> clue:
>
>
>     [Sun Jul 29 08:37:36 2007] [error] proxy: error reading status line
from
> remote server 127.0.0.1
>     [Sun Jul 29 08:37:36 2007] [error] proxy: Error reading from remote
> server returned by /sessions
>
>
> Restarting the Mongrel cluster resolves the problem until the next time it
> happens.  I have done a lot of reading about this issue online and a number
> of sources -- including the Mongrel FAQ -- point to being able to fix a
> ''hanging mongrel'' situation by setting this value in 
environment.rb:
>
>
>     ActiveRecord::Base.verification_timeout = 14400
>
>
> This make the ActiveRecord timeout value less than the MySQL default of
> 28800.  I have made this change and it doesn''t appear to make a
difference.
>
>
> Can anyone advise me on what to try next to diagnose this issue? 
I''m
> quickly running out of ideas and I''d appreciate a bit of help!
>
>
> Here are the vitals from our Ubuntu 6.0.6 server:
>
>
>     MySQL
>         Ver 14.12 Distrib 5.0.22, for pc-linux-gnu (x86_64) using readline
> 5.1
>         connect_timeout                 5
>         interactive_timeout             28800
>         max_connect_errors           10
>         max_connections               100
>         max_user_connections       0
>         wait_timeout                      28800
>
>
>     ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]
>         cgi_multipart_eof_fix ( 2.2)
>         fastthread (1.0)
>         mongrel (1.0.1)
>         mongrel_cluster (1.0.2)
>
>
>     Apache/2.2.4 (Unix)
>
>
>     Mongrel Cluster Config:
>         port: "8000"
>         environment: production
>         address: 127.0.0.1
>         pid_file: log/mongrel.pid
>         servers: 2
>         user: [the user]
>         group: [the group]
>
>
>     Apache Load Balancer setup:
>         <Proxy balancer://appname_cluster>
>         BalancerMember http://127.0.0.1:8000
>         BalancerMember  http://127.0.0.1:8001
>         </Proxy>
>
>
>     Apache Loaded Modules:
>         core_module (static)
>          authn_file_module (static)
>         authn_default_module (static)
>         authz_host_module (static)
>         authz_groupfile_module (static)
>         authz_user_module (static)
>         authz_default_module (static)
>         auth_basic_module (static)
>         cache_module (static)
>         include_module (static)
>         filter_module (static)
>         deflate_module (static)
>         log_config_module (static)
>          env_module (static)
>         headers_module (static)
>         setenvif_module (static)
>         proxy_module (static)
>         proxy_connect_module (static)
>         proxy_ftp_module (static)
>         proxy_http_module (static)
>         proxy_ajp_module (static)
>         proxy_balancer_module (static)
>         ssl_module (static)
>         mpm_prefork_module (static)
>         http_module (static)
>         mime_module (static)
>          status_module (static)
>         autoindex_module (static)
>         asis_module (static)
>         cgi_module (static)
>         negotiation_module (static)
>         dir_module (static)
>         actions_module (static)
>         userdir_module (static)
>         alias_module (static)
>         rewrite_module (static)
>         so_module (static)
>         php5_module (shared)
>         info_module (shared)
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

-- 
Evan Weaver
Cloudburst, LLC

I get the same issue behind Lighttpd, and so I don''t believe this is  
an Apache bug.

Facts:

- Mongrel stops working after a period of inactivity
- ActiveRecord timeout is not the issue (recommended fix changes  
nothing)
- Happens behind lighttpd and apache
- Nothing in the logs to give any clues

Workaround:

I haven''t found a solution yet, but if I setup monit to contact the  
mongrels every few minutes, I''m able to prevent this issue, since  
it''s related to inactivity.

Hope this helps,
Pete

PS. Does anyone know if Zed has any future bug-fix releases or  
updates planned?  I believe the last release was January, right?


On Jul 29, 2007, at 4:47 PM, Evan Weaver wrote:
> Looks like you hit the venerable Apache bug 39499 (and/or 37770),  
> maybe.
>
> Try adding
>
>   SetEnv force-proxy-request-1.0 1
>   SetEnv proxy-nokeepalive 1
>
> to your VirtualHost block for the Rails app.
>
> Evan
>
> On 7/29/07, Olly Lylo <list at lylo.co.uk> wrote:
>> Hi
>>
>>
>> I posted this to the Ruby on Rails Talk group but I thought
I''d
>> post it here
>> too as it''s probably a more appropriate group.  Hope this is
ok.
>>
>>
>> I''m running a 2-instance Mongrel cluster behind Apache 2.2.4
with
>> Rails
>> 1.2.3.  If no requests are received by the application for several  
>> hours
>> (this usually happens overnight) then Mongrel stops
>> responding and no requests are detected by Rails (at least nothing  
>> is in the
>> Rails log).  Nothing untoward is in the Mongrel log.
>>
>>
>> If you try and visit the application, the request times out with a  
>> 502 Proxy
>> Error.  Apache is still up and running and serving static files  
>> without a
>> problem, but the following is present in the Apache error log  
>> (from trying
>> to hit the SessionController, identified by / sessions) which may  
>> provide a
>> clue:
>>
>>
>>     [Sun Jul 29 08:37:36 2007] [error] proxy: error reading status  
>> line from
>> remote server 127.0.0.1
>>     [Sun Jul 29 08:37:36 2007] [error] proxy: Error reading from  
>> remote
>> server returned by /sessions
>>
>>
>> Restarting the Mongrel cluster resolves the problem until the next  
>> time it
>> happens.  I have done a lot of reading about this issue online and  
>> a number
>> of sources -- including the Mongrel FAQ -- point to being able to  
>> fix a
>> ''hanging mongrel'' situation by setting this value in 
environment.rb:
>>
>>
>>     ActiveRecord::Base.verification_timeout = 14400
>>
>>
>> This make the ActiveRecord timeout value less than the MySQL  
>> default of
>> 28800.  I have made this change and it doesn''t appear to make
a
>> difference.
>>
>>
>> Can anyone advise me on what to try next to diagnose this issue? 
I''m
>> quickly running out of ideas and I''d appreciate a bit of help!
>>
>>
>> Here are the vitals from our Ubuntu 6.0.6 server:
>>
>>
>>     MySQL
>>         Ver 14.12 Distrib 5.0.22, for pc-linux-gnu (x86_64) using  
>> readline
>> 5.1
>>         connect_timeout                 5
>>         interactive_timeout             28800
>>         max_connect_errors           10
>>         max_connections               100
>>         max_user_connections       0
>>         wait_timeout                      28800
>>
>>
>>     ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]
>>         cgi_multipart_eof_fix ( 2.2)
>>         fastthread (1.0)
>>         mongrel (1.0.1)
>>         mongrel_cluster (1.0.2)
>>
>>
>>     Apache/2.2.4 (Unix)
>>
>>
>>     Mongrel Cluster Config:
>>         port: "8000"
>>         environment: production
>>         address: 127.0.0.1
>>         pid_file: log/mongrel.pid
>>         servers: 2
>>         user: [the user]
>>         group: [the group]
>>
>>
>>     Apache Load Balancer setup:
>>         <Proxy balancer://appname_cluster>
>>         BalancerMember http://127.0.0.1:8000
>>         BalancerMember  http://127.0.0.1:8001
>>         </Proxy>
>>
>>
>>     Apache Loaded Modules:
>>         core_module (static)
>>          authn_file_module (static)
>>         authn_default_module (static)
>>         authz_host_module (static)
>>         authz_groupfile_module (static)
>>         authz_user_module (static)
>>         authz_default_module (static)
>>         auth_basic_module (static)
>>         cache_module (static)
>>         include_module (static)
>>         filter_module (static)
>>         deflate_module (static)
>>         log_config_module (static)
>>          env_module (static)
>>         headers_module (static)
>>         setenvif_module (static)
>>         proxy_module (static)
>>         proxy_connect_module (static)
>>         proxy_ftp_module (static)
>>         proxy_http_module (static)
>>         proxy_ajp_module (static)
>>         proxy_balancer_module (static)
>>         ssl_module (static)
>>         mpm_prefork_module (static)
>>         http_module (static)
>>         mime_module (static)
>>          status_module (static)
>>         autoindex_module (static)
>>         asis_module (static)
>>         cgi_module (static)
>>         negotiation_module (static)
>>         dir_module (static)
>>         actions_module (static)
>>         userdir_module (static)
>>         alias_module (static)
>>         rewrite_module (static)
>>         so_module (static)
>>         php5_module (shared)
>>         info_module (shared)
>> _______________________________________________
>> Mongrel-users mailing list
>> Mongrel-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>
>
>
> -- 
> Evan Weaver
> Cloudburst, LLC
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070729/fe997a29/attachment-0001.html

On Sun, 29 Jul 2007 19:23:06 -0700
Pete DeLaurentis <pete at nextengine.com> wrote:
> PS. Does anyone know if Zed has any future bug-fix releases or  
> updates planned?  I believe the last release was January, right?
Nope, Zed doesn''t.

You should run strace on your "dead" processes and gdb.  If I can find
the time I''ll do a write-up, but what you''re looking for is
Ruby cycling over one or many open files.  These will usually be something like
memcache, database, or some other external service.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

It could also be a log rotation issue. Check to see if it''s happening
at the same time every night.

If so, search the archives, as it''s come up over and over.

On 7/29/07, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
> On Sun, 29 Jul 2007 19:23:06 -0700
> Pete DeLaurentis <pete at nextengine.com> wrote:
>
> > PS. Does anyone know if Zed has any future bug-fix releases or
> > updates planned?  I believe the last release was January, right?
>
> Nope, Zed doesn''t.
>
> You should run strace on your "dead" processes and gdb.  If I can
find the time I''ll do a write-up, but what you''re looking for
is Ruby cycling over one or many open files.  These will usually be something
like memcache, database, or some other external service.
>
> --
> Zed A. Shaw
> - Hate: http://savingtheinternetwithhate.com/
> - Good: http://www.zedshaw.com/
> - Evil: http://yearofevil.com/
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

make sure you install the ruby-mysql gem.  We had the same problem, and that
fixed it.

--Brian

On 7/29/07, Joey Geiger <jgeiger at gmail.com>
wrote:
>
> It could also be a log rotation issue. Check to see if it''s
happening
> at the same time every night.
>
> If so, search the archives, as it''s come up over and over.
>
> On 7/29/07, Zed A. Shaw <zedshaw at zedshaw.com> wrote:
> > On Sun, 29 Jul 2007 19:23:06 -0700
> > Pete DeLaurentis <pete at nextengine.com> wrote:
> >
> > > PS. Does anyone know if Zed has any future bug-fix releases or
> > > updates planned?  I believe the last release was January, right?
> >
> > Nope, Zed doesn''t.
> >
> > You should run strace on your "dead" processes and gdb.  If
I can find
> the time I''ll do a write-up, but what you''re looking for
is Ruby cycling
> over one or many open files.  These will usually be something like
memcache,
> database, or some other external service.
> >
> > --
> > Zed A. Shaw
> > - Hate: http://savingtheinternetwithhate.com/
> > - Good: http://www.zedshaw.com/
> > - Evil: http://yearofevil.com/
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070729/5aa0a002/attachment.html

I have same problem with Oracle DB. I am using nagios to ping all mongrel
processes every 15 minutes as a temp. solution.

On 7/30/07, Brian Williams <bwillenator at gmail.com>
wrote:
>
> make sure you install the ruby-mysql gem.  We had the same problem, and
> that fixed it.
>
> --Brian
>
> On 7/29/07, Joey Geiger < jgeiger at gmail.com> wrote:
> >
> > It could also be a log rotation issue. Check to see if it''s
happening
> > at the same time every night.
> >
> > If so, search the archives, as it''s come up over and over.
> >
> > On 7/29/07, Zed A. Shaw <zedshaw at zedshaw.com> wrote:
> > > On Sun, 29 Jul 2007 19:23:06 -0700
> > > Pete DeLaurentis <pete at nextengine.com> wrote:
> > >
> > > > PS. Does anyone know if Zed has any future bug-fix releases
or
> > > > updates planned?  I believe the last release was January,
right?
> > >
> > > Nope, Zed doesn''t.
> > >
> > > You should run strace on your "dead" processes and gdb.
If I can find
> > the time I''ll do a write-up, but what you''re looking
for is Ruby cycling
> > over one or many open files.  These will usually be something like
memcache,
> > database, or some other external service.
> > >
> > > --
> > > Zed A. Shaw
> > > - Hate: http://savingtheinternetwithhate.com/
> > > - Good: http://www.zedshaw.com/
> > > - Evil: http://yearofevil.com/
> > > _______________________________________________
> > > Mongrel-users mailing list
> > > Mongrel-users at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/mongrel-users
> > >
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070730/e02e99b4/attachment.html

Check two things:

- Database connection might be getting closed by the DB server (we''ve 
had this exact problem with MySQL)
- Logs might be getting rotated (although you''d probably get a
different
error if this were the case)


Olly Lylo wrote:
> Hi
> I posted this to the Ruby on Rails Talk group but I thought I''d
post it here
> too as it''s probably a more appropriate group.  Hope this is ok.
>
> I''m running a 2-instance Mongrel cluster behind Apache 2.2.4 with
Rails
> 1.2.3.  If no requests are received by the application for several hours
> (this usually happens overnight) then Mongrel stops
> responding and no requests are detected by Rails (at least nothing is in
the
> Rails log).  Nothing untoward is in the Mongrel log.
>
> If you try and visit the application, the request times out with a 502
Proxy
> Error.  Apache is still up and running and serving static files without a
> problem, but the following is present in the Apache error log (from trying
> to hit the SessionController, identified by / sessions) which may provide a
> clue:
>
>     [Sun Jul 29 08:37:36 2007] [error] proxy: error reading status line
from
> remote server 127.0.0.1
>     [Sun Jul 29 08:37:36 2007] [error] proxy: Error reading from remote
> server returned by /sessions
>
> Restarting the Mongrel cluster resolves the problem until the next time it
> happens.  I have done a lot of reading about this issue online and a number
> of sources -- including the Mongrel FAQ -- point to being able to fix a
> ''hanging mongrel'' situation by setting this value in
environment.rb:
>
>     ActiveRecord::Base.verification_timeout = 14400
>
> This make the ActiveRecord timeout value less than the MySQL default of
> 28800.  I have made this change and it doesn''t appear to make a
difference.
>
> Can anyone advise me on what to try next to diagnose this issue?
>  I''m quickly running out of ideas and I''d appreciate a
bit of help!
>
> Here are the vitals from our Ubuntu 6.0.6 server:
>
>     MySQL
>         Ver 14.12 Distrib 5.0.22, for pc-linux-gnu (x86_64) using readline
> 5.1
>         connect_timeout                 5
>         interactive_timeout             28800
>         max_connect_errors           10
>         max_connections               100
>         max_user_connections       0
>         wait_timeout                      28800
>
>     ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]
>         cgi_multipart_eof_fix (2.2)
>         fastthread (1.0)
>         mongrel (1.0.1)
>         mongrel_cluster (1.0.2)
>
>     Apache/2.2.4 (Unix)
>
>     Mongrel Cluster Config:
>         port: "8000"
>         environment: production
>         address: 127.0.0.1
>         pid_file: log/mongrel.pid
>         servers: 2
>         user: [the user]
>         group: [the group]
>
>     Apache Load Balancer setup:
>         <Proxy balancer://appname_cluster>
>         BalancerMember http://127.0.0.1:8000
>         BalancerMember http://127.0.0.1:8001
>         </Proxy>
>
>     Apache Loaded Modules:
>         core_module (static)
>         authn_file_module (static)
>         authn_default_module (static)
>         authz_host_module (static)
>         authz_groupfile_module (static)
>         authz_user_module (static)
>         authz_default_module (static)
>         auth_basic_module (static)
>         cache_module (static)
>         include_module (static)
>         filter_module (static)
>         deflate_module (static)
>         log_config_module (static)
>         env_module (static)
>         headers_module (static)
>         setenvif_module (static)
>         proxy_module (static)
>         proxy_connect_module (static)
>         proxy_ftp_module (static)
>         proxy_http_module (static)
>         proxy_ajp_module (static)
>         proxy_balancer_module (static)
>         ssl_module (static)
>         mpm_prefork_module (static)
>         http_module (static)
>         mime_module (static)
>         status_module (static)
>         autoindex_module (static)
>         asis_module (static)
>         cgi_module (static)
>         negotiation_module (static)
>         dir_module (static)
>         actions_module (static)
>         userdir_module (static)
>         alias_module (static)
>         rewrite_module (static)
>         so_module (static)
>         php5_module (shared)
>         info_module (shared)
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users


This e-mail message is privileged, confidential and subject to copyright. Any
unauthorized use or disclosure is prohibited.
Le contenu du pr''esent courriel est privil''egi''e,
confidentiel et soumis `a des droits d''auteur. Il est interdit de
l''utiliser ou de le divulguer sans autorisation.

had there been a error, things would have been very easier then :P

On 7/30/07, Matt Zukowski <mzukowski at urbacon.net>
wrote:
>
> Check two things:
>
> - Database connection might be getting closed by the DB server
(we''ve
> had this exact problem with MySQL)
> - Logs might be getting rotated (although you''d probably get a
different
> error if this were the case)
>
>
> Olly Lylo wrote:
> > Hi
> > I posted this to the Ruby on Rails Talk group but I thought
I''d post it
> here
> > too as it''s probably a more appropriate group.  Hope this is
ok.
> >
> > I''m running a 2-instance Mongrel cluster behind Apache 2.2.4
with Rails
> > 1.2.3.  If no requests are received by the application for several
hours
> > (this usually happens overnight) then Mongrel stops
> > responding and no requests are detected by Rails (at least nothing is
in
> the
> > Rails log).  Nothing untoward is in the Mongrel log.
> >
> > If you try and visit the application, the request times out with a 502
> Proxy
> > Error.  Apache is still up and running and serving static files
without
> a
> > problem, but the following is present in the Apache error log (from
> trying
> > to hit the SessionController, identified by / sessions) which may
> provide a
> > clue:
> >
> >     [Sun Jul 29 08:37:36 2007] [error] proxy: error reading status
line
> from
> > remote server 127.0.0.1
> >     [Sun Jul 29 08:37:36 2007] [error] proxy: Error reading from
remote
> > server returned by /sessions
> >
> > Restarting the Mongrel cluster resolves the problem until the next
time
> it
> > happens.  I have done a lot of reading about this issue online and a
> number
> > of sources -- including the Mongrel FAQ -- point to being able to fix
a
> > ''hanging mongrel'' situation by setting this value in
environment.rb:
> >
> >     ActiveRecord::Base.verification_timeout = 14400
> >
> > This make the ActiveRecord timeout value less than the MySQL default
of
> > 28800.  I have made this change and it doesn''t appear to make
a
> difference.
> >
> > Can anyone advise me on what to try next to diagnose this issue?
> >  I''m quickly running out of ideas and I''d appreciate
a bit of help!
> >
> > Here are the vitals from our Ubuntu 6.0.6 server:
> >
> >     MySQL
> >         Ver 14.12 Distrib 5.0.22, for pc-linux-gnu (x86_64) using
> readline
> > 5.1
> >         connect_timeout                 5
> >         interactive_timeout             28800
> >         max_connect_errors           10
> >         max_connections               100
> >         max_user_connections       0
> >         wait_timeout                      28800
> >
> >     ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]
> >         cgi_multipart_eof_fix (2.2)
> >         fastthread (1.0)
> >         mongrel (1.0.1)
> >         mongrel_cluster (1.0.2)
> >
> >     Apache/2.2.4 (Unix)
> >
> >     Mongrel Cluster Config:
> >         port: "8000"
> >         environment: production
> >         address: 127.0.0.1
> >         pid_file: log/mongrel.pid
> >         servers: 2
> >         user: [the user]
> >         group: [the group]
> >
> >     Apache Load Balancer setup:
> >         <Proxy balancer://appname_cluster>
> >         BalancerMember http://127.0.0.1:8000
> >         BalancerMember http://127.0.0.1:8001
> >         </Proxy>
> >
> >     Apache Loaded Modules:
> >         core_module (static)
> >         authn_file_module (static)
> >         authn_default_module (static)
> >         authz_host_module (static)
> >         authz_groupfile_module (static)
> >         authz_user_module (static)
> >         authz_default_module (static)
> >         auth_basic_module (static)
> >         cache_module (static)
> >         include_module (static)
> >         filter_module (static)
> >         deflate_module (static)
> >         log_config_module (static)
> >         env_module (static)
> >         headers_module (static)
> >         setenvif_module (static)
> >         proxy_module (static)
> >         proxy_connect_module (static)
> >         proxy_ftp_module (static)
> >         proxy_http_module (static)
> >         proxy_ajp_module (static)
> >         proxy_balancer_module (static)
> >         ssl_module (static)
> >         mpm_prefork_module (static)
> >         http_module (static)
> >         mime_module (static)
> >         status_module (static)
> >         autoindex_module (static)
> >         asis_module (static)
> >         cgi_module (static)
> >         negotiation_module (static)
> >         dir_module (static)
> >         actions_module (static)
> >         userdir_module (static)
> >         alias_module (static)
> >         rewrite_module (static)
> >         so_module (static)
> >         php5_module (shared)
> >         info_module (shared)
> >
> >
> >
------------------------------------------------------------------------
> >
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
>
>
>
> This e-mail message is privileged, confidential and subject to copyright.
> Any unauthorized use or disclosure is prohibited.
> Le contenu du pr''esent courriel est
privil''egi''e, confidentiel et soumis
> `a des droits d''auteur. Il est interdit de l''utiliser ou
de le divulguer
> sans autorisation.
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070731/d557e7e4/attachment.html

Thanks for the tip.  We''re not using log rotation and have already  
tried adjusting the activerecord timeout.

One thing not tried yet: the mysql gem

-Pete



On Jul 30, 2007, at 11:57 PM, Piyush Ranjan wrote:
> had there been a error, things would have been very easier then :P
>
> On 7/30/07, Matt Zukowski <mzukowski at urbacon.net > wrote:
> Check two things:
>
> - Database connection might be getting closed by the DB server
(we''ve
> had this exact problem with MySQL)
> - Logs might be getting rotated (although you''d probably get a  
> different
> error if this were the case)
>
>
> Olly Lylo wrote:
> > Hi
> > I posted this to the Ruby on Rails Talk group but I thought
I''d
> post it here
> > too as it''s probably a more appropriate group.  Hope this is
ok.
> >
> > I''m running a 2-instance Mongrel cluster behind Apache 2.2.4
with
> Rails
> > 1.2.3.  If no requests are received by the application for  
> several hours
> > (this usually happens overnight) then Mongrel stops
> > responding and no requests are detected by Rails (at least  
> nothing is in the
> > Rails log).  Nothing untoward is in the Mongrel log.
> >
> > If you try and visit the application, the request times out with  
> a 502 Proxy
> > Error.  Apache is still up and running and serving static files  
> without a
> > problem, but the following is present in the Apache error log  
> (from trying
> > to hit the SessionController, identified by / sessions) which may  
> provide a
> > clue:
> >
> >     [Sun Jul 29 08:37:36 2007] [error] proxy: error reading  
> status line from
> > remote server 127.0.0.1
> >     [Sun Jul 29 08:37:36 2007] [error] proxy: Error reading from  
> remote
> > server returned by /sessions
> >
> > Restarting the Mongrel cluster resolves the problem until the  
> next time it
> > happens.  I have done a lot of reading about this issue online  
> and a number
> > of sources -- including the Mongrel FAQ -- point to being able to  
> fix a
> > ''hanging mongrel'' situation by setting this value in
environment.rb:
> >
> >     ActiveRecord::Base.verification_timeout = 14400
> >
> > This make the ActiveRecord timeout value less than the MySQL  
> default of
> > 28800.  I have made this change and it doesn''t appear to make
a
> difference.
> >
> > Can anyone advise me on what to try next to diagnose this issue?
> >  I''m quickly running out of ideas and I''d appreciate
a bit of help!
> >
> > Here are the vitals from our Ubuntu 6.0.6 server:
> >
> >     MySQL
> >         Ver 14.12 Distrib 5.0.22, for pc-linux-gnu (x86_64) using  
> readline
> > 5.1
> >         connect_timeout                 5
> >         interactive_timeout             28800
> >         max_connect_errors           10
> >         max_connections               100
> >         max_user_connections       0
> >         wait_timeout                      28800
> >
> >     ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]
> >         cgi_multipart_eof_fix (2.2)
> >         fastthread (1.0)
> >         mongrel (1.0.1)
> >         mongrel_cluster (1.0.2)
> >
> >     Apache/2.2.4 (Unix)
> >
> >     Mongrel Cluster Config:
> >         port: "8000"
> >         environment: production
> >         address: 127.0.0.1
> >         pid_file: log/mongrel.pid
> >         servers: 2
> >         user: [the user]
> >         group: [the group]
> >
> >     Apache Load Balancer setup:
> >         <Proxy balancer://appname_cluster>
> >         BalancerMember http://127.0.0.1:8000
> >         BalancerMember http://127.0.0.1:8001
> >         </Proxy>
> >
> >     Apache Loaded Modules:
> >         core_module (static)
> >         authn_file_module (static)
> >         authn_default_module (static)
> >         authz_host_module (static)
> >         authz_groupfile_module (static)
> >         authz_user_module (static)
> >         authz_default_module (static)
> >         auth_basic_module (static)
> >         cache_module (static)
> >         include_module (static)
> >         filter_module (static)
> >         deflate_module (static)
> >         log_config_module (static)
> >         env_module (static)
> >         headers_module (static)
> >         setenvif_module (static)
> >         proxy_module (static)
> >         proxy_connect_module (static)
> >         proxy_ftp_module (static)
> >         proxy_http_module (static)
> >         proxy_ajp_module (static)
> >         proxy_balancer_module (static)
> >         ssl_module (static)
> >         mpm_prefork_module (static)
> >         http_module (static)
> >         mime_module (static)
> >         status_module (static)
> >         autoindex_module (static)
> >         asis_module (static)
> >         cgi_module (static)
> >         negotiation_module (static)
> >         dir_module (static)
> >         actions_module (static)
> >         userdir_module (static)
> >         alias_module (static)
> >         rewrite_module (static)
> >         so_module (static)
> >         php5_module (shared)
> >         info_module (shared)
> >
> >
> >  
> ---------------------------------------------------------------------- 
> --
> >
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
>
>
>
> This e-mail message is privileged, confidential and subject to  
> copyright. Any unauthorized use or disclosure is prohibited.
> Le contenu du pr''esent courriel est
privil''egi''e, confidentiel et
> soumis `a des droits d''auteur. Il est interdit de
l''utiliser ou de
> le divulguer sans autorisation.
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070731/773437db/attachment.html

Some have reported that changing the ActiveRecord verification_timeout 
value doesn''t help with this problem. Try increasing the 
interactive_timeout and/or wait_timeout in your MySQL server and see if 
that helps.

Pete DeLaurentis wrote:
> Thanks for the tip.  We''re not using log rotation and have already
> tried adjusting the activerecord timeout.
>
> One thing not tried yet: the mysql gem
>
> -Pete
>
>
>
> On Jul 30, 2007, at 11:57 PM, Piyush Ranjan wrote:
>
>> had there been a error, things would have been very easier then :P
>>
>> On 7/30/07, Matt Zukowski <mzukowski at urbacon.net > wrote:
>> Check two things:
>>
>> - Database connection might be getting closed by the DB server
(we''ve
>> had this exact problem with MySQL)
>> - Logs might be getting rotated (although you''d probably get a
different
>> error if this were the case)
>>
>>
>> Olly Lylo wrote:
>> > Hi
>> > I posted this to the Ruby on Rails Talk group but I thought
I''d
>> post it here
>> > too as it''s probably a more appropriate group.  Hope this
is ok.
>> >
>> > I''m running a 2-instance Mongrel cluster behind Apache
2.2.4 with
>> Rails
>> > 1.2.3.  If no requests are received by the application for several
>> hours
>> > (this usually happens overnight) then Mongrel stops
>> > responding and no requests are detected by Rails (at least nothing
>> is in the
>> > Rails log).  Nothing untoward is in the Mongrel log.
>> >
>> > If you try and visit the application, the request times out with a
>> 502 Proxy
>> > Error.  Apache is still up and running and serving static files 
>> without a
>> > problem, but the following is present in the Apache error log
(from
>> trying
>> > to hit the SessionController, identified by / sessions) which may 
>> provide a
>> > clue:
>> >
>> >     [Sun Jul 29 08:37:36 2007] [error] proxy: error reading status
>> line from
>> > remote server 127.0.0.1
>> >     [Sun Jul 29 08:37:36 2007] [error] proxy: Error reading from 
>> remote
>> > server returned by /sessions
>> >
>> > Restarting the Mongrel cluster resolves the problem until the next
>> time it
>> > happens.  I have done a lot of reading about this issue online and
>> a number
>> > of sources -- including the Mongrel FAQ -- point to being able to 
>> fix a
>> > ''hanging mongrel'' situation by setting this
value in environment.rb:
>> >
>> >     ActiveRecord::Base.verification_timeout = 14400
>> >
>> > This make the ActiveRecord timeout value less than the MySQL 
>> default of
>> > 28800.  I have made this change and it doesn''t appear to
make a
>> difference.
>> >
>> > Can anyone advise me on what to try next to diagnose this issue?
>> >  I''m quickly running out of ideas and I''d
appreciate a bit of help!
>> >
>> > Here are the vitals from our Ubuntu 6.0.6 server:
>> >
>> >     MySQL
>> >         Ver 14.12 Distrib 5.0.22, for pc-linux-gnu (x86_64) using 
>> readline
>> > 5.1
>> >         connect_timeout                 5
>> >         interactive_timeout             28800
>> >         max_connect_errors           10
>> >         max_connections               100
>> >         max_user_connections       0
>> >         wait_timeout                      28800
>> >
>> >     ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]
>> >         cgi_multipart_eof_fix (2.2)
>> >         fastthread (1.0)
>> >         mongrel (1.0.1)
>> >         mongrel_cluster (1.0.2)
>> >
>> >     Apache/2.2.4 (Unix)
>> >
>> >     Mongrel Cluster Config:
>> >         port: "8000"
>> >         environment: production
>> >         address: 127.0.0.1
>> >         pid_file: log/mongrel.pid
>> >         servers: 2
>> >         user: [the user]
>> >         group: [the group]
>> >
>> >     Apache Load Balancer setup:
>> >         <Proxy balancer://appname_cluster>
>> >         BalancerMember http://127.0.0.1:8000
>> >         BalancerMember http://127.0.0.1:8001
>> >         </Proxy>
>> >
>> >     Apache Loaded Modules:
>> >         core_module (static)
>> >         authn_file_module (static)
>> >         authn_default_module (static)
>> >         authz_host_module (static)
>> >         authz_groupfile_module (static)
>> >         authz_user_module (static)
>> >         authz_default_module (static)
>> >         auth_basic_module (static)
>> >         cache_module (static)
>> >         include_module (static)
>> >         filter_module (static)
>> >         deflate_module (static)
>> >         log_config_module (static)
>> >         env_module (static)
>> >         headers_module (static)
>> >         setenvif_module (static)
>> >         proxy_module (static)
>> >         proxy_connect_module (static)
>> >         proxy_ftp_module (static)
>> >         proxy_http_module (static)
>> >         proxy_ajp_module (static)
>> >         proxy_balancer_module (static)
>> >         ssl_module (static)
>> >         mpm_prefork_module (static)
>> >         http_module (static)
>> >         mime_module (static)
>> >         status_module (static)
>> >         autoindex_module (static)
>> >         asis_module (static)
>> >         cgi_module (static)
>> >         negotiation_module (static)
>> >         dir_module (static)
>> >         actions_module (static)
>> >         userdir_module (static)
>> >         alias_module (static)
>> >         rewrite_module (static)
>> >         so_module (static)
>> >         php5_module (shared)
>> >         info_module (shared)
>> >
>> >
>> > 
>>
------------------------------------------------------------------------
>> >
>> > _______________________________________________
>> > Mongrel-users mailing list
>> > Mongrel-users at rubyforge.org
>> > http://rubyforge.org/mailman/listinfo/mongrel-users
>>
>>
>>
>> This e-mail message is privileged, confidential and subject to 
>> copyright. Any unauthorized use or disclosure is prohibited.
>> Le contenu du pr''esent courriel est
privil''egi''e, confidentiel et
>> soumis `a des droits d''auteur. Il est interdit de
l''utiliser ou de le
>> divulguer sans autorisation.
>>
>>
>> _______________________________________________
>> Mongrel-users mailing list
>> Mongrel-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

On Jul 31, 2007, at 14:37 , Pete DeLaurentis wrote:
> Thanks for the tip.  We''re not using log rotation and have already
> tried adjusting the activerecord timeout.
>
> One thing not tried yet: the mysql gem
>
> -Pete
Pete,

We found that we saw this issue would arise when the MySQL gem was  
either uninstalled or not installed properly (eg make failed).

--
Wayne E. Seguin
Sr. Systems Architect & Systems Admin
wayneseguin at gmail.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070731/007cf897/attachment-0001.html

Just a suggestion. I''d get some kind of log rotation going sooner than
later.

On 7/31/07, Wayne E. Seguin <wayneeseguin at gmail.com>
wrote:
>
>
> On Jul 31, 2007, at 14:37 , Pete DeLaurentis wrote:
>
> Thanks for the tip.  We''re not using log rotation and have already
tried
> adjusting the activerecord timeout.
>
> One thing not tried yet: the mysql gem
>
> -Pete
>
> Pete,
>
> We found that we saw this issue would arise when the MySQL gem was either
> uninstalled or not installed properly (eg make failed).
>
> --
> Wayne E. Seguin
> Sr. Systems Architect & Systems Admin
> wayneseguin at gmail.com
>
>
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

On Sun, 29 Jul 2007 22:57:29 +0100
"Olly Lylo" <list at lylo.co.uk> wrote:
> Hi
> I posted this to the Ruby on Rails Talk group but I thought I''d
post it here
> too as it''s probably a more appropriate group.  Hope this is ok.

Alright, I''ve got a few minutes to write this down since it seems to be
biting a few people.

Here''s the rules for resolving a "RAILS in Mongrel dies"
problem (notice the RAILS part?):

1) 1000''s of sites run mongrel without this problem so look to what you
have installed first.
2) Make sure that every single bit of software you are running is the most
recent version and everything is installed and used correctly.  Common culprits
are:
  a) MySQL -- install the gem manually and make sure that the most recent is the
only one.  DO NOT USE THE RAILS DEFAULT.
  b) Memcached -- Use the very latest from Eric Hodel''s project and do
NOT put any keys in that have spaces or null (\0) chars.  Yes, your keys cannot
have spaces.  YES YOUR KEYS CANNOT HAVE SPACES.
  c) net::http to some web site.  This can cycle forever.
3) Next, once you''ve made sure all the above is isolated then you can
proceed to stage 2.

STAGE 2

1) You must let your application run in the best configuration and be ready to
pounce on it the second a mongrel dies.
2) Log in to your server and find out the PID file of the mongrel that
isn''t responding.  You do this by hitting the process on it''s
actual port (like 8000, 8001, 8002, NOT apache/nginx''s port of 80).
3) Once you know what port it is, then use: sudo lsof -i -P | grep PORT to find
what process PID is on that port.
4) Next, attach to this process with: sudo strace -p PID.  What you''ll
see in a healthy mongrel is lots of variation.  What you''ll see in a
dead mongrel is probably either a bunch of calls to select/poll for the exact
same filedescriptors, or nothing.
5) If you see it doing a select on the file descriptors, then you need to find
out what is on that FD that is causing it to wait.  Again, use: lsof | grep
FILEDESCR
  a) This will potentially tell you where it''s connected, etc.  Once
you do this you know what is being read/written and can go find out what in your
rails app is using it, and apply even more debugging tools.
  b) You can also just force it closed externally.  There''s a few
mentions of this but I don''t remember the exact procedure.
  c) The reason this typically happens is that you have a socket that ruby has
written to and not read from yet, but that socket is closed.  Another cause is
that you are simply waiting for data (which is what happens with memcached and
putting a space in your keys).
6) If your mongrel is completely dead then move on to stage 3 and also
you''re kind of fucked.

STAGE 3

1) Get your good shoes on because you''re now in GDB land and C.
2) http://eigenclass.org/hiki.rb?ruby+live+process+introspection  Explains
attaching to your mongrel process (you''ve identified above) using GDB,
loading some GDB scripts, and then stopping it, inspecting it, and forcing an
exception.
3) Do those thigns.  Attach.  Pause it. Inspect variables.  Get stack traces.
Force an exception.  See where it''s stopped.  Look for where OUTSIDE of
mongrel it''s coming from.
4) That''s all I can say right now.  Let''s hope other people
can expand on this.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

I thought I''d follow up on my progress with this thread.  I
haven''t seen the
problem now for a couple of days so (fingers crossed) my problem appears to
be fixed.
Firstly, I DO have logrotate scheduled to run daily using cron but I
didn''t
touch this, so I don''t believe this is the problem in my case.


I initiall tried setting the following in environment.rb

        ActiveRecord::Base.verification_timeout = 14400

This did NOT solve my problem, but is probably a change which should be made
anyway.

I then made two significant changes:


1. I didn''t have the MySQL gem installed, so I have installed this
(version
2.7).

2. I made the following change to my Apache Virtual Host setting to prevent
Apache from losing the connection to the proxy (according to
http://httpd.apache.org/docs/2.0/mod/mod_proxy.html)

        SetEnv force-proxy-request-1.0 1
        SetEnv proxy-nokeepalive 1


After making these two changes and restarting MySQL, Apache and the Mongrel
Cluster, the problem hasn''t reoccurred.

Hope this helps.



Cheers, Olly
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070801/8dab6287/attachment.html

Hi
Thank you for your suggestions.
I have a few sites running on mongrel. Only one of them has this "sleep
disorder". I  later realized that due to constant load (throughout the day)
on other sites they were working fine whereas this forum had load only
during the day and hence did die overnight. I have inspected the PIDs and
found that mongrel remain in the sleeping mode even after repeated
connection request. For now I have a nagios looking after it(pinging it
every 15 mins).
I must point out here that the DB here is Oracle and mysql is not in the
picture at all


On 8/1/07, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
>
> On Sun, 29 Jul 2007 22:57:29 +0100
> "Olly Lylo" <list at lylo.co.uk> wrote:
>
> > Hi
> > I posted this to the Ruby on Rails Talk group but I thought
I''d post it
> here
> > too as it''s probably a more appropriate group.  Hope this is
ok.
>
>
> Alright, I''ve got a few minutes to write this down since it seems
to be
> biting a few people.
>
> Here''s the rules for resolving a "RAILS in Mongrel dies"
problem (notice
> the RAILS part?):
>
> 1) 1000''s of sites run mongrel without this problem so look to
what you
> have installed first.
> 2) Make sure that every single bit of software you are running is the most
> recent version and everything is installed and used correctly.  Common
> culprits are:
>   a) MySQL -- install the gem manually and make sure that the most recent
> is the only one.  DO NOT USE THE RAILS DEFAULT.
>   b) Memcached -- Use the very latest from Eric Hodel''s project
and do NOT
> put any keys in that have spaces or null (\0) chars.  Yes, your keys cannot
> have spaces.  YES YOUR KEYS CANNOT HAVE SPACES.
>   c) net::http to some web site.  This can cycle forever.
> 3) Next, once you''ve made sure all the above is isolated then you
can
> proceed to stage 2.
>
> STAGE 2
>
> 1) You must let your application run in the best configuration and be
> ready to pounce on it the second a mongrel dies.
> 2) Log in to your server and find out the PID file of the mongrel that
> isn''t responding.  You do this by hitting the process on
it''s actual port
> (like 8000, 8001, 8002, NOT apache/nginx''s port of 80).
> 3) Once you know what port it is, then use: sudo lsof -i -P | grep PORT to
> find what process PID is on that port.
> 4) Next, attach to this process with: sudo strace -p PID.  What
you''ll see
> in a healthy mongrel is lots of variation.  What you''ll see in a
dead
> mongrel is probably either a bunch of calls to select/poll for the exact
> same filedescriptors, or nothing.
> 5) If you see it doing a select on the file descriptors, then you need to
> find out what is on that FD that is causing it to wait.  Again, use: lsof |
> grep FILEDESCR
>   a) This will potentially tell you where it''s connected, etc. 
Once you
> do this you know what is being read/written and can go find out what in
your
> rails app is using it, and apply even more debugging tools.
>   b) You can also just force it closed externally.  There''s a few
mentions
> of this but I don''t remember the exact procedure.
>   c) The reason this typically happens is that you have a socket that ruby
> has written to and not read from yet, but that socket is closed.  Another
> cause is that you are simply waiting for data (which is what happens with
> memcached and putting a space in your keys).
> 6) If your mongrel is completely dead then move on to stage 3 and also
> you''re kind of fucked.
>
> STAGE 3
>
> 1) Get your good shoes on because you''re now in GDB land and C.
> 2) http://eigenclass.org/hiki.rb?ruby+live+process+introspection  Explains
> attaching to your mongrel process (you''ve identified above) using
GDB,
> loading some GDB scripts, and then stopping it, inspecting it, and forcing
> an exception.
> 3) Do those thigns.  Attach.  Pause it. Inspect variables.  Get stack
> traces. Force an exception.  See where it''s stopped.  Look for
where OUTSIDE
> of mongrel it''s coming from.
> 4) That''s all I can say right now.  Let''s hope other
people can expand on
> this.
>
> --
> Zed A. Shaw
> - Hate: http://savingtheinternetwithhate.com/
> - Good: http://www.zedshaw.com/
> - Evil: http://yearofevil.com/
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070801/6b73793c/attachment-0001.html

I don''t know if this contributes much to the conversation, but I  
wonder if some of these occasional deadlocks are related to this:

http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-core/11756? 
11593-11836

Corey


On Jul 31, 2007, at 5:15 PM, Zed A. Shaw wrote:
> On Sun, 29 Jul 2007 22:57:29 +0100
> "Olly Lylo" <list at lylo.co.uk> wrote:
>
>> Hi
>> I posted this to the Ruby on Rails Talk group but I thought
I''d
>> post it here
>> too as it''s probably a more appropriate group.  Hope this is
ok.
>
>
> Alright, I''ve got a few minutes to write this down since it seems
> to be biting a few people.
>
> Here''s the rules for resolving a "RAILS in Mongrel dies"
problem
> (notice the RAILS part?):
>
> 1) 1000''s of sites run mongrel without this problem so look to
what
> you have installed first.
> 2) Make sure that every single bit of software you are running is  
> the most recent version and everything is installed and used  
> correctly.  Common culprits are:
>   a) MySQL -- install the gem manually and make sure that the most  
> recent is the only one.  DO NOT USE THE RAILS DEFAULT.
>   b) Memcached -- Use the very latest from Eric Hodel''s project
and
> do NOT put any keys in that have spaces or null (\0) chars.  Yes,  
> your keys cannot have spaces.  YES YOUR KEYS CANNOT HAVE SPACES.
>   c) net::http to some web site.  This can cycle forever.
> 3) Next, once you''ve made sure all the above is isolated then you
> can proceed to stage 2.
>
> STAGE 2
>
> 1) You must let your application run in the best configuration and  
> be ready to pounce on it the second a mongrel dies.
> 2) Log in to your server and find out the PID file of the mongrel  
> that isn''t responding.  You do this by hitting the process on
it''s
> actual port (like 8000, 8001, 8002, NOT apache/nginx''s port of
80).
> 3) Once you know what port it is, then use: sudo lsof -i -P | grep  
> PORT to find what process PID is on that port.
> 4) Next, attach to this process with: sudo strace -p PID.  What  
> you''ll see in a healthy mongrel is lots of variation.  What
you''ll
> see in a dead mongrel is probably either a bunch of calls to select/ 
> poll for the exact same filedescriptors, or nothing.
> 5) If you see it doing a select on the file descriptors, then you  
> need to find out what is on that FD that is causing it to wait.   
> Again, use: lsof | grep FILEDESCR
>   a) This will potentially tell you where it''s connected, etc.   
> Once you do this you know what is being read/written and can go  
> find out what in your rails app is using it, and apply even more  
> debugging tools.
>   b) You can also just force it closed externally.  There''s a few
> mentions of this but I don''t remember the exact procedure.
>   c) The reason this typically happens is that you have a socket  
> that ruby has written to and not read from yet, but that socket is  
> closed.  Another cause is that you are simply waiting for data  
> (which is what happens with memcached and putting a space in your  
> keys).
> 6) If your mongrel is completely dead then move on to stage 3 and  
> also you''re kind of fucked.
>
> STAGE 3
>
> 1) Get your good shoes on because you''re now in GDB land and C.
> 2) http://eigenclass.org/hiki.rb?ruby+live+process+introspection   
> Explains attaching to your mongrel process (you''ve identified  
> above) using GDB, loading some GDB scripts, and then stopping it,  
> inspecting it, and forcing an exception.
> 3) Do those thigns.  Attach.  Pause it. Inspect variables.  Get  
> stack traces. Force an exception.  See where it''s stopped.  Look  
> for where OUTSIDE of mongrel it''s coming from.
> 4) That''s all I can say right now.  Let''s hope other
people can
> expand on this.
>
> -- 
> Zed A. Shaw
> - Hate: http://savingtheinternetwithhate.com/
> - Good: http://www.zedshaw.com/
> - Evil: http://yearofevil.com/
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users