I''ve got a Rails application running at Dreamhost using Fast CGI. This was running more or less okay with 0.13.1. I''ve now upgraded to 0.14.2 and I''m having issues with lots of stale dispatch.fcgi processes getting left around. The issue appears to be caused by the parent Apache process being terminated. With ps, I see the parent PID change from what I guess must be the Apache process (Dreamhost''s security prevents me from seeing processes belonging to other users) to the PID 1. At the same time I see "asked to terminate ASAP" in the fastcgi.crash.log. According to fcgi_handler.rb, this must mean the process was sent a TERM signal. Once the process has been sent the TERM, it receives no further requests. As this is a new issue with 0.14.2 that I wasn''t having with 0.13.1, I''ve had a quick look in SVN to see whats changed in fcgi_handler. In the 0.13.1 version, TERM appears to be handled by terminating immediately if there is a request in progress. If the dispatcher is busy it waits until the request has finished and then terminates. In 0.14.2, the immediate termination if not busy behaviour has been removed and the dispatcher now always waits until the next request to terminate (http://dev.rubyonrails.com/changeset/1867). Since the processes are receiving a TERM and then never get passed any more requests, they are never terminating. I''m going to try reverting this change for now with my application to see if it makes any difference. I would be interested to know if anyone else was seeing similar problems or has a better solution though. Thanks, Phil
Hammed Malik
2005-Nov-02 00:04 UTC
Re: dispatch.fcgi processes not terminating with 0.14.2
I''ve noticed a similar problem on Textdrive after I upgraded to a new version of typo and edge rails. Hammed On 01/11/05, Philip Ross <phil.ross-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> > I''ve got a Rails application running at Dreamhost using Fast CGI. This > was running more or less okay with 0.13.1. I''ve now upgraded to 0.14.2 > and I''m having issues with lots of stale dispatch.fcgi processes getting > left around. > > The issue appears to be caused by the parent Apache process being > terminated. With ps, I see the parent PID change from what I guess must > be the Apache process (Dreamhost''s security prevents me from seeing > processes belonging to other users) to the PID 1. At the same time I see > "asked to terminate ASAP" in the fastcgi.crash.log. According to > fcgi_handler.rb, this must mean the process was sent a TERM signal. Once > the process has been sent the TERM, it receives no further requests. > > As this is a new issue with 0.14.2 that I wasn''t having with 0.13.1, > I''ve had a quick look in SVN to see whats changed in fcgi_handler. > > In the 0.13.1 version, TERM appears to be handled by terminating > immediately if there is a request in progress. If the dispatcher is busy > it waits until the request has finished and then terminates. > > In 0.14.2, the immediate termination if not busy behaviour has been > removed and the dispatcher now always waits until the next request to > terminate (http://dev.rubyonrails.com/changeset/1867). Since the > processes are receiving a TERM and then never get passed any more > requests, they are never terminating. > > I''m going to try reverting this change for now with my application to > see if it makes any difference. I would be interested to know if anyone > else was seeing similar problems or has a better solution though. > > Thanks, > > Phil > > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >_______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails
Philip Ross wrote:> In 0.14.2, the immediate termination if not busy behaviour has been > removed and the dispatcher now always waits until the next request to > terminate (http://dev.rubyonrails.com/changeset/1867). Since the > processes are receiving a TERM and then never get passed any more > requests, they are never terminating. > > I''m going to try reverting this change for now with my application to > see if it makes any difference. I would be interested to know if anyone > else was seeing similar problems or has a better solution though.I''ve now been running with this change reverted for almost a day. I''m no longer seeing any processes left hanging around after receiving a TERM signal. Phil
Jeremy Kemper
2005-Nov-02 21:07 UTC
Re: Re: dispatch.fcgi processes not terminating with 0.14.2
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Nov 2, 2005, at 12:47 PM, Philip Ross wrote:> Philip Ross wrote: >> In 0.14.2, the immediate termination if not busy behaviour has >> been removed and the dispatcher now always waits until the next >> request to terminate (http://dev.rubyonrails.com/changeset/1867). >> Since the processes are receiving a TERM and then never get passed >> any more requests, they are never terminating. >> I''m going to try reverting this change for now with my application >> to see if it makes any difference. I would be interested to know >> if anyone else was seeing similar problems or has a better >> solution though. > > I''ve now been running with this change reverted for almost a day. > I''m no longer seeing any processes left hanging around after > receiving a TERM signal.Try svn trunk. http://dev.rubyonrails.org/changeset/2847 Changeset 1867 was intentional: users should not see 500 errors when you upgrade your app. However, sending USR2 to the FastCGI instances you have spawned is the preferred method, so immediate exit on TERM is back in (may the sysadmins rejoice.) jeremy -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (Darwin) iD8DBQFDaSqmAQHALep9HFYRAhs8AKDSSZlA6OFcykhzmQf4kShxzne9jACg0o0c DVajwuvgZXMGbbKOELdYQF8=pm4t -----END PGP SIGNATURE-----
Brian V. Hughes
2005-Nov-02 21:10 UTC
Re: Re: dispatch.fcgi processes not terminating with 0.14.2
Philip Ross wrote:> Philip Ross wrote: >> In 0.14.2, the immediate termination if not busy behaviour has been >> removed and the dispatcher now always waits until the next request to >> terminate (http://dev.rubyonrails.com/changeset/1867). Since the >> processes are receiving a TERM and then never get passed any more >> requests, they are never terminating. >> >> I''m going to try reverting this change for now with my application to >> see if it makes any difference. I would be interested to know if >> anyone else was seeing similar problems or has a better solution though. > > I''ve now been running with this change reverted for almost a day. I''m no > longer seeing any processes left hanging around after receiving a TERM > signal.OK. I''m not sure I''m following exactly what you''ve been seeing, Phil. But it seemed similar to what I was seeing with 0.14.2 on my Tiger server, with latest Lighttpd. But I think my problem is more related to ./script/process/reaper not working. It doesn''t matter whether I run it as a normal user, or via sudo, it always says it can''t find any processes related to my app''s path: /Library/WebServer/Rails/myapp/public/dispatch.fcgi Even though the ps -ax report shows 3 processes: 3661 ?? S 1:53.84 /usr/local/bin/ruby /Library/Webserver/Rails/myapp/public/dispatch.fcgi 3662 ?? S 0:01.89 /usr/local/bin/ruby /Library/Webserver/Rails/myapp/public/dispatch.fcgi 3663 ?? S 5:05.35 /usr/local/bin/ruby /Library/Webserver/Rails/myapp/public/dispatch.fcgi Is what I''m seeing related to what you were seeing, Phil? Or, is there some thing that I''m just missing with reaper? I couldn''t find any examples outside of what''s in the -h info. -Brian