Rails - Mar 2005 - dispatch.fcgi dieing randomly

Recently my application has refused to start on my production
environment.  The setup is debian(unstable)/apache2/fastcgi/mysql
Rails 0.10 (upgrading to 0.10.1 as we speak)

When I first start up apache the machine grinds to a crawl starting up
the dispatch.cfi processes.  It''s a VPS with 256M of ram, but still
(in addition to user time) it''s spending 30% of time in system
resources on start up.  Once everything has started up, the machine
pretty much is at a 0.00 or 0.01 load time.  (Cool.)

But if I hit the site before everything has settled down, a 500 with
no further useful information.  Just waiting and retrying seem to make
things better, but it''s maybe a minute before anything works.  The
apache logs are discriptive thusly:

 FastCGI: incomplete headers (0 bytes) received from server
"/home/ws/myelinate/public/dispatch.fcgi"

and from /var/log/apache2/error.log
FastCGI: (dynamic) server "/home/ws/myelinate/public/dispatch.fcgi"
(pid 19296) terminated by calling exit with status ''0''

How do I figure out what''s going on so I can fix this?  Running
script/console production takes a second or 2 to start up, so I''m
don''t understand what''s killing dispatch.fcgi.

Thanks,
w

Have you tried increasing the time out for fastcgi in the httpd.conf file?
Also I notice you''re using mod_fcgi with Apache2, I''m probably
wrong as
things have maybe improved since I last checked, but I heard that 
mod_fcgi is unstable under Apache2.
On my Apache 2 setup I use mod_fcgid instead which works nicely without 
hiccups.

Hope this helps


Will Schenk wrote:
> Recently my application has refused to start on my production
> environment.  The setup is debian(unstable)/apache2/fastcgi/mysql
> Rails 0.10 (upgrading to 0.10.1 as we speak)
> 
> When I first start up apache the machine grinds to a crawl starting up
> the dispatch.cfi processes.  It''s a VPS with 256M of ram, but
still
> (in addition to user time) it''s spending 30% of time in system
> resources on start up.  Once everything has started up, the machine
> pretty much is at a 0.00 or 0.01 load time.  (Cool.)
> 
> But if I hit the site before everything has settled down, a 500 with
> no further useful information.  Just waiting and retrying seem to make
> things better, but it''s maybe a minute before anything works.  The
> apache logs are discriptive thusly:
> 
>  FastCGI: incomplete headers (0 bytes) received from server
> "/home/ws/myelinate/public/dispatch.fcgi"
> 
> and from /var/log/apache2/error.log
> FastCGI: (dynamic) server
"/home/ws/myelinate/public/dispatch.fcgi"
> (pid 19296) terminated by calling exit with status ''0''
> 
> How do I figure out what''s going on so I can fix this?  Running
> script/console production takes a second or 2 to start up, so I''m
> don''t understand what''s killing dispatch.fcgi.
> 
> Thanks,
> w

On Thu, 10 Mar 2005, Will Schenk wrote:
> Recently my application has refused to start on my production environment.
> The setup is debian(unstable)/apache2/fastcgi/mysql Rails 0.10 (upgrading
to
> 0.10.1 as we speak)
>
> When I first start up apache the machine grinds to a crawl starting up the
> dispatch.cfi processes.  It''s a VPS with 256M of ram, but still
(in addition
> to user time) it''s spending 30% of time in system resources on
start up.
> Once everything has started up, the machine pretty much is at a 0.00 or
0.01
> load time.  (Cool.)
>
> But if I hit the site before everything has settled down, a 500 with no
> further useful information.  Just waiting and retrying seem to make things
> better, but it''s maybe a minute before anything works.  The apache
logs are
> discriptive thusly:
>
> FastCGI: incomplete headers (0 bytes) received from server
> "/home/ws/myelinate/public/dispatch.fcgi"
>
> and from /var/log/apache2/error.log
> FastCGI: (dynamic) server
"/home/ws/myelinate/public/dispatch.fcgi"
> (pid 19296) terminated by calling exit with status ''0''
>
> How do I figure out what''s going on so I can fix this?  Running
> script/console production takes a second or 2 to start up, so I''m
don''t
> understand what''s killing dispatch.fcgi.
where is your fastcgi ipc dir?

-a
-- 
==============================================================================|
EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: 303.497.6469
| When you do something, you should burn yourself completely, like a good
| bonfire, leaving no trace of yourself.  --Shunryu Suzuki
===============================================================================

I had a similar problem which I fixed by increasing the default FastCGI 
timeout from the default 30 to 60 second -- this is in my httpd.conf file:

   FastCgiConfig -idle-timeout 60 -maxClassProcesses 5


Cheers, Stuart
-- 
Stuart Rackham
http://www.methods.co.nz/asciidoc/  -- Text based document generation


Will Schenk wrote:
> Recently my application has refused to start on my production
> environment.  The setup is debian(unstable)/apache2/fastcgi/mysql
> Rails 0.10 (upgrading to 0.10.1 as we speak)
> 
> When I first start up apache the machine grinds to a crawl starting up
> the dispatch.cfi processes.  It''s a VPS with 256M of ram, but
still
> (in addition to user time) it''s spending 30% of time in system
> resources on start up.  Once everything has started up, the machine
> pretty much is at a 0.00 or 0.01 load time.  (Cool.)
> 
> But if I hit the site before everything has settled down, a 500 with
> no further useful information.  Just waiting and retrying seem to make
> things better, but it''s maybe a minute before anything works.  The
> apache logs are discriptive thusly:
> 
>  FastCGI: incomplete headers (0 bytes) received from server
> "/home/ws/myelinate/public/dispatch.fcgi"
> 
> and from /var/log/apache2/error.log
> FastCGI: (dynamic) server
"/home/ws/myelinate/public/dispatch.fcgi"
> (pid 19296) terminated by calling exit with status ''0''
> 
> How do I figure out what''s going on so I can fix this?  Running
> script/console production takes a second or 2 to start up, so I''m
> don''t understand what''s killing dispatch.fcgi.
> 
> Thanks,
> w
> _______________________________________________
> Rails mailing list
> Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>

geeze. Put that to an hour or so. 


On Fri, 11 Mar 2005 08:22:29 +1300, Stuart Rackham
<srackham-yZTdQaUi3UlBWQWeTLFoew@public.gmane.org>
wrote:
> I had a similar problem which I fixed by increasing the default FastCGI
> timeout from the default 30 to 60 second -- this is in my httpd.conf file:

-- 
Tobi
http://www.snowdevil.ca - Snowboards that don''t suck
http://www.hieraki.org   - Open source book authoring
http://blog.leetsoft.com - Technical weblog

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mar 10, 2005, at 4:54 AM, Will Schenk wrote:
> Recently my application has refused to start on my production
> environment.  The setup is debian(unstable)/apache2/fastcgi/mysql
> Rails 0.10 (upgrading to 0.10.1 as we speak)
>
> and from /var/log/apache2/error.log
> FastCGI: (dynamic) server
"/home/ws/myelinate/public/dispatch.fcgi"
> (pid 19296) terminated by calling exit with status ''0''
>
Tobias and Stuart on the right track. A status 0 is a normal "error"
in
fastcgi because it''s the error sent out by the process manager when it 
kills off children before a timeout. For example, when the FastCGI 
process manager receives a SIGHUP, it sends a SIGTERM to each of its 
children, and then resumes its normal operations. When it receives a 
SIGTERM, it sends a SIGTERM to each of its children, sets an alarm(3) 
``die timeout'''' handler, and waits for each of its children to
die. If
all children die before this timeout, the process manager exits with 
return status 0. If all children do not die by the time the ``die 
timeout'''' occurs, the process manager sends a SIGKILL to each
of the
remaining children, and exists with return status 1.

So ... bump up your timeout to 600-1200 seconds.

The other thing you might think about is instead of using a dynamic 
fastcgi process, do it was an external server, write yourself a 
wrapper/spawner and start is up with sh. Then in Apache you''ll use the 
fastcgi external server directive (which remember is also not affected 
by fastcgiconfig).

And with 256MB RAM you might want to tune down Apache and MySQL (you''ll
at least notice it starting faster, just don''t get a lot of hits). The 
default prefork is StartServers 5, MinSpareServers 5 and 
MaxSpareServers 10, and these 5 that start up are 30-40MB each, add to 
that 125MB+ mysql process (for non threaded) and a single 20MB 
dispatch.fcgi and your OS has to spend sometime figure out how to 
accommodate these all.

Man, and I thought I was going to do a quick one liner back.

- - Jason

-----BEGIN PGP SIGNATURE-----
Version: PGP 8.1

iQA/AwUBQjKP9VUyB+ajXkCLEQKIXACcC9nNK527UOowvpn7s6HJjx9424gAoKhz
yH0HubX10obStAeDjkA97AfN
=G8SX
-----END PGP SIGNATURE-----