Rails - Apr 2006 - Runaway FCGI Processes with Debian, Apache

Hi all.

There is an issue with Rails, FastCGI and Apache2 on
Debian that I''ve been dealing with since I started
Rails development in late 2005.  I''ve developed
various workarounds but have not been able to solve
the problem at the root.

Every so often, for reasons I''ve never been able to
determine, a load of dispatch.fcgi processes get
spawned and take over my whole server, effectively
rendering the web site useless.  The only solution is
to manually kill the processes.

I''ve seen other posts about this issue but I''ve never
seen a *real* solution -- only workarounds like having
a script that periodically searches for these
dispatch.fcgi processes and kills them.

Our web site is currently only in a test or beta
stage, so the issue isn''t too bad -- but soon we''ll be
attracting a lot more traffic (I hope), and I don''t
feel comfortable not knowing why my server is killed
by dispatch.fcgi processes.

If anyone can provide information about this, I''ll
collate it and write it up properly and make it
accessible.  My questions are:

1) What''s the cause?  Is it a problem with the FCGI
module, Apache, or the Rails dispatcher?

2) Does the problem only exist for certain Rails
versions, or for Rails on Debian, or all Rails
versions on all *nix platforms?

3) If you''ve never experienced any problem like this,
and you use Apache 2 and FCGI with Rails, what version
of Rails do you use, and what *nix distro?

4) Finally, what''s a *real* solution to the problem? 
I''ll accept having to change to a different web
server, a different version of Rails, or a different
distro if I absolutely know it will solve my problem.


Many thanks in advance for any info.


Matt Jones

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

I''m running Debian 3.1 and had exactly the same issues. Apache2 and
mod_fastcgi doesn''t play well together. I''ve switched to
mod_fcgid and it
works fine so far.

Kent.

On 4/3/06, Matt Jones <matt_binary@yahoo.com>
wrote:
>
> Hi all.
>
> There is an issue with Rails, FastCGI and Apache2 on
> Debian that I''ve been dealing with since I started
> Rails development in late 2005.  I''ve developed
> various workarounds but have not been able to solve
> the problem at the root.
>
> Every so often, for reasons I''ve never been able to
> determine, a load of dispatch.fcgi processes get
> spawned and take over my whole server, effectively
> rendering the web site useless.  The only solution is
> to manually kill the processes.
>
> I''ve seen other posts about this issue but I''ve never
> seen a *real* solution -- only workarounds like having
> a script that periodically searches for these
> dispatch.fcgi processes and kills them.
>
> Our web site is currently only in a test or beta
> stage, so the issue isn''t too bad -- but soon we''ll be
> attracting a lot more traffic (I hope), and I don''t
> feel comfortable not knowing why my server is killed
> by dispatch.fcgi processes.
>
> If anyone can provide information about this, I''ll
> collate it and write it up properly and make it
> accessible.  My questions are:
>
> 1) What''s the cause?  Is it a problem with the FCGI
> module, Apache, or the Rails dispatcher?
>
> 2) Does the problem only exist for certain Rails
> versions, or for Rails on Debian, or all Rails
> versions on all *nix platforms?
>
> 3) If you''ve never experienced any problem like this,
> and you use Apache 2 and FCGI with Rails, what version
> of Rails do you use, and what *nix distro?
>
> 4) Finally, what''s a *real* solution to the problem?
> I''ll accept having to change to a different web
> server, a different version of Rails, or a different
> distro if I absolutely know it will solve my problem.
>
>
> Many thanks in advance for any info.
>
>
> Matt Jones
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>


--
Kent
---
http://www.datanoise.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060403/98255632/attachment-0001.html

Nice one.  I''ve changed to fcgid; I''ll let it run for
a few days and see how it fares.  If it works I''ll add
a note to the Rails wiki and other documentation.

Thanks

--- Kent Sibilev <ksruby@gmail.com> wrote:
> I''m running Debian 3.1 and had exactly the same
> issues. Apache2 and
> mod_fastcgi doesn''t play well together. I''ve
> switched to mod_fcgid and it
> works fine so far.
> 
> Kent.
> 
> On 4/3/06, Matt Jones <matt_binary@yahoo.com> wrote:
> >
> > Hi all.
> >
> > There is an issue with Rails, FastCGI and Apache2
> on
> > Debian that I''ve been dealing with since I started
> > Rails development in late 2005.  I''ve developed
> > various workarounds but have not been able to
> solve
> > the problem at the root.
> >
> > Every so often, for reasons I''ve never been able
> to
> > determine, a load of dispatch.fcgi processes get
> > spawned and take over my whole server, effectively
> > rendering the web site useless.  The only solution
> is
> > to manually kill the processes.
> >
> > I''ve seen other posts about this issue but I''ve
> never
> > seen a *real* solution -- only workarounds like
> having
> > a script that periodically searches for these
> > dispatch.fcgi processes and kills them.
> >
> > Our web site is currently only in a test or beta
> > stage, so the issue isn''t too bad -- but soon
> we''ll be
> > attracting a lot more traffic (I hope), and I
> don''t
> > feel comfortable not knowing why my server is
> killed
> > by dispatch.fcgi processes.
> >
> > If anyone can provide information about this, I''ll
> > collate it and write it up properly and make it
> > accessible.  My questions are:
> >
> > 1) What''s the cause?  Is it a problem with the
> FCGI
> > module, Apache, or the Rails dispatcher?
> >
> > 2) Does the problem only exist for certain Rails
> > versions, or for Rails on Debian, or all Rails
> > versions on all *nix platforms?
> >
> > 3) If you''ve never experienced any problem like
> this,
> > and you use Apache 2 and FCGI with Rails, what
> version
> > of Rails do you use, and what *nix distro?
> >
> > 4) Finally, what''s a *real* solution to the
> problem?
> > I''ll accept having to change to a different web
> > server, a different version of Rails, or a
> different
> > distro if I absolutely know it will solve my
> problem.
> >
> >
> > Many thanks in advance for any info.
> >
> >
> > Matt Jones
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam
> protection around
> > http://mail.yahoo.com
> > _______________________________________________
> > Rails mailing list
> > Rails@lists.rubyonrails.org
> >
> http://lists.rubyonrails.org/mailman/listinfo/rails
> >
> 
> 
> 
> --
> Kent
> ---
> http://www.datanoise.com
> > _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
> 
__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

Matt,

How has this gone for you? I am using Lighttpd 1.4.11 on debian
testing and I get the same behaviour you were talking about. I''m
manually spawning my FCGI instances and running them on a socket so I
would have thought that any adaptive spawning would have been out of
the question... but I guess that''s just one of the reasons why this
problem is so irritating :)

If mod_fcgid under apache is the way to go I''m keen to hear it.

Cheers,
Dan

--
Dan Sketcher
www.peoplehub.com.au
www.dansketcher.com

I''ve not come up with a good solution to this either.

I don''t get the "randomly spawning" processes, but it seems
that after a web
restart some FCGI dispatchers are left hanging around. Over time they can
build up and take over the machine.

I have 3 production sites on Apache 2 / (insert unix here Debian | FBSD) /
FCGI and experience the problem on all 3.

What''s a clean way to handle this? I don''t want to have to
resort to writing
a script to reap the processes ala dreamhost. That would suck.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060411/0300afe3/attachment.html

* Dan Sketcher (dansketcher@gmail.com) [060410 18:30]:
> How has this gone for you? I am using Lighttpd 1.4.11 on debian
> testing and I get the same behaviour you were talking about. I''m
> manually spawning my FCGI instances and running them on a socket so I
> would have thought that any adaptive spawning would have been out of
> the question... but I guess that''s just one of the reasons why
this
> problem is so irritating :)
> 
> If mod_fcgid under apache is the way to go I''m keen to hear it.
FWIW, on Debian, using Apache2 to front lighttpd with lighttpd
delegating to fcgi Rails has worked well for us (i.e., zero problems).
Using Apache{1,2}, on Debian, to serve FCGI Rails directly has been a
constant headache and I do not recommend it at all.  Intermittent 500
errors, "Application Error (Rails)" at random times, buildup of fcgi
processes, and eventual swamping of the machine were consistently seen.

Rick
-- 
 http://www.rickbradley.com    MUPRN: 702
                       |  him speak before. Too
   random email haiku  |  bad. .understandable of
                       |  course. but too bad.

Try mod_fcgid: http://datanoise.com/articles/2006/04/06/apache2-with-mod_fcgid


On 4/10/06, subimage interactive <subimage@gmail.com>
wrote:
> I''ve not come up with a good solution to this either.
>
> I don''t get the "randomly spawning" processes, but it
seems that after a web
> restart some FCGI dispatchers are left hanging around. Over time they can
> build up and take over the machine.
>
> I have 3 production sites on Apache 2 / (insert unix here Debian | FBSD) /
> FCGI and experience the problem on all 3.
>
> What''s a clean way to handle this? I don''t want to have
to resort to writing
> a script to reap the processes ala dreamhost. That would suck.
>
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>
>
>

--
Kent
---
http://www.datanoise.com

On 4/11/06, subimage interactive <subimage@gmail.com>
wrote:
> I''ve not come up with a good solution to this either.
>
> I don''t get the "randomly spawning" processes, but it
seems that after a web
> restart some FCGI dispatchers are left hanging around. Over time they can
> build up and take over the machine.
>
That''s the reason I stopped letting lighttpd do the spawning and set
up static fcgi processes like this:
http://jamis.jamisbuck.org/articles/2006/02/11/tip-textdrive-and-lighttpd
The difference for me is that I moved from a FreeBSD server to a
Debian one and this problem started. I think I''ll have a crack at
Mongrel once I work out how Capistrano will interact with it (can''t
find much info on this).

Cheers,
Dan

--
Dan Sketcher
www.peoplehub.com.au
www.dansketcher.com

I agree wholeheartedly.   I had a problem with random fcgi processes 
before I switched and it has disappeared completely.  The one gotcha I 
encountered was to set production mode in environment.rb b/c I couldn''t
get the environment variables to pass, but that is no biggie really.

Nicholas P. Mueller

Kent Sibilev wrote:
> Try mod_fcgid:
http://datanoise.com/articles/2006/04/06/apache2-with-mod_fcgid
>
>
> On 4/10/06, subimage interactive <subimage@gmail.com> wrote:
>   
>> I''ve not come up with a good solution to this either.
>>
>> I don''t get the "randomly spawning" processes, but
it seems that after a web
>> restart some FCGI dispatchers are left hanging around. Over time they
can
>> build up and take over the machine.
>>
>> I have 3 production sites on Apache 2 / (insert unix here Debian |
FBSD) /
>> FCGI and experience the problem on all 3.
>>
>> What''s a clean way to handle this? I don''t want to
have to resort to writing
>> a script to reap the processes ala dreamhost. That would suck.
>>
>> _______________________________________________
>> Rails mailing list
>> Rails@lists.rubyonrails.org
>> http://lists.rubyonrails.org/mailman/listinfo/rails
>>
>>
>>
>>     
>
>
> --
> Kent
> ---
> http://www.datanoise.com
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>

I should clarify that I still get zombie processes even when I have
static fcgi processes on Debian Etch - on FreeBSD 5.4/6.0 I had no
problems.

--
Dan Sketcher
www.peoplehub.com.au
www.dansketcher.com

I''ve also set static fcgi processes and I have to force into production
mode, but I still get zombie fcgi''s from time to time. It''s
almost as if
they don''t get shut down properly on an apache restart.

This is a little off topic, but related - how are you all restarting your
app when pushing new code to production? I''m under the impression that
you
have to actually restart apache in order for the changes to take effect.

On 4/10/06, Dan Sketcher <dansketcher@gmail.com>
wrote:
>
> I should clarify that I still get zombie processes even when I have
> static fcgi processes on Debian Etch - on FreeBSD 5.4/6.0 I had no
> problems.
>
> --
> Dan Sketcher
> www.peoplehub.com.au
> www.dansketcher.com
> _______________________________________________
> Rails mailing list
> Rails@lists.rubyonrails.org
> http://lists.rubyonrails.org/mailman/listinfo/rails
>


--
seth at subimage interactive
http://www.subimage.com/sublog/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://wrath.rubyonrails.org/pipermail/rails/attachments/20060411/5ad7adb0/attachment.html

I''ve found a few posts that might help some having this problem -

http://www.fastcgi.com/archives/fastcgi-developers/2005-May/003720.html
which links to
http://rubyists.com/articles/2005/05/03/spawn-fcgi-in-the-foreground

There is a ticket to get it integrated directly into Lighttpd in
http://trac.lighttpd.net/trac/ticket/216 - although from what I gather
in the patch it''s more of fastcgi rather than lighty

Doesn''t help me though because I can''t get the right build
environment
on debian testing for some reason - probably because I am a debian
noob. I wanted to have a crack at the changes from
http://trac.lighttpd.net/trac/ticket/554 to see if it fixed anything
but I guess I''ll just have to accept it for now...

Does anyone have a good script to kill off the extra zombies?

--
Dan Sketcher
www.peoplehub.com.au
www.dansketcher.com