hi all, I''m getting these errors: [Tue Jun 21 02:19:57 2005] [error] [client (my IP)] FastCGI: incomplete headers (0 bytes) received from server "/var/www/beta/site/trunk/public/dispatch.fcgi", referer: http://url/company/investors Occasionally - perhaps every 100 requests or so. Obviously I can''t go into production with "Internal Server Error" printed by apache every (x) page requests... any ideas? FastCGI is a pain in the bootay, it doesn''t know how to handle problems at all :( My apache2 FCGI config: <IfModule mod_fastcgi.c> FastCgiIpcDir /tmp/fcgi_ipc/ AddHandler fastcgi-script .fcgi FastCgiConfig -minProcesses 4 -maxProcesses 40 -singleThreshold 100 -killInterval 120 -autoUpdate -initial-env RAILS_ENV=development -idle-timeout 120 -pass-header HTTP_AUTHORIZATION </IfModule> (which was cobbled together from this list and by preying to the fcgi voodoo gods) if there are any 43things people listening - please - paste a production apache fcgi config for all to see... you have the knowledge.... ;) thanks, _alex -- alex black, founder the turing studio, inc. 510.666.0074 root-16h2cdTTKgpzNNFeSAH1EA@public.gmane.org http://www.turingstudio.com 2600 10th street, suite 635 berkeley, ca 94710
On Jun 21, 2005, at 1:26 AM, alex black wrote:> [Tue Jun 21 02:19:57 2005] [error] [client (my IP)] FastCGI: > incomplete headers (0 bytes) received from server "/var/www/beta/ > site/trunk/public/dispatch.fcgi", referer: http://url/company/ > investors > > Occasionally - perhaps every 100 requests or so. > > Obviously I can''t go into production with "Internal Server Error" > printed by apache every (x) page requests... any ideas? > > FastCGI is a pain in the bootay, it doesn''t know how to handle > problems at all :( > > My apache2 FCGI config: > > <IfModule mod_fastcgi.c> > FastCgiIpcDir /tmp/fcgi_ipc/ > AddHandler fastcgi-script .fcgi > FastCgiConfig -minProcesses 4 -maxProcesses 40 -singleThreshold 100 > -killInterval 120 -autoUpdate -initial-env RAILS_ENV=development - > idle-timeout 120 -pass-header HTTP_AUTHORIZATION > </IfModule>I would take out -singleThreshold -killInterval and -autoUpdate. Your application is likely not handling them well and because you''re under production and getting hit, they''re too aggressive. The singleThreshold is used by the process manager to figure out how long something should be left idle (the default is 0, which means it leaves idle procs alone, a value 1 is oddly enough the good balance between 0 and 100). Your killinterval is about 1/3 of the default and launches an aggressive killing policy. Both of these configs don''t really go with doing something in the RAILS_ENV=development environment, they go with having your application do a LOT of constant hits, and being able to rapidly kick out a "bad" process. These settings make sense that stuff is getting kicked out after about a 100ish hits. Then your maxProcesses is actually less then the default 50, min is normally 5. So ... I''d take out all from FastCgiConfig except for RAIL_ENV and - idle-timeout 360 (give yourself time), and then if it''s still happening, I''d turn my attention to the application. - Jason
i am not good in voodooing, but here is the configuration we use for bellybutton.de: FastCgiServer /var/www/prod_shop/public/dispatch.fcgi -initial-env RAILS_ENV=production -idle-timeout 60 -processes 15 simple and works fine! Am Dienstag, den 21.06.2005, 01:26 -0700 schrieb alex black:> hi all, > > I''m getting these errors: > > [Tue Jun 21 02:19:57 2005] [error] [client (my IP)] FastCGI: incomplete > headers (0 bytes) received from server > "/var/www/beta/site/trunk/public/dispatch.fcgi", referer: > http://url/company/investors > > Occasionally - perhaps every 100 requests or so. > > Obviously I can''t go into production with "Internal Server Error" > printed by apache every (x) page requests... any ideas? > > FastCGI is a pain in the bootay, it doesn''t know how to handle problems > at all :( > > My apache2 FCGI config: > > <IfModule mod_fastcgi.c> > FastCgiIpcDir /tmp/fcgi_ipc/ > AddHandler fastcgi-script .fcgi > FastCgiConfig -minProcesses 4 -maxProcesses 40 -singleThreshold 100 > -killInterval 120 -autoUpdate -initial-env RAILS_ENV=development > -idle-timeout 120 -pass-header HTTP_AUTHORIZATION > </IfModule> > > (which was cobbled together from this list and by preying to the fcgi > voodoo gods) > > if there are any 43things people listening - please - paste a > production apache fcgi config for all to see... you have the > knowledge.... ;) > > thanks, > > _alex > >-- Norman Timmler Holländische Reihe 31 22765 Hamburg +49 (0)40 / 43 25 10 80 mailto:norman-QkIQCVqxERM@public.gmane.org _ ASCII ribbon campaign ( ) - against HTML email X & vCards / \
On 6/21/05, alex black <enigma-16h2cdTTKgpzNNFeSAH1EA@public.gmane.org> wrote:> hi all, > > I''m getting these errors: > > [Tue Jun 21 02:19:57 2005] [error] [client (my IP)] FastCGI: incomplete > headers (0 bytes) received from server > "/var/www/beta/site/trunk/public/dispatch.fcgi", referer: > http://url/company/investors > > Occasionally - perhaps every 100 requests or so. > > Obviously I can''t go into production with "Internal Server Error" > printed by apache every (x) page requests... any ideas? > > FastCGI is a pain in the bootay, it doesn''t know how to handle problems > at all :( > > My apache2 FCGI config: > > <IfModule mod_fastcgi.c> > FastCgiIpcDir /tmp/fcgi_ipc/ > AddHandler fastcgi-script .fcgi > FastCgiConfig -minProcesses 4 -maxProcesses 40 -singleThreshold 100 > -killInterval 120 -autoUpdate -initial-env RAILS_ENV=development > -idle-timeout 120 -pass-header HTTP_AUTHORIZATION > </IfModule> > > (which was cobbled together from this list and by preying to the fcgi > voodoo gods) > > if there are any 43things people listening - please - paste a > production apache fcgi config for all to see... you have the > knowledge.... ;) > > thanks, > > _alex > > > -- > alex black, founder > the turing studio, inc. > > 510.666.0074 > root-16h2cdTTKgpzNNFeSAH1EA@public.gmane.org > http://www.turingstudio.com > > 2600 10th street, suite 635 > berkeley, ca 94710 > > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails >1. Run with RAILS_ENV=production rather than development environment. 2. If you haven''t yet, grab the dispatch.fcgi out of the svn trunk. Jason
Jason A.Hoffman wrote:> So ... I''d take out all from FastCgiConfig except for RAIL_ENV and - > idle-timeout 360 (give yourself time), and then if it''s still > happening, I''d turn my attention to the application.Yet another message on this list that I flag as "important" and keep for future reference. There''s some really useful, in-depth advice going on here! This is a great mailing list!! Cheers, ~Dave -- Dave Silvester Rent-A-Monkey Website Development Web: http://www.rentamonkey.com/
> I would take out -singleThreshold -killInterval and -autoUpdate.Ok. autoUpdate seems necessary for development. Agreed, I would remove this in production...> Then your maxProcesses is actually less then the default 50, min is > normally 5.Ah.. It seems apache fcgi needs a LOT of work before it''s a really good, stable system. for example, I wouldn''t even expect to tune these values to avoid outright errors. ;(> So ... I''d take out all from FastCgiConfig except for RAIL_ENV and > -idle-timeout 360 (give yourself time), and then if it''s still > happening, I''d turn my attention to the application.Will do - frustrating to deal with this and not the app... This last line disturbs me: "turn my attention to the app" - I would expect the code I''m writing or having written to be shielded from the kinds of errors I''m seeing - if I get a rails error message in a browser, that makes sense. It doesn''t make sense that apache''s communication with a fastcgi process craps out mid-stream and forces apache to return a 500 (where it should just try the request elsewhere again).. For production I''m going to take norman''s advice with a small mod on idle_timeout. FastCgiServer /var/www/my_app/public/dispatch.fcgi -initial-env RAILS_ENV=production -idle-timeout 120 -processes 15 For development, though, it''s still scary and bad for clients to see "Internal Server Error" and have my response be "oh, well, that won''t be there in production". thanks _very_much_ for your help ;) _a -- alex black, founder the turing studio, inc. 510.666.0074 root-16h2cdTTKgpzNNFeSAH1EA@public.gmane.org http://www.turingstudio.com 2600 10th street, suite 635 berkeley, ca 94710
Jason A. Hoffman
2005-Jun-21 22:21 UTC
Re: Random Internal Server Error 500 / Apache2 FCGI
Hi Alex, On Jun 21, 2005, at 11:59 AM, alex black wrote:> It seems apache fcgi needs a LOT of work before it''s a really good, > stable system. for example, I wouldn''t even expect to tune these > values to avoid outright errors. ;(The defaults are usually sufficient. Granted FCGI hasn''t change much since 1995, but it''s pretty solid. The mod_fastcgi for Apache is "better" under Apache1, but it''s exactly what''s exciting either the mod_fcgid module (which works with ruby and at least in PHP is powering portal sites in China that are doing 10-20 million visitors a day: the developer refers to his day job as "China''s Yahoo"), and the support of fastcgi by up-and-coming web servers like lighttpd. But I hear you, 500 errors under PHP-FCGI (which is "mature") can be too frequent under some circumstances and you might end up running strace to see where the cause is, usually it''s the process manager not kicking dynamic fcgis out fast enough (this is where mod_fcgid "fixed" mod_fastcgi). If that''s the case and one sticks with Apache then the other things to do (besides mod_fcgid) is to realize is that the "simple" config for mod_fastcgi and FastCGIConf only launches/ affects _dynamic FCGIs_, the same as lighttpd''s adaptive spawning. There is another settting and that is to have mod_fastcgi connect to your application as an external server (FastCGI server settings) where you spawn it from the commandline. Often when there''s a problem, one will try doing this and see if generic errors disappear. But that said, I know (as in know know, as in been doing mod_fastcgi for closer to a decade) that lighttpd''s mod_fastcgi is better and you won''t see 500s under it (well, at least I haven''t).>> So ... I''d take out all from FastCgiConfig except for RAIL_ENV >> and -idle-timeout 360 (give yourself time), and then if it''s still >> happening, I''d turn my attention to the application. > > Will do - frustrating to deal with this and not the app... This > last line disturbs me: "turn my attention to the app" - I would > expect the code I''m writing or having written to be shielded from > the kinds of errors I''m seeing - if I get a rails error message in > a browser, that makes sense. It doesn''t make sense that apache''s > communication with a fastcgi process craps out mid-stream and > forces apache to return a 500 (where it should just try the request > elsewhere again)..You are getting a "0" error and that can be due to something happening in the application, and then Apache throws a 500 because the backend fastcgi is no good. The problem with both of those errors is that they are as generic as they come, and the true problem can lie anywhere along the chain. If your on *nix, you can use truss/strace to see what your application is doing at the time the request is made.> For production I''m going to take norman''s advice with a small mod > on idle_timeout. > FastCgiServer /var/www/my_app/public/dispatch.fcgi -initial-env > RAILS_ENV=production -idle-timeout 120 -processes 15 > For development, though, it''s still scary and bad for clients to > see "Internal Server Error" and have my response be "oh, well, that > won''t be there in production". > thanks _very_much_ for your help ;)Well, I know what I do (and for some reason helping deploy a new rails site in production has become a very frequent -- sometimes daily -- thing), and that is that I don''t deploy production sites under Apache anymore. Not that anything is wrong with Apache, lighttpd is simply better, and 1.3.14 really fixed the issues that was preventing me from saying that 100% of the time (and there''s a few cool cool --at least to me -- things I''ll be writing about soon). - Jason
David Heinemeier Hansson
2005-Jun-21 22:31 UTC
Re: Random Internal Server Error 500 / Apache2 FCGI
Be sure to keep an eye on log/fastcgi.crash.log. Often times it''s "your own fault", meaning that the application logic has a problem. FastCGI can be rather unforgiving in those instances. -- David Heinemeier Hansson http://www.loudthinking.com -- Broadcasting Brain http://www.basecamphq.com -- Online project management http://www.backpackit.com -- Personal information manager http://www.rubyonrails.com -- Web-application framework