hi all, I can blab, but here is a sample of my /var/log/apache2/error.log, it has all the info: any ideas what I can do prevent 500s from hitting my users? (besides redirecting them with javascript to the same URL which will be fine on the next request? :) thanks, _alex [Fri Jun 24 12:30:45 2005] [error] [client 172.20.0.175] FastCGI: incomplete headers (0 bytes) received from server "/var/www/project_name/public/dispatch.fcgi" [Fri Jun 24 12:30:45 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" (pid 9337) terminated due to uncaught signal ''6'' (Aborted) [Fri Jun 24 12:30:45 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" restarted (pid 9376) [Fri Jun 24 12:31:40 2005] [notice] caught SIGTERM, shutting down [Fri Jun 24 12:31:41 2005] [notice] FastCGI: process manager initialized (pid 9853) [Fri Jun 24 12:31:41 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 9860) [Fri Jun 24 12:31:41 2005] [notice] Apache/2.0.54 (Debian GNU/Linux) mod_fastcgi/2.4.2 configured -- resuming normal operations [Fri Jun 24 12:31:42 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 9930) [Fri Jun 24 12:31:43 2005] [notice] caught SIGTERM, shutting down [Fri Jun 24 12:31:43 2005] [alert] (4)Interrupted system call: FastCGI: read() from pipe failed (0) [Fri Jun 24 12:31:43 2005] [alert] (4)Interrupted system call: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye [Fri Jun 24 12:31:43 2005] [notice] Apache/2.0.54 (Debian GNU/Linux) mod_fastcgi/2.4.2 configured -- resuming normal operations [Fri Jun 24 12:31:43 2005] [notice] FastCGI: process manager initialized (pid 9941) [Fri Jun 24 12:31:43 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 9955) [Fri Jun 24 12:31:44 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10000) [Fri Jun 24 12:31:45 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10001) [Fri Jun 24 12:31:46 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10010) [Fri Jun 24 12:31:47 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10025) [Fri Jun 24 12:31:48 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10044) [Fri Jun 24 12:31:49 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10063) [Fri Jun 24 12:31:51 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10084) [Fri Jun 24 12:31:52 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10107) [Fri Jun 24 12:31:53 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10130) [Fri Jun 24 12:31:54 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10155) [Fri Jun 24 12:31:55 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10176) [Fri Jun 24 12:31:56 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10199) [Fri Jun 24 12:31:57 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10218) [Fri Jun 24 12:31:58 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" started (pid 10241) /usr/lib/ruby/gems/1.8/gems/activesupport-1.0.4/lib/active_support/ core_ext/hash/keys.rb:27: [BUG] gc_sweep(): unknown data type 0x407560d0(0) ruby 1.8.2 (2005-04-11) [i386-linux] [Fri Jun 24 12:35:58 2005] [error] [client 172.20.0.175] FastCGI: incomplete headers (0 bytes) received from server "/var/www/project_name/public/dispatch.fcgi" [Fri Jun 24 12:35:58 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" (pid 10241) terminated due to uncaught signal ''6'' (Aborted) [Fri Jun 24 12:35:58 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" restarted (pid 11044) /usr/lib/ruby/gems/1.8/gems/actionpack-1.8.1/lib/action_view/helpers/ tag_helper.rb:30: [BUG] Segmentation fault ruby 1.8.2 (2005-04-11) [i386-linux] [Fri Jun 24 12:35:58 2005] [error] [client 172.20.0.175] FastCGI: incomplete headers (0 bytes) received from server "/var/www/project_name/public/dispatch.fcgi" [Fri Jun 24 12:35:58 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" (pid 10130) terminated due to uncaught signal ''6'' (Aborted) ?PAP(?@:72: [BUG] Segmentation fault ruby 1.8.2 (2005-04-11) [i386-linux] [Fri Jun 24 12:36:02 2005] [error] [client 172.20.0.175] FastCGI: incomplete headers (0 bytes) received from server "/var/www/project_name/public/dispatch.fcgi" [Fri Jun 24 12:36:02 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" (pid 10155) terminated due to uncaught signal ''6'' (Aborted) /usr/lib/ruby/1.8/erb.rb:478: [BUG] Segmentation fault ruby 1.8.2 (2005-04-11) [i386-linux] [Fri Jun 24 12:36:02 2005] [error] [client 172.20.0.175] FastCGI: incomplete headers (0 bytes) received from server "/var/www/project_name/public/dispatch.fcgi" [Fri Jun 24 12:36:02 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" (pid 10199) terminated due to uncaught signal ''6'' (Aborted) [Fri Jun 24 12:36:02 2005] [warn] FastCGI: server "/var/www/project_name/public/dispatch.fcgi" has remained running for more than 30 seconds, its restart interval has been restored to 5 seconds -- alex black, founder the turing studio, inc. 510.666.0074 root-16h2cdTTKgpzNNFeSAH1EA@public.gmane.org http://www.turingstudio.com 2600 10th street, suite 635 berkeley, ca 94710
Duane Johnson
2005-Jun-27 15:00 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
On Jun 24, 2005, at 1:48 PM, alex black wrote:> hi all, > > I can blab, but here is a sample of my /var/log/apache2/error.log, > it has all the info: > > any ideas what I can do prevent 500s from hitting my users? > (besides redirecting them with javascript to the same URL which > will be fine on the next request? :) > > thanks, > > _alex > > > > [Fri Jun 24 12:30:45 2005] [error] [client 172.20.0.175] FastCGI: > incomplete headers (0 bytes) received from server "/var/www/ > project_name/public/dispatch.fcgi" > [Fri Jun 24 12:30:45 2005] [warn] FastCGI: server "/var/www/ > project_name/public/dispatch.fcgi" (pid 9337) terminated due to > uncaught signal ''6'' (Aborted)I''m getting this too: [Sat Jun 25 11:14:41 2005] [error] [client 67.50.46.38] (61) Connection refused: FastCGI: failed to connect to server "/home/users/ filmfury/public_html/dispatch.fcgi": connect() failed [Sat Jun 25 11:14:41 2005] [error] [client 67.50.46.38] FastCGI: incomplete headers (0 bytes) received from server "/home/users/ filmfury/public_html/dispatch.fcgi" Has anyone come up with something? Duane Johnson (canadaduane) _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails
David Heinemeier Hansson
2005-Jun-27 15:05 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
Check your fastcgi.crash.logs. And be sure that you''re using the latest dispatch.fcgi if at all possible (which currently means running beta gems/edge rails). A 0.13 release is near forthcoming and will dish out the much strengthened dispatch.fcgi to all. -- David Heinemeier Hansson http://www.loudthinking.com -- Broadcasting Brain http://www.basecamphq.com -- Online project management http://www.backpackit.com -- Personal information manager http://www.rubyonrails.com -- Web-application framework
Duane Johnson
2005-Jun-27 16:06 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
On Jun 27, 2005, at 9:05 AM, David Heinemeier Hansson wrote:> Check your fastcgi.crash.logs. And be sure that you''re using the > latest dispatch.fcgi if at all possible (which currently means running > beta gems/edge rails). A 0.13 release is near forthcoming and will > dish out the much strengthened dispatch.fcgi to all.I''m running edge rails. Here''s what the crash log says (repeated many times): [Fri Jun 24 21:46:18 MDT 2005] Dispatcher failed to catch: exit (SystemExit) /usr/local/lib/ruby/site_ruby/1.8/fcgi.rb:10:in `exit'' /usr/local/lib/ruby/site_ruby/1.8/fcgi.rb:10 /usr/local/lib/ruby/site_ruby/1.8/fcgi.rb:10:in `call'' /usr/local/lib/ruby/site_ruby/1.8/fcgi.rb:598:in `each_cgi'' /usr/local/lib/ruby/site_ruby/1.8/fcgi.rb:597:in `each'' /usr/local/lib/ruby/site_ruby/1.8/fcgi.rb:597:in `each_cgi'' /home/users/filmfury/public_html/dispatch.fcgi:18 FCGI process 22529 killed by this error However, these entries are curiously from Friday and there are no entries in the crash log for Saturday--but saturday is when we were getting the fcgi-caused error 500s. So I wonder, is this SystemExit the cause of the problem or a symptom of it? Or is it simply unrelated? Duane Johnson (canadaduane) _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails
Belorion
2005-Jun-27 16:13 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
*snip*> However, these entries are curiously from Friday and there are no entries in > the crash log for Saturday--but saturday is when we were getting the > fcgi-caused error 500s. > > So I wonder, is this SystemExit the cause of the problem or a symptom of it? > Or is it simply unrelated? > > Duane Johnson > (canadaduane)I encountered similar errors on a few occassions, and *if* I recall correctly, in my case it was a configuration problem ... ie I had a syntax error in one of my config (such as environment.rm) files that is loaded up at the beginning. Matt
Jason A. Hoffman
2005-Jun-27 16:14 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
On Jun 27, 2005, at 8:00 AM, Duane Johnson wrote:> On Jun 24, 2005, at 1:48 PM, alex black wrote: >> >> [Fri Jun 24 12:30:45 2005] [error] [client 172.20.0.175] FastCGI: >> incomplete headers (0 bytes) received from server "/var/www/ >> project_name/public/dispatch.fcgi" >> [Fri Jun 24 12:30:45 2005] [warn] FastCGI: server "/var/www/ >> project_name/public/dispatch.fcgi" (pid 9337) terminated due to >> uncaught signal ''6'' (Aborted) > > I''m getting this too: > > [Sat Jun 25 11:14:41 2005] [error] [client 67.50.46.38] (61) > Connection refused: FastCGI: failed to connect to server "/home/ > users/filmfury/public_html/dispatch.fcgi": connect() failed > [Sat Jun 25 11:14:41 2005] [error] [client 67.50.46.38] FastCGI: > incomplete headers (0 bytes) received from server "/home/users/ > filmfury/public_html/dispatch.fcgi" > > Has anyone come up with something? > > Duane Johnson > (canadaduane)It doesn''t seem that you''re getting the signal 6 error, your''s looks like fastcgi kicking old fcgi processes out. Do you actually see failures in the site from coming up or it''s just these in the log? - Jason _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails
Trevor Squires
2005-Jun-27 16:59 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
Hi, I might as well add another war story to this thread... I''ve found that occasionally things go squiffy after a restart of apache and it can only be cleared by stopping apache and removing the fcgi ipc directory before restarting apache again. I''ve not had the time to see why... HTH Trevor On 27-Jun-05, at 9:13 AM, Belorion wrote:> *snip* >> However, these entries are curiously from Friday and there are no >> entries in >> the crash log for Saturday--but saturday is when we were getting the >> fcgi-caused error 500s. >> >> So I wonder, is this SystemExit the cause of the problem or a symptom >> of it? >> Or is it simply unrelated? >> >> Duane Johnson >> (canadaduane) > > I encountered similar errors on a few occassions, and *if* I recall > correctly, in my case it was a configuration problem ... ie I had a > syntax error in one of my config (such as environment.rm) files that > is loaded up at the beginning. > > Matt > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails
Matias Pelenur
2005-Jun-28 02:58 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
I have a related question: I''ve found that whenever I restart apache, one or two FastCGI processes stay alive. I can see on the fastcgi.crash.log that the other processes receive SIGHUP and die, and am wondering why the remaining one or two processes don''t also get SIGHUP''d and die. My guess is that they do get the SIGHUP, but they are in some hung state and refuse to die (I have to kill them with -9). Has anyone experienced this problem? Thanks, matias Trevor Squires wrote:> Hi, > > I might as well add another war story to this thread... > > I''ve found that occasionally things go squiffy after a restart of apache > and it can only be cleared by stopping apache and removing the fcgi ipc > directory before restarting apache again. I''ve not had the time to see > why... > > HTH > Trevor > > On 27-Jun-05, at 9:13 AM, Belorion wrote: > >> *snip* >> >>> However, these entries are curiously from Friday and there are no >>> entries in >>> the crash log for Saturday--but saturday is when we were getting the >>> fcgi-caused error 500s. >>> >>> So I wonder, is this SystemExit the cause of the problem or a symptom >>> of it? >>> Or is it simply unrelated? >>> >>> Duane Johnson >>> (canadaduane) >> >> >> I encountered similar errors on a few occassions, and *if* I recall >> correctly, in my case it was a configuration problem ... ie I had a >> syntax error in one of my config (such as environment.rm) files that >> is loaded up at the beginning. >> >> Matt >> _______________________________________________ >> Rails mailing list >> Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org >> http://lists.rubyonrails.org/mailman/listinfo/rails > > > _______________________________________________ > Rails mailing list > Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org > http://lists.rubyonrails.org/mailman/listinfo/rails
Lon Baker
2005-Jun-28 15:03 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
I have been seeing that issue under Apache 1.3 on Mac OS X. But have not stressed over it since we hope to move to Lighttpd soon. But it would be good to here if there are any solutions to this issue. Some clients don''t want to deploy Lighttpd. -- Lon Baker Speedymac LLC http://www.speedymac.com AIM: spdemac Skype: spdemac On Jun 27, 2005, at 11:17 PM, rails-request-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org wrote:> I have a related question: I''ve found that whenever I restart > apache, one or two > FastCGI processes stay alive. I can see on the fastcgi.crash.log > that the other > processes receive SIGHUP and die, and am wondering why the > remaining one or two > processes don''t also get SIGHUP''d and die. My guess is that they do > get the > SIGHUP, but they are in some hung state and refuse to die (I have > to kill them > with -9). > > Has anyone experienced this problem? >
Robert Cottrell
2005-Jun-28 17:58 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
On Jun 27, 2005, at 7:58 PM, Matias Pelenur wrote:> I have a related question: I''ve found that whenever I restart > apache, one or two > FastCGI processes stay alive. I can see on the fastcgi.crash.log > that the other > processes receive SIGHUP and die, and am wondering why the > remaining one or two > processes don''t also get SIGHUP''d and die. My guess is that they do > get the > SIGHUP, but they are in some hung state and refuse to die (I have > to kill them > with -9).Ruby has some tricky signal handling logic that can interfere with the FastCGI processes. There is some code in the FCGI module that tries to ensure that calls into the C layer accept() and select() won''t block. Unfortunately, we''ve noticed that this doesn''t always work and some processes will block in the C layer and not receive the termination signal. Our solution to this was simply to run a cron that periodically looks for these orphaned processes (they will be owned by init and not the FastCGI process manager), and kills them. _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails
Duane Johnson
2005-Jun-29 16:15 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
On Jun 27, 2005, at 10:14 AM, Jason A. Hoffman wrote:>>> [Fri Jun 24 12:30:45 2005] [error] [client 172.20.0.175] FastCGI: >>> incomplete headers (0 bytes) received from server "/var/www/ >>> project_name/public/dispatch.fcgi" >>> [Fri Jun 24 12:30:45 2005] [warn] FastCGI: server "/var/www/ >>> project_name/public/dispatch.fcgi" (pid 9337) terminated due to >>> uncaught signal ''6'' (Aborted) >> >> [Sat Jun 25 11:14:41 2005] [error] [client 67.50.46.38] (61) >> Connection refused: FastCGI: failed to connect to server "/home/ >> users/filmfury/public_html/dispatch.fcgi": connect() failed >> [Sat Jun 25 11:14:41 2005] [error] [client 67.50.46.38] FastCGI: >> incomplete headers (0 bytes) received from server "/home/users/ >> filmfury/public_html/dispatch.fcgi" > > It doesn''t seem that you''re getting the signal 6 error, your''s > looks like fastcgi kicking old fcgi processes out. Do you actually > see failures in the site from coming up or it''s just these in the log?I''m still working on this, but it seems that the server will work for a certain period of time or for a certain number of requests and then start dishing out the Application Error page. Each request results in the quoted "Connection refused" and the "incomplete headers" messages above (from /var/log/httpd-errors.log). I''m still very new at debugging / analyzing apache and fcgi. When I use "ps aux" and grep it for fcgi, I''m not getting anything. Is there something else I should be searching for, or could it be that, indeed, fcgi isn''t running anymore? Any guidance here would be appreciated. Thanks, Duane Johnson (canadaduane) _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails
Duane Johnson wrote:> I''m still very new at debugging / analyzing apache and fcgi. When I use > "ps aux" and grep it for fcgi, I''m not getting anything.Sometimes ps chops the end of process names off making it look like something is not running when it is, try pgrep instead: pgrep -l fcgi Will.
Duane Johnson
2005-Jun-29 16:29 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
On Jun 27, 2005, at 10:14 AM, Jason A. Hoffman wrote:> It doesn''t seem that you''re getting the signal 6 error, your''s > looks like fastcgi kicking old fcgi processes out. Do you actually > see failures in the site from coming up or it''s just these in the log?When I use Apache benchmark, with n > 1, I always get failed requests. What''s going on here? $ ab -n 5 filmfury.com/ <snip> Benchmarking filmfury.com (be patient).....done Server Software: Apache/1.3.33 Server Hostname: filmfury.com Server Port: 80 Document Path: / Document Length: 5819 bytes Concurrency Level: 1 Time taken for tests: 0.355 seconds Complete requests: 5 Failed requests: 4 <-- Why is this failing? (Connect: 0, Length: 4, Exceptions: 0) Broken pipe errors: 0 Total transferred: 30313 bytes HTML transferred: 29008 bytes Requests per second: 14.08 [#/sec] (mean) Time per request: 71.00 [ms] (mean) Time per request: 71.00 [ms] (mean, across all concurrent requests) Transfer rate: 85.39 [Kbytes/sec] received I can use many concurrent requests, apparently, without failure; however, anything with n > 1 with or without concurrent requests will cause those failures (n-1) to appear. I appreciate your help. We''re trying to make this Rails site live today! :) Duane Johnson (canadaduane) _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails
Todd Grimason
2005-Jun-29 16:33 UTC
Re: apache2 fastcgi (fcgi) internal server errors, still :(
* will [2005-06-29 12:28]:> Duane Johnson wrote: > >I''m still very new at debugging / analyzing apache and fcgi. When I use > >"ps aux" and grep it for fcgi, I''m not getting anything. > > Sometimes ps chops the end of process names off making it look like > something is not running when it is, try pgrep instead: > > pgrep -l fcgi >Or if you don''t have pgrep around you can still use ps -- add 1 or 2 ''-w'' flags. man ps: -w Use 132 columns to display information, instead of the default which is your window size. If the -w option is specified more than once, ps will use as many columns as necessary without regard for your window size. -- ________________________________ toddgrimason*todd[ at ]slack.net
On Jun 29, 2005, at 10:29 AM, Duane Johnson wrote:> When I use Apache benchmark, with n > 1, I always get failed > requests. What''s going on here? > > $ ab -n 5 filmfury.com/ > <snip> > Benchmarking filmfury.com (be patient).....done > Server Software: Apache/1.3.33 > Server Hostname: filmfury.com > Server Port: 80 > > Document Path: / > Document Length: 5819 bytes > > Concurrency Level: 1 > Time taken for tests: 0.355 seconds > Complete requests: 5 > Failed requests: 4 <-- Why is this failing? > (Connect: 0, Length: 4, Exceptions: 0) > Broken pipe errors: 0 > Total transferred: 30313 bytes > HTML transferred: 29008 bytes > Requests per second: 14.08 [#/sec] (mean) > Time per request: 71.00 [ms] (mean) > Time per request: 71.00 [ms] (mean, across all concurrent > requests) > Transfer rate: 85.39 [Kbytes/sec] received > > I can use many concurrent requests, apparently, without failure; > however, anything with n > 1 with or without concurrent requests > will cause those failures (n-1) to appear.For anyone who is interested, it turns out the "length" failures simply indicate that the page length (number of bytes) is different each time apache benchmark loads the page--which is 100% ok since the page contains dynamic content. Duane Johnson (canadaduane) _______________________________________________ Rails mailing list Rails-1W37MKcQCpIf0INCOvqR/iCwEArCW2h5@public.gmane.org http://lists.rubyonrails.org/mailman/listinfo/rails