On Zed''s suggestion, I caught two new spinning mongrels and sent a SIGUSR2. The code appears to be stuck in read_multipart for both processes: # kill -USR2 6109 ** USR2 signal received. Thu Sep 21 14:55:39 EDT 2006: Reaping 1 threads for slow workers because of ''shutdown'' Thread #<Thread:0x419d7ce0 run> is too old, killing. Waiting for 1 requests to finish, could take 60 seconds.Error calling Dispatcher.dispatch #<Mongrel::TimeoutError: Timed out thread.> /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:600:in `loop'' /usr/lib/ruby/1.8/cgi.rb:984:in `read_multipart'' /usr/lib/ruby/gems/1.8/gems/actionpack-1.12.5/lib/action_controller/cgi_ext/raw_post_data_fix.rb:20:in `initialize_query'' /usr/lib/ruby/1.8/cgi.rb:2270:in `initialize'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel/cgi.rb:47:in `initialize'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel/rails.rb:67:in `process'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:551:in `process_client'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:550:in `process_client'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:636:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:636:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:625:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:956:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:955:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/bin/mongrel_rails:127:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel/command.rb:199:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/bin/mongrel_rails:235 /usr/bin/mongrel_rails:18 The same for process 6100. We''ve had multipart problems before, but most of those were due to malformed multipart MIME posts. Has anyone else experienced multipart mime posts causing mongrel to spin out of control? - Ian C. Blenke <ian at blenke.com> http://ian.blenke.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: ian.vcf Type: text/x-vcard Size: 117 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20060921/c8d42e05/attachment.vcf
This is a repost, apparently my earlier message attempts were eaten by rubyforge. On Zed''s suggestion, I caught two new spinning mongrels and sent a SIGUSR2 (yeah, not a SIGUSR1, 0.3.13.3 and all). The code appears to be stuck in read_multipart for both processes: # kill -USR2 6109 ** USR2 signal received. Thu Sep 21 14:55:39 EDT 2006: Reaping 1 threads for slow workers because of ''shutdown'' Thread #<Thread:0x419d7ce0 run> is too old, killing. Waiting for 1 requests to finish, could take 60 seconds.Error calling Dispatcher.dispatch #<Mongrel::TimeoutError: Timed out thread.> /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:600:in `loop'' /usr/lib/ruby/1.8/cgi.rb:984:in `read_multipart'' /usr/lib/ruby/gems/1.8/gems/actionpack-1.12.5/lib/action_controller/cgi_ext/raw_post_data_fix.rb:20:in `initialize_query'' /usr/lib/ruby/1.8/cgi.rb:2270:in `initialize'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel/cgi.rb:47:in `initialize'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel/rails.rb:67:in `process'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:551:in `process_client'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:550:in `process_client'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:636:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:636:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:625:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:956:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel.rb:955:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/bin/mongrel_rails:127:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/lib/mongrel/command.rb:199:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.3/bin/mongrel_rails:235 /usr/bin/mongrel_rails:18 The same for process 6100. We''ve had multipart problems before, but most of those were due to malformed multipart MIME posts. Has anyone else experienced multipart mime posts causing mongrel to spin out of control? - Ian C. Blenke <ian at blenke.com> http://ian.blenke.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: ian.vcf Type: text/x-vcard Size: 117 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20060922/53f5558b/attachment.vcf
On Thu, 2006-09-21 at 20:57 -0400, Ian C. Blenke wrote:> On Zed''s suggestion, I caught two new spinning mongrels and sent a > SIGUSR2. The code appears to be stuck in read_multipart for both processes:Hey Ian, can you give me some more information on this? 1) What is being uploaded and what are the average sizes and content types? 2) Does the process stay pegged for a long time (like never ends) or is it possible that the mime processing is just taking longer than you expect? 3) How often does this happen? 4) Operating system particulars, and Ruby version. Thanks. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Zed Shaw wrote:> On Thu, 2006-09-21 at 20:57 -0400, Ian C. Blenke wrote: > >> On Zed''s suggestion, I caught two new spinning mongrels and sent a >> SIGUSR2. The code appears to be stuck in read_multipart for both processes: >> > > Hey Ian, can you give me some more information on this? > > 1) What is being uploaded and what are the average sizes and content > types? >XML encapsulated HTML pages from various public websites. Browser extensions are uploading the pages to a Rails server using a RESTful JSON based multi-part MIME API post. Some are pretty large, most are fairly small.> 2) Does the process stay pegged for a long time (like never ends) or is > it possible that the mime processing is just taking longer than you > expect? >The processes stay pegged for a long time, they never seem to end.> 3) How often does this happen? >During the early development cycle of the browser extensions, this was a huge problem, as the developers were sending malformed multi-part MIME boundaries. Regardless of the size of the submitted webpage, mongrel would go into a tight spin. After giving each developer a local VMWare player runtime with the entire application environment to beat up against, they cleaned up their bugs and the problem "went away". Today, we''re only getting this problem occasionally. On a low volume site, a couple times a day. It "feels like" the extensions are getting interrupted midway through their posts to the server (TCP streams reset, etc), but I haven''t verified this conclusively yet.> 4) Operating system particulars, and Ruby version. >Multicore AMD64X2 4G whitebox servers, a Debian woody based homegrown distro with many many backports (virtually sarge), linux 2.6.13 kernel. backported Ruby 1.8.4 (-1, not the newer buggier builds). Rails 1.1.6 (from gems). - Ian C. Blenke <ian at blenke.com> http://ian.blenke.com/