Hey folks, I ran into a few people at RubyConf who were having 99% CPU issues. Please contact me if you meet the following criteria: 1. You are running a production site. 2. You are experiencing 99% CPU errors. 3. This is frequent enough that you cannot manage it. Thank you. Please contact me off-list about it. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
I''m not having a CPU problem, but I was just installing and messing around with Mephisto 0.7.0 and noticed the ruby process was consuming around 150 megs after an hour or so. I shut it down and restarted and started clicking around the blog. The memory usage climbed steadily as I clicked. (This occurred in both production and dev modes.) I shut it down again and started WEBRick. Memory hit about 39 megs and just stayed there. Mac OS Tiger latest update Ruby 1.8.4 (MacPorts) Rails Edge Mephisto 0.7.0 Mongrel 0.3.13.4 Jamie On Oct 24, 2006, at 2:39 AM, Zed A. Shaw wrote:> Hey folks, > > I ran into a few people at RubyConf who were having 99% CPU > issues. Please contact me if you meet the following criteria: > > 1. You are running a production site. > 2. You are experiencing 99% CPU errors. > 3. This is frequent enough that you cannot manage it. > > Thank you. Please contact me off-list about it. > > -- > Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu > http://www.zedshaw.com/ > http://mongrel.rubyforge.org/ > http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On 10/24/06, Jamie Orchard-Hays <jamie at dangosaur.us> wrote:> I''m not having a CPU problem, but I was just installing and messing > around with Mephisto 0.7.0 and noticed the ruby process was consuming > around 150 megs after an hour or so. I shut it down and restarted and > started clicking around the blog. The memory usage climbed steadily > as I clicked. (This occurred in both production and dev modes.) I > shut it down again and started WEBRick. Memory hit about 39 megs and > just stayed there. > > Mac OS Tiger latest update > Ruby 1.8.4 (MacPorts) > Rails Edge > Mephisto 0.7.0 > Mongrel 0.3.13.4 >Mephisto hovers at around 50-60MB depending on how big the site is. This is from my experience on mephistoblog.com and riding rails. -- Rick Olson http://weblog.techno-weenie.net http://mephistoblog.com
Mine just kept going up. It had been at 150 or 160 when I restarted. I saw it go past 60 both times I tried it out. I''ll play with it again and see what happens (if it goes over 100). That seems huge, considering I''ve got a vanilla installation running. If you or Zed want me to try anything in particular, just let me know. Jamie On Oct 24, 2006, at 11:56 AM, Rick Olson wrote:> On 10/24/06, Jamie Orchard-Hays <jamie at dangosaur.us> wrote: >> I''m not having a CPU problem, but I was just installing and messing >> around with Mephisto 0.7.0 and noticed the ruby process was consuming >> around 150 megs after an hour or so. I shut it down and restarted and >> started clicking around the blog. The memory usage climbed steadily >> as I clicked. (This occurred in both production and dev modes.) I >> shut it down again and started WEBRick. Memory hit about 39 megs and >> just stayed there. >> >> Mac OS Tiger latest update >> Ruby 1.8.4 (MacPorts) >> Rails Edge >> Mephisto 0.7.0 >> Mongrel 0.3.13.4 >> > > Mephisto hovers at around 50-60MB depending on how big the site is. > This is from my experience on mephistoblog.com and riding rails. > > -- > Rick Olson > http://weblog.techno-weenie.net > http://mephistoblog.com > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
Zed- We''re seeing processes go off and become "dead", they continue to accept connections, but the don''t do anything useful. We had manually applied your one-liner from r356 and the problem went down quite a bit. We are still seeing an occasional process drift off and now we are getting a trace in the log. I''m responding to this here as it is a production site and a fairly high volume one http://jibjab.com , but we are not seeing high CPU usage, in fact the dead processes aren''t using any to speak of. Thanks in advance for you time and any tips on how we can track this down. Michael Trace follows: Thread #<Thread:0xb5cf9bb4 sleep> is too old, killing. Tue Oct 24 15:19:15 UTC 2006: Error calling Dispatcher.dispatch #<Sync_m::Err::UnknownLocker: Thread(#<Thread:0xb5cf9bb4 run>) not locked.> /usr/lib/ruby/1.8/sync.rb:57:in `Fail'' /usr/lib/ruby/1.8/sync.rb:63:in `Fail'' /usr/lib/ruby/1.8/sync.rb:183:in `sync_unlock'' /usr/lib/ruby/1.8/sync.rb:231:in `synchronize'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel/rails.rb: 83:in `process''knownLocker: Thread(#<Thread:0xb5cf9bb4 run>) not locked.> /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel.rb:580:in `process_client''r/lib/ruby/1.8/sync.rb:63:in `Fail'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel.rb:579:in `process_client''r/lib/ruby/1.8/sync.rb:231:in `synchronize'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel.rb:686:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel.rb:686:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel.rb:673:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel/ configurator.rb:267:in `run''lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/ lib/mongrel.rb:686:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel/ configurator.rb:266:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/bin/mongrel_rails:127:in `run'' /usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel/command.rb: 211:in `run''/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/bin/ mongrel_rails:231 /usr/bin/mongrel_rails:18 On Oct 23, 2006, at 11:39 PM, Zed A. Shaw wrote:> Hey folks, > > I ran into a few people at RubyConf who were having 99% CPU > issues. Please contact me if you meet the following criteria: > > 1. You are running a production site. > 2. You are experiencing 99% CPU errors. > 3. This is frequent enough that you cannot manage it. > > Thank you. Please contact me off-list about it. > > -- > Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu > http://www.zedshaw.com/ > http://mongrel.rubyforge.org/ > http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On Oct 24, 2006, at 3:10 PM, Zed A. Shaw wrote:> Go grab the 0.3.13.5 pre-release and run it under USR1 debugging in > production. It''s got a tweak to the debugging output so that you > can better understand what action is blocking the mongrel. > Something like this: > > $ gem install mongrel --source=http://mongrel.rubyforge.org/releases > $ <start mongrel_rails however you do> > $ killall -USR1 mongrel_rails > > Then, you''re looking for lines in mongrel.log: > > Tue Oct 24 15:07:07 PDT 2006: 0 threads sync_waiting for /test, 1 > still active > in Mongrel.Here''s a handful of the ones we logged yesterday, as you can see it wasn''t in a happy state. I''ll run off and grab the pre-release for one of the boxes and give it another spin. Mon Oct 23 21:35:33 UTC 2006: 331 threads sync_waiting for /, 337 still active in mongrel. Mon Oct 23 21:35:44 UTC 2006: 332 threads sync_waiting for /health/ db, 338 still active in mongrel. Mon Oct 23 21:36:05 UTC 2006: 334 threads sync_waiting for /, 340 still active in mongrel. Mon Oct 23 21:36:21 UTC 2006: 335 threads sync_waiting for /feeds/ player/22722/27312, 341 still active in mongrel. Mon Oct 23 21:36:34 UTC 2006: 337 threads sync_waiting for /usage/ 207205/2, 343 still active in mongrel. Mon Oct 23 21:36:52 UTC 2006: 339 threads sync_waiting for /, 345 still active in mongrel. Mon Oct 23 21:37:16 UTC 2006: 341 threads sync_waiting for /feeds/ gse_player/232062, 347 still active in mongrel. Mon Oct 23 21:37:31 UTC 2006: 343 threads sync_waiting for / great_sketch_experiment/register/, 349 still active in mongrel.> Yep, this sounds like you have a particular Rails action that is > blocking the process. What kinds of things is your Rails > application doing?I was trying to track something down in there yesterday based on the FAQ, but it looks like it''s all over? Maybe I''m reading it wrong or maybe I need to have it enabled before it goes bad and see who''s the first to complain?. As for what the site is doing, for the above actions, nothing special. The tricky bits are Ferret search over DRb, File uploads (which could be causing problems, it''s not using m_u_p yet), now that a take some time to think about it (it''s been hectic around here) I''m going to guess my next task is m_u_p. Thanks again- Michael
On Tue, 24 Oct 2006 11:05:49 -0700 Michael Moen <mi-mongrel at moensolutions.com> wrote:> Zed- We''re seeing processes go off and become "dead", they continue > to accept connections, but the don''t do anything useful. > > We had manually applied your one-liner from r356 and the problem went > down quite a bit. We are still seeing an occasional process drift off > and now we are getting a trace in the log. >Go grab the 0.3.13.5 pre-release and run it under USR1 debugging in production. It''s got a tweak to the debugging output so that you can better understand what action is blocking the mongrel. Something like this: $ gem install mongrel --source=http://mongrel.rubyforge.org/releases $ <start mongrel_rails however you do> $ killall -USR1 mongrel_rails Then, you''re looking for lines in mongrel.log: Tue Oct 24 15:07:07 PDT 2006: 0 threads sync_waiting for /test, 1 still active in Mongrel.> I''m responding to this here as it is a production site and a fairly > high volume one http://jibjab.com , but we are not seeing high CPU > usage, in fact the dead processes aren''t using any to speak of.Yep, this sounds like you have a particular Rails action that is blocking the process. What kinds of things is your Rails application doing? -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
On Tue, 24 Oct 2006 13:27:01 -0700 Michael Moen <mi-mongrel at moensolutions.com> wrote:> On Oct 24, 2006, at 3:10 PM, Zed A. Shaw wrote: > > Go grab the 0.3.13.5 pre-release and run it under USR1 debugging in > > production. It''s got a tweak to the debugging output so that you > > can better understand what action is blocking the mongrel. > > Something like this: > > > > $ gem install mongrel --source=http://mongrel.rubyforge.org/releases > > $ <start mongrel_rails however you do> > > $ killall -USR1 mongrel_rails > > > > Then, you''re looking for lines in mongrel.log: > > > > Tue Oct 24 15:07:07 PDT 2006: 0 threads sync_waiting for /test, 1 > > still active > > in Mongrel. > > Here''s a handful of the ones we logged yesterday, as you can see it > wasn''t in a happy state. I''ll run off and grab the pre-release for > one of the boxes and give it another spin.Is this with the 0.3.13.5 pre-release? It looks like you don''t have the change that makes it clearer what is blocking (or that it''s not really working well). -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
On Oct 24, 2006, at 5:35 PM, Zed A. Shaw wrote:> Is this with the 0.3.13.5 pre-release? It looks like you don''t > have the change that makes it clearer what is blocking (or that > it''s not really working well).Sorry- That was .13.4. I have 0.3.14 on one box currently, what''s the stability like in the pre-release? Is it safe to run across the entire cluster? Problem is since we the r356 change it happens a lot less and only running it on a single box could take a long time to show its face. Thanks again-
On Wed, 25 Oct 2006 08:05:13 -0700 Michael Moen <mi-mongrel at moensolutions.com> wrote:> > On Oct 24, 2006, at 5:35 PM, Zed A. Shaw wrote: > > Is this with the 0.3.13.5 pre-release? It looks like you don''t > > have the change that makes it clearer what is blocking (or that > > it''s not really working well). > > Sorry- That was .13.4. I have 0.3.14 on one box currently, what''s the > stability like in the pre-release? Is it safe to run across the > entire cluster? Problem is since we the r356 change it happens a lot > less and only running it on a single box could take a long time to > show its face.Should be good for you to run the latest pre-release I just, but definitely test it on a staging server first. Main thing that you''ll want to check is that if you use mongrel_upload_progress then just quickly make sure it works, and look for any EOFError reports in mongrel.log. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://safari.oreilly.com/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.