Matthew
2007-Mar-04 03:54 UTC
[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster
I''ve been troubleshooting this problem for several hours now and haven''t been able to figure out what''s wrong. I have two boxes, dev and prod, both are identical except that prod uses Ruby 1.8.5 and dev uses 1.8.4. Both are running Slave 1.2.0, Daemons 1.0.5, and Backroundrb 0.2.1. The behavior I''m seeing is this. When I first start Backgroundrb, this is what the process list looks like: cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0:03.36/usr/local/bin/ruby18 /usr/local/bin/mon cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mon cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48/usr/local/bin/ruby18 /usr/local/bin/mon cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mon cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 ruby: backgroundrb (ruby) cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: backgroundrb_logger (ruby) cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: backgroundrb_results (ruby) My four Mongrels in the cluster sitting out there, and the three backgroundrb processes. I then proceed to generate a worker using MiddleMan via IRB, get the auto generated job key, look up the worker, and then delete him. All goes well, and the process list still looks like what I posted above. I can do this as many times as I want to via IRB and nothing goes wrong. However, when I try to call the same code through a Rails action, Backgroundrb doesn''t behave. The browser just sits there spinning. Examining the process list yields: cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 ruby: backgroundrb (ruby) cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 ruby: backgroundrb (ruby) cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0:03.36/usr/local/bin/ruby18 /usr/local/bin/mong cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mong cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51/usr/local/bin/ruby18 /usr/local/bin/mong cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mong cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: backgroundrb_logger (ruby) cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: backgroundrb_results (ruby) cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 grep ruby Aha! An extra Backgroundrb process that''s spiking the CPU. I''ve let the browser sit there until it times out, and the Backgroundrb process will still be there, spiking the CPU. If I stop Backgroundrb at this point, the extra process is still there. I have to kill it manually to get rid of it. Something else I''ve noticed is that, once this process is out there, even if I try creating new workers via IRB, the call to MiddleMan.new_worker just sits there. So this definitely appears to be a problem with the way Backgroundrb and my Mongrel clusters are interacting, but I''m not sure where to even begin. It''s frustrating that this isn''t happening on the dev box. I''ve tried to figure out what could be different between the boxes, but other than the Ruby versions, I''m not seeing anything. I''m out of options. Does anyone have any ideas? Thanks, Matthew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070303/3c154ef6/attachment.html
Mason Hale
2007-Mar-04 16:33 UTC
[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster
Matthew - Trying running netstat to see what ports your mongrels and backgroundrb processes are using. I''ve run into some problems with those two trying to use the same ports in past. Mason On 3/3/07, Matthew <pelargir at gmail.com> wrote:> > I''ve been troubleshooting this problem for several hours now and haven''t > been able to figure out what''s wrong. I have two boxes, dev and prod, both > are identical except that prod uses Ruby 1.8.5 and dev uses 1.8.4 . Both > are running Slave 1.2.0, Daemons 1.0.5, and Backroundrb 0.2.1. > > The behavior I''m seeing is this. When I first start Backgroundrb, this is > what the process list looks like: > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48/usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 ruby: > backgroundrb (ruby) > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: > backgroundrb_logger (ruby) > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: > backgroundrb_results (ruby) > > My four Mongrels in the cluster sitting out there, and the three > backgroundrb processes. I then proceed to generate a worker using MiddleMan > via IRB, get the auto generated job key, look up the worker, and then delete > him. All goes well, and the process list still looks like what I posted > above. I can do this as many times as I want to via IRB and nothing goes > wrong. However, when I try to call the same code through a Rails action, > Backgroundrb doesn''t behave. The browser just sits there spinning. Examining > the process list yields: > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 ruby: > backgroundrb (ruby) > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 ruby: > backgroundrb (ruby) > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51/usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: > backgroundrb_logger (ruby) > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: > backgroundrb_results (ruby) > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 grep ruby > > Aha! An extra Backgroundrb process that''s spiking the CPU. I''ve let the > browser sit there until it times out, and the Backgroundrb process will > still be there, spiking the CPU. If I stop Backgroundrb at this point, the > extra process is still there. I have to kill it manually to get rid of it. > > Something else I''ve noticed is that, once this process is out there, even > if I try creating new workers via IRB, the call to MiddleMan.new_workerjust sits there. So this definitely appears to be a problem with the way > Backgroundrb and my Mongrel clusters are interacting, but I''m not sure where > to even begin. It''s frustrating that this isn''t happening on the dev box. > I''ve tried to figure out what could be different between the boxes, but > other than the Ruby versions, I''m not seeing anything. > > I''m out of options. Does anyone have any ideas? > > Thanks, > Matthew > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070304/cc6c1beb/attachment.html
Matthew
2007-Mar-05 20:40 UTC
[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster
Thanks for the suggestion, Mason. I did that and noticed that each time I attempted to create a new worker, I got a new entry. This is what I ended up with in netstat after trying several attempts: -bash-2.05b$ netstat -an | grep 2001 c4cb5af0 stream 0 0 0 c8cb5af0 0 0 ///tmp/backgroundrbunix_localhost_2001 c8d58a64 stream 0 0 0 0 0 0 ///tmp/backgroundrbunix_localhost_2001 c4dcc94c stream 0 0 0 c97ac94c 0 0 ///tmp/backgroundrbunix_localhost_2001 c8d20690 stream 0 0 0 c8cb5000 0 0 ///tmp/backgroundrbunix_localhost_2001 c8cca348 stream 0 0 0 c4dcc690 0 0 ///tmp/backgroundrbunix_localhost_2001 c8cf7b7c stream 0 0 0 c8d209d8 0 0 ///tmp/backgroundrbunix_localhost_2001 c4cb62bc stream 0 0 c6b63d68 0 0 0 ///tmp/backgroundrbunix_localhost_2001 My process list looked like this: -bash-2.05b$ ps aux | grep ruby cwarner 30993 31.9 1.3 46932 39748 ?? R 3:27PM 2:22.28 ruby: pdf_generator_worker_dde6f63034382c4727 cwarner 31131 31.6 1.3 47344 40160 ?? R 3:30PM 0:26.14 ruby: pdf_generator_worker_031025c9be743ca12e cwarner 31079 31.5 1.3 47072 39888 ?? R 3:29PM 0:43.74 ruby: pdf_generator_worker_c617aa5cc57c7b59b7 cwarner 23921 0.1 3.1 103420 97144 ?? S 10:35AM 6:08.78 ruby /usr/local/lib/ruby/gems/1.8/gems/mongre cwarner 23910 0.0 4.5 148668 140424 ?? S 10:35AM 11:28.06 ruby /usr/local/lib/ruby/gems/1.8/gems/mongre cwarner 23914 0.0 3.1 107792 98560 ?? S 10:35AM 7:13.36 ruby /usr/local/lib/ruby/gems/1.8/gems/mongre cwarner 23918 0.0 3.1 106548 97144 ?? S 10:35AM 6:39.64 ruby /usr/local/lib/ruby/gems/1.8/gems/mongre cwarner 30836 0.0 1.3 47480 40300 ?? S 3:24PM 0:02.42 ruby: backgroundrb (ruby) cwarner 30837 0.0 0.7 21512 20552 ?? S 3:24PM 0:00.06 ruby: backgroundrb_logger (ruby) cwarner 30838 0.0 0.7 21468 20560 ?? S 3:24PM 0:00.06 ruby: backgroundrb_results (ruby) cwarner 31162 0.0 0.0 1484 832 p3 S+ 3:31PM 0:00.00 grep ruby Looks like a new pdf_generator_worker process gets created and immediately freezes each time I try to create a worker. I can''t communicate with any of these workers and have to manually kill them to get them to go away. Anyone else have an idea of what could be going on? Matthew On 3/4/07, Mason Hale <masonhale at gmail.com> wrote:> > Matthew - > > Trying running netstat to see what ports your mongrels and backgroundrb > processes are using. I''ve run into some problems with those two trying to > use the same ports in past. > > Mason > > On 3/3/07, Matthew <pelargir at gmail.com> wrote: > > > I''ve been troubleshooting this problem for several hours now and haven''t > > been able to figure out what''s wrong. I have two boxes, dev and prod, both > > are identical except that prod uses Ruby 1.8.5 and dev uses 1.8.4 . Both > > are running Slave 1.2.0, Daemons 1.0.5, and Backroundrb 0.2.1. > > > > The behavior I''m seeing is this. When I first start Backgroundrb, this > > is what the process list looks like: > > > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48/usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 ruby: > > backgroundrb (ruby) > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: > > backgroundrb_logger (ruby) > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: > > backgroundrb_results (ruby) > > > > My four Mongrels in the cluster sitting out there, and the three > > backgroundrb processes. I then proceed to generate a worker using MiddleMan > > via IRB, get the auto generated job key, look up the worker, and then delete > > him. All goes well, and the process list still looks like what I posted > > above. I can do this as many times as I want to via IRB and nothing goes > > wrong. However, when I try to call the same code through a Rails action, > > Backgroundrb doesn''t behave. The browser just sits there spinning. Examining > > the process list yields: > > > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 ruby: > > backgroundrb (ruby) > > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 ruby: > > backgroundrb (ruby) > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51/usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: > > backgroundrb_logger (ruby) > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: > > backgroundrb_results (ruby) > > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 grep > > ruby > > > > Aha! An extra Backgroundrb process that''s spiking the CPU. I''ve let the > > browser sit there until it times out, and the Backgroundrb process will > > still be there, spiking the CPU. If I stop Backgroundrb at this point, the > > extra process is still there. I have to kill it manually to get rid of it. > > > > Something else I''ve noticed is that, once this process is out there, > > even if I try creating new workers via IRB, the call to > > MiddleMan.new_worker just sits there. So this definitely appears to be a > > problem with the way Backgroundrb and my Mongrel clusters are interacting, > > but I''m not sure where to even begin. It''s frustrating that this isn''t > > happening on the dev box. I''ve tried to figure out what could be different > > between the boxes, but other than the Ruby versions, I''m not seeing > > anything. > > > > I''m out of options. Does anyone have any ideas? > > > > Thanks, > > Matthew > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070305/b18ef904/attachment-0001.html
Matthew
2007-Mar-05 20:49 UTC
[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster
Something I''m also noticing that may be related is this message. I get
it
every so often when I try to stop backgroundrb:
-bash-2.05b$ script/backgroundrb stop
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in
`initialize'': No such file or directory -
/usr/home/cwarner/rails/log/backgroundrb.pid (Errno::ENOENT)
from
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in
`open''
from
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in
`pid''
from
/usr/home/cwarner/www/html/rails/test/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:350:in
`stop''
from
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:133:in
`stop_all''
from
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:129:in
`each''
from
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:129:in
`stop_all''
from
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/controller.rb:74:in
`run''
from
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:182:in
`run_proc''
from
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:in
`call''
from
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:in
`catch_exceptions''
from
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in
`run_proc''
from
/usr/home/cwarner/www/html/rails/test/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:301:in
`run''
from script/backgroundrb:29
I''m also seeing a file called "backgroundrb.ppid" (yes, with
two Ps) in my
log dir. "backgroundrb.pid" shows up there when backgroundrb is
running, but
it gets deleted when I stop. On the other hand, "backgroundrb.ppid"
stays
there unless I delete it manually.
Matthew
On 3/5/07, Matthew <pelargir at gmail.com> wrote:>
> Thanks for the suggestion, Mason. I did that and noticed that each time I
> attempted to create a new worker, I got a new entry. This is what I ended
up
> with in netstat after trying several attempts:
>
> -bash-2.05b$ netstat -an | grep 2001
> c4cb5af0 stream 0 0 0 c8cb5af0 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c8d58a64 stream 0 0 0 0 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c4dcc94c stream 0 0 0 c97ac94c 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c8d20690 stream 0 0 0 c8cb5000 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c8cca348 stream 0 0 0 c4dcc690 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c8cf7b7c stream 0 0 0 c8d209d8 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c4cb62bc stream 0 0 c6b63d68 0 0 0
> ///tmp/backgroundrbunix_localhost_2001
>
> My process list looked like this:
>
> -bash-2.05b$ ps aux | grep ruby
> cwarner 30993 31.9 1.3 46932 39748 ?? R 3:27PM 2: 22.28 ruby:
> pdf_generator_worker_dde6f63034382c4727
> cwarner 31131 31.6 1.3 47344 40160 ?? R 3:30PM 0:26.14 ruby:
> pdf_generator_worker_031025c9be743ca12e
> cwarner 31079 31.5 1.3 47072 39888 ?? R 3:29PM 0: 43.74 ruby:
> pdf_generator_worker_c617aa5cc57c7b59b7
> cwarner 23921 0.1 3.1 103420 97144 ?? S 10:35AM 6:08.78 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23910 0.0 4.5 148668 140424 ?? S 10:35AM 11: 28.06 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23914 0.0 3.1 107792 98560 ?? S 10:35AM 7:13.36 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23918 0.0 3.1 106548 97144 ?? S 10:35AM 6: 39.64 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 30836 0.0 1.3 47480 40300 ?? S 3:24PM 0:02.42 ruby:
> backgroundrb (ruby)
> cwarner 30837 0.0 0.7 21512 20552 ?? S 3:24PM 0:00.06 ruby:
> backgroundrb_logger (ruby)
> cwarner 30838 0.0 0.7 21468 20560 ?? S 3:24PM 0:00.06 ruby:
> backgroundrb_results (ruby)
> cwarner 31162 0.0 0.0 1484 832 p3 S+ 3:31PM 0:00.00 grep ruby
>
> Looks like a new pdf_generator_worker process gets created and immediately
> freezes each time I try to create a worker. I can''t communicate
with any of
> these workers and have to manually kill them to get them to go away.
>
> Anyone else have an idea of what could be going on?
>
> Matthew
>
>
> On 3/4/07, Mason Hale <masonhale at gmail.com > wrote:
> >
> > Matthew -
> >
> > Trying running netstat to see what ports your mongrels and
backgroundrb
> > processes are using. I''ve run into some problems with those
two trying to
> > use the same ports in past.
> >
> > Mason
> >
> > On 3/3/07, Matthew <pelargir at gmail.com> wrote:
> >
> > > I''ve been troubleshooting this problem for several
hours now and
> > > haven''t been able to figure out what''s wrong. I
have two boxes, dev and
> > > prod, both are identical except that prod uses Ruby 1.8.5 and dev
uses
> > > 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and
Backroundrb
> > > 0.2.1.
> > >
> > > The behavior I''m seeing is this. When I first start
Backgroundrb, this
> > > is what the process list looks like:
> > >
> > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0:
03.36/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM
0:03.59/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM
0:03.48/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM
0:03.26/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43
ruby:
> > > backgroundrb (ruby)
> > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07
ruby:
> > > backgroundrb_logger (ruby)
> > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06
ruby:
> > > backgroundrb_results (ruby)
> > >
> > > My four Mongrels in the cluster sitting out there, and the three
> > > backgroundrb processes. I then proceed to generate a worker using
MiddleMan
> > > via IRB, get the auto generated job key, look up the worker, and
then delete
> > > him. All goes well, and the process list still looks like what I
posted
> > > above. I can do this as many times as I want to via IRB and
nothing goes
> > > wrong. However, when I try to call the same code through a Rails
action,
> > > Backgroundrb doesn''t behave. The browser just sits there
spinning. Examining
> > > the process list yields:
> > >
> > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07
ruby:
> > > backgroundrb (ruby)
> > > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60
ruby:
> > > backgroundrb (ruby)
> > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0:
03.36/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM
0:03.59/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM
0:03.51/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM
0:03.26/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07
ruby:
> > > backgroundrb_logger (ruby)
> > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06
ruby:
> > > backgroundrb_results (ruby)
> > > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00
grep
> > > ruby
> > >
> > > Aha! An extra Backgroundrb process that''s spiking the
CPU. I''ve let
> > > the browser sit there until it times out, and the Backgroundrb
process will
> > > still be there, spiking the CPU. If I stop Backgroundrb at this
point, the
> > > extra process is still there. I have to kill it manually to get
rid of it.
> > >
> > > Something else I''ve noticed is that, once this process
is out there,
> > > even if I try creating new workers via IRB, the call to
> > > MiddleMan.new_worker just sits there. So this definitely appears
to be
> > > a problem with the way Backgroundrb and my Mongrel clusters are
interacting,
> > > but I''m not sure where to even begin. It''s
frustrating that this isn''t
> > > happening on the dev box. I''ve tried to figure out what
could be different
> > > between the boxes, but other than the Ruby versions, I''m
not seeing
> > > anything.
> > >
> > > I''m out of options. Does anyone have any ideas?
> > >
> > > Thanks,
> > > Matthew
> > >
> > > _______________________________________________
> > > Backgroundrb-devel mailing list
> > > Backgroundrb-devel at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> > >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070305/e1134c2d/attachment.html
Ezra Zygmuntowicz
2007-Mar-05 21:08 UTC
[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster
Mathew- This may be the same worker class name length overflow error. If your classname for your worker is too long then slave can fail to create the proper unix domain socket. Can you try to shorten the name of your worker class and try again? -Ezra On Mar 5, 2007, at 12:40 PM, Matthew wrote:> Thanks for the suggestion, Mason. I did that and noticed that each > time I attempted to create a new worker, I got a new entry. This is > what I ended up with in netstat after trying several attempts: > > -bash-2.05b$ netstat -an | grep 2001 > c4cb5af0 stream 0 0 0 c8cb5af0 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c8d58a64 stream 0 0 0 0 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c4dcc94c stream 0 0 0 c97ac94c 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c8d20690 stream 0 0 0 c8cb5000 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c8cca348 stream 0 0 0 c4dcc690 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c8cf7b7c stream 0 0 0 c8d209d8 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c4cb62bc stream 0 0 c6b63d68 0 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > > My process list looked like this: > > -bash-2.05b$ ps aux | grep ruby > cwarner 30993 31.9 1.3 46932 39748 ?? R 3:27PM 2: 22.28 > ruby: pdf_generator_worker_dde6f63034382c4727 > cwarner 31131 31.6 1.3 47344 40160 ?? R 3:30PM 0:26.14 > ruby: pdf_generator_worker_031025c9be743ca12e > cwarner 31079 31.5 1.3 47072 39888 ?? R 3:29PM 0: 43.74 > ruby: pdf_generator_worker_c617aa5cc57c7b59b7 > cwarner 23921 0.1 3.1 103420 97144 ?? S 10:35AM 6:08.78 > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 23910 0.0 4.5 148668 140424 ?? S 10:35AM 11: 28.06 > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 23914 0.0 3.1 107792 98560 ?? S 10:35AM 7:13.36 > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 23918 0.0 3.1 106548 97144 ?? S 10:35AM 6: 39.64 > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 30836 0.0 1.3 47480 40300 ?? S 3:24PM 0:02.42 > ruby: backgroundrb (ruby) > cwarner 30837 0.0 0.7 21512 20552 ?? S 3:24PM 0:00.06 > ruby: backgroundrb_logger (ruby) > cwarner 30838 0.0 0.7 21468 20560 ?? S 3:24PM 0:00.06 > ruby: backgroundrb_results (ruby) > cwarner 31162 0.0 0.0 1484 832 p3 S+ 3:31PM 0:00.00 > grep ruby > > Looks like a new pdf_generator_worker process gets created and > immediately freezes each time I try to create a worker. I can''t > communicate with any of these workers and have to manually kill > them to get them to go away. > > Anyone else have an idea of what could be going on? > > Matthew > > > On 3/4/07, Mason Hale <masonhale at gmail.com > wrote:Matthew - > > Trying running netstat to see what ports your mongrels and > backgroundrb processes are using. I''ve run into some problems with > those two trying to use the same ports in past. > > Mason > > On 3/3/07, Matthew <pelargir at gmail.com> wrote: > I''ve been troubleshooting this problem for several hours now and > haven''t been able to figure out what''s wrong. I have two boxes, dev > and prod, both are identical except that prod uses Ruby 1.8.5 and > dev uses 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and > Backroundrb 0.2.1. > > The behavior I''m seeing is this. When I first start Backgroundrb, > this is what the process list looks like: > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48 / > usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 > ruby: backgroundrb (ruby) > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > ruby: backgroundrb_logger (ruby) > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > ruby: backgroundrb_results (ruby) > > My four Mongrels in the cluster sitting out there, and the three > backgroundrb processes. I then proceed to generate a worker using > MiddleMan via IRB, get the auto generated job key, look up the > worker, and then delete him. All goes well, and the process list > still looks like what I posted above. I can do this as many times > as I want to via IRB and nothing goes wrong. However, when I try to > call the same code through a Rails action, Backgroundrb doesn''t > behave. The browser just sits there spinning. Examining the process > list yields: > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 > ruby: backgroundrb (ruby) > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 > ruby: backgroundrb (ruby) > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51 / > usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > ruby: backgroundrb_logger (ruby) > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > ruby: backgroundrb_results (ruby) > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 > grep ruby > > Aha! An extra Backgroundrb process that''s spiking the CPU. I''ve let > the browser sit there until it times out, and the Backgroundrb > process will still be there, spiking the CPU. If I stop > Backgroundrb at this point, the extra process is still there. I > have to kill it manually to get rid of it. > > Something else I''ve noticed is that, once this process is out > there, even if I try creating new workers via IRB, the call to > MiddleMan.new_worker just sits there. So this definitely appears to > be a problem with the way Backgroundrb and my Mongrel clusters are > interacting, but I''m not sure where to even begin. It''s frustrating > that this isn''t happening on the dev box. I''ve tried to figure out > what could be different between the boxes, but other than the Ruby > versions, I''m not seeing anything. > > I''m out of options. Does anyone have any ideas? > > Thanks, > Matthew > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel-- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273)
Matthew
2007-Mar-05 21:25 UTC
[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster
Thanks for the reply, Ezra. Just tried that and I''m still experiencing the same problem. As before, running from either script/console or the browser works fine on the first go round. I can generate a worker, query it, and delete it. Trying to generate a worker the second time is a different story, though. In the browser the generate appears to work, but when I try to get a worker for the returned key I get a nil object. In script/console, the generate just sits there spinning until I kill the backgroundrb process. Something interesting I noticed as well is the dir that gets created in /tmp with the PID on the end does not match the actual PID of my backgroundrb process. Is it supposed to? -bash-2.05b$ ls /tmp/ba* /tmp/backgroundrbunix_localhost_2020 /tmp/backgroundrb.32589 -bash-2.05b$ ps aux | grep backgroundrb cwarner 32591 0.0 0.7 21528 20564 ?? S 4:13PM 0:00.17 ruby: backgroundrb (ruby) cwarner 32592 0.0 0.7 21512 20552 ?? S 4:13PM 0:00.06 ruby: backgroundrb_logger (ruby) cwarner 32593 0.0 0.7 21468 20560 ?? S 4:13PM 0:00.05 ruby: backgroundrb_results (ruby) On an aside, thanks for building a really great tool here. It''s simple to use and perfect for what I''m trying to do. I''m sure the issue is on my end with configuration or something. Matthew On 3/5/07, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:> > Mathew- > > This may be the same worker class name length overflow error. If > your classname for your worker is too long then slave can fail to > create the proper unix domain socket. Can you try to shorten the name > of your worker class and try again? > > -Ezra > > > On Mar 5, 2007, at 12:40 PM, Matthew wrote: > > > Thanks for the suggestion, Mason. I did that and noticed that each > > time I attempted to create a new worker, I got a new entry. This is > > what I ended up with in netstat after trying several attempts: > > > > -bash-2.05b$ netstat -an | grep 2001 > > c4cb5af0 stream 0 0 0 c8cb5af0 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c8d58a64 stream 0 0 0 0 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c4dcc94c stream 0 0 0 c97ac94c 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c8d20690 stream 0 0 0 c8cb5000 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c8cca348 stream 0 0 0 c4dcc690 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c8cf7b7c stream 0 0 0 c8d209d8 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c4cb62bc stream 0 0 c6b63d68 0 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > > My process list looked like this: > > > > -bash-2.05b$ ps aux | grep ruby > > cwarner 30993 31.9 1.3 46932 39748 ?? R 3:27PM 2: 22.28 > > ruby: pdf_generator_worker_dde6f63034382c4727 > > cwarner 31131 31.6 1.3 47344 40160 ?? R 3:30PM 0:26.14 > > ruby: pdf_generator_worker_031025c9be743ca12e > > cwarner 31079 31.5 1.3 47072 39888 ?? R 3:29PM 0: 43.74 > > ruby: pdf_generator_worker_c617aa5cc57c7b59b7 > > cwarner 23921 0.1 3.1 103420 97144 ?? S 10:35AM 6:08.78 > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > cwarner 23910 0.0 4.5 148668 140424 ?? S 10:35AM 11: 28.06 > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > cwarner 23914 0.0 3.1 107792 98560 ?? S 10:35AM 7:13.36 > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > cwarner 23918 0.0 3.1 106548 97144 ?? S 10:35AM 6: 39.64 > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > cwarner 30836 0.0 1.3 47480 40300 ?? S 3:24PM 0:02.42 > > ruby: backgroundrb (ruby) > > cwarner 30837 0.0 0.7 21512 20552 ?? S 3:24PM 0:00.06 > > ruby: backgroundrb_logger (ruby) > > cwarner 30838 0.0 0.7 21468 20560 ?? S 3:24PM 0:00.06 > > ruby: backgroundrb_results (ruby) > > cwarner 31162 0.0 0.0 1484 832 p3 S+ 3:31PM 0:00.00 > > grep ruby > > > > Looks like a new pdf_generator_worker process gets created and > > immediately freezes each time I try to create a worker. I can''t > > communicate with any of these workers and have to manually kill > > them to get them to go away. > > > > Anyone else have an idea of what could be going on? > > > > Matthew > > > > > > On 3/4/07, Mason Hale <masonhale at gmail.com > wrote:Matthew - > > > > Trying running netstat to see what ports your mongrels and > > backgroundrb processes are using. I''ve run into some problems with > > those two trying to use the same ports in past. > > > > Mason > > > > On 3/3/07, Matthew <pelargir at gmail.com> wrote: > > I''ve been troubleshooting this problem for several hours now and > > haven''t been able to figure out what''s wrong. I have two boxes, dev > > and prod, both are identical except that prod uses Ruby 1.8.5 and > > dev uses 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and > > Backroundrb 0.2.1. > > > > The behavior I''m seeing is this. When I first start Backgroundrb, > > this is what the process list looks like: > > > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > > usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > > usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48 / > > usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > > usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 > > ruby: backgroundrb (ruby) > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > > ruby: backgroundrb_logger (ruby) > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > > ruby: backgroundrb_results (ruby) > > > > My four Mongrels in the cluster sitting out there, and the three > > backgroundrb processes. I then proceed to generate a worker using > > MiddleMan via IRB, get the auto generated job key, look up the > > worker, and then delete him. All goes well, and the process list > > still looks like what I posted above. I can do this as many times > > as I want to via IRB and nothing goes wrong. However, when I try to > > call the same code through a Rails action, Backgroundrb doesn''t > > behave. The browser just sits there spinning. Examining the process > > list yields: > > > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 > > ruby: backgroundrb (ruby) > > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 > > ruby: backgroundrb (ruby) > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > > usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > > usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51 / > > usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > > usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > > ruby: backgroundrb_logger (ruby) > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > > ruby: backgroundrb_results (ruby) > > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 > > grep ruby > > > > Aha! An extra Backgroundrb process that''s spiking the CPU. I''ve let > > the browser sit there until it times out, and the Backgroundrb > > process will still be there, spiking the CPU. If I stop > > Backgroundrb at this point, the extra process is still there. I > > have to kill it manually to get rid of it. > > > > Something else I''ve noticed is that, once this process is out > > there, even if I try creating new workers via IRB, the call to > > MiddleMan.new_worker just sits there. So this definitely appears to > > be a problem with the way Backgroundrb and my Mongrel clusters are > > interacting, but I''m not sure where to even begin. It''s frustrating > > that this isn''t happening on the dev box. I''ve tried to figure out > > what could be different between the boxes, but other than the Ruby > > versions, I''m not seeing anything. > > > > I''m out of options. Does anyone have any ideas? > > > > Thanks, > > Matthew > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070305/4fafd351/attachment-0001.html
Matthew
2007-Mar-07 19:23 UTC
[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster
Good news. I solved the problem by downgrading to Ruby 1.8.4 to match the dev box. Backgroundrb immediately started behaving correctly. There must be some conflict between Backgroundrb and Mongrel when running under Ruby 1.8.5on FreeBSD. I''m running 1.8.5 locally and Backgroundrb worked. Very strange, but I''m very glad to have this resolved now. Backgroundrb is quite solid on the prod box now. I have another question, but I''m going to post it in a new thread since it has nothing to do with this topic. Matthew On 3/5/07, Matthew <pelargir at gmail.com> wrote:> > Thanks for the reply, Ezra. Just tried that and I''m still experiencing the > same problem. > > As before, running from either script/console or the browser works fine on > the first go round. I can generate a worker, query it, and delete it. Trying > to generate a worker the second time is a different story, though. In the > browser the generate appears to work, but when I try to get a worker for the > returned key I get a nil object. In script/console, the generate just sits > there spinning until I kill the backgroundrb process. > > Something interesting I noticed as well is the dir that gets created in > /tmp with the PID on the end does not match the actual PID of my > backgroundrb process. Is it supposed to? > > -bash-2.05b$ ls /tmp/ba* > /tmp/backgroundrbunix_localhost_2020 > /tmp/backgroundrb.32589 > > -bash-2.05b$ ps aux | grep backgroundrb > cwarner 32591 0.0 0.7 21528 20564 ?? S 4:13PM 0:00.17 ruby: > backgroundrb (ruby) > cwarner 32592 0.0 0.7 21512 20552 ?? S 4:13PM 0:00.06 ruby: > backgroundrb_logger (ruby) > cwarner 32593 0.0 0.7 21468 20560 ?? S 4:13PM 0:00.05 ruby: > backgroundrb_results (ruby) > > > On an aside, thanks for building a really great tool here. It''s simple to > use and perfect for what I''m trying to do. I''m sure the issue is on my end > with configuration or something. > > Matthew > > > On 3/5/07, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote: > > > > Mathew- > > > > This may be the same worker class name length overflow error. If > > your classname for your worker is too long then slave can fail to > > create the proper unix domain socket. Can you try to shorten the name > > of your worker class and try again? > > > > -Ezra > > > > > > On Mar 5, 2007, at 12:40 PM, Matthew wrote: > > > > > Thanks for the suggestion, Mason. I did that and noticed that each > > > time I attempted to create a new worker, I got a new entry. This is > > > what I ended up with in netstat after trying several attempts: > > > > > > -bash-2.05b$ netstat -an | grep 2001 > > > c4cb5af0 stream 0 0 0 c8cb5af0 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c8d58a64 stream 0 0 0 0 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c4dcc94c stream 0 0 0 c97ac94c 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c8d20690 stream 0 0 0 c8cb5000 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c8cca348 stream 0 0 0 c4dcc690 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c8cf7b7c stream 0 0 0 c8d209d8 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c4cb62bc stream 0 0 c6b63d68 0 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > > > > My process list looked like this: > > > > > > -bash-2.05b$ ps aux | grep ruby > > > cwarner 30993 31.9 1.3 46932 39748 ?? R 3:27PM 2: 22.28 > > > ruby: pdf_generator_worker_dde6f63034382c4727 > > > cwarner 31131 31.6 1.3 47344 40160 ?? R 3:30PM 0:26.14 > > > ruby: pdf_generator_worker_031025c9be743ca12e > > > cwarner 31079 31.5 1.3 47072 39888 ?? R 3:29PM 0: 43.74 > > > ruby: pdf_generator_worker_c617aa5cc57c7b59b7 > > > cwarner 23921 0.1 3.1 103420 97144 ?? S 10:35AM 6:08.78 > > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > > cwarner 23910 0.0 4.5 148668 140424 ?? S 10:35AM 11: 28.06 > > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > > cwarner 23914 0.0 3.1 107792 98560 ?? S 10:35AM 7:13.36 > > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > > cwarner 23918 0.0 3.1 106548 97144 ?? S 10:35AM 6: 39.64 > > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > > cwarner 30836 0.0 1.3 47480 40300 ?? S 3:24PM 0:02.42 > > > ruby: backgroundrb (ruby) > > > cwarner 30837 0.0 0.7 21512 20552 ?? S 3:24PM 0:00.06 > > > ruby: backgroundrb_logger (ruby) > > > cwarner 30838 0.0 0.7 21468 20560 ?? S 3:24PM 0:00.06 > > > ruby: backgroundrb_results (ruby) > > > cwarner 31162 0.0 0.0 1484 832 p3 S+ 3:31PM 0:00.00 > > > grep ruby > > > > > > Looks like a new pdf_generator_worker process gets created and > > > immediately freezes each time I try to create a worker. I can''t > > > communicate with any of these workers and have to manually kill > > > them to get them to go away. > > > > > > Anyone else have an idea of what could be going on? > > > > > > Matthew > > > > > > > > > On 3/4/07, Mason Hale <masonhale at gmail.com > wrote:Matthew - > > > > > > Trying running netstat to see what ports your mongrels and > > > backgroundrb processes are using. I''ve run into some problems with > > > those two trying to use the same ports in past. > > > > > > Mason > > > > > > On 3/3/07, Matthew < pelargir at gmail.com> wrote: > > > I''ve been troubleshooting this problem for several hours now and > > > haven''t been able to figure out what''s wrong. I have two boxes, dev > > > and prod, both are identical except that prod uses Ruby 1.8.5 and > > > dev uses 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and > > > Backroundrb 0.2.1. > > > > > > The behavior I''m seeing is this. When I first start Backgroundrb, > > > this is what the process list looks like: > > > > > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > > > usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > > > usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48 / > > > usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > > > usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 > > > ruby: backgroundrb (ruby) > > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > > > ruby: backgroundrb_logger (ruby) > > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > > > ruby: backgroundrb_results (ruby) > > > > > > My four Mongrels in the cluster sitting out there, and the three > > > backgroundrb processes. I then proceed to generate a worker using > > > MiddleMan via IRB, get the auto generated job key, look up the > > > worker, and then delete him. All goes well, and the process list > > > still looks like what I posted above. I can do this as many times > > > as I want to via IRB and nothing goes wrong. However, when I try to > > > call the same code through a Rails action, Backgroundrb doesn''t > > > behave. The browser just sits there spinning. Examining the process > > > list yields: > > > > > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 > > > ruby: backgroundrb (ruby) > > > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 > > > ruby: backgroundrb (ruby) > > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > > > usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > > > usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51 / > > > usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > > > usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > > > ruby: backgroundrb_logger (ruby) > > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > > > ruby: backgroundrb_results (ruby) > > > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 > > > grep ruby > > > > > > Aha! An extra Backgroundrb process that''s spiking the CPU. I''ve let > > > the browser sit there until it times out, and the Backgroundrb > > > process will still be there, spiking the CPU. If I stop > > > Backgroundrb at this point, the extra process is still there. I > > > have to kill it manually to get rid of it. > > > > > > Something else I''ve noticed is that, once this process is out > > > there, even if I try creating new workers via IRB, the call to > > > MiddleMan.new_worker just sits there. So this definitely appears to > > > be a problem with the way Backgroundrb and my Mongrel clusters are > > > interacting, but I''m not sure where to even begin. It''s frustrating > > > that this isn''t happening on the dev box. I''ve tried to figure out > > > what could be different between the boxes, but other than the Ruby > > > versions, I''m not seeing anything. > > > > > > I''m out of options. Does anyone have any ideas? > > > > > > Thanks, > > > Matthew > > > > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > -- Ezra Zygmuntowicz > > -- Lead Rails Evangelist > > -- ez at engineyard.com > > -- Engine Yard, Serious Rails Hosting > > -- (866) 518-YARD (9273) > > > > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070307/0354fced/attachment-0001.html