Greetings all, Unit tests are passing, but I''m having mucho problems getting brb to run without exceptions for more than 6 hours. I''m seeing a mixture of the following 2 exceptions: 20070605-11:26:56 (21497) failed to find slave socket - (RuntimeError) 20070605-11:26:56 (21497) /usr/local/lib/ruby/gems/1.8/gems/ slave-1.2.1/lib/slave.rb:435:in `initialize'' 20070605-11:26:56 (21497) /var/www/html/the-soup/vendor/plugins/ backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new_worker'' and 20070605-11:26:56 (21497) ~~~~~~~~~ Can''t find job! ''job_key_1'' - (Exception) 20070605-11:26:56 (21497) undefined method `initial_do_work'' for []:Array - (NoMethodError) 20070605-11:26:56 (21497) /var/www/html/the-soup/vendor/plugins/ backgroundrb/server/lib/backgroundrb/middleman.rb:342:in `schedule_worker'' (the above is a check I added when a job key lookup fails) environment: - RHEL 4 - ruby 1.8.4 (2005-12-24) [x86_64-linux] - rails 1.2.3 - slave (1.2.1) - daemons (1.0.5) - not using middleman - only the scheduler - mysql 5 The tasks are scheduled to run every hour, and don''t run for more than a few minutes. Talking with Ezra at railsconf he indicated that a new team member was joining - can I provide any info to help debug this? Jodi
Re: Jodi''s post, I''m having the same problem just today (don''t think I''ve had this before), and only when I feed it a custom background.yml file. Here''s the error: 20070627-11:36:28 (1804) Starting BackgrounDRb Server 20070627-11:36:28 (1804) config: RAILS_ROOT/config/backgroundrb.yml 20070627-11:36:28 (1804) rails_env: development 20070627-11:36:28 (1804) socket_dir: RAILS_ROOT/tmp/backgroundrb.1804 20070627-11:36:28 (1804) pool_size: 5 20070627-11:36:28 (1804) host: localhost 20070627-11:36:28 (1804) acl: orderdenyallowdenyallallowlocalhost 127.0.0.1 20070627-11:36:28 (1804) temp_dir: RAILS_ROOT/tmp 20070627-11:36:28 (1804) port: 2000 20070627-11:36:28 (1804) worker_dir: RAILS_ROOT/lib/workers 20070627-11:36:28 (1804) protocol: drbunix 20070627-11:36:28 (1804) uri: drbunix:///RAILS_ROOT/tmp/backgroundrbunix_localhost_2000 20070627-11:36:29 (1806) Starting worker: BackgrounDRb::Worker::WorkerLogger backgroundrb_logger (backgroundrb_logger) () 20070627-11:36:29 (1806) failed to find slave socket - (RuntimeError) 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/slave-1.2.1/lib/slave.rb:435:in `initialize'' 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new'' 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new_worker'' 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:36:in `dispatch'' 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `initialize'' 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `new'' 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `dispatch'' 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:199:in `new_worker'' 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:103:in `setup'' 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:306:in `run'' 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.6/lib/daemons/application.rb:193:in `call'' 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.6/lib/daemons/application.rb:193:in `start_proc'' 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.6/lib/daemons/daemonize.rb:192:in `call'' 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.6/lib/daemons/daemonize.rb:192:in `call_as_daemon'' 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.6/lib/daemons/application.rb:197:in `start_proc'' 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.6/lib/daemons/application.rb:233:in `start'' 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.6/lib/daemons/controller.rb:69:in `run'' 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.6/lib/daemons.rb:185:in `run_proc'' 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.6/lib/daemons/cmdline.rb:105:in `call'' 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.6/lib/daemons/cmdline.rb:105:in `catch_exceptions'' 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.6/lib/daemons.rb:184:in `run_proc'' 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:301:in `run'' 20070627-11:36:29 (1806) script/backgroundrb:29 20070627-11:36:29 (1806) Loading Worker Class File: RAILS_ROOT/lib/workers/my_worker.rb And I only used a minimal config file to change the tmp directory: --- :temp_dir: "RAILS_ROOT/tmp" (the RAILS_ROOT in all of this is just a replacement for the paths to my app) I just noticed this problem and haven''t even started to dig into it yet, but it looks like feeding it a custom temp dir messes up the slave somehow? I tried replacing the temp_dir via the command line args and it still happens. environment: - OSX 10.4.10 - ruby 1.8.4 (2005-12-24) [x86_64-linux] - ruby 1.8.5 (2006-12-25 patchlevel 12) [i686-darwin8.8.5] - rails EDGE - slave (1.2.1) - daemons (1.0.6) - mysql 5.0.27 -tieg -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070627/aff60354/attachment.html
Greetings Tieg, On 27-Jun-07, at 11:48 AM, Tieg Zaharia wrote:> Re: Jodi''s post, > > I''m having the same problem just today (don''t think I''ve had this > before), and only when I feed it a custom background.yml file. > Here''s the error: > > 20070627-11:36:28 (1804) Starting BackgrounDRb Server > 20070627-11:36:28 (1804) config: RAILS_ROOT/config/backgroundrb.yml > 20070627-11:36:28 (1804) rails_env: development > 20070627-11:36:28 (1804) socket_dir: RAILS_ROOT/tmp/backgroundrb.1804 > 20070627-11:36:28 (1804) pool_size: 5 > 20070627-11:36:28 (1804) host: localhost > 20070627-11:36:28 (1804) acl: orderdenyallowdenyallallowlocalhost > 127.0.0.1 > 20070627-11:36:28 (1804) temp_dir: RAILS_ROOT/tmp > 20070627-11:36:28 (1804) port: 2000 > 20070627-11:36:28 (1804) worker_dir: RAILS_ROOT/lib/workers > 20070627-11:36:28 (1804) protocol: drbunix > 20070627-11:36:28 (1804) uri: drbunix:///RAILS_ROOT/tmp/ > backgroundrbunix_localhost_2000 > 20070627-11:36:29 (1806) Starting worker: > BackgrounDRb::Worker::WorkerLogger backgroundrb_logger > (backgroundrb_logger) () > 20070627-11:36:29 (1806) failed to find slave socket - (RuntimeError) > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > slave-1.2.1/lib/slave.rb:435:in `initialize'' > 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/middleman.rb:210:in `new'' > 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/middleman.rb:210:in `new_worker'' > 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/thread_pool.rb:36:in `dispatch'' > 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/thread_pool.rb:22:in `initialize'' > 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/thread_pool.rb:22:in `new'' > 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/thread_pool.rb:22:in `dispatch'' > 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/middleman.rb:199:in `new_worker'' > 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/middleman.rb:103:in `setup'' > 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ > server/lib/backgroundrb_server.rb:306:in `run'' > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > daemons-1.0.6/lib/daemons/application.rb:193:in `call'' > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > daemons-1.0.6/lib/daemons/application.rb:193:in `start_proc'' > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > daemons-1.0.6/lib/daemons/daemonize.rb:192:in `call'' > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > daemons-1.0.6/lib/daemons/daemonize.rb:192:in `call_as_daemon'' > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > daemons-1.0.6/lib/daemons/application.rb:197:in `start_proc'' > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > daemons-1.0.6/lib/daemons/application.rb:233:in `start'' > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > daemons-1.0.6/lib/daemons/controller.rb:69:in `run'' > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > daemons-1.0.6/lib/daemons.rb:185:in `run_proc'' > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > daemons-1.0.6/lib/daemons/cmdline.rb:105:in `call'' > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > daemons-1.0.6/lib/daemons/cmdline.rb:105:in `catch_exceptions'' > 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ > daemons-1.0.6/lib/daemons.rb:184:in `run_proc'' > 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ > server/lib/backgroundrb_server.rb:301:in `run'' > 20070627-11:36:29 (1806) script/backgroundrb:29 > 20070627-11:36:29 (1806) Loading Worker Class File: RAILS_ROOT/lib/ > workers/my_worker.rb > > And I only used a minimal config file to change the tmp directory: > --- > :temp_dir: "RAILS_ROOT/tmp" > > (the RAILS_ROOT in all of this is just a replacement for the paths > to my app) > I just noticed this problem and haven''t even started to dig into it > yet, but it looks like feeding it a custom temp dir messes up the > slave somehow? I tried replacing the temp_dir via the command line > args and it still happens. > > environment: > - OSX 10.4.10 > - ruby 1.8.4 (2005-12-24) [x86_64-linux] > - ruby 1.8.5 (2006-12-25 patchlevel 12) [i686-darwin8.8.5] > - rails EDGE > - slave (1.2.1) > - daemons (1.0.6) > - mysql 5.0.27 >I have resorted to using Monit to keep backgroundrb up. Last week the app went 4 days without downtime - woop! I''m quite convinced it''s not an app specific problem - my tests are quite complete. And dev, staging and production show the same symptoms - unpredictable failures. The only guess I have so far is that it maybe a resource contention issue - as long as I keep. My brb tasks ratchet up the ram usage to +80%. Funny (not so actually) just checked my prod box and cp was at +95%, and 3 separate backgroundrb tasks were running (the dangers of using monit to heal processes!) good luck Jodi -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070630/6d791709/attachment.html