Jonathan del Strother
2007-Feb-07 13:45 UTC
[Backgroundrb-devel] Repeatedly dying with "failed to find slave socket"
I''m struggling to keep backgroundrb v2 running for more than 24 hours. It appears to be running fine for a while, then tries to fire off a worker (exactly the same worker it''s been running for the past few hours), and it suddenly dies. I haven''t found any pattern for the cause of this. In backgroundrb_server.log, the final entry is this : 20070206-10:41:45 (5189) Schedule triggered: #<struct #<Class: 0x2afb643b56c0> job=#<Proc:0x00002afb63ed83c8 at .../vendor/plugins/ backgroundrb/server/lib/backgroundrb/middleman.rb:355>, trigger=#<BackgrounDRb::Trigger:0x2afb6546ccc0 @repeat_interval=600, @end_time=nil, @start_time=Tue Feb 06 03:41:45 -0800 2007>, earliest=Tue Feb 06 10:41:45 -0800 2007, last=Tue Feb 06 10:41:45 -0800 2007> 20070206-10:41:45 (5189) failed to find slave socket - (RuntimeError) 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/slave-1.2.0/lib/ slave.rb:435:in `initialize'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/middleman.rb:210:in `new'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/middleman.rb:210:in `new_worker'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/thread_pool.rb:36:in `dispatch'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/thread_pool.rb:22:in `initialize'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/thread_pool.rb:22:in `new'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/thread_pool.rb:22:in `dispatch'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/middleman.rb:199:in `new_worker'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/middleman.rb:357:in `schedule_worker'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:46:in `call'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:46:in `run'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:40:in `each'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:40:in `run'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:35:in `loop'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:35:in `run'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:95:in `run'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:94:in `initialize'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:94:in `new'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/scheduler.rb:94:in `run'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb/middleman.rb:106:in `setup'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb_server.rb:306:in `run'' 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ lib/daemons/application.rb:186:in `call'' 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ lib/daemons/application.rb:186:in `start_proc'' 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ lib/daemons/daemonize.rb:192:in `call'' 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ lib/daemons/daemonize.rb:192:in `call_as_daemon'' 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ lib/daemons/application.rb:190:in `start_proc'' 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ lib/daemons/application.rb:226:in `start'' 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ lib/daemons/controller.rb:69:in `run'' 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ lib/daemons.rb:182:in `run_proc'' 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ lib/daemons/cmdline.rb:94:in `call'' 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ lib/daemons/cmdline.rb:94:in `catch_exceptions'' 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ lib/daemons.rb:181:in `run_proc'' 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ backgroundrb_server.rb:301:in `run'' 20070206-10:41:45 (5189) .../script/backgroundrb:29 This happens both on my EngineYard server, and running locally on intel OS X 10.4.8. Any suggestions? Anyone else experiencing this? Jon
Mason Hale
2007-Feb-07 16:16 UTC
[Backgroundrb-devel] Repeatedly dying with "failed to find slave socket"
Hi Jon - I have see similar errors like this in the past, but I don''t have any in my backgroundrb log now. I didn''t specifically track down and fix this error, but I made some other changes that may have inadvertantly avoided this problem. Those changes were: 1.) Avoid using the results feature at all. All status from workers is stored in the db now. 2.) Call self.delete at the end of the worker doing its work 3.) Simplifying the workers themselves. They now just load arguments and call other classes with those arguments. (This step also makes testing easier). I have a scheduled worker that is long running and runs once per minute. That worker looks for pending jobs and spawns new workers to handle those jobs as needed. Those spawner workers take from a few minutes to several hours to do their work before they finish. We have up to 10 workers running at a time. We also have a cron job that checks every minute to see if the scheduled worker has run in the last 3 minutes. If not, it restarts the backgroundrb server and sends a notification email. It''s been a little over a week now since the backgroundrb server was auto-restarted. Mason On 2/7/07, Jonathan del Strother <maillist at steelskies.com> wrote:> I''m struggling to keep backgroundrb v2 running for more than 24 > hours. It appears to be running fine for a while, then tries to fire > off a worker (exactly the same worker it''s been running for the past > few hours), and it suddenly dies. I haven''t found any pattern for the > cause of this. > > In backgroundrb_server.log, the final entry is this : > > 20070206-10:41:45 (5189) Schedule triggered: #<struct #<Class: > 0x2afb643b56c0> job=#<Proc:0x00002afb63ed83c8 at .../vendor/plugins/ > backgroundrb/server/lib/backgroundrb/middleman.rb:355>, > trigger=#<BackgrounDRb::Trigger:0x2afb6546ccc0 @repeat_interval=600, > @end_time=nil, @start_time=Tue Feb 06 03:41:45 -0800 2007>, > earliest=Tue Feb 06 10:41:45 -0800 2007, last=Tue Feb 06 10:41:45 > -0800 2007> > 20070206-10:41:45 (5189) failed to find slave socket - (RuntimeError) > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/slave-1.2.0/lib/ > slave.rb:435:in `initialize'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/middleman.rb:210:in `new'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/middleman.rb:210:in `new_worker'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/thread_pool.rb:36:in `dispatch'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/thread_pool.rb:22:in `initialize'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/thread_pool.rb:22:in `new'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/thread_pool.rb:22:in `dispatch'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/middleman.rb:199:in `new_worker'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/middleman.rb:357:in `schedule_worker'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/scheduler.rb:46:in `call'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/scheduler.rb:46:in `run'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/scheduler.rb:40:in `each'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/scheduler.rb:40:in `run'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/scheduler.rb:35:in `loop'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/scheduler.rb:35:in `run'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/scheduler.rb:95:in `run'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/scheduler.rb:94:in `initialize'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/scheduler.rb:94:in `new'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/scheduler.rb:94:in `run'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb/middleman.rb:106:in `setup'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb_server.rb:306:in `run'' > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/application.rb:186:in `call'' > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/application.rb:186:in `start_proc'' > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/daemonize.rb:192:in `call'' > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/daemonize.rb:192:in `call_as_daemon'' > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/application.rb:190:in `start_proc'' > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/application.rb:226:in `start'' > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/controller.rb:69:in `run'' > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons.rb:182:in `run_proc'' > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/cmdline.rb:94:in `call'' > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/cmdline.rb:94:in `catch_exceptions'' > 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons.rb:181:in `run_proc'' > 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/ > backgroundrb_server.rb:301:in `run'' > 20070206-10:41:45 (5189) .../script/backgroundrb:29 > > > > This happens both on my EngineYard server, and running locally on > intel OS X 10.4.8. > > > Any suggestions? Anyone else experiencing this? > > Jon > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel >