Pascal Rettig
2007-Apr-09 18:01 UTC
[Backgroundrb-devel] Drb Connection error on multiple dispatch.fcgi ''s
Hi All, I''m using Backgroundrb as a general purpose long-running-task back-end (upload processing, email sending, etc), and it''s been a great solution. However I''ve recently run into some some intermittent connection issues that have me baffled. I''m running on apache2/fcgid and the problem occurs in both devlopment and production mode. The problem seems to only trigger when there''s enough load on the server to create a second dispatch.fcgi instance. When this happens, all of a sudden 1/2 of my new workers fail. I haven''t tried a mongrel setup but I found a message from January 30th from jacobatzen.dk that had the same backtrace error, so I don''t think it''s uniquely a fcgid issue. Anyone come across this before or have any ideas where to look to fix? The worker I''m using for testing right now just spits out a log message (code at the bottom). Thanks, -Pascal>From log/background_server.log :Connection reset by peer - (DRb::DRbConnError) /usr/lib/ruby/1.8/drb/drb.rb:563:in `read'' /usr/lib/ruby/1.8/drb/drb.rb:563:in `load'' /usr/lib/ruby/1.8/drb/drb.rb:629:in `recv_reply'' /usr/lib/ruby/1.8/drb/drb.rb:918:in `recv_reply'' /usr/lib/ruby/1.8/drb/drb.rb:1192:in `send_message'' /usr/lib/ruby/1.8/drb/drb.rb:1083:in `method_missing'' /usr/lib/ruby/1.8/drb/drb.rb:1167:in `open'' /usr/lib/ruby/1.8/drb/drb.rb:1082:in `method_missing'' /usr/lib/ruby/1.8/drb/drb.rb:1100:in `with_friend'' /usr/lib/ruby/1.8/drb/drb.rb:1081:in `method_missing'' /usr/lib/ruby/1.8/drb/drb.rb:1069:in `respond_to?'' /usr/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:454:in `initialize'' /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new_worker'' /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:36:in `dispatch'' /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `dispatch'' /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:199:in `new_worker'' /usr/lib/ruby/1.8/drb/drb.rb:1552:in `perform_without_block'' /usr/lib/ruby/1.8/drb/drb.rb:1512:in `perform'' /usr/lib/ruby/1.8/drb/drb.rb:1586:in `main_loop'' /usr/lib/ruby/1.8/drb/drb.rb:1582:in `main_loop'' /usr/lib/ruby/1.8/drb/drb.rb:1578:in `main_loop'' /usr/lib/ruby/1.8/drb/drb.rb:1427:in `run'' /usr/lib/ruby/1.8/drb/drb.rb:1424:in `run'' /usr/lib/ruby/1.8/drb/drb.rb:1344:in `initialize'' /usr/lib/ruby/1.8/drb/drb.rb:1624:in `start_service'' /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:315:in `run'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application.rb:187:in `start_proc'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application.rb:197:in `start_proc'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application.rb:227:in `start'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/controller.rb:72:in `run'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:182:in `run_proc'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:in `catch_exceptions'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in `run_proc'' /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:301:in `run'' ./script/backgroundrb:29 The output from ./script/backgroundrb run: undefined method `wait'' for nil:NilClass - (NoMethodError) /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:217:in `new_worker'' /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:36:in `dispatch'' /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `dispatch'' /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:199:in `new_worker'' /usr/lib/ruby/1.8/drb/drb.rb:1552:in `perform_without_block'' /usr/lib/ruby/1.8/drb/drb.rb:1512:in `perform'' /usr/lib/ruby/1.8/drb/drb.rb:1586:in `main_loop'' /usr/lib/ruby/1.8/drb/drb.rb:1582:in `main_loop'' /usr/lib/ruby/1.8/drb/drb.rb:1578:in `main_loop'' /usr/lib/ruby/1.8/drb/drb.rb:1427:in `run'' /usr/lib/ruby/1.8/drb/drb.rb:1424:in `run'' /usr/lib/ruby/1.8/drb/drb.rb:1344:in `initialize'' /usr/lib/ruby/1.8/drb/drb.rb:1624:in `start_service'' /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:315:in `run'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application.rb:187:in `start_proc'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application.rb:197:in `start_proc'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application.rb:227:in `start'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/controller.rb:72:in `run'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:182:in `run_proc'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:in `catch_exceptions'' /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in `run_proc'' /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:301:in `run'' ./script/backgroundrb:29 In lib/workers/other_worker.rb: class OtherWorker < BackgrounDRb::Rails def do_work(args) logger.warn("In Do Work:") @processed = true end def finished? @processed == true end end OtherWorker.register
Ezra Zygmuntowicz
2007-Apr-09 19:19 UTC
[Backgroundrb-devel] Drb Connection error on multiple dispatch.fcgi ''s
I think what is happening is that when apache spawns a new dispatch.fcgi it forks the first one to do so. And the drb connection cannot survive across a fork. The quikc fix is to use a set number of dispatchers so they each start with their own drb connection. I''m sorry to say that I haven''t used bdrb with fastcgi as I gave up on fcgi a while ago. But it does appear that apache-fcgid forking a new process from the first is what causes the second forked process to not have a good drb connection. -Ezra On Apr 9, 2007, at 11:01 AM, Pascal Rettig wrote:> Hi All, > > > I''m using Backgroundrb as a general purpose long-running-task back-end > (upload processing, email sending, etc), and it''s been a great > solution. > However I''ve recently run into some some intermittent connection > issues > that have me baffled. I''m running on apache2/fcgid and the problem > occurs in both devlopment and production mode. The problem seems to > only > trigger when there''s enough load on the server to create a second > dispatch.fcgi instance. When this happens, all of a sudden 1/2 of > my new > workers fail. I haven''t tried a mongrel setup but I found a message > from > January 30th from jacobatzen.dk that had the same backtrace error, > so I > don''t think it''s uniquely a fcgid issue. > > Anyone come across this before or have any ideas where to look to fix? > The worker I''m using for testing right now just spits out a log > message > (code at the bottom). > > Thanks, > > -Pascal > > >> From log/background_server.log : > > Connection reset by peer - (DRb::DRbConnError) > /usr/lib/ruby/1.8/drb/drb.rb:563:in `read'' > /usr/lib/ruby/1.8/drb/drb.rb:563:in `load'' > /usr/lib/ruby/1.8/drb/drb.rb:629:in `recv_reply'' > /usr/lib/ruby/1.8/drb/drb.rb:918:in `recv_reply'' > /usr/lib/ruby/1.8/drb/drb.rb:1192:in `send_message'' > /usr/lib/ruby/1.8/drb/drb.rb:1083:in `method_missing'' > /usr/lib/ruby/1.8/drb/drb.rb:1167:in `open'' > /usr/lib/ruby/1.8/drb/drb.rb:1082:in `method_missing'' > /usr/lib/ruby/1.8/drb/drb.rb:1100:in `with_friend'' > /usr/lib/ruby/1.8/drb/drb.rb:1081:in `method_missing'' > /usr/lib/ruby/1.8/drb/drb.rb:1069:in `respond_to?'' > /usr/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:454:in > `initialize'' > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/middleman.rb:210:in `new_worker'' > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/thread_pool.rb:36:in `dispatch'' > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/thread_pool.rb:22:in `dispatch'' > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/middleman.rb:199:in `new_worker'' > /usr/lib/ruby/1.8/drb/drb.rb:1552:in `perform_without_block'' > /usr/lib/ruby/1.8/drb/drb.rb:1512:in `perform'' > /usr/lib/ruby/1.8/drb/drb.rb:1586:in `main_loop'' > /usr/lib/ruby/1.8/drb/drb.rb:1582:in `main_loop'' > /usr/lib/ruby/1.8/drb/drb.rb:1578:in `main_loop'' > /usr/lib/ruby/1.8/drb/drb.rb:1427:in `run'' > /usr/lib/ruby/1.8/drb/drb.rb:1424:in `run'' > /usr/lib/ruby/1.8/drb/drb.rb:1344:in `initialize'' > /usr/lib/ruby/1.8/drb/drb.rb:1624:in `start_service'' > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb_server.rb:315:in `run'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ > application.rb:187:in `start_proc'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ > application.rb:197:in `start_proc'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ > application.rb:227:in `start'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/controller.rb: > 72:in `run'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:182:in > `run_proc'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb: > 105:in > `catch_exceptions'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in > `run_proc'' > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb_server.rb:301:in `run'' > ./script/backgroundrb:29 > > The output from ./script/backgroundrb run: > > undefined method `wait'' for nil:NilClass - (NoMethodError) > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/middleman.rb:217:in `new_worker'' > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/thread_pool.rb:36:in `dispatch'' > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/thread_pool.rb:22:in `dispatch'' > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb/middleman.rb:199:in `new_worker'' > /usr/lib/ruby/1.8/drb/drb.rb:1552:in `perform_without_block'' > /usr/lib/ruby/1.8/drb/drb.rb:1512:in `perform'' > /usr/lib/ruby/1.8/drb/drb.rb:1586:in `main_loop'' > /usr/lib/ruby/1.8/drb/drb.rb:1582:in `main_loop'' > /usr/lib/ruby/1.8/drb/drb.rb:1578:in `main_loop'' > /usr/lib/ruby/1.8/drb/drb.rb:1427:in `run'' > /usr/lib/ruby/1.8/drb/drb.rb:1424:in `run'' > /usr/lib/ruby/1.8/drb/drb.rb:1344:in `initialize'' > /usr/lib/ruby/1.8/drb/drb.rb:1624:in `start_service'' > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb_server.rb:315:in `run'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ > application.rb:187:in `start_proc'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ > application.rb:197:in `start_proc'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/ > application.rb:227:in `start'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/controller.rb: > 72:in `run'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:182:in > `run_proc'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb: > 105:in > `catch_exceptions'' > /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in > `run_proc'' > /home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/ > server/lib/backgroundrb_server.rb:301:in `run'' > ./script/backgroundrb:29 > > > In lib/workers/other_worker.rb: > > class OtherWorker < BackgrounDRb::Rails > > def do_work(args) > > logger.warn("In Do Work:") > @processed = true > end > > def finished? > @processed == true > end > > > end > OtherWorker.register > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel >-- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273)