Nick, the ''db gone away'' thing is a real problem. I ended up
with a
RobustWorker that all our other workers subclass, providing a kind of
max tries mechanism as well as emails to me when that try count is
exceeded. I think it would be great to see the library handle this
more transparently, since before I learned how to handle all this I
was beginning to think that all the naysayers may have been right
about backgroundrb.
Adam Williams
On Apr 24, 2009, at 12:06 PM, nick at bytemark.co.uk wrote:
> My name''s Nick, and I work for a company called Bytemark - we use
> backgroundrb
> in a range of internal projects (all our internal apps are Ruby on
> Rails, so
> it makes sense).
>
> Basically, and as I mentioned on IRC, I''ve been tasked with making
> backgroundrb "better" over the next month or so, and I''d
like to
> push as much
> upstream as I can while I''m at it (although some stuff I can
> guarantee you
> won''t want - I''ll my making my own local branch use a
login plugin
> with a
> different syntax, for instance).
>
> First on the agenda is this backtrace:
> /home/bmrack/bmrack/releases/20090404011130/vendor/rails/
> activerecord/lib/active_record/connection_adapters/
> abstract_adapter.rb:147:in
> `log'': Mysql::Error: MySQL server has gone away: SELECT * FROM
> `bdrb_job_queues` WHERE ( worker_name =
''dhshell_export_worker''
> AND taken
> = 0 AND scheduled_at <= ''2009-04-24 10:41:44'' ) LIMIT
1 FOR UPDATE
> (ActiveRecord::StatementInvalid)
> from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/
> activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:
> 302:in
> `execute''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/
> activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:
> 537:in
> `select''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/
> activerecord/lib/active_record/connection_adapters/abstract/
> database_statements.rb:7:in
> `select_all_without_query_cache''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/
> activerecord/lib/active_record/connection_adapters/abstract/
> query_cache.rb:61:in
> `select_all''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/
> activerecord/lib/active_record/base.rb:586:in
> `find_by_sql''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/
> activerecord/lib/active_record/base.rb:1345:in
> `find_every''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/
> activerecord/lib/active_record/base.rb:1307:in
> `find_initial''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/
> activerecord/lib/active_record/base.rb:538:in
> `find''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/
> backgroundrb/lib/backgroundrb/bdrb_job_queue.rb:11:in
> `find_next''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/
> activerecord/lib/active_record/connection_adapters/abstract/
> database_statements.rb:66:in
> `transaction''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/
> activerecord/lib/active_record/transactions.rb:79:in
> `transaction''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/
> backgroundrb/lib/backgroundrb/bdrb_job_queue.rb:8:in
> `find_next''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/
> backgroundrb/server/lib/meta_worker.rb:271:in
> `check_for_enqueued_tasks''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/
> backgroundrb/server/lib/meta_worker.rb:133:in
> `worker_init''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/
> packet_periodic_event.rb:23:in
> `call''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/
> packet_periodic_event.rb:23:in
> `run''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/
> packet_core.rb:301:in
> `check_for_timer_events''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/
> packet_core.rb:301:in
> `each''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/
> packet_core.rb:301:in
> `check_for_timer_events''
> from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/
> backgroundrb/server/lib/meta_worker.rb:296:in
> `check_for_timer_events''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/
> packet_core.rb:140:in
> `start_reactor''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/
> packet_core.rb:139:in
> `loop''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/
> packet_core.rb:139:in
> `start_reactor''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/
> packet_worker.rb:20:in
> `start_worker''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/
> packet_worker_runner:33:in
> `load_worker''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/
> packet_worker_runner:26:in
> `initialize''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/
> packet_worker_runner:47:in
> `new''
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/
> packet_worker_runner:47
> from /usr/bin/packet_worker_runner:19:in `load''
> from /usr/bin/packet_worker_runner:19
>
> (not latest git, but definitely git - we''re just updating to
packet
> 0.1.15,
> but I doubt that''ll affect this particular error) - this kills the
> server
> from time to time for us (we have our MySQL server set to go away
> after half
> a day) - it''s not the full story, since for that to happen, worker
> requests
> must have stopped too (or something else), but fixing this is number
> 1 on my
> list. I''ve not delved much into the source yet (5pm on a Friday is
> *not* the
> time to start with that!); I was wondering if you guys have a
> preferred /
> global strategy for dealing with errors. My approach to the above
> would be to
> catch the error, and respond by attempting to re-establish the
> connection. If
> that succeeds, I''d re-do the scheduled item (so the database going
> away would
> be transparent to the client); if not... hmm. Do we have a method of
> reporting a failed task back to the worker?
>
> Another problem I was seeing that led me to stop using the
> backgroundrb cache
> object was that requests into a worker via the MiddleMan API would
> just
> freeze up from time to time, leaving the Rails controller method
> hanging on
> for it. That''s another one for me to investigate (we
weren''t using
> memcached,
> mind, but it annoys me when the Hash - a far simpler system - is less
> reliable).
>
> I''m also tasked with redeploying the current backgroundrb setup -
> right now,
> we have one server per application using it (which is about 4,5
> backgroundrb
> servers in total), with the backgroundrb table stored in the
> application''s
> database. I''m kind of moving towards a scheme where we have a pair
of
> backgroundrb servers (transparently load-balanced) used by all
> applications.
> Each backgroundrb server would have its own separate *SQLite*
> database.
> Requests would go to one or another of the backgroundrb servers; if
> one of
> the servers died, all the requests would go to the other server, and
> vice-versa. If possible (I know this is a -devel list), I''d like
to
> get
> comments on whether that''s a good setup or not, and suggestions
for
> improvement. If not, well, just consider it to be a bit of context ;)
>
> Regards,
> --
> Nick Thomas
> Bytemark Hosting Limited
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel