Backgroundrb devel - Jan 2007 - Yet Another Problem with BackgroundRB

Hi!

My cron-based worker is being indeed invoked by backgroundRB at
correct times. But... After several runs it can no longer find DB
columns! The same query, which was running OK an hour ago starts to
throw MySQL error about unkown column in where clause. If I restart
backgroundrb it works for some time but stops working after several
invocations.

Has anyone epxerienced the similar beahviour? Are there any ideas of
what can be wrong? Any hints? Anything I can check?

-- 
Regards,
Mike

Hi Mike --

Can you please provide a stack trace or error message?

If you run the same worker over and over, can you recreate the problem?
Or does this happen after no worker is run for some extended period of time?
(I''m guessing
here that the db connection might be timing out)

Do you know if the problem is triggered by a certain number of innvocations,
or is it a lack of activity, or neither?

Mason

On 1/22/07, Mike Aizatsky <mike.aizatsky at gmail.com>
wrote:
>
> Hi!
>
> My cron-based worker is being indeed invoked by backgroundRB at
> correct times. But... After several runs it can no longer find DB
> columns! The same query, which was running OK an hour ago starts to
> throw MySQL error about unkown column in where clause. If I restart
> backgroundrb it works for some time but stops working after several
> invocations.
>
> Has anyone epxerienced the similar beahviour? Are there any ideas of
> what can be wrong? Any hints? Anything I can check?
>
> --
> Regards,
> Mike
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070122/e584312e/attachment.html

long running scripts lose connection to the mysql server if they don''t 
have any activity for a while. the connection library is supposed to 
reconnect when this happens but in my experience it doesn''t.

i''m not sure if this is the problem you''re hitting, but if so,
one
workaround is to have the script perform a simple query every 5 minutes. 
this ensures that the mysql server never closes the connection due.

-Jacob


Mike Aizatsky wrote:
> Hi!
> 
> My cron-based worker is being indeed invoked by backgroundRB at
> correct times. But... After several runs it can no longer find DB
> columns! The same query, which was running OK an hour ago starts to
> throw MySQL error about unkown column in where clause. If I restart
> backgroundrb it works for some time but stops working after several
> invocations.
> 
> Has anyone epxerienced the similar beahviour? Are there any ideas of
> what can be wrong? Any hints? Anything I can check?
>

Mason,

The stacktrace is at the end of the message.
> If you run the same worker over and over, can you recreate the problem?
> Or does this happen after no worker is run for some extended period of
time?
> (I''m guessing
> here that the db connection might be timing out)
I''m running my worker every 10 minutes. According to log files
sometimes it takes just 3-4 executions to break. But I''ve also
experienced a longer periods of successfull runs (3-4 hours).
>
> Do you know if the problem is triggered by a certain number of
innvocations,
> or is it a lack of activity, or neither?
Nope, I don''t. However it does look like something is wrong with
connection.

---------
20070122-15:21:00 (22914) Mysql::Error: Unknown column
''drill_notification'' in ''where clause'':
SELECT * FROM users WHERE
(drill_notification = 1 AND drill_notification_last_send <
''20070122''
AND drill_notification_time < 44460 )  -
(ActiveRecord::StatementInvalid)rs/abstract_adapter.rb:128:in
`log''0070122-15:21:00 (22914)
/opt/local/lib/ruby/gems/1.8/gems/activerecord-1.15.1/lib/active_record/connection_adapters/mysql_adapter.rb:243:in
`execute''070122-15:21:00 (22914)
/opt/local/lib/ruby/gems/1.8/gems/activerecord-1.15.1/lib/active_record/connection_adapters/mysql_adapter.rb:395:in
`select''0070122-15:21:00 (22914)
/opt/local/lib/ruby/gems/1.8/gems/activerecord-1.15.1/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in
`select_all''(22914)
/opt/local/lib/ruby/gems/1.8/gems/activerecord-1.15.1/lib/active_record/base.rb:424:in
`find_by_sql''
20070122-15:21:00 (22914)
/opt/local/lib/ruby/gems/1.8/gems/activerecord-1.15.1/lib/active_record/base.rb:994:in
`find_every''
20070122-15:21:00 (22914)
/opt/local/lib/ruby/gems/1.8/gems/activerecord-1.15.1/lib/active_record/base.rb:415:in
`find''
20070122-15:21:00 (22914)
/u/apps/learnbyrote/releases/20070121210300/config/../app/models/user.rb:74:in
`find_all_to_send_drill_reminder''
20070122-15:21:00 (22914)
/u/apps/learnbyrote/releases/20070121210300/lib/workers/drill_reminder_worker.rb:10:in
`do_work''
20070122-15:21:00 (22914)
/u/apps/learnbyrote/releases/20070121210300/vendor/plugins/backgroundrb/server/lib/backgroundrb/worker.rb:55:in
`send''
20070122-15:21:00 (22914)
/u/apps/learnbyrote/releases/20070121210300/vendor/plugins/backgroundrb/server/lib/backgroundrb/worker.rb:55:in
`work_thread''070122-15:21:00 (22914)
/u/apps/learnbyrote/releases/20070121210300/vendor/plugins/backgroundrb/server/lib/backgroundrb/worker.rb:69:in
`call''
20070122-15:21:00 (22914)
/u/apps/learnbyrote/releases/20070121210300/vendor/plugins/backgroundrb/server/lib/backgroundrb/worker.rb:69:in
`work_thread''070122-15:21:00 (22914)
/u/apps/learnbyrote/releases/20070121210300/vendor/plugins/backgroundrb/server/lib/backgroundrb/worker.rb:67:in
`initialize''0070122-15:21:00 (22914)
/u/apps/learnbyrote/releases/20070121210300/vendor/plugins/backgroundrb/server/lib/backgroundrb/worker.rb:67:in
`new''
20070122-15:21:00 (22914)
/u/apps/learnbyrote/releases/20070121210300/vendor/plugins/backgroundrb/server/lib/backgroundrb/worker.rb:67:in
`work_thread''070122-15:21:00 (22914)
/opt/local/lib/ruby/1.8/drb/drb.rb:1555:in `__send__''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1555:in
`perform_without_block''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1515:in
`perform''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1589:in
`main_loop''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1585:in
`loop''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1585:in
`main_loop''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1581:in
`start''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1581:in
`main_loop''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1430:in
`run''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1427:in
`start''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1427:in
`run''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1347:in
`initialize''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1627:in
`new''
20070122-15:21:00 (22914) /opt/local/lib/ruby/1.8/drb/drb.rb:1627:in
`start_service''
20070122-15:21:00 (22914)
/opt/local/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:396:in
`initialize''
20070122-15:21:00 (22914)
/opt/local/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:391:in
`times''
20070122-15:21:00 (22914)
/opt/local/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:391:in
`initialize''


On 1/22/07, Mason Hale <masonhale at gmail.com>
wrote:
> Hi Mike --
>
> Can you please provide a stack trace or error message?
>
> If you run the same worker over and over, can you recreate the problem?
> Or does this happen after no worker is run for some extended period of
time?
> (I''m guessing
> here that the db connection might be timing out)
>
> Do you know if the problem is triggered by a certain number of
innvocations,
> or is it a lack of activity, or neither?
>
> Mason
>
>
>  On 1/22/07, Mike Aizatsky <mike.aizatsky at gmail.com> wrote:
> >
> > Hi!
> >
> > My cron-based worker is being indeed invoked by backgroundRB at
> > correct times. But... After several runs it can no longer find DB
> > columns! The same query, which was running OK an hour ago starts to
> > throw MySQL error about unkown column in where clause. If I restart
> > backgroundrb it works for some time but stops working after several
> > invocations.
> >
> > Has anyone epxerienced the similar beahviour? Are there any ideas of
> > what can be wrong? Any hints? Anything I can check?
> >
> > --
> > Regards,
> > Mike
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> >
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> >
>
>

-- 
Regards,
Mike

Mason,

I''ve just had an idea and it turned out to be true in my situation. It
turned out, that after a while backgroundrb connects to my
_development_ database instead of production one. I''m invoking
backgroundrb with the following command:

script/backgroundrb start -- -r production

It looks like -r doesn''t propagate nicely.

On 1/22/07, Mason Hale <masonhale at gmail.com>
wrote:
> Hi Mike --
>
> Can you please provide a stack trace or error message?
>
> If you run the same worker over and over, can you recreate the problem?
> Or does this happen after no worker is run for some extended period of
time?
> (I''m guessing
> here that the db connection might be timing out)
>
> Do you know if the problem is triggered by a certain number of
innvocations,
> or is it a lack of activity, or neither?
>
> Mason
>
>
>  On 1/22/07, Mike Aizatsky <mike.aizatsky at gmail.com> wrote:
> >
> > Hi!
> >
> > My cron-based worker is being indeed invoked by backgroundRB at
> > correct times. But... After several runs it can no longer find DB
> > columns! The same query, which was running OK an hour ago starts to
> > throw MySQL error about unkown column in where clause. If I restart
> > backgroundrb it works for some time but stops working after several
> > invocations.
> >
> > Has anyone epxerienced the similar beahviour? Are there any ideas of
> > what can be wrong? Any hints? Anything I can check?
> >
> > --
> > Regards,
> > Mike
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> >
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> >
>
>

-- 
Regards,
Mike

That sounds like a BackgroundRb bug.

FWIW -- I''ve got a production backgroundrb app running smoothly,
but I use the RAILS_ENV setting in the backgroundrb.yml to
set it to a production environment. I do not use the -r setting.

Mason

On 1/22/07, Mike Aizatsky <mike.aizatsky at gmail.com>
wrote:
>
> Mason,
>
> I''ve just had an idea and it turned out to be true in my
situation. It
> turned out, that after a while backgroundrb connects to my
> _development_ database instead of production one. I''m invoking
> backgroundrb with the following command:
>
> script/backgroundrb start -- -r production
>
> It looks like -r doesn''t propagate nicely.
>
> On 1/22/07, Mason Hale <masonhale at gmail.com> wrote:
> > Hi Mike --
> >
> > Can you please provide a stack trace or error message?
> >
> > If you run the same worker over and over, can you recreate the
problem?
> > Or does this happen after no worker is run for some extended period of
> time?
> > (I''m guessing
> > here that the db connection might be timing out)
> >
> > Do you know if the problem is triggered by a certain number of
> innvocations,
> > or is it a lack of activity, or neither?
> >
> > Mason
> >
> >
> >  On 1/22/07, Mike Aizatsky <mike.aizatsky at gmail.com> wrote:
> > >
> > > Hi!
> > >
> > > My cron-based worker is being indeed invoked by backgroundRB at
> > > correct times. But... After several runs it can no longer find DB
> > > columns! The same query, which was running OK an hour ago starts
to
> > > throw MySQL error about unkown column in where clause. If I
restart
> > > backgroundrb it works for some time but stops working after
several
> > > invocations.
> > >
> > > Has anyone epxerienced the similar beahviour? Are there any ideas
of
> > > what can be wrong? Any hints? Anything I can check?
> > >
> > > --
> > > Regards,
> > > Mike
> > > _______________________________________________
> > > Backgroundrb-devel mailing list
> > > Backgroundrb-devel at rubyforge.org
> > >
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> > >
> >
> >
>
>
> --
> Regards,
> Mike
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070122/0ddb110b/attachment-0001.html

Mason,

How do you test your workers in development?

On 1/22/07, Mason Hale <masonhale at gmail.com>
wrote:
> That sounds like a BackgroundRb bug.
>
> FWIW -- I''ve got a production backgroundrb app running smoothly,
> but I use the RAILS_ENV setting in the backgroundrb.yml to
> set it to a production environment. I do not use the -r setting.

I definitely see a lot of opportunity in improving the test support of
backgroundrb.

The tests I have written start a worker in the test using
MiddleMan.new_worker and then check its results (stored in a database)
externally. [Note: there are some issues with the ResultsWorker at present
that make it unreliable as a means of checking
the state of your worker processes, thus all state is stored in a database.]

This has the advantage of testing the full stack in a real-world-ish
scenario. It has the downside
that it only works with relatively simple workers.

Here''s an example:

==  /test/unit/backgroundrb_test ===========
require File.dirname(__FILE__) + ''/../test_helper''

class BackgroundrbTest < Test::Unit::TestCase

  def setup
    # start backgroundrb server
    `../../script/backgroundrb start`
    puts "starting backgroundrb server..."
    sleep 2 # give it time to startup
  end

  def teardown
    # stop backgroundrb server
    `../../script/backgroundrb stop`
    puts "backgroundrb server stopped"
  end

  def test_drb
    keys = []
    10.times do |i|
      key = MiddleMan.new_worker(:class => :example_worker)
      puts "Started process: #{key}"
      keys << key
    end

    keys.each_with_index do |k, i|
      puts "Checking #{k}"
      mm = MiddleMan[k]
      assert_not_nil mm, "checking job_key #{k} on iteration #{i}"
      puts "found job: #{mm}"
      mw = MiddleMan.worker(k)
      assert_not_nil mw, "checking job_key #{k} on iteration #{i}"
      puts "found worker: #{mw}"
    end

  end
end

=========================
This test just creates 10 workers and then makes sure they are reachable
later via the generated job keys.

The test could just as easily create a worker, sleep for a few seconds to
let
it do it''s work, then check the database for the results.

I have adopted a habit of writing dead-simple workers, where all work is
done via an external class, either
a model class or library that is loaded by the worker. I can then test these
classes independently
because they do not have a dependency on running within or being loaded by
the backgroundrb_server.
The worker should just extract the params it needs from the args passed to
it, instantiate/load the object/class it needs to
do it''s work -- and then call a method on that object/class. These
external
objects can be tested normally.

The tricky part is checking the results of a separate process. I use a
database for as a medium for that now.
I don''t have a way to test a worker in the same process, although
I''m sure
that can be worked out.

This setup is not ideal, and it won''t work for everyone, but for now
it''s
workable for me.

Mason



On 1/22/07, Mike Aizatsky <mike.aizatsky at gmail.com>
wrote:
>
> Mason,
>
> How do you test your workers in development?
>
> On 1/22/07, Mason Hale <masonhale at gmail.com> wrote:
> > That sounds like a BackgroundRb bug.
> >
> > FWIW -- I''ve got a production backgroundrb app running
smoothly,
> > but I use the RAILS_ENV setting in the backgroundrb.yml to
> > set it to a production environment. I do not use the -r setting.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070122/24627658/attachment.html