Backgroundrb devel - Sep 2008 - not finding activerecord data in worker

Hey guys,

Firstly, I''m new to the list and great work with backgroundrb,
it''s
sweet!!! I''ve just started noticing an issue and i''m wondering
if
there is a standard fix. The situation is as follows:

1. In my model''s after_save I call my worker asynchronously with the  
active record object''s id as a parameter (the record that has just  
been saved).
2. In my worker, the first thing I do is find the record from the  
database using the id which has been passed in.

What i''m finding is occasionally, in step 2. the worker cannot find  
the record in the db. Having log_bin set in my mysql configuration  
seems to drastically increase the frequency of this happening. It is  
quite clear from the rails logs that the INSERT always happens before  
the call to the backgroundrb worker.

I''m puzzled as I thought the mysql insert query would not return until
the inserted data is actually in the database and ready to be  
retrieved. It seems to be not the case. The current workaround i''m  
thinking is to put some retrying/sleeping in the backgroundrb worker  
in order to try to wait it out until the record is there. This seems  
to me a last resort. Would love to hear what your experiences are and  
of a better solution if it exists.

many thanks,
zol

On Mon, 2008-09-15 at 12:39 +1000, Zoltan Olah wrote:
> Hey guys,
> 
> Firstly, I''m new to the list and great work with backgroundrb,
it''s
> sweet!!! I''ve just started noticing an issue and i''m
wondering if
> there is a standard fix. The situation is as follows:
> 
> 1. In my model''s after_save I call my worker asynchronously with
the
> active record object''s id as a parameter (the record that has just
> been saved).
> 2. In my worker, the first thing I do is find the record from the  
> database using the id which has been passed in.
> 
> What i''m finding is occasionally, in step 2. the worker cannot
find
> the record in the db. Having log_bin set in my mysql configuration  
> seems to drastically increase the frequency of this happening. It is  
> quite clear from the rails logs that the INSERT always happens before  
> the call to the backgroundrb worker.
> 
> I''m puzzled as I thought the mysql insert query would not return
until
> the inserted data is actually in the database and ready to be  
> retrieved. It seems to be not the case. The current workaround i''m
> thinking is to put some retrying/sleeping in the backgroundrb worker  
> in order to try to wait it out until the record is there. This seems  
> to me a last resort. Would love to hear what your experiences are and  
> of a better solution if it exists.
> 
Yes, this issue came up before on the list and mostly went unresolved.
I think, this is especially true for databases thats remotely located (I
am unable to reproduce this with my local test setup). Currently your
best bet is to retry, something like this can be a good idea:

  def foobar ar_id
    t = User.find(ar_id)
    unless t
      # better than sleep, because your worker can
      # process other requests while this timer gets ready
      add_timer(1) { foobar(ar_id) }
    else
      # do the normal processing here
    end
  end

Thanks for your reply. After doing some more research, it seems likely  
that the cause of the problem is after_save being called from inside  
the transaction which wraps save. After some very short testing, the  
problem seems fixed by downloading the after_commit plugin from git:// 
github.com/GUI/after_commit.git . This adds an after_commit AR  
callback which happens after the save transaction has been commited to  
the db. Make the async call to backgroundrb from the after_commit  
callback instead after_save . This ensures that the save transaction  
has been committed to the db and subsequent selects *should* work  
barring any mysql bugs :)

zol


On 15/09/2008, at 2:26 PM, hemant kumar wrote:
> On Mon, 2008-09-15 at 12:39 +1000, Zoltan Olah wrote:
>> Hey guys,
>>
>> Firstly, I''m new to the list and great work with backgroundrb,
it''s
>> sweet!!! I''ve just started noticing an issue and i''m
wondering if
>> there is a standard fix. The situation is as follows:
>>
>> 1. In my model''s after_save I call my worker asynchronously
with the
>> active record object''s id as a parameter (the record that has
just
>> been saved).
>> 2. In my worker, the first thing I do is find the record from the
>> database using the id which has been passed in.
>>
>> What i''m finding is occasionally, in step 2. the worker cannot
find
>> the record in the db. Having log_bin set in my mysql configuration
>> seems to drastically increase the frequency of this happening. It is
>> quite clear from the rails logs that the INSERT always happens before
>> the call to the backgroundrb worker.
>>
>> I''m puzzled as I thought the mysql insert query would not
return
>> until
>> the inserted data is actually in the database and ready to be
>> retrieved. It seems to be not the case. The current workaround
i''m
>> thinking is to put some retrying/sleeping in the backgroundrb worker
>> in order to try to wait it out until the record is there. This seems
>> to me a last resort. Would love to hear what your experiences are and
>> of a better solution if it exists.
>>
>
> Yes, this issue came up before on the list and mostly went unresolved.
> I think, this is especially true for databases thats remotely  
> located (I
> am unable to reproduce this with my local test setup). Currently your
> best bet is to retry, something like this can be a good idea:
>
>  def foobar ar_id
>    t = User.find(ar_id)
>    unless t
>      # better than sleep, because your worker can
>      # process other requests while this timer gets ready
>      add_timer(1) { foobar(ar_id) }
>    else
>      # do the normal processing here
>    end
>  end
>
>
>