Backgroundrb devel - Sep 2008 - Worker''s not created quick enough?

Just starting out with BackgroundDRb and have some troubles with  
calling a worker''s async process right after I try to create it.

Here''s my code:

In my book model I have the following:

# Starts a new BackgroundDRb worker for searching
def start_search(worker_key, job_key, search_params)
   MiddleMan.new_worker(:worker => :texis_worker, :worker_key =>  
worker_key.to_s)
   MiddleMan.worker(:texis_worker, worker_key.to_s).async_search(:arg  
=> {:book => self, :search_params => search_params}, :job_key =>  
job_key)
end

# Get Status of Worker
def self.search_status(worker_key, job_key)
   status = MiddleMan.worker(:texis_worker,  
worker_key.to_s).ask_result("#{worker_key.to_s}_#{job_key.to_s}")
end

My controller''s search method has this line:

@book.start_search(session[:user], @book.id, search_params)

Here''s my worker:

class TexisWorker < BackgrounDRb::MetaWorker
   set_worker_name :texis_worker
   set_no_auto_load true

   def create(args = nil)
     logger.info "Creating #{worker_key}"
   end

  def search(arg)
     logger.info "Testing #{worker_key} -- #{job_key} --  
#{arg[:book].name}"
     cache["#{worker_key.to_s}_#{job_key.to_s}"] = arg[:book].name
   end
end


When a user runs a search in my books controller I call start_search  
passing in a user id as the worker_key, book_id as job_key and a hash  
of search parameters.

I tail the log and then browse to the controller. In my log I will get:
   Creating 293460168

But nothing else.

Eventually, say 30 seconds to a minute and switching between different  
books I finally start to get something like:
   Testing 293460168 -- 3806 -- Workers Compensation

If I do this from script/console everything works fine.

I''m running mongrel in development mode, using packet 0.1.13,  
backgroundrb 1.0.4, cent os 5

It seems initially creating the worker is not done for a while? and so  
the calls to async are failing; once the worker has been intialized  
everything is fine?

Any help would be appreciated.

Thanks,
Curtis

Yeah, you pretty much have to throw a sleep call in between creating the
worker and actually asking it to do work.  It only takes a fraction of a
second.  I was told this was only a problem on OS X, but I''ve also
witnessed
it on Gentoo.  I see this question on the mailing list enough that I think
it might as well be included in the docs somewhere.

-Brent


On Fri, Sep 19, 2008 at 11:14 PM, Mitchell Curtis Hatter <
curtis.hatter at insightbb.com> wrote:
> Just starting out with BackgroundDRb and have some troubles with calling a
> worker''s async process right after I try to create it.
>
> Here''s my code:
>
> In my book model I have the following:
>
> # Starts a new BackgroundDRb worker for searching
> def start_search(worker_key, job_key, search_params)
>  MiddleMan.new_worker(:worker => :texis_worker, :worker_key =>
> worker_key.to_s)
>  MiddleMan.worker(:texis_worker, worker_key.to_s).async_search(:arg =>
> {:book => self, :search_params => search_params}, :job_key =>
job_key)
> end
>
> # Get Status of Worker
> def self.search_status(worker_key, job_key)
>  status = MiddleMan.worker(:texis_worker,
> worker_key.to_s).ask_result("#{worker_key.to_s}_#{job_key.to_s}")
> end
>
> My controller''s search method has this line:
>
> @book.start_search(session[:user], @book.id, search_params)
>
> Here''s my worker:
>
> class TexisWorker < BackgrounDRb::MetaWorker
>  set_worker_name :texis_worker
>  set_no_auto_load true
>
>  def create(args = nil)
>    logger.info "Creating #{worker_key}"
>  end
>
>  def search(arg)
>    logger.info "Testing #{worker_key} -- #{job_key} --
#{arg[:book].name}"
>    cache["#{worker_key.to_s}_#{job_key.to_s}"] = arg[:book].name
>  end
> end
>
>
> When a user runs a search in my books controller I call start_search
> passing in a user id as the worker_key, book_id as job_key and a hash of
> search parameters.
>
> I tail the log and then browse to the controller. In my log I will get:
>  Creating 293460168
>
> But nothing else.
>
> Eventually, say 30 seconds to a minute and switching between different
> books I finally start to get something like:
>  Testing 293460168 -- 3806 -- Workers Compensation
>
> If I do this from script/console everything works fine.
>
> I''m running mongrel in development mode, using packet 0.1.13,
backgroundrb
> 1.0.4, cent os 5
>
> It seems initially creating the worker is not done for a while? and so the
> calls to async are failing; once the worker has been intialized everything
> is fine?
>
> Any help would be appreciated.
>
> Thanks,
> Curtis
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>


-- 
Brent Collier | 919.564.6915 | www.BrentCollier.com | www.acts-as-blogr.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080919/c9d56180/attachment.html>

Thanks, yeah I specifically moved my development off my laptop to the  
CentOS system because I saw a thread that said it was only a problem  
on OS X. This is unfortunate, as when it does work it''s great.
I''ll
try the sleep call and see what starts to happen.

Thanks again,
Curtis

On Sep 19, 2008, at 11:58 PM, Brent Collier wrote:
> Yeah, you pretty much have to throw a sleep call in between creating  
> the worker and actually asking it to do work.  It only takes a  
> fraction of a second.  I was told this was only a problem on OS X,  
> but I''ve also witnessed it on Gentoo.  I see this question on the
> mailing list enough that I think it might as well be included in the  
> docs somewhere.
>
> -Brent
>
>
> On Fri, Sep 19, 2008 at 11:14 PM, Mitchell Curtis Hatter <curtis.hatter
at insightbb.com
> > wrote:
> Just starting out with BackgroundDRb and have some troubles with  
> calling a worker''s async process right after I try to create it.
>
> Here''s my code:
>
> In my book model I have the following:
>
> # Starts a new BackgroundDRb worker for searching
> def start_search(worker_key, job_key, search_params)
>  MiddleMan.new_worker(:worker => :texis_worker, :worker_key =>  
> worker_key.to_s)
>  MiddleMan.worker(:texis_worker, worker_key.to_s).async_search(:arg  
> => {:book => self, :search_params => search_params}, :job_key
=>
> job_key)
> end
>
> # Get Status of Worker
> def self.search_status(worker_key, job_key)
>  status = MiddleMan.worker(:texis_worker,  
> worker_key.to_s).ask_result("#{worker_key.to_s}_#{job_key.to_s}")
> end
>
> My controller''s search method has this line:
>
> @book.start_search(session[:user], @book.id, search_params)
>
> Here''s my worker:
>
> class TexisWorker < BackgrounDRb::MetaWorker
>  set_worker_name :texis_worker
>  set_no_auto_load true
>
>  def create(args = nil)
>    logger.info "Creating #{worker_key}"
>  end
>
>  def search(arg)
>    logger.info "Testing #{worker_key} -- #{job_key} --  
> #{arg[:book].name}"
>    cache["#{worker_key.to_s}_#{job_key.to_s}"] = arg[:book].name
>  end
> end
>
>
> When a user runs a search in my books controller I call start_search  
> passing in a user id as the worker_key, book_id as job_key and a  
> hash of search parameters.
>
> I tail the log and then browse to the controller. In my log I will  
> get:
>  Creating 293460168
>
> But nothing else.
>
> Eventually, say 30 seconds to a minute and switching between  
> different books I finally start to get something like:
>  Testing 293460168 -- 3806 -- Workers Compensation
>
> If I do this from script/console everything works fine.
>
> I''m running mongrel in development mode, using packet 0.1.13,  
> backgroundrb 1.0.4, cent os 5
>
> It seems initially creating the worker is not done for a while? and  
> so the calls to async are failing; once the worker has been  
> intialized everything is fine?
>
> Any help would be appreciated.
>
> Thanks,
> Curtis
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
>
>
> -- 
> Brent Collier | 919.564.6915 | www.BrentCollier.com | www.acts-as-blogr.com

On Fri, 2008-09-19 at 23:58 -0400, Brent Collier wrote:
> Yeah, you pretty much have to throw a sleep call in between creating
> the worker and actually asking it to do work.  It only takes a
> fraction of a second.  I was told this was only a problem on OS X, but
> I''ve also witnessed it on Gentoo.  I see this question on the
mailing
> list enough that I think it might as well be included in the docs
> somewhere.

First what exactly is async_xxx "failed"? As far as my thinking goes,
since in a worker, usually full rails environment is loaded, it is
naturally a bit slow to start, but any sync/async task execution should
not get "lost" between the time worker is loading the rails
environment.

For example, on my ubuntu box, with packet-0.1.13 and bdrb from git:

require File.join(File.dirname(__FILE__) + "/../config/environment")

a = MiddleMan.new_worker(:worker => :wow_worker,:worker_key =>
"hello_world")

10.times do |i|
  MiddleMan.worker(:wow_worker,"hello_world").async_run_crap(:arg
=>
"hello world : #{i}")
end

I won''t loose any of the "async_run_crap" calls, even though
there is no
"sleep" between starting a worker and asking it to execute a method.
Sure, worker may take time, to be fully ready, but by that time, tasks
should be "queued" up and thats what happens in Linux and behaviour is
correct.

On OSX though, you will loose async_xxx calls and hence its a bug.So,
mitchell, are you loosing async_xxx method calls? Or it gets invoked
eventually but slightly delayed (while its loading rails env)? If a
method call is lost, its a bug, I will need to fix it.

On OSX, ruby nonblocking APIs aren''t working well and hence tasks
aren''t
getting enqueued in socket buffer too and hence we are loosing messages.
But this works, quite well in Linux, in my experience.

I''m actually losing the call to the async_* method.

Anytime I first create a new worker I lose any async_* calls to it for  
approximately 10 to 15 seconds.

I even tried this:

     MiddleMan.new_worker(:worker => :texis_worker, :worker_key =>  
worker_key.to_s)
     sleep(5)
     MiddleMan.worker(:texis_worker,  
worker_key.to_s).async_search(:arg => {:book => self, :search_params  
=> search_params}, :job_key => job_key)

And got nothing. After the worker creates it''s fine. But I tried just  
one call and waited about 30 seconds, got nothing, then made another  
call and got results right away.

Again, when I do it from script/console it never fails.

On Sep 20, 2008, at 12:36 AM, hemant kumar wrote:
> On Fri, 2008-09-19 at 23:58 -0400, Brent Collier wrote:
>> Yeah, you pretty much have to throw a sleep call in between creating
>> the worker and actually asking it to do work.  It only takes a
>> fraction of a second.  I was told this was only a problem on OS X,  
>> but
>> I''ve also witnessed it on Gentoo.  I see this question on the
mailing
>> list enough that I think it might as well be included in the docs
>> somewhere.
>
>
> First what exactly is async_xxx "failed"? As far as my thinking
goes,
> since in a worker, usually full rails environment is loaded, it is
> naturally a bit slow to start, but any sync/async task execution  
> should
> not get "lost" between the time worker is loading the rails  
> environment.
>
> For example, on my ubuntu box, with packet-0.1.13 and bdrb from git:
>
> require File.join(File.dirname(__FILE__) +
"/../config/environment")
>
> a = MiddleMan.new_worker(:worker => :wow_worker,:worker_key =>
> "hello_world")
>
> 10.times do |i|
>  MiddleMan.worker(:wow_worker,"hello_world").async_run_crap(:arg
=>
> "hello world : #{i}")
> end
>
> I won''t loose any of the "async_run_crap" calls, even
though there
> is no
> "sleep" between starting a worker and asking it to execute a
method.
> Sure, worker may take time, to be fully ready, but by that time, tasks
> should be "queued" up and thats what happens in Linux and
behaviour is
> correct.
>
> On OSX though, you will loose async_xxx calls and hence its a bug.So,
> mitchell, are you loosing async_xxx method calls? Or it gets invoked
> eventually but slightly delayed (while its loading rails env)? If a
> method call is lost, its a bug, I will need to fix it.
>
> On OSX, ruby nonblocking APIs aren''t working well and hence tasks
> aren''t
> getting enqueued in socket buffer too and hence we are loosing  
> messages.
> But this works, quite well in Linux, in my experience.
>
>
>
>
>

Mitchel, can you send me tar.gz file of your app, which is sufficient
to reproduce mentioned behaviour?
I mean, don''t send me entire app (if it contains confidential info),
just enough of code, so as I can reproduce this. Also, in the
meanwhile, you can try git version and see if it fixes this.



On Sat, Sep 20, 2008 at 10:55 AM, Mitchell Curtis Hatter
<curtis.hatter at insightbb.com> wrote:
> I''m actually losing the call to the async_* method.
>
> Anytime I first create a new worker I lose any async_* calls to it for
> approximately 10 to 15 seconds.
>
> I even tried this:
>
>    MiddleMan.new_worker(:worker => :texis_worker, :worker_key =>
> worker_key.to_s)
>    sleep(5)
>    MiddleMan.worker(:texis_worker, worker_key.to_s).async_search(:arg =>
> {:book => self, :search_params => search_params}, :job_key =>
job_key)
>
> And got nothing. After the worker creates it''s fine. But I tried
just one
> call and waited about 30 seconds, got nothing, then made another call and
> got results right away.
>
> Again, when I do it from script/console it never fails.
>
> On Sep 20, 2008, at 12:36 AM, hemant kumar wrote:
>
>> On Fri, 2008-09-19 at 23:58 -0400, Brent Collier wrote:
>>>
>>> Yeah, you pretty much have to throw a sleep call in between
creating
>>> the worker and actually asking it to do work.  It only takes a
>>> fraction of a second.  I was told this was only a problem on OS X,
but
>>> I''ve also witnessed it on Gentoo.  I see this question on
the mailing
>>> list enough that I think it might as well be included in the docs
>>> somewhere.
>>
>>
>> First what exactly is async_xxx "failed"? As far as my
thinking goes,
>> since in a worker, usually full rails environment is loaded, it is
>> naturally a bit slow to start, but any sync/async task execution should
>> not get "lost" between the time worker is loading the rails
environment.
>>
>> For example, on my ubuntu box, with packet-0.1.13 and bdrb from git:
>>
>> require File.join(File.dirname(__FILE__) +
"/../config/environment")
>>
>> a = MiddleMan.new_worker(:worker => :wow_worker,:worker_key =>
>> "hello_world")
>>
>> 10.times do |i|
>> 
MiddleMan.worker(:wow_worker,"hello_world").async_run_crap(:arg =>
>> "hello world : #{i}")
>> end
>>
>> I won''t loose any of the "async_run_crap" calls,
even though there is no
>> "sleep" between starting a worker and asking it to execute a
method.
>> Sure, worker may take time, to be fully ready, but by that time, tasks
>> should be "queued" up and thats what happens in Linux and
behaviour is
>> correct.
>>
>> On OSX though, you will loose async_xxx calls and hence its a bug.So,
>> mitchell, are you loosing async_xxx method calls? Or it gets invoked
>> eventually but slightly delayed (while its loading rails env)? If a
>> method call is lost, its a bug, I will need to fix it.
>>
>> On OSX, ruby nonblocking APIs aren''t working well and hence
tasks aren''t
>> getting enqueued in socket buffer too and hence we are loosing
messages.
>> But this works, quite well in Linux, in my experience.
>>
>>
>>
>>
>>
>
>


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

Found this: http://gnufied.org/2008/02/12/backgroundrb-best-practises/

Which you wrote of course. Point #2 seems to have solved my problem. I  
removed where I was passing the entire Book AR object and just passed  
the Book''s id and things start working as expected within the full  
Rails environment.

Any chance a "best practices" could be added somewhere to the  
backgroundrb.rubyforge.org site and have those comments added.

If I have any more problems with this I''ll post to this thread again  
but for now seems to be working.

Thanks,
Curtis

On Sep 20, 2008, at 12:36 AM, hemant kumar wrote:
> On Fri, 2008-09-19 at 23:58 -0400, Brent Collier wrote:
>> Yeah, you pretty much have to throw a sleep call in between creating
>> the worker and actually asking it to do work.  It only takes a
>> fraction of a second.  I was told this was only a problem on OS X,  
>> but
>> I''ve also witnessed it on Gentoo.  I see this question on the
mailing
>> list enough that I think it might as well be included in the docs
>> somewhere.
>
>
> First what exactly is async_xxx "failed"? As far as my thinking
goes,
> since in a worker, usually full rails environment is loaded, it is
> naturally a bit slow to start, but any sync/async task execution  
> should
> not get "lost" between the time worker is loading the rails  
> environment.
>
> For example, on my ubuntu box, with packet-0.1.13 and bdrb from git:
>
> require File.join(File.dirname(__FILE__) +
"/../config/environment")
>
> a = MiddleMan.new_worker(:worker => :wow_worker,:worker_key =>
> "hello_world")
>
> 10.times do |i|
>  MiddleMan.worker(:wow_worker,"hello_world").async_run_crap(:arg
=>
> "hello world : #{i}")
> end
>
> I won''t loose any of the "async_run_crap" calls, even
though there
> is no
> "sleep" between starting a worker and asking it to execute a
method.
> Sure, worker may take time, to be fully ready, but by that time, tasks
> should be "queued" up and thats what happens in Linux and
behaviour is
> correct.
>
> On OSX though, you will loose async_xxx calls and hence its a bug.So,
> mitchell, are you loosing async_xxx method calls? Or it gets invoked
> eventually but slightly delayed (while its loading rails env)? If a
> method call is lost, its a bug, I will need to fix it.
>
> On OSX, ruby nonblocking APIs aren''t working well and hence tasks
> aren''t
> getting enqueued in socket buffer too and hence we are loosing  
> messages.
> But this works, quite well in Linux, in my experience.
>
>
>
>
>