Backgroundrb devel - Dec 2007 - Possible Bug & Question

Hi there,

Amazing changes! I really like what you''ve done with BackgrounDRb and I
hope
you continue developing it, since it was definitely somewhat languishing
before.

One small error stuck out for me, though. The create method requires an
argument even if one isn''t used, such that if you have:

def create
  @results = Hash.new
end

The server throws an error about trying to send an argument to create when
it doesn''t accept any. Obviously the fix on our end is to just do def
create(arg) and ignore the argument if one isn''t used, but you might
want to
make the server friendlier to create methods that don''t accept
arguments.

And now that I''ve made a bug report, I have a question for you. I have
a
very long running task (sometimes taking up to two minutes) that I offloaded
onto the previous version of BackgrounDRb. I made a new worker (called
correlator) for every user and killed it after it returned results for them.
This was a pretty ugly method of doing things and occassionally I''d
lose
Result hashes or the workers wouldn''t get created at all, so with the
new
version I''m trying to use only one worker and report individual
correlations
through a results hash, like so:

def CorrelatorWorker < BackgrounDRb::MetaWorker
  set_worker_name :correlator_worker

def create(arg)
  @results = Hash.new
end

def correlate(user_id)
  # This stuff takes awhile...
  @results[user_id] = progress
  register_status(@results)
end

end

So in my code I can say
MiddleMan.ask_status(:worker=>:correlator_worker)[user_id]
to find the progress of correlations for a single user. But the previous
version of BackgrounDRb did have one big advantage in that each worker was a
separate thread. In this version, when I start correlations for one user,
the correlator_worker blocks until the method is finished, and then starts
in on the next correlation, and so on. I want all the correlations to happen
concurrently, so I was thinking of replacing correlate with this:

def correlate(user_id)
  Thread.new do
    # This stuff takes awhile...
    @results[user_id] = progress
    register_status(@results)
  end
end

And splitting each correlation into its own thread. I have no idea how the
new BackgrounDRb acts with this sort of thing, though. Would you recommend
this as the developer, or is there some more obvious method that I''m
missing
to allow one worker to do a method without blocking further calls to that
method?

Josh Symonds
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071213/1d91d126/attachment-0001.html

On Thu, 2007-12-13 at 11:13 -0600, Josh Symonds wrote:
> Hi there,
> 
> Amazing changes! I really like what you''ve done with BackgrounDRb
and
> I hope you continue developing it, since it was definitely somewhat
> languishing before.
> 
> One small error stuck out for me, though. The create method requires
> an argument even if one isn''t used, such that if you have: 
> 
> def create
>   @results = Hash.new
> end
> 
> The server throws an error about trying to send an argument to create
> when it doesn''t accept any. Obviously the fix on our end is to
just do
> def create(arg) and ignore the argument if one isn''t used, but you
> might want to make the server friendlier to create methods that
don''t
> accept arguments. 
Ok, Will be fixed.
> 
> And now that I''ve made a bug report, I have a question for you. I
have
> a very long running task (sometimes taking up to two minutes) that I
> offloaded onto the previous version of BackgrounDRb. I made a new
> worker (called correlator) for every user and killed it after it
> returned results for them. This was a pretty ugly method of doing
> things and occassionally I''d lose Result hashes or the workers
> wouldn''t get created at all, so with the new version I''m
trying to use
> only one worker and report individual correlations through a results
> hash, like so: 
> 
> def CorrelatorWorker < BackgrounDRb::MetaWorker
>   set_worker_name :correlator_worker
> 
> def create(arg)
>   @results = Hash.new
> end
> 
> def correlate(user_id)
>   # This stuff takes awhile...
>   @results[user_id] = progress
>   register_status(@results)
> end
> 
> end
> 
> So in my code I can say
> MiddleMan.ask_status(:worker=>:correlator_worker)[user_id] to find the
> progress of correlations for a single user. But the previous version
> of BackgrounDRb did have one big advantage in that each worker was a
> separate thread. In this version, when I start correlations for one
> user, the correlator_worker blocks until the method is finished, and
> then starts in on the next correlation, and so on. I want all the
> correlations to happen concurrently, so I was thinking of replacing
> correlate with this: 
> 
> def correlate(user_id)
>   Thread.new do
>     # This stuff takes awhile...
>     @results[user_id] = progress
>     register_status(@results)
>   end
> end
> 
> And splitting each correlation into its own thread. I have no idea how
> the new BackgrounDRb acts with this sort of thing, though. Would you
> recommend this as the developer, or is there some more obvious method
> that I''m missing to allow one worker to do a method without
blocking
> further calls to that method? 
Since ruby has green threads, you got to be careful about what you are
doing in those threads. If your task is CPU bound then ruby threads will
not help, but if its reading from network or something, then you may use
threads. However, i am in process of finalizing usage of threads in bdrb
and would update the code soon.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org