Backgroundrb devel - Oct 2008 - questions about persistent jobs

I''ve been having problems with losing async''ed tasks, so
I''ve moved to
trying to use enqueued tasks.

I''m trying to offload user queries to the background, using a thread  
pool size of 40 and two workers (on different machines). It seems that  
if I exceed the thread pool size those queries get lost, or for some  
other reason I can''t determine yet I''m losing my tasks when
there are
lots of them at once.

For that reason I''ve moved to trying enqued tasks. This method seems  
to be fine for not losing jobs but I can''t seem to run more then 2 at  
a time? (one for each worker). Is there a way to up the number of  
enqueued tasks that will be ran at a time? This way I could make use  
of my thread pool, and the enqueueing tasks.

The behavior I expected from this method is that it checks every  
second (like I have configured) for enqueued tasks. Then running as  
many enqueued tasks as the thread pool limit for that worker. Is there  
a way to get this behavior from the system? Right now it seems to pull  
out one at a time and run them till they finish and then pull out the  
next.

If my expected behavior isn''t how it''s supposed to work, would
there
be a way to do that?

I''m running version 1.0.4 with packet 0.1.14

My drb config:
---
:backgroundrb:
   :ip: 0.0.0.0
   :port: 11006
   :persistent_delay: 1
   :debug_log: true
   :environment: development
   :result_storage: memcache

:memcache: "xxx.xxx.xxx.xxx:11211,xxx.xxx.xxx.xxx:11211"
:client: "xxx.xxx.xxx.xxx:11006,xxx.xxx.xxx.xxx:11006"

# :schedules: # optional task scheduling


My worker:

require ''hpricot''
require ''open-uri''

class TexisWorker < BackgrounDRb::MetaWorker
   set_worker_name :texis_worker
   pool_size 40


   def create(args = nil)
     logger.info "Creating Texis Worker Instance"
   end

   def search(arg)
     thread_pool.defer(:perform_search, arg)
   end

   def perform_search(arg)
     # calls out to a server runs a search, gets parsed by hpricot and  
then caches the results
     persistent_job.finish!
   end
end


Thanks,
Curtis