similar to: Threadpool and queuing of tasks

Greetings all, Unit tests are passing, but I''m having mucho problems getting brb to run without exceptions for more than 6 hours. I''m seeing a mixture of the following 2 exceptions: 20070605-11:26:56 (21497) failed to find slave socket - (RuntimeError) 20070605-11:26:56 (21497) /usr/local/lib/ruby/gems/1.8/gems/ slave-1.2.1/lib/slave.rb:435:in `initialize''

Hi All, It might be lack of sleep, but I am struggling to accurately limit our pool size. It seems like I can specify it on the server with the -s command line option and also on the client via the YAML pool_size. Is that right? Which one wins? Our problem is that we are getting about 40 threads on each backgroundrb box and it''s flooring our db and each bgrb box. We want around 8.

I have a previous question regarding long tasks and the thread_pool (sorry for the dup, I didn''t see the first go through). To try and track things down, I made a change based on a suggestion found in the archives. I moved my import contacts worker to its own file and set the pool_size to 1. class ImportContactsWorker < BackgrounDRb::MetaWorker set_worker_name

I have a scheduled worker that runs every minute, checking for new eBay auctions to post. If it finds an auction, then it posts it. If there are no auctions in the database that are queued up, it simply does nothing. I got the worker to finally run as a scheduled worker, every minute. However, after about 30-45 scheduled runs, it simply stops running. My backgroundrb_schedules.yml file: ------

Hi, i''m going to implement a syndication-service, which will get lists in xml with some meta-data an enclosed video files, which will get encoded at the end. The syndication run will be startet every five minutes of a full hour. So i thought to build 4 Worker. One for checking which feeds to syndicate (syndication_worker) at a specific time, one for processing the list

I''ve been checking out all the async gems and plugins for Rails, and decided to give BackgroundRb a shot. Unfortunately, I''ve not even been able to get past the initial stages. Whenever I call MiddleMan.new_worker, it invariably returns nil. The log file remains empty. I''ve only tried this from the console so far. If I can''t even do this successfully,

I''m trying to run a single worker that could perform a periodic task for a given user. >From a controller, I imagine something like: def start_job MiddleMan.ask_work(:worker => :foo_worker, :worker_method => :perform_task, :data => { :user_id = current_user.id }) end def check_job @status = MiddleMan.ask_status(:worker => :foo_worker)[current_user.id] end

After some initial stumblings, I think I''ve got the hang of backgroundrb. It''s great! I''d been thinking for many many months how cool something like this would be! I''m trying to make a "job queue". That is, a pool of worker threads monitor a queue. When a job appears, one of the workers grabs it and executes the task. When complete, the worker

Hi, Newbie here. I''ve got a worker (for generating PDF reports) that uses the "thread_pool" to allow processing multiple reports simultaneously and queue up any requests that exceed the thread pool (pool_size = 10 currently). def process_pdf(user) thread_pool.defer(user) do |user| makepdf(user) end end My question is: I use a mutex to handle

My Rails site uses BackgroundDRb and I have a couple of questions: 1. Can someone point me to some sample code/examples for how to use thread_pool? The website doc says to add a line ====== pool_size 10 ====== in my Worker class which seems straightforward. I wasn''t able to understand this part though: ========= thread_pool.defer(wiki_scrap_url) { |wiki_url| scrap_wikipedia(wiki_url) }

I use BDRb to process RSS feeds for users on my site (http://feedflix.com). I have a batch job that queries the DB for records that haven''t been updated in the last so many hours and kicks off a background job for each of them. If N records are returned by the DB, N background jobs gets queued and get done serially. As long as N is 255 or under, everything works like a charm.

I use backgroundrb for many long tasks in my system, but I''m having issues with one in particular. Two large tasks for me are importing people and updating companies. def import_contacts(args = nil) thread_pool.defer(args) do |job_id| begin job = ImportJob.find(job_id) job.process_job rescue => err logger.error "MscWorker#import_contacts

I''m having some trouble with using brb for polling for work. Basically, what I want to do is update the state of a record and have brb notice and start a process. I do not want to call ask_work for each process because there may be several places where the object''s state is changed, and I only want to process once. Plus, I can schedule things better this way. My first thought

Forgive me if this has been addressed somewhere, but I have searched and can''t come up with anything. I am basically trying to distribute several web page scraping tasks among different threads, and have the results from each added to an Array which is ultimately returned by the backgroundrb worker. Here is an example of what I''m trying to do in a worker method: pages =

Hello, I am trying to make DRb calls to a external server from a backgrounDRb worker, but I''m getting always the next exception when I make the remote call. > ERROR: > drbunix:///tmp/backgroundrb.11175/experimenter_worker_3e899de945f0803a82cd257cfac316d0_0_0.0738904928422051 > - #<Errno::ENOENT: No such file or directory - >

Hi everyone, I''ve a class that try to spawn two workers, every 5 seconds. The code is something like: while true do MiddleMan.new_worker(:class => :researches_worker,:job_key => :researches) MiddleMan.new_worker(:class => :products_worker,:job_key => :products) sleep(5) end I''m using ":job_key", so if a worker hasn''t already ended its work

I''ve got some workers that I want to have running all the time. Right now I''m just launching them manually, by requesting a special page in my rails app that includes lines like: MiddleMan.new_worker(:class=> :receiver, :job_key=>:r, :args=>{:sleep_time=>10}) This strikes me as a really weak way to fire up my workers. I basically want to automate things so

I''m struggling to keep backgroundrb v2 running for more than 24 hours. It appears to be running fine for a while, then tries to fire off a worker (exactly the same worker it''s been running for the past few hours), and it suddenly dies. I haven''t found any pattern for the cause of this. In backgroundrb_server.log, the final entry is this : 20070206-10:41:45

We have a fairly expensive task that we''d rather not schedule too many instances of in parallel. Couldn''t get pool_size to limit the amount of workers, and I figure that''s hardly optimal anyway, so would like to have the work queued up for a single named worker instead, running as soon as possible. Using MiddleMan.schedule_worker() with :job_key does start off a

I have a job that I am starting from a Rails controller in production mode using MiddleMan: MiddleMan.new_worker(:class => :import_ccg_category_worker, :args => { :category_id => @category.id, :remote_category_id => params[:remote_category_id], :description => "Importing CCG products for #{@ category.name}" }) Both of the category ID variables are just basic integers. I