Displaying 20 results from an estimated 52 matches for "thread_pooling".
2008 May 09
1
register_status for excess thread_pool?
Hi,
Newbie here. I''ve got a worker (for generating PDF reports) that
uses the "thread_pool" to allow processing multiple reports
simultaneously and queue up any requests that exceed the thread
pool (pool_size = 10 currently).
def process_pdf(user)
thread_pool.defer(user) do |user|
makepdf(user)
end
end
My question is: I use a mutex to handle
2008 Jan 03
1
Thread_pool bug?
I have a previous question regarding long tasks and the thread_pool (sorry
for the dup, I didn''t see the first go through). To try and track things
down, I made a change based on a suggestion found in the archives. I moved
my import contacts worker to its own file and set the pool_size to 1.
class ImportContactsWorker < BackgrounDRb::MetaWorker
set_worker_name
2007 Dec 19
6
thread_pooling sleeping
I''m trying to run a single worker that could perform a periodic task
for a given user.
>From a controller, I imagine something like:
def start_job
MiddleMan.ask_work(:worker => :foo_worker, :worker_method => :perform_task,
:data => { :user_id = current_user.id })
end
def check_job
@status = MiddleMan.ask_status(:worker => :foo_worker)[current_user.id]
end
2008 Feb 03
1
Defer multiple methods within the same worker to the thread pool?
Within a threaded worker, I would like to have multiple methods. Will
the following code work?
def method_1(args)
thread_pool.defer(args) do |args|
#work
end
end
def method_2(args)
thread_pool.defer(args) do |args|
#work
end
end
. . . or, should I have one "meta"-method and pass which sub-method
to perform via args?
def meta_method(args)
2007 Jan 30
1
Backgroundrb weirdness with multiple mongrels
Hi list,
I''m having some issues getting backgroundrb to play nice with more than
one mongrel. When I run my app on a single mongrel instance it works
fine.
I have two different problems show up in my backgrounrb_server.log. The
top of the backtrace is shown here:
20070130-12:03:03 (92473) failed to find slave socket - (RuntimeError)
20070130-12:03:03 (92473)
2007 Feb 22
1
failed to find slave socket - (RuntimeError)
Hi everyone,
I''ve a class that try to spawn two workers, every 5 seconds.
The code is something like:
while true do
MiddleMan.new_worker(:class => :researches_worker,:job_key
=> :researches)
MiddleMan.new_worker(:class => :products_worker,:job_key => :products)
sleep(5)
end
I''m using ":job_key", so if a worker hasn''t already ended its work
2008 Feb 01
3
Job Queue
After some initial stumblings, I think I''ve got the hang of
backgroundrb. It''s great! I''d been thinking for many many months how
cool something like this would be!
I''m trying to make a "job queue". That is, a pool of worker threads
monitor a queue. When a job appears, one of the workers grabs it and
executes the task. When complete, the worker
2007 Aug 08
3
Cannot start workers in production mode
I have a job that I am starting from a Rails controller in production mode
using MiddleMan:
MiddleMan.new_worker(:class => :import_ccg_category_worker, :args => {
:category_id => @category.id, :remote_category_id =>
params[:remote_category_id], :description => "Importing CCG products for #{@
category.name}" })
Both of the category ID variables are just basic integers. I
2007 Jan 23
0
Error while creating one worker from another
for some strange reason this error occours when i create one worker from
another BUT only when i derive worker class from Worker::Base instead or
Worker::RailsBase - when change it back it works fine, child worker can
have empty do_work, but this error still ocours
Bad file descriptor (Errno::EBADF)
/var/lib/gems/1.8/gems/slave-1.2.0/lib/slave.rb:214:in `for_fd''
2008 Jan 03
1
Memory leak and long process problem
I use backgroundrb for many long tasks in my system, but I''m having issues
with one in particular. Two large tasks for me are importing people and
updating companies.
def import_contacts(args = nil)
thread_pool.defer(args) do |job_id|
begin
job = ImportJob.find(job_id)
job.process_job
rescue => err
logger.error "MscWorker#import_contacts
2008 May 20
1
Couple questions on BDRb and concurrent processing
My Rails site uses BackgroundDRb and I have a couple of questions:
1. Can someone point me to some sample code/examples for how to use
thread_pool? The website doc says to add a line
======
pool_size 10
======
in my Worker class which seems straightforward.
I wasn''t able to understand this part though:
=========
thread_pool.defer(wiki_scrap_url) { |wiki_url| scrap_wikipedia(wiki_url) }
2008 Jun 10
4
adding results from threads to a collection and returning it
Forgive me if this has been addressed somewhere, but I have searched and
can''t come up with anything.
I am basically trying to distribute several web page scraping tasks among
different threads, and have the results from each added to an Array which is
ultimately returned by the backgroundrb worker. Here is an example of what
I''m trying to do in a worker method:
pages =
2007 Feb 07
1
Repeatedly dying with "failed to find slave socket"
I''m struggling to keep backgroundrb v2 running for more than 24
hours. It appears to be running fine for a while, then tries to fire
off a worker (exactly the same worker it''s been running for the past
few hours), and it suddenly dies. I haven''t found any pattern for the
cause of this.
In backgroundrb_server.log, the final entry is this :
20070206-10:41:45
2007 Jun 06
2
struggling with stability
Greetings all,
Unit tests are passing, but I''m having mucho problems getting brb to
run without exceptions for more than 6 hours.
I''m seeing a mixture of the following 2 exceptions:
20070605-11:26:56 (21497) failed to find slave socket - (RuntimeError)
20070605-11:26:56 (21497) /usr/local/lib/ruby/gems/1.8/gems/
slave-1.2.1/lib/slave.rb:435:in `initialize''
2006 Dec 03
0
worker is not executed
Hey,
I''ve got a problem where my worker is not being properly executed....
it used to work in an earlier version of brb, but i realized my
vendor/plugins was out of whack, upgraded to the newer schedule-
oriented release (0.2.1). I can start brb, and run the new_worker
middleman, but ntohing happens.
When putting brb in ''run'' mode, i see this:
undefined method
2007 Dec 17
1
Some more updates, enhancements and fixes
Hi Folks,
Well, Yesterday I pushed some more updates that I was cooking since
last couple of days. Here is a brief list:
1. Results from worker stays even after a worker dies.
2. You can ask backgroundrb to store status objects in memcache
cluster now. by default its stored in master process memory, but I
have introduced an option to allow it to be stored in a memcache
cluster:
2008 Jan 22
2
Threadpool and queuing of tasks
I recently switched over to v1.0, and things are rolling along pretty well.
However, one thing that has always been a little confusing to me is knowing
when to use thread_pool. Since most of my bgrb workers are called from my
web app to process rather than being scheduled, I''m using the thread_pool
for every call. Unfortunately, that means that I have to split up workers
by how many
2007 Apr 09
1
Drb Connection error on multiple dispatch.fcgi ''s
Hi All,
I''m using Backgroundrb as a general purpose long-running-task back-end
(upload processing, email sending, etc), and it''s been a great solution.
However I''ve recently run into some some intermittent connection issues
that have me baffled. I''m running on apache2/fcgid and the problem
occurs in both devlopment and production mode. The problem seems to
2008 Apr 12
2
Sequence of multiple background process submissions
I have a question on how multiple b/g processes that are kicked off get
handled in terms of sequence/schedule.
On my site, a user enters an RSS feed to be processed and since this takes
about 5-10 secs, I pass the process off to a background job and meanwhile do
some Ajaxy spinners and tap-dancing until the job completes and then
redirect_to the appropriate page. This works great.
I also need to
2007 Mar 13
3
Scheduled worker dies after about 30-45 runs
I have a scheduled worker that runs every minute, checking for new eBay
auctions to post. If it finds an auction, then it posts it. If there are no
auctions in the database that are queued up, it simply does nothing. I got
the worker to finally run as a scheduled worker, every minute. However,
after about 30-45 scheduled runs, it simply stops running.
My backgroundrb_schedules.yml file:
------