Displaying 20 results from an estimated 52 matches for "thread_pool".
2008 May 09
1
register_status for excess thread_pool?
Hi,
Newbie here. I''ve got a worker (for generating PDF reports) that
uses the "thread_pool" to allow processing multiple reports
simultaneously and queue up any requests that exceed the thread
pool (pool_size = 10 currently).
def process_pdf(user)
thread_pool.defer(user) do |user|
makepdf(user)
end
end
My question is: I use a mutex to handle synchronizing...
2008 Jan 03
1
Thread_pool bug?
I have a previous question regarding long tasks and the thread_pool (sorry
for the dup, I didn''t see the first go through). To try and track things
down, I made a change based on a suggestion found in the archives. I moved
my import contacts worker to its own file and set the pool_size to 1.
class ImportContactsWorker < BackgrounDRb::MetaWorker
set...
2007 Dec 19
6
thread_pooling sleeping
...current_user.id]
end
My worker is something like:
class FooWorker < BackgrounDRb::MetaWorker
set_worker_name :foo_worker
def create(args=nil)
@mutex = Mutex.new
@mutex.synchronize do
@statuses = {}
register_status(@statuses)
end
end
def do_task(some_user_id)
thread_pool.defer(some_user_id) do |user_id|
user = User.find user_id
save_status user_id, :progress, "Starting Task"
user.do_some_database_stuff
save_status user_id, :progress, "Task Stage 2"
user.do_some_other_database_stuff
save_status user_id, :progre...
2008 Feb 03
1
Defer multiple methods within the same worker to the thread pool?
Within a threaded worker, I would like to have multiple methods. Will
the following code work?
def method_1(args)
thread_pool.defer(args) do |args|
#work
end
end
def method_2(args)
thread_pool.defer(args) do |args|
#work
end
end
. . . or, should I have one "meta"-method and pass which sub-method
to perform via args?
def meta_method(args)
thread_pool.defer(args) do |args|
if a...
2007 Jan 30
1
Backgroundrb weirdness with multiple mongrels
...2:03:03 (92473) /usr/local/www/rails-apps/myapp/releases/20070130101502/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new_worker''
20070130-12:03:03 (92473) /usr/local/www/rails-apps/myapp/releases/20070130101502/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:36:in `dispatch''
20070130-12:03:03 (92473) /usr/local/www/rails-apps/myapp/releases/20070130101502/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `initialize''
20070130-12:03:03 (92473) /usr/local/www/rails-apps/myapp/releases/20070130101502/vendor/plug...
2007 Feb 22
1
failed to find slave socket - (RuntimeError)
...w''
20070222-15:13:37 (15574) /Users/abernesco/Documents/ruby/alfaomega/
vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:
210:in `new_worker''
20070222-15:13:37 (15574) /Users/abernesco/Documents/ruby/alfaomega/
vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:
36:in `dispatch''
20070222-15:13:37 (15574) /Users/abernesco/Documents/ruby/alfaomega/
vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:
22:in `initialize''
20070222-15:13:37 (15574) /Users/abernesco/Documents/ruby/alfaomega/
vendor/plugins/backgroundrb/ser...
2008 Feb 01
3
Job Queue
After some initial stumblings, I think I''ve got the hang of
backgroundrb. It''s great! I''d been thinking for many many months how
cool something like this would be!
I''m trying to make a "job queue". That is, a pool of worker threads
monitor a queue. When a job appears, one of the workers grabs it and
executes the task. When complete, the worker
2007 Aug 08
3
Cannot start workers in production mode
...it'' for nil:NilClass - (NoMethodError)
/mnt/home/magicarsenal/admin/releases/20070807014939/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:217:in
`new_worker''
/mnt/home/magicarsenal/admin/releases/20070807014939/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:36:in
`dispatch''
/mnt/home/magicarsenal/admin/releases/20070807014939/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in
`initialize''
/mnt/home/magicarsenal/admin/releases/20070807014939/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22...
2007 Jan 23
0
Error while creating one worker from another
...r/plugins/backgroundrb/server/lib/back
groundrb/middleman.rb:210:in `new''
/home/swistak/rails+backgroundrb/vendor/plugins/backgroundrb/server/lib/back
groundrb/middleman.rb:210:in `new_worker''
/home/swistak/rails+backgroundrb/vendor/plugins/backgroundrb/server/lib/back
groundrb/thread_pool.rb:36:in `dispatch''
/home/swistak/rails+backgroundrb/vendor/plugins/backgroundrb/server/lib/back
groundrb/thread_pool.rb:22:in `initialize''
/home/swistak/rails+backgroundrb/vendor/plugins/backgroundrb/server/lib/back
groundrb/thread_pool.rb:22:in `new''
/home/swistak/rai...
2008 Jan 03
1
Memory leak and long process problem
I use backgroundrb for many long tasks in my system, but I''m having issues
with one in particular. Two large tasks for me are importing people and
updating companies.
def import_contacts(args = nil)
thread_pool.defer(args) do |job_id|
begin
job = ImportJob.find(job_id)
job.process_job
rescue => err
logger.error "MscWorker#import_contacts failed! #{err.class}:
#{err}"
end
end
end
def update_company_from_vendor(args = nil)
thread_pool.defer...
2008 May 20
1
Couple questions on BDRb and concurrent processing
My Rails site uses BackgroundDRb and I have a couple of questions:
1. Can someone point me to some sample code/examples for how to use
thread_pool? The website doc says to add a line
======
pool_size 10
======
in my Worker class which seems straightforward.
I wasn''t able to understand this part though:
=========
thread_pool.defer(wiki_scrap_url) { |wiki_url| scrap_wikipedia(wiki_url) }
=========
Can someone explain this code pleas...
2008 Jun 10
4
adding results from threads to a collection and returning it
...web page scraping tasks among
different threads, and have the results from each added to an Array which is
ultimately returned by the backgroundrb worker. Here is an example of what
I''m trying to do in a worker method:
pages = Array.new
pages_to_scrape.each do |url|
thread_pool.defer(url) do |url|
begin
# model object performs the scraping
page = ScrapedPage.new(page.url)
pages << page
rescue
logger.info "page scrape failed"
end
end
end
end...
2007 Feb 07
1
Repeatedly dying with "failed to find slave socket"
...../vendor/plugins/backgroundrb/server/lib/
backgroundrb/middleman.rb:210:in `new''
20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
backgroundrb/middleman.rb:210:in `new_worker''
20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
backgroundrb/thread_pool.rb:36:in `dispatch''
20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
backgroundrb/thread_pool.rb:22:in `initialize''
20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
backgroundrb/thread_pool.rb:22:in `new''
20070206-10:41:45 (5189)...
2007 Jun 06
2
struggling with stability
Greetings all,
Unit tests are passing, but I''m having mucho problems getting brb to
run without exceptions for more than 6 hours.
I''m seeing a mixture of the following 2 exceptions:
20070605-11:26:56 (21497) failed to find slave socket - (RuntimeError)
20070605-11:26:56 (21497) /usr/local/lib/ruby/gems/1.8/gems/
slave-1.2.1/lib/slave.rb:435:in `initialize''
2006 Dec 03
0
worker is not executed
...undefined method `wait'' for nil:NilClass - (NoMethodError)
/Users/james/Sites/client/backend/trunk/vendor/plugins/backgroundrb/
server/lib/backgroundrb/middleman.rb:217:in `new_worker''
/Users/james/Sites/client/backend/trunk/vendor/plugins/backgroundrb/
server/lib/backgroundrb/thread_pool.rb:36:in `dispatch''
/Users/james/Sites/client/backend/trunk/vendor/plugins/backgroundrb/
server/lib/backgroundrb/thread_pool.rb:22:in `initialize''
/Users/james/Sites/client/backend/trunk/vendor/plugins/backgroundrb/
server/lib/backgroundrb/thread_pool.rb:22:in `new''
/Us...
2007 Dec 17
1
Some more updates, enhancements and fixes
...memory, but I
have introduced an option to allow it to be stored in a memcache
cluster:
#backgroundbr.yml
:result_storage:
:memcache: "10.10.10.2.11211,10.10.10.6:11211"
3. Check for Ruby version.
4. Introduced Thread pool now, all the workers have access to
"thread_pool" object, which can be used to run concurrent tasks in
ruby threads:
An example,
def scrap_wikipedia(text)
thread_pool.defer(text) do |text|
scrap_and_store_text(text)
end
end
Please consult backgroundrb documentation for more information ab...
2008 Jan 22
2
Threadpool and queuing of tasks
I recently switched over to v1.0, and things are rolling along pretty well.
However, one thing that has always been a little confusing to me is knowing
when to use thread_pool. Since most of my bgrb workers are called from my
web app to process rather than being scheduled, I''m using the thread_pool
for every call. Unfortunately, that means that I have to split up workers
by how many threads I can have. It would be great if one worker could
partition a single...
2007 Apr 09
1
Drb Connection error on multiple dispatch.fcgi ''s
...ib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:454:in `initialize''
/home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new_worker''
/home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:36:in `dispatch''
/home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `dispatch''
/home/pascal/projects/work/sitenoodler/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:199:in `new_worker''
/usr/...
2008 Apr 12
2
Sequence of multiple background process submissions
I have a question on how multiple b/g processes that are kicked off get
handled in terms of sequence/schedule.
On my site, a user enters an RSS feed to be processed and since this takes
about 5-10 secs, I pass the process off to a background job and meanwhile do
some Ajaxy spinners and tap-dancing until the job completes and then
redirect_to the appropriate page. This works great.
I also need to
2007 Mar 13
3
Scheduled worker dies after about 30-45 runs
...rb/server/lib/backgroundrb/middleman.rb:210:in
`new''
20070312-18:57:00 (24028)
/home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:210:in
`new_worker''
20070312-18:57:00 (24028)
/home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:36:in
`dispatch''
20070312-18:57:00 (24028)
/home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in
`initialize''
20070312-18:57:00 (24028)
/home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in
`new'&...