Displaying 1 result from an estimated 1 matches for "pages_to_scrape".
2008 Jun 10
4
adding results from threads to a collection and returning it
...m basically trying to distribute several web page scraping tasks among
different threads, and have the results from each added to an Array which is
ultimately returned by the backgroundrb worker. Here is an example of what
I''m trying to do in a worker method:
pages = Array.new
pages_to_scrape.each do |url|
thread_pool.defer(url) do |url|
begin
# model object performs the scraping
page = ScrapedPage.new(page.url)
pages << page
rescue
logger.info "page scrape failed"
end...