search for: scheduled_crawl

...while worker is processing (thats , while its doing actual crawling, not sitting idle for schedule to reach) task, it won''t be able to respond to requests from rails immediately. You can mitigate this easily by adding scheduled tasks to thread pool as well. What I mean is, say method "scheduled_crawling" initiates scheduled crawling of pages preferred by all users on current website. You can schedule "scheduled_crawling" through cron scheduler or normal add_periodic_timer method. Now, since you do not want scheduled processing to block rest of the request/response cycle, you can...