I have a general question about a web service. My app will (hopefully) parse RSS (XML) so that it returns newly updated feeds. The parsing job is not the problem. The problem is how to let the server do it efficiently. Obviously, one of the methods is the cron job. But it cannot process all if the number of the feeds is too large. Somehow the server does the job little by little. For instance, I have some experience in AppEngine which provides Task Queue that finishes the job little by little maintaining the server intact. Is there such functionality that Ruby on Rails has? so that my app can finish parsing without reaching the server limit? soichi -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/1bf5ba0c58cb520eb815e4f3fa54be41%40ruby-forum.com?hl=en-US. For more options, visit https://groups.google.com/groups/opt_out.
On Wed, May 29, 2013 at 9:14 PM, Soichi Ishida <lists-fsXkhYbjdPsEEoCn2XhGlw@public.gmane.org> wrote:> Obviously, one of the methods is the cron job. But it cannot process > all if the number of the feeds is too large. Somehow the server does > the job little by little. > > For instance, I have some experience in AppEngine which provides Task > Queue that finishes the job little by little maintaining the server > intact. > > Is there such functionality that Ruby on Rails has? so that my app can > finish parsing without reaching the server limit?If you want to queue projects why would you use Cron at all? And Rails 4 does it''s called Queuing but you don''t need Rails to do that, you could use Resque or Sidkiq which Queueing does but in with an integrated API. And your apps limit is only defined by your servers limit. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/CAM5XQnzrar_gAcGgM%2B4kofGX0GA8SgavZgsajEas85%2BOsSzMsw%40mail.gmail.com?hl=en-US. For more options, visit https://groups.google.com/groups/opt_out.
thanks! Resque and Sidliq sound great! I haven''t even heard about them until now. -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/35c86d788fc02ff1f5bfa7368a99fd83%40ruby-forum.com?hl=en-US. For more options, visit https://groups.google.com/groups/opt_out.
If you are talking about a system where a polling happens for jobs, and then the processing of the jobs happens independently, I''d suggest decoupling the two parts. This is how I would go about it ( using AWS ) * 1 EC2 instance to do the polling. * an SQS( a queue ) into which the polling EC2 pushes jobs. * An autoscaling EC2 group which scales based on the number of jobs in the queue and processes the jobs in the queue. - Emil On 30 May 2013 08:21, Soichi Ishida <lists-fsXkhYbjdPsEEoCn2XhGlw@public.gmane.org> wrote:> thanks! Resque and Sidliq sound great! I haven''t even heard about them > until now. > > -- > Posted via http://www.ruby-forum.com/. > > -- > You received this message because you are subscribed to the Google Groups > "Ruby on Rails: Talk" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > To view this discussion on the web visit > https://groups.google.com/d/msgid/rubyonrails-talk/35c86d788fc02ff1f5bfa7368a99fd83%40ruby-forum.com?hl=en-US > . > For more options, visit https://groups.google.com/groups/opt_out. > > >-- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/CAJ%3Dox-C_Bqqbx8xwM3VP7pxwiPQoxaOfwTkNFy2gbsVvsetaig%40mail.gmail.com?hl=en-US. For more options, visit https://groups.google.com/groups/opt_out.