Hi, I have been researching about this for couple of days now. My rails application needs to send email notifications to large number of customers. Before sending the message, application customizes content for each recipient. The requirement compels me to call ActionMailer''s deliver method for each email address. I would like to send notifications asynchronously. I looked into backgroundRB. I have been playing around with it for almost two days and I can''t get it working on cygwin. However, my main concern is the poor performance of backgroundRB for moderate or high scalability needs. I also looked into Threads in ruby. Since rails is single threaded and especially ActiveRecord is not thread safe, I dropped that idea too. What are the options generally used for asynchronous jobs in rails ? From google search I find only two options mentioned above. Please tell me your experience with either of them. Is there any other solution that I have overlooked ? Thank you! --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Hi, I have been researching about this for couple of days now. My rails application needs to send email notifications to large number of customers. Before sending the message, application customizes content for each recipient. The requirement compels me to call ActionMailer''s deliver method for each email address. I would like to send notifications asynchronously. I looked into backgroundRB. I have been playing around with it for almost two days and I can''t get it working on cygwin. However, my main concern is the poor performance of backgroundRB for moderate or high scalability needs. I also looked into Threads in ruby. Since rails is single threaded and especially ActiveRecord is not thread safe, I dropped that idea too. What are the options generally used for asynchronous jobs in rails ? From google search I find only two options mentioned above. Please tell me your experience with either of them. Is there any other solution that I have overlooked ? Thank you! --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
william.harding-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
2008-Jun-10 21:37 UTC
Re: Asynchronous jobs in Rails
We''ve used BackgroundRb for our light-to-this-point asynchronous processing needs. It was a pain in the butt to get working initially (and we can still only run it on our Linux boxes, not on Windows), but I''ve perceived it to be probably the most frequently used asynchronous solution for Rails, so we''re going to go with it until we can''t. Though it''s a memory and process hog, I don''t think there would be trouble sending many mails with it, as long as its running on a machine with a lot of memory, and as long as those mails aren''t spam sent to me inbox. :) Another popular asynchronous option is to combine a queuing system with rake tasks that run jobs as they are put in the queue. Starling is a fairly reliable and easy to setup queuing solution developed and used by the folks at Twitter. You can use it with Workling to automatically process your tasks, or you can write a rake task that runs via crontab and checks for jobs to run on some timed interval. There''s some documentation on setting up Starling on my blog. Bill http://www.williambharding.com/blog On Jun 10, 2:16 pm, procyon <kunal...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> Hi, > > I have been researching about this for couple of days now. My rails > application needs to send email notifications to large number of > customers. Before sending the message, application customizes content > for each recipient. The requirement compels me to call ActionMailer''s > deliver method for each email address. > > I would like to send notifications asynchronously. I looked into > backgroundRB. I have been playing around with it for almost two days > and I can''t get it working on cygwin. However, my main concern is the > poor performance of backgroundRB for moderate or high scalability > needs. > > I also looked into Threads in ruby. Since rails is single threaded and > especially ActiveRecord is not thread safe, I dropped that idea too. > > What are the options generally used for asynchronous jobs in rails ? > From google search I find only two options mentioned above. Please > tell me your experience with either of them. Is there any other > solution that I have overlooked ? > > Thank you!--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Thanks for you response. I was thinking about Amazon queue service and if that might help me here. Our production servers are on linux but my dev machine ( to my dismay) is a windows machine. BackgroundRB seems promising as along as I am able to get it to run. On Jun 10, 2:37 pm, "william.hard...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org" <william.hard...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> We''ve used BackgroundRb for our light-to-this-point asynchronous > processing needs. It was a pain in the butt to get working initially > (and we can still only run it on our Linux boxes, not on Windows), but > I''ve perceived it to be probably the most frequently used asynchronous > solution for Rails, so we''re going to go with it until we can''t. > Though it''s a memory and process hog, I don''t think there would be > trouble sending many mails with it, as long as its running on a > machine with a lot of memory, and as long as those mails aren''t spam > sent to me inbox. :) > > Another popular asynchronous option is to combine a queuing system > with rake tasks that run jobs as they are put in the queue. Starling > is a fairly reliable and easy to setup queuing solution developed and > used by the folks at Twitter. You can use it with Workling to > automatically process your tasks, or you can write a rake task that > runs via crontab and checks for jobs to run on some timed interval. > > There''s some documentation on setting up Starling on my blog. > > Billhttp://www.williambharding.com/blog > > On Jun 10, 2:16 pm, procyon <kunal...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > Hi, > > > I have been researching about this for couple of days now. My rails > > application needs to send email notifications to large number of > > customers. Before sending the message, application customizes content > > for each recipient. The requirement compels me to call ActionMailer''s > > deliver method for each email address. > > > I would like to send notifications asynchronously. I looked into > > backgroundRB. I have been playing around with it for almost two days > > and I can''t get it working on cygwin. However, my main concern is the > > poor performance of backgroundRB for moderate or high scalability > > needs. > > > I also looked into Threads in ruby. Since rails is single threaded and > > especially ActiveRecord is not thread safe, I dropped that idea too. > > > What are the options generally used for asynchronous jobs in rails ? > > From google search I find only two options mentioned above. Please > > tell me your experience with either of them. Is there any other > > solution that I have overlooked ? > > > Thank you!--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Hey Procyon, I''ve recently set up backgroundrb to work with my app and it performs an emailing function as well. Althought I haven''t put it through a rigorous stress test yet, one test that I did conduct was setting my Worker method to run every 1 second and having it process 200 emails each time. The server was a 512mb slice from slicehost. It has 2 rails sites on it right now. At normal operations (with bgrb stopped), it has about 403mb free and 108mb used. During my test for about 5 minutes, the server memory usage went to around 200mb free. I was using sendmail to send off the emails... and I didn''t check how much of the memory usage that was using. I should also note that the Worker code did not use the thread_pool feature so that might help bring memory usage down as well. For my purpose, I think I won''t be sending more than 1000 emails per hour and so it looks like it will be able to serve my needs easily unless my traffic goes crazy like FaceStat. Hope this helps. ~ mel On Jun 10, 2:53 pm, procyon <kunal...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> Thanks for you response. I was thinking about Amazon queue service and > if that might help me here. Our production servers are on linux but my > dev machine ( to my dismay) is a windows machine. BackgroundRB seems > promising as along as I am able to get it to run. > > On Jun 10, 2:37 pm, "william.hard...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org" > > <william.hard...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > We''ve used BackgroundRb for our light-to-this-point asynchronous > > processing needs. It was a pain in the butt to get working initially > > (and we can still only run it on our Linux boxes, not on Windows), but > > I''ve perceived it to be probably the most frequently used asynchronous > > solution for Rails, so we''re going to go with it until we can''t. > > Though it''s a memory and process hog, I don''t think there would be > > trouble sending many mails with it, as long as its running on a > > machine with a lot of memory, and as long as those mails aren''t spam > > sent to me inbox. :) > > > Another popular asynchronous option is to combine a queuing system > > with rake tasks that run jobs as they are put in the queue. Starling > > is a fairly reliable and easy to setup queuing solution developed and > > used by the folks at Twitter. You can use it with Workling to > > automatically process your tasks, or you can write a rake task that > > runs via crontab and checks for jobs to run on some timed interval. > > > There''s some documentation on setting up Starling on my blog. > > > Billhttp://www.williambharding.com/blog > > > On Jun 10, 2:16 pm, procyon <kunal...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > Hi, > > > > I have been researching about this for couple of days now. My rails > > > application needs to send email notifications to large number of > > > customers. Before sending the message, application customizes content > > > for each recipient. The requirement compels me to call ActionMailer''s > > > deliver method for each email address. > > > > I would like to send notifications asynchronously. I looked into > > > backgroundRB. I have been playing around with it for almost two days > > > and I can''t get it working on cygwin. However, my main concern is the > > > poor performance of backgroundRB for moderate or high scalability > > > needs. > > > > I also looked into Threads in ruby. Since rails is single threaded and > > > especially ActiveRecord is not thread safe, I dropped that idea too. > > > > What are the options generally used for asynchronous jobs in rails ? > > > From google search I find only two options mentioned above. Please > > > tell me your experience with either of them. Is there any other > > > solution that I have overlooked ? > > > > Thank you!--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
PS: If you need any help with getting things set up, feel free to IM me on Gtalk at melvin @-AT=@ volcanicmarketing.com. I''m no expert with it but I''ll try to help On Jun 10, 2:53 pm, procyon <kunal...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> Thanks for you response. I was thinking about Amazon queue service and > if that might help me here. Our production servers are on linux but my > dev machine ( to my dismay) is a windows machine. BackgroundRB seems > promising as along as I am able to get it to run. > > On Jun 10, 2:37 pm, "william.hard...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org" > > <william.hard...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > We''ve used BackgroundRb for our light-to-this-point asynchronous > > processing needs. It was a pain in the butt to get working initially > > (and we can still only run it on our Linux boxes, not on Windows), but > > I''ve perceived it to be probably the most frequently used asynchronous > > solution for Rails, so we''re going to go with it until we can''t. > > Though it''s a memory and process hog, I don''t think there would be > > trouble sending many mails with it, as long as its running on a > > machine with a lot of memory, and as long as those mails aren''t spam > > sent to me inbox. :) > > > Another popular asynchronous option is to combine a queuing system > > with rake tasks that run jobs as they are put in the queue. Starling > > is a fairly reliable and easy to setup queuing solution developed and > > used by the folks at Twitter. You can use it with Workling to > > automatically process your tasks, or you can write a rake task that > > runs via crontab and checks for jobs to run on some timed interval. > > > There''s some documentation on setting up Starling on my blog. > > > Billhttp://www.williambharding.com/blog > > > On Jun 10, 2:16 pm, procyon <kunal...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > Hi, > > > > I have been researching about this for couple of days now. My rails > > > application needs to send email notifications to large number of > > > customers. Before sending the message, application customizes content > > > for each recipient. The requirement compels me to call ActionMailer''s > > > deliver method for each email address. > > > > I would like to send notifications asynchronously. I looked into > > > backgroundRB. I have been playing around with it for almost two days > > > and I can''t get it working on cygwin. However, my main concern is the > > > poor performance of backgroundRB for moderate or high scalability > > > needs. > > > > I also looked into Threads in ruby. Since rails is single threaded and > > > especially ActiveRecord is not thread safe, I dropped that idea too. > > > > What are the options generally used for asynchronous jobs in rails ? > > > From google search I find only two options mentioned above. Please > > > tell me your experience with either of them. Is there any other > > > solution that I have overlooked ? > > > > Thank you!--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
You should also check out Starling with the Workling plugin. I found it to be a much easier, robust, and efficient method for doing background processing. Check out http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/ for a series I did after not being pleased with how backgroundrb was working. On Jun 10, 8:10 pm, mel ram <mel...-IA9i2KS8NFBpEKysQ+xqfPpXobYPEAuW@public.gmane.org> wrote:> PS: If you need any help with getting things set up, feel free to IM > me on Gtalk at melvin @-AT=@ volcanicmarketing.com. I''m no expert with > it but I''ll try to help > > On Jun 10, 2:53 pm, procyon <kunal...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > Thanks for you response. I was thinking about Amazon queue service and > > if that might help me here. Our production servers are on linux but my > > dev machine ( to my dismay) is a windows machine. BackgroundRB seems > > promising as along as I am able to get it to run. > > > On Jun 10, 2:37 pm, "william.hard...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org" > > > <william.hard...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > We''ve used BackgroundRb for our light-to-this-point asynchronous > > > processing needs. It was a pain in the butt to get working initially > > > (and we can still only run it on our Linux boxes, not on Windows), but > > > I''ve perceived it to be probably the most frequently used asynchronous > > > solution for Rails, so we''re going to go with it until we can''t. > > > Though it''s a memory and process hog, I don''t think there would be > > > trouble sending many mails with it, as long as its running on a > > > machine with a lot of memory, and as long as those mails aren''t spam > > > sent to me inbox. :) > > > > Another popular asynchronous option is to combine a queuing system > > > with rake tasks that run jobs as they are put in the queue. Starling > > > is a fairly reliable and easy to setup queuing solution developed and > > > used by the folks at Twitter. You can use it with Workling to > > > automatically process your tasks, or you can write a rake task that > > > runs via crontab and checks for jobs to run on some timed interval. > > > > There''s some documentation on setting up Starling on my blog. > > > > Billhttp://www.williambharding.com/blog > > > > On Jun 10, 2:16 pm, procyon <kunal...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > > Hi, > > > > > I have been researching about this for couple of days now. My rails > > > > application needs to send email notifications to large number of > > > > customers. Before sending the message, application customizes content > > > > for each recipient. The requirement compels me to call ActionMailer''s > > > > deliver method for each email address. > > > > > I would like to send notifications asynchronously. I looked into > > > > backgroundRB. I have been playing around with it for almost two days > > > > and I can''t get it working on cygwin. However, my main concern is the > > > > poor performance of backgroundRB for moderate or high scalability > > > > needs. > > > > > I also looked into Threads in ruby. Since rails is single threaded and > > > > especially ActiveRecord is not thread safe, I dropped that idea too. > > > > > What are the options generally used for asynchronous jobs in rails ? > > > > From google search I find only two options mentioned above. Please > > > > tell me your experience with either of them. Is there any other > > > > solution that I have overlooked ? > > > > > Thank you!--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On Wed, Jun 11, 2008 at 3:07 AM, william.harding-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org <william.harding-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> > We''ve used BackgroundRb for our light-to-this-point asynchronous > processing needs. It was a pain in the butt to get working initially > (and we can still only run it on our Linux boxes, not on Windows), but > I''ve perceived it to be probably the most frequently used asynchronous > solution for Rails, so we''re going to go with it until we can''t. > Though it''s a memory and process hog, I don''t think there would be > trouble sending many mails with it, as long as its running on a > machine with a lot of memory, and as long as those mails aren''t spam > sent to me inbox. :)I am afraid, that has changed unless you are running old svn version of backgroundrb: http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/ Many users have found git version to be quite stable and since now bdrb uses fork and exec ( meaning it replaces old process image with new one), there is no sharing of resources between workers, which used to prevent garbage collection of objects, which were allocated before fork. Its not a problem of bdrb per se, since Ruby interpretor is known to be not COW(copy on write) friendly (for uninitiated, it means that, in ruby, when you fork, all the objects in scope have a bit set in them, so as they get copied in forked process. This is fine, but this prevents, those objects from garbage collected. Ideally, it should copy only when an object''s value is changed in child , hence copy on write . But Ruby doesn''t do that. You will find plenty of material about this on internet) Github version of bdrb uses fork and exec, and hence it means that, objects are no longer shared and hence garbage collection can work as usual. In fact, if you want access to rails environment, this is as much as lean and thin you can get. You can''t have your cake and eat it too.> > Another popular asynchronous option is to combine a queuing system > with rake tasks that run jobs as they are put in the queue. Starling > is a fairly reliable and easy to setup queuing solution developed and > used by the folks at Twitter. You can use it with Workling to > automatically process your tasks, or you can write a rake task that > runs via crontab and checks for jobs to run on some timed interval. > > There''s some documentation on setting up Starling on my blog. > > Bill > http://www.williambharding.com/blog > > On Jun 10, 2:16 pm, procyon <kunal...-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> Hi, >> >> I have been researching about this for couple of days now. My rails >> application needs to send email notifications to large number of >> customers. Before sending the message, application customizes content >> for each recipient. The requirement compels me to call ActionMailer''s >> deliver method for each email address. >> >> I would like to send notifications asynchronously. I looked into >> backgroundRB. I have been playing around with it for almost two days >> and I can''t get it working on cygwin. However, my main concern is the >> poor performance of backgroundRB for moderate or high scalability >> needs. >> >> I also looked into Threads in ruby. Since rails is single threaded and >> especially ActiveRecord is not thread safe, I dropped that idea too. >> >> What are the options generally used for asynchronous jobs in rails ? >> From google search I find only two options mentioned above. Please >> tell me your experience with either of them. Is there any other >> solution that I have overlooked ? >> >> Thank you! > > >-- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---