Greetings, I''ve recently deployed an app that is having some serious memory consumption issue. Dropping mongrels quite frequently when the threshold is met. Initially, we chalked this problem up to RMagick (I know, I know, but we needed watermarking). We quickly optimized things so all images were only processed once then neatly stored in their own folder. When an image is requested, we confirm that it doesn''t exist before processing it. Been tested, and this is working as it should. According to some memory profiling we''ve done, there are a couple of key pages that are creating large amounts of strings that are not being reclaimed. Has anyone ever experienced issues similar to this? Cheers, DJ -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
What is memory limit per process? On Apr 1, 2:49 pm, Dave John <rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> wrote:> Greetings, > > I''ve recently deployed an app that is having some serious memory > consumption issue. Dropping mongrels quite frequently when the > threshold is met. > > Initially, we chalked this problem up to RMagick (I know, I know, but we > needed watermarking). We quickly optimized things so all images were > only processed once then neatly stored in their own folder. When an > image is requested, we confirm that it doesn''t exist before processing > it. Been tested, and this is working as it should. > > According to some memory profiling we''ve done, there are a couple of key > pages that are creating large amounts of strings that are not being > reclaimed. > > Has anyone ever experienced issues similar to this? > > Cheers, > DJ > -- > Posted viahttp://www.ruby-forum.com/.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Dave - On 1-Apr-08, at 2:49 PM, Dave John wrote:> > Greetings, > > I''ve recently deployed an app that is having some serious memory > consumption issue. Dropping mongrels quite frequently when the > threshold is met. > > Initially, we chalked this problem up to RMagick (I know, I know, > but we > needed watermarking). We quickly optimized things so all images were > only processed once then neatly stored in their own folder. When an > image is requested, we confirm that it doesn''t exist before processing > it. Been tested, and this is working as it should. > > According to some memory profiling we''ve done, there are a couple of > key > pages that are creating large amounts of strings that are not being > reclaimed. > > Has anyone ever experienced issues similar to this? > > Cheers, > DJI too am presently using monit to manage this until I can track down the leak. Our mongrel''s high water mark is 280 mb, and each of the 11 restart a few times per day. Rmagick is in place, but isolated, and likely not the source. There''s an article somewhere (lost in my booksmarks) that helps track down leaks. As I suspect my leaks are very specific, I have not appealed to the general group, but perhaps this thread will turn out something interesting. Jodi --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Hey Jodi, I believe our mongrels memory threshold is in the same ballpark. Here are a couple of articles we''ve been looking at, following along with these processes and using the memory analysis tool in the second has revealed that the strings not getting reclaimed are blank.. http://codehappy.wordpress.com/2007/01/17/tracking-a-memory-leak-in-rails/ http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling-with-rails We''ve also been using bleak_house: http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html yet still can''t nail down exactly whats going on here. Jodi, what version of Rails are you using for you project? Thanks, Dave -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Thanx Dave - On 1-Apr-08, at 5:10 PM, Dave John wrote:> > Hey Jodi, > > > I believe our mongrels memory threshold is in the same ballpark. Here > are a couple of articles we''ve been looking at, following along with > these processes and using the memory analysis tool in the second has > revealed that the strings not getting reclaimed are blank.. > > http://codehappy.wordpress.com/2007/01/17/tracking-a-memory-leak-in-rails/ > http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling-with-rails > > We''ve also been using bleak_house: > http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html > yet still can''t nail down exactly whats going on here. > > Jodi, what version of Rails are you using for you project? > > Thanks, > DaveWe''re on 1.2.3, mysql 5, ubunutu 6.06 LTS, ruby 1.8.6 (i686). I''m deploying a new server (throw money at the problem), but will be getting to the leaks next week. We''ve had the leaks since day 1, and although we have 1.8million plus pages, controller actions are limited - so I should be able to localize the area quickly. I''ll update this thread with anything learned. thank you again Dave for the references. Jodi --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On Apr 1, 2:49 pm, Dave John <rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> wrote:> Greetings, > > I''ve recently deployed an app that is having some serious memory > consumption issue. Dropping mongrels quite frequently when the > threshold is met. > > Initially, we chalked this problem up to RMagick (I know, I know, but we > needed watermarking).If RMagick is not the source of leaks currently then this won''t help, but I just do the following: `convert -resize 480x360 #{temp_file_path} #{file_path}` My image uploading is low volume, so this works fine for me. I think it would work well even for much higher volumes, but I haven''t benchmarked the difference between a command line invocation and RMagick. It''s not likely to leak though. If you track down the problem please post it. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Greetings mem-leakers On 1-Apr-08, at 8:13 PM, Brian Adkins wrote:> > On Apr 1, 2:49 pm, Dave John <rails-mailing-l...-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org> wrote: >> Greetings, >> >> I''ve recently deployed an app that is having some serious memory >> consumption issue. Dropping mongrels quite frequently when the >> threshold is met. >> >> Initially, we chalked this problem up to RMagick (I know, I know, >> but we >> needed watermarking). > > If RMagick is not the source of leaks currently then this won''t help, > but I just do the following: > > `convert -resize 480x360 #{temp_file_path} #{file_path}` > > My image uploading is low volume, so this works fine for me. I think > it would work well even for much higher volumes, but I haven''t > benchmarked the difference between a command line invocation and > RMagick. It''s not likely to leak though. > > If you track down the problem please post it.I don''t have any answers yet, a few clues, but have some tools to suggest: re: clues. this article looks promising: http://blog.pluron.com/2008/01/ruby-on-rails-i.html As for tools, I''m trying out Evan Weaver''s bleak-house: http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html as it patches ruby, you''ll want to use this on your staging server (Evan says he uses the patched ruby in production) this is the output from Bleak: ./homestars/shared/log $ bleak /tmp/bleak.3242.0.dump 632837 total objects Final heap size 632837 filled, 732169 free Displaying top 20 most common line/class pairs 510101 __null__:__null__:__node__ 29450 __null__:__null__:String 5511 /usr/local/lib/ruby/gems/1.8/gems/memcached-0.8.1/lib/ memcached/memcached.rb:295:String 4692 (eval):1:__node__ 3442 /var/www/apps/homestars/releases/20080407140541/vendor/rails/ actionpack/lib/action_controller/routing.rb:836:String 3442 /var/www/apps/homestars/releases/20080407140541/vendor/rails/ actionpack/lib/action_controller/routing.rb: 836:ActionController::Routing::DividerSegment 2084 __null__:__null__:Regexp 1902 /usr/local/lib/ruby/gems/1.8/specifications/ tzinfo-0.3.7.gemspec:12:String 1686 generated_code_(/var/www/apps/homestars/releases/ 20080407140541/vendor/rails/actionpack/lib/action_controller/ routing.rb:388):1:__node__ 1491 /var/www/apps/homestars/releases/20080407140541/vendor/rails/ actionpack/lib/action_controller/routing.rb:834:String 1491 /var/www/apps/homestars/releases/20080407140541/vendor/rails/ actionpack/lib/action_controller/routing.rb: 834:ActionController::Routing::StaticSegment 1287 /var/www/apps/homestars/releases/20080407140541/vendor/rails/ activerecord/lib/../../activesupport/lib/active_support/deprecation.rb: 92:__node__ 1106 /var/www/apps/homestars/releases/20080407140541/vendor/rails/ actionpack/lib/action_controller/routing.rb: 827:ActionController::Routing::DynamicSegment 1056 /usr/local/lib/ruby/gems/1.8/specifications/ rmagick-2.2.2.gemspec:13:String 1051 /usr/local/lib/ruby/1.8/yaml.rb:133:String 976 /var/www/apps/homestars/releases/20080407140541/vendor/rails/ activerecord/lib/../../activesupport/lib/active_support/core_ext/ module/aliasing.rb:28:__node__ 938 /var/www/apps/homestars/releases/20080407140541/vendor/rails/ activerecord/lib/../../activesupport/lib/active_support/core_ext/ module/aliasing.rb:29:__node__ 891 __null__:__null__:Class 845 /var/www/apps/homestars/releases/20080407140541/vendor/rails/ actionpack/lib/action_controller/routing.rb: 945:ActionController::Routing::Route 845 /var/www/apps/homestars/releases/20080407140541/vendor/rails/ actionpack/lib/action_controller/routing.rb:865:Hash (the above is from a single run of hitting a single url for 15 minutes. working out a test approach presently) The above is after a 30 minute run - showing 5511 objects lost in memached (funny enough also written by Evan - I''ve asked him about the leak). The __ leaks are supposedly internal to ruby - boy, that''s a load of lost objects - other than tzinfo (which I''ve heard is a culprit), it seems to me that routing is generating a load. Anybody have any info on routing and leaks? I''m at a bit of a loss when digging info ruby leaks/garbage collection. If anybody has a good ref I''d appreciate it. rails 1.2.6 ruby 1.8.6 (2008-03-03 patchlevel 901) [i686-linux] thanx. Jodi --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Hey all, We''ll we tracked down our memory issues. We had some complex join queries going on and it appears that the AR objects created from said queries were not getting reclaimed. So if you are still having issues, and have some complex find_by_sql or find statements using a lot/nested includes. Try breaking those up into two queries.. Vast improvement here after doing that. Cheers, Dave -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Thanx Dave - On 7-Apr-08, at 4:04 PM, Dave John wrote:> > Hey all, > > We''ll we tracked down our memory issues. We had some complex join > queries going on and it appears that the AR objects created from said > queries were not getting reclaimed. > > So if you are still having issues, and have some complex find_by_sql > or > find statements using a lot/nested includes. Try breaking those up > into > two queries.. Vast improvement here after doing that. > > Cheers, > DaveI''m interested in how your determined they weren''t be reclaimed? ie. how did you zero in on the opportunity? Jodi --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---