I''ve got an app that is eating up lots of memory. When it first starts up, it uses ~125 megs for a mongrel process, which isn''t terrible I suppose. However after being used for a while it balloons to 250-350 megs. It would appear there''s a big memory leak somewhere, but I have no clue where to even begin looking. Any ideas? Pat --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On 7/5/07, Pat Maddox <pergesu-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> > I''ve got an app that is eating up lots of memory. When it first > starts up, it uses ~125 megs for a mongrel process, which isn''t > terrible I suppose.That''s huge, are you sure that''s ram and not virtual memory? However after being used for a while it balloons> to 250-350 megs. It would appear there''s a big memory leak somewhere, > but I have no clue where to even begin looking. Any ideas? >Pin down when it started and then find out what changed at that time. That''s where I would start. Normally I have at least an idea of where something like that is coming from. Do you have large data sets you are processing in some way, or retrieving through activerecord? Chris --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Pat Maddox wrote:> I''ve got an app that is eating up lots of memory. When it first > starts up, it uses ~125 megs for a mongrel process, which isn''t > terrible I suppose. However after being used for a while it balloons > to 250-350 megs. It would appear there''s a big memory leak somewhere, > but I have no clue where to even begin looking. Any ideas? > > PatDo you use RMagick? If so, try using GC.start to start Garbage Collection regulary. Just to rule it out... Greetings Skade -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On 7/5/07, snacktime <snacktime-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> > On 7/5/07, Pat Maddox <pergesu-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > > I''ve got an app that is eating up lots of memory. When it first > > starts up, it uses ~125 megs for a mongrel process, which isn''t > > terrible I suppose. > > That''s huge, are you sure that''s ram and not virtual memory?I''m checking it out on my MBP, watching the process with the Activity Monitor tool When I first start it up, it uses 32 megs of Real Memory, 29 of Private, and 77 of Virtual.> However after being used for a while it balloons > > to 250-350 megs. It would appear there''s a big memory leak somewhere, > > but I have no clue where to even begin looking. Any ideas? > > > > Pin down when it started and then find out what changed at that time. > That''s where I would start. Normally I have at least an idea of where > something like that is coming from. Do you have large data sets you > are processing in some way, or retrieving through activerecord?Here''s one line where I found something weird: Company.top(10).collect {|c| c.complete_videos.size} class Company < ActiveRecord::Base def self.top(n) find_by_sql ["SELECT companies.*, COUNT(videos.company_id) AS num_videos FROM companies LEFT JOIN videos ON companies.id=videos.company_id WHERE videos.status=''complete'' AND videos.deleted_at IS NULL GROUP BY companies.id, companies.name, companies.nickname, companies.plan_id, companies.deleted_at ORDER BY num_videos DESC LIMIT ?", n] end def complete_videos(options = {}) @complete_videos ||= Video.find :all, { :conditions => "company_id=#{id} AND status=''complete''" }.merge(options) end end When I run that line in console, I get the proper results, and there''s no memory spike. One thing that''s really weird though is that if I run 2.times { Company.top(10).collect {|c| c.complete_videos.size} } then it jumps to 48/45/93 and never comes back down. My only guess is that the companies and videos are kept in a closure and not released. In the rails app itself, the offending code is: <% @top_companies.each do |c| %> <tr><td><%= link_to c.name, company_url(c) %></td><td><%c.complete_videos.size %></td></tr> <% end %> I''m not doing anything like 2.times.. there. One thing I have checked out is http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling-with-rails When I collect the top company''s video sizes, it increases the number of Video objects by 5689, but I never see it come back down. I just noticed that if I run Company.find(2).complete_videos.size twice in quick succession, the memory jumps up to the high 49/46/94. So maybe it has to do with how quickly a connection is being made to the database? I''m using Rails 1.2.3, Ruby 1.8.6, and PostgreSQL 8.2.3. Pat --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On 7/6/07, Pat Maddox <pergesu-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> One thing I have checked out is > http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling-with-rails > When I collect the top company''s video sizes, it increases the > number of Video objects by 5689, but I never see it come back down. > > I just noticed that if I run > Company.find(2).complete_videos.size > > twice in quick succession, the memory jumps up to the high 49/46/94. > > So maybe it has to do with how quickly a connection is being made to > the database?I want to elaborate on this a tad. When I run Company.find(2).complete_videos.size, the MemoryProfiler class reports 3744 new Video objects (which is what complete_videos.size returns). I never see the number be reduced. However, the memory usage doesn''t change at all. When I run 2.times { Company.find(2).complete_videos.size }, it reports another 3744 Videos for a total of 7488. Memory usage increases slightly to 35/32/81. When I run 2.times { Company.find(2).complete_videos.size } one more time, there''s no change in Video objects, but the usage jumps up to 50/47/95.> > I''m using Rails 1.2.3, Ruby 1.8.6, and PostgreSQL 8.2.3. > > Pat >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On 7/6/07, Pat Maddox <pergesu-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> So maybe it has to do with how quickly a connection is being made to > the database? > > I''m using Rails 1.2.3, Ruby 1.8.6, and PostgreSQL 8.2.3.I just dumped all the data into mysql and tried it, same issue. So it''s not a mysql vs postgresql thing. Pat --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On 7/5/07, snacktime <snacktime-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> Do you have large data sets you are processing in some way, or > retrieving through activerecord?You specifically asked about this, so perhaps you know something I don''t...but anyway, some of the data sets are in the 10k range, and yes they''re coming through activerecord. I don''t understand why the memory isn''t being freed up though. Am I just misunderstanding how Ruby''s memory allocation works? If a mongrel needs 300 megs, then it''ll hold onto that forever? In that case do I just need to allow very high memory limits? That doesn''t seem right to me... Pat --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On Jul 6, 2007, at 2:49 PM, Pat Maddox wrote:> > On 7/5/07, snacktime <snacktime-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> Do you have large data sets you are processing in some way, or >> retrieving through activerecord? > > You specifically asked about this, so perhaps you know something I > don''t...but anyway, some of the data sets are in the 10k range, and > yes they''re coming through activerecord. > > I don''t understand why the memory isn''t being freed up though. > > Am I just misunderstanding how Ruby''s memory allocation works? If a > mongrel needs 300 megs, then it''ll hold onto that forever? In that > case do I just need to allow very high memory limits? That doesn''t > seem right to me... > > PatYou are going to see massive memory usage like this anytime you load more then 500 -1000 activerecord objects at once. There is quite a bit of memory overhead per AR object instantiation so ruby allocates a bunch or ram to hold the records. Once the dataset is no longer needed ruby doesn''t like to give ram back to the OS, instead it tends to keep the memory around for use by the interpreter later. Do you actually need 10k AR objects at once? You are going to need a lot of ram to handle result sets like that. If there is any way to do more calculation in the dn itself and pull smaller sets at a time you wil keep your memory usage lower. But 10k AR''s is going to eat the ram hard. Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez-NLltGlunAUd/unjJdyJNww@public.gmane.org -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On 7/6/07, Ezra Zygmuntowicz <ezmobius-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> > > On Jul 6, 2007, at 2:49 PM, Pat Maddox wrote: > > > > > On 7/5/07, snacktime <snacktime-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > >> Do you have large data sets you are processing in some way, or > >> retrieving through activerecord? > > > > You specifically asked about this, so perhaps you know something I > > don''t...but anyway, some of the data sets are in the 10k range, and > > yes they''re coming through activerecord. > > > > I don''t understand why the memory isn''t being freed up though. > > > > Am I just misunderstanding how Ruby''s memory allocation works? If a > > mongrel needs 300 megs, then it''ll hold onto that forever? In that > > case do I just need to allow very high memory limits? That doesn''t > > seem right to me... > > > > Pat > > > You are going to see massive memory usage like this anytime you load > more then 500 -1000 activerecord objects at once. There is quite a > bit of memory overhead per AR object instantiation so ruby allocates > a bunch or ram to hold the records. Once the dataset is no longer > needed ruby doesn''t like to give ram back to the OS, instead it tends > to keep the memory around for use by the interpreter later.Okay, that''s what I suspected. I just wasn''t sure about that.> Do you actually need 10k AR objects at once? You are going to need a > lot of ram to handle result sets like that. If there is any way to do > more calculation in the dn itself and pull smaller sets at a time you > wil keep your memory usage lower. But 10k AR''s is going to eat the > ram hard.You''re right, we definitely don''t. There''s plenty of nasty code that I need to get cleaned up :) Thanks for the explanation. Pat --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On Jul 6, 2007, at 3:04 PM, Pat Maddox wrote:>> >> You are going to see massive memory usage like this >> anytime you load >> more then 500 -1000 activerecord objects at once. There is quite a >> bit of memory overhead per AR object instantiation so ruby allocates >> a bunch or ram to hold the records. Once the dataset is no longer >> needed ruby doesn''t like to give ram back to the OS, instead it tends >> to keep the memory around for use by the interpreter later. > > Okay, that''s what I suspected. I just wasn''t sure about that. > > >> Do you actually need 10k AR objects at once? You are going >> to need a >> lot of ram to handle result sets like that. If there is any way to do >> more calculation in the dn itself and pull smaller sets at a time you >> wil keep your memory usage lower. But 10k AR''s is going to eat the >> ram hard. > > You''re right, we definitely don''t. There''s plenty of nasty code that > I need to get cleaned up :) > > Thanks for the explanation.You may find this explains the GC a bit more: http://whytheluckystiff.net/articles/theFullyUpturnedBin.html Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez-NLltGlunAUd/unjJdyJNww@public.gmane.org -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
On 7/6/07, Ezra Zygmuntowicz <ezmobius-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:> > > On Jul 6, 2007, at 3:04 PM, Pat Maddox wrote: > > >> > >> You are going to see massive memory usage like this > >> anytime you load > >> more then 500 -1000 activerecord objects at once. There is quite a > >> bit of memory overhead per AR object instantiation so ruby allocates > >> a bunch or ram to hold the records. Once the dataset is no longer > >> needed ruby doesn''t like to give ram back to the OS, instead it tends > >> to keep the memory around for use by the interpreter later. > > > > Okay, that''s what I suspected. I just wasn''t sure about that. > > > > > >> Do you actually need 10k AR objects at once? You are going > >> to need a > >> lot of ram to handle result sets like that. If there is any way to do > >> more calculation in the dn itself and pull smaller sets at a time you > >> wil keep your memory usage lower. But 10k AR''s is going to eat the > >> ram hard. > > > > You''re right, we definitely don''t. There''s plenty of nasty code that > > I need to get cleaned up :) > > > > Thanks for the explanation. > > > You may find this explains the GC a bit more: > > http://whytheluckystiff.net/articles/theFullyUpturnedBin.htmlThanks. One thing I don''t see it addressed is what happens to the memory when the GC runs. My current understanding is that it''s just cleared out so that Ruby can store later objects in it, rather than being returned to the OS. So if I have a big query that bumps the usage up to 250 megs, after GC runs the ruby process may only need 125, but it''s still holding onto the full 250. Is that accurate? Pat --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---
Pat Maddox wrote:> On 7/6/07, Ezra Zygmuntowicz <ezmobius-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> >> needed ruby doesn''t like to give ram back to the OS, instead it tends >> >> ram hard. >> > >> > You''re right, we definitely don''t. There''s plenty of nasty code that >> > I need to get cleaned up :) >> > >> > Thanks for the explanation. >> >> >> You may find this explains the GC a bit more: >> >> http://whytheluckystiff.net/articles/theFullyUpturnedBin.html > > Thanks. One thing I don''t see it addressed is what happens to the > memory when the GC runs. My current understanding is that it''s just > cleared out so that Ruby can store later objects in it, rather than > being returned to the OS. So if I have a big query that bumps the > usage up to 250 megs, after GC runs the ruby process may only need > 125, but it''s still holding onto the full 250. Is that accurate? > > PatPat, I''m exepriencing the same problem in my RAILS application. I seldomly need to retrieve about 10K records, and my applications bumps to 400MB... and never releases the memory again. Did you find a way to solve this problem, or I have to rewrite my code? Thanks Massimo -- Posted via http://www.ruby-forum.com/. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en -~----------~----~----~----~------~----~------~--~---