Hi guys,
I must be missing something obvious...
array_of_stuff = [ contains a lot of fairly small AR objects ]
# RAM at 30MB
array_of_stuff.each do |foo|
foo.save
end
array_of_stuff = nil
# RAM at 60+MB
The iteration itself takes 30+MB of RAM (I measure with "memory_usage
from #{pcaller} at l.#{pline}: "+`ps -o rss= -p #{$$}`.to_i.to_s).
I don''t get it. Why is this using so much memory?
Thanks a lot,
Pierre
--
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.
PierreW wrote:> Hi guys, > > I must be missing something obvious... > > array_of_stuff = [ contains a lot of fairly small AR objects ] > > # RAM at 30MB > > array_of_stuff.each do |foo| > foo.save > end > array_of_stuff = nilThis is terrible! You should never have a DB query inside a loop. Instead, generate one query to insert all the records. The ar-extensions plugin can help with that.> > # RAM at 60+MB > > The iteration itself takes 30+MB of RAM (I measure with "memory_usage > from #{pcaller} at l.#{pline}: "+`ps -o rss= -p #{$$}`.to_i.to_s). > > I don''t get it. Why is this using so much memory? > > Thanks a lot, > PierreBest, -- Marnen Laibow-Koser http://www.marnen.org marnen-sbuyVjPbboAdnm+yROfE0A@public.gmane.org -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
I am sorry but I don''t have an answer about the RAM question. However I would like to answer Marnen’s comment. While I agree that letting the DB do the work for mass record processing should be the best and most efficient way to go by reading the OP one cannot assume that is the way things are in this case. The array used by the OP could very well contain tons of different types of objects, used for very different purposes and not necessarily related to each other. Pierre never gave any indication it was one way or the other. On Jan 30, 8:37 am, Marnen Laibow-Koser <li...-fsXkhYbjdPsEEoCn2XhGlw@public.gmane.org> wrote:> PierreW wrote: > > Hi guys, > > > I must be missing something obvious... > > > array_of_stuff = [ contains a lot of fairly small AR objects ] > > > # RAM at 30MB > > > array_of_stuff.each do |foo| > > foo.save > > end > > array_of_stuff = nil > > This is terrible! You should never have a DB query inside a loop. > Instead, generate one query to insert all the records. The > ar-extensions plugin can help with that. > > > > > # RAM at 60+MB > > > The iteration itself takes 30+MB of RAM (I measure with "memory_usage > > from #{pcaller} at l.#{pline}: "+`ps -o rss= -p #{$$}`.to_i.to_s). > > > I don''t get it. Why is this using so much memory? > > > Thanks a lot, > > Pierre > > Best, > -- > Marnen Laibow-Koserhttp://www.marnen.org > mar...-sbuyVjPbboAdnm+yROfE0A@public.gmane.org > -- > Posted viahttp://www.ruby-forum.com/.-- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
On Fri, Jan 29, 2010 at 8:10 PM, PierreW <wamrewam-gM/Ye1E23mwN+BqQ9rBEUg@public.gmane.org> wrote:> Hi guys, > > I must be missing something obvious... > > array_of_stuff = [ contains a lot of fairly small AR objects ] > > # RAM at 30MB > > array_of_stuff.each do |foo| > foo.save > end > array_of_stuff = nil > > # RAM at 60+MB > > The iteration itself takes 30+MB of RAM (I measure with "memory_usage > from #{pcaller} at l.#{pline}: "+`ps -o rss= -p #{$$}`.to_i.to_s). > > I don''t get it. Why is this using so much memory? > > Thanks a lot, > Pierre > >Pierre, you might be interested in the following thread within this mailing list because it covered a similar topic: "activerecord 2.3.5''s create & find slower than activerecord 2.1.2" Good luck, -Conrad> -- > You received this message because you are subscribed to the Google Groups > "Ruby on Rails: Talk" group. > To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org > To unsubscribe from this group, send email to > rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org<rubyonrails-talk%2Bunsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org> > . > For more options, visit this group at > http://groups.google.com/group/rubyonrails-talk?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
pepe wrote:> I am sorry but I don''t have an answer about the RAM question. However > I would like to answer Marnen�s comment. While I agree that letting > the DB do the work for mass record processing should be the best and > most efficient way to go by reading the OP one cannot assume that is > the way things are in this case.No, but it''s likely.> The array used by the OP could very > well contain tons of different types of objects, used for very > different purposes and not necessarily related to each other.Then for the purpose of saving, they should be separated out by type. Queries don''t go in loops. Period.> Pierre > never gave any indication it was one way or the other.True. So why "correct" me with an unlikely exception to the general principle? Best, -- Marnen Laibow-Koser http://www.marnen.org marnen-sbuyVjPbboAdnm+yROfE0A@public.gmane.org -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
On Feb 1, 7:53 am, Marnen Laibow-Koser <li...-fsXkhYbjdPsEEoCn2XhGlw@public.gmane.org> wrote:> pepe wrote: > > I am sorry but I don''t have an answer about the RAM question. However > > I would like to answer Marnen s comment. While I agree that letting > > the DB do the work for mass record processing should be the best and > > most efficient way to go by reading the OP one cannot assume that is > > the way things are in this case. > > No, but it''s likely.Anything is likely.> > > The array used by the OP could very > > well contain tons of different types of objects, used for very > > different purposes and not necessarily related to each other. > > Then for the purpose of saving, they should be separated out by type. > Queries don''t go in loops. Period.Says who? The point I was making is that it would depend on the situation and the solution the OP is trying to give to his particular problem.> > > Pierre > > never gave any indication it was one way or the other. > > True. So why "correct" me with an unlikely exception to the general > principle?I wasn''t trying to "correct" you. I was trying to offer a different point of view and have an open mind. Cheers.> > Best, > -- > Marnen Laibow-Koserhttp://www.marnen.org > mar...-sbuyVjPbboAdnm+yROfE0A@public.gmane.org > -- > Posted viahttp://www.ruby-forum.com/.-- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.
pepe wrote:> On Feb 1, 7:53�am, Marnen Laibow-Koser <li...-fsXkhYbjdPsEEoCn2XhGlw@public.gmane.org> wrote: >> pepe wrote: >> > I am sorry but I don''t have an answer about the RAM question. However >> > I would like to answer Marnen s comment. While I agree that letting >> > the DB do the work for mass record processing should be the best and >> > most efficient way to go by reading the OP one cannot assume that is >> > the way things are in this case. >> >> No, but it''s likely. > > Anything is likely.Anything is *possible*...but it''s...er...unusual to be saving an array of unrelated objects.> >> >> > The array used by the OP could very >> > well contain tons of different types of objects, used for very >> > different purposes and not necessarily related to each other. >> >> Then for the purpose of saving, they should be separated out by type. >> Queries don''t go in loops. �Period. > > Says who?Anyone who understands how to use databases efficiently end effectively.> The point I was making is that it would depend on the > situation and the solution the OP is trying to give to his particular > problem. > >> >> > Pierre >> > never gave any indication it was one way or the other. >> >> True. �So why "correct" me with an unlikely exception to the general >> principle? > > I wasn''t trying to "correct" you.Yes, I realized that after I posted.> I was trying to offer a different > point of view and have an open mind.OK, but in this case I don''t think it was relevant.> > Cheers.Best, -- Marnen Laibow-Koser http://www.marnen.org marnen-sbuyVjPbboAdnm+yROfE0A@public.gmane.org -- Posted via http://www.ruby-forum.com/. -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org To unsubscribe from this group, send email to rubyonrails-talk+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/rubyonrails-talk?hl=en.