Rick Bradley forwarded me this:
http://groups.google.com/group/god-rb/browse_thread/thread/1cca2b7c4a581c2
This idiom leaks memory pretty quickly in Ruby:
def whatever
value.split(/regex)
end
That is, splitting without assigning to any local variables. If you
assign any local variables at all -- even a no-op ''x = nil'',
or just
''x = value.split...'' -- then you get no leak. Go ahead, test
it in
your ruby version.
Here was my test (done with instance methods, since I wanted to see if
it only leaked with class methods):
class Bar
def class_name
name.split(/::/).collect { |i| r = i*2 }
end
attr_reader :name
def initialize(name)
@name = name
end
end
bar = Bar.new(''yay::boo'')
loop { bar.class_name }
I''ve found at least 10 or so instances of this idiom in Puppet, and
I''ve confirmed that local variables in an attached block also leaks:
def whatever
whatever.split(/::/) do |val| a = val end
end
This still leaks.
I''m going to walk through the code and see if I can fix all of the
places where I use this idiom, and comment them appropriately. I''m
assuming this is already filed with the Ruby guys, but if someone can
verify that, it''d be great.
--
A complex system that works is invariably found to have evolved from a
simple system that works. -- John Gaule
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
Well played.
Myself and Adam (holoway) are currently GDBing my leaky puppetmaster to try and
find exactly what arrays and where are leaking - he has more GDB-fu than me, and
the ruby-live-process introspection stuff is pretty magical.
Here''s hoping we can track this one down!
Arjuna Christensen | Systems Engineer
Maximum Internet Ltd
DDI: + 64 9 913 9683 | Ph: +64 9 915 1825 | Fax:: +64 9 300 7227
arjuna.christensen@maxnet.co.nz| www.maxnet.co.nz
-----Original Message-----
From: puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] On
Behalf Of Luke Kanies
Sent: Tuesday, 18 March 2008 12:33 p.m.
To: puppet-users@googlegroups.com
Subject: [Puppet Users] Likely source of memory leak
Rick Bradley forwarded me this:
http://groups.google.com/group/god-rb/browse_thread/thread/1cca2b7c4a581c2
This idiom leaks memory pretty quickly in Ruby:
def whatever
value.split(/regex)
end
That is, splitting without assigning to any local variables. If you
assign any local variables at all -- even a no-op ''x = nil'',
or just
''x = value.split...'' -- then you get no leak. Go ahead, test
it in
your ruby version.
Here was my test (done with instance methods, since I wanted to see if
it only leaked with class methods):
class Bar
def class_name
name.split(/::/).collect { |i| r = i*2 }
end
attr_reader :name
def initialize(name)
@name = name
end
end
bar = Bar.new(''yay::boo'')
loop { bar.class_name }
I''ve found at least 10 or so instances of this idiom in Puppet, and
I''ve confirmed that local variables in an attached block also leaks:
def whatever
whatever.split(/::/) do |val| a = val end
end
This still leaks.
I''m going to walk through the code and see if I can fix all of the
places where I use this idiom, and comment them appropriately. I''m
assuming this is already filed with the Ruby guys, but if someone can
verify that, it''d be great.
--
A complex system that works is invariably found to have evolved from a
simple system that works. -- John Gaule
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
Note that (as mentioned in the ticket) I uploaded a branch named memory_leak_split_fix in my personal repository in which I did my best to implement the fix from this thread. If someone who can reproduce this problem can test this branch out, I''d appreciate it. On Mar 17, 2008, at 8:33 PM, Arjuna Christensen wrote:> > Well played. > > Myself and Adam (holoway) are currently GDBing my leaky puppetmaster > to try and find exactly what arrays and where are leaking - he has > more GDB-fu than me, and the ruby-live-process introspection stuff > is pretty magical. > > Here''s hoping we can track this one down! > > Arjuna Christensen | Systems Engineer > Maximum Internet Ltd > DDI: + 64 9 913 9683 | Ph: +64 9 915 1825 | Fax:: +64 9 300 7227 > arjuna.christensen@maxnet.co.nz| www.maxnet.co.nz > > > -----Original Message----- > From: puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com > ] On Behalf Of Luke Kanies > Sent: Tuesday, 18 March 2008 12:33 p.m. > To: puppet-users@googlegroups.com > Subject: [Puppet Users] Likely source of memory leak > > > Rick Bradley forwarded me this: > > http://groups.google.com/group/god-rb/browse_thread/thread/1cca2b7c4a581c2 > > This idiom leaks memory pretty quickly in Ruby: > > def whatever > value.split(/regex) > end > > That is, splitting without assigning to any local variables. If you > assign any local variables at all -- even a no-op ''x = nil'', or just > ''x = value.split...'' -- then you get no leak. Go ahead, test it in > your ruby version. > > Here was my test (done with instance methods, since I wanted to see if > it only leaked with class methods): > > class Bar > def class_name > name.split(/::/).collect { |i| r = i*2 } > end > > attr_reader :name > def initialize(name) > @name = name > end > end > > bar = Bar.new(''yay::boo'') > > loop { bar.class_name } > > I''ve found at least 10 or so instances of this idiom in Puppet, and > I''ve confirmed that local variables in an attached block also leaks: > > def whatever > whatever.split(/::/) do |val| a = val end > end > > This still leaks. > > I''m going to walk through the code and see if I can fix all of the > places where I use this idiom, and comment them appropriately. I''m > assuming this is already filed with the Ruby guys, but if someone can > verify that, it''d be great. > > -- > A complex system that works is invariably found to have evolved from a > simple system that works. -- John Gaule > --------------------------------------------------------------------- > Luke Kanies | http://reductivelabs.com | http://madstop.com > > > > > >-- When one admits that nothing is certain one must, I think, also admit that some things are much more nearly certain than others. -- Bertrand Russell --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Mon, Mar 17, 2008 at 9:48 PM, Luke Kanies <luke@madstop.com> wrote:>... > > Myself and Adam (holoway) are currently GDBing my leaky puppetmaster > > to try and find exactly what arrays and where are leaking - he has > > more GDB-fu than me, and the ruby-live-process introspection stuff > > is pretty magical. > > > > Here''s hoping we can track this one down! > >There is also some work being done by dtrace people: http://joyeur.com/2007/05/07/dtrace-for-ruby-is-available, which may prove fruitful. I''m sure they would welcome additional tests & input. Steven --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Mar 18, 2008, at 8:31 AM, Steven Jenkins wrote:> There is also some work being done by dtrace people: > http://joyeur.com/2007/05/07/dtrace-for-ruby-is-available, which may > prove fruitful. I''m sure they would welcome additional tests & input.I''ve messed around with dtrace and ruby some already; it might be possible to figure it out with dtrace, but it''s so darn unintuitive... I might give it a try if it can''t be figured out other ways, but it seems like Adam''s making great progress with gdb. -- Never regret. If it''s good, it''s wonderful. If it''s bad, it''s experience. -- Victoria Holt --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Tue, Mar 18, 2008 at 11:25 AM, Luke Kanies <luke@madstop.com> wrote:> > On Mar 18, 2008, at 8:31 AM, Steven Jenkins wrote: > > > There is also some work being done by dtrace people: > > http://joyeur.com/2007/05/07/dtrace-for-ruby-is-available, which may > > prove fruitful. I''m sure they would welcome additional tests & input. > > > I''ve messed around with dtrace and ruby some already; it might be > possible to figure it out with dtrace, but it''s so darn unintuitive... > > I might give it a try if it can''t be figured out other ways, but it > seems like Adam''s making great progress with gdb. >Apparently, none of our clients have been bitten by the ruby memory leak issues, otherwise I could get someone to spend some time looking at it. Steven --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On 3/18/08, Steven Jenkins <steven.jenkins@gmail.com> wrote:> Apparently, none of our clients have been bitten by the ruby memory > leak issues, otherwise I could get someone to spend some time looking > at it.I would be kind of surprised if everyones puppetmasterd wasn''t leaking over time. There is plenty of evidence, both from what Luke talked about earlier, and what I saw with gdb. We''re trying to track down whether the leaking (bug #1131) is really related to #1010 (the file resource corruption.) Hopefully we''ll have more on the topic tonight.. Adam -- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: adam@hjksolutions.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---