Puppet users - Nov 2008 - puppetmasterd taking a lot of memory

I installed the blastware packages and then used gary''s CSWpuppet
(0.24.4) package from http://www.garylaw.net/packages/

I''m seeing puppetmasterd using a lot of memory (850MB+).

It seems that ruby might have a memory leak from the forum posts I''m
reading.

Is there a known resolution to this or should a self compile specific
versions on my own?

Thanks,
Karl


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

I''ve noticed a significant drop in memory usage since disabling  
storeconfigs (and thus, rails ''n stuff). Previously I''d have
to
crontab restart the masters.

On 15/11/2008, at 9:51 AM, Loobster <karl.rossing@gmail.com> wrote:
>
> I installed the blastware packages and then used gary''s CSWpuppet
> (0.24.4) package from http://www.garylaw.net/packages/
>
> I''m seeing puppetmasterd using a lot of memory (850MB+).
>
> It seems that ruby might have a memory leak from the forum posts
I''m
> reading.
>
> Is there a known resolution to this or should a self compile specific
> versions on my own?
>
> Thanks,
> Karl
>
>
> >
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

2008/11/14 Loobster <karl.rossing@gmail.com>
>
> I installed the blastware packages and then used gary''s CSWpuppet
> (0.24.4) package from http://www.garylaw.net/packages/
>
> I''m seeing puppetmasterd using a lot of memory (850MB+).
>
> It seems that ruby might have a memory leak from the forum posts
I''m
> reading.
>
> Is there a known resolution to this or should a self compile specific
> versions on my own?
>
>
I would upgrade to the latest stable (0.24.6) to start with.  You might also
find an answer to your problem by either searching the mailing list or the
ticketing system - this issue has widely been reported though there has not
been a definitive solution.

http://groups.google.com/group/puppet-users/search?group=puppet-users&q=memory+leak&qt_g=Search+this+group

http://projects.reductivelabs.com/search/index/puppet?q=memory+leak

.r''

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

How many clients and how much file serving?

I''d love for someone with deep profiling experience to take a look at
this
but so far I haven''t found a leak so much as just a lot of memory
consumption and poor GC.

With the Ruby implementation of XMLRPC puppet uses now, file serving results
in 3x the file size in memory which gets garbage collected sometime between
now and the heat death of the universe.  So if you have a lot of file
serving and enough clients to make frequent requests the puppetmaster
process can balloon.

You can mitigate this by pulling the fileserving and the catalogs into
separate processes or any strategy that can minimize file serving (like
maybe packaging some of the configs so puppet:// isn''t the source.

Fileserving is essentially rewritten in the master branch and preliminary
test show that fixes a lot of this. (master branch is an unstable
development branch)

Regards,
Andrew




On Fri, Nov 14, 2008 at 2:31 PM, RijilV <rijilv@riji.lv> wrote:
> 2008/11/14 Loobster <karl.rossing@gmail.com>
>
>>
>> I installed the blastware packages and then used gary''s
CSWpuppet
>> (0.24.4) package from http://www.garylaw.net/packages/
>>
>> I''m seeing puppetmasterd using a lot of memory (850MB+).
>>
>> It seems that ruby might have a memory leak from the forum posts
I''m
>> reading.
>>
>> Is there a known resolution to this or should a self compile specific
>> versions on my own?
>>
>>
> I would upgrade to the latest stable (0.24.6) to start with.  You might
> also find an answer to your problem by either searching the mailing list or
> the ticketing system - this issue has widely been reported though there has
> not been a definitive solution.
>
>
>
http://groups.google.com/group/puppet-users/search?group=puppet-users&q=memory+leak&qt_g=Search+this+group
>
> http://projects.reductivelabs.com/search/index/puppet?q=memory+leak
>
> .r''
>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Nov 14, 6:53 pm, "Andrew Shafer" <and...@reductivelabs.com>
wrote:
> How many clients and how much file serving?
3 clients. The puppet server was a client.

puppetd was simply transfering firefox 3.0.4

    file { "/usr/local/firefox":
        source => "puppet://puppet/applications/firefox/
firefox-3.0.4.en-US.opensolaris-i386",
        recurse => "true"
    }

    file { "/usr/bin/firefox":
        ensure => "/usr/local/firefox/firefox.sh"
    }

I downloaded puppet 0.24.6, facter 1.5.2 and ruby 1.8.7-p72. It seemed
to use less memory but the usage was still rather high(500MB).

I''ll have to do some more testing next week.

I guess i could change
    source => "puppet://puppet/applications/firefox/firefox-3.0.4.en-
US.opensolaris-i386"
to
    source => "/export/applications/firefox/firefox-3.0.4.en-
US.opensolaris-i386",
and
    setup an rsync on the clients to copy /export/applications from
the rsync server.

Being new to puppet, is master safe to use and do people us puppet to
upgrade puppet?

Karl

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On 14/11/08 21:51, Loobster wrote:
> I installed the blastware packages and then used gary''s CSWpuppet
> (0.24.4) package from http://www.garylaw.net/packages/
> 
> I''m seeing puppetmasterd using a lot of memory (850MB+).
> 
> It seems that ruby might have a memory leak from the forum posts
I''m
> reading.
> 
> Is there a known resolution to this or should a self compile specific
> versions on my own?
I never could reproduce any memory leak (ruby or puppet) myself (linux, 
debian), but I practiced a little bit on how to find such issues, so
I''m
offering there a few ideas to explore:

Since you are on a DTrace enabled platform you can build a ruby 
interpreter containing DTrace probes[1] to check for memory 
allocations/deallocations (see the DTrace Toolkit[2], they have a script 
that almost does that). My own DTrace memory allocation checks were 
really not satisfying under Mac OSX (ruby crashes), but I think on 
Solaris you might have better luck than me.

Or you can use gdb live inspection as described in:
http://eigenclass.org/hiki.rb?ruby+live+process+introspection
(This works really fine)

Or you can also use a specially patched ruby called bleak house:
http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html


[1]: http://www.joyeur.com/2007/05/07/dtrace-for-ruby-is-available
[2]: http://opensolaris.org/os/community/dtrace/dtracetoolkit/

-- 
Brice Figureau
Days of Wonder
http://www.daysofwonder.com

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Aj wrote:
> I''ve noticed a significant drop in memory usage since disabling  
> storeconfigs (and thus, rails ''n stuff). Previously I''d
have to
> crontab restart the masters.
Upgrading activerecord and associated libs to version 2.1.1 fixed a
similar memory leak issues we had.

François

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Nov 14, 6:53 pm, "Andrew Shafer" <and...@reductivelabs.com>
wrote:
>
> Fileserving is essentially rewritten in the master branch and preliminary
> test show that fixes a lot of this. (master branch is an unstable
> development branch)
>
Is there a way of downloading the master as tar?

Karl
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

You can pull it from github, but I really don''t recommend you use it
for
anything but testing at the moment.

Of course, we''d be happy to hear about any issues that you find, I just
don''t want you finding them on anything important.

The head of trunk is Luke''s github
git://github.com/lak/puppet.git

Cheers,
Andrew







On Mon, Nov 17, 2008 at 2:17 PM, Loobster <karl.rossing@gmail.com> wrote:
>
> On Nov 14, 6:53 pm, "Andrew Shafer"
<and...@reductivelabs.com> wrote:
> >
> > Fileserving is essentially rewritten in the master branch and
preliminary
> > test show that fixes a lot of this. (master branch is an unstable
> > development branch)
> >
>
> Is there a way of downloading the master as tar?
>
> Karl
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi
>> I''ve noticed a significant drop in memory usage since
disabling
>> storeconfigs (and thus, rails ''n stuff). Previously
I''d have to
>> crontab restart the masters.
> 
> Upgrading activerecord and associated libs to version 2.1.1 fixed a
> similar memory leak issues we had.
François: thanks for mentioning that!

I have now installed activerecord and activesupport 2.1.2 by gem besides
the ones provided by rpm and I can see some significant memory drop:

https://durito.cronopios.org/puppetmaster_memory_activerecord.png

I would say it dropped nearly about a third of the memory usage. I can
confirm that from a second puppetmaster installation.

interesting. It seems that not only puppet needs memory fixes :-/

cheers pete



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi
> I have now installed activerecord and activesupport 2.1.2 by gem besides
> the ones provided by rpm and I can see some significant memory drop:
> 
> https://durito.cronopios.org/puppetmaster_memory_activerecord.png
> 
> I would say it dropped nearly about a third of the memory usage. I can
> confirm that from a second puppetmaster installation.
> 
> interesting. It seems that not only puppet needs memory fixes :-/
ohh and what I forgot to mention is that this is still with puppetmaster
running 0.24.5 and the gem versions before were 2.0.2

And what I see now, by looking at some more graphs is that since I
installed the new libs, that the amount of mysql throughput dropped to
the half of before and there seems to be really a lot more cache_hits of
mysql than before.

so even the mysql stuff is a lot better with the new libs.

greets pete

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---