we´ve upgraded puppet from 0.24.8 to 0.25.5 last week, since then we
´re experiencing load/memory (and then swapping) problems. Our setup
is based on nginx (0.8.20, four worker processes) balancing to five
puppetmaster instances (servertype mongrel, package:rubygem-
mongrel-1.0.1-6.el5), hardware is one quad-core with 12 GB RAM with
approx. 500 clients (puppet runs only once a day), OS is RHEL5.5 64
Bit.
this was fine until our upgrade, but now we´re running out of memory
every morning:
Mem: 12296768k total, 9891280k used, 2405488k free, 3784k
buffers
Swap: 2096472k total, 1908084k used, 188388k free, 95972k
cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13851 puppet 25 0 2526m 2.3g 992 R 95.0 19.3 15:38.54
puppetmasterd
14010 puppet 18 0 2555m 2.0g 112 D 14.3 17.4 13:40.84
puppetmasterd
13957 puppet 18 0 2907m 2.3g 112 D 0.7 19.2 13:51.26
puppetmasterd
13904 puppet 18 0 2929m 2.0g 112 D 0.3 17.0 12:29.75
puppetmasterd
1 root 15 0 10348 80 48 S 0.0 0.0 2:53.71 init
...
nginx active connection peak is ~ 200, memory usage with 0.24.8 was
stable with 2-4 GB.
we´ve not changed anything within our puppet configuration, are there
any topics within
moving 0.24.x to 0.25.x which need to be reconsidered?
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1> we´ve upgraded puppet from 0.24.8 to 0.25.5 last week, since then we > ´re experiencing load/memory (and then swapping) problems. Our setup > is based on nginx (0.8.20, four worker processes) balancing to five > puppetmaster instances (servertype mongrel, package:rubygem- > mongrel-1.0.1-6.el5), hardware is one quad-core with 12 GB RAM with > approx. 500 clients (puppet runs only once a day), OS is RHEL5.5 64 > Bit. > > this was fine until our upgrade, but now we´re running out of memory > every morning:this is odd.> nginx active connection peak is ~ 200, memory usage with 0.24.8 was > stable with 2-4 GB.I have 5 instances running using ~400MB of memory.> we´ve not changed anything within our puppet configuration, are there > any topics within > moving 0.24.x to 0.25.x which need to be reconsidered?not that I''m aware of. Actually the upgrade to 0.25.x even reduced memory for me so that I could fire up another mongrel instance ;) What does your setup look like? * are you using custom functions? might they leak? * are you using stored configs? (maybe updating the installed required gems might help) * are all your clients also on 0.25.x or are some still on 0.24.x? I''m aksing because then the xmlrpc api would still be used which might cause a problem * are you using default ruby? Which is 1.8.5 on RHEL 5.5, right? I got also some memory reduction by backporting latest fedora ruby to rhel5.5 (just take the SRPM and rebuild it for rhel5.5) * are you putting out a lot files with puppet? actually 0.25.x should have reduced that footprint heavily, but you never know... maybe you could push them out directly via nginx, see [1] for more. I''m really just guessing and giving you some points where you could look at. I think some of the points shouldn''t actually be a problem (like files) but you never know. I ordered them in the way that the most possible in my opinion are on top. You are imho the first person talking about memory problems after switching to 0.25.x but I don''t know how special your setup is etc. Can you pinpoint how and when (after 1 client, after 2, after X, after certain clients) your memory grows? I''m pretty sure that if you can elaborate the problem more and maybe get some first points where the problem might be, some puppet-devs might help you more. cheers pete [1] http://www.masterzen.fr/2010/03/21/more-puppet-offloading/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwEprYACgkQbwltcAfKi39DzwCeLOmsFOebKRnT0P/00T16pCjA CuQAnjVLvmvMyv2bIh6LkxE7A0OYiXH2 =EAbj -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
client and server run 0.25.5
I think our problem is the missing section [puppetmasterd] within our
puppet.conf on our puppet server.
I´ve added the section with all stuff (including storeconfigs = false)
and reloaded all puppetmaster instances.
but after that I´ll get the following error on my clients:
[root@clientXXX puppet]# clear; /usr/sbin/puppetd --test --verbose
info: Retrieving plugin
err: /File[/var/lib/puppet/lib]: Failed to retrieve current state of
resource: Could not retrieve information from source(s)
puppet://puppetB.example.com/plugins
.....
err: Could not retrieve catalog from remote server: Error 400 on
SERVER: Could not find class hostsfile at /etc/puppet/manifests/
site.pp:10 on node
seems that puppet tries to use plugindest, which defaults to libdir,
which is $vardir/lib (empty folder)
(the fileserver mount plugins is in our fileserver.conf). Next he
tries to load the first module within site.pp,
which fails. Somehow he cannot determine the modulepath = /var/lib/
puppet/modules ??!?
client puppet.conf
[main]
vardir = /var/lib/puppet
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl
factpath = $vardir/lib/facter/
factdest = $vardir/lib/facter/
modulepath = /var/lib/puppet/modules
pluginsync = true
#preferred_serialization_format = yaml
[puppetd]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
server = puppetB.example.com
runinterval = 1200
report = true
summarize = true
server: puppet.conf
[puppetmasterd]
reports = false
logdest=syslog
confdir = /etc/puppet
modulepath = /var/lib/puppet/modules
manifestdir = /etc/puppet/manifests
storeconfigs = false
rest_authconfig = /etc/puppet/auth.conf
yamldir = /var/lib/puppet/yaml
fileserverconfig = /etc/puppet/fileserver.conf
masterlog = /var/log/puppet/puppetmaster.log
libdir = /var/lib/puppet/lib
plugindest = /var/lib/puppet/modules
thin_storeconfigs = false
factsource = puppet://puppetB.example.com/facts/
certdir = /var/lib/puppet/ssl/certs
pluginsource = puppet://puppetB.example.com/plugins
#factpath = /var/lib/puppet/facts/
factsync = true
vardir = /var/lib/puppet
rundir = /var/run/puppet
ssldir = /var/lib/puppet/ssl
logdir = /var/log/puppet
templatedir = /var/lib/puppet/templates
certname = puppetB.example.com
certdnsname = puppetB.example.com
bind = 4.4.4.4
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
I´ve disabled pluginsync on my client, which eliminated the first error message. Now I´m still having problems while using modules: err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class hostsfile at /etc/puppet/manifests/ site.pp:10 on node xyz.example.com I´m only using the default-node in my site.pp, which includes all classes. any ideas why all modules (13) are not loaded? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
I´ve revamped my puppet.conf and updated to ruby 1.8.6/gem 1.3.7 and it seems that everything´s fine again ..... thanks for all input! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.