Yes, thanks. I''ll let you know what the results are.
On a similar note, I''ve realised that my puppetmasterd tends to send
corrupt resources when memory usage is high:
Jan 17 21:13:04 vps-harveys puppetd[8977]:
(//Node[vps-harveys]/monitoring/File[/etc/nagios/nrpe.cfg]) Filebucketed to main
with sum %
23%23%23+WARNING+THIS+FILE+IS+CONTROLLED+BY+PUPPET%0A%23%23%23+ANY+CHANGES+MADE+NOT+VIA+PUPPET+WILL+BE+OVERWRITTEN%0A%0A%23%23%23%23%2
3%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%
23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%0A%23+Sample+NRPE+Config+File+%0A%23+Written+by%3A
+Ethan+Galstad+%28nagios%40nagios.org%29%0A%23+%0A%23+Last+Modified%3A+02-23-2006%0A%23%0A%23+NOTES%3A%0A%23+This+is+a+sample+configur
ation+file+for+the+NRPE+daemon.++It+needs+to+be%0A%23+located+on+the+remote+host+that+is+running+the+NRPE+daemon%2C+not+the+host%0A%23
+from+which+the+check_nrpe+client+is+being+executed.%0A%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%
23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23%23
%23%23%23%23%23%23%0A%0A%0A%23+PID+FILE%0A%23+The+name+of+the+file+in+which+the+NRPE+daemon+should+write+it%27s+process+ID%0A%23+numbe
r.++The+file+is+only+written+if+the+NRPE+daemon+is+started+by+the+root%0A%23+user+and+is+running+in+standalone+mode.%0A%0Apid_file%3D%
2Fvar%2Frun%2Fnrpe.pid%0A%0A%0A%0A%23+PORT+NUMBER%0A%23+Port+number+we+should+wait+for+connections+on.%0A%23+NOTE%3A+This+must+be+a+no
n-priviledged+port+%28i.e.+%3E+1024%29.%0A%23+NOTE%3A+This+option+is+ignored+if+NRPE+is+running+under+either+inetd+or+xinetd%0A%0Aserv
er_port%3D5666%0A%0A%0A%0A%23+SERVER+ADDRESS%0A%23+Address+that+nrpe+should+bind+to+in+case+there+are+more+than+one+interface%0A%23+an
d+you+do+not+want+nrpe+to+bind+on+all+interfaces.%0A%23+NOTE%3A+This+option+is+ignored+if+NRPE+is+running+under+either+inetd+or+xinetd
%0A%0A%23server_address%3D192.168.1.1%0A%0A%0A%0A%23+NRPE+USER%0A%23+This+determines+the+effective+user+that+the+NRPE+daemon+should+ru
n+as.++%0A%23+You+can+either+supply+a+username+or+a+UID.%0A%23+%0A%23+NOTE%3A+This+option+is+ignored+if+NRPE+is+running+under+either+i
netd+or+xinetd%0A%0Anrpe_user%3Dnagios%0A%0A%0A%0A%23+NRPE+GROUP%0A%23+This+determines+the+effective+group+that+the+NRPE+daemon+should
+run+as.++%0A%23+You+can+either+supply+a+group+name+or+a+GID.%0A%23+%0A%23+NOTE%3A+This+option+is+ignored+if+NRPE+is+running+under+eit
her+inetd+or+xinetd%0A%0Anrpe_group%3Dnagios%0A%0A%0A%0A%23+ALLOWED+HOST+ADDRESSES%0A%23+This+is+an+optional+comma-delimited+list+of+I
P+address+or+hostnames+%0A%23+that+are+allowed+to+talk+to+the+NRPE+daemon.%0A%23%0A%23+Note%3A+The+daemon+only+does+rudimentary+checki
ng+of+the+client%27s+IP%0A%23+address.++I+would+highly+recommend+adding+entries+in+your+%2Fetc%2Fhosts.allow%0A%23+file+to+allow+only+
the+specified+host+to+connect+to+the+port%0A%23+you+are+running+this+daemon+on.%0A%23%0A%23+NOTE%3A+This+option+is+ignored+if+NRPE+is+
running+under+either+inetd+or+xinetd%0A%0A%23allowed_hosts%3D127.0.0.1%2C192.168.0.2%0A+%0A%0A%0A%23+COMMAND+ARGUMENT+PROCESSING%0A%23
+This+option+determines+whether+or+not+the+NRPE+daemon+will+allow+clients%0A%23+to+specify+arguments+to+commands+that+are+executed.++T
his+option+only+works%0A%23+if+the+daemon+was+configured+with+the+--enable-command-args+configure+script%0A%23+option.++%0A%23%0A%23+%
2A%2A%2A+ENABLING+THIS+OPTION+IS+A+SECURITY+RISK%21+%2A%2A%2A+%0A%23+Read+the+SECURITY+file+for+information+on+some+of+the+security+im
plications%0A%23+of+enabling+this+variable.%0A%23%0A%23+Values%3A+0%3Ddo+not+allow+arguments%2C+1%3Dallow+command+arguments%0A%0Adont_
blame_nrpe%3D1%0A%0A%0A%0A%23+COMMAND+PREFIX%0A%23+This+option+allows+you+to+prefix+all+commands+with+a+user-defined+string.%0A%23+A+s
pace+is+automatically+added+between+the+specified+prefix+string+and+the%0A%23+command+line+from+the+command+definition.%0A%23%0A%23+%2
A%2A%2A+THIS+EXAMPLE+MAY+POSE+A+POTENTIAL+SECURITY+RISK%2C+SO+USE+WITH+CAUTION%21+%2A%2A%2A%0A%23+Usage+scenario%3A+%0A%23+Execute+res
tricted+commmands+using+sudo.++For+this+to+work%2C+you+need+to+add%0A%23+the+nagios+user+to+your+%2Fetc%2Fsudoers.++An+example+entry+f
or+alllowing+%0A%23+execution+of+the+plugins+from+might+be%3A%0A%23%0A%23+nagios++++++++++ALL%3D%28ALL%29+NOPASSWD%3A+%2Fusr%2Flib%2Fn
agios%2Fplugins%2F%0A%23%0A%23+This+lets+the+nagios+user+run+all+commands+in+that+directory+%28and+only+them%29%0A%23+without+asking+f
or+a+password.++If+you+do+this%2C+make+sure+you+don%27t+give%0A%23+random+users+write+access+to+that+directory+or+its+contents%21%0A%0
A%23+command_prefix%3D%2Fusr%2Fbin%2Fsudo+%0A%0A%0A%0A%23+DEBUGGING+OPTION%0A%23+This+option+determines+whether+or+not+debugging+messa
ges+are+logged+to+the%0A%23+syslog+facility.%0A%23+Values%3A+0%3Ddebugging+off%2C+1%3Ddebugging+on%0A%0Adebug%3D0%0A%0A%0A%0A%23+COMMA
ND+TIMEOUT%0A%23+This+specifies+the+maximum+number+of+seconds+that+the+NRPE+daemon+will%0A%23+allow+plugins+to+finish+executing+before
+killing+them+off.%0A%0Acommand_timeout%3D60%0A%0A%0A%0A%23+WEEK+RANDOM+SEED+OPTION%0A%23+This+directive+allows+you+to+use+SSL+even+if
+your+system+does+not+have%0A%23+a+%2Fdev%2Frandom+or+%2Fdev%2Furandom+%28on+purpose+or+because+the+necessary+patches%0A%23+were+not+a
pplied%29.+The+random+number+generator+will+be+seeded+from+a+file%0A%23+which+is+either+a+file+pointed+to+by+the+environment+valiable+
%24RANDFILE%0A%23+or+%24HOME%2F.rnd.+If+neither+exists%2C+the+pseudo+random+number+generator+will%0A%23+be+initialized+and+a+warning+w
ill+be+issued.%0A%23+Values%3A+0%3Donly+seed+from+%2Fdev%2F%5Bu%5Drandom%2C+1%3Dalso+seed+from+weak+randomness%0A%0A%23allow_weak_rand
om_seed%3D1%0A%0A%0A%0A%23+INCLUDE+CONFIG+FILE%0A%23+This+directive+allows+you+to+include+definitions+from+an+external+config+file.%0A
%0A%23include%3D%3Csomefile.cfg%3E%0A%0A%0A%0A%23+INCLUDE+CONFIG+DIRECTORY%0A%23+This+directive+allows+you+to+include+definitions+from
+config+files+%28with+a%0A%23+.cfg+extension%29+in+one+or+more+directories+%28with+recursion%29.%0A%0A%23include_dir%3D%3Csomedirector
y%3E%0A%23include_dir%3D%3Csomeotherdirectory%3E%0A%0A%0A%0A%23+COMMAND+DEFINITIONS%0A%23+Command+definitions+that+this+daemon+will+ru
n.++Definitions%0A%23+are+in+the+following+format%3A%0A%23%0A%23+command%5B%3Ccommand_name%3E%5D%3D%3Ccommand_line%3E%0A%23%0A%23+When
+the+daemon+receives+a+request+to+return+the+results+of+%3Ccommand_name%3E%0A%23+it+will+execute+the+command+specified+by+the+%3Ccomma
nd_line%3E+argument.%0A%23%0A%23+Unlike+Nagios%2C+the+command+line+cannot+contain+macros+-+it+must+be%0A%23+typed+exactly+as+it+should
+be+executed.%0A%23%0A%23+Note%3A+Any+plugins+that+are+used+in+the+command+lines+must+reside%0A%23+on+the+machine+that+this+daemon+is+
running+on%21++The+examples+below%0A%23+assume+that+you+have+plugins+installed+in+a+%2Fusr%2Flocal%2Fnagios%2Flibexec%0A%23+directory.
++Also+note+that+you+will+have+to+modify+the+definitions+below%0A%23+to+match+the+argument+format+the+plugins+expect.++Remember%2C+the
se+are%0A%23+examples+only%21%0A%0A%23+The+following+examples+use+hardcoded+command+arguments...%0A%0A%23command%5Bcheck_users%5D%3D%2
Fusr%2Flib%2Fnagios%2Fplugins%2Fcheck_users+-w+5+-c+10%0A%23command%5Bcheck_load%5D%3D%2Fusr%2Flib%2Fnagios%2Fplugins%2Fcheck_load+-w+
15%2C10%2C5+-c+30%2C25%2C20%0A%23command%5Bcheck_disk1%5D%3D%2Fusr%2Flib%2Fnagios%2Fplugins%2Fcheck_disk+-w+20+-c+10+-p+%2Fdev%2Fhda1%
0A%23command%5Bcheck_disk2%5D%3D%2Fusr%2Flib%2Fnagios%2Fplugins%2Fcheck_disk+-w+20+-c+10+-p+%2Fdev%2Fhdb1%0A%23command%5Bcheck_zombie_
procs%5D%3D%2Fusr%2Flib%2Fnagios%2Fplugins%2Fcheck_procs+-w+5+-c+10+-s+Z%0A%23command%5Bcheck_total_procs%5D%3D%2Fusr%2Flib%2Fnagios%2
Fplugins%2Fcheck_procs+-w+150+-c+200+%0A%0A%23+The+following+examples+allow+user-supplied+arguments+and+can%0A%23+only+be+used+if+the+
NRPE+daemon+was+compiled+with+support+for+%0A%23+command+arguments+%2AAND%2A+the+dont_blame_nrpe+directive+in+this%0A%23+config+file+i
s+set+to+%271%27...%0A%0A%23command%5Bcheck_users%5D%3D%2Fusr%2Flib%2Fnagios%2Fplugins%2Fcheck_users+-w+%24ARG1%24+-c+%24ARG2%24%0Acom
mand%5Bcheck_load%5D%3D%2Fusr%2Flib%2Fnagios%2Fplugins%2Fcheck_load+-w+%24ARG1%24+-c+%24ARG2%24%0Acommand%5Bcheck_disk%5D%3D%2Fusr%2Fl
ib%2Fnagios%2Fplugins%2Fch
Jan 17 21:13:04 vps-harveys puppetd[8977]:
(//Node[vps-harveys]/monitoring/File[/etc/nagios/nrpe.cfg]/source) replacing
from source pu
ppet:///files/nrpe.cfg with contents {md5}62350842fae499561653fa11bbece8f8
Jan 17 21:13:04 vps-harveys puppetd[8977]:
(//Node[vps-harveys]/monitoring/File[/etc/nagios/nrpe.cfg]) Scheduling refresh
of Service[n
agios-nrpe-server]
Jan 17 21:13:04 vps-harveys puppetd[8977]:
(//Node[vps-harveys]/vps-systems/File[/etc/bash.bashrc]/source) No specified
sources exist
Jan 17 21:13:04 vps-harveys puppetd[8977]:
(//Node[vps-harveys]/monitoring/Service[nagios-nrpe-server]) Triggering
''refresh'' from 1 de
pendencies
Jan 17 21:13:04 vps-harveys nrpe[1025]: Caught SIGTERM - shutting down...
Anyone else seen similar behaviour?
-----Original Message-----
From: puppet-users-bounces@madstop.com on behalf of Luke Kanies
Sent: Thu 17/01/2008 8:34 PM
To: Puppet User Discussion
Subject: Re: [Puppet-users] Memory Leak
On Jan 15, 2008, at 5:36 PM, Arjuna Christensen wrote:
> So, I''ve stumbled across bleak_house
(http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html
> ) - this looks like it may help me track down what is causing the
> Memory leak I''m seeing (be it me, Ubuntu, Debian, or Puppet).
>
> First step: I''m going to clone the repo, branch
> Second step: Learn Ruby - shouldn''t be too hard.
> Third step: Do exactly what bleak_house says!
>
> At the start of your app, put:
>
> require ''rubygems''
> require ''bleak_house''
> $memlogger = BleakHouse::Logger.new
> File.delete($logfile = "/path/to/logfile") rescue nil
>
> Now, at the points of interest, put:
>
> $memlogger.snapshot($logfile, "tag/subtag", false)
>
> Run your app. Once you are done, analyze your data:
>
> bleak /path/to/logfile
>
> From this, I''m assuming that ''points of
interest'' would be whenever
> we parse a resource, or whenever we send a resource to a puppet -
> may be wrong though - comments?
If the server is leaking memory, it''s almost guaranteed to be in one
of two places: Parsing, or file-serving.
If it''s parsing, then it''s either in the build-up of the AST
tree
during actual parsing, or the creation of the catalog during
compilation. If the former, then look for subclasses of
Puppet::Parser::AST; if the latter, then look for instances of
Puppet::Parser::Resource.
If it''s during file serving, then look for instances of
Puppet::Type::File.
Is that what you''re looking for?
--
One of the Ten Commandments for Technicians:
(7) Work thou not on energized equipment, for if thou dost, thy
fellow workers will surely buy beers for thy widow and
console her in other ways.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users
_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users