I don''t like how much of a pig puppetd uses memory. puppetd using 100MB+ on a VPS that has only 256 MB (which isn''t that uncommon) is just plain awful. Yes I know I can make puppetd into a cron job but then loose the ability to run puppet via puppetrun command. I haven''t gone down this road yet, but I''m wondering if I can use inetd/xinetd to fire up a puppetrun request? --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Sun, Jun 22, 2008 at 7:57 AM, Larry Ludwig <larrylud@gmail.com> wrote:> > I don''t like how much of a pig puppetd uses memory. puppetd using > 100MB+ on a VPS that has only 256 MB (which isn''t that uncommon) is > just plain awful. Yes I know I can make puppetd into a cron job but > then loose the ability to run puppet via puppetrun command. > > I haven''t gone down this road yet, but I''m wondering if I can use > inetd/xinetd to fire up a puppetrun request?Why not use Capistrano instead? A simple cap recipe for this might be: role :puppet, server1, sever2, server3 namespace :puppet do desc <<-DESC Run puppet on each. You can set the Puppet Server with the SERVER env variable. You can specify a --tags value with TAGS. DESC task :default do puppet_server = nil if ENV.has_key?("SERVER") puppet_server = ENV["SERVER"] else puppet_server = `hostname -f`.chomp! end tags = "" if ENV.has_key?("TAGS") tags = "--tags #{ENV["TAGS"]}" end sudo("sleep $(expr $RANDOM \% 10); sudo puppetd --onetime --verbose --ignorecache --server # {puppet_server} #{tags}") end end That is untested, but cribbed from working code (that uses iClassify to get on-the-fly lists of nodes to run puppet on.) That would allow you to trigger puppet in the same way that puppetrun will, but without the overhead of a running puppetd. Adam -- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: adam@hjksolutions.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I''m also not happy with puppet memory footprint, What is the common practice in other places? do you run puppet from a cronjob or do you run it as a daemon? for me, the main reason to run puppet as a daemon mode was to be able to execute it remotely and to have a randomize drift between client runs (i.e. after some time each client is fetching the configs in a more or less random time). I would be happy to have an xinetd ( or similar ) option officially support, or maybe there is a way to reduce puppet footprint (split between puppet daemon and puppet actual client?). Ohad On Sun, Jun 22, 2008 at 11:59 PM, Adam Jacob <adam@hjksolutions.com> wrote:> > On Sun, Jun 22, 2008 at 7:57 AM, Larry Ludwig <larrylud@gmail.com> wrote: > > > > I don''t like how much of a pig puppetd uses memory. puppetd using > > 100MB+ on a VPS that has only 256 MB (which isn''t that uncommon) is > > just plain awful. Yes I know I can make puppetd into a cron job but > > then loose the ability to run puppet via puppetrun command. > > > > I haven''t gone down this road yet, but I''m wondering if I can use > > inetd/xinetd to fire up a puppetrun request? > > Why not use Capistrano instead? > > A simple cap recipe for this might be: > > role :puppet, server1, sever2, server3 > > namespace :puppet do > > desc <<-DESC > Run puppet on each. You can set the Puppet Server with the SERVER > env variable. > > You can specify a --tags value with TAGS. > DESC > task :default do > puppet_server = nil > if ENV.has_key?("SERVER") > puppet_server = ENV["SERVER"] > else > puppet_server = `hostname -f`.chomp! > end > tags = "" > if ENV.has_key?("TAGS") > tags = "--tags #{ENV["TAGS"]}" > end > sudo("sleep $(expr $RANDOM \% 10); sudo puppetd --onetime > --verbose --ignorecache --server # > {puppet_server} #{tags}") > end > end > > That is untested, but cribbed from working code (that uses iClassify > to get on-the-fly lists of nodes to run puppet on.) > > That would allow you to trigger puppet in the same way that puppetrun > will, but without the overhead of a running puppetd. > > Adam > > -- > HJK Solutions - We Launch Startups - http://www.hjksolutions.com > Adam Jacob, Senior Partner > T: (206) 508-4759 E: adam@hjksolutions.com > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I''m equally frustrated. I''m about to cron this: perl -e "sleep int(rand(20));" ; puppetd --onetime Where 20 (in seconds) is closer to 25min, and the cron runs on the half hour.. What is the puppetd client holding on to in between runs? Shouldn''t it just be a scheduler anyway? --joel On Tue, Jun 24, 2008 at 8:56 PM, Ohad Levy <ohadlevy@gmail.com> wrote:> I''m also not happy with puppet memory footprint, What is the common practice > in other places? do you run puppet from a cronjob or do you run it as a > daemon? > > for me, the main reason to run puppet as a daemon mode was to be able to > execute it remotely and to have a randomize drift between client runs (i.e. > after some time each client is fetching the configs in a more or less random > time). > > I would be happy to have an xinetd ( or similar ) option officially support, > or maybe there is a way to reduce puppet footprint (split between puppet > daemon and puppet actual client?). > > Ohad > > On Sun, Jun 22, 2008 at 11:59 PM, Adam Jacob <adam@hjksolutions.com> wrote: >> >> On Sun, Jun 22, 2008 at 7:57 AM, Larry Ludwig <larrylud@gmail.com> wrote: >> > >> > I don''t like how much of a pig puppetd uses memory. puppetd using >> > 100MB+ on a VPS that has only 256 MB (which isn''t that uncommon) is >> > just plain awful. Yes I know I can make puppetd into a cron job but >> > then loose the ability to run puppet via puppetrun command. >> > >> > I haven''t gone down this road yet, but I''m wondering if I can use >> > inetd/xinetd to fire up a puppetrun request? >> >> Why not use Capistrano instead? >> >> A simple cap recipe for this might be: >> >> role :puppet, server1, sever2, server3 >> >> namespace :puppet do >> >> desc <<-DESC >> Run puppet on each. You can set the Puppet Server with the SERVER >> env variable. >> >> You can specify a --tags value with TAGS. >> DESC >> task :default do >> puppet_server = nil >> if ENV.has_key?("SERVER") >> puppet_server = ENV["SERVER"] >> else >> puppet_server = `hostname -f`.chomp! >> end >> tags = "" >> if ENV.has_key?("TAGS") >> tags = "--tags #{ENV["TAGS"]}" >> end >> sudo("sleep $(expr $RANDOM \% 10); sudo puppetd --onetime >> --verbose --ignorecache --server # >> {puppet_server} #{tags}") >> end >> end >> >> That is untested, but cribbed from working code (that uses iClassify >> to get on-the-fly lists of nodes to run puppet on.) >> >> That would allow you to trigger puppet in the same way that puppetrun >> will, but without the overhead of a running puppetd. >> >> Adam >> >> -- >> HJK Solutions - We Launch Startups - http://www.hjksolutions.com >> Adam Jacob, Senior Partner >> T: (206) 508-4759 E: adam@hjksolutions.com >> >> > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Wed, Jun 25, 2008 at 11:15 AM, Joel Krauska <jkrauska@gmail.com> wrote:> > I''m equally frustrated. > > I''m about to cron this: > perl -e "sleep int(rand(20));" ; puppetd --onetime > > Where 20 (in seconds) is closer to 25min, and the cron runs on the half hour.. > > What is the puppetd client holding on to in between runs? > Shouldn''t it just be a scheduler anyway? >How are you guess running puppetd? It has almost no footprint on any of the servers or workstations. Even the puppetmasterd is not really using much in the way of resource when idle. I am using 2.24.4 on CentOS boxes with the EPEL puppet rpms. Evan --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
2008/6/25 Evan Hisey <ehisey@gmail.com>:> > On Wed, Jun 25, 2008 at 11:15 AM, Joel Krauska <jkrauska@gmail.com> wrote: >> >> I''m equally frustrated. >> >> I''m about to cron this: >> perl -e "sleep int(rand(20));" ; puppetd --onetime >> >> Where 20 (in seconds) is closer to 25min, and the cron runs on the half hour.. >> >> What is the puppetd client holding on to in between runs? >> Shouldn''t it just be a scheduler anyway? >> > How are you guess running puppetd? It has almost no footprint on any > of the servers or workstations. Even the puppetmasterd is not really > using much in the way of resource when idle. I am using 2.24.4 on > CentOS boxes with the EPEL puppet rpms.The original poster said it was chewing 100MB+ on a 256 MB VM. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetd On a smaller machine, that is a significant chunk of RAM. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
what modules do you have installed. I don''t know if using 64 bit makes it also use more memory (which is not that uncommon) Here is from one VPS. USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31695 0.1 15.6 127668 82284 ? Ss Jun23 2:21 /usr/ bin/ruby /usr/sbin/puppetd On Jun 25, 12:31 pm, "Duncan Hill" <bajand...@googlemail.com> wrote:> 2008/6/25 Evan Hisey <ehi...@gmail.com>: > > > > > > > On Wed, Jun 25, 2008 at 11:15 AM, Joel Krauska <jkrau...@gmail.com> wrote: > > >> I''m equally frustrated. > > >> I''m about to cron this: > >> perl -e "sleep int(rand(20));" ; puppetd --onetime > > >> Where 20 (in seconds) is closer to 25min, and the cron runs on the half hour.. > > >> What is the puppetd client holding on to in between runs? > >> Shouldn''t it just be a scheduler anyway? > > > How are you guess running puppetd? It has almost no footprint on any > > of the servers or workstations. Even the puppetmasterd is not really > > using much in the way of resource when idle. I am using 2.24.4 on > > CentOS boxes with the EPEL puppet rpms. > > The original poster said it was chewing 100MB+ on a 256 MB VM. > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetd > > On a smaller machine, that is a significant chunk of RAM.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Wed, Jun 25, 2008 at 9:31 AM, Duncan Hill <bajandude@googlemail.com> wrote:> The original poster said it was chewing 100MB+ on a 256 MB VM. > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetdHere''s an example host of mine right now. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 29102 root 15 0 132m 53m 2504 S 0 0.3 0:03.74 ruby We tend to use certain machines as memory caches and try to max out our memory usage on them. So it''s not just low memory machines, but you also get problems when you''re trying to use memory efficiently. When there''s low memory available, the background puppetd won''t even fire. --joel --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Wed, Jun 25, 2008 at 11:42 AM, Joel Krauska <jkrauska@gmail.com> wrote:> > On Wed, Jun 25, 2008 at 9:31 AM, Duncan Hill <bajandude@googlemail.com> wrote: >> The original poster said it was chewing 100MB+ on a 256 MB VM. >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetd > > Here''s an example host of mine right now. > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 29102 root 15 0 132m 53m 2504 S 0 0.3 0:03.74 ruby > > We tend to use certain machines as memory caches and try to max out > our memory usage on them. > So it''s not just low memory machines, but you also get problems when > you''re trying to use memory efficiently. > > When there''s low memory available, the background puppetd won''t even fire. > > --joel >I was unaware of that issue, but not not really suprised. Low memory does tend to do that. I wonder if puppet does not like virtual hosts. I saw another postin gon the list were a user was having high memory issues on a Virtual host. Evan --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
2008/6/25 Evan Hisey <ehisey@gmail.com>:> I was unaware of that issue, but not not really suprised. Low memory > does tend to do that. I wonder if puppet does not like virtual hosts. > I saw another postin gon the list were a user was having high memory > issues on a Virtual host.We''re not running it on any virtual hosts and are seeing large memory footprints as well. Also seeing really large footprints for the puppetmaster (though I don''t care as much about that). Sorry about the formatting. USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 8399 0.8 2.0 166316 83484 ? Ssl 17:25 0:19 /usr/bin/ruby /usr/sbin/puppetd puppet 15543 0.0 5.3 303764 216352 ? Ssl Jun24 1:04 /usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel --masterport 18140 --pidfile /var/run/puppet/puppetmaster-18140.pid puppet 15569 0.0 2.1 172812 85180 ? Ssl Jun24 0:38 /usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel --masterport 18141 --pidfile /var/run/puppet/puppetmaster-18141.pid puppet 15595 0.1 3.1 214396 126456 ? Ssl Jun24 2:09 /usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel --masterport 18142 --pidfile /var/run/puppet/puppetmaster-18142.pid puppet 15621 0.0 2.5 191304 103760 ? Ssl Jun24 0:37 /usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel --masterport 18143 --pidfile /var/run/puppet/puppetmaster-18143.pid puppet 15647 0.0 1.7 157968 70560 ? Ssl Jun24 0:43 /usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel --masterport 18144 --pidfile /var/run/puppet/puppetmaster-18144.pid puppet 15673 0.0 1.8 164452 76900 ? Ssl Jun24 0:32 /usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel --masterport 18145 --pidfile /var/run/puppet/puppetmaster-18145.pid puppet 15699 0.0 1.8 161176 73764 ? Ssl Jun24 0:44 /usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel --masterport 18146 --pidfile /var/run/puppet/puppetmaster-18146.pid puppet 15725 0.0 2.4 184812 97252 ? Ssl Jun24 0:41 /usr/bin/ruby /usr/sbin/puppetmasterd --servertype mongrel --masterport 18147 --pidfile /var/run/puppet/puppetmaster-18147.pid --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On 6/25/2008 12:09 PM, Evan Hisey wrote:> I was unaware of that issue, but not not really suprised. Low memory > does tend to do that. I wonder if puppet does not like virtual hosts. > I saw another postin gon the list were a user was having high memory > issues on a Virtual host.It''s certainly not a guarantee that you''ll have memory problems on low-memory virtual hosts: ==== # free total used free shared buffers cached Mem: 262292 252920 9372 0 6052 46736 -/+ buffers/cache: 200132 62160 Swap: 1048568 99448 949120 # uname -r 2.6.18-4-xen-686 # puppetd --version 0.23.1 # ps aux | grep [p]uppetd root 21479 0.2 17.6 57608 46248 ? Ss Jun04 69:35 ruby /usr/sbin/puppetd --server gold.cae.tntech.edu --factsync ==== Most of my memory on this box is sucked up by Zope/Plone and Apache, not by Puppet. -- Mike Renfro / R&D Engineer, Center for Manufacturing Research, 931 372-3601 / Tennessee Technological University --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
RijilV <rijilv@gmail.com> writes:> We''re not running it on any virtual hosts and are seeing large memory > footprints as well. Also seeing really large footprints for the > puppetmaster (though I don''t care as much about that).puppetmaster still leaks like a sieve for us. :/ We have to restart it nightly to keep the system from running out of memory, and even that doesn''t always work. It''s a problem that''s been going on for a long time, unfortunately, and it''s not at all clear what''s causing it. We''re currently running a slightly patched version of 0.24.4. -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/> --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi>> We''re not running it on any virtual hosts and are seeing large memory >> footprints as well. Also seeing really large footprints for the >> puppetmaster (though I don''t care as much about that). > > puppetmaster still leaks like a sieve for us. :/ We have to restart it > nightly to keep the system from running out of memory, and even that > doesn''t always work. It''s a problem that''s been going on for a long time, > unfortunately, and it''s not at all clear what''s causing it. > > We''re currently running a slightly patched version of 0.24.4.I have the same problem on centos 5 boxes, x86_64 as well i386, all running in xenguests and 0.24.4 (as well the guests). I deploy a cron on puppetmasters to restart puppetmaster every 4 hours. Just to be on the safe side. As well I''m having some big problems with certain modules and use cases [1]. However currently I simply get problems like these working with workarounds. :-/ PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6562 root 15 0 183m 96m 2260 S 0 32.2 6:35.53 puppetd on a host with 300mb ram. I couldn''t yet figure out what is causing this problem, but it certainly exists. Unfortunately I don''t know yet a lot how to debug problems like these and I''m hoping that problems like these might go away with the switch away from xmlrpc, as I could see a lot of memory problems there. but now i''m just guessing. however I certainly don''t have anymore any corrupted file due to memory leaks! :) if anybody would like to dig into this problem and would like to have data from hosts i''m having access to it, please ping me on irc (duritong). I''d really like to help on this problem. greets pete [1] http://groups.google.com/group/puppet-users/browse_thread/thread/26aa8025de704cc4/1c9608f58360cdbb?hl=en&lnk=gst&q=djbdns#1c9608f58360cdbb --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Jun 25, 2008, at 1:22 PM, Peter Meier wrote:> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > 6562 root 15 0 183m 96m 2260 S 0 32.2 6:35.53 puppetd > > on a host with 300mb ram.This is a pretty clear indication that the next release should be focused on ram usage. -- A person''s maturity consists in having found again the seriousness one had as a child, at play. --Friedrich Nietzsche --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I think that there is a bit difference between a puppetmaster and a client in terms of ram usage. I for one, don''t care that much how much RAM a puppet master is using (as long as its stable), because that''s a service, which needs system resources to perform its duty,however, as far as for the client, I cant find any good argument why it should use a lot of memory when all its suppose to do is to wait for puppetrun requests and count until its next run. It would be really nice, if puppetd could release its system resources when in idle mode. Thanks, Ohad On Thu, Jun 26, 2008 at 6:35 AM, Luke Kanies <luke@madstop.com> wrote:> > On Jun 25, 2008, at 1:22 PM, Peter Meier wrote: > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > > > > 6562 root 15 0 183m 96m 2260 S 0 32.2 6:35.53 puppetd > > > > on a host with 300mb ram. > > > This is a pretty clear indication that the next release should be > focused on ram usage. > > -- > A person''s maturity consists in having found again the seriousness one > had as a child, at play. --Friedrich Nietzsche > --------------------------------------------------------------------- > Luke Kanies | http://reductivelabs.com | http://madstop.com > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
"Ohad Levy" <ohadlevy@gmail.com> writes:> I think that there is a bit difference between a puppetmaster and a > client in terms of ram usage. > > I for one, don''t care that much how much RAM a puppet master is using > (as long as its stable), because that''s a service, which needs system > resources to perform its duty,however, as far as for the client, I cant > find any good argument why it should use a lot of memory when all its > suppose to do is to wait for puppetrun requests and count until its next > run.I care about both because the memory consumption of the puppetmaster daemon is currently the limiting factor on how many of them I can run on a single system, which is currently my performance limitation in scaling Puppet. Right now, we can''t run our infrastructure reliably on a single puppetmaster server in large part because the system has 4GB of memory, which isn''t sufficient for even 10 puppetmasterds restarted daily. When we try to put another system with only 2GB of memory into the rotation, it doesn''t last long enough to be useful before it runs out of memory, although we could probably play more games with balancing between the two. We have new hardware about to be racked, but really, I don''t think puppetmaster should be this heavy. -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/> --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
By-the-way puppetd --onetime --no-client does not appear to work as I would expect.. (run once and exit) How do you folks running puppetd client in crond call it? --test works, but it''s verbose... --joel --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Joel Krauska wrote:> By-the-way > > puppetd --onetime --no-client > does not appear to work as I would expect.. (run once and exit) > > How do you folks running puppetd client in crond call it? > > --test works, but it''s verbose... > > --joel >/usr/sbin/puppetd -t --noop --puppetdlockfile /var/lib/puppet/state/puppetcronlock, is how I run it --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
2008/6/25 Peter Meier <peter.meier@immerda.ch>:> > Hi > >>> We''re not running it on any virtual hosts and are seeing large memory >>> footprints as well. Also seeing really large footprints for the >>> puppetmaster (though I don''t care as much about that). >> >> puppetmaster still leaks like a sieve for us. :/ We have to restart it >> nightly to keep the system from running out of memory, and even that> I have the same problem on centos 5 boxes, x86_64 as well i386, all > running in xenguests and 0.24.4 (as well the guests). I deploy a cron on > puppetmasters to restart puppetmaster every 4 hours. Just to be on thePuppetmaster 1 - catalog server. VIRT 99.6, RES 82. Puppetmaster 2 - file server. VIRT 52M, RES 37M I''ve also got a restart cron job, but it only smacks the catalog server, not the file server. CentOS 5, x86, VMware Server guest. Before I split the file server out (and I sent a patch to dlutter for the RHEL init script, perhaps it belongs elsewhere), I''d routinely see puppetmasterd at 200+ MB of memory on a VM that had 256. Add in the backup software taking VIRT 139/RES 4.5, and a postgres instance for storeconfigs, and I''d also be 200 MB into swap, and compile times would hit hundreds of seconds. Even with the cron to restart it, this happened. So perhaps splitting the file server out has benefits beyond file serving hogging CPU on the catalog generation? --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
> I care about both because the memory consumption of the puppetmaster > daemon is currently the limiting factor on how many of them I can run on a > single system, which is currently my performance limitation in scaling > Puppet. Right now, we can''t run our infrastructure reliably on a single > puppetmaster server in large part because the system has 4GB of memory, > which isn''t sufficient for even 10 puppetmasterds restarted daily. > > When we try to put another system with only 2GB of memory into the > rotation, it doesn''t last long enough to be useful before it runs out of > memory, although we could probably play more games with balancing between > the two. > > We have new hardware about to be racked, but really, I don''t think > puppetmaster should be this heavy.What is case you to need to 10 puppetmasterds on a server? From what I have seen on the list, a single instance can handle a very large work load. The usual trick for multiple puppetmasterds is when they are in split out over several regions and there is a link speed issue. There may be a better way to handle you setup, or I need to be aware of why you need to do that as I push to spread puppet out to more of our areas. Evan --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
"Evan Hisey" <ehisey@gmail.com> writes:> What is case you to need to 10 puppetmasterds on a server? From what I > have seen on the list, a single instance can handle a very large work > load.A single instance wasn''t even remotely handling a large work load for us. Running multiple puppetmasterds behind Apache to load-balance helped considerably. We''re planning on switching to Pound for the load-balancing since we''ve heard it works a lot better.> The usual trick for multiple puppetmasterds is when they are in split > out over several regions and there is a link speed issue. There may be > a better way to handle you setup, or I need to be aware of why you need > to do that as I push to spread puppet out to more of our areas.It''s possible that you''ve just not scaled quite as far as we have. I''m not sure. We don''t have that many nodes but they''re nearly all different from each other and we do a very wide variety of stuff to them. lsdb:/srv/puppet-ca/ssl/ca# find signed -type f -print | wc -l 310 windlord:~/work/puppet> find . -name \*.pp -print | wc -l 302 windlord:~/work/puppet> find . -name \*.pp -print0 | xargs -0 wc -l | tail -1 24797 total windlord:~/work/puppet> find */*/files -type f -print | wc -l 5489 On the average, it takes fifteen seconds to compile a manifest, which is already too slow for my taste. henson2:/var/log> grep ''Compiled configuration'' puppetmaster.log | sed -e ''s/.* in //'' -e ''s/ seconds.*//'' | awk ''{ count++; total += $1 } END { print total / count }'' 15.853 -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/> --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
"Duncan Hill" <bajandude@googlemail.com> writes:> Before I split the file server out (and I sent a patch to dlutter for > the RHEL init script, perhaps it belongs elsewhere), I''d routinely see > puppetmasterd at 200+ MB of memory on a VM that had 256. Add in the > backup software taking VIRT 139/RES 4.5, and a postgres instance for > storeconfigs, and I''d also be 200 MB into swap, and compile times would > hit hundreds of seconds. Even with the cron to restart it, this > happened. > > So perhaps splitting the file server out has benefits beyond file > serving hogging CPU on the catalog generation?That''s entirely possible. We do a *lot* of file service. -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/> --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Russ Allbery <rra@stanford.edu> writes:> windlord:~/work/puppet> find */*/files -type f -print | wc -l > 5489Sorry, that''s a bogus number. I always forget about Subversion''s annoying overhead. Here''s the correct number. windlord:~/work/puppet> find */*/files -name .svn -prune -o -type f -print | wc -l 1566 -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/> --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Thu, Jun 26, 2008 at 7:34 PM, Russ Allbery <rra@stanford.edu> wrote:> > "Evan Hisey" <ehisey@gmail.com> writes: > >> What is case you to need to 10 puppetmasterds on a server? From what I >> have seen on the list, a single instance can handle a very large work >> load. > > A single instance wasn''t even remotely handling a large work load for us. > Running multiple puppetmasterds behind Apache to load-balance helped > considerably. We''re planning on switching to Pound for the load-balancing > since we''ve heard it works a lot better. > >> The usual trick for multiple puppetmasterds is when they are in split >> out over several regions and there is a link speed issue. There may be >> a better way to handle you setup, or I need to be aware of why you need >> to do that as I push to spread puppet out to more of our areas. > > It''s possible that you''ve just not scaled quite as far as we have. I''m > not sure. We don''t have that many nodes but they''re nearly all different > from each other and we do a very wide variety of stuff to them. > > lsdb:/srv/puppet-ca/ssl/ca# find signed -type f -print | wc -l > 310 > > windlord:~/work/puppet> find . -name \*.pp -print | wc -l > 302 > windlord:~/work/puppet> find . -name \*.pp -print0 | xargs -0 wc -l | tail -1 > 24797 total > > windlord:~/work/puppet> find */*/files -type f -print | wc -l > 5489 > > On the average, it takes fifteen seconds to compile a manifest, which is > already too slow for my taste. > > henson2:/var/log> grep ''Compiled configuration'' puppetmaster.log | sed -e ''s/.* in //'' -e ''s/ seconds.*//'' | awk ''{ count++; total += $1 } END { print total / count }'' > 15.853 >We have definitely not scaled out that far yet, part of the reason I was asking. I like learning from others mistakes/success. !0 puppet masters on one server still strikes me as counter productive from a speed stand point. Each instance is then increasing the load by N on the server, or has this not been a facter so much as the memory foot print? Evan --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
"Evan Hisey" <ehisey@gmail.com> writes:> We have definitely not scaled out that far yet, part of the reason I was > asking. I like learning from others mistakes/success. !0 puppet masters > on one server still strikes me as counter productive from a speed stand > point. Each instance is then increasing the load by N on the server, or > has this not been a facter so much as the memory foot print?The load is a bit of an issue but hasn''t been as much of a factor as the memory footprint. The system has two CPUs and eight logical processes with hyperthreading, and we definitely saw a performance increase when going from five puppetmasters to ten. Of course, with less beefy hardware, it would make sense to spread out across more machines instead, but it''s getting to the point where that''s almost a standard server box. -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/> --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
> henson2:/var/log> grep ''Compiled configuration'' puppetmaster.log | sed -e ''s/.* in //'' -e ''s/ seconds.*//'' | awk ''{ count++; total += $1 } END { print total / count }'' > 15.853server1# grep ''Compiled catalog'' /var/log/puppet/puppetmaster.log | sed -e ''s/.* in //'' -e ''s/ seconds.*//'' | awk ''{ count++; total += $1 } END { print total / count }'' 14.2223 running 4 instances load balanced with nginx. server2# grep ''Compiled catalog'' /var/log/puppet/puppetmaster.log | sed -e ''s/.* in //'' -e ''s/ seconds.*//'' | awk ''{ count++; total += $1 } END { print total / count }'' 65.5803 currently only running 1 instance with webrick. soon switching to nginx and as well 4 instances. will see if that changes anything. both are yet only serving a bunch of hosts. more to come soon. actually I don''t really care how long it takes to compile the catalogs (only when testing new features :P ) but more than a minute seems also long for me. 15 seconds are currently ok. btw: in which Version changed the log entry from Compiled configuration to Compiled catalog? greets pete --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
2008/6/25 Duncan Hill <bajandude@googlemail.com>:> The original poster said it was chewing 100MB+ on a 256 MB VM. > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 22533 root 15 0 141m 61m 2696 S 0 6.1 9:12.38 puppetd > > On a smaller machine, that is a significant chunk of RAM.Coming to a VM near you, When Puppetmasterd Goes Bad. Starring Puppet 0.24.4, CentOS 5.1 and VMware Server 1.04. Mem: 340780k total, 337032k used, 3748k free, 156k buffers Swap: 819192k total, 409596k used, 409596k free, 9268k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17923 puppet 18 0 400m 249m 2828 R 51.0 74.9 80:56.51 puppetmasterd 6856 puppet 15 0 56484 6144 1456 S 2.3 1.8 524:38.33 puppetmasterd The first one is the manifest puppetmaster. The second is the file server. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
> > The load is a bit of an issue but hasn''t been as much of a factor as the > memory footprint. The system has two CPUs and eight logical processes > with hyperthreading, and we definitely saw a performance increase when > going from five puppetmasters to ten.I assume you know this but will throw it out just in case. hyperthreading != multi-core CPUs. In some cases hyperthreading performs much worse when enabled. I don''t know about with a puppetmaster server setup, but you may want to test this. Also the newer Xeons (woodcrests and newer) have a much lower power consumption, and blow the pants off of the older Xeons. Right now from what I see, I believe puppetmaster needs much more optimization than anything else. -L -- Larry Ludwig Empowering Media 1-866-792-0489 x600 Managed and Unmanaged Xen VPSes http://www.hostcube.com/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Larry Ludwig <larrylud@gmail.com> writes:> I assume you know this but will throw it out just in case. > > hyperthreading != multi-core CPUs. In some cases hyperthreading > performs much worse when enabled. I don''t know about with a > puppetmaster server setup, but you may want to test this.Yeah, I know. We haven''t experimented yet with this. We might with the new system. -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/> --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi Luke Kanies wrote:> On Jun 25, 2008, at 1:22 PM, Peter Meier wrote: > >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> >> >> 6562 root 15 0 183m 96m 2260 S 0 32.2 6:35.53 puppetd >> >> on a host with 300mb ram. > > > This is a pretty clear indication that the next release should be > focused on ram usage.should I open a ticket, wich tracks a bit the problems and points raised in this discussion? so we have something that these points don''t get lost. greets pete --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Jul 1, 2008, at 8:56 AM, Peter Meier wrote:> should I open a ticket, wich tracks a bit the problems and points > raised > in this discussion? so we have something that these points don''t get > lost.Yes. -- Chase after truth like hell and you''ll free yourself, even though you never touch its coat-tails. -- Clarence Darrow --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi>> should I open a ticket, wich tracks a bit the problems and points >> raised >> in this discussion? so we have something that these points don''t get >> lost. > > > Yes.I tried to summarize the recent discussion in bug #1395 [1]. I tried to collect the interesting points of the discussion and tried to summarize the interesting data. I hope I noticed everything and that the important data is in the bug report. For me it is the first attempt for this kind of bugreport. I hope it is useful and that it might help to investigate the problem further. I''d like to help to investigate on this problem, as I have a lot of xen guests having not much memory, which should do just a bunch of little tasks. (dns for example) However I still like to manage them with puppet, which is kind of problematic at the current memory usage. As I don''t have much of experience in debugging of ruby applications especially on memory usage, it would be nice to have some people with experience joining the investigation(s). If somebody teaches me to collect any data needed from running puppetds, I''m happy to provide you with additional data. greets pete [1] http://reductivelabs.com/redmine/issues/show/1395 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---