Hi, I''m running multiple vservers and have some aspects managed via puppet. I noticed that puppetd consumes from 20 to 60 mb RAM and the longer the process runs, the more RAM it takes. It looks like 60MB is the top what I measured. Having a couple of vservers (11 in my case) this can already take up to 500MB just for puppet! I''m wondering if there''s any downside in disabling the puppet daemon behavior and controlling it via cron and the --onetime switch? thanks, - Markus --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Aug 4, 2008, at 4:15 PM, Markus Fischer wrote:> > Hi, > > I''m running multiple vservers and have some aspects managed via > puppet. I > noticed that puppetd consumes from 20 to 60 mb RAM and the longer > the process > runs, the more RAM it takes. It looks like 60MB is the top what I > measured. > > Having a couple of vservers (11 in my case) this can already take up > to 500MB > just for puppet! I''m wondering if there''s any downside in disabling > the puppet > daemon behavior and controlling it via cron and the --onetime switch?Not at this point, no. Hopefully over time we''ll add awesome services that will encourage you to run the daemon, but we''ll also hopefully fix those memory leaks. BTW, Andrew has nailed down some pretty consistent mechanisms for finding memory leaks, so if anyone is experiencing them and willing to delve a bit into ruby, ping Andrew to have him help you track them down. I''d really like to get rid of them, but we can only fix the ones we''re experiencing. -- You can''t wait for inspiration. You have to go after it with a club. -- Jack London --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
> BTW, Andrew has nailed down some pretty consistent mechanisms for > finding memory leaks, so if anyone is experiencing them and willing to > delve a bit into ruby, ping Andrew to have him help you track them > down. I''d really like to get rid of them, but we can only fix the > ones we''re experiencing. >In our case now it''s not the memory leaks (at least with puppetd) it''s the memory usage. It gets big and stays big. Big being 100-150MB. The latest release of puppetmaster (24.5) seemed to have helped with the memory issues. -L --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Mon, Aug 4, 2008 at 8:17 PM, Luke Kanies <luke@madstop.com> wrote:> > On Aug 4, 2008, at 4:15 PM, Markus Fischer wrote: > > > > > Hi, > > > > I''m running multiple vservers and have some aspects managed via > > puppet. I > > noticed that puppetd consumes from 20 to 60 mb RAM and the longer > > the process > > runs, the more RAM it takes. It looks like 60MB is the top what I > > measured. > > > > Having a couple of vservers (11 in my case) this can already take up > > to 500MB > > just for puppet! I''m wondering if there''s any downside in disabling > > the puppet > > daemon behavior and controlling it via cron and the --onetime switch? > > Not at this point, no. Hopefully over time we''ll add awesome services > that will encourage you to run the daemon, but we''ll also hopefully > fix those memory leaks.Luke, maybe I am missing something. Wouldn''t using cron break the ability to do a puppetrun from the puppetmasterd host? (IE: break the ability to say "update hosts NOW"). Thanks, Brian> BTW, Andrew has nailed down some pretty consistent mechanisms for > finding memory leaks, so if anyone is experiencing them and willing to > delve a bit into ruby, ping Andrew to have him help you track them > down. I''d really like to get rid of them, but we can only fix the > ones we''re experiencing. > > -- > You can''t wait for inspiration. You have to go after it with a club. > -- Jack London > --------------------------------------------------------------------- > Luke Kanies | http://reductivelabs.com | http://madstop.com > > > > >-- - Brian Gupta http://opensolaris.org/os/project/nycosug/ http://www.genunix.org/wiki/index.php/OpenSolaris_New_User_FAQ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Mon, Aug 4, 2008 at 6:17 PM, Brian Gupta <brian.gupta@gmail.com> wrote:>> Not at this point, no. Hopefully over time we''ll add awesome services >> that will encourage you to run the daemon, but we''ll also hopefully >> fix those memory leaks. > > Luke, maybe I am missing something. Wouldn''t using cron break the ability to > do a puppetrun from the puppetmasterd host? (IE: break the ability to say > "update hosts NOW").Yes, although you can also use other tools for triggering puppetruns... Capistrano, for example. Adam -- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: adam@hjksolutions.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hallo Markus! Markus Fischer schrieb:> Hi, > > I''m running multiple vservers and have some aspects managed via puppet. I > noticed that puppetd consumes from 20 to 60 mb RAM and the longer the process > runs, the more RAM it takes. It looks like 60MB is the top what I measured. > > Having a couple of vservers (11 in my case) this can already take up to 500MB > just for puppet! I''m wondering if there''s any downside in disabling the puppet > daemon behavior and controlling it via cron and the --onetime switch?As Brian said, you''re missing out on triggering puppet runs with puppetrun. I''m running puppetd from cron since the beginning for exactly the same reasons. It''s way to easy to trigger a run from the vserver host than to think about puppetrun. Also, you can loop over all your vservers from the host and run puppet synchronously, so you can reduce the load without having to depend on splay. Regards, DavidS --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Aug 4, 2008, at 6:14 PM, Larry Ludwig wrote:> > > >> BTW, Andrew has nailed down some pretty consistent mechanisms for >> finding memory leaks, so if anyone is experiencing them and willing >> to >> delve a bit into ruby, ping Andrew to have him help you track them >> down. I''d really like to get rid of them, but we can only fix the >> ones we''re experiencing. >> > > In our case now it''s not the memory leaks (at least with puppetd) it''s > the memory usage. It gets big and stays big. Big being 100-150MB. > > The latest release of puppetmaster (24.5) seemed to have helped with > the memory issues.I''d qualify that as a leak, even if it eventually stops growing -- there''s absolutely no reason for puppetd to take that much ram. -- You can''t wait for inspiration. You have to go after it with a club. -- Jack London --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
> > I''d qualify that as a leak, even if it eventually stops growing -- > there''s absolutely no reason for puppetd to take that much ram. >Even on a 64 bit platform? it''s double the size. What is considered "normal"? -L --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
ok, I think I''m moving to cron... here is a report on a 64bit machine: PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND 11925 root 16 0 0 35:47.77 5.1 333m 300m 2584 S puppetd On Wed, Aug 6, 2008 at 2:43 AM, Larry Ludwig <larrylud@gmail.com> wrote:> > > >> >> I''d qualify that as a leak, even if it eventually stops growing -- >> there''s absolutely no reason for puppetd to take that much ram. >> > > Even on a 64 bit platform? it''s double the size. > > What is considered "normal"? > > -L > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Wed, 2008-08-06 at 09:56 +0800, Ohad Levy wrote:> ok, I think I''m moving to cron... > > here is a report on a 64bit machine: > PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND > 11925 root 16 0 0 35:47.77 5.1 333m 300m 2584 S puppetdDo you use File resource with recurse? I''ve noticed that if the local side is full of a deep file hierarchy puppet will build a huge hash/array that holds all the local files information during the comparison of source and destination. I''ve seen some puppetd growing to more than 1GB while transfering a couple of files on my home directory (I used to have a recursive file resource to copy my .zshrc and ssh keys). For more information see: http://reductivelabs.com/redmine/issues/show/1469 Hope that helps, -- Brice Figureau <brice-puppet@daysofwonder.com> --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi, Not using recursive file copies... but my state/localconfig.yaml is 720kb... I guess until the memory abuse will stop, I have to move to cron. Anyone knows how hard would it be to implemnt xinetd solution for puppetrun? Ohad On Wed, Aug 6, 2008 at 4:06 PM, Brice Figureau <brice-puppet@daysofwonder.com> wrote:> > On Wed, 2008-08-06 at 09:56 +0800, Ohad Levy wrote: >> ok, I think I''m moving to cron... >> >> here is a report on a 64bit machine: >> PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND >> 11925 root 16 0 0 35:47.77 5.1 333m 300m 2584 S puppetd > > Do you use File resource with recurse? > > I''ve noticed that if the local side is full of a deep file hierarchy > puppet will build a huge hash/array that holds all the local files > information during the comparison of source and destination. I''ve seen > some puppetd growing to more than 1GB while transfering a couple of > files on my home directory (I used to have a recursive file resource to > copy my .zshrc and ssh keys). > For more information see: > http://reductivelabs.com/redmine/issues/show/1469 > > Hope that helps, > -- > Brice Figureau <brice-puppet@daysofwonder.com> > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Sigh... I''m moving to cron for two reasons: - memory usage - puppetd itself seems to die at times and nagios has to restart it (we look at the age of the /var/lib/puppet/localconfig.yaml to determine if puppet should be restarted) -L -- Larry Ludwig Empowering Media 1-866-792-0489 x600 Fully Managed VPSes http://www.hostcube.com/ On Aug 5, 9:56 pm, "Ohad Levy" <ohadl...@gmail.com> wrote:> ok, I think I''m moving to cron... > > here is a report on a 64bit machine: > PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND > 11925 root 16 0 0 35:47.77 5.1 333m 300m 2584 S puppetd > > On Wed, Aug 6, 2008 at 2:43 AM, Larry Ludwig <larry...@gmail.com> wrote: > > >> I''d qualify that as a leak, even if it eventually stops growing -- > >> there''s absolutely no reason for puppetd to take that much ram. > > > Even on a 64 bit platform? it''s double the size. > > > What is considered "normal"? > > > -L--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
One other comment I should add. We just added --splay true option to our puppetd and made a world of difference on the puppetmaster load. I would suggest making this the default for puppetd. It smoothed out the load of puppetmaster quite a bit. Also another recommendation when using storeconfigs is make sure your MySQL is optimized. (of course if using mysql) If not familiar with tuning mysql, this application is a good start: http://rackerhacker.com/mysqltuner/ Maybe there should be a wiki page(s) to tips on how to optimize puppetd/puppetmaster? -L -- Larry Ludwig Empowering Media 1-866-792-0489 x600 Fully Managed VPSes http://www.hostcube.com/ On Aug 6, 7:54 am, Larry Ludwig <larry...@gmail.com> wrote:> Sigh... I''m moving to cron for two reasons: > - memory usage > - puppetd itself seems to die at times and nagios has to restart it > (we look at the age of the /var/lib/puppet/localconfig.yaml to > determine if puppet should be restarted) > > -L > > -- > Larry Ludwig > Empowering Media > 1-866-792-0489 x600 > Fully Managed VPSeshttp://www.hostcube.com/ > > On Aug 5, 9:56 pm, "Ohad Levy" <ohadl...@gmail.com> wrote: > > > ok, I think I''m moving to cron... > > > here is a report on a 64bit machine: > > PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND > > 11925 root 16 0 0 35:47.77 5.1 333m 300m 2584 S puppetd > > > On Wed, Aug 6, 2008 at 2:43 AM, Larry Ludwig <larry...@gmail.com> wrote: > > > >> I''d qualify that as a leak, even if it eventually stops growing -- > > >> there''s absolutely no reason for puppetd to take that much ram. > > > > Even on a 64 bit platform? it''s double the size. > > > > What is considered "normal"? > > > > -L--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Wed, 2008-08-06 at 05:05 -0700, Larry Ludwig wrote:> [snipped useful splay advice] > > Also another recommendation when using storeconfigs is make sure your > MySQL is optimized. (of course if using mysql)Even with an optimized MySQL[*], my own findings shown that: * for a 472 resources node, about 2900 SQL requests are sent to MySQL (I still have to find why it tries to recreate the whole resources even when nothing had changed (or maybe I have something wrong in my environment)). * when resources/params are modified, it seems puppet deletes them and inserts the modified version instead of using any kind of SQL update (or active record save). This alone hurts the most (especially since there are lots of parameters per resources). * on mysql, my storeconfig had duplicated index after puppet created the schema. Each table had a primary key _and_ a regular index containing the primary key column. This also hurts writing. On my system, about 80% of the compile time is storeconfig...which I need for exported/collected resources. I didn''t had time yet to exactly look what happens, but as soon as I find some free spare time, I''m going to try to fix the second point (please tell me if I''m wrong about that). I''ll also check the index to see if we can have clever covering indexes for some of the (most used) queries active-record generates. [*]: mysql should be optimized for a _write_ workload, which is usually a bit different than for mostly-read/OLTP workload. If you, like me, use your mysql server for other critical things, you can''t apply the various optimization that will give you most gains in write workload (ie innodb_flush_log_at_trx_commit=2 or things like that). -- Brice Figureau <brice-puppet@daysofwonder.com> --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Wed, Aug 6, 2008 at 9:09 PM, Brice Figureau < brice-puppet@daysofwonder.com> wrote:> > On my system, about 80% of the compile time is storeconfig...which I > need for exported/collected resourcesThat, and the reason that we have multiple puppet masters in different geographical locations, is the main reason why I''m not using store config. I end up using or own internal custom function to do the same as puppet does, but in a controlled manner (e.g. import all facts to db, query for whatever...). Ohad --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
> (I still have to find why it tries to recreate the whole resources even > when nothing had changed (or maybe I have something wrong in my > environment)).Yes I get that issue too.. Aug 6 09:44:59 puppetmaster puppetmasterd[3980]: Expiring the node cache of node.empoweringmedia.net Aug 6 09:44:59 puppetmaster puppetmasterd[3980]: Not using expired node for node.empoweringmedia.net from cache; expired at Wed Aug 06 09:43:59 -0400 2008 Even if I rerun puppetd just a few min later --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
> I''ve noticed that if the local side is full of a deep file hierarchy > puppet will build a huge hash/array that holds all the local filesIt doesn''t have to be that deep, even. :) I have a shallow hierarchy with about 15,500 files in it. When I removed the file{} from that directory (which was just confirming ownership), memory usage went down dramatically -- the difference is about 300 Megs resident (!!). Now a fresh puppetd is only 278/187M instead of 550M new -- of course the one i killed yesterday had (again) grown to 1.2 Gb resident. I still think it''s leaking memory somewhere. d. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Isn''t --splay a cfengine thing? To my knowledge, none of the puppet binaries have such an option.> > We just added --splay true option to our puppetd and made a world of > difference on the puppetmaster load.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
windowsrefund wrote:> Isn''t --splay a cfengine thing? To my knowledge, none of the puppet > binaries have such an option. >That''s not correct. The --splay option does exist please see: http://reductivelabs.com/trac/puppet/wiki/ConfigurationReference Regards James Turnbull -- Author of: * Pulling Strings with Puppet (http://www.amazon.com/gp/product/1590599780/) * Pro Nagios 2.0 (http://www.amazon.com/gp/product/1590596099/) * Hardening Linux (http://www.amazon.com/gp/product/1590594444/)