Well, this doesn''t look good... Nov 14 21:42:09 gumby puppetd[4358]: Could not retrieve catalog: Cannot allocate memory - /bin/cat /proc/uptime Is puppetd out of memory??? How do I increase? Actually, I just noticed that puppetd was using approx 1.5Gb of memory before it barfed. Nice... Doug --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi Douglas, Douglas Garstang wrote:> Actually, I just noticed that puppetd was using approx 1.5Gb of memory > before it barfed. Nice...If you''re running on RHEL/CentOS, there is a known issue with the Ruby that ships with that distribution that causes memory leaks. I have a cronjob that restarts puppetd each night to ensure that it doesn''t use too much memory. An alternative is to not run puppetd as a service, but rather as a cronjob with the --no-daemonize and --onetime command-line options. That way, puppetd never remains resident. Hope that helps, Avi --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Avi, I doubt that is my problem. I''m running this in a test environment, and I''m doing a full ''service puppet restart'' every single time. I''ve seen this memory problem occur on the very first run of puppet after the restart. I read about the ruby memory leak, but I don''t see where I can get a ruby 1.8.7 RPM from. :( Doug On Sun, Nov 15, 2009 at 12:45 PM, Avi Miller <avi.miller@gmail.com> wrote:> > Hi Douglas, > > Douglas Garstang wrote: >> Actually, I just noticed that puppetd was using approx 1.5Gb of memory >> before it barfed. Nice... > > If you''re running on RHEL/CentOS, there is a known issue with the Ruby > that ships with that distribution that causes memory leaks. I have a > cronjob that restarts puppetd each night to ensure that it doesn''t use > too much memory. > > An alternative is to not run puppetd as a service, but rather as a > cronjob with the --no-daemonize and --onetime command-line options. That > way, puppetd never remains resident. > > Hope that helps, > Avi > > > >-- Regards, Douglas Garstang http://www.linkedin.com/in/garstang Email: doug.garstang@gmail.com Cell: +1-805-340-5627 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi Douglas, First, can you bottom post? It makes way easier to read (long) threads. On 15/11/09 22:50, Douglas Garstang wrote:> I doubt that is my problem. I''m running this in a test environment, > and I''m doing a full ''service puppet restart'' every single time. I''ve > seen this memory problem occur on the very first run of puppet after > the restart. > > I read about the ruby memory leak, but I don''t see where I can get a > ruby 1.8.7 RPM from. :(If you''re just on a test system and you want to rule out ruby as the culprit of the memory issue, I think you can just either compile your own 1.8.7 locally (ie you don''t need to install ruby in your system paths), or even use the ruby enterprise edition installer which setups most of the thing for you easily. Once done, you''ll know if the problem is puppet or ruby itself. And if it turns out to be puppet, please file a bug report with the more information you can, including manifest, versions...> On Sun, Nov 15, 2009 at 12:45 PM, Avi Miller<avi.miller@gmail.com> wrote: >> >> Hi Douglas, >> >> Douglas Garstang wrote: >>> Actually, I just noticed that puppetd was using approx 1.5Gb of memory >>> before it barfed. Nice... >> >> If you''re running on RHEL/CentOS, there is a known issue with the Ruby >> that ships with that distribution that causes memory leaks. I have a >> cronjob that restarts puppetd each night to ensure that it doesn''t use >> too much memory. >> >> An alternative is to not run puppetd as a service, but rather as a >> cronjob with the --no-daemonize and --onetime command-line options. That >> way, puppetd never remains resident. >> >> Hope that helps, >> Avi-- Brice Figureau My Blog: http://www.masterzen.fr/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Sun, Nov 15, 2009 at 2:10 PM, Brice Figureau <brice-puppet@daysofwonder.com> wrote:> > Hi Douglas, > > First, can you bottom post? > It makes way easier to read (long) threads. > > On 15/11/09 22:50, Douglas Garstang wrote: >> I doubt that is my problem. I''m running this in a test environment, >> and I''m doing a full ''service puppet restart'' every single time. I''ve >> seen this memory problem occur on the very first run of puppet after >> the restart. >> >> I read about the ruby memory leak, but I don''t see where I can get a >> ruby 1.8.7 RPM from. :( > > If you''re just on a test system and you want to rule out ruby as the > culprit of the memory issue, I think you can just either compile your > own 1.8.7 locally (ie you don''t need to install ruby in your system > paths), or even use the ruby enterprise edition installer which setups > most of the thing for you easily. > Once done, you''ll know if the problem is puppet or ruby itself. > And if it turns out to be puppet, please file a bug report with the more > information you can, including manifest, versions... >So, I downloaded, compiled, and installed ruby 1.9. --- SAME PROBLEM --- Ruby/Puppet uses up approx 1.4Gb of memory before I run out. Doug>> On Sun, Nov 15, 2009 at 12:45 PM, Avi Miller<avi.miller@gmail.com> wrote: >>> >>> Hi Douglas, >>> >>> Douglas Garstang wrote: >>>> Actually, I just noticed that puppetd was using approx 1.5Gb of memory >>>> before it barfed. Nice... >>> >>> If you''re running on RHEL/CentOS, there is a known issue with the Ruby >>> that ships with that distribution that causes memory leaks. I have a >>> cronjob that restarts puppetd each night to ensure that it doesn''t use >>> too much memory. >>> >>> An alternative is to not run puppetd as a service, but rather as a >>> cronjob with the --no-daemonize and --onetime command-line options. That >>> way, puppetd never remains resident. >>> >>> Hope that helps, >>> Avi > > > -- > Brice Figureau > My Blog: http://www.masterzen.fr/ > > > > >-- Regards, Douglas Garstang http://www.linkedin.com/in/garstang Email: doug.garstang@gmail.com Cell: +1-805-340-5627 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On 16/11/09 20:55, Douglas Garstang wrote:> > On Sun, Nov 15, 2009 at 2:10 PM, Brice Figureau > <brice-puppet@daysofwonder.com> wrote: >> >> Hi Douglas, >> >> First, can you bottom post? >> It makes way easier to read (long) threads. >> >> On 15/11/09 22:50, Douglas Garstang wrote: >>> I doubt that is my problem. I''m running this in a test environment, >>> and I''m doing a full ''service puppet restart'' every single time. I''ve >>> seen this memory problem occur on the very first run of puppet after >>> the restart. >>> >>> I read about the ruby memory leak, but I don''t see where I can get a >>> ruby 1.8.7 RPM from. :( >> >> If you''re just on a test system and you want to rule out ruby as the >> culprit of the memory issue, I think you can just either compile your >> own 1.8.7 locally (ie you don''t need to install ruby in your system >> paths), or even use the ruby enterprise edition installer which setups >> most of the thing for you easily. >> Once done, you''ll know if the problem is puppet or ruby itself. >> And if it turns out to be puppet, please file a bug report with the more >> information you can, including manifest, versions... >> > > So, I downloaded, compiled, and installed ruby 1.9.Note I suggested 1.8.7. I don''t have any experience with ruby 1.9.> --- SAME PROBLEM --- > > Ruby/Puppet uses up approx 1.4Gb of memory before I run out.It would be interesting to know what are doing your manifests, and more specifically how many resources, and how many recursive file resources. Warning what follows is not for the faint-hearted: Also, it would be interesting to introspect the ruby runtime (sorry 1.8.x only), like what is explained here: http://eigenclass.org/hiki/ruby+live+process+introspection Then after each run you can try to print the objectspace. Another possibility is to use ruby-prof in memory allocation mode: http://ruby-prof.rubyforge.org/ and http://rubyforge.org/tracker/index.php?func=detail&aid=17676&group_id=1814&atid=7062 -- Brice Figureau My Blog: http://www.masterzen.fr/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
does puppet run at all? e.g. do you see anything on the console? if not, it might be an old facter 1.54 bug? if your puppetd process is using 1.5GB or ram, something is wrong or you are doing way to many file transfers (probably recursive ones). cheers, Ohad On Mon, Nov 16, 2009 at 3:26 AM, Douglas Garstang <doug.garstang@gmail.com>wrote:> > Well, this doesn''t look good... > > Nov 14 21:42:09 gumby puppetd[4358]: Could not retrieve catalog: > Cannot allocate memory - /bin/cat /proc/uptime > > Is puppetd out of memory??? How do I increase? > > Actually, I just noticed that puppetd was using approx 1.5Gb of memory > before it barfed. Nice... > > Doug > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Ohad Levy <ohadlevy@gmail.com> writes:> does puppet run at all? e.g. do you see anything on the console? > if not, it might be an old facter 1.54 bug? > > if your puppetd process is using 1.5GB or ram, something is wrong or you are > doing way to many file transfers (probably recursive ones).We''ve also seen tidy use lots of RAM when there are tons of files to be tidied. This was with 0.24.x; I haven''t tested with 0.25.x yet. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi Douglas, We are currently working on a patch related to memory consumption, and would appreciate some information to see if this patch may resolve this issue. Can I get a little more information about the characteristics that are leading to this issue? Is this just for a few machines or all? Is this re creatable? Does it consistently happen every run for one machine, or set of manifests? do you have a lot of file resources? Are you using recurse=true? Do you know what the total size for all files managed in this catalog is? Is there anything special about this catalog? How many resources total would you say are managed? regards, Dan -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=.