We''ve been having a huge number of problems with puppetmaster seizing up on CentOS 4. We started with 0.23.2 and it worked, but the clients were leaking memory and we hoped upgrading to 0.24.5 would help. Since then we''ve had far worse problems, primarily that puppetmaster gets into a state where it simply stops doing anything and any already connected client similarly freezes. Part of the troubleshooting involved upgrading ruby, first to a hand-built RPM, then to the RubyWorks RPMs, but nothing really improved. I finally decided to revert to 0.23.2 and we''re still seeing the lockups. I believe replacing ruby with the original OS RPM might get us back, but I am also looking at building a dedicated VM. Since I have the option to install any OS I want, I was hoping folks might have some suggestions on which have been found to work best for running puppetmaster in terms of just making sure all the parts work properly together as well as simplifying implementation of the scalability options (this was painful on CentOS 4). I''m leaning toward Debian Etch, but I''m open to anything at this point -- suggestions welcome! Thanks, Mark --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Mark D. Nagel wrote:> We''ve been having a huge number of problems with puppetmaster seizing up > on CentOS 4. We started with 0.23.2 and it worked, but the clients were > leaking memory and we hoped upgrading to 0.24.5 would help. Since then > we''ve had far worse problems, primarily that puppetmaster gets into a > state where it simply stops doing anything and any already connected > client similarly freezes. Part of the troubleshooting involved > upgrading ruby, first to a hand-built RPM, then to the RubyWorks RPMs, > but nothing really improved. I finally decided to revert to 0.23.2 and > we''re still seeing the lockups. I believe replacing ruby with the > original OS RPM might get us back, but I am also looking at building a > dedicated VM. Since I have the option to install any OS I want, I was > hoping folks might have some suggestions on which have been found to > work best for running puppetmaster in terms of just making sure all the > parts work properly together as well as simplifying implementation of > the scalability options (this was painful on CentOS 4). I''m leaning > toward Debian Etch, but I''m open to anything at this point -- > suggestions welcome! >I use Debian Etch and don''t have any performance issues but I only have 25 hosts under management (mostly test VMs) and don''t hammer the master as much as others do. I should point out that I also run 0.24.x HEAD rather than a specific release. I would recommend using 0.24.6 though as a production platform. Regards James Turnbull -- Author of: * Pulling Strings with Puppet (http://www.amazon.com/gp/product/1590599780/) * Pro Nagios 2.0 (http://www.amazon.com/gp/product/1590596099/) * Hardening Linux (http://www.amazon.com/gp/product/1590594444/) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Mark, Are you using webrick or mongrel? How many clients do you have? Andrew On Tue, Nov 11, 2008 at 9:22 PM, James Turnbull <james@lovedthanlost.net>wrote:> > Mark D. Nagel wrote: > > We''ve been having a huge number of problems with puppetmaster seizing up > > on CentOS 4. We started with 0.23.2 and it worked, but the clients were > > leaking memory and we hoped upgrading to 0.24.5 would help. Since then > > we''ve had far worse problems, primarily that puppetmaster gets into a > > state where it simply stops doing anything and any already connected > > client similarly freezes. Part of the troubleshooting involved > > upgrading ruby, first to a hand-built RPM, then to the RubyWorks RPMs, > > but nothing really improved. I finally decided to revert to 0.23.2 and > > we''re still seeing the lockups. I believe replacing ruby with the > > original OS RPM might get us back, but I am also looking at building a > > dedicated VM. Since I have the option to install any OS I want, I was > > hoping folks might have some suggestions on which have been found to > > work best for running puppetmaster in terms of just making sure all the > > parts work properly together as well as simplifying implementation of > > the scalability options (this was painful on CentOS 4). I''m leaning > > toward Debian Etch, but I''m open to anything at this point -- > > suggestions welcome! > > > I use Debian Etch and don''t have any performance issues but I only have > 25 hosts under management (mostly test VMs) and don''t hammer the master > as much as others do. I should point out that I also run 0.24.x HEAD > rather than a specific release. I would recommend using 0.24.6 though > as a production platform. > > Regards > > James Turnbull > > -- > Author of: > * Pulling Strings with Puppet > (http://www.amazon.com/gp/product/1590599780/) > * Pro Nagios 2.0 > (http://www.amazon.com/gp/product/1590596099/) > * Hardening Linux > (http://www.amazon.com/gp/product/1590594444/) > > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I am using debian etch and have 4 puppetmasters handling 3000+ clients with no problems (hourly checkins). cheers, ski On Tue, 2008-11-11 at 23:08 -0700, Andrew Shafer wrote:> Mark, > > Are you using webrick or mongrel? > > How many clients do you have? > > Andrew > > > On Tue, Nov 11, 2008 at 9:22 PM, James Turnbull > <james@lovedthanlost.net> wrote: > > Mark D. Nagel wrote: > > We''ve been having a huge number of problems with > puppetmaster seizing up > > on CentOS 4. We started with 0.23.2 and it worked, but the > clients were > > leaking memory and we hoped upgrading to 0.24.5 would help. > Since then > > we''ve had far worse problems, primarily that puppetmaster > gets into a > > state where it simply stops doing anything and any already > connected > > client similarly freezes. Part of the troubleshooting > involved > > upgrading ruby, first to a hand-built RPM, then to the > RubyWorks RPMs, > > but nothing really improved. I finally decided to revert to > 0.23.2 and > > we''re still seeing the lockups. I believe replacing ruby > with the > > original OS RPM might get us back, but I am also looking at > building a > > dedicated VM. Since I have the option to install any OS I > want, I was > > hoping folks might have some suggestions on which have been > found to > > work best for running puppetmaster in terms of just making > sure all the > > parts work properly together as well as simplifying > implementation of > > the scalability options (this was painful on CentOS 4). I''m > leaning > > toward Debian Etch, but I''m open to anything at this point > -- > > suggestions welcome! > > > > I use Debian Etch and don''t have any performance issues but I > only have > 25 hosts under management (mostly test VMs) and don''t hammer > the master > as much as others do. I should point out that I also run > 0.24.x HEAD > rather than a specific release. I would recommend using > 0.24.6 though > as a production platform. > > Regards > > James Turnbull > > -- > Author of: > * Pulling Strings with Puppet > (http://www.amazon.com/gp/product/1590599780/) > * Pro Nagios 2.0 > (http://www.amazon.com/gp/product/1590596099/) > * Hardening Linux > (http://www.amazon.com/gp/product/1590594444/) > > > > > >-- "When we try to pick out anything by itself, we find it connected to the entire universe" John Muir Chris "Ski" Kacoroski, kacoroski@gmail.com, 206-501-9803 or ski98033 on most IM services --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Using Mongrel and monit to keep the memory leak in check works quite well. I use RHE5, and if your clients are RH I would not suggest to use 0.24-6 because of the selinux slow down (use 24-4 instead). Ohad On Wed, Nov 12, 2008 at 12:22 PM, James Turnbull <james@lovedthanlost.net>wrote:> > Mark D. Nagel wrote: > > We''ve been having a huge number of problems with puppetmaster seizing up > > on CentOS 4. We started with 0.23.2 and it worked, but the clients were > > leaking memory and we hoped upgrading to 0.24.5 would help. Since then > > we''ve had far worse problems, primarily that puppetmaster gets into a > > state where it simply stops doing anything and any already connected > > client similarly freezes. Part of the troubleshooting involved > > upgrading ruby, first to a hand-built RPM, then to the RubyWorks RPMs, > > but nothing really improved. I finally decided to revert to 0.23.2 and > > we''re still seeing the lockups. I believe replacing ruby with the > > original OS RPM might get us back, but I am also looking at building a > > dedicated VM. Since I have the option to install any OS I want, I was > > hoping folks might have some suggestions on which have been found to > > work best for running puppetmaster in terms of just making sure all the > > parts work properly together as well as simplifying implementation of > > the scalability options (this was painful on CentOS 4). I''m leaning > > toward Debian Etch, but I''m open to anything at this point -- > > suggestions welcome! > > > I use Debian Etch and don''t have any performance issues but I only have > 25 hosts under management (mostly test VMs) and don''t hammer the master > as much as others do. I should point out that I also run 0.24.x HEAD > rather than a specific release. I would recommend using 0.24.6 though > as a production platform. > > Regards > > James Turnbull > > -- > Author of: > * Pulling Strings with Puppet > (http://www.amazon.com/gp/product/1590599780/) > * Pro Nagios 2.0 > (http://www.amazon.com/gp/product/1590596099/) > * Hardening Linux > (http://www.amazon.com/gp/product/1590594444/) > > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
2008/11/12 Mark D. Nagel <mnagel@willingminds.com>> > We''ve been having a huge number of problems with puppetmaster seizing up > on CentOS 4. We started with 0.23.2 and it worked, but the clients were > leaking memory and we hoped upgrading to 0.24.5 would help. Since then > we''ve had far worse problems, primarily that puppetmaster gets into a > state where it simply stops doing anything and any already connected > client similarly freezes. Part of the troubleshooting involved > upgrading ruby, first to a hand-built RPM, then to the RubyWorks RPMs, > but nothing really improved. I finally decided to revert to 0.23.2 and > we''re still seeing the lockups. I believe replacing ruby with the > original OS RPM might get us back, but I am also looking at building a > dedicated VM. Since I have the option to install any OS I want, I was > hoping folks might have some suggestions on which have been found to > work best for running puppetmaster in terms of just making sure all the > parts work properly together as well as simplifying implementation of > the scalability options (this was painful on CentOS 4). I''m leaning > toward Debian Etch, but I''m open to anything at this point -- > suggestions welcome! > > Thanks, > Mark > > > Debian etch here with 16 real hosts, and 48 Xen hosts, No problems as yet.This is a test env, hopefully puppet will be production soon with apporx 250 hosts. so we''ll see how that go''s :) --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Ohad Levy wrote:> Using Mongrel and monit to keep the memory leak in check works quite well. > I use RHE5, and if your clients are RH I would not suggest to use 0.24-6 > because of the selinux slow down (use 24-4 instead).The SELinux support is being refactored in 0.24.7 BTW. That will remove these issues. The 0.24.7 release is probably 2 weeks away at this stage. Regards James Turnbull -- Author of: * Pulling Strings with Puppet (http://www.amazon.com/gp/product/1590599780/) * Pro Nagios 2.0 (http://www.amazon.com/gp/product/1590596099/) * Hardening Linux (http://www.amazon.com/gp/product/1590594444/)
Mark D. Nagel wrote:> We''ve been having a huge number of problems with puppetmaster seizing up > on CentOS 4. We started with 0.23.2 and it worked, but the clients were > leaking memory and we hoped upgrading to 0.24.5 would help. Since then > we''ve had far worse problems, primarily that puppetmaster gets into a > state where it simply stops doing anything and any already connected > client similarly freezes. Part of the troubleshooting involved > upgrading ruby, first to a hand-built RPM, then to the RubyWorks RPMs, > but nothing really improved. I finally decided to revert to 0.23.2 and > we''re still seeing the lockups. I believe replacing ruby with the > original OS RPM might get us back, but I am also looking at building a > dedicated VM. Since I have the option to install any OS I want, I was > hoping folks might have some suggestions on which have been found to > work best for running puppetmaster in terms of just making sure all the > parts work properly together as well as simplifying implementation of > the scalability options (this was painful on CentOS 4). I''m leaning > toward Debian Etch, but I''m open to anything at this point -- > suggestions welcome!Are you by any chance using storeconfigs and mysql? If so, make sure you''re using the mysql gem connector, rather than the very simple one that ships with rails. http://reductivelabs.com/trac/puppet/wiki/UsingStoredConfiguration#mysql -- Frank Sweetser fs at wpi.edu | For every problem, there is a solution that WPI Senior Network Engineer | is simple, elegant, and wrong. - HL Mencken GPG fingerprint = 6174 1257 129E 0D21 D8D4 E8A3 8E39 29E3 E2E8 8CEC --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---