We are having a problem with some of the puppet clients dying during the weekly log rotation reload. We are using Centos 4 with puppet 0.24.6. It seems to be a race condition with clients that reload when the puppetmaster is restarting exit. The redhat logrotate script does "/etc/init.d/puppetmaster condrestart" and "/etc/init.d/puppet reload". Reloading puppet when the puppetmaster is stopped doesn''t seem to be a problem. My guess is that there is a period when the puppetmaster is starting when the puppetd fails when it contacts the server. Unfortunately, I don''t see anything in the logs on the clients. Has anyone seen this problem? - Ian --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Andrew Shafer
2009-Feb-04 01:39 UTC
[Puppet Users] Re: Log rotation and puppet clients dying
Ian, When you say they die, do they hang somewhere or the process exits? What is in the logs on the dead reloaded puppets? (also what log level are they running at) Andrew On Tue, Feb 3, 2009 at 12:01 PM, Ian Burrell <ianburrell@gmail.com> wrote:> > We are having a problem with some of the puppet clients dying during > the weekly log rotation reload. We are using Centos 4 with puppet > 0.24.6. It seems to be a race condition with clients that reload when > the puppetmaster is restarting exit. The redhat logrotate script does > "/etc/init.d/puppetmaster condrestart" and "/etc/init.d/puppet > reload". Reloading puppet when the puppetmaster is stopped doesn''t > seem to be a problem. My guess is that there is a period when the > puppetmaster is starting when the puppetd fails when it contacts the > server. Unfortunately, I don''t see anything in the logs on the > clients. > > Has anyone seen this problem? > > - Ian > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Luke Kanies
2009-Feb-05 15:47 UTC
[Puppet Users] Re: Log rotation and puppet clients dying
On Feb 3, 2009, at 1:01 PM, Ian Burrell wrote:> > We are having a problem with some of the puppet clients dying during > the weekly log rotation reload. We are using Centos 4 with puppet > 0.24.6. It seems to be a race condition with clients that reload when > the puppetmaster is restarting exit. The redhat logrotate script does > "/etc/init.d/puppetmaster condrestart" and "/etc/init.d/puppet > reload". Reloading puppet when the puppetmaster is stopped doesn''t > seem to be a problem. My guess is that there is a period when the > puppetmaster is starting when the puppetd fails when it contacts the > server. Unfortunately, I don''t see anything in the logs on the > clients. > > Has anyone seen this problem?Can you get a stack trace from the clients? The last few releases have seen us greatly reduce the number of cases where puppetd can exit on failure, and we might have already resolved the one you''re seeing. I know that either 0.24.6 or 0.24.7 had a problem where if the first connection to the master failed, the client would exit. I hope that was fixed in 0.24.7, because that could mean your problem''s already fixed. :) -- At my lemonade stand I used to give the first glass away free and charge five dollars for the second glass. The refill contained the antidote. -- Emo Phillips --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Ian Burrell
2009-Feb-05 22:04 UTC
[Puppet Users] Re: Log rotation and puppet clients dying
On Thu, Feb 5, 2009 at 7:47 AM, Luke Kanies <luke@madstop.com> wrote:> On Feb 3, 2009, at 1:01 PM, Ian Burrell wrote: >> >> We are having a problem with some of the puppet clients dying during >> the weekly log rotation reload. We are using Centos 4 with puppet >> 0.24.6. It seems to be a race condition with clients that reload when >> the puppetmaster is restarting exit. The redhat logrotate script does >> "/etc/init.d/puppetmaster condrestart" and "/etc/init.d/puppet >> reload". Reloading puppet when the puppetmaster is stopped doesn''t >> seem to be a problem. My guess is that there is a period when the >> puppetmaster is starting when the puppetd fails when it contacts the >> server. Unfortunately, I don''t see anything in the logs on the >> clients. >> >> Has anyone seen this problem? > > Can you get a stack trace from the clients? > > The last few releases have seen us greatly reduce the number of cases > where puppetd can exit on failure, and we might have already resolved > the one you''re seeing. I know that either 0.24.6 or 0.24.7 had a > problem where if the first connection to the master failed, the client > would exit. I hope that was fixed in 0.24.7, because that could mean > your problem''s already fixed. :) >There are no errors from the restarted puppetd. The redhat puppet.logrotate script redirects all the output. There are also no logs in the rotated file but it is likely that puppetd is failing before opening the logs. The last line in the old logs are: Sun Feb 01 04:02:11 -0800 2009 Puppet (notice): Restarting with ''/usr/sbin/puppetd --server=edev2.rtkinternal --logdest=/var/log/puppet/puppet.log --waitforcert=500'' Sun Feb 01 04:02:11 -0800 2009 Puppet (notice): Shutting down I am going to patch the logrotate script to keep all the output and send it through cron email. We are running 0.24.6. I may start upgrading to 0.24.7. - Ian --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---