Hi, 3 days ago, we had a production issue where the puppetmaster became unreachable for 20 minute. All of the puppet clients which tried to connect at this time are now failing : "Run of Puppet configuration client already in progress; skipping" Further investigation show that the puppet agent process is still running on all of this clients 3 days later. I tried to kille the process on a machine, and puppet service came back to normal for this machine. Is there a way to kill the process on all of the client ? Is there a way to specify a timeout on the process ? This would permit to discard the process without human action Regards Yannig Configuration : - Puppet master => RHEL5, Puppet 3.2.2 - Puppet client => RHEL5, Puppet 3.1.1 & 3.2.2 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
On Wednesday, July 17, 2013 12:00:13 PM UTC-5, yannig rousseau wrote:> > Hi, > > 3 days ago, we had a production issue where the puppetmaster became > unreachable for 20 minute. > All of the puppet clients which tried to connect at this time are now > failing : "Run of Puppet configuration client already in progress; > skipping" > > Further investigation show that the puppet agent process is still running > on all of this clients 3 days later. I tried to kille the process on a > machine, and puppet service came back to normal for this machine. > > Is there a way to kill the process on all of the client ? >Puppet does not provide a built-in mechanism for this. There is a variety of tools that could do it, but you would have to had already set them up.> Is there a way to specify a timeout on the process ? This would permit to > discard the process without human action >Again, Puppet does not provide this as a built-in feature, but it should be possible to add external instrumentation to make it happen. That would probably be facilitated by launching the agent periodically via a scheduler (such as cron) instead of running it in daemon mode. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
I managed to kill all of the stuck process with a simple ssh loop in a script. Nevertheless, that''s not very cool to proceed so... Don''t you think it would be a good option to implement a timeout in the puppet agent ? If the connection / catalog application is too long (this treshold could be set with a specific argument), the puppet agent discard the action and request the puppet master again... Or is it too complicated to implement safely ? Yannig On Thursday, July 18, 2013 3:11:46 PM UTC+2, jcbollinger wrote:> > > > On Wednesday, July 17, 2013 12:00:13 PM UTC-5, yannig rousseau wrote: >> >> Hi, >> >> 3 days ago, we had a production issue where the puppetmaster became >> unreachable for 20 minute. >> All of the puppet clients which tried to connect at this time are now >> failing : "Run of Puppet configuration client already in progress; >> skipping" >> >> Further investigation show that the puppet agent process is still running >> on all of this clients 3 days later. I tried to kille the process on a >> machine, and puppet service came back to normal for this machine. >> >> Is there a way to kill the process on all of the client ? >> > > > Puppet does not provide a built-in mechanism for this. There is a variety > of tools that could do it, but you would have to had already set them up. > > > >> Is there a way to specify a timeout on the process ? This would permit to >> discard the process without human action >> > > > Again, Puppet does not provide this as a built-in feature, but it should > be possible to add external instrumentation to make it happen. That would > probably be facilitated by launching the agent periodically via a scheduler > (such as cron) instead of running it in daemon mode. > > > John > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.