Jo Rhett
2012-Aug-18 00:50 UTC
[Puppet Users] don''t push out facter-1.6.11 without testing ; causes puppetd hang
At least on CentOS 5 and CentOS 6, after upgrading to facter 1.6.11 our hosts stopped checking in. Stale puppetdlock problem. -- Jo Rhett Net Consonance : net philanthropy to improve open source and internet projects. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Stuart Cracraft
2012-Aug-18 00:55 UTC
Re: [Puppet Users] don''t push out facter-1.6.11 without testing ; causes puppetd hang
Can you kick them somehow? On Aug 17, 2012, at 5:50 PM, Jo Rhett <jrhett@netconsonance.com> wrote:> At least on CentOS 5 and CentOS 6, after upgrading to facter 1.6.11 our hosts stopped checking in. Stale puppetdlock problem. > > -- > Jo Rhett > Net Consonance : net philanthropy to improve open source and internet projects. > > > > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jo Rhett
2012-Aug-20 20:02 UTC
Re: [Puppet Users] don''t push out facter-1.6.11 without testing ; causes puppetd hang
Nope, they think they are running. We had to reset the policy to downgrade facter, then login to each host and "service puppet stop ; puppet agent --test --ignoreschedules" (our systems are set to only upgrade packages in certain hours) to get the hosts back online. This looks similar to the old problem with a kernel that changed proc semantics, but it''s not the kernel this time. Reverting facter to 1.6.10 resolves the issue. On Aug 17, 2012, at 5:55 PM, Stuart Cracraft wrote:> Can you kick them somehow? > > On Aug 17, 2012, at 5:50 PM, Jo Rhett <jrhett@netconsonance.com> wrote: > >> At least on CentOS 5 and CentOS 6, after upgrading to facter 1.6.11 our hosts stopped checking in. Stale puppetdlock problem. >> >> -- >> Jo Rhett >> Net Consonance : net philanthropy to improve open source and internet projects. >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.-- Jo Rhett Net Consonance : net philanthropy to improve open source and internet projects. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Justin Stoller
2012-Aug-20 20:38 UTC
Re: [Puppet Users] don''t push out facter-1.6.11 without testing ; causes puppetd hang
Hey Jo, Do you have any debugging information about this issue? Stack traces, systems, versions, ruby, custom facts, etc would all be helpful. It certainly seems from what you said that the version of Facter had something to do with this error, but I''m not exactly sure how Facter would affect Puppet''s lock file.... - Justin On Mon, Aug 20, 2012 at 1:02 PM, Jo Rhett <jrhett@netconsonance.com> wrote:> Nope, they think they are running. We had to reset the policy to downgrade > facter, then login to each host and "service puppet stop ; puppet agent > --test --ignoreschedules" (our systems are set to only upgrade packages in > certain hours) to get the hosts back online. > > This looks similar to the old problem with a kernel that changed proc > semantics, but it''s not the kernel this time. Reverting facter to 1.6.10 > resolves the issue. > > On Aug 17, 2012, at 5:55 PM, Stuart Cracraft wrote: > > Can you kick them somehow? > > On Aug 17, 2012, at 5:50 PM, Jo Rhett <jrhett@netconsonance.com> wrote: > > At least on CentOS 5 and CentOS 6, after upgrading to facter 1.6.11 our > hosts stopped checking in. Stale puppetdlock problem. > > -- > Jo Rhett > Net Consonance : net philanthropy to improve open source and internet > projects. > > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > > -- > Jo Rhett > Net Consonance : net philanthropy to improve open source and internet > projects. > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jo Rhett
2012-Aug-20 21:58 UTC
Re: [Puppet Users] don''t push out facter-1.6.11 without testing ; causes puppetd hang
Lots of people have logged information in ticket #10418 http://projects.puppetlabs.com/issues/10418 I don''t have much time to track this today, but our environment is fairly bone-stock CentOS 5.6, Ruby 1.8.7, Puppet 1.7.18. The only custom facts come from puppetlabs-stdlib. On Aug 20, 2012, at 1:38 PM, Justin Stoller wrote:> Do you have any debugging information about this issue? Stack traces, > systems, versions, ruby, custom facts, etc would all be helpful. It > certainly seems from what you said that the version of Facter had > something to do with this error, but I''m not exactly sure how Facter > would affect Puppet''s lock file.... > > - Justin > > On Mon, Aug 20, 2012 at 1:02 PM, Jo Rhett <jrhett@netconsonance.com> wrote: >> Nope, they think they are running. We had to reset the policy to downgrade >> facter, then login to each host and "service puppet stop ; puppet agent >> --test --ignoreschedules" (our systems are set to only upgrade packages in >> certain hours) to get the hosts back online. >> >> This looks similar to the old problem with a kernel that changed proc >> semantics, but it''s not the kernel this time. Reverting facter to 1.6.10 >> resolves the issue. >> >> On Aug 17, 2012, at 5:55 PM, Stuart Cracraft wrote: >> >> Can you kick them somehow? >> >> On Aug 17, 2012, at 5:50 PM, Jo Rhett <jrhett@netconsonance.com> wrote: >> >> At least on CentOS 5 and CentOS 6, after upgrading to facter 1.6.11 our >> hosts stopped checking in. Stale puppetdlock problem. >> >> -- >> Jo Rhett >> Net Consonance : net philanthropy to improve open source and internet >> projects. >> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >> >> -- >> Jo Rhett >> Net Consonance : net philanthropy to improve open source and internet >> projects. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- Jo Rhett Net Consonance : net philanthropy to improve open source and internet projects. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Eric Sorenson
2012-Aug-21 17:45 UTC
Re: [Puppet Users] don''t push out facter-1.6.11 without testing ; causes puppetd hang
Ugh, the only code changes between 1.6.10 and 1.6.11 are /proc related changes for selinux detection. https://github.com/puppetlabs/facter/commit/a836eeee764790be14acc276ab039080316f65b2 So this is somehow tickling the buggy select() behaviour. Will spend more time on tracking this down today; could anyone with affected systems please ping me (eric0) on irc? -=Eric On Monday, August 20, 2012 2:58:16 PM UTC-7, Jo wrote:> > Lots of people have logged information in ticket #10418 > http://projects.puppetlabs.com/issues/10418 > > I don''t have much time to track this today, but our environment is fairly > bone-stock CentOS 5.6, Ruby 1.8.7, Puppet 1.7.18. The only custom facts > come from puppetlabs-stdlib. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/hhgbXIJkzqkJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.