Puppet users - Nov 2011 - How to debug intermittent puppet catalog failure

We''ve recently developed a problem where a puppetd scheduled catalog
run
will hang or fail, then never resume.  The puppetdlock file is present, but
I can''t find any logged errors.  If I stop the daemon, it will restart
and
run the catalog without problem.

Any suggestions on how to identify the problem?

Puppet version 2.7.6 on both client and server and Foreman is managing
nodes.

Thanks

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Also, the timestamp on several clients puppetdlock file will be within
minutes of each other, so it appears to be some sort of infrastructure
issue.  The puppet master is a VM and the backend mysql server physical.

On Thu, Nov 10, 2011 at 8:24 AM, Len Rugen <lenrugen@gmail.com> wrote:
> We''ve recently developed a problem where a puppetd scheduled
catalog run
> will hang or fail, then never resume.  The puppetdlock file is present, but
> I can''t find any logged errors.  If I stop the daemon, it will
restart and
> run the catalog without problem.
>
> Any suggestions on how to identify the problem?
>
> Puppet version 2.7.6 on both client and server and Foreman is managing
> nodes.
>
> Thanks
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

It seems like I have two sets of similar symptoms.  I have another set of
servers where puppetd -tv runs fine, but the daemon will never finish a
catalog run.  Again, puppetdlock lingers.

I think the first problem may track to a database backup.  The clients with
similar timestamps are near that time.  I''ve changed that process and
will
see if that fixes the problem.

On Thu, Nov 10, 2011 at 8:32 AM, Len Rugen <lenrugen@gmail.com> wrote:
> Also, the timestamp on several clients puppetdlock file will be within
> minutes of each other, so it appears to be some sort of infrastructure
> issue.  The puppet master is a VM and the backend mysql server physical.
>
> On Thu, Nov 10, 2011 at 8:24 AM, Len Rugen <lenrugen@gmail.com>
wrote:
>
>> We''ve recently developed a problem where a puppetd scheduled
catalog run
>> will hang or fail, then never resume.  The puppetdlock file is present,
but
>> I can''t find any logged errors.  If I stop the daemon, it will
restart and
>> run the catalog without problem.
>>
>> Any suggestions on how to identify the problem?
>>
>> Puppet version 2.7.6 on both client and server and Foreman is managing
>> nodes.
>>
>> Thanks
>>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Are you running the latest centos or redhat-based kernel?  If so, 274.7.1 is
corked. Backgrade from 274.7.1 to 274.3.1 and the problem will disappear. This
is documented in

https://projects.puppetlabs.com/issues/10418

And RedHat bug 
 https://bugzilla.redhat.com/show_bug.cgi?id=751214

On Nov 10, 2011, at 11:27 AM, Len Rugen wrote:
> It seems like I have two sets of similar symptoms.  I have another set of
servers where puppetd -tv runs fine, but the daemon will never finish a catalog
run.  Again, puppetdlock lingers.
> 
> I think the first problem may track to a database backup.  The clients with
similar timestamps are near that time.  I''ve changed that process and
will see if that fixes the problem.
> 
> On Thu, Nov 10, 2011 at 8:32 AM, Len Rugen <lenrugen@gmail.com>
wrote:
> Also, the timestamp on several clients puppetdlock file will be within
minutes of each other, so it appears to be some sort of infrastructure issue. 
The puppet master is a VM and the backend mysql server physical.
> 
> On Thu, Nov 10, 2011 at 8:24 AM, Len Rugen <lenrugen@gmail.com>
wrote:
> We''ve recently developed a problem where a puppetd scheduled
catalog run will hang or fail, then never resume.  The puppetdlock file is
present, but I can''t find any logged errors.  If I stop the daemon, it
will restart and run the catalog without problem.
> 
> Any suggestions on how to identify the problem?  
> 
> Puppet version 2.7.6 on both client and server and Foreman is managing
nodes.
> 
> Thanks
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
-- 
Jo Rhett
Net Consonance : consonant endings by net philanthropy, open source and other
randomness

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Len,

On Thu, Nov 10, 2011 at 11:27 AM, Len Rugen <lenrugen@gmail.com> wrote:
> It seems like I have two sets of similar symptoms.  I have another set of
> servers where puppetd -tv runs fine, but the daemon will never finish a
> catalog run.  Again, puppetdlock lingers.
>
> I think the first problem may track to a database backup.  The clients
> with similar timestamps are near that time.  I''ve changed that
process and
> will see if that fixes the problem.
>
> On Thu, Nov 10, 2011 at 8:32 AM, Len Rugen <lenrugen@gmail.com>
wrote:
>
>> Also, the timestamp on several clients puppetdlock file will be within
>> minutes of each other, so it appears to be some sort of infrastructure
>> issue.  The puppet master is a VM and the backend mysql server
physical.
>>
>> On Thu, Nov 10, 2011 at 8:24 AM, Len Rugen <lenrugen@gmail.com>
wrote:
>>
>>> We''ve recently developed a problem where a puppetd
scheduled catalog run
>>> will hang or fail, then never resume.  The puppetdlock file is
present, but
>>> I can''t find any logged errors.  If I stop the daemon, it
will restart and
>>> run the catalog without problem.
>>>
>>> Any suggestions on how to identify the problem?
>>>
>>>
I would run truss/strace/dtrace to see what puppet is doing when it''s
"hung".

> Puppet version 2.7.6 on both client and server and Foreman is managing
>> nodes.
>>
>> Thanks
>>
>
We recently fixed a bug[1] that caused puppet to take a long time to run
when managing many files (more than an order of magnitude slower in 2.7
than 2.6). For example, managing a directory recursively with 1k+ files.
The bug is present in 2.7.0-2.7.6, and fixed in 2.7.7rc2, released
yesterday. If the trace shows puppet stat''ing files with long pauses in
between, then this may be part of the problem.

Josh

[1] https://projects.puppetlabs.com/issues/9671

-- 
Josh Cooper
Developer, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Jo Rhett wrote:
> Are you running the latest centos or redhat-based kernel?  If so,
> 274.7.1 is corked. Backgrade from 274.7.1 to 274.3.1 and the problem
> will disappear. This is documented in
>
> https://projects.puppetlabs.com/issues/10418
>
> And RedHat bug
>
> https://bugzilla.redhat.com/show_bug.cgi?id=751214
Thanks for filing that Jo.  I''ve found the same issue on my CentOS
systems.  Can you mark that bug as public or ask the RHEL folks to do
so?  Right now, it''s not accessible to most folks -- myself included.

As one of the EPEL puppet maintainers, I''d very much like to be able
to follow the discussion there.  Perhaps if the bug can''t be made
public, I could get on the Cc?

Thanks,

-- 
Todd        OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If God had meant for us to be naked, we would have been born that way.