Few of my puppet clients (puppet 0.24.8, ruby 1.8.5, facter 1.5.2) are dying with out any error messages on on syslog. I cannot see much system resource usage on these nodes. What is the best way to troubleshoot the root cause of the problem? -Basil -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Christopher Wood
2011-Sep-02 15:41 UTC
Re: [Puppet Users] Puppet client dying without errors
On Thu, Sep 01, 2011 at 08:31:55PM -0400, Basil Baby wrote:> Few of my puppet clients (puppet 0.24.8,� ruby 1.8.5, facter 1.5.2) are > dying with out any error messages on on syslog.� I cannot see much system > resource usage on these nodes. > > What is the best way to troubleshoot the root cause of the problem?Without having huge scads of puppet knowledge, if they die frequently enough perhaps running one of these in the foreground with debug/tee to see if they spit out any interesting messages? See "puppet agent -h" for your exact command line parameters, but something like: puppet agent --no-daemonize --debug 2>&1 | tee /var/tmp/p1.log> -Basil > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
From the debug mode logs, ruby was having trouble with LDAP. Debug Error message: ruby: ../../../libraries/libldap/error.c:273: ldap_parse_result: Assertion `r != ((void *)0)'' failed Is there way I can exclude puppet not to look up LDAP to avoid this situation? I did try adding following entry in /etc/ldap.conf but didn''t help. "nss_initgroups_ignoreusers root" -Basil On Fri, Sep 2, 2011 at 11:41 AM, Christopher Wood < christopher_wood@pobox.com> wrote:> On Thu, Sep 01, 2011 at 08:31:55PM -0400, Basil Baby wrote: > > Few of my puppet clients (puppet 0.24.8,� ruby 1.8.5, facter 1.5.2) > are > > dying with out any error messages on on syslog.� I cannot see much > system > > resource usage on these nodes. > > > > What is the best way to troubleshoot the root cause of the problem? > > Without having huge scads of puppet knowledge, if they die frequently > enough perhaps running one of these in the foreground with debug/tee to see > if they spit out any interesting messages? > > See "puppet agent -h" for your exact command line parameters, but something > like: > > puppet agent --no-daemonize --debug 2>&1 | tee /var/tmp/p1.log > > > > -Basil > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to > > puppet-users+unsubscribe@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/puppet-users?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
This looks much like the error we had, which was: ruby: ../../../libraries/libldap/result.c:113: ldap_result: Assertion `ld !((void *)0)'' failed. Under CentOS/RHEL, you need to upgrade the version of nss_ldap to 253-37.el5_6.1 (or later) Bug details: https://bugzilla.redhat.com/show_bug.cgi?id=684889 and https://bugzilla.redhat.com/show_bug.cgi?id=703831 Bug fix advisory: http://rhn.redhat.com/errata/RHBA-2011-0514.html Note that things other than puppet will be affected by this problem, such as ssh connections! BTW, your nss_initgroups_ignoreusers addition looks correct. On 23 September 2011 07:53, Basil Baby <basilbaby@gmail.com> wrote:> From the debug mode logs, ruby was having trouble with LDAP. > Debug Error message: > ruby: ../../../libraries/libldap/error.c:273: ldap_parse_result: Assertion `r != ((void *)0)'' failed > > > Is there way I can exclude puppet not to look up LDAP to avoid this situation? I did try adding following entry in /etc/ldap.conf but didn''t help. > "nss_initgroups_ignoreusers root" > > -Basil > > On Fri, Sep 2, 2011 at 11:41 AM, Christopher Wood < > christopher_wood@pobox.com> wrote: > >> On Thu, Sep 01, 2011 at 08:31:55PM -0400, Basil Baby wrote: >> > Few of my puppet clients (puppet 0.24.8,� ruby 1.8.5, facter 1.5.2) >> are >> > dying with out any error messages on on syslog.� I cannot see much >> system >> > resource usage on these nodes. >> > >> > What is the best way to troubleshoot the root cause of the problem? >> >> Without having huge scads of puppet knowledge, if they die frequently >> enough perhaps running one of these in the foreground with debug/tee to see >> if they spit out any interesting messages? >> >> See "puppet agent -h" for your exact command line parameters, but >> something like: >> >> puppet agent --no-daemonize --debug 2>&1 | tee /var/tmp/p1.log >> >> >> > -Basil >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Puppet Users" group. >> > To post to this group, send email to puppet-users@googlegroups.com. >> > To unsubscribe from this group, send email to >> > puppet-users+unsubscribe@googlegroups.com. >> > For more options, visit this group at >> > http://groups.google.com/group/puppet-users?hl=en. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Thanks for the help! As per RedHat release note, the updates on the package is promising. Updated nss_ldap on couple hosts to see whether it fixes the problem or not. -Basil On Thu, Sep 22, 2011 at 7:08 PM, Iain Sutton <iainsutton@gmail.com> wrote:> This looks much like the error we had, which was: > > ruby: ../../../libraries/libldap/result.c:113: ldap_result: Assertion `ld > != ((void *)0)'' failed. > > Under CentOS/RHEL, you need to upgrade the version of nss_ldap to > 253-37.el5_6.1 (or later) > Bug details: https://bugzilla.redhat.com/show_bug.cgi?id=684889 and > https://bugzilla.redhat.com/show_bug.cgi?id=703831 > > Bug fix advisory: > http://rhn.redhat.com/errata/RHBA-2011-0514.html > Note that things other than puppet will be affected by this problem, such > as ssh connections! > > BTW, your nss_initgroups_ignoreusers addition looks correct. > On 23 September 2011 07:53, Basil Baby <basilbaby@gmail.com> wrote: > >> From the debug mode logs, ruby was having trouble with LDAP. >> >> >> Debug Error message: >> ruby: ../../../libraries/libldap/error.c:273: ldap_parse_result: Assertion `r != ((void *)0)'' failed >> >> >> Is there way I can exclude puppet not to look up LDAP to avoid this situation? I did try adding following entry in /etc/ldap.conf but didn''t help. >> "nss_initgroups_ignoreusers root" >> >> >> >> -Basil >> >> On Fri, Sep 2, 2011 at 11:41 AM, Christopher Wood < >> christopher_wood@pobox.com> wrote: >> >>> On Thu, Sep 01, 2011 at 08:31:55PM -0400, Basil Baby wrote: >>> > Few of my puppet clients (puppet 0.24.8,� ruby 1.8.5, facter 1.5.2) >>> are >>> > dying with out any error messages on on syslog.� I cannot see much >>> system >>> > resource usage on these nodes. >>> > >>> > What is the best way to troubleshoot the root cause of the problem? >>> >>> Without having huge scads of puppet knowledge, if they die frequently >>> enough perhaps running one of these in the foreground with debug/tee to see >>> if they spit out any interesting messages? >>> >>> See "puppet agent -h" for your exact command line parameters, but >>> something like: >>> >>> puppet agent --no-daemonize --debug 2>&1 | tee /var/tmp/p1.log >>> >>> >>> > -Basil >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups >>> > "Puppet Users" group. >>> > To post to this group, send email to puppet-users@googlegroups.com. >>> > To unsubscribe from this group, send email to >>> > puppet-users+unsubscribe@googlegroups.com. >>> > For more options, visit this group at >>> > http://groups.google.com/group/puppet-users?hl=en. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Puppet Users" group. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> To unsubscribe from this group, send email to >>> puppet-users+unsubscribe@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/puppet-users?hl=en. >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.