I''ve got a handful of nodes (3?) out of about 400 that are giving me
grief... puppet will run either manually or in the service mode. However, in the
service mode the puppet process dies after an hour or so. I got an strace of the
failure:
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(8, "", 4096) = 0
rt_sigaction(SIGPIPE, {0x1, [], SA_RESTORER, 0x31af00eb10}, {0x31af88c700, [],
SA_RESTORER|SA_RESTART, 0x31af00eb10}, 8) = 0
write(2, "ruby: ../../../libraries/libldap"..., 98ruby:
../../../libraries/libldap/result.c:113: ldap_result: Assertion `ld != ((void
*)0)'' failed.
) = 98
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
write(4, ":XMLRPC::WEBrickServlet:0x2afa17"..., 495) = 495
tgkill(1267, 1267, SIGABRT) = 0
--- SIGABRT (Aborted) @ 0 (0) ---
+++ killed by SIGABRT (core dumped) +++
[root@bastion100 ~]#
I''m running Puppet 2.6.5 on CentOS 5.5 x86_64... any thoughts?
—Matt
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
On Wed, Mar 16, 2011 at 20:16, Matt Wise <wise@wiredgeek.net> wrote:> I''ve got a handful of nodes (3?) out of about 400 that are giving me grief... puppet will run either manually or in the service mode. However, in the service mode the puppet process dies after an hour or so. I got an strace of the failure: > > rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 > rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 > read(8, "", 4096) = 0 > rt_sigaction(SIGPIPE, {0x1, [], SA_RESTORER, 0x31af00eb10}, {0x31af88c700, [], SA_RESTORER|SA_RESTART, 0x31af00eb10}, 8) = 0 > write(2, "ruby: ../../../libraries/libldap"..., 98ruby: ../../../libraries/libldap/result.c:113: ldap_result: Assertion `ld != ((void *)0)'' failed. > ) = 98 > rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0 > write(4, ":XMLRPC::WEBrickServlet:0x2afa17"..., 495) = 495 > tgkill(1267, 1267, SIGABRT) = 0 > --- SIGABRT (Aborted) @ 0 (0) --- > +++ killed by SIGABRT (core dumped) +++ > [root@bastion100 ~]# > > I''m running Puppet 2.6.5 on CentOS 5.5 x86_64... any thoughts?Something is making libldap unhappy, which is being used by Ruby, used in turn by Puppet. Which triggers the unfortunate error handling behaviour of "abort the entire process now!!!" that I just *love* from libraries I depend on. So, I would go hunting for details about what causes that particular assertion to fire in the LDAP library you are using. My guess would be that there is a bug in there, tickled by some particular data set, that those hosts hit, and that a newer release would probably fix it. Other that turning off LDAP in the Puppet code, there isn''t likely much we can do about it. Daniel -- ⎋ Puppet Labs Developer – http://puppetlabs.com ✉ Daniel Pittman <daniel@puppetlabs.com> ✆ Contact me via gtalk, email, or phone: +1 (877) 575-9775 ♲ Made with 100 percent post-consumer electrons -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ok, for what its worth, I think I solved this a while back.. but I ran into it again the other day, and couldn''t remember the fix. I found this thread again while searching around — so I figure I should update it so that everyone knows what went wrong. We are using not only local files, and LDAP for auth.. but we''re using ''nsscache'' as a backup if the LDAP service is down for any extended period of time. The problem seems to be that occasionally nsscache writes out its DB files in a way that upsets ruby''s ldap library significantly. Disabling nsscache, or rebuilding its DB files from scratch seems to solve this problem. —Matt On Mar 16, 2011, at 8:46 PM, Daniel Pittman wrote:> On Wed, Mar 16, 2011 at 20:16, Matt Wise <wise@wiredgeek.net> wrote: > >> I''ve got a handful of nodes (3?) out of about 400 that are giving me grief... puppet will run either manually or in the service mode. However, in the service mode the puppet process dies after an hour or so. I got an strace of the failure: >> >> rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 >> rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 >> read(8, "", 4096) = 0 >> rt_sigaction(SIGPIPE, {0x1, [], SA_RESTORER, 0x31af00eb10}, {0x31af88c700, [], SA_RESTORER|SA_RESTART, 0x31af00eb10}, 8) = 0 >> write(2, "ruby: ../../../libraries/libldap"..., 98ruby: ../../../libraries/libldap/result.c:113: ldap_result: Assertion `ld != ((void *)0)'' failed. >> ) = 98 >> rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0 >> write(4, ":XMLRPC::WEBrickServlet:0x2afa17"..., 495) = 495 >> tgkill(1267, 1267, SIGABRT) = 0 >> --- SIGABRT (Aborted) @ 0 (0) --- >> +++ killed by SIGABRT (core dumped) +++ >> [root@bastion100 ~]# >> >> I''m running Puppet 2.6.5 on CentOS 5.5 x86_64... any thoughts? > > Something is making libldap unhappy, which is being used by Ruby, used > in turn by Puppet. Which triggers the unfortunate error handling > behaviour of "abort the entire process now!!!" that I just *love* from > libraries I depend on. > > So, I would go hunting for details about what causes that particular > assertion to fire in the LDAP library you are using. My guess would > be that there is a bug in there, tickled by some particular data set, > that those hosts hit, and that a newer release would probably fix it. > > Other that turning off LDAP in the Puppet code, there isn''t likely > much we can do about it. > > Daniel > -- > ⎋ Puppet Labs Developer – http://puppetlabs.com > ✉ Daniel Pittman <daniel@puppetlabs.com> > ✆ Contact me via gtalk, email, or phone: +1 (877) 575-9775 > ♲ Made with 100 percent post-consumer electrons-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Just following up on this a bit more. One additional thing I needed to do as well was install ruby-ldap. It wasnt installed by default, and caused all kinds of random failures that were hard to troubleshoot. Once installed, all the warning messages went away and the failures stopped. On Apr 19, 2011, at 8:14 AM, Matt Wise wrote:> Ok, for what its worth, I think I solved this a while back.. but I ran into it again the other day, and couldn''t remember the fix. I found this thread again while searching around — so I figure I should update it so that everyone knows what went wrong. > > We are using not only local files, and LDAP for auth.. but we''re using ''nsscache'' as a backup if the LDAP service is down for any extended period of time. The problem seems to be that occasionally nsscache writes out its DB files in a way that upsets ruby''s ldap library significantly. Disabling nsscache, or rebuilding its DB files from scratch seems to solve this problem. > > —Matt > > On Mar 16, 2011, at 8:46 PM, Daniel Pittman wrote: > >> On Wed, Mar 16, 2011 at 20:16, Matt Wise <wise@wiredgeek.net> wrote: >> >>> I''ve got a handful of nodes (3?) out of about 400 that are giving me grief... puppet will run either manually or in the service mode. However, in the service mode the puppet process dies after an hour or so. I got an strace of the failure: >>> >>> rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 >>> rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 >>> read(8, "", 4096) = 0 >>> rt_sigaction(SIGPIPE, {0x1, [], SA_RESTORER, 0x31af00eb10}, {0x31af88c700, [], SA_RESTORER|SA_RESTART, 0x31af00eb10}, 8) = 0 >>> write(2, "ruby: ../../../libraries/libldap"..., 98ruby: ../../../libraries/libldap/result.c:113: ldap_result: Assertion `ld != ((void *)0)'' failed. >>> ) = 98 >>> rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0 >>> write(4, ":XMLRPC::WEBrickServlet:0x2afa17"..., 495) = 495 >>> tgkill(1267, 1267, SIGABRT) = 0 >>> --- SIGABRT (Aborted) @ 0 (0) --- >>> +++ killed by SIGABRT (core dumped) +++ >>> [root@bastion100 ~]# >>> >>> I''m running Puppet 2.6.5 on CentOS 5.5 x86_64... any thoughts? >> >> Something is making libldap unhappy, which is being used by Ruby, used >> in turn by Puppet. Which triggers the unfortunate error handling >> behaviour of "abort the entire process now!!!" that I just *love* from >> libraries I depend on. >> >> So, I would go hunting for details about what causes that particular >> assertion to fire in the LDAP library you are using. My guess would >> be that there is a bug in there, tickled by some particular data set, >> that those hosts hit, and that a newer release would probably fix it. >> >> Other that turning off LDAP in the Puppet code, there isn''t likely >> much we can do about it. >> >> Daniel >> -- >> ⎋ Puppet Labs Developer – http://puppetlabs.com >> ✉ Daniel Pittman <daniel@puppetlabs.com> >> ✆ Contact me via gtalk, email, or phone: +1 (877) 575-9775 >> ♲ Made with 100 percent post-consumer electrons >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Belated update on this one from another ''victim''... Our CentOS and RHEL hosts don''t have ruby-ldap, but we had the same problem, with ldap-related assertion failures, including a couple of times outside of a puppet run where ssh sessions would be terminated. The fix: upgrade the version of nss_ldap to 253-37.el5_6.1 . Bug details: https://bugzilla.redhat.com/show_bug.cgi?id=684889 and https://bugzilla.redhat.com/show_bug.cgi?id=703831 Bug fix advisory: http://rhn.redhat.com/errata/RHBA-2011-0514.html On 26 April 2011 12:02, Matt Wise <wise@wiredgeek.net> wrote:> Just following up on this a bit more. One additional thing I needed to do > as well was install ruby-ldap. It wasnt installed by default, and caused all > kinds of random failures that were hard to troubleshoot. Once installed, all > the warning messages went away and the failures stopped. > > On Apr 19, 2011, at 8:14 AM, Matt Wise wrote: > > > Ok, for what its worth, I think I solved this a while back.. but I ran > into it again the other day, and couldn''t remember the fix. I found this > thread again while searching around — so I figure I should update it so that > everyone knows what went wrong. > > > > We are using not only local files, and LDAP for auth.. but we''re using > ''nsscache'' as a backup if the LDAP service is down for any extended period > of time. The problem seems to be that occasionally nsscache writes out its > DB files in a way that upsets ruby''s ldap library significantly. Disabling > nsscache, or rebuilding its DB files from scratch seems to solve this > problem. > > > > —Matt > > > > On Mar 16, 2011, at 8:46 PM, Daniel Pittman wrote: > > > >> On Wed, Mar 16, 2011 at 20:16, Matt Wise <wise@wiredgeek.net> wrote: > >> > >>> I''ve got a handful of nodes (3?) out of about 400 that are giving me > grief... puppet will run either manually or in the service mode. However, in > the service mode the puppet process dies after an hour or so. I got an > strace of the failure: > >>> > >>> rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 > >>> rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 > >>> read(8, "", 4096) = 0 > >>> rt_sigaction(SIGPIPE, {0x1, [], SA_RESTORER, 0x31af00eb10}, > {0x31af88c700, [], SA_RESTORER|SA_RESTART, 0x31af00eb10}, 8) = 0 > >>> write(2, "ruby: ../../../libraries/libldap"..., 98ruby: > ../../../libraries/libldap/result.c:113: ldap_result: Assertion `ld !> ((void *)0)'' failed. > >>> ) = 98 > >>> rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0 > >>> write(4, ":XMLRPC::WEBrickServlet:0x2afa17"..., 495) = 495 > >>> tgkill(1267, 1267, SIGABRT) = 0 > >>> --- SIGABRT (Aborted) @ 0 (0) --- > >>> +++ killed by SIGABRT (core dumped) +++ > >>> [root@bastion100 ~]# > >>> > >>> I''m running Puppet 2.6.5 on CentOS 5.5 x86_64... any thoughts? > >> > >> Something is making libldap unhappy, which is being used by Ruby, used > >> in turn by Puppet. Which triggers the unfortunate error handling > >> behaviour of "abort the entire process now!!!" that I just *love* from > >> libraries I depend on. > >> > >> So, I would go hunting for details about what causes that particular > >> assertion to fire in the LDAP library you are using. My guess would > >> be that there is a bug in there, tickled by some particular data set, > >> that those hosts hit, and that a newer release would probably fix it. > >> > >> Other that turning off LDAP in the Puppet code, there isn''t likely > >> much we can do about it. > >> > >> Daniel > >> -- > >> ⎋ Puppet Labs Developer – http://puppetlabs.com > >> ✉ Daniel Pittman <daniel@puppetlabs.com> > >> ✆ Contact me via gtalk, email, or phone: +1 (877) 575-9775 > >> ♲ Made with 100 percent post-consumer electrons > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.