Hi, My puppet install was working fine for quite a while, but at some point has stopped working. I have puppetmasterd running on the server and listening on the correct port (8140). # rpm -qa|grep ruby ruby-libs-1.8.5-5.el5_1.1 ruby-1.8.5-5.el5_1.1 # rpm -qa|grep pup puppet-0.24.2-1.el5 puppet-server-0.24.2-1.el5 When I try to start up puppetd on any client, it just plain hangs: #puppetd --verbose --test --server cob.local I tried to strace it, but I don''t really understand the results. It seems to generate mostly lines like this: rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 puppetd doesn''t print anything to stdout or stderr or the log. puppetmasterd also prints nothing to its log after the regular startup sequence. Using lsof|grep puppetmaster, I can see the network connection to the client. On the client: # rpm -qa|grep pup puppet-0.24.1-1.el5 What''s the best way to troubleshoot this? I can connect to port 8140 on the server using telnet from the client; there''s no selinux or firewall in the way. Thanks, Alex --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Luke Kanies
2008-Mar-24 19:18 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
On Mar 24, 2008, at 2:05 PM, chalex wrote:> > Hi, > > My puppet install was working fine for quite a while, but at some > point has stopped working. I have puppetmasterd running on the server > and listening on the correct port (8140). > # rpm -qa|grep ruby > ruby-libs-1.8.5-5.el5_1.1 > ruby-1.8.5-5.el5_1.1 > # rpm -qa|grep pup > puppet-0.24.2-1.el5 > puppet-server-0.24.2-1.el5 > > When I try to start up puppetd on any client, it just plain hangs: > #puppetd --verbose --test --server cob.local > > I tried to strace it, but I don''t really understand the results. It > seems to generate mostly lines like this: > rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0This is ruby''s verbose way of having an event loop; just ignore all of them.> > puppetd doesn''t print anything to stdout or stderr or the log. > puppetmasterd also prints nothing to its log after the regular startup > sequence. Using lsof|grep puppetmaster, I can see the network > connection to the client. > > On the client: > # rpm -qa|grep pup > puppet-0.24.1-1.el5 > > What''s the best way to troubleshoot this? I can connect to port 8140 > on the server using telnet from the client; there''s no selinux or > firewall in the way.Erm, this is a complete new one on me. It''s got to be something in your environment, but I can''t for the life of me think of what it would be, other than the firewall/selinux solutions you''ve already rejected. -- There is only one success - to be able to spend your life in your own way. -- Christopher Morley --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
RijilV
2008-Mar-24 19:24 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
On 24/03/2008, chalex <alex.chekholko@gmail.com> wrote:> > > What''s the best way to troubleshoot this? I can connect to port 8140 > on the server using telnet from the client; there''s no selinux or > firewall in the way.What does a packet capture look like on both the client and the master when the normal puppet client attempts to connect? Should look pretty similar to the telnet session. .r'' --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
chalex
2008-Mar-24 20:05 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
On Mar 24, 3:24 pm, RijilV <rij...@gmail.com> wrote:> On 24/03/2008, chalex <alex.chekho...@gmail.com> wrote: > > > What''s the best way to troubleshoot this? I can connect to port 8140 > > on the server using telnet from the client; there''s no selinux or > > firewall in the way. > > What does a packet capture look like on both the client and the master when > the normal puppet client attempts to connect? Should look pretty similar to > the telnet session.I tried tcpdump and tshark. The results don''t tell me a whole lot, but I agree that they look similar for the puppet session and the telnet session. On the client: # puppetd --verbose --test --server cob.local Cancelling startup I waited a bit over a minute before hitting Ctrl-C, which results in "cancelling startup". On the server: # tshark -i eth1 tcp port 8140 Running as user "root" and group "root". This could be dangerous. Capturing on eth1 0.000000 10.10.70.15 -> 10.10.70.5 TCP 52480 > 8140 [SYN] Seq=0 Win=15920 Len=0 MSS=7960 TSV=258845901 TSER=0 WS=8 0.000008 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1364660 TSER=258845901 WS=7 0.000092 10.10.70.15 -> 10.10.70.5 TCP 52480 > 8140 [ACK] Seq=1 Ack=1 Win=16128 Len=0 TSV=258845901 TSER=1364660 0.000476 10.10.70.15 -> 10.10.70.5 TCP 52480 > 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258845902 TSER=1364660 0.205623 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480> 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258846107TSER=1364660 0.615982 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480> 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258846517TSER=1364660 1.435822 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480> 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258847337TSER=1364660 3.076220 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480> 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258848977TSER=1364660 3.587284 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1365558 TSER=258848977 WS=7 3.587402 10.10.70.15 -> 10.10.70.5 TCP [TCP Dup ACK 8#1] 52480 > 8140 [ACK] Seq=118 Ack=1 Win=16128 Len=0 TSV=258849488 TSER=1365558 SLE=0 SRE=1 6.356085 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480> 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258852257TSER=1365558 9.591637 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1367059 TSER=258852257 WS=7 9.591759 10.10.70.15 -> 10.10.70.5 TCP [TCP Dup ACK 11#1] 52480 > 8140 [ACK] Seq=118 Ack=1 Win=16128 Len=0 TSV=258855492 TSER=1367059 SLE=0 SRE=1 12.916143 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480> 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258858817TSER=1367059 21.604388 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1370062 TSER=258858817 WS=7 21.604512 10.10.70.15 -> 10.10.70.5 TCP [TCP Dup ACK 14#1] 52480 > 8140 [ACK] Seq=118 Ack=1 Win=16128 Len=0 TSV=258867504 TSER=1370062 SLE=0 SRE=1 26.037840 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480> 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258871937TSER=1370062 45.629871 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1376068 TSER=258871937 WS=7 45.629988 10.10.70.15 -> 10.10.70.5 TCP [TCP Dup ACK 17#1] 52480 > 8140 [ACK] Seq=118 Ack=1 Win=16128 Len=0 TSV=258891528 TSER=1376068 SLE=0 SRE=1 52.278715 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480> 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258898177TSER=1376068 75.329119 10.10.70.15 -> 10.10.70.5 TCP 52480 > 8140 [FIN, ACK] Seq=118 Ack=1 Win=16128 Len=0 TSV=258921225 TSER=1376068 93.880927 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1388130 TSER=258898177 WS=7 93.881050 10.10.70.15 -> 10.10.70.5 TCP [TCP Dup ACK 21#1] 52480 > 8140 [ACK] Seq=119 Ack=1 Win=16128 Len=0 TSV=258939776 TSER=1388130 SLE=0 SRE=1 The log on the server: [2008-03-24 14:24:49] INFO WEBrick 1.3.1 [2008-03-24 14:24:49] INFO ruby 1.8.5 (2006-08-25) [x86_64-linux] [2008-03-24 14:24:49] INFO Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha1WithRSAEncryption Issuer: CN=cob.genomics.upenn.edu Validity Not Before: Nov 29 02:56:05 2007 GMT Not After : Nov 27 02:56:05 2012 GMT Subject: CN=cob.genomics.upenn.edu Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public Key: (1024 bit) Modulus (1024 bit): 00:bb:56:8b:63:ed:c5:f9:12:3e:b7:b0:b8:fb:4b: c2:e5:9c:4f:f6:6c:59:bf:c5:6c:99:e1:4b:64:33: 2b:5b:70:45:a5:ae:e3:3a:7c:b6:62:aa:9e:45:fd: 26:26:86:61:da:be:9f:ed:12:51:80:e4:c2:f8:48: c9:88:c1:c8:00:34:12:f6:ed:c1:af:a8:28:c1:13: f0:a7:14:80:dc:c4:5a:dc:89:a1:3d:af:3a:bd:c9: 44:aa:a1:62:86:06:d7:4d:54:39:99:89:61:57:9e: 74:5a:1c:fa:74:e8:79:98:06:50:6b:b6:87:77:d6: c1:c2:53:be:42:23:61:5a:31 Exponent: 65537 (0x10001) X509v3 extensions: Netscape Comment: Puppet Ruby/OpenSSL Generated Certificate X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: AB:55:EF:F2:5E:04:B4:BE:D7:D8:1A:F4:EE:61:4A:EF: 37:8E:DB:00 X509v3 Key Usage: Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication Signature Algorithm: sha1WithRSAEncryption 86:a1:dc:d7:ee:47:32:39:ec:e7:a8:7f:fb:68:cc:36:4d:eb: 61:49:e9:8e:04:24:5e:4c:ba:44:85:5c:6e:41:93:16:a2:8b: bf:d4:5f:65:a1:63:8e:4b:38:ef:3f:37:48:e7:78:da:a2:0a: 52:47:a5:61:01:d6:4c:43:87:e8:81:86:06:ff:c0:5c:4f:3d: 6e:83:81:88:ef:ea:d0:7f:2f:7f:d0:c5:e0:1d:d8:ac:1e:93: 8d:d5:88:ed:87:96:74:95:9f:26:09:ea:95:c5:9a:25:49:47: 75:06:01:47:b8:8b:2c:f5:0a:1f:88:e6:d7:75:e1:00:76:a3: 61:ce It doesn''t seem to write anything except this one bit after it starts up. Thanks, Alex --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Simon Coggins
2008-Mar-24 20:47 UTC
[Puppet Users] puppetd silently hangs - how to troubleshoot
This may have nothing to do with it, but I''ve noticed my puppetd clients recently locking up and hanging after they get their config from the server. It turns out to be an ldap lookup problem. For some reason a query it does to lookup group/user information hangs. If I kick nscd and restart it, puppet continues on it''s way. I''m pretty sure this won''t be your problem as you appear to be on redhat (I''m on solaris). But just incase it is I thought I''d mention it. Regards Simon On Tue, Mar 25, 2008 at 6:05 AM, chalex <alex.chekholko@gmail.com> wrote: > > On Mar 24, 3:24 pm, RijilV <rij...@gmail.com> wrote: > > > On 24/03/2008, chalex <alex.chekho...@gmail.com> wrote: > > > > > What''s the best way to troubleshoot this? I can connect to port 8140 > > > on the server using telnet from the client; there''s no selinux or > > > firewall in the way. > > > > What does a packet capture look like on both the client and the master when > > the normal puppet client attempts to connect? Should look pretty similar to > > the telnet session. > > I tried tcpdump and tshark. The results don''t tell me a whole lot, > but I agree that they look similar for the puppet session and the > telnet session. > > On the client: > # puppetd --verbose --test --server cob.local > Cancelling startup > > I waited a bit over a minute before hitting Ctrl-C, which results in > "cancelling startup". > > On the server: > # tshark -i eth1 tcp port 8140 > Running as user "root" and group "root". This could be dangerous. > Capturing on eth1 > 0.000000 10.10.70.15 -> 10.10.70.5 TCP 52480 > 8140 [SYN] Seq=0 > Win=15920 Len=0 MSS=7960 TSV=258845901 TSER=0 WS=8 > 0.000008 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] > Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1364660 TSER=258845901 WS=7 > 0.000092 10.10.70.15 -> 10.10.70.5 TCP 52480 > 8140 [ACK] Seq=1 > Ack=1 Win=16128 Len=0 TSV=258845901 TSER=1364660 > 0.000476 10.10.70.15 -> 10.10.70.5 TCP 52480 > 8140 [PSH, ACK] > Seq=1 Ack=1 Win=16128 Len=117 TSV=258845902 TSER=1364660 > 0.205623 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480 > > 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258846107 > TSER=1364660 > 0.615982 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480 > > 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258846517 > TSER=1364660 > 1.435822 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480 > > 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258847337 > TSER=1364660 > 3.076220 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480 > > 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258848977 > TSER=1364660 > 3.587284 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] > Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1365558 TSER=258848977 WS=7 > 3.587402 10.10.70.15 -> 10.10.70.5 TCP [TCP Dup ACK 8#1] 52480 > > 8140 [ACK] Seq=118 Ack=1 Win=16128 Len=0 TSV=258849488 TSER=1365558 > SLE=0 SRE=1 > 6.356085 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480 > > 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258852257 > TSER=1365558 > 9.591637 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] > Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1367059 TSER=258852257 WS=7 > 9.591759 10.10.70.15 -> 10.10.70.5 TCP [TCP Dup ACK 11#1] 52480 > > 8140 [ACK] Seq=118 Ack=1 Win=16128 Len=0 TSV=258855492 TSER=1367059 > SLE=0 SRE=1 > 12.916143 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480 > > 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258858817 > TSER=1367059 > 21.604388 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] > Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1370062 TSER=258858817 WS=7 > 21.604512 10.10.70.15 -> 10.10.70.5 TCP [TCP Dup ACK 14#1] 52480 > > 8140 [ACK] Seq=118 Ack=1 Win=16128 Len=0 TSV=258867504 TSER=1370062 > SLE=0 SRE=1 > 26.037840 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480 > > 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258871937 > TSER=1370062 > 45.629871 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] > Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1376068 TSER=258871937 WS=7 > 45.629988 10.10.70.15 -> 10.10.70.5 TCP [TCP Dup ACK 17#1] 52480 > > 8140 [ACK] Seq=118 Ack=1 Win=16128 Len=0 TSV=258891528 TSER=1376068 > SLE=0 SRE=1 > 52.278715 10.10.70.15 -> 10.10.70.5 TCP [TCP Retransmission] 52480 > > 8140 [PSH, ACK] Seq=1 Ack=1 Win=16128 Len=117 TSV=258898177 > TSER=1376068 > 75.329119 10.10.70.15 -> 10.10.70.5 TCP 52480 > 8140 [FIN, ACK] > Seq=118 Ack=1 Win=16128 Len=0 TSV=258921225 TSER=1376068 > 93.880927 10.10.70.5 -> 10.10.70.15 TCP 8140 > 52480 [SYN, ACK] > Seq=0 Ack=1 Win=15896 Len=0 MSS=7960 TSV=1388130 TSER=258898177 WS=7 > 93.881050 10.10.70.15 -> 10.10.70.5 TCP [TCP Dup ACK 21#1] 52480 > > 8140 [ACK] Seq=119 Ack=1 Win=16128 Len=0 TSV=258939776 TSER=1388130 > SLE=0 SRE=1 > > The log on the server: > [2008-03-24 14:24:49] INFO WEBrick 1.3.1 > [2008-03-24 14:24:49] INFO ruby 1.8.5 (2006-08-25) [x86_64-linux] > [2008-03-24 14:24:49] INFO > Certificate: > Data: > Version: 3 (0x2) > Serial Number: 1 (0x1) > Signature Algorithm: sha1WithRSAEncryption > Issuer: CN=cob.genomics.upenn.edu > Validity > Not Before: Nov 29 02:56:05 2007 GMT > Not After : Nov 27 02:56:05 2012 GMT > Subject: CN=cob.genomics.upenn.edu > Subject Public Key Info: > Public Key Algorithm: rsaEncryption > RSA Public Key: (1024 bit) > Modulus (1024 bit): > 00:bb:56:8b:63:ed:c5:f9:12:3e:b7:b0:b8:fb:4b: > c2:e5:9c:4f:f6:6c:59:bf:c5:6c:99:e1:4b:64:33: > 2b:5b:70:45:a5:ae:e3:3a:7c:b6:62:aa:9e:45:fd: > 26:26:86:61:da:be:9f:ed:12:51:80:e4:c2:f8:48: > c9:88:c1:c8:00:34:12:f6:ed:c1:af:a8:28:c1:13: > f0:a7:14:80:dc:c4:5a:dc:89:a1:3d:af:3a:bd:c9: > 44:aa:a1:62:86:06:d7:4d:54:39:99:89:61:57:9e: > 74:5a:1c:fa:74:e8:79:98:06:50:6b:b6:87:77:d6: > c1:c2:53:be:42:23:61:5a:31 > Exponent: 65537 (0x10001) > X509v3 extensions: > Netscape Comment: > Puppet Ruby/OpenSSL Generated Certificate > X509v3 Basic Constraints: critical > CA:FALSE > X509v3 Subject Key Identifier: > AB:55:EF:F2:5E:04:B4:BE:D7:D8:1A:F4:EE:61:4A:EF: > 37:8E:DB:00 > X509v3 Key Usage: > Digital Signature, Key Encipherment > X509v3 Extended Key Usage: > TLS Web Server Authentication, TLS Web Client > Authentication > Signature Algorithm: sha1WithRSAEncryption > 86:a1:dc:d7:ee:47:32:39:ec:e7:a8:7f:fb:68:cc:36:4d:eb: > 61:49:e9:8e:04:24:5e:4c:ba:44:85:5c:6e:41:93:16:a2:8b: > bf:d4:5f:65:a1:63:8e:4b:38:ef:3f:37:48:e7:78:da:a2:0a: > 52:47:a5:61:01:d6:4c:43:87:e8:81:86:06:ff:c0:5c:4f:3d: > 6e:83:81:88:ef:ea:d0:7f:2f:7f:d0:c5:e0:1d:d8:ac:1e:93: > 8d:d5:88:ed:87:96:74:95:9f:26:09:ea:95:c5:9a:25:49:47: > 75:06:01:47:b8:8b:2c:f5:0a:1f:88:e6:d7:75:e1:00:76:a3: > 61:ce > > It doesn''t seem to write anything except this one bit after it starts > up. > > Thanks, > Alex > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Luke Kanies
2008-Mar-24 20:48 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
On Mar 24, 2008, at 3:05 PM, chalex wrote:> It doesn''t seem to write anything except this one bit after it starts > up.Look in syslog, not the webserver log. -- A complex system that works is invariably found to have evolved from a simple system that works. -- John Gaule --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
chalex
2008-Mar-24 20:59 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
On Mar 24, 4:48 pm, Luke Kanies <l...@madstop.com> wrote:> On Mar 24, 2008, at 3:05 PM, chalex wrote: > > > It doesn''t seem to write anything except this one bit after it starts > > up. > > Look in syslog, not the webserver log. >Nope, nothing there either. It definitely used to write all the useful information into /var/log/messages. Here''s the latest relevant messages from there: Mar 13 20:18:35 cob puppetmasterd[3580]: Compiled configuration for node-r1-u18-c17-p11-o13.local in 0.15 seconds Mar 13 20:48:40 cob puppetmasterd[3580]: Compiled configuration for node-r1-u18-c17-p11-o13.local in 0.07 seconds Mar 13 21:35:35 cob puppetmasterd[3580]: Caught TERM; shutting down Mar 13 21:35:35 cob puppetmasterd[3580]: Shutting down Mar 13 21:53:09 cob puppetmasterd[1152]: Reopening log files Mar 13 21:53:09 cob puppetmasterd[1152]: Starting Puppet server version 0.24.1 Mar 14 00:07:33 cob puppetmasterd[1160]: Reopening log files Mar 14 00:07:33 cob puppetmasterd[1160]: Starting Puppet server version 0.24.1 Mar 14 00:10:34 cob Updated: puppet.noarch 0.24.2-1.el5 Mar 14 00:10:46 cob Updated: puppet-server.noarch 0.24.2-1.el5 Mar 14 00:10:49 cob puppetmasterd[1160]: Caught TERM; shutting down Mar 14 00:10:49 cob puppetmasterd[1160]: Shutting down Mar 14 00:10:51 cob puppetmasterd[2011]: Reopening log files Mar 14 00:10:51 cob puppetmasterd[2011]: Starting Puppet server version 0.24.2 Mar 14 17:10:42 cob puppetmasterd[2011]: Caught TERM; shutting down Mar 14 17:10:42 cob puppetmasterd[2011]: Shutting down Mar 14 17:10:47 cob puppetmasterd[26689]: Reopening log files Mar 14 17:10:47 cob puppetmasterd[26689]: Starting Puppet server version 0.24.2 Mar 16 04:02:15 cob puppetmasterd[26689]: Caught TERM; shutting down Mar 16 04:02:15 cob puppetmasterd[26689]: Shutting down Mar 16 04:02:21 cob puppetmasterd[32215]: Reopening log files Mar 16 04:02:21 cob puppetmasterd[32215]: Starting Puppet server version 0.24.2 [root@cob.genomics.upenn.edu] ~ [0] # grep pup /var/log/messages.1 Mar 23 04:02:11 cob puppetmasterd[32215]: Caught TERM; shutting down Mar 23 04:02:11 cob puppetmasterd[32215]: Shutting down Mar 23 04:02:11 cob puppetmasterd[32215]: Could not remove PID file / var/run/puppet/puppetmasterd.pid Mar 23 04:02:18 cob puppetmasterd[23266]: Could not create PID file: / var/run/puppet/puppetmasterd.pid [root@cob.genomics.upenn.edu] ~ [0] # grep pup /var/log/messages Mar 24 12:09:16 cob puppetmasterd[4240]: Could not create PID file: / var/run/puppet/puppetmasterd.pid Mar 24 12:10:45 cob puppetmasterd[4314]: Could not create PID file: / var/run/puppet/puppetmasterd.pid Mar 24 12:26:23 cob puppetmasterd[4732]: Could not create PID file: / var/run/puppet/puppetmasterd.pid Mar 24 12:26:55 cob puppetmasterd[4802]: Reopening log files Mar 24 12:26:55 cob puppetmasterd[4802]: Starting Puppet server version 0.24.2 Mar 24 13:51:14 cob puppetmasterd[4802]: Caught TERM; shutting down Mar 24 13:51:14 cob puppetmasterd[4802]: Shutting down Mar 24 13:53:04 cob puppetmasterd[5044]: Reopening log files Mar 24 13:53:04 cob puppetmasterd[5044]: Starting Puppet server version 0.24.2 Mar 24 14:24:08 cob puppetmasterd[5044]: Caught TERM; shutting down Mar 24 14:24:08 cob puppetmasterd[5044]: Shutting down Mar 24 14:24:49 cob puppetmasterd[1151]: Reopening log files Mar 24 14:24:49 cob puppetmasterd[1151]: Starting Puppet server version 0.24.2 I should have mentioned it earlier, but the filesystem filled up at some point, and I guess puppetmaster crashed, and that''s why the PID file was still there. I freed up some space and deleted the PID file and then restarted puppetmaster. Might there be some other files I need to clear out? # ls -alh /var/run/puppet/ total 16K drwxrwxrwt 2 puppet puppet 4.0K Mar 24 14:24 . drwxr-xr-x 18 root root 4.0K Mar 24 15:40 .. -rw-r--r-- 1 puppet puppet 4 Mar 24 14:24 puppetmasterd.pid Thanks, Alex --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
chalex
2008-Mar-24 21:03 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
On Mar 24, 4:47 pm, "Simon Coggins" <scogg...@gmail.com> wrote:> This may have nothing to do with it, but I''ve noticed my puppetd > clients recently locking up and hanging after they get their config > from the server. It turns out to be an ldap lookup problem. For some > reason a query it does to lookup group/user information hangs. If I > kick nscd and restart it, puppet continues on it''s way. > > I''m pretty sure this won''t be your problem as you appear to be on > redhat (I''m on solaris). But just incase it is I thought I''d mention > it.I am using LDAP on all the clients but not on the puppetmaster, but nscd is turned off everywhere. It wasn''t an issue before, and LDAP is working fine now. Thanks, Alex --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Mathieu Sauve-Frankel
2008-Mar-25 01:39 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
On Mon, Mar 24, 2008 at 02:18:53PM -0500, Luke Kanies wrote:> > On Mar 24, 2008, at 2:05 PM, chalex wrote: > > > > > Hi, > > > > My puppet install was working fine for quite a while, but at some > > point has stopped working. I have puppetmasterd running on the server > > and listening on the correct port (8140). > > # rpm -qa|grep ruby > > ruby-libs-1.8.5-5.el5_1.1 > > ruby-1.8.5-5.el5_1.1 > > # rpm -qa|grep pup > > puppet-0.24.2-1.el5 > > puppet-server-0.24.2-1.el5 > > > > When I try to start up puppetd on any client, it just plain hangs: > > #puppetd --verbose --test --server cob.local > > > > I tried to strace it, but I don''t really understand the results. It > > seems to generate mostly lines like this: > > rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 > > This is ruby''s verbose way of having an event loop; just ignore all of > them. > > > > > puppetd doesn''t print anything to stdout or stderr or the log. > > puppetmasterd also prints nothing to its log after the regular startup > > sequence. Using lsof|grep puppetmaster, I can see the network > > connection to the client. > > > > On the client: > > # rpm -qa|grep pup > > puppet-0.24.1-1.el5 > > > > What''s the best way to troubleshoot this? I can connect to port 8140 > > on the server using telnet from the client; there''s no selinux or > > firewall in the way. > > Erm, this is a complete new one on me. It''s got to be something in > your environment, but I can''t for the life of me think of what it > would be, other than the firewall/selinux solutions you''ve already > rejected.McBride and I have observed puppet hanging silently when dns on the client is hosed, we believe it''s related to facter, but no definite answers yet. -- Mathieu Sauve-Frankel --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
DerekW
2008-Mar-25 08:01 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
Complete shot in the dark, but the only time this happens to me is when my cwd goes away, e.g. when an NFS file system (e.g. my homedir) is remounted (e.g. when Puppet has restarted autofs). That''s Ruby choking. Derek On Mar 25, 1:39 am, m...@kisoku.net (Mathieu Sauve-Frankel) wrote:> On Mon, Mar 24, 2008 at 02:18:53PM -0500, Luke Kanies wrote: > > > On Mar 24, 2008, at 2:05 PM, chalex wrote: > > > > Hi, > > > > My puppet install was working fine for quite a while, but at some > > > point has stopped working. I have puppetmasterd running on the server > > > and listening on the correct port (8140). > > > # rpm -qa|grep ruby > > > ruby-libs-1.8.5-5.el5_1.1 > > > ruby-1.8.5-5.el5_1.1 > > > # rpm -qa|grep pup > > > puppet-0.24.2-1.el5 > > > puppet-server-0.24.2-1.el5 > > > > When I try to start up puppetd on any client, it just plain hangs: > > > #puppetd --verbose --test --server cob.local > > > > I tried to strace it, but I don''t really understand the results. It > > > seems to generate mostly lines like this: > > > rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 > > > This is ruby''s verbose way of having an event loop; just ignore all of > > them. > > > > puppetd doesn''t print anything to stdout or stderr or the log. > > > puppetmasterd also prints nothing to its log after the regular startup > > > sequence. Using lsof|grep puppetmaster, I can see the network > > > connection to the client. > > > > On the client: > > > # rpm -qa|grep pup > > > puppet-0.24.1-1.el5 > > > > What''s the best way to troubleshoot this? I can connect to port 8140 > > > on the server using telnet from the client; there''s no selinux or > > > firewall in the way. > > > Erm, this is a complete new one on me. It''s got to be something in > > your environment, but I can''t for the life of me think of what it > > would be, other than the firewall/selinux solutions you''ve already > > rejected. > > McBride and I have observed puppet hanging silently when dns on the client > is hosed, we believe it''s related to facter, but no definite answers yet. > > -- > Mathieu Sauve-Frankel--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
chalex
2008-Mar-31 20:42 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
Hi all, Still trying to figure this out. Upgraded puppetmaster and client to 0.24.4 Running puppetmasterd with --debug gets me the following messages in / var/log/messages: Mar 31 16:35:38 cob puppetmasterd[32745]: Starting server for Puppet version 0.24.4 Mar 31 16:35:38 cob puppetmasterd[32745]: (mount[dist]) allowing 10.0.0.0/8 access Mar 31 16:35:38 cob puppetmasterd[32745]: (mount[dist]) allowing 128.91.79.0/24 access Mar 31 16:35:39 cob puppetmasterd[32745]: Listening on port 8140 Mar 31 16:35:39 cob puppetmasterd[32763]: Reopening log files Mar 31 16:35:39 cob puppetmasterd[32763]: Starting Puppet server version 0.24.4 Running puppetd with --debug gets me the following: # puppetd --test --server cob.local --waitforcert 1 --debug debug: Creating default schedules debug: Failed to load library ''shadow'' for feature ''libshadow'' debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/lib/ puppet/lib]: Autorequiring File[/var/lib/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/lib/ puppet/ssl/public_keys/beta.genomics.upenn.edu.pem]: Autorequiring File[/var/lib/puppet/ssl/public_keys] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/lib/ puppet/ssl]: Autorequiring File[/var/lib/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/lib/ puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/lib/ puppet/ssl/certs/ca.pem]: Autorequiring File[/var/lib/puppet/ssl/ certs] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/lib/ puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[puppetd]/File[/var/ lib/puppet/classes.txt]: Autorequiring File[/var/lib/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/lib/ puppet/state]: Autorequiring File[/var/lib/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/lib/ puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/lib/ puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/lib/ puppet/ssl/csr_beta.genomics.upenn.edu.pem]: Autorequiring File[/var/ lib/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[puppetd]/File[/var/ lib/puppet/state/state.yaml]: Autorequiring File[/var/lib/puppet/ state] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/lib/ puppet/ssl/certs/beta.genomics.upenn.edu.pem]: Autorequiring File[/var/ lib/puppet/ssl/certs] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/lib/ puppet/ssl/private_keys/beta.genomics.upenn.edu.pem]: Autorequiring File[/var/lib/puppet/ssl/private_keys] debug: /Settings[/etc/puppet/puppet.conf]/Settings[puppetd]/File[/etc/ puppet/puppet.conf]: Autorequiring File[/etc/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[puppetd]/File[/var/ lib/puppet/state/state.yaml]: Changing mode debug: /Settings[/etc/puppet/puppet.conf]/Settings[puppetd]/File[/var/ lib/puppet/state/state.yaml]: 1 change(s) debug: /Settings[/etc/puppet/puppet.conf]/Settings[puppetd]/File[/var/ lib/puppet/state/state.yaml]/mode: mode changed ''640'' to ''660'' debug: Finishing transaction 23456267419720 with 1 changes debug: Loaded state in 0.00 seconds debug: Retrieved facts in 0.20 seconds notice: Ignoring cache debug: Retrieving catalog debug: Calling puppetmaster.getconfig It sits there a while then says err: Configuration retrieval timed out warning: Not using cache on failed catalog and quits. No messages from puppetmaster during that time. Is there any way to print out what the puppetmaster is doing? Thanks, Alex On Mar 24, 3:05 pm, chalex <alex.chekho...@gmail.com> wrote:> Hi, > > My puppet install was working fine for quite a while, but at some > point has stopped working. I have puppetmasterd running on the server > and listening on the correct port (8140). > # rpm -qa|grep ruby > ruby-libs-1.8.5-5.el5_1.1 > ruby-1.8.5-5.el5_1.1 > # rpm -qa|grep pup > puppet-0.24.2-1.el5 > puppet-server-0.24.2-1.el5 > > When I try to start up puppetd on any client, it just plain hangs: > #puppetd --verbose --test --server cob.local > > I tried to strace it, but I don''t really understand the results. It > seems to generate mostly lines like this: > rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 > > puppetd doesn''t print anything to stdout or stderr or the log. > puppetmasterd also prints nothing to its log after the regular startup > sequence. Using lsof|grep puppetmaster, I can see the network > connection to the client. > > On the client: > # rpm -qa|grep pup > puppet-0.24.1-1.el5 > > What''s the best way to troubleshoot this? I can connect to port 8140 > on the server using telnet from the client; there''s no selinux or > firewall in the way. > > Thanks, > Alex--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Luke Kanies
2008-Mar-31 21:12 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
On Mar 31, 2008, at 3:42 PM, chalex wrote:> No messages from puppetmaster during that time. Is there any way to > print out what the puppetmaster is doing?If the master isn''t printing anything, then either it''s never getting the connection, or it''s a problem with ssl/http, which is too low- level for my code to log problems with. It sounds like it''s a connectivity issue between the client and server, either at a networking/dns level or in http/ssl. No real idea how to daignose that, other than using lots of strace/truss on client and maybe server. -- Kai''s Example Dilemma: A good analogy is like a diagonal frog. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
chalex
2008-Apr-01 20:27 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
On Mar 31, 5:12 pm, Luke Kanies <l...@madstop.com> wrote:> On Mar 31, 2008, at 3:42 PM, chalex wrote: > > > No messages from puppetmaster during that time. Is there any way to > > print out what the puppetmaster is doing? > > If the master isn''t printing anything, then either it''s never getting > the connection, or it''s a problem with ssl/http, which is too low- > level for my code to log problems with. > > It sounds like it''s a connectivity issue between the client and > server, either at a networking/dns level or in http/ssl. No real idea > how to daignose that, other than using lots of strace/truss on client > and maybe server.What should be the normal behavior of puppetmasterd if I telnet to it? Can I issue some HTTP request and get some response? Or does it need to be HTTPS? Thanks, Alex --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Luke Kanies
2008-Apr-01 21:44 UTC
[Puppet Users] Re: puppetd silently hangs - how to troubleshoot
On Apr 1, 2008, at 3:27 PM, chalex wrote:> What should be the normal behavior of puppetmasterd if I telnet to > it? Can I issue some HTTP request and get some response? Or does it > need to be HTTPS?It has to be https. You should be able to follow the examples in ext/puppet-test to create a simple xmlrpc client of the ''status'' namespace to test this. I also recently posted a script that should exactly do this, in a thread about monitoring. -- The Chico, California, City Council enacted a ban on nuclear weapons, setting a $500 fine for anyone detonating one within city limits. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---