I have a couple of hosts which are having trouble talking to the puppet VIP: puppetd[4554]: could not retrieve catalog from remote server: ssl_connect returned=1 errno=0 state=sslv3 read server certificate b: certificate verify failed puppetd[4554]: Not using cache on failed catalog puppetd[4554]: Could not retrieve catalog; skipping run puppetd[4961]: Retrieving plugin puppetd[4961]: (/File[/var/lib/puppet/lib]) Failed to generate additional resources using ''eval_generate'': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed puppetd[4961]: (/File[/var/lib/puppet/lib]) Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet://puppet/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed I''ve gone through the usual SSL troubleshooting: the clocks are in sync, the client cert matches the one issued to it by the server (and is decodable by the private_key). When I use tshark to watch the ssl traffic, I see that the client is rejecting the server with the following ssl error. The connection never makes it to the back-end server, because the client hangs up. (10.1.1.1 is this client, 10.0.0.1 is the puppet vip) [root@db9 /var/lib/puppet/ssl]# tshark -n -i bond0 -d tcp.port==8140,ssl -s2000 ''port 8140 and len > 60'' 0.000000 10.1.1.1 -> 10.0.0.1 TCP 29718 > 8140 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=1862094055 TSER=0 WS=7 0.001585 10.1.1.1 -> 10.0.0.1 SSLv2 Client Hello 0.001713 10.0.0.1 -> 10.1.1.1 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done 0.002208 10.1.1.1 -> 10.0.0.1 TLSv1 Alert (Level: Fatal, Description: Unknown CA) But openssl with the same cert and key that puppet is using passes verification and connects successfully: openssl s_client -connect puppet:8140 -cert certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem -showcerts -state -verify 2 103.115871 10.1.1.1 -> 10.0.0.1 TCP 40758 > 8140 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=1862197169 TSER=0 WS=7 103.116949 10.1.1.1 -> 10.0.0.1 SSLv2 Client Hello 103.117078 10.0.0.1 -> 10.1.1.1 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done 103.121057 10.1.1.1 -> 10.0.0.1 TLSv1 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message 103.122162 10.0.0.1 -> 10.1.1.1 TLSv1 Change Cipher Spec, Encrypted Handshake Message Any thoughts on what could be causing this failure? I''ve seen quite a few odd ones (#3120, #4948 for example) but I''ve been gnawing at this one all day and haven''t figured it out. -=Eric -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Fri, 2011-02-18 at 00:44 -0800, Eric Sorenson wrote:> I have a couple of hosts which are having trouble talking to the puppet VIP: > > puppetd[4554]: could not retrieve catalog from remote server: ssl_connect returned=1 errno=0 state=sslv3 read server certificate b: certificate verify failed > puppetd[4554]: Not using cache on failed catalog > puppetd[4554]: Could not retrieve catalog; skipping run > puppetd[4961]: Retrieving plugin > puppetd[4961]: (/File[/var/lib/puppet/lib]) Failed to generate additional resources using ''eval_generate'': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed > puppetd[4961]: (/File[/var/lib/puppet/lib]) Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet://puppet/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failedThis certainly means your local node CA cert is not able to verify the server proposed certificate (probably because the server advertised cert hasn''t been signed by this CA, or you use a CA chain but don''t send the full chain to the client).> I''ve gone through the usual SSL troubleshooting: the clocks are in > sync, the client cert matches the one issued to it by the server (and > is decodable by the private_key). > > When I use tshark to watch the ssl traffic, I see that the client is > rejecting the server with the following ssl error. The connection > never makes it to the back-end server, because the client hangs up. > (10.1.1.1 is this client, 10.0.0.1 is the puppet vip) > > [root@db9 /var/lib/puppet/ssl]# tshark -n -i bond0 -d tcp.port==8140,ssl -s2000 ''port 8140 and len > 60'' > 0.000000 10.1.1.1 -> 10.0.0.1 TCP 29718 > 8140 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=1862094055 TSER=0 WS=7 > 0.001585 10.1.1.1 -> 10.0.0.1 SSLv2 Client Hello > 0.001713 10.0.0.1 -> 10.1.1.1 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done > 0.002208 10.1.1.1 -> 10.0.0.1 TLSv1 Alert (Level: Fatal, Description: Unknown CA) > > But openssl with the same cert and key that puppet is using passes verification and connects successfully: > > openssl s_client -connect puppet:8140 -cert certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem -showcerts -state -verify 2You didn''t ask openssl s_client to actually check the server certificate against the CA cert of the client. Can you try: openssl s_client -connect puppet:8140 -CAfile certs/ca.pem -cert certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem -showcerts -state -verify 2> 103.115871 10.1.1.1 -> 10.0.0.1 TCP 40758 > 8140 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=1862197169 TSER=0 WS=7 > 103.116949 10.1.1.1 -> 10.0.0.1 SSLv2 Client Hello > 103.117078 10.0.0.1 -> 10.1.1.1 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done > 103.121057 10.1.1.1 -> 10.0.0.1 TLSv1 Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message > 103.122162 10.0.0.1 -> 10.1.1.1 TLSv1 Change Cipher Spec, Encrypted Handshake Message > > Any thoughts on what could be causing this failure? I''ve seen quite a > few odd ones (#3120, #4948 for example) but I''ve been gnawing at this > one all day and haven''t figured it out.For an unknown reason your local node CA cert is not correct. You can solve this by overwriting it with the main CA cert, or check that your server certificate is indeed correctly signed by the CA you think it was signed with. -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Hi Brice, thanks for the reply. On Feb 18, 1:32 am, Brice Figureau <brice-pup...@daysofwonder.com> wrote:> This certainly means your local node CA cert is not able to verify the > server proposed certificate (probably because the server advertised cert > hasn''t been signed by this CA, or you use a CA chain but don''t send the > full chain to the client).Normally I agree but I don''t understand why all the other hosts connecting to the same VIP are fine. There are just a couple of them that exhibit this problem and scores that are fine.> You didn''t ask openssl s_client to actually check the server certificate > against the CA cert of the client. > > Can you try: > openssl s_client -connect puppet:8140 -CAfile certs/ca.pem -cert certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem -showcerts -state -verify 2 >Ah, I had actually had that in a different command but did not paste it correctly. It also succeeds: (echo "GET /" ; echo; ) | openssl s_client -connect puppet:8140 -cert certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem - showcerts -state -verify 2 -CAfile certs/ca.pem 2>&1 | grep -i verify verify depth is 2 verify return:1 verify return:1 SSL_connect:SSLv3 write certificate verify A Verify return code: 0 (ok)> For an unknown reason your local node CA cert is not correct. > You can solve this by overwriting it with the main CA cert, or check > that your server certificate is indeed correctly signed by the CA you > think it was signed with.I renamed the local file to ca-not.pem then put the cert from the CA shown with openssl -showcerts into puppet''s certs dir and compared them: [/var/lib/puppet/ssl/certs]# openssl x509 -in ca.pem -modulus -noout Modulus=B7491A13EFEDBBD259F90D914365DA8E2BF11F8AB6762723E3C91C1868AE999EAB3E0003F10C99363C7D9A5260B474EA193F52F05C42456FD0089E2284CDE9FD26F3F4F82D7A6B7E6A9B7A27EA2B2C94B46272926932A4FB6D6EDF7D6EEC720781B76261382927799E63FFA8DEF5EA6239D5590ADD34A0C06DB9818834DBB6BF [/var/lib/puppet/ssl/certs]# openssl x509 -in ca-not.pem -modulus - noout Modulus=B7491A13EFEDBBD259F90D914365DA8E2BF11F8AB6762723E3C91C1868AE999EAB3E0003F10C99363C7D9A5260B474EA193F52F05C42456FD0089E2284CDE9FD26F3F4F82D7A6B7E6A9B7A27EA2B2C94B46272926932A4FB6D6EDF7D6EEC720781B76261382927799E63FFA8DEF5EA6239D5590ADD34A0C06DB9818834DBB6BF I am really starting to think something is amiss with the ruby->openssl libraries on the affected hosts. They show the same behaviourconnecting to other puppet vips in the environment, including one that has the SSL connection terminated at Apache instead of the loadbalancer. Thanks very much Brice. I will keep at it. -=Eric -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
I figured it out, partway. There''s something about the CRL that my puppetmaster is generating that kills the client. When I mv $ssldir/ca/ca_crl.pem out of the way on the server (so it can''t be re-downloaded) and rm $ssldir/crl.pem on the client, they all work. Put it back in place, or generate a new one with puppet cert -- clean <somehost>, and I get this error. This explains why openssl was working - It doesn''t use a CRL. 100% repeatable. Still working out why the CRL is fatal. I had actually suspected this because of my earlier trouble with it (#4948) but did not see the VIP (nor the endpoints) certs in the CRL so I discounted it. On Feb 18, 9:16 am, Eric Sorenson <eric.soren...@me.com> wrote:> Hi Brice, thanks for the reply. > > On Feb 18, 1:32 am, Brice Figureau <brice-pup...@daysofwonder.com> > wrote: > > > This certainly means your local node CA cert is not able to verify the > > server proposed certificate (probably because the server advertised cert > > hasn''t been signed by this CA, or you use a CA chain but don''t send the > > full chain to the client). > > Normally I agree but I don''t understand why all the other hosts > connecting to the same VIP are fine. There are just a couple of them > that exhibit this problem and scores that are fine. > > > You didn''t ask openssl s_client to actually check the server certificate > > against the CA cert of the client. > > > Can you try: > > openssl s_client -connect puppet:8140 -CAfile certs/ca.pem -cert certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem -showcerts -state -verify 2 > > Ah, I had actually had that in a different command but did not paste > it correctly. It also succeeds: > > (echo "GET /" ; echo; ) | openssl s_client -connect puppet:8140 -cert > certs/db9.domain.com.pem -key private_keys/db9.domain.com.pem - > showcerts -state -verify 2 -CAfile certs/ca.pem 2>&1 | grep -i verify > verify depth is 2 > verify return:1 > verify return:1 > SSL_connect:SSLv3 write certificate verify A > Verify return code: 0 (ok) > > > For an unknown reason your local node CA cert is not correct. > > You can solve this by overwriting it with the main CA cert, or check > > that your server certificate is indeed correctly signed by the CA you > > think it was signed with. > > I renamed the local file to ca-not.pem then put the cert from the CA > shown with openssl -showcerts into puppet''s certs dir and compared > them: > > [/var/lib/puppet/ssl/certs]# openssl x509 -in ca.pem -modulus -noout > Modulus=B7491A13EFEDBBD259F90D914365DA8E2BF11F8AB6762723E3C91C1868AE999EAB3 E0003F10C99363C7D9A5260B474EA193F52F05C42456FD0089E2284CDE9FD26F3F4F82D7A6B 7E6A9B7A27EA2B2C94B46272926932A4FB6D6EDF7D6EEC720781B76261382927799E63FFA8D EF5EA6239D5590ADD34A0C06DB9818834DBB6BF > [/var/lib/puppet/ssl/certs]# openssl x509 -in ca-not.pem -modulus - > noout > Modulus=B7491A13EFEDBBD259F90D914365DA8E2BF11F8AB6762723E3C91C1868AE999EAB3 E0003F10C99363C7D9A5260B474EA193F52F05C42456FD0089E2284CDE9FD26F3F4F82D7A6B 7E6A9B7A27EA2B2C94B46272926932A4FB6D6EDF7D6EEC720781B76261382927799E63FFA8D EF5EA6239D5590ADD34A0C06DB9818834DBB6BF > > I am really starting to think something is amiss with the ruby->openssl libraries on the affected hosts. They show the same behaviour > > connecting to other puppet vips in the environment, including one that > has the SSL connection terminated at Apache instead of the > loadbalancer. > > Thanks very much Brice. I will keep at it. > > -=Eric-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Fri, Feb 18, 2011 at 7:34 PM, Eric Sorenson <eric.sorenson@me.com> wrote:> > Still working out why the CRL is fatal. I had actually suspected this > because of my earlier trouble with it (#4948) but did not see the VIP > (nor the endpoints) certs in the CRL so I discounted it. >Thanks for the follow up Eric, please let us know if you figure this out. I suspect I''m going to run into this as well and may have working with someone in training yesterday. -Jeff -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Feb 18, 2011, at 5:46 PM, Jeff McCune wrote:> > Thanks for the follow up Eric, please let us know if you figure this > out. I suspect I''m going to run into this as well and may have > working with someone in training yesterday.I believe it''s is related to changing the name of the issuer for the CA, from here: http://www.mail-archive.com/puppet-dev@googlegroups.com/msg09176.html When I have an empty CRL, the clients work fine. The crl looks like: # openssl crl -noout -in ca_crl.pem -text Certificate Revocation List (CRL): Version 2 (0x1) Signature Algorithm: sha1WithRSAEncryption Issuer: /CN=ca Last Update: Feb 19 01:46:28 2011 GMT Next Update: Feb 18 01:46:28 2016 GMT CRL extensions: X509v3 CRL Number: 0 No Revoked Certificates. and it passes validation: # openssl crl -noout -CAfile ./ca_crt.pem -in ca_crl.pem -issuer verify OK issuer=/CN=ca As soon as I --clean a client, the CRL gets rewritten and starts failing: # openssl crl -noout -in ca_crl_fatal.pem -text Certificate Revocation List (CRL): Version 2 (0x1) Signature Algorithm: sha1WithRSAEncryption Issuer: /CN=Puppet CA: puppetmaster001 Last Update: Feb 19 01:21:00 2011 GMT Next Update: Feb 18 01:21:00 2016 GMT CRL extensions: X509v3 CRL Number: 7 [some revoked certs here] # openssl crl -noout -CAfile ./ca_crt.pem -in ca_crl_fatal.pem -text Error getting CRL issuer certificate Could it be that the issuer name change is causing the ssl client libraries to fail to match up the CRL with the issuing CA? - Eric Sorenson - N37 17.255 W121 55.738 - http://twitter.com/ahpook - -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 02/19/2011 03:43 AM, Eric Sorenson wrote:> > On Feb 18, 2011, at 5:46 PM, Jeff McCune wrote: >> >> Thanks for the follow up Eric, please let us know if you figure this >> out. I suspect I''m going to run into this as well and may have >> working with someone in training yesterday. > > I believe it''s is related to changing the name of the issuer for the CA, from here: > > http://www.mail-archive.com/puppet-dev@googlegroups.com/msg09176.html > > When I have an empty CRL, the clients work fine. The crl looks like: > > # openssl crl -noout -in ca_crl.pem -text > Certificate Revocation List (CRL): > Version 2 (0x1) > Signature Algorithm: sha1WithRSAEncryption > Issuer: /CN=ca > Last Update: Feb 19 01:46:28 2011 GMT > Next Update: Feb 18 01:46:28 2016 GMT > CRL extensions: > X509v3 CRL Number: > 0 > No Revoked Certificates. > > and it passes validation: > # openssl crl -noout -CAfile ./ca_crt.pem -in ca_crl.pem -issuer > verify OK > issuer=/CN=ca > > > As soon as I --clean a client, the CRL gets rewritten and starts failing: > # openssl crl -noout -in ca_crl_fatal.pem -text > Certificate Revocation List (CRL): > Version 2 (0x1) > Signature Algorithm: sha1WithRSAEncryption > Issuer: /CN=Puppet CA: puppetmaster001 > Last Update: Feb 19 01:21:00 2011 GMT > Next Update: Feb 18 01:21:00 2016 GMT > CRL extensions: > X509v3 CRL Number: > 7 > [some revoked certs here] > # openssl crl -noout -CAfile ./ca_crt.pem -in ca_crl_fatal.pem -text > Error getting CRL issuer certificate > > Could it be that the issuer name change is causing the ssl client libraries to fail to match up the CRL with the issuing CA?Definitely. This CRL is not related to your CA cert in any way. It''s probably a bug that the CRL is created this way. Can you find a CA cert with the Subject CN as in the Issuer field above anywhere on your system? Regards, Felix -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Feb 21, 5:53 am, Felix Frank <felix.fr...@alumni.tu-berlin.de> wrote:> On 02/19/2011 03:43 AM, Eric Sorenson wrote: > > > Could it be that the issuer name change is causing the ssl client libraries to fail to match up the CRL with the issuing CA? > > Definitely. This CRL is not related to your CA cert in any way. It''s > probably a bug that the CRL is created this way. > > Can you find a CA cert with the Subject CN as in the Issuer field above > anywhere on your system?Hah, got it. There was not a *cert* with that Subject. There was however, a CRL, outside of the CA area, which puppet-cert read and reused the Issuer field to create the new CRL. I think this is a bug because the client and CA SSL areas ought to be distinct, but it''s probably quite rare for this situation to arise. I updated my previous "bad ssl error messages" bug with this case and hope it will help other people. If I had to hazard a guess as to why it does this, it looks like the ssl/certificate_revocation_list.rb uses the same indirector file location (:hostcrl from indirector/certificate_revocation_list/ file.rb) whether its reading or writing a CRL. But I don''t actually understand the indirector so this could be completely off-base. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.