John Warburton
2010-Sep-01 05:47 UTC
[Puppet Users] Puppet Scalability - Centralised Puppet SSL Cert Issues
Hi All I am trying to use the section on Centralised Puppet Infrastructure on the Scaling Puppet page - http://projects.puppetlabs.com/projects/1/wiki/Puppet_Scalability No matter what I do, I always end up with the client contacting a puppet server and rejecting the configuration with a dreaded "certificate verify failed": err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet:// engnsvr002.example.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed I have started from completely fresh servers, and repeated this behavior a number of times, with clean puppet configs - you can see a very detailed working below. I am stumped as to what to do next, but suspect a number of things: - the example given was for Mongrel - is Passenger different? - there are a number SSL cert chaining tickets in the issues list My goal is to have any puppet client be able to talk to any puppet server, so that if one.s designated puppet server died, we could repoint its CNAME to another puppet server in another datacentre and the client would continue working as if nothing happened. Does anyone have a working configuration that fits this scenario? Thanks John I have Solaris 10 Update 8 0.25.5 puppeteer, client and server, and Apache 2.2.15 with rack and the following gems: fastthread (1.0.7) passenger (2.2.14) rack (1.1.0) rake (0.8.7) I start with a clean config on my puppeteer: cornadm010# nslookup puppet.example.com Server: 1.2.3.4 Address: 4.5.6.7#53 puppet.example.com canonical name = cornadm010.example.com. Name: cornadm010.example.com cornadm010# /opt/local/sbin/puppetmasterd --server puppet.example.com--certname puppet.example.com --certdnsname `uname -n`.example.com:puppet.example.com--genconfig --vardir=/local/puppet/var --confdir=/local/puppet/etc --pluginsync --ssl_client_header=SSL_CLIENT_S_DN --ssl_client_verify_header=SSL_CLIENT_VERIFY --reports store --autosign /local/puppet/etc/autosign.conf --node_terminus exec --external_nodes /local/puppet/bin/node_classifier.pl | sed -e ''s/genconfig = true/genconfig = false/'' > /local/puppet/etc/puppetmasterd.conf cornadm010# \rm -rf /local/puppet/etc/ssl root@cornadm010# /opt/local/sbin/puppetmasterd --no-daemonize --verbose --config /local/puppet/etc/puppetmasterd.conf info: Creating a new SSL key for ca info: Creating a new SSL certificate request for ca notice: Signed certificate request for ca notice: Rebuilding inventory file info: Creating a new certificate revocation list info: Creating a new SSL key for puppet.example.com info: Creating a new SSL certificate request for puppet.example.com notice: puppet.example.com has a waiting certificate request info: authstore: defaulting to no access for puppet.example.com notice: Signed certificate request for puppet.example.com notice: Removing file Puppet::SSL::CertificateRequest puppet.example.com at ''/local/puppet/etc/ssl/ca/requests/puppet.example.com.pem'' notice: Removing file Puppet::SSL::CertificateRequest puppet.example.com at ''/local/puppet/etc/ssl/certificate_requests/puppet.example.com.pem'' notice: Starting Puppet server version 0.25.5 root@engnsvr002# /opt/local/sbin/puppetmasterd --server `uname -n`. example.com --certname `uname -n`.example.com --certdnsname `uname -n`. example.com --genconfig --vardir=/local/puppet/var --confdir=/local/puppet/etc --pluginsync --ssl_client_header=SSL_CLIENT_S_DN --ssl_client_verify_header=SSL_CLIENT_VERIFY --reports store --autosign /local/puppet/etc/autosign.conf --node_terminus exec --external_nodes /local/puppet/bin/node_classifier.pl | sed -e ''s/genconfig = true/genconfig = false/'' > /local/puppet/etc/puppetmasterd.conf root@engnsvr002# \rm -rf /local/puppet/etc/ssl root@engnsvr002# /opt/local/sbin/puppetmasterd --no-daemonize --verbose --config /local/puppet/etc/puppetmasterd.conf info: Creating a new SSL key for ca info: Creating a new SSL certificate request for ca notice: Signed certificate request for ca notice: Rebuilding inventory file info: Creating a new certificate revocation list info: Creating a new SSL key for engnsvr002.example.com info: Creating a new SSL certificate request for engnsvr002.example.com notice: engnsvr002.example.com has a waiting certificate request notice: Signed certificate request for engnsvr002.example.com notice: Removing file Puppet::SSL::CertificateRequest engnsvr002.example.comat ''/local/puppet/etc/ssl/ca/requests/engnsvr002.example.com.pem'' notice: Removing file Puppet::SSL::CertificateRequest engnsvr002.example.comat ''/local/puppet/etc/ssl/certificate_requests/engnsvr002.example.com.pem'' notice: Starting Puppet server version 0.25.5 root@engnsvr002# egrep example.com /tmp/openssl.cnf commonName = engnsvr002.example.com nsCaRevocationUrl = https://puppet.example.com/ca_crl.pem root@engnsvr002# openssl req -new -nodes -key /local/puppet/etc/ssl/ca/ca_key.pem -config /tmp/openssl.cnf -out /tmp/`uname -n`.example.com.csr -passin file:/local/puppet/etc/ssl/ca/private/ca.pass puppet@cornadm010% scp root@engnsvr002:/tmp/engnsvr002.example.com.csr . puppet@cornadm010% touch /local/puppet/etc/ssl/index puppet@cornadm010% egrep example.com /tmp/openssl.cnf commonName = puppet.example.com nsCaRevocationUrl = https://puppet.example.com/ca_crl.pem puppet@cornadm010% /opt/local/bin/openssl ca -config /tmp/openssl.cnf -extfile /tmp/openssl.cnf -extensions v3_ca -in engnsvr002.example.com.csr -out engnsvr002.example.com.pem -passin file:/local/puppet/etc/ssl/ca/private/ca.pass -batch Using configuration from /tmp/openssl.cnf Check that the request matches the signature Signature ok Certificate Details: Serial Number: 3 (0x3) Validity Not Before: Sep 1 05:09:00 2010 GMT Not After : Aug 29 05:09:00 2020 GMT Subject: commonName = engnsvr002.example.com X509v3 extensions: X509v3 Subject Key Identifier: 70:86:83:1E:C0:73:53:F8:3D:98:BD:58:C8:A7:49:E9:81:70:2F:C3 X509v3 Authority Key Identifier: keyid:FC:86:06:92:FB:99:75:EC:58:F2:83:F7:50:77:38:6F:17:62:04:74 DirName:/CN=ca serial:01 X509v3 Basic Constraints: critical CA:TRUE X509v3 Key Usage: Certificate Sign, CRL Sign Certificate is to be certified until Aug 29 05:09:00 2020 GMT (3650 days) Write out database with 1 new entries Data Base Updated puppet@cornadm010% scp engnsvr002.example.com.pem root@engnsvr002 :/tmp/engnsvr002.example.com.pem root@engnsvr002# cp /local/puppet/etc/ssl/ca/ca_crt.pem /local/puppet/etc/ssl/ca/ca_crt.pem.orig root@engnsvr002# cp /tmp/`uname -n`.example.com.pem /local/puppet/etc/ssl/ca/ca_crt.pem puppet@cornadm010% cat ssl/ca/ca_crt.pem -----BEGIN CERTIFICATE----- MIICCTCCAXKgAwIBAgIBATANBgkqhkiG9w0BAQUFADANMQswCQYDVQQDDAJjYTAe Fw0xMDA4MzEwMjU0MjBaFw0xNTA4MzAwMjU0MjBaMA0xCzAJBgNVBAMMAmNhMIGf MA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCuPbG6LHp/5nIEPMFQbuiqUGHedrRc 5aKJpWOAqXvAiVXnwYP6vBl+jVlxCJG4xHVaLcIIp1lHVBweyz8VwZ/aw60/2333 6v6GsLo4UYrz9a/SWKT4JNPQABBvbY/8rU7H/Yuvop3nhXBbQVMtvqCgQDFpkpx2 KYz2zXi6MJoiMQIDAQABo3kwdzA4BglghkgBhvhCAQ0EKxYpUHVwcGV0IFJ1Ynkv T3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwDwYDVR0TAQH/BAUwAwEB/zAd BgNVHQ4EFgQU/IYGkvuZdexY8oP3UHc4bxdiBHQwCwYDVR0PBAQDAgEGMA0GCSqG SIb3DQEBBQUAA4GBAEk7XQV7ohMMFjzJnd+AVc/VJaw7QAUdtjJYPthlBZKv4guO iy9BpSLZn2ChHNh1ANBAnRGIIFzljMHN6i4MXhhzfxKk6Vz0sAg74A3dE2Ots8F4 BF4BtunVFt7fyTPw/GFf3UibTM1xRXRpHq79fM5XTiuSu71pxQDCclYP2MPH -----END CERTIFICATE----- engnsvr003# vi /var/puppet/confdir/ssl/certs/ca.pem <with above> puppet@cornadm010% grep ^ServerName /local/apache-infra/conf/httpd.conf ServerName puppet.example.com:80 puppet@cornadm010% less /local/apache-infra/conf.d/puppetmasterd.conf <VirtualHost *:8140> ServerName puppet.example.com SSLEngine on SSLProtocol -ALL +SSLv3 +TLSv1 SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP SSLCertificateFile /local/puppet/etc/ssl/certs/puppet.example.com.pem SSLCertificateKeyFile /local/puppet/etc/ssl/private_keys/puppet.example.com.pem SSLCertificateChainFile /local/puppet/etc/ssl/ca/ca_crt.pem SSLCACertificateFile /local/puppet/etc/ssl/ca/ca_crt.pem # If Apache complains about invalid signatures on the CRL, you can try disabling # CRL checking by commenting the next line, but this is not recommended. #SSLCARevocationFile /local/puppet/etc/ssl/ca/ca_crl.pem SSLVerifyClient optional SSLVerifyDepth 1 SSLOptions +StdEnvVars cornadm010# svcadm restart apache-infra root@engnsvr002# grep ^ServerName /local/apache-infra/conf/httpd.conf ServerName engnsvr002.example.com:80 root@engnsvr002# less /local/apache-infra/conf.d/puppetmasterd.conf <VirtualHost *:8140> ServerName engnsvr002.example.com SSLEngine on SSLProtocol -ALL +SSLv3 +TLSv1 SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP SSLCertificateFile /local/puppet/etc/ssl/certs/engnsvr002.example.com.pem SSLCertificateKeyFile /local/puppet/etc/ssl/private_keys/engnsvr002.example.com.pem SSLCertificateChainFile /local/puppet/etc/ssl/ca/ca_crt.pem SSLCACertificateFile /local/puppet/etc/ssl/ca/ca_crt.pem # If Apache complains about invalid signatures on the CRL, you can try disabling # CRL checking by commenting the next line, but this is not recommended. #SSLCARevocationFile /local/puppet/etc/ssl/ca/ca_crl.pem SSLVerifyClient optional SSLVerifyDepth 1 SSLOptions +StdEnvVars root@engnsvr003# mkdir /var/puppet/confdir root@engnsvr003# /opt/local/sbin/puppetd --confdir /var/puppet/confdir --vardir /var/puppet/confdir/var --server engnsvr002.example.com--pluginsync --report --genconfig | sed -e ''s/genconfig = true/genconfig false/'' > /var/puppet/confdir/puppetd.conf root@engnsvr003# mkdir -p /var/puppet/confdir/ssl/certs root@engnsvr003# /opt/local/sbin/puppetd --verbose --onetime --no-daemonize --ignorecache --no-usecacheonfailure --config /var/puppet/confdir/puppetd.conf --environment lab --debug info: Creating a new SSL key for engnsvr003.example.com debug: Using cached certificate for ca warning: peer certificate won''t be verified in this SSL session warning: peer certificate won''t be verified in this SSL session info: Creating a new SSL certificate request for engnsvr003.example.com warning: peer certificate won''t be verified in this SSL session debug: Using cached certificate for ca warning: peer certificate won''t be verified in this SSL session info: Caching certificate for engnsvr003.example.com debug: Finishing transaction 7818336 with 0 changes info: Retrieving plugin debug: Using cached certificate for ca debug: Using cached certificate for engnsvr003.example.com err: /File[/var/puppet/confdir/var/lib]: Failed to generate additional resources using ''eval_generate'': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet:// engnsvr002.example.com/plugins: SSL_connect re turned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed debug: Finishing transaction 7755204 with 0 changes debug: catalog supports formats: b64_zlib_yaml marshal pson raw yaml; using pson err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed warning: Not using cache on failed catalog err: Could not retrieve catalog; skipping run Delete & recreate ssl dirs on 002 & 003 with no chained cert, and all is OK: root@engnsvr003# /opt/local/sbin/puppetd --verbose --onetime --no-daemonize --ignorecache --no-usecacheonfailure --config /var/puppet/confdir/puppetd.conf --environment lab notice: running from engnsvr002.example.com on engnsvr003.example.com notice: //Notify[running from engnsvr002.example.com on engnsvr003.example.com]/message: defined ''message'' as ''running from engnsvr002.example.com on engnsvr003.example.com'' -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Patrick
2010-Sep-01 06:14 UTC
Re: [Puppet Users] Puppet Scalability - Centralised Puppet SSL Cert Issues
On Aug 31, 2010, at 10:47 PM, John Warburton wrote:> Hi All > > I am trying to use the section on Centralised Puppet Infrastructure on the Scaling Puppet page - http://projects.puppetlabs.com/projects/1/wiki/Puppet_Scalability > > No matter what I do, I always end up with the client contacting a puppet server and rejecting the configuration with a dreaded "certificate verify failed": > > err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet://engnsvr002.example.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed > > I have started from completely fresh servers, and repeated this behavior a number of times, with clean puppet configs - you can see a very detailed working below. > > I am stumped as to what to do next, but suspect a number of things: > - the example given was for Mongrel - is Passenger different? > - there are a number SSL cert chaining tickets in the issues list > > My goal is to have any puppet client be able to talk to any puppet server, so that if one.s designated puppet server died, we could repoint its CNAME to another puppet server in another datacentre and the client would continue working as if nothing happened. Does anyone have a working configuration that fits this scenario?I''ve done it 2 ways. 1) Just copy the ca folder to the other servers. (Warning, breaks certificate revocation because of duplicate serial numbers) 2) Use one server as the ca for everything, but have local servers for everything else. (Not as much reliability, but close. You can''t sign when the ca goes down, but everything else works.) I have tried using that method, but I''ve had horrible luck and didn''t manage to make it work. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ohad Levy
2010-Sep-01 06:37 UTC
Re: [Puppet Users] Puppet Scalability - Centralised Puppet SSL Cert Issues
There is an open bug with 0.25.x (and 2.6) which breaks certificate chaining. this works well for the 0.24.x series, and I hope that will work again sometime in the near future with 2.6.x series. I would recommend you at the moment to use one machine as the CA, if you can accept the fact that its a single point of failure for creating new certificates. Ohad On Wed, Sep 1, 2010 at 9:14 AM, Patrick <kc7zzv@gmail.com> wrote:> > On Aug 31, 2010, at 10:47 PM, John Warburton wrote: > > Hi All > > I am trying to use the section on Centralised Puppet Infrastructure on the > Scaling Puppet page - > http://projects.puppetlabs.com/projects/1/wiki/Puppet_Scalability > > No matter what I do, I always end up with the client contacting a puppet > server and rejecting the configuration with a dreaded "certificate verify > failed": > > err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state > of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server > certificate B: certificate verify failed Could not retrieve file metadata > for puppet://engnsvr002.example.com/plugins: SSL_connect returned=1 > errno=0 state=SSLv3 read server certificate B: certificate verify failed > > I have started from completely fresh servers, and repeated this behavior a > number of times, with clean puppet configs - you can see a very detailed > working below. > > I am stumped as to what to do next, but suspect a number of things: > - the example given was for Mongrel - is Passenger different? > - there are a number SSL cert chaining tickets in the issues list > > My goal is to have any puppet client be able to talk to any puppet server, > so that if one.s designated puppet server died, we could repoint its CNAME > to another puppet server in another datacentre and the client would continue > working as if nothing happened. Does anyone have a working configuration > that fits this scenario? > > > I''ve done it 2 ways. > 1) Just copy the ca folder to the other servers. (Warning, breaks > certificate revocation because of duplicate serial numbers) > 2) Use one server as the ca for everything, but have local servers for > everything else. (Not as much reliability, but close. You can''t sign when > the ca goes down, but everything else works.) > > I have tried using that method, but I''ve had horrible luck and didn''t > manage to make it work. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
John Warburton
2010-Sep-01 23:54 UTC
Re: [Puppet Users] Puppet Scalability - Centralised Puppet SSL Cert Issues
Thanks Ohad I have updated the Wiki entry with a warning (where''s the <blink> tag?) and references to the bugs on certificate chaining I''m not 100% comfortable with a single CA, so would it be possible to do the following: ca_server = puppet-ca.example.com rsync the ssl dir every 5 minutes to puppet-ca2.example.com If puppet-ca dies, I would swing the CNAME over to puppet-ca2.example.com Thanks John On 1 September 2010 16:37, Ohad Levy <ohadlevy@gmail.com> wrote:> There is an open bug with 0.25.x (and 2.6) which breaks certificate > chaining. > this works well for the 0.24.x series, and I hope that will work again > sometime in the near future with 2.6.x series. > > I would recommend you at the moment to use one machine as the CA, if you > can accept the fact that its a single point of failure for creating new > certificates. > > Ohad > > > On Wed, Sep 1, 2010 at 9:14 AM, Patrick <kc7zzv@gmail.com> wrote: > >> >> On Aug 31, 2010, at 10:47 PM, John Warburton wrote: >> >> Hi All >> >> I am trying to use the section on Centralised Puppet Infrastructure on the >> Scaling Puppet page - >> http://projects.puppetlabs.com/projects/1/wiki/Puppet_Scalability >> >> No matter what I do, I always end up with the client contacting a puppet >> server and rejecting the configuration with a dreaded "certificate verify >> failed": >> >> err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state >> of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server >> certificate B: certificate verify failed Could not retrieve file metadata >> for puppet://engnsvr002.example.com/plugins: SSL_connect returned=1 >> errno=0 state=SSLv3 read server certificate B: certificate verify failed >> >> I have started from completely fresh servers, and repeated this behavior a >> number of times, with clean puppet configs - you can see a very detailed >> working below. >> >> I am stumped as to what to do next, but suspect a number of things: >> - the example given was for Mongrel - is Passenger different? >> - there are a number SSL cert chaining tickets in the issues list >> >> My goal is to have any puppet client be able to talk to any puppet server, >> so that if one.s designated puppet server died, we could repoint its CNAME >> to another puppet server in another datacentre and the client would continue >> working as if nothing happened. Does anyone have a working configuration >> that fits this scenario? >> >> >> I''ve done it 2 ways. >> 1) Just copy the ca folder to the other servers. (Warning, breaks >> certificate revocation because of duplicate serial numbers) >> 2) Use one server as the ca for everything, but have local servers for >> everything else. (Not as much reliability, but close. You can''t sign when >> the ca goes down, but everything else works.) >> >> I have tried using that method, but I''ve had horrible luck and didn''t >> manage to make it work. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- John Warburton Ph: 0417 299 600 Email: jwarburton@gmail.com -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.