John Warburton
2010-Sep-01 05:47 UTC
[Puppet Users] Puppet Scalability - Centralised Puppet SSL Cert Issues
Hi All
I am trying to use the section on Centralised Puppet Infrastructure on the
Scaling Puppet page -
http://projects.puppetlabs.com/projects/1/wiki/Puppet_Scalability
No matter what I do, I always end up with the client contacting a puppet
server and rejecting the configuration with a dreaded "certificate verify
failed":
err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state of
resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate
B: certificate verify failed Could not retrieve file metadata for puppet://
engnsvr002.example.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3
read server certificate B: certificate verify failed
I have started from completely fresh servers, and repeated this behavior a
number of times, with clean puppet configs - you can see a very detailed
working below.
I am stumped as to what to do next, but suspect a number of things:
- the example given was for Mongrel - is Passenger different?
- there are a number SSL cert chaining tickets in the issues list
My goal is to have any puppet client be able to talk to any puppet server,
so that if one.s designated puppet server died, we could repoint its CNAME
to another puppet server in another datacentre and the client would continue
working as if nothing happened. Does anyone have a working configuration
that fits this scenario?
Thanks
John
I have Solaris 10 Update 8 0.25.5 puppeteer, client and server, and Apache
2.2.15 with rack and the following gems:
fastthread (1.0.7)
passenger (2.2.14)
rack (1.1.0)
rake (0.8.7)
I start with a clean config on my puppeteer:
cornadm010# nslookup puppet.example.com
Server: 1.2.3.4
Address: 4.5.6.7#53
puppet.example.com canonical name = cornadm010.example.com.
Name: cornadm010.example.com
cornadm010# /opt/local/sbin/puppetmasterd --server puppet.example.com--certname
puppet.example.com --certdnsname `uname
-n`.example.com:puppet.example.com--genconfig
--vardir=/local/puppet/var --confdir=/local/puppet/etc
--pluginsync --ssl_client_header=SSL_CLIENT_S_DN
--ssl_client_verify_header=SSL_CLIENT_VERIFY --reports store --autosign
/local/puppet/etc/autosign.conf --node_terminus exec --external_nodes
/local/puppet/bin/node_classifier.pl | sed -e ''s/genconfig =
true/genconfig
= false/'' > /local/puppet/etc/puppetmasterd.conf
cornadm010# \rm -rf /local/puppet/etc/ssl
root@cornadm010# /opt/local/sbin/puppetmasterd --no-daemonize --verbose
--config /local/puppet/etc/puppetmasterd.conf
info: Creating a new SSL key for ca
info: Creating a new SSL certificate request for ca
notice: Signed certificate request for ca
notice: Rebuilding inventory file
info: Creating a new certificate revocation list
info: Creating a new SSL key for puppet.example.com
info: Creating a new SSL certificate request for puppet.example.com
notice: puppet.example.com has a waiting certificate request
info: authstore: defaulting to no access for puppet.example.com
notice: Signed certificate request for puppet.example.com
notice: Removing file Puppet::SSL::CertificateRequest puppet.example.com at
''/local/puppet/etc/ssl/ca/requests/puppet.example.com.pem''
notice: Removing file Puppet::SSL::CertificateRequest puppet.example.com at
''/local/puppet/etc/ssl/certificate_requests/puppet.example.com.pem''
notice: Starting Puppet server version 0.25.5
root@engnsvr002# /opt/local/sbin/puppetmasterd --server `uname -n`.
example.com --certname `uname -n`.example.com --certdnsname `uname -n`.
example.com --genconfig --vardir=/local/puppet/var
--confdir=/local/puppet/etc --pluginsync --ssl_client_header=SSL_CLIENT_S_DN
--ssl_client_verify_header=SSL_CLIENT_VERIFY --reports store --autosign
/local/puppet/etc/autosign.conf --node_terminus exec --external_nodes
/local/puppet/bin/node_classifier.pl | sed -e ''s/genconfig =
true/genconfig
= false/'' > /local/puppet/etc/puppetmasterd.conf
root@engnsvr002# \rm -rf /local/puppet/etc/ssl
root@engnsvr002# /opt/local/sbin/puppetmasterd --no-daemonize --verbose
--config /local/puppet/etc/puppetmasterd.conf
info: Creating a new SSL key for ca
info: Creating a new SSL certificate request for ca
notice: Signed certificate request for ca
notice: Rebuilding inventory file
info: Creating a new certificate revocation list
info: Creating a new SSL key for engnsvr002.example.com
info: Creating a new SSL certificate request for engnsvr002.example.com
notice: engnsvr002.example.com has a waiting certificate request
notice: Signed certificate request for engnsvr002.example.com
notice: Removing file Puppet::SSL::CertificateRequest
engnsvr002.example.comat
''/local/puppet/etc/ssl/ca/requests/engnsvr002.example.com.pem''
notice: Removing file Puppet::SSL::CertificateRequest
engnsvr002.example.comat
''/local/puppet/etc/ssl/certificate_requests/engnsvr002.example.com.pem''
notice: Starting Puppet server version 0.25.5
root@engnsvr002# egrep example.com /tmp/openssl.cnf
commonName = engnsvr002.example.com
nsCaRevocationUrl = https://puppet.example.com/ca_crl.pem
root@engnsvr002# openssl req -new -nodes -key
/local/puppet/etc/ssl/ca/ca_key.pem -config /tmp/openssl.cnf -out
/tmp/`uname -n`.example.com.csr -passin
file:/local/puppet/etc/ssl/ca/private/ca.pass
puppet@cornadm010% scp root@engnsvr002:/tmp/engnsvr002.example.com.csr .
puppet@cornadm010% touch /local/puppet/etc/ssl/index
puppet@cornadm010% egrep example.com /tmp/openssl.cnf
commonName = puppet.example.com
nsCaRevocationUrl = https://puppet.example.com/ca_crl.pem
puppet@cornadm010% /opt/local/bin/openssl ca -config /tmp/openssl.cnf
-extfile /tmp/openssl.cnf -extensions v3_ca -in engnsvr002.example.com.csr
-out engnsvr002.example.com.pem -passin
file:/local/puppet/etc/ssl/ca/private/ca.pass -batch
Using configuration from /tmp/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 3 (0x3)
Validity
Not Before: Sep 1 05:09:00 2010 GMT
Not After : Aug 29 05:09:00 2020 GMT
Subject:
commonName = engnsvr002.example.com
X509v3 extensions:
X509v3 Subject Key Identifier:
70:86:83:1E:C0:73:53:F8:3D:98:BD:58:C8:A7:49:E9:81:70:2F:C3
X509v3 Authority Key Identifier:
keyid:FC:86:06:92:FB:99:75:EC:58:F2:83:F7:50:77:38:6F:17:62:04:74
DirName:/CN=ca
serial:01
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Key Usage:
Certificate Sign, CRL Sign
Certificate is to be certified until Aug 29 05:09:00 2020 GMT (3650 days)
Write out database with 1 new entries
Data Base Updated
puppet@cornadm010% scp engnsvr002.example.com.pem root@engnsvr002
:/tmp/engnsvr002.example.com.pem
root@engnsvr002# cp /local/puppet/etc/ssl/ca/ca_crt.pem
/local/puppet/etc/ssl/ca/ca_crt.pem.orig
root@engnsvr002# cp /tmp/`uname -n`.example.com.pem
/local/puppet/etc/ssl/ca/ca_crt.pem
puppet@cornadm010% cat ssl/ca/ca_crt.pem
-----BEGIN CERTIFICATE-----
MIICCTCCAXKgAwIBAgIBATANBgkqhkiG9w0BAQUFADANMQswCQYDVQQDDAJjYTAe
Fw0xMDA4MzEwMjU0MjBaFw0xNTA4MzAwMjU0MjBaMA0xCzAJBgNVBAMMAmNhMIGf
MA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCuPbG6LHp/5nIEPMFQbuiqUGHedrRc
5aKJpWOAqXvAiVXnwYP6vBl+jVlxCJG4xHVaLcIIp1lHVBweyz8VwZ/aw60/2333
6v6GsLo4UYrz9a/SWKT4JNPQABBvbY/8rU7H/Yuvop3nhXBbQVMtvqCgQDFpkpx2
KYz2zXi6MJoiMQIDAQABo3kwdzA4BglghkgBhvhCAQ0EKxYpUHVwcGV0IFJ1Ynkv
T3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwDwYDVR0TAQH/BAUwAwEB/zAd
BgNVHQ4EFgQU/IYGkvuZdexY8oP3UHc4bxdiBHQwCwYDVR0PBAQDAgEGMA0GCSqG
SIb3DQEBBQUAA4GBAEk7XQV7ohMMFjzJnd+AVc/VJaw7QAUdtjJYPthlBZKv4guO
iy9BpSLZn2ChHNh1ANBAnRGIIFzljMHN6i4MXhhzfxKk6Vz0sAg74A3dE2Ots8F4
BF4BtunVFt7fyTPw/GFf3UibTM1xRXRpHq79fM5XTiuSu71pxQDCclYP2MPH
-----END CERTIFICATE-----
engnsvr003# vi /var/puppet/confdir/ssl/certs/ca.pem
<with above>
puppet@cornadm010% grep ^ServerName /local/apache-infra/conf/httpd.conf
ServerName puppet.example.com:80
puppet@cornadm010% less /local/apache-infra/conf.d/puppetmasterd.conf
<VirtualHost *:8140>
ServerName puppet.example.com
SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile
/local/puppet/etc/ssl/certs/puppet.example.com.pem
SSLCertificateKeyFile
/local/puppet/etc/ssl/private_keys/puppet.example.com.pem
SSLCertificateChainFile /local/puppet/etc/ssl/ca/ca_crt.pem
SSLCACertificateFile /local/puppet/etc/ssl/ca/ca_crt.pem
# If Apache complains about invalid signatures on the CRL, you can
try disabling
# CRL checking by commenting the next line, but this is not
recommended.
#SSLCARevocationFile /local/puppet/etc/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars
cornadm010# svcadm restart apache-infra
root@engnsvr002# grep ^ServerName /local/apache-infra/conf/httpd.conf
ServerName engnsvr002.example.com:80
root@engnsvr002# less /local/apache-infra/conf.d/puppetmasterd.conf
<VirtualHost *:8140>
ServerName engnsvr002.example.com
SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile
/local/puppet/etc/ssl/certs/engnsvr002.example.com.pem
SSLCertificateKeyFile
/local/puppet/etc/ssl/private_keys/engnsvr002.example.com.pem
SSLCertificateChainFile /local/puppet/etc/ssl/ca/ca_crt.pem
SSLCACertificateFile /local/puppet/etc/ssl/ca/ca_crt.pem
# If Apache complains about invalid signatures on the CRL, you can
try disabling
# CRL checking by commenting the next line, but this is not
recommended.
#SSLCARevocationFile /local/puppet/etc/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars
root@engnsvr003# mkdir /var/puppet/confdir
root@engnsvr003# /opt/local/sbin/puppetd --confdir /var/puppet/confdir
--vardir /var/puppet/confdir/var --server
engnsvr002.example.com--pluginsync --report --genconfig | sed -e
''s/genconfig = true/genconfig false/'' >
/var/puppet/confdir/puppetd.conf
root@engnsvr003# mkdir -p /var/puppet/confdir/ssl/certs
root@engnsvr003# /opt/local/sbin/puppetd --verbose --onetime --no-daemonize
--ignorecache --no-usecacheonfailure --config
/var/puppet/confdir/puppetd.conf --environment lab --debug
info: Creating a new SSL key for engnsvr003.example.com
debug: Using cached certificate for ca
warning: peer certificate won''t be verified in this SSL session
warning: peer certificate won''t be verified in this SSL session
info: Creating a new SSL certificate request for engnsvr003.example.com
warning: peer certificate won''t be verified in this SSL session
debug: Using cached certificate for ca
warning: peer certificate won''t be verified in this SSL session
info: Caching certificate for engnsvr003.example.com
debug: Finishing transaction 7818336 with 0 changes
info: Retrieving plugin
debug: Using cached certificate for ca
debug: Using cached certificate for engnsvr003.example.com
err: /File[/var/puppet/confdir/var/lib]: Failed to generate additional
resources using ''eval_generate'': SSL_connect returned=1
errno=0 state=SSLv3
read server certificate B: certificate verify failed
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml;
using pson
err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state of
resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate
B: certificate verify failed Could not retrieve file metadata for puppet://
engnsvr002.example.com/plugins: SSL_connect re
turned=1 errno=0 state=SSLv3 read server certificate B: certificate verify
failed
debug: Finishing transaction 7755204 with 0 changes
debug: catalog supports formats: b64_zlib_yaml marshal pson raw yaml; using
pson
err: Could not retrieve catalog from remote server: SSL_connect returned=1
errno=0 state=SSLv3 read server certificate B: certificate verify failed
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
Delete & recreate ssl dirs on 002 & 003 with no chained cert, and all is
OK:
root@engnsvr003# /opt/local/sbin/puppetd --verbose --onetime --no-daemonize
--ignorecache --no-usecacheonfailure --config
/var/puppet/confdir/puppetd.conf --environment lab
notice: running from engnsvr002.example.com on engnsvr003.example.com
notice: //Notify[running from engnsvr002.example.com on
engnsvr003.example.com]/message: defined ''message'' as
''running from
engnsvr002.example.com on engnsvr003.example.com''
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Patrick
2010-Sep-01 06:14 UTC
Re: [Puppet Users] Puppet Scalability - Centralised Puppet SSL Cert Issues
On Aug 31, 2010, at 10:47 PM, John Warburton wrote:> Hi All > > I am trying to use the section on Centralised Puppet Infrastructure on the Scaling Puppet page - http://projects.puppetlabs.com/projects/1/wiki/Puppet_Scalability > > No matter what I do, I always end up with the client contacting a puppet server and rejecting the configuration with a dreaded "certificate verify failed": > > err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet://engnsvr002.example.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed > > I have started from completely fresh servers, and repeated this behavior a number of times, with clean puppet configs - you can see a very detailed working below. > > I am stumped as to what to do next, but suspect a number of things: > - the example given was for Mongrel - is Passenger different? > - there are a number SSL cert chaining tickets in the issues list > > My goal is to have any puppet client be able to talk to any puppet server, so that if one.s designated puppet server died, we could repoint its CNAME to another puppet server in another datacentre and the client would continue working as if nothing happened. Does anyone have a working configuration that fits this scenario?I''ve done it 2 ways. 1) Just copy the ca folder to the other servers. (Warning, breaks certificate revocation because of duplicate serial numbers) 2) Use one server as the ca for everything, but have local servers for everything else. (Not as much reliability, but close. You can''t sign when the ca goes down, but everything else works.) I have tried using that method, but I''ve had horrible luck and didn''t manage to make it work. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ohad Levy
2010-Sep-01 06:37 UTC
Re: [Puppet Users] Puppet Scalability - Centralised Puppet SSL Cert Issues
There is an open bug with 0.25.x (and 2.6) which breaks certificate chaining. this works well for the 0.24.x series, and I hope that will work again sometime in the near future with 2.6.x series. I would recommend you at the moment to use one machine as the CA, if you can accept the fact that its a single point of failure for creating new certificates. Ohad On Wed, Sep 1, 2010 at 9:14 AM, Patrick <kc7zzv@gmail.com> wrote:> > On Aug 31, 2010, at 10:47 PM, John Warburton wrote: > > Hi All > > I am trying to use the section on Centralised Puppet Infrastructure on the > Scaling Puppet page - > http://projects.puppetlabs.com/projects/1/wiki/Puppet_Scalability > > No matter what I do, I always end up with the client contacting a puppet > server and rejecting the configuration with a dreaded "certificate verify > failed": > > err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state > of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server > certificate B: certificate verify failed Could not retrieve file metadata > for puppet://engnsvr002.example.com/plugins: SSL_connect returned=1 > errno=0 state=SSLv3 read server certificate B: certificate verify failed > > I have started from completely fresh servers, and repeated this behavior a > number of times, with clean puppet configs - you can see a very detailed > working below. > > I am stumped as to what to do next, but suspect a number of things: > - the example given was for Mongrel - is Passenger different? > - there are a number SSL cert chaining tickets in the issues list > > My goal is to have any puppet client be able to talk to any puppet server, > so that if one.s designated puppet server died, we could repoint its CNAME > to another puppet server in another datacentre and the client would continue > working as if nothing happened. Does anyone have a working configuration > that fits this scenario? > > > I''ve done it 2 ways. > 1) Just copy the ca folder to the other servers. (Warning, breaks > certificate revocation because of duplicate serial numbers) > 2) Use one server as the ca for everything, but have local servers for > everything else. (Not as much reliability, but close. You can''t sign when > the ca goes down, but everything else works.) > > I have tried using that method, but I''ve had horrible luck and didn''t > manage to make it work. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
John Warburton
2010-Sep-01 23:54 UTC
Re: [Puppet Users] Puppet Scalability - Centralised Puppet SSL Cert Issues
Thanks Ohad I have updated the Wiki entry with a warning (where''s the <blink> tag?) and references to the bugs on certificate chaining I''m not 100% comfortable with a single CA, so would it be possible to do the following: ca_server = puppet-ca.example.com rsync the ssl dir every 5 minutes to puppet-ca2.example.com If puppet-ca dies, I would swing the CNAME over to puppet-ca2.example.com Thanks John On 1 September 2010 16:37, Ohad Levy <ohadlevy@gmail.com> wrote:> There is an open bug with 0.25.x (and 2.6) which breaks certificate > chaining. > this works well for the 0.24.x series, and I hope that will work again > sometime in the near future with 2.6.x series. > > I would recommend you at the moment to use one machine as the CA, if you > can accept the fact that its a single point of failure for creating new > certificates. > > Ohad > > > On Wed, Sep 1, 2010 at 9:14 AM, Patrick <kc7zzv@gmail.com> wrote: > >> >> On Aug 31, 2010, at 10:47 PM, John Warburton wrote: >> >> Hi All >> >> I am trying to use the section on Centralised Puppet Infrastructure on the >> Scaling Puppet page - >> http://projects.puppetlabs.com/projects/1/wiki/Puppet_Scalability >> >> No matter what I do, I always end up with the client contacting a puppet >> server and rejecting the configuration with a dreaded "certificate verify >> failed": >> >> err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state >> of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server >> certificate B: certificate verify failed Could not retrieve file metadata >> for puppet://engnsvr002.example.com/plugins: SSL_connect returned=1 >> errno=0 state=SSLv3 read server certificate B: certificate verify failed >> >> I have started from completely fresh servers, and repeated this behavior a >> number of times, with clean puppet configs - you can see a very detailed >> working below. >> >> I am stumped as to what to do next, but suspect a number of things: >> - the example given was for Mongrel - is Passenger different? >> - there are a number SSL cert chaining tickets in the issues list >> >> My goal is to have any puppet client be able to talk to any puppet server, >> so that if one.s designated puppet server died, we could repoint its CNAME >> to another puppet server in another datacentre and the client would continue >> working as if nothing happened. Does anyone have a working configuration >> that fits this scenario? >> >> >> I''ve done it 2 ways. >> 1) Just copy the ca folder to the other servers. (Warning, breaks >> certificate revocation because of duplicate serial numbers) >> 2) Use one server as the ca for everything, but have local servers for >> everything else. (Not as much reliability, but close. You can''t sign when >> the ca goes down, but everything else works.) >> >> I have tried using that method, but I''ve had horrible luck and didn''t >> manage to make it work. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- John Warburton Ph: 0417 299 600 Email: jwarburton@gmail.com -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.