Daniele Sluijters
2013-Oct-24 13:59 UTC
[Puppet Users] multi-master with single CA, certificate woes
Hey,
I''ve been battling this all day so I hop some people over here have
some
good ideas. I''m trying to set up a multi-master puppet environment with
a
single CA.
This is what I''ve done so far:
* node 1, the CA:
* install puppetmaster
* stop it
* wipe `vardir/ssl`
* configure certname and dns_alt_names
* start it up
* node 2, an actual master that will handle requests but no CA:
* install puppetmaster
* stop it
* wipe `vardir/ssl`
* configure certname, dns_alt_names and ca_server to point to node1, set
ca=false in the [master] block and point server to itself
* do a puppet agent -t
* sign the CSR on node1
* get acknowledgement that node2 grabbed its cert from node1
Now I want to actually make the Puppet master on node2 usable. My Puppet
masters run in Passenger Standalone with an nginx in front of them proxying
the requests on port 8140 to them.
The complete SSL configuration in nginx now looks like this:
ssl on;
ssl_certificate ssl/puppetmaster/node2.pem; # this is
`vardir/ssl/certs/node2.pem`
ssl_certificate_key ssl/puppetmaster/node2.key; # this is
`vardir/ssl/private_keys/node2.pem`
ssl_client_certificate ssl/puppetmaster/ca.pem; # this is
`vardir/ssl/certs/ca.pem`
ssl_crl ssl/puppetmaster/crl.pem; # this was
copied from the master at `vardir/ssl/crl.pem`
ssl_verify_client on; #
since we do only master, no CA we can require a client certificate
location / {
[..] bunch of other proxy_set_header directives [..]
proxy_set_header X-Client-Verify $ssl_client_verify;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-SSL-Subject $ssl_client_s_dn;
proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
}
I started the Puppet master and reloaded the nginx for the configuration to
take effect.
First I tried to connect with openssl s_client:
root@node2 # openssl s_client -connect localhost:8140 -cert
/var/lib/puppet/ssl/certs/node2.pem -key
/var/lib/puppet/ssl/private_keys/node2.pem
CONNECTED(00000003)
depth=1 CN = Puppet CA: node1
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/CN=node2
i:/CN=Puppet CA: node1
1 s:/CN=Puppet CA: node1
i:/CN=Puppet CA: node1
---
[..]
subject=/CN=node2
issuer=/CN=Puppet CA: node1
---
Acceptable client certificate CA names
/CN=Puppet CA: node1
---
SSL handshake has read 5314 bytes and written 2445 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
[..]
Start Time: 1382622609
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
---
GET /production/node/node2
Forbidden request: node2l(10.120.12.73) access to /node/node2 [find] at
:115closed
This looks fine, It complains about the self-signed certificate in the
chain but other than that verify return = 0 which as far as I''m aware
means
success. Once the handshake is complete I can actually do a GET for that
node but that seems to fail because of something in auth.conf, which is
slightly odd since the authentication succeeded so the ACL shouldn''t
trip
over it but that''s step two (I''m probably forgetting to
proxy/set a header).
Now, running `puppet agent -t` however gives me this:
Warning: Unable to fetch my node definition, but the agent run will
continue:
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate
B: certificate verify failed: [certificate signature failure for /CN=node1]
At this point I''m lost. Ntpd''s are running, nodes are within
far less than
a second in sync of each other. Certificate is valid from yesterday to
yesterday in 2018 so I doubt that''s an issue either. If it were the
`openssl s_client -connect` should have failed too but it''s obviously
perfectly happy with it.
What am I missing here? I fear it''s staring me in the face but
I''m just not
seeing it.
--
Daniele Sluijters
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.
Daniele Sluijters
2013-Oct-24 19:19 UTC
[Puppet Users] Re: multi-master with single CA, certificate woes
Hello, With a lot of help from Hunter Haugen (Puppetlabs) we managed to figure out what was wrong and fix it. The errors always pointed in the direction of SSL issues but I never managed to figure out why. Turns out this had something to do with the way I was starting the master to create the CA. The first issue we found was that ca.pem on node2 wasn''t actually matching ca_crt.pem on node1 which clued us in that something really was wrong with the way the CA was behaving. My initial approach of bootstrapping the CA server turned out to be the issue. I just cleaned the ssl dir, did a `puppet agent -t` followed by a `puppet cert list` which would automatically say something about signing the CA and considered it done. Turns out that''s the wrong way to do it. What eventually worked was wiping vardir/ssl (again) and then starting a Puppet master in the foreground: `puppet master --no-daemonize -v`. It then actually correctly bootstrapped the CA. After that everything started to work. Running `puppet agent -t --server=node1` (itself) passed and after having requests a cert for node2 and signed it that too started to work. I''ll shortly document and publish the complete configuration and send a pull request towards Puppetlabs to clarify a few things in the docs about how to achieve all this. -- Daniele Sluijters On Thursday, 24 October 2013 15:59:13 UTC+2, Daniele Sluijters wrote:> > Hey, > > I''ve been battling this all day so I hop some people over here have some > good ideas. I''m trying to set up a multi-master puppet environment with a > single CA. > > This is what I''ve done so far: > * node 1, the CA: > * install puppetmaster > * stop it > * wipe `vardir/ssl` > * configure certname and dns_alt_names > * start it up > > * node 2, an actual master that will handle requests but no CA: > * install puppetmaster > * stop it > * wipe `vardir/ssl` > * configure certname, dns_alt_names and ca_server to point to node1, set > ca=false in the [master] block and point server to itself > * do a puppet agent -t > * sign the CSR on node1 > * get acknowledgement that node2 grabbed its cert from node1 > > Now I want to actually make the Puppet master on node2 usable. My Puppet > masters run in Passenger Standalone with an nginx in front of them proxying > the requests on port 8140 to them. > > The complete SSL configuration in nginx now looks like this: > ssl on; > ssl_certificate ssl/puppetmaster/node2.pem; # this is > `vardir/ssl/certs/node2.pem` > ssl_certificate_key ssl/puppetmaster/node2.key; # this is > `vardir/ssl/private_keys/node2.pem` > ssl_client_certificate ssl/puppetmaster/ca.pem; # this is > `vardir/ssl/certs/ca.pem` > ssl_crl ssl/puppetmaster/crl.pem; # this was > copied from the master at `vardir/ssl/crl.pem` > ssl_verify_client on; # > since we do only master, no CA we can require a client certificate > > location / { > [..] bunch of other proxy_set_header directives [..] > proxy_set_header X-Client-Verify $ssl_client_verify; > proxy_set_header X-Client-DN $ssl_client_s_dn; > proxy_set_header X-SSL-Subject $ssl_client_s_dn; > proxy_set_header X-SSL-Issuer $ssl_client_i_dn; > } > > I started the Puppet master and reloaded the nginx for the configuration > to take effect. > > First I tried to connect with openssl s_client: > root@node2 # openssl s_client -connect localhost:8140 -cert > /var/lib/puppet/ssl/certs/node2.pem -key > /var/lib/puppet/ssl/private_keys/node2.pem > CONNECTED(00000003) > depth=1 CN = Puppet CA: node1 > verify error:num=19:self signed certificate in certificate chain > verify return:0 > --- > Certificate chain > 0 s:/CN=node2 > i:/CN=Puppet CA: node1 > 1 s:/CN=Puppet CA: node1 > i:/CN=Puppet CA: node1 > --- > [..] > subject=/CN=node2 > issuer=/CN=Puppet CA: node1 > --- > Acceptable client certificate CA names > /CN=Puppet CA: node1 > --- > SSL handshake has read 5314 bytes and written 2445 bytes > --- > New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256 > Server public key is 4096 bit > Secure Renegotiation IS supported > Compression: NONE > Expansion: NONE > SSL-Session: > Protocol : TLSv1.2 > Cipher : ECDHE-RSA-AES128-GCM-SHA256 > [..] > Start Time: 1382622609 > Timeout : 300 (sec) > Verify return code: 19 (self signed certificate in certificate chain) > --- > GET /production/node/node2 > Forbidden request: node2l(10.120.12.73) access to /node/node2 [find] at > :115closed > > This looks fine, It complains about the self-signed certificate in the > chain but other than that verify return = 0 which as far as I''m aware means > success. Once the handshake is complete I can actually do a GET for that > node but that seems to fail because of something in auth.conf, which is > slightly odd since the authentication succeeded so the ACL shouldn''t trip > over it but that''s step two (I''m probably forgetting to proxy/set a header). > > Now, running `puppet agent -t` however gives me this: > Warning: Unable to fetch my node definition, but the agent run will > continue: > Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server > certificate B: certificate verify failed: [certificate signature failure > for /CN=node1] > > At this point I''m lost. Ntpd''s are running, nodes are within far less than > a second in sync of each other. Certificate is valid from yesterday to > yesterday in 2018 so I doubt that''s an issue either. If it were the > `openssl s_client -connect` should have failed too but it''s obviously > perfectly happy with it. > > What am I missing here? I fear it''s staring me in the face but I''m just > not seeing it. > > -- > Daniele Sluijters >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.