thr3ads.net - Puppet users - [Puppet Users] Seperate CA''s/Master behind load balancer [Dec 2011]

If this information is useful, please help other people find it:
Share via:

OlliesDad@googlemail.com

2011-Dec-16 12:40 UTC

[Puppet Users] Seperate CA''s/Master behind load balancer

Hello,

Attempting to setup a CA primary/standby as well as seperate
puppetmaster servers (all running Apache/Passenger) behind another
Apache/Passenger type load balancer.

Clients are not getting certs:-
err: Could not request certificate: Could not intern from s: nested
asn1 error

Clearly an SSL issue but not something I know a great deal about.

loadbalancer.conf

# Puppet Load Balancing Thing

<Proxy balancer://puppetmaster>
  BalancerMember http://puppetmst1:18140
  BalancerMember http://puppetmst2:18140
</Proxy>

<Proxy balancer://puppet_ca>
  # Only one member
  BalancerMember http://cahost1:18140
  # Hot Standby if the primary is offline
  BalancerMember http://cahost2:18140 status=+H
</Proxy>

Listen 8140
<VirtualHost *:8140>

  ServerName loadbalancer

  SSLEngine on
  # SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
  SSLProtocol -ALL +SSLv3 +TLSv1
  SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
  # Puppet master should generate initial CA certificate.
  # ensure certs are located in etc/puppet/ssl
  SSLCertificateFile /etc/puppet/ssl/certs/loadbalancer.pem
  SSLCertificateKeyFile /etc/puppet/ssl/private_keys/loadbalancer.pem
  SSLCertificateChainFile /etc/puppet/ssl/certs/ca.pem
  SSLCACertificateFile /etc/puppet/ssl/ca/ca_crt.pem
  # CRL checking should be enabled
  # disable next line if Apache complains about CRL
  #SSLCARevocationFile /etc/puppet/ssl/ca/ca_crl.pem
  # optional to allow CSR request, required if certificates
distributed to client during provisioning.
  SSLVerifyClient optional
  SSLVerifyDepth 1
  SSLOptions +StdEnvVars
  # The following client headers record authentication information for
down stream workers.
  RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
  # WARNING THIS SHOULD HAVE ACCESS CONTROL.
  # I''ve left it wide open for testing.
  <Location /balancer-manager>
    SetHandler balancer-manager
    Order allow,deny
    Allow from all
  </Location>

  <Location /server-status>
    SetHandler server-status
    Order allow,deny
    Allow from all
  </Location>

  # Optional status
  ProxyStatus On

  # Don''t load balance requests to the status page
  ProxyPass /balancer-manager !
  ProxyPass /server-status !

  # Ordering of ProxyPass directives is important
  # Direct all Puppet Agent CA requests to a specific set of workers.
  ProxyPassMatch ^(/.*?)/(certificate.*?)/(.*)$ balancer://puppet_ca/
  ProxyPassReverse ^(/.*?)/(certificate.*?)/(.*)$ balancer://puppet_ca/
  # Direct all other Puppet Agent requests to the default set of
workers.
  ProxyPass / balancer://puppetmaster/
  ProxyPassReverse / balancer://puppetmaster/

  ProxyPreserveHost On

  # Logging for this front end
  ErrorLog  /var/log/httpd/frontend_puppet_error.log
  CustomLog /var/log/httpd/frontend_puppet_access.log combined
  CustomLog /var/log/httpd/frontend_puppet_ssl_requests.log "%t %h %
{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>

cahost.conf:-

Listen 18140
<VirtualHost *:18140>
  ServerName cahost1
  SSLEngine on
  SSLProtocol -ALL +SSLv3 +TLSv1
  SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
  SSLCertificateFile /etc/puppet/ssl/certs/cahost1.pem
  SSLCertificateKeyFile /etc/puppet/ssl/private_keys/cahost1.pem
  SSLCertificateChainFile /etc/puppet/ssl/certs/ca.pem
  SSLCACertificateFile /etc/puppet/ssl/ca/ca_crt.pem
  # CRL checking should be enabled
  # disable next line if Apache complains about CRL
  SSLCARevocationFile /etc/puppet/ssl/ca/ca_crl.pem
  # optional to allow CSR request, required if certificates
distributed to client during provisioning.
  SSLVerifyClient optional
  SSLVerifyDepth 1
  SSLOptions +StdEnvVars
  # The following client headers record authentication information for
down stream workers.
  RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
  RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
  RackAutoDetect On
  DocumentRoot /etc/puppet/rack/puppetmaster/public/
  <Directory /etc/puppet/rack/puppetmaster/>
    Options None
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>

Hostname have been changed to protect the innocent.

Any pointers or obviously "you idiot" things here ?

Thanks
Paul

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Brice Figureau

2011-Dec-16 15:34 UTC

head link

Re: [Puppet Users] Seperate CA''s/Master behind load balancer

On Fri, 2011-12-16 at 04:40 -0800, OlliesDad@googlemail.com
wrote:> Hello,
> 
> Attempting to setup a CA primary/standby as well as seperate
> puppetmaster servers (all running Apache/Passenger) behind another
> Apache/Passenger type load balancer.
> 
> Clients are not getting certs:-
> err: Could not request certificate: Could not intern from s: nested
> asn1 error
> 
> Clearly an SSL issue but not something I know a great deal about.
Your primary load-balancer is the SSL endpoint, so when the requests
arrive in your puppet_ca nodes it is in clear text.
But apparently the ca_host configuration tell the server that it will
receive SSL content.
> [snipped]-- 
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

OlliesDad@googlemail.com

2011-Dec-16 15:53 UTC

head link

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

Thanks I realised that when I sent it. Dialled back the CA to:-
Listen 18140
<VirtualHost *:18140>
  SSLEngine off
  ServerName <CA FQDN>
  RackAutoDetect On
  DocumentRoot /etc/puppet/rack/puppetmaster/public/
  <Directory /etc/puppet/rack/puppetmaster/>
    Options None
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>
</VirtualHost>

Now clients are getting cert requests signed but not going any further
info: Creating a new SSL key for <CLIENT FQDN>
warning: peer certificate won''t be verified in this SSL session
info: Caching certificate for ca
warning: peer certificate won''t be verified in this SSL session
warning: peer certificate won''t be verified in this SSL session
info: Creating a new SSL certificate request for <CLIENT FQDN>
info: Certificate Request fingerprint (md5): 51:D6:6B:58:EA:CC:
11:14:4B:48:E1:B4:C1:8B:A5:A6
warning: peer certificate won''t be verified in this SSL session
warning: peer certificate won''t be verified in this SSL session
info: Caching certificate for <CLIENT FQDN>
info: Retrieving plugin
err: /File[/var/lib/puppet/plugins]: Failed to generate additional
resources using ''eval_generate: certificate verify failed.  This is
often because the time is out of sync on the server or client
err: /File[/var/lib/puppet/plugins]: Could not evaluate: certificate
verify failed.  This is often because the time is out of sync on the
server or client Could not retrieve file metadata for puppet://<LOAD
BALANCER FQDN>
/plugins: certificate verify failed.  This is often because the time
is out of sync on the server or client
err: Could not retrieve catalog from remote server: certificate verify
failed.  This is often because the time is out of sync on the server
or client
warning: Not using cache on failed catalog

I know the time is in sync OK

Certs look the same.

On CA:-
# openssl x509 -text -noout -in <CLIENT FQDN>
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 8 (0x8)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: CN=Puppet CA: <CAFQDN>
        Validity
            Not Before: Dec 15 15:29:00 2011 GMT
            Not After : Dec  9 15:29:00 2036 GMT
        Subject: CN=<CLIENT FQDN>
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:ad:62:d2:47:05:33:94:e2:5c:55:c1:e4:c4:7b:
                    27:7d:21:ec:0b:34:29:39:4f:56:1c:81:9f:94:3d:
                    9c:4c:ed:b0:ae:6b:82:85:d9:2f:3f:c9:bc:9a:31:
                    3d:5b:e7:d0:f9:19:7c:71:e0:ad:7f:18:fa:fe:53:
                    38:d2:35:67:d6:e6:dd:f8:df:fc:3c:46:ae:50:b1:
                    7b:66:04:a1:39:b3:bf:55:39:2e:47:ad:ee:59:ae:
                    17:36:43:2b:76:c3:ad:90:5e:03:67:aa:7e:8e:9e:
                    ca:1a:19:c6:3f:2e:c0:ea:33:bc:2a:01:63:2b:85:
                    e5:b1:4a:75:ff:0f:8b:b2:4d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            Netscape Comment:
                Puppet Ruby/OpenSSL Internal Certificate
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Subject Key Identifier:
                C7:AB:D6:D1:DC:D9:30:E6:0B:DE:60:4E:FB:25:37:AF:7F:
43:E1:B7
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client
Authentication
    Signature Algorithm: sha1WithRSAEncryption
        00:dc:82:79:6b:2f:30:2c:e2:8b:9d:52:78:6f:e7:86:56:8c:
        98:7f:3a:46:8a:86:49:12:de:e7:2e:c3:d9:ed:96:e6:5e:1e:
        fe:0d:cc:86:ed:86:2e:44:13:c3:4b:ec:e7:3c:9a:1d:bf:2d:
        07:a5:c7:65:dc:ec:10:80:85:f5:de:25:83:8b:66:4f:46:17:
        73:97:5d:1b:8c:9e:1c:bc:7a:51:5a:11:d7:b4:75:21:44:0b:
        39:29:06:77:51:9a:b0:3f:5b:80:6b:6d:13:99:ed:6a:7d:35:
        c2:0c:6c:f2:98:f0:d1:f0:4f:ed:d6:53:98:7c:8a:5b:07:4a:
        54:e0

On client:-
# openssl x509 -text -noout -in <CLIENT FQDN>
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 8 (0x8)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: CN=Puppet CA: <CA FQDN>
        Validity
            Not Before: Dec 15 15:29:00 2011 GMT
            Not After : Dec  9 15:29:00 2036 GMT
        Subject: CN=<CLIENT FQDN>
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (1024 bit)
                Modulus (1024 bit):
                    00:ad:62:d2:47:05:33:94:e2:5c:55:c1:e4:c4:7b:
                    27:7d:21:ec:0b:34:29:39:4f:56:1c:81:9f:94:3d:
                    9c:4c:ed:b0:ae:6b:82:85:d9:2f:3f:c9:bc:9a:31:
                    3d:5b:e7:d0:f9:19:7c:71:e0:ad:7f:18:fa:fe:53:
                    38:d2:35:67:d6:e6:dd:f8:df:fc:3c:46:ae:50:b1:
                    7b:66:04:a1:39:b3:bf:55:39:2e:47:ad:ee:59:ae:
                    17:36:43:2b:76:c3:ad:90:5e:03:67:aa:7e:8e:9e:
                    ca:1a:19:c6:3f:2e:c0:ea:33:bc:2a:01:63:2b:85:
                    e5:b1:4a:75:ff:0f:8b:b2:4d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            Netscape Comment:
                Puppet Ruby/OpenSSL Internal Certificate
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Subject Key Identifier:
                C7:AB:D6:D1:DC:D9:30:E6:0B:DE:60:4E:FB:25:37:AF:7F:
43:E1:B7
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client
Authentication
    Signature Algorithm: sha1WithRSAEncryption
        00:dc:82:79:6b:2f:30:2c:e2:8b:9d:52:78:6f:e7:86:56:8c:
        98:7f:3a:46:8a:86:49:12:de:e7:2e:c3:d9:ed:96:e6:5e:1e:
        fe:0d:cc:86:ed:86:2e:44:13:c3:4b:ec:e7:3c:9a:1d:bf:2d:
        07:a5:c7:65:dc:ec:10:80:85:f5:de:25:83:8b:66:4f:46:17:
        73:97:5d:1b:8c:9e:1c:bc:7a:51:5a:11:d7:b4:75:21:44:0b:
        39:29:06:77:51:9a:b0:3f:5b:80:6b:6d:13:99:ed:6a:7d:35:
        c2:0c:6c:f2:98:f0:d1:f0:4f:ed:d6:53:98:7c:8a:5b:07:4a:
        54:e0

So now I am really puzzled :(

Cheers
Paul

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Brice Figureau

2011-Dec-16 16:21 UTC

head link

Re: [Puppet Users] Re: Seperate CA''s/Master behind load balancer

On Fri, 2011-12-16 at 07:53 -0800, OlliesDad@googlemail.com
wrote:> Thanks I realised that when I sent it. Dialled back the CA to:-
> Listen 18140
> <VirtualHost *:18140>
>   SSLEngine off
>   ServerName <CA FQDN>
>   RackAutoDetect On
>   DocumentRoot /etc/puppet/rack/puppetmaster/public/
>   <Directory /etc/puppet/rack/puppetmaster/>
>     Options None
>     AllowOverride None
>     Order allow,deny
>     allow from all
>   </Directory>
> </VirtualHost>
> 
> Now clients are getting cert requests signed but not going any further
> info: Creating a new SSL key for <CLIENT FQDN>
> warning: peer certificate won''t be verified in this SSL session
> info: Caching certificate for ca
> warning: peer certificate won''t be verified in this SSL session
> warning: peer certificate won''t be verified in this SSL session
> info: Creating a new SSL certificate request for <CLIENT FQDN>
> info: Certificate Request fingerprint (md5): 51:D6:6B:58:EA:CC:
> 11:14:4B:48:E1:B4:C1:8B:A5:A6
> warning: peer certificate won''t be verified in this SSL session
> warning: peer certificate won''t be verified in this SSL session
> info: Caching certificate for <CLIENT FQDN>
> info: Retrieving plugin
> err: /File[/var/lib/puppet/plugins]: Failed to generate additional
> resources using ''eval_generate: certificate verify failed.  This
is
> often because the time is out of sync on the server or client
> err: /File[/var/lib/puppet/plugins]: Could not evaluate: certificate
> verify failed.  This is often because the time is out of sync on the
> server or client Could not retrieve file metadata for puppet://<LOAD
> BALANCER FQDN>
> /plugins: certificate verify failed.  This is often because the time
> is out of sync on the server or client
> err: Could not retrieve catalog from remote server: certificate verify
> failed.  This is often because the time is out of sync on the server
> or client
> warning: Not using cache on failed catalog
OK, so when it tried to pluginsync it complained the server certificate
could not be verified.

Are you sure the puppetmaster _server_ certificate has been signed by
the same CA as this node _client_ certificate.

In other words is the following working:
openssl s_client -host puppet -port 8140 \
 -CAfile /var/lib/puppet/ssl/certs/ca.pem \
 -cert /var/lib/puppet/ssl/certs/<CLIENT FQDN>.pem \
 -key /var/lib/puppet/ssl/private_keys/<CLIENT FQDN>.pem

If not, it might give you more information (especially with -debug).

Also, it might be worth checking on the apache error log.
> I know the time is in sync OK
> 
> Certs look the same.
To be really sure compare the certificate fingerprints.

-- 
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

OlliesDad@googlemail.com

2011-Dec-19 11:05 UTC

head link

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

Thanks,

On our older infrastrcture if we wanted to scale out we just copied
the ssldir and changed the filenames to the FQDN of the new master
server. certdnsnames would be wildcarded.

Now using 2.7.9 how do we do certificates so we could scale out
horizontally from behind this loadbalancer ?

Tring this approach leads now to this:-

# puppet cert --list --all
warning: The `certdnsnames` setting is no longer functional,
after CVE-2011-3872. We ignore the value completely.

For your own certificate request you can set `dns_alt_names` in the
configuration and it will apply locally.  There is no configuration
option to
set DNS alt names, or any other `subjectAltName` value, for another
nodes
certificate.

Alternately you can use the `--dns_alt_names` command line option to
set the
labels added while generating your own CSR.
- <CLIENT FQDN> (FA:C4:68:C1:30:E2:95:9E:48:AB:ED:E4:A7:BF:3F:19)
(certificate signature failure)

Going around in circles somewhat trying to get a modern puppet setup
with a potential to scale horizontally.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Brice Figureau

2011-Dec-19 18:24 UTC

head link

Re: [Puppet Users] Re: Seperate CA''s/Master behind load balancer

On 19/12/11 12:05, OlliesDad@googlemail.com wrote:> Thanks,
> 
> On our older infrastrcture if we wanted to scale out we just copied
> the ssldir and changed the filenames to the FQDN of the new master
> server. certdnsnames would be wildcarded.
The problem with this way of scaling is that you won''t be able to
revoke
a certificate. The reason is that more than one certificate can have the
same serial.

I believe it''s better to dedicate a master to be a CA only master. Then
you point your clients to this ca.
If you fear the SPOF, then you can use a pair of CA server sharing
ssldir either through rsync or anything else allowing sharing files.
> Now using 2.7.9 how do we do certificates so we could scale out
> horizontally from behind this loadbalancer ?
There''s no reasons you can''t do what you were doing before
upgrading to
the 2.7.9 version. If what you were doing doesn''t work anymore, then it
might be a bug you should report.
> Tring this approach leads now to this:-
> 
> # puppet cert --list --all
> warning: The `certdnsnames` setting is no longer functional,
> after CVE-2011-3872. We ignore the value completely.
> 
> For your own certificate request you can set `dns_alt_names` in the
> configuration and it will apply locally.  There is no configuration
> option to
> set DNS alt names, or any other `subjectAltName` value, for another
> nodes
> certificate.
> 
> Alternately you can use the `--dns_alt_names` command line option to
> set the
> labels added while generating your own CSR.
> - <CLIENT FQDN> (FA:C4:68:C1:30:E2:95:9E:48:AB:ED:E4:A7:BF:3F:19)
> (certificate signature failure)
> 
> Going around in circles somewhat trying to get a modern puppet setup
> with a potential to scale horizontally.
The command just complains about the certdnsnames option that has been
removed. You can stil use dns_alt_names to generate clients and/or
server certificates with embedded subjectAltName extension.

-- 
Brice Figureau
My Blog: http://www.masterzen.fr/

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

OlliesDad@googlemail.com

2011-Dec-20 15:14 UTC

head link

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

Thanks.

I assume that the section in this:-
http://projects.puppetlabs.com/projects/puppet/wiki/Puppet_Scalability

Stating that is doesn''t work for 0.25 & 2.6 also applies to the
2.7.9
release that is the latest ?

Sharing an area via NFS/iSCSI/rsync''ing or whatever is potentially
viable does anyone know how this would be possible with different
hostnames serving the certs and the traffic being directed via a load-
balancer ?

Maybe it''s just not possible right now and I am flogging a dead horse
and should accept a SPOF for a CA but can easily scale out the
puppetmasters fine.

Thanks all.

Paul

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Brice Figureau

2011-Dec-20 15:31 UTC

head link

Re: [Puppet Users] Re: Seperate CA''s/Master behind load balancer

On Tue, 2011-12-20 at 07:14 -0800, OlliesDad@googlemail.com
wrote:> Thanks.
> 
> I assume that the section in this:-
http://projects.puppetlabs.com/projects/puppet/wiki/Puppet_Scalability
> 
> Stating that is doesn''t work for 0.25 & 2.6 also applies to
the 2.7.9
> release that is the latest ?
Yes, I believe chained CA are still not working in 2.7.x, if that''s
what
you meant.
> Sharing an area via NFS/iSCSI/rsync''ing or whatever is potentially
> viable does anyone know how this would be possible with different
> hostnames serving the certs and the traffic being directed via a load-
> balancer ?
That''s easy: dedicate two host to be CAs only. One is the hot standby
of
the first one. You can either manually bring it up when the first one
fails, or use something like drbd+pacemaker to do it automatically.
Then have all your other masters run in "no ca" mode. Each can have a
different server CN, or they can share the same server certificate.
This is explained in length in the Pro puppet [1] book if you need.
> Maybe it''s just not possible right now and I am flogging a dead
horse
> and should accept a SPOF for a CA but can easily scale out the
> puppetmasters fine.
The simplest architecture for load balanced puppet is the single CA one,
of course that means you can live with the SPOF. BTW, the SPOF is only
at certificate signing. In the event your CA becomes unresponsive, it
won''t prevent your actual nodes to get a catalog.

I highly recommend you to get a copy of the "Pro Puppet" book. It
contains an extensive chapter on load balancing puppet master (both with
the SPOF and without it).

[1]: http://www.apress.com/9781430230571
-- 
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

OlliesDad@googlemail.com

2011-Dec-20 16:02 UTC

head link

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

> That''s easy: dedicate two host to be CAs only. One is the hot
standby of
> the first one. You can either manually bring it up when the first one
> fails, or use something like drbd+pacemaker to do it automatically.
> Then have all your other masters run in "no ca" mode. Each can
have a
> different server CN, or they can share the same server certificate.
> This is explained in length in the Pro puppet [1] book if you need.
>
> > Maybe it''s just not possible right now and I am flogging a
dead horse
> > and should accept a SPOF for a CA but can easily scale out the
> > puppetmasters fine.
>
> The simplest architecture for load balanced puppet is the single CA one,
> of course that means you can live with the SPOF. BTW, the SPOF is only
> at certificate signing. In the event your CA becomes unresponsive, it
> won''t prevent your actual nodes to get a catalog.
>
> I highly recommend you to get a copy of the "Pro Puppet" book. It
> contains an extensive chapter on load balancing puppet master (both with
> the SPOF and without it).
Thanks.

Have got a copy of the book and that is what I was working from. As
per the
example in the book it''s fine running the CA''s in the
localhost sort
of mode
but when switching from locahost to other servers off the load-
balancer server
I get the cert errors:-

err: /File[/var/lib/puppet/lib]: Failed to generate additional
resources using ''eval_generate: certificate verify failed.  This is
often because the time is out of sync on the server or client


Do I have to clean out the puppetmaster setup on the load-balancer
host ?

On the CA servers I removed the ssldir and ran "puppet master" to
generate a
new ssl data.

Then with a new client I get the new cert generated but then the above
error.


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Brice Figureau

2011-Dec-20 16:16 UTC

head link

Re: [Puppet Users] Re: Seperate CA''s/Master behind load balancer

On Tue, 2011-12-20 at 08:02 -0800, OlliesDad@googlemail.com
wrote:> > That''s easy: dedicate two host to be CAs only. One is the hot
standby of
> > the first one. You can either manually bring it up when the first one
> > fails, or use something like drbd+pacemaker to do it automatically.
> > Then have all your other masters run in "no ca" mode. Each
can have a
> > different server CN, or they can share the same server certificate.
> > This is explained in length in the Pro puppet [1] book if you need.
> >
> > > Maybe it''s just not possible right now and I am flogging
a dead horse
> > > and should accept a SPOF for a CA but can easily scale out the
> > > puppetmasters fine.
> >
> > The simplest architecture for load balanced puppet is the single CA
one,
> > of course that means you can live with the SPOF. BTW, the SPOF is only
> > at certificate signing. In the event your CA becomes unresponsive, it
> > won''t prevent your actual nodes to get a catalog.
> >
> > I highly recommend you to get a copy of the "Pro Puppet"
book. It
> > contains an extensive chapter on load balancing puppet master (both
with
> > the SPOF and without it).
> 
> Thanks.
> 
> Have got a copy of the book and that is what I was working from. As
> per the
> example in the book it''s fine running the CA''s in the
localhost sort
> of mode
> but when switching from locahost to other servers off the load-
> balancer server
> I get the cert errors:-
> 
> err: /File[/var/lib/puppet/lib]: Failed to generate additional
> resources using ''eval_generate: certificate verify failed.  This
is
> often because the time is out of sync on the server or client
> 
> 
> Do I have to clean out the puppetmaster setup on the load-balancer
> host ?
> 
> On the CA servers I removed the ssldir and ran "puppet master" to
> generate a
> new ssl data.
> 
> Then with a new client I get the new cert generated but then the above
> error.
That''s expected because when the client connects to one of your
loadbalanced server it receives a certificate that was signed/generated
under the previous CA. You actually need your loadbalanced masters to
get a certificate from your current CA. This certificate will then be
used when talking to your nodes.

-- 
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

OlliesDad@googlemail.com

2011-Dec-20 16:25 UTC

head link

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

On Dec 20, 4:16 pm, Brice Figureau <brice-pup...@daysofwonder.com>
wrote:> On Tue, 2011-12-20 at 08:02 -0800, Ollies...@googlemail.com wrote:
> > > That''s easy: dedicate two host to be CAs only. One is
the hot standby of
> > > the first one. You can either manually bring it up when the first
one
> > > fails, or use something like drbd+pacemaker to do it
automatically.
> > > Then have all your other masters run in "no ca" mode.
Each can have a
> > > different server CN, or they can share the same server
certificate.
> > > This is explained in length in the Pro puppet [1] book if you
need.
>
> > > > Maybe it''s just not possible right now and I am
flogging a dead horse
> > > > and should accept a SPOF for a CA but can easily scale out
the
> > > > puppetmasters fine.
>
> > > The simplest architecture for load balanced puppet is the single
CA one,
> > > of course that means you can live with the SPOF. BTW, the SPOF is
only
> > > at certificate signing. In the event your CA becomes
unresponsive, it
> > > won''t prevent your actual nodes to get a catalog.
>
> > > I highly recommend you to get a copy of the "Pro
Puppet" book. It
> > > contains an extensive chapter on load balancing puppet master
(both with
> > > the SPOF and without it).
>
> > Thanks.
>
> > Have got a copy of the book and that is what I was working from. As
> > per the
> > example in the book it''s fine running the CA''s in
the localhost sort
> > of mode
> > but when switching from locahost to other servers off the load-
> > balancer server
> > I get the cert errors:-
>
> > err: /File[/var/lib/puppet/lib]: Failed to generate additional
> > resources using ''eval_generate: certificate verify failed.
 This is
> > often because the time is out of sync on the server or client
>
> > Do I have to clean out the puppetmaster setup on the load-balancer
> > host ?
>
> > On the CA servers I removed the ssldir and ran "puppet
master" to
> > generate a
> > new ssl data.
>
> > Then with a new client I get the new cert generated but then the above
> > error.
>
> That''s expected because when the client connects to one of your
> loadbalanced server it receives a certificate that was signed/generated
> under the previous CA. You actually need your loadbalanced masters to
> get a certificate from your current CA. This certificate will then be
> used when talking to your nodes.
But the Apache LB settings are sending the certificate stuff to the
seperate
CA server (I can see this in the logs) and the CA has the signed cert
in the
puppet cert --list --all but it still complains on the client.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Brice Figureau

2011-Dec-20 16:31 UTC

head link

Re: [Puppet Users] Re: Seperate CA''s/Master behind load balancer

On Tue, 2011-12-20 at 08:25 -0800, OlliesDad@googlemail.com
wrote:> 
> On Dec 20, 4:16 pm, Brice Figureau <brice-pup...@daysofwonder.com>
> wrote:
> > On Tue, 2011-12-20 at 08:02 -0800, Ollies...@googlemail.com wrote:
> > > > That''s easy: dedicate two host to be CAs only. One
is the hot standby of
> > > > the first one. You can either manually bring it up when the
first one
> > > > fails, or use something like drbd+pacemaker to do it
automatically.
> > > > Then have all your other masters run in "no ca"
mode. Each can have a
> > > > different server CN, or they can share the same server
certificate.
> > > > This is explained in length in the Pro puppet [1] book if
you need.
> >
> > > > > Maybe it''s just not possible right now and I
am flogging a dead horse
> > > > > and should accept a SPOF for a CA but can easily scale
out the
> > > > > puppetmasters fine.
> >
> > > > The simplest architecture for load balanced puppet is the
single CA one,
> > > > of course that means you can live with the SPOF. BTW, the
SPOF is only
> > > > at certificate signing. In the event your CA becomes
unresponsive, it
> > > > won''t prevent your actual nodes to get a catalog.
> >
> > > > I highly recommend you to get a copy of the "Pro
Puppet" book. It
> > > > contains an extensive chapter on load balancing puppet
master (both with
> > > > the SPOF and without it).
> >
> > > Thanks.
> >
> > > Have got a copy of the book and that is what I was working from.
As
> > > per the
> > > example in the book it''s fine running the CA''s
in the localhost sort
> > > of mode
> > > but when switching from locahost to other servers off the load-
> > > balancer server
> > > I get the cert errors:-
> >
> > > err: /File[/var/lib/puppet/lib]: Failed to generate additional
> > > resources using ''eval_generate: certificate verify
failed.  This is
> > > often because the time is out of sync on the server or client
> >
> > > Do I have to clean out the puppetmaster setup on the
load-balancer
> > > host ?
> >
> > > On the CA servers I removed the ssldir and ran "puppet
master" to
> > > generate a
> > > new ssl data.
> >
> > > Then with a new client I get the new cert generated but then the
above
> > > error.
> >
> > That''s expected because when the client connects to one of
your
> > loadbalanced server it receives a certificate that was
signed/generated
> > under the previous CA. You actually need your loadbalanced masters to
> > get a certificate from your current CA. This certificate will then be
> > used when talking to your nodes.
> 
> But the Apache LB settings are sending the certificate stuff to the
> seperate
> CA server (I can see this in the logs) and the CA has the signed cert
> in the
> puppet cert --list --all but it still complains on the client.
The client is supposed to validate the certificate server. It does this
by checking the certificate the server sent against its locally cached
CA certificate.

In your case, depending on how your LB is working, it might be possible
the SSL endpoint is your LB. In which case this is the one that will
send the server certificate. Make sure this one sends a certificate that
was generated by the loadbalanced CA.

-- 
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

OlliesDad@googlemail.com

2011-Dec-21 13:15 UTC

head link

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

Thanks for all the help but I have failed in this and decided to
change tack a little.
With no better results :)

Here is what the current attempt is.

2 Apache Load-Balancers which will be CA''s and therefore entry points
for clients. These
will be in active/standby solution on RHEL6 with heartbeat.

The virtual IP will be called puppet.domain.fqdn the physical node
names will be different
of course.

The PuppetMaster processes will be a load balanced pool underneath to
allow the horizontal
scaling and with active/standby LB''s and CA''s should not have
a SPOF

Client will then connect to the VIP which will do the certificate
checks on the LB server
and then pass to the puppetmaster pool.

Load-Balancer/CA structures kept in place with rsync or some-such.

Still failing with the CA setup against it''s realname, setup with
generate on the puppet.domain.fqdn
name, and setup with certname = puppet.domain.fqdn and then calling
puppet master removing the ssldir
between each.

They all failed to allow clients to get a valid cert:-
err: Could not request certificate: Retrieved certificate does not
match private key; please remove certificate from server and
regenerate it with the current key

Now I just assume I am being brain-dead and doing silly things with
fatigue to get this working with a VIP.
Or is this setup without a SPOF really just not possible ?

Thanks
Paul

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Seemingly Similar Threads

Search for more reasonably related threads

Puppet users - Dec 2011 - Seperate CA's/Master behind load balancer

[Puppet Users] Seperate CA''s/Master behind load balancer

Re: [Puppet Users] Seperate CA''s/Master behind load balancer

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

Re: [Puppet Users] Re: Seperate CA''s/Master behind load balancer

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

Re: [Puppet Users] Re: Seperate CA''s/Master behind load balancer

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

Re: [Puppet Users] Re: Seperate CA''s/Master behind load balancer

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

Re: [Puppet Users] Re: Seperate CA''s/Master behind load balancer

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

Re: [Puppet Users] Re: Seperate CA''s/Master behind load balancer

[Puppet Users] Re: Seperate CA''s/Master behind load balancer

Seemingly Similar Threads