Puppet users - Feb 2011 - multiple puppetmasters (w/ Passenger) behind load balancer

Hello Gang,

     I''m working on scaling my puppet solution, and I''m
deploying multiple
masters w/ passenger that are going sit behind a load balancer.  If anyone 
is using these type of setup, would you share how you deal with the SSL 
certs?  I''ve been following Bode''s Blog
(http://bodepd.com/wordpress/?p=7),
and it''s not working to good for me.  For instance I generate the ssl
cert
from the 1st master, and I copy it over to the 2nd puppetmaster:


puppet1>scp private_keys/puppetmaster2.pem
root@puppet2:/var/lib/puppet/ssl/private_keys/puppetmaster2.pem
puppet1>scp ca/signed/puppetmaster2.pem
root@puppet2:/var/lib/puppet/ssl/certs/puppetmaster2.pem
puppet1>scp ca/ca_crt.pem root@puppet2:/var/lib/puppet/ssl/certs/ca.pem

I get the following error:

"Starting puppetmaster: Could not prepare for execution: Retrieved
certificate does not match private key; please remove certificate from server
and regenerate it with the current key"

Does anyone have a better system for handling the generation and distribution of
SSL certs?

Please advise.

Thanks,

 


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I''m using puppet 0.25.5

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 8, 2011, at 1:09 PM, CraftyTech wrote:
> Hello Gang,
> 
>      I''m working on scaling my puppet solution, and I''m
deploying multiple masters w/ passenger that are going sit behind a load
balancer.  If anyone is using these type of setup, would you share how you deal
with the SSL certs?  I''ve been following Bode''s Blog
(http://bodepd.com/wordpress/?p=7), and it''s not working to good for
me.  For instance I generate the ssl cert from the 1st master, and I copy it
over to the 2nd puppetmaster:
> 
If you can, post the puppet.conf from "puppetmaster".  I''d
guess your problem is there.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Here''s the  puppet.conf that i''m distributing to the
puppetmasters:

[main]
    vardir = /var/lib/puppet
    logdir = /var/log/puppet
    rundir = /var/run/puppet
    reports= log, foreman
    clientyamldir = /var/lib/puppet/yaml/node
    pluginsync = true
    external_nodes = /etc/puppet/external_node.rb
    node_terminus = exec
    manifest  = /etc/puppet/manifests/site.pp
    modulepath = /apps/iadmin/puppet/modules


[puppetmasterd]
    certname=puppetmaster2.dev.domain.com
    ca= false
    ssl_client_header = SSL_CLIENT_S_DN
    ssl_client_verify_header = SSL_CLIENT_VERIFY
    templatedir = /apps/iadmin/puppet/templates
    clientyamldir = /var/lib/puppet/yaml/node
    reportdir = /var/lib/puppet/reports
    storeconfigs = true
    dbadapter = mysql
    dbuser = app_puppet
    dbpassword = xxxxxxxxxxxxxx
    dbserver = puppetdb.glb.domain.com
    dbsocket = /sql/mysql/mysql.sock
    rrddir = /var/lib/puppet/rrd
    rrdinterval = $runinterval
    rrdgraph = true


[puppetd]
    report = true
    clientbucketdir = /backup/puppetClientBucketDir
    ca_server = puppetmaster1.dev.domain.com
    #node_name = facter
    listen = true
    ssldir = /etc/puppet/ssl
    #certname = puppetmaster2


My main idea is to have one puppetmaster per datacenter syncing with the 
main puppetmaster.  I''m going to make sure that each datacenter
resolves to
their local puppetmaster via load balancer appliance (bigIP). The main 
puppetmaster also hosts the mysql database, which is dual replicated with 
another server, and which is also accessed via virtual name/ Load balance. 
 I know it''s a little ambitious but ironing out the SSL certs situation
would certainly put the last nail on the coffin.

Thanks,

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 8, 2011, at 9:02 PM, CraftyTech wrote:
> Here''s the  puppet.conf that i''m distributing to the
puppetmasters:
> 
> [puppetmasterd]
>     certname=puppetmaster2.dev.domain.com
>    
> 
> 
> My main idea is to have one puppetmaster per datacenter syncing with the
main puppetmaster.  I''m going to make sure that each datacenter
resolves to their local puppetmaster via load balancer appliance (bigIP). The
main puppetmaster also hosts the mysql database, which is dual replicated with
another server, and which is also accessed via virtual name/ Load balance.  I
know it''s a little ambitious but ironing out the SSL certs situation
would certainly put the last nail on the coffin.
> 
> Thanks,
So, you tell the puppetmaster the certname is
"puppetmaster2.dev.domain.com", but you''re generating a cert
named "puppetmaster2".  You should either change one or the other.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks for your reply.  At this juncture, I''m doing it by the book and
it''s
still failing for me.  Can someone spot what I''m doing wrong here?:

puppetmasterd --no-daemonize --verbose --certname 
puppet-test01.dev.domain.com
"Could not prepare for execution: Retrieved certificate does not match 
private key; please remove certificate from server and regenerate it with 
the current key"

Puppet.conf:
[main]
    vardir = /var/lib/puppet
    logdir = /var/log/puppet
    rundir = /var/run/puppet
    reports= log, foreman
    clientyamldir = /var/lib/puppet/yaml/node
    pluginsync = true
    external_nodes = /etc/puppet/external_node.rb
    node_terminus = exec
    manifest  = /etc/puppet/manifests/site.pp
    modulepath = /apps/iadmin/puppet/modules


[puppetmasterd]
    certname = puppet-test01.dev.domain.com
    ca= false
    ssl_client_header = SSL_CLIENT_S_DN
    ssl_client_verify_header = SSL_CLIENT_VERIFY
    templatedir = /apps/iadmin/puppet/templates
    clientyamldir = /var/lib/puppet/yaml/node
    reportdir = /var/lib/puppet/reports
    storeconfigs = true
    dbadapter = mysql
    dbuser = app_puppet
    dbpassword = CoNf1Gpupp3T
    dbserver = puppetdb.glb.domain.com
    dbsocket = /sql/mysql/mysql.sock
    rrddir = /var/lib/puppet/rrd
    rrdinterval = $runinterval
    rrdgraph = true


[puppetd]
    report = true
    clientbucketdir = /backup/puppetClientBucketDir
    ca_server = puppetmaster1.dev.domain.com
    #node_name = facter
    listen = true
    ssldir = /etc/puppet/ssl
    certname = puppet-test01.dev.domain.com



Generated from Main Puppetmaster with:
sudo /usr/sbin/puppetca --generate --certdnsnames 
puppet:puppet.glb.domain.com  puppet-test01.dev.domain.com


The Files are there:
puppet-test01 puppet # ls -ltr /etc/puppet/ssl/private_keys/
total 4
-rw------- 1 puppet root 887 Feb  9 10:08 puppet-test01.dev.domain.com.pem
puppet-test01 puppet # ls -ltr /etc/puppet/ssl/certs
total 8
-rw-r--r-- 1 puppet root 960 Feb  9 10:09 puppet-test01.dev.domain.com.pem
-rw-r--r-- 1 puppet root 765 Feb  9 10:11 ca.pem

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 9, 2011, at 7:27 AM, CraftyTech wrote:
> Thanks for your reply.  At this juncture, I''m doing it by the book
and it''s still failing for me.  Can someone spot what I''m
doing wrong here?:
> 
> puppetmasterd --no-daemonize --verbose --certname
puppet-test01.dev.domain.com
> "Could not prepare for execution: Retrieved certificate does not match
private key; please remove certificate from server and regenerate it with the
current key"
> 
> 
> Generated from Main Puppetmaster with:
> sudo /usr/sbin/puppetca --generate --certdnsnames
puppet:puppet.glb.domain.com  puppet-test01.dev.domain.com
> 
> 
> The Files are there:
> puppet-test01 puppet # ls -ltr /etc/puppet/ssl/private_keys/
> total 4
> -rw------- 1 puppet root 887 Feb  9 10:08 puppet-test01.dev.domain.com.pem
> puppet-test01 puppet # ls -ltr /etc/puppet/ssl/certs
> total 8
> -rw-r--r-- 1 puppet root 960 Feb  9 10:09 puppet-test01.dev.domain.com.pem
> -rw-r--r-- 1 puppet root 765 Feb  9 10:11 ca.pem
With your current config file, "there" should be /var/lib/puppet/ssl,
not /etc/puppet/ssl.  Check by running this on puppet-test1:
puppetmasterd --genconfig | grep ''ssldir =''

It will probably return "ssldir = /var/lib/puppet/ssl" telling you the
ssl directory is in "/var/lib/puppet/ssl".

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

The SSLDIR is where it should be:
puppetmasterd --genconfig | grep ''ssldir =''
    ssldir = /etc/puppet/ssl

The crazy thing is that running puppetd with the same cert works fine.  Only 
puppetmaster is the issue.  I run it to sync up with the main puppetmaster. 
 


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

These are the files that I copy:

puppetmaster1>cd /etc/puppet/ssl
puppetmaster1>scp private_keys/puppet-test01.pem 
root@puppet-test01:/etc/puppet/ssl/private_keys/puppet-test01.pem
puppetmaster1>scp ca/signed/puppet-test01.pem 
root@puppet-test01:/etc/puppet/ssl/certs/puppet-test01.pem
puppetmaster1>scp ca/ca_crt.pem 
root@puppet-test01:/etc/puppet/ssl/certs/ca.pem

To use Passenger (I''d need also ca_crt.pem and ca_crl.pem), so I also
copy:
puppetmaster1>scp /etc/puppet/ssl/ca/ca_crt.pem 
root@puppet-test01:/etc/puppet/ssl/ca/ca_crt.pem 
puppetmaster1>scp /etc/puppet/ssl/ca/ca_crl.pem 
root@puppet-test01:/etc/puppet/ssl/ca/ca_crl.pem 


I don''t get why it works for the client side, and not the servers
side......

All ideas welcome :)

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 9, 2011, at 10:51 AM, CraftyTech wrote:
> These are the files that I copy:
> 
> puppetmaster1>cd /etc/puppet/ssl
> puppetmaster1>scp private_keys/puppet-test01.pem
root@puppet-test01:/etc/puppet/ssl/private_keys/puppet-test01.pem
> puppetmaster1>scp ca/signed/puppet-test01.pem
root@puppet-test01:/etc/puppet/ssl/certs/puppet-test01.pem
> puppetmaster1>scp ca/ca_crt.pem
root@puppet-test01:/etc/puppet/ssl/certs/ca.pem
> 
> To use Passenger (I''d need also ca_crt.pem and ca_crl.pem), so I
also copy:
> puppetmaster1>scp /etc/puppet/ssl/ca/ca_crt.pem
root@puppet-test01:/etc/puppet/ssl/ca/ca_crt.pem
> puppetmaster1>scp /etc/puppet/ssl/ca/ca_crl.pem
root@puppet-test01:/etc/puppet/ssl/ca/ca_crl.pem
> 
> 
> I don''t get why it works for the client side, and not the servers
side......
> 
> All ideas welcome :)
The only line I see that''s different that I think matters is this:
ca_server = puppetmaster1.dev.domain.com

Try putting that line in the puppetmasterd section.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

If I just leave it with
[puppetmasterd]
certname = puppet-test01.dev.domain.com
ca = falsa

it works......  it just breaks after I put in the rest of the options....  I 
need to work my way through all the options....

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I run two 0.25.5 puppetmasters behind an haproxy load balancer with
both production and QA puppetmasters on different ports on each host.
These resolve to puppet.arces.net and puppet-qa.arces.net on our load
balancer and go to host-a and host-b on 8140 for production and 8150
for QA.  It all works great.  We evolved up to this over the last 3.5
years going from Webrick to Mongrel to some intermediate stuff and
finally to passenger.  Our puppetmasters are in the same datacenter,
so the manifest/module directory is shared via NFS.  It sounds exactly
like what you''re trying to do, except for the NFS part.  If
you''re
replicating the contents of var and etc between systems, then it
shouldn''t matter that it''s not NFS.  My puppetmasterd block in
puppet.conf looks like this:

[puppetmasterd]
    vardir = /var/www/localhost/puppet/var
    confdir = /var/www/localhost/puppet/etc
    modulepath = $confdir/modules
    bucketdir = $vardir/bucket
    clientbucketdir = $vardir/clientbucket
    fileserverconfig = $confdir/fileserver.conf
    ssldir = $vardir/ssl
    yamldir = $vardir/yaml
    manifestdir = $confdir/manifests
    logdest = syslog
    bindaddress = 127.0.0.1
    user = puppet
    storeconfigs = true
    dbadapter = mysql
    dbuser = xxxxxxxxxx
    dbname = xxxxxxxxxxx
    dbpassword = xxxxxxxxxx
    dbserver = xxxxxxxxxx
    reports = puppet_dashboard
    ssl_client_header = SSL_CLIENT_S_DN
    ssl_client_verify_header = SSL_CLIENT_VERIFY

We have a single key/cert for the master named "puppet.arces.net" (or
puppet-qa.arces.net for the QA one).  I don''t designate a cert name
anywhere - I just have a cert generated for the puppetmasters that
matches the hostname that the clients use to connect to the load
balancer, not a cert name for the hosts themselves.  This is shared
via the NFS mount, but it could just as easily be on both servers.

My suggestion is to get it working behind the F5 with only one node.
When you have that working, copy the vardir to the other node and
switch your config to use only that node.  If it still works, activate
both nodes behind the VIP and set your session persistence to bind by
clientip so you don''t have flipflop behavior between requests from the
clients.  You should be good to go.

Hope that helps.

Adrian Goins
Arces Network, LLC
http://www.arces.net

On Feb 10, 10:27 am, CraftyTech <hmmed...@gmail.com>
wrote:
> If I just leave it with
> [puppetmasterd]
> certname = puppet-test01.dev.domain.com
> ca = falsa
>
> it works......  it just breaks after I put in the rest of the options....
 I
> need to work my way through all the options....
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks Adrian !! I''ll certainly try it out in QA.  On the other hand,
for
those who followed this thread, I''ve found problem, and corrected it. 
It
turns out, that space after any of the values, would basically break your 
config.  I just edited the file, and removed all extra spaces, and 
everything is back to normal.  I needed to have to going so that I can 
deploy puppetmasters via a puppet class.  Thanks kc7zzv and Adrian.

Cheers,

Henry

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.