Dear puppeteers I am trying to build a tree hierarchy of puppetmasters. The architecture is aimed to distribute the load among a number of datacenters, while keeping the puppetmasters in sync by means of puppet itself. The architecture I am trying to build is: - one "main puppetmaster"; - many "distribution servers", that will be client of the main puppetmaster, and masters to other clients - plain clients Unfortunately puppetmasterd gets in the way (maybe thinking it''s so smart?), screwing up the SSL setup. This was discussed yesterday on IRC; Volcano suspects that there something in the certificates is at the root of the problem, and that''s why I added a certdnsnames directive, but with no result so far. I am testing this setup on VirtualBox VMs on my desktop (which is actually a luck since I can use snapshot and rewind back and forth to different working states). The main puppetmaster is called mastertest.oslo.osa and has address 192.168.56.108; the distribution server is called distserver.oslo.osa and has address 192.168.56.111. Both are on each other''s /etc/hosts file. First, I configure distserver as a plain puppet client of mastertest. A couple of runs of puppetd --test will bring it up to speed, and it will work as expected. Then, on mastertest, I create a node file for distserver, which will define it as a distribution server, and run puppetd again. /etc/puppet/puppet.conf is rewritten so that it contains the following certdnsnames in the puppetmasterd section: certdnsnames="distserver.oslo.osa:distserver" while the server directive is the puppetd section is: server=mastertest.oslo.osa Eventually, after the new puppet.conf is already in place, puppetmasterd starts, and screws up the SSL setup:> Jul 13 14:00:38 distserver puppetmasterd[2861]: Creating a new SSL key for ca > Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate for ca, good until Sun Jul 05 12:44:33 UTC 2015 > Jul 13 14:00:38 distserver puppetmasterd[2861]: Expiring the certificate cache of ca > Jul 13 14:00:38 distserver puppetmasterd[2861]: Removing file Puppet::SSL::Certificate ca at ''/var/lib/puppet/ssl/certs/ca.pem'' > Jul 13 14:00:38 distserver puppetmasterd[2861]: Retrieved certificate does not match private key > Jul 13 14:00:38 distserver puppetmasterd[2861]: Creating a new SSL certificate request for ca > Jul 13 14:00:38 distserver puppetmasterd[2861]: Signed certificate request for ca > Jul 13 14:00:38 distserver puppetmasterd[2861]: Rebuilding inventory file > Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate_revocation_list for ca, good until > Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate for ca, good until Sat Jul 11 12:00:38 UTC 2015 > Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate for distserver.oslo.osa, good until Sat Jul 11 09:25:03 UTC 2015 > Jul 13 14:00:38 distserver puppetmasterd[2888]: Reopening log files(note the "Removing file" line...) Now, next time I run puppetd --test, all I get is:> Jul 13 14:01:08 distserver puppetd[3212]: Could not retrieve catalog from remote server: undefined method `closed?'' for nil:NilClass > Jul 13 14:01:08 distserver puppetd[3212]: Not using cache on failed catalog > Jul 13 14:01:08 distserver puppetd[3212]: Could not retrieve catalog; skipping runNeedless to say, if I stop puppetmasterd and put the old, "client" files back in place into /var/lib/puppet, this machine starts working again as a client. On mastertest (which has a nginx reverse proxy to four puppetmasterd instances, again for scalability) I see that the request from distserver is wrong. In fact, for distserver I have:> 192.168.56.111 - - [13/Jul/2010:15:30:09 +0200] "-" 400 0 "-" "-"while for working clients (e.g.: mastertest itself) I have something like:> 192.168.56.109 - - [13/Jul/2010:15:30:28 +0200] "GET /production/catalog/mastertest.oslo.osa?facts_format=b64_zlib_yaml&facts=LONG_BASE64_STRING_HEREI honestly can''t understand what is going on here... Is there a way to make this all work as intended? Thanks in advance Ciao --bronto -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Patrick Mohr
2010-Jul-13 17:05 UTC
Re: [Puppet Users] puppetmasterd screwing the SSL setup
On Jul 13, 2010, at 6:54 AM, Marco Marongiu wrote:> Dear puppeteers > > I am trying to build a tree hierarchy of puppetmasters. The architecture > is aimed to distribute the load among a number of datacenters, while > keeping the puppetmasters in sync by means of puppet itself. > > The architecture I am trying to build is: > > - one "main puppetmaster"; > - many "distribution servers", that will be client of the main > puppetmaster, and masters to other clients > - plain clients > > > Unfortunately puppetmasterd gets in the way (maybe thinking it''s so > smart?), screwing up the SSL setup. This was discussed yesterday on IRC; > Volcano suspects that there something in the certificates is at the root > of the problem, and that''s why I added a certdnsnames directive, but > with no result so far. > > I am testing this setup on VirtualBox VMs on my desktop (which is > actually a luck since I can use snapshot and rewind back and forth to > different working states). The main puppetmaster is called > mastertest.oslo.osa and has address 192.168.56.108; the distribution > server is called distserver.oslo.osa and has address 192.168.56.111. > Both are on each other''s /etc/hosts file. > > First, I configure distserver as a plain puppet client of mastertest. A > couple of runs of puppetd --test will bring it up to speed, and it will > work as expected. > > Then, on mastertest, I create a node file for distserver, which will > define it as a distribution server, and run puppetd again. > /etc/puppet/puppet.conf is rewritten so that it contains the following > certdnsnames in the puppetmasterd section: > > certdnsnames="distserver.oslo.osa:distserver" > > while the server directive is the puppetd section is: > > server=mastertest.oslo.osa > > Eventually, after the new puppet.conf is already in place, puppetmasterd > starts, and screws up the SSL setup: > >> Jul 13 14:00:38 distserver puppetmasterd[2861]: Creating a new SSL key for ca >> Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate for ca, good until Sun Jul 05 12:44:33 UTC 2015 >> Jul 13 14:00:38 distserver puppetmasterd[2861]: Expiring the certificate cache of ca >> Jul 13 14:00:38 distserver puppetmasterd[2861]: Removing file Puppet::SSL::Certificate ca at ''/var/lib/puppet/ssl/certs/ca.pem'' >> Jul 13 14:00:38 distserver puppetmasterd[2861]: Retrieved certificate does not match private key >> Jul 13 14:00:38 distserver puppetmasterd[2861]: Creating a new SSL certificate request for ca >> Jul 13 14:00:38 distserver puppetmasterd[2861]: Signed certificate request for ca >> Jul 13 14:00:38 distserver puppetmasterd[2861]: Rebuilding inventory file >> Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate_revocation_list for ca, good until >> Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate for ca, good until Sat Jul 11 12:00:38 UTC 2015 >> Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate for distserver.oslo.osa, good until Sat Jul 11 09:25:03 UTC 2015 >> Jul 13 14:00:38 distserver puppetmasterd[2888]: Reopening log files > > (note the "Removing file" line...) > > > Now, next time I run puppetd --test, all I get is: > >> Jul 13 14:01:08 distserver puppetd[3212]: Could not retrieve catalog from remote server: undefined method `closed?'' for nil:NilClass >> Jul 13 14:01:08 distserver puppetd[3212]: Not using cache on failed catalog >> Jul 13 14:01:08 distserver puppetd[3212]: Could not retrieve catalog; skipping run > > Needless to say, if I stop puppetmasterd and put the old, "client" files > back in place into /var/lib/puppet, this machine starts working again as > a client. > > On mastertest (which has a nginx reverse proxy to four puppetmasterd > instances, again for scalability) I see that the request from distserver > is wrong. In fact, for distserver I have: > >> 192.168.56.111 - - [13/Jul/2010:15:30:09 +0200] "-" 400 0 "-" "-" > > while for working clients (e.g.: mastertest itself) I have something like: > >> 192.168.56.109 - - [13/Jul/2010:15:30:28 +0200] "GET /production/catalog/mastertest.oslo.osa?facts_format=b64_zlib_yaml&facts=LONG_BASE64_STRING_HERE > > I honestly can''t understand what is going on here...Basically, the puppet packages you are using (and I suspect most others) assume that the client and the server on a given machine are part of the same PKI. It also might be assuming a couple of other things, but my experiments never got that far.> Is there a way to make this all work as intended?WARNING: This fix is almost as destructive as rm -Rf /var/lib/puppet I think everything will just work if you set puppetd and puppetmaster to have a different "ssldir" like this: [main] #remove the ssldir entry from here. [puppetmasterd] ssldir=/var/lib/puppet/ssl_server [puppetd] ssldir=/var/lib/puppet/ssl_client I won''t say this is working as intended. The normal way is to make a real PKI that includes all the servers, but this is probably much easier, and will probably do what you want. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Garrett Honeycutt
2010-Jul-13 23:44 UTC
[Puppet Users] Re: puppetmasterd screwing the SSL setup
On Jul 13, 6:54 am, Marco Marongiu <brontoli...@gmail.com> wrote:> Dear puppeteers > > I am trying to build a tree hierarchy of puppetmasters. The architecture > is aimed to distribute the load among a number of datacenters, while > keeping the puppetmasters in sync by means of puppet itself. > > The architecture I am trying to build is: > > - one "main puppetmaster"; > - many "distribution servers", that will be client of the main > puppetmaster, and masters to other clients > - plain clients ><snip>> Thanks in advance > > Ciao > --brontoI suggest staying away from a hierarchical puppetmaster approach as it adds unneeded complexity. At a former employer we solved the problem of dealing with multiple datacenters or POP''s by having a puppetmaster (or load balanced puppetmasters) in each POP. We then had a configuration file in our VCS that mapped POP''s to branches in our VCS. The puppetmasters would check the contents of that file and if it changed they would do an svn switch to the new branch/tag. Through this architecture we were able to easily update the code used by the puppetmasters. This also gave us the ability to horizontally scale puppetmasters as appropriate at each POP. Working on having the code available on the Puppet Forge this week. -g -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Marco Marongiu
2010-Jul-14 12:50 UTC
Re: [Puppet Users] puppetmasterd screwing the SSL setup
Hello Thanks to the ones who replied. Patrick Mohr ha scritto:> Basically, the puppet packages you are using (and I suspect most > others) assume that the client and the server on a given machine are > part of the same PKI. It also might be assuming a couple of other > things, but my experiments never got that far.Hm, I see...>> Is there a way to make this all work as intended? > > > WARNING: This fix is almost as destructive as rm -Rf /var/lib/puppet > > > I think everything will just work if you set puppetd and puppetmaster > to have a different "ssldir" like this: > > [main] #remove the ssldir entry from here. > > [puppetmasterd] ssldir=/var/lib/puppet/ssl_server > > [puppetd] ssldir=/var/lib/puppet/ssl_clientThat did the trick, thanks.> I won''t say this is working as intended. The normal way is to make a > real PKI that includes all the servers, but this is probably much > easier, and will probably do what you want. >I''m not sure if you''re suggesting to use a single machine to host a CA for the whole infrastructure. In case, would that scale? Thanks Ciao --bronto -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Wed, Jul 14, 2010 at 8:50 PM, Marco Marongiu <brontolinux@gmail.com>wrote:> Hello > > > I''m not sure if you''re suggesting to use a single machine to host a CA > for the whole infrastructure. In case, would that scale? >It would scale, but it would be a single point of failure for adding new puppet clients. I would say this is your best setup at the moment (if you want to keep the revoke option), while a chained ca is possible (I''m doing it) its currently broken with the latests version of puppet. as a side note, you might want to look at my multi puppetmaster module (a bit out of date, but maybe useful for a multi puppetmaster setup) - http://github.com/ohadlevy/puppet-multipuppetmaster Ohad -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Marco Marongiu
2010-Jul-14 14:03 UTC
Re: [Puppet Users] puppetmasterd screwing the SSL setup
Ohad Levy ha scritto:> > > On Wed, Jul 14, 2010 at 8:50 PM, Marco Marongiu <brontolinux@gmail.com > <mailto:brontolinux@gmail.com>> wrote: > > Hello > > > I''m not sure if you''re suggesting to use a single machine to host a CA > for the whole infrastructure. In case, would that scale? > > It would scale, but it would be a single point of failure for adding new > puppet clients.Yep. OK> I would say this is your best setup at the moment (if you want to keep > the revoke option), while a chained ca is possible (I''m doing it) its > currently broken with the latests version of puppet.I see. Thanks for the warning ;)> as a side note, you might want to look at my multi puppetmaster module > (a bit out of date, but maybe useful for a multi puppetmaster setup) > - http://github.com/ohadlevy/puppet-multipuppetmasterI gave it a glance, will look into it better tomorrow (early wakeup this morning, ouch...). I understand it is aimed to a hiearchical setup as the one I am building, is it? Thanks again Ciao --bronto -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Cesar Avalos
2010-Jul-15 14:32 UTC
Re: [Puppet Users] puppetmasterd screwing the SSL setup
I had the same problem and just fixed with Patrick solution. I just wanted to say thanks everybody. -- Att, Cesar Avalos -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Marco Marongiu
2010-Jul-16 20:50 UTC
Re: [Puppet Users] puppetmasterd screwing the SSL setup
Cesar Avalos ha scritto:> I had the same problem and just fixed with Patrick solution. I just > wanted to say thanks everybody.Hello Just as a round-up, You may want to add a [puppetca] section to your puppet.conf, setting the same ssldir as you did in [puppetmasterd], so that you don''t have to specify the ssldir on the command line each time you use the puppetca command. Ciao --bronto -- Marco Marongiu System Administrator - Technical Author - Perl Programmer -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.