Hi, Since puppet doesn''t have HA/fail over capabilities as of now. How does one mitigate a puppet master failure( Ex. Hardware). When you replace the server and configure the Puppet masterd, the clients may no longer be able to communicate with the server, since the server''s SSL certificates would have changed. BTW, I am using puppet to manage about 700+ hosts, I am beginning to worry about the scenario as mentioned above. I definitely don''t want to login to each host to clear the "ssl" directory to make it request new certificate. Thanks, -- LOhit -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Le mercredi 31 mars 2010 à 16:22 +0530, LOhit a écrit :> Hi,Hi,> Since puppet doesn''t have HA/fail over capabilities as of now. How > does one mitigate a puppet master failure( Ex. Hardware). When you > replace the server and configure the Puppet masterd, the clients may > no longer be able to communicate with the server, since the server''s > SSL certificates would have changed.I have 2 masters, certs are available on both. If one fails the second one handles the job. Quite easy with nginx + mongrel (my setup).> BTW, I am using puppet to manage about 700+ hosts, I am beginning to > worry about the scenario as mentioned above. I definitely don''t want > to login to each host to clear the "ssl" directory to make it request > new certificate.Tools like fabric can do this quite easily in 5 minutes of work Nico.
We had exactly the same problem at our site over the weekend and resolved it by... logging in to every node, stopping puppetd, cleaning out /var/lib/puppet, and starting puppetd once again. We still use a twisted-python tool to run one-off commands and executables in parallel with ssh, and this is a case where it helps. Also, we will sometimes use it to undo obsolete policy then simply change puppet manifests to reflect the new policy. Until we upgraded our master to run 12 puppetmasterds behind nginx, it was nearly impossible to make a policy change affecting hundreds of nodes in seconds any other way. In an emergency, having the ssh trick in our back pocket can be a lifesaver. On Mar 31, 2010, at 5:52, LOhit <lohit.b@gmail.com> wrote:> Hi, > > Since puppet doesn''t have HA/fail over capabilities as of now. How > does one mitigate a puppet master failure( Ex. Hardware). When you > replace the server and configure the Puppet masterd, the clients may > no longer be able to communicate with the server, since the server''s > SSL certificates would have changed. > > BTW, I am using puppet to manage about 700+ hosts, I am beginning to > worry about the scenario as mentioned above. I definitely don''t want > to login to each host to clear the "ssl" directory to make it > request new certificate. > > Thanks, > -- > LOhit > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com > . > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en > .-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 3/31/10 6:52 AM, "LOhit" <lohit.b@gmail.com> wrote:> BTW, I am using puppet to manage about 700+ hosts,Before we started using rsync and running puppet locally on each host, we actually added the SSL certs to SVN. Cheesy, but we could quickly, and easily, move clients from master to master. -- Brian Akins -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
hello, The master doesn''t actually need a copy of the signed certs on the its drives. All the master needs is the CA that signed the nodes. If you set your masters up that you access them via a CNAME such as ''puppet'' all you need to really bother with is making sure your new master is on that name and has the same CA as before. You might need to set certname=puppet in puppet.conf in the [puppetmasterd] section to prevent your new master from creating certs when a new hostname is found. You only generally need the client certs to be able to do revokes and such. I have many regional masters, they all just share the same CA. The more correct way is to do the whole CA and Sub CA setup etc, but it just seemed too much of a hassle when I set mine up. With the same CA everywhere any one of my nodes can talk to any one of my masters, like I designate one master as the place where reports go, any machine can send in reports even if another master signed them. Also makes it trivial to recover from failure or to scale. ----- "Brian Akins" <Brian.Akins@turner.com> wrote:> On 3/31/10 6:52 AM, "LOhit" <lohit.b@gmail.com> wrote: > > BTW, I am using puppet to manage about 700+ hosts, > > Before we started using rsync and running puppet locally on each host, > we > actually added the SSL certs to SVN. Cheesy, but we could quickly, > and > easily, move clients from master to master. > > -- > Brian Akins > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- R.I.Pienaar -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
R.I.Pienaar <rip@devco.net> wrote:> All the master needs is the CA that signed the nodes. If > you set your masters up that you access them via a CNAME > such as ''puppet'' all you need to really bother with > is making sure your new master is on that name and has > the same CA as before.Excuse me for the question, but how do you do that exactly? -- Cosimo -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
hello, ----- "Cosimo Streppone" <cosimo@streppone.it> wrote:> R.I.Pienaar <rip@devco.net> wrote: > > > All the master needs is the CA that signed the nodes. If > > you set your masters up that you access them via a CNAME > > such as ''puppet'' all you need to really bother with > > is making sure your new master is on that name and has > > the same CA as before. > > Excuse me for the question, but how do you do that exactly?I think all you need is to copy /var/lib/puppet/ssl/ca/ and set appropriate certname entries -- R.I.Pienaar -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
LOhit, The main two things to take into account are: 1. keep your manifests elsewhere, IMHO puppetmasters always gets RO copy of your puppet data (e.g. from a VCS). 2. Solve the SSL hell, then everything is simple. I''ve been using in my setup (with approx 15 productive puppetmasters and about 20 development puppetmasters) a chained CA. What it means is that you have one puppet master, which signs other puppet masters. in turn, the masters sign the clients, and as they are trusting each other, you can connect from any server (or client) to any other server without SSL errors. the top level CA can be shut down (even better to keep the SSL data some where offline) and use it only when you need to sign a new puppet master. I find this way relatively clean, and nothing usually happens if i end up blowing up a puppet master or killing its certificate (as currently 025.4 is doing but its already fixed for 025.5). Ohad On Wed, Mar 31, 2010 at 6:52 PM, LOhit <lohit.b@gmail.com> wrote:> Hi, > > Since puppet doesn''t have HA/fail over capabilities as of now. How does one > mitigate a puppet master failure( Ex. Hardware). When you replace the server > and configure the Puppet masterd, the clients may no longer be able to > communicate with the server, since the server''s SSL certificates would have > changed. > > BTW, I am using puppet to manage about 700+ hosts, I am beginning to worry > about the scenario as mentioned above. I definitely don''t want to login to > each host to clear the "ssl" directory to make it request new certificate. > > Thanks, > -- > LOhit > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
LOhit writes: > Hi, > > Since puppet doesn''t have HA/fail over capabilities as of now. How does one > mitigate a puppet master failure( Ex. Hardware). When you replace the server > and configure the Puppet masterd, the clients may no longer be able to > communicate with the server, since the server''s SSL certificates would have > changed. Back up your puppetmaster SSL keys and certificates and reinstall them if you should ever have to resurrect your puppetmaster for the dead. There is absolutely no reason you should have to create a new puppetmaster certificate if you already have one. But you shouldn''t have to totally reinstall your puppetmaster if you back it up properly, why reinstall when you could just restore the backup? You can perfectly well use standard HA techniques to make your puppetmaster a high-availability service. You could have a standby node with a duplicate puppetmaster installation ready to activate, or even use standard load-balancing to have multiple puppetmasters. > BTW, I am using puppet to manage about 700+ hosts, I am beginning to worry > about the scenario as mentioned above. I definitely don''t want to login to > each host to clear the "ssl" directory to make it request new certificate. > > Thanks, > -- > LOhit -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Can you describe how to set this up? On Wed, Mar 31, 2010 at 11:23 AM, Ohad Levy <ohadlevy@gmail.com> wrote:> LOhit, > > The main two things to take into account are: > > 1. keep your manifests elsewhere, IMHO puppetmasters always gets RO > copy of your puppet data (e.g. from a VCS). > 2. Solve the SSL hell, then everything is simple. > > I''ve been using in my setup (with approx 15 productive puppetmasters and > about 20 development puppetmasters) a chained CA. > > What it means is that you have one puppet master, which signs other puppet > masters. > in turn, the masters sign the clients, and as they are trusting each other, > you can connect from any server (or client) to any other server without SSL > errors. > > the top level CA can be shut down (even better to keep the SSL data some > where offline) and use it only when you need to sign a new puppet master. > > I find this way relatively clean, and nothing usually happens if i end up > blowing up a puppet master or killing its certificate (as currently 025.4 is > doing but its already fixed for 025.5). > > Ohad > > > On Wed, Mar 31, 2010 at 6:52 PM, LOhit <lohit.b@gmail.com> wrote: > >> Hi, >> >> Since puppet doesn''t have HA/fail over capabilities as of now. How does >> one mitigate a puppet master failure( Ex. Hardware). When you replace the >> server and configure the Puppet masterd, the clients may no longer be able >> to communicate with the server, since the server''s SSL certificates would >> have changed. >> >> BTW, I am using puppet to manage about 700+ hosts, I am beginning to worry >> about the scenario as mentioned above. I definitely don''t want to login to >> each host to clear the "ssl" directory to make it request new certificate. >> >> Thanks, >> -- >> LOhit >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.