I''ve been struggling to find a good way of preventing myself from shooting myself in the foot when pushing new puppet profiles to my servers. I try to do rigorous testing, but there is always the odd system which was setup years ago and keels over when puppet makes the changes. The approach I''ve taken is to run two puppetmasters[1] and only push the changes to one of them at a time, and waiting for the changes to stew before pushing to the other. This way I won''t take down a whole site at once. How have you solved this problem? [1] <http://blogs.sun.com/martin> cheers, /Martin -- Martin Englund, Security Engineer, Web Engineering, Sun Microsystems Inc. Email: martin.englund@sun.com Time Zone: GMT+1 PGP: 1024D/AA514677 "The question is not if you are paranoid, it is if you are paranoid enough." --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
One of the things we started doing is: 1) running puppet via cron. 2) on critical production systems running with --dry-run enabled so that NOTHING happens completely automatically. 3) emailing the output of the dry run if changes show up on a production server. We also have a script on an administrative box that runs puppet over ssh for any server or group of servers. This also means we can schedule change windows as well. On Tue, Jun 23, 2009 at 3:11 PM, martin <martin.englund@sun.com> wrote:> > I''ve been struggling to find a good way of preventing myself from > shooting myself in the foot when pushing new puppet profiles to my > servers. I try to do rigorous testing, but there is always the odd > system which was setup years ago and keels over when puppet makes the > changes. > > The approach I''ve taken is to run two puppetmasters[1] and only push > the changes to one of them at a time, and waiting for the changes to > stew before pushing to the other. This way I won''t take down a whole > site at once. > > How have you solved this problem? > > [1] <http://blogs.sun.com/martin> > > cheers, > /Martin > -- > Martin Englund, Security Engineer, Web Engineering, Sun Microsystems > Inc. > Email: martin.englund@sun.com Time Zone: GMT+1 PGP: 1024D/AA514677 > "The question is not if you are paranoid, it is if you are paranoid > enough." > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Tue, Jun 23, 2009 at 3:11 PM, martin<martin.englund@sun.com> wrote:> > I''ve been struggling to find a good way of preventing myself from > shooting myself in the foot when pushing new puppet profiles to my > servers. I try to do rigorous testing, but there is always the odd > system which was setup years ago and keels over when puppet makes the > changes. > > The approach I''ve taken is to run two puppetmasters[1] and only push > the changes to one of them at a time, and waiting for the changes to > stew before pushing to the other. This way I won''t take down a whole > site at once. > > How have you solved this problem?Why not use environments to actually have an unstable->testing->stable release process? Alternatively you could have a canary environment you push to first, and have a certain percentage of your clients, like 5%, randomly use the canary environment, that way you''re only breaking 5% of your fleet at max when pushing a change.> > [1] <http://blogs.sun.com/martin> > > cheers, > /Martin > -- > Martin Englund, Security Engineer, Web Engineering, Sun Microsystems > Inc. > Email: martin.englund@sun.com Time Zone: GMT+1 PGP: 1024D/AA514677 > "The question is not if you are paranoid, it is if you are paranoid > enough." > > >-- Nigel Kersten nigelk@google.com System Administrator Google, Inc. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
martin wrote:> I''ve been struggling to find a good way of preventing myself from > shooting myself in the foot when pushing new puppet profiles to my > servers. I try to do rigorous testing, but there is always the odd > system which was setup years ago and keels over when puppet makes the > changes. > > The approach I''ve taken is to run two puppetmasters[1] and only push > the changes to one of them at a time, and waiting for the changes to > stew before pushing to the other. This way I won''t take down a whole > site at once. > > How have you solved this problem? > > [1] <http://blogs.sun.com/martin> > > cheers, > /Martin > -- > Martin Englund, Security Engineer, Web Engineering, Sun Microsystems > Inc. > Email: martin.englund@sun.com Time Zone: GMT+1 PGP: 1024D/AA514677 > "The question is not if you are paranoid, it is if you are paranoid > enough." > > >Pre-commits will take care of the low-hanging syntax error fruit, then you can cover the logic errors by having a staging environment. That could be a team of VMs or it could be something like I believe Nigel has suggested (which is what I do ATM because there isn''t budget for a new VM server), push to a puppetmaster that only serves a small small percentage of your nodes. If you have the err tag setup in tagmail, and after some time passes you don''t get notified, you know you''re fairly safe to really go live. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---