Puppet users - Oct 2010 - How do you ensure safety from errors/bugs in large deployments?

Hello all,

This is not a puppet proper issue of course.. but I was wondering if
any of you could share some thoughts...

When you deploy a system like Puppet at a large park of systems, you
instantly increase the efficacy of mistakes and bugs in destroying the
environment....
How do you deal with that? I would be interested in any experience or
input, especially puppet proper best practices...

BTW, does puppet have any means to say something like "don''t ever
do
anything that takes more than 5 minutes? or that involves more than
1000 files/inodes?

Thanks a lot.
Mohamed.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Oct 5, 2010, at 6:05 PM, Mohamed Lrhazi wrote:
> Hello all,
> 
> This is not a puppet proper issue of course.. but I was wondering if
> any of you could share some thoughts...
> 
> When you deploy a system like Puppet at a large park of systems, you
> instantly increase the efficacy of mistakes and bugs in destroying the
> environment....
> How do you deal with that? I would be interested in any experience or
> input, especially puppet proper best practices...
The answer is simple, although simple is not the same as quick or easy.  You can
avoid most problems like that by having different environments and testing your
changes in a development environment on a representative sample of you machines
you are managing.


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Oct 5, 2010 at 6:05 PM, Mohamed Lrhazi <lrhazi@gmail.com>
wrote:
> Hello all,
>
> This is not a puppet proper issue of course.. but I was wondering if
> any of you could share some thoughts...
>
> When you deploy a system like Puppet at a large park of systems, you
> instantly increase the efficacy of mistakes and bugs in destroying the
> environment....
> How do you deal with that? I would be interested in any experience or
> input, especially puppet proper best practices...

Lots of precommit hooks to catch the obvious human errors
Easy to use testing environment so you can test your changes quickly.
Strict release process. We follow unstable -> testing -> stable like
Debian.

You absolutely need to sort your release process out.


>
> BTW, does puppet have any means to say something like "don''t
ever do
> anything that takes more than 5 minutes? or that involves more than
> 1000 files/inodes?
>
> Thanks a lot.
> Mohamed.
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>


-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Oct 5, 2010 at 6:05 PM, Mohamed Lrhazi <lrhazi@gmail.com>
wrote:
> Hello all,
>
> This is not a puppet proper issue of course.. but I was wondering if
> any of you could share some thoughts...
>
> When you deploy a system like Puppet at a large park of systems, you
> instantly increase the efficacy of mistakes and bugs in destroying the
> environment....
With great power comes great responsibility.
> How do you deal with that? I would be interested in any experience or
> input, especially puppet proper best practices...
In addition to the other responses (testing, some more testing, and a
well defined release process) I also recommend a "Big red stop button"
of sorts.  Few sites seem to employ this, but the ones who have have
reported it to be incredibly useful when necessary and enjoy a little
more peace of mind.

The big red stop button can take many forms.  I personally use a
wrapper script to launch puppet agent runs from cron, so the stop
button in my case  could be a simple curl command before running
puppet agent and the agent proceeds if and only if the curl command
does not get a valid 200 response from the HTTP request.

This allow you to quickly touch a file somewhere and know it will halt
all puppet agents on all of your nodes.

-- 
Jeff McCune
http://www.puppetlabs.com/

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Oct 5, 2010 at 7:39 PM, Jeff McCune <jeff@puppetlabs.com>
wrote:
> On Tue, Oct 5, 2010 at 6:05 PM, Mohamed Lrhazi <lrhazi@gmail.com>
wrote:
>> Hello all,
>>
>> This is not a puppet proper issue of course.. but I was wondering if
>> any of you could share some thoughts...
>>
>> When you deploy a system like Puppet at a large park of systems, you
>> instantly increase the efficacy of mistakes and bugs in destroying the
>> environment....
>
> With great power comes great responsibility.
>
>> How do you deal with that? I would be interested in any experience or
>> input, especially puppet proper best practices...
>
> In addition to the other responses (testing, some more testing, and a
> well defined release process) I also recommend a "Big red stop
button"
> of sorts.  Few sites seem to employ this, but the ones who have have
> reported it to be incredibly useful when necessary and enjoy a little
> more peace of mind.
>
> The big red stop button can take many forms.  I personally use a
> wrapper script to launch puppet agent runs from cron, so the stop
> button in my case  could be a simple curl command before running
> puppet agent and the agent proceeds if and only if the curl command
> does not get a valid 200 response from the HTTP request.
>
> This allow you to quickly touch a file somewhere and know it will halt
> all puppet agents on all of your nodes.

We have a slightly more sophisticated BigRedButton that we use.

The clients make an HTTP request, supplying some core facts as
parameters, like domain, hardware-type, major OS version, release
track, etc.

If they get anything other than a 404 for the response, they stop.

Server side it''s something you could whip up in any language quickly,
where we have a simple YAML file that the app reads to work out
whether it should be returning non-404 depending upon the parameters
provided.

This allows you to have slightly more fine-grained control over a
disparate fleet, where you can block individual geographic regions,
releases, hardware models from executing their normal update.

You could implement all of this in puppet pre-run hooks nowadays
rather than a wrapper script, but we have other reasons to want a
wrapper. Admittedly these reasons are getting smaller and smaller each
puppet release. :)

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Oct 6, 2010 at 10:43 AM, Nigel Kersten
<nigel@explanatorygap.net>wrote:
> On Tue, Oct 5, 2010 at 7:39 PM, Jeff McCune <jeff@puppetlabs.com>
wrote:
> > On Tue, Oct 5, 2010 at 6:05 PM, Mohamed Lrhazi
<lrhazi@gmail.com> wrote:
> >> Hello all,
> >>
> >> This is not a puppet proper issue of course.. but I was wondering
if
> >> any of you could share some thoughts...
> >>
> >> When you deploy a system like Puppet at a large park of systems,
you
> >> instantly increase the efficacy of mistakes and bugs in destroying
the
> >> environment....
> >
> > With great power comes great responsibility.
> >
> >> How do you deal with that? I would be interested in any experience
or
> >> input, especially puppet proper best practices...
> >
> > In addition to the other responses (testing, some more testing, and a
> > well defined release process) I also recommend a "Big red stop
button"
> > of sorts.  Few sites seem to employ this, but the ones who have have
> > reported it to be incredibly useful when necessary and enjoy a little
> > more peace of mind.
> >
> > The big red stop button can take many forms.  I personally use a
> > wrapper script to launch puppet agent runs from cron, so the stop
> > button in my case  could be a simple curl command before running
> > puppet agent and the agent proceeds if and only if the curl command
> > does not get a valid 200 response from the HTTP request.
> >
> > This allow you to quickly touch a file somewhere and know it will halt
> > all puppet agents on all of your nodes.
>
>
> We have a slightly more sophisticated BigRedButton that we use.
>
> The clients make an HTTP request, supplying some core facts as
> parameters, like domain, hardware-type, major OS version, release
> track, etc.
>
> If they get anything other than a 404 for the response, they stop.
>
> Server side it''s something you could whip up in any language
quickly,
> where we have a simple YAML file that the app reads to work out
> whether it should be returning non-404 depending upon the parameters
> provided.
>
> This allows you to have slightly more fine-grained control over a
> disparate fleet, where you can block individual geographic regions,
> releases, hardware models from executing their normal update.
>
> You could implement all of this in puppet pre-run hooks nowadays
> rather than a wrapper script, but we have other reasons to want a
> wrapper. Admittedly these reasons are getting smaller and smaller each
> puppet release. :)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
Write a script that does a diff between what''s in revision control, and
what''s in /etc/puppet. Show those changes to the user and prompt before
updating the local working copy. Then, do a puppetrun.

Doug.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.