Puppet users - Mar 2011 - Testing changes before deployment

Hi guys,

I''m going to deploy Puppet 2.6 into an environment consisting of a 100+
machines of different kinds. Large group are web servers, other group 
are DB servers, then there are some IT infrastructure ones like LDAP or 
Mail servers.

For start I''m running Puppet on a couple of non-important nodes and if
I
do some changes they either work or not and I rework it. Not a big deal. 
But that''s not possible for production deployment. Things just must
work
before being put out on the prod puppet server.

I''m thinking having a separate test server, do all the changes on that,
verify them on a test client, then commit to SVN or GIT, update the 
catalog on the prod puppet master and get it deployed across the fleet.

But what if I need to do a web-server specific change? And then 
Mail-server specific change? If I had only one test client machine I 
would have to reinstall it every time I needed to do verify a change for 
a different group of servers. That''s hardly a way to go. There must be
a
better approach.

How do you guys test puppet changes before they go live?

Thanks!

GiBo

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

* Giovanni Bordello [2011-03-20]:
> How do you guys test puppet changes before they go live?
We have several lines of defense:
1) we develop our puppet manifests on a development machine where we can
test many changes locally
2) we use cucumber-puppet to verify catalog compilation for all our
servers and ensure some kind of consistency
3) we deploy automatically to our puppetmaster''s testing environment
4) we manually verify changes with "--environment=testing --noop" on
the
target server
5) we manually update our production environment from the testing
environment

Let me know, if you are interested in any details.

cheers,

Nikolay

--
"It''s all part of my Can''t-Do approach to life."
Wally

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mar 19, 2011 9:05 PM, "Giovanni Bordello"
<gibo@gentlemail.com> wrote:
> I''m going to deploy Puppet 2.6 into an environment consisting of a
100+
machines of different kinds. Large group are web servers, other group are DB
servers, then there are some IT infrastructure ones like LDAP or Mail
servers.
>
> For start I''m running Puppet on a couple of non-important nodes
and if I
do some changes they either work or not and I rework it. Not a big deal. But
that''s not possible for production deployment. Things just must work
before
being put out on the prod puppet server.
[…]
> But what if I need to do a web-server specific change? And then
Mail-server specific change? If I had only one test client machine I would
have to reinstall it every time I needed to do verify a change for a
different group of servers. That''s hardly a way to go. There must be a
better approach.

Well, aside from using container based VMs, that was exactly the way we did
our testing, and I don''t think I would strongly advocate any other
approach.

Getting the bootstrap cost of a system down to less than 30 minutes is
pretty easy with puppet, and if things are going right then there should be
a sharp reduction in the number of emergency changes you are needing to
make.

Regards,
    Daniel
-- 
Puppet Labs Developer –http://puppetlabs.com
Daniel Pittman <daniel@puppetlabs.com>
Contact me via gtalk, email, or phone: +1 (877) 575-9775
Sent from a mobile device. Please forgive me if this is briefer than usual.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Nikolay,

I am interested in the details using cucumber-puppet to verify catalog
compilation. Could you please elaborate on it?

Thanks,
Hai

On Sun, Mar 20, 2011 at 3:59 AM, Nikolay Sturm
<google@erisiandiscord.de>wrote:
> * Giovanni Bordello [2011-03-20]:
> > How do you guys test puppet changes before they go live?
>
> We have several lines of defense:
> 1) we develop our puppet manifests on a development machine where we can
> test many changes locally
> 2) we use cucumber-puppet to verify catalog compilation for all our
> servers and ensure some kind of consistency
> 3) we deploy automatically to our puppetmaster''s testing
environment
> 4) we manually verify changes with "--environment=testing --noop"
on the
> target server
> 5) we manually update our production environment from the testing
> environment
>
> Let me know, if you are interested in any details.
>
> cheers,
>
> Nikolay
>
> --
> "It''s all part of my Can''t-Do approach to
life." Wally
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Giovanni Bordello wrote:
> But what if I need to do a web-server specific change? And then 
> Mail-server specific change? If I had only one test client machine I 
> would have to reinstall it every time I needed to do verify a change for 
> a different group of servers. That''s hardly a way to go. There
must be a
> better approach.
> 
> How do you guys test puppet changes before they go live?
First of all, we do have multiple environments, so we can test
new versions of the manifests before taking them into production.
There is of course the an environment named "production" which
clients are usually running from.  Then each sysadmin has their
own personal environment, and some of us have multiple personal
environments.  We do our changes in our personal environment,
test them, and when we feel they are ready, we push them to the
central Git repository, and then do a ''git pull'' in the
production
environment.  I highly recommend this.

Our approach to test machines vary a bit depending on how extensive,
invasive and risky changes we are making.  If I''m doing something
small and simple, like adding a member to a mail alias on the email
server, I just do that, commit, push, pull and let puppetd on the
email server do its job.  (Since I only run puppetd every fourth
hour, not every thirty minutes, I might do a manual run of puppetd
to apply my change quicker.)

For somewhat larger changes, but changes where I feel the risk of
actually harming the server is very low, I would first run puppetd
manually in no-op mode on the live client against my environment.
If that looks okay, I will then run puppetd again and let it do its
changes, and then I test that my changes work as I intended.  If
they don''t, then I revert them (usually manually), and go back to
fixing my manifests.

When the risk gets higher, or if I think something will take me
some time to implement, I install a test server.  That will usually
be a Xen guest.  I make a new node definition in my manifests that
is a copy of the real server I want to change, except that it has
different hostname, IP address, and MAC address.  Then I install
CentOS on that with kickstart (ca 5 minutes), and run Puppet on
it (less than 15 minutes).  I have Puppet generate Xen config
files and a kickstart file for the test machine on the Xen host,
so it is fairly painless.  I do need to manually create the LVM
volumes for the virtual disks, but all in all I can easily have
a clone of server up and running in 30 minutes, including adding
the test machine to DNS and DHCP, installing OS on it and running
Puppet on it.

When I have my test server up and running, I do all my testing
on that.  When I''m close to finished with extensive changes, I
often re-install my test server from scratch to check that
everything really works.  Often I find that they don''t (typically
some missed dependencies) and have to fix that, and then I do
a new re-install.

Since virtual servers are fairly cheap (and I can often give
them less CPU, memory and disk than the real server needs), I
sometimes have several such test servers running, if I am doing
work on several different features at the same time.  (That would
typically be because I started on doing something, then some
other change with higher priority came up and I had to put my
original work aside for a few days or weeks.)


Where the limit is for something you dare test directly on live
production servers, would vary between organisations.  We can
usually tolerate the occasional unplanned downtime if they are
short enough, so I probably have a higher threshold than many
others before I install a separate test machine.

But regardless of how high risks you can take with your production
machines, I heartily recommend that you make it easy to create and
install test machines.  Virtualization is really nice for that.


	/Bellman

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I actually wrote up a post about using multiple environments for this
purpose. It includes the process flow chart as well.

http://cwebber.ucr.edu/2011/03/multiple-environments-in-puppet/

--cwebber

On 3/22/11 2:10 PM, "Thomas Bellman" <bellman@nsc.liu.se> wrote:
>Giovanni Bordello wrote:
>
>> But what if I need to do a web-server specific change? And then
>> Mail-server specific change? If I had only one test client machine I
>> would have to reinstall it every time I needed to do verify a change
>>for 
>> a different group of servers. That''s hardly a way to go. There
must be
>>a 
>> better approach.
>> 
>> How do you guys test puppet changes before they go live?
>
>First of all, we do have multiple environments, so we can test
>new versions of the manifests before taking them into production.
>There is of course the an environment named "production" which
>clients are usually running from.  Then each sysadmin has their
>own personal environment, and some of us have multiple personal
>environments.  We do our changes in our personal environment,
>test them, and when we feel they are ready, we push them to the
>central Git repository, and then do a ''git pull'' in the
production
>environment.  I highly recommend this.
>
>Our approach to test machines vary a bit depending on how extensive,
>invasive and risky changes we are making.  If I''m doing something
>small and simple, like adding a member to a mail alias on the email
>server, I just do that, commit, push, pull and let puppetd on the
>email server do its job.  (Since I only run puppetd every fourth
>hour, not every thirty minutes, I might do a manual run of puppetd
>to apply my change quicker.)
>
>For somewhat larger changes, but changes where I feel the risk of
>actually harming the server is very low, I would first run puppetd
>manually in no-op mode on the live client against my environment.
>If that looks okay, I will then run puppetd again and let it do its
>changes, and then I test that my changes work as I intended.  If
>they don''t, then I revert them (usually manually), and go back to
>fixing my manifests.
>
>When the risk gets higher, or if I think something will take me
>some time to implement, I install a test server.  That will usually
>be a Xen guest.  I make a new node definition in my manifests that
>is a copy of the real server I want to change, except that it has
>different hostname, IP address, and MAC address.  Then I install
>CentOS on that with kickstart (ca 5 minutes), and run Puppet on
>it (less than 15 minutes).  I have Puppet generate Xen config
>files and a kickstart file for the test machine on the Xen host,
>so it is fairly painless.  I do need to manually create the LVM
>volumes for the virtual disks, but all in all I can easily have
>a clone of server up and running in 30 minutes, including adding
>the test machine to DNS and DHCP, installing OS on it and running
>Puppet on it.
>
>When I have my test server up and running, I do all my testing
>on that.  When I''m close to finished with extensive changes, I
>often re-install my test server from scratch to check that
>everything really works.  Often I find that they don''t (typically
>some missed dependencies) and have to fix that, and then I do
>a new re-install.
>
>Since virtual servers are fairly cheap (and I can often give
>them less CPU, memory and disk than the real server needs), I
>sometimes have several such test servers running, if I am doing
>work on several different features at the same time.  (That would
>typically be because I started on doing something, then some
>other change with higher priority came up and I had to put my
>original work aside for a few days or weeks.)
>
>
>Where the limit is for something you dare test directly on live
>production servers, would vary between organisations.  We can
>usually tolerate the occasional unplanned downtime if they are
>short enough, so I probably have a higher threshold than many
>others before I install a separate test machine.
>
>But regardless of how high risks you can take with your production
>machines, I heartily recommend that you make it easy to create and
>install test machines.  Virtualization is really nice for that.
>
>
>    /Bellman
>
>-- 
>You received this message because you are subscribed to the Google Groups
>"Puppet Users" group.
>To post to this group, send email to puppet-users@googlegroups.com.
>To unsubscribe from this group, send email to
>puppet-users+unsubscribe@googlegroups.com.
>For more options, visit this group at
>http://groups.google.com/group/puppet-users?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

* hai wu [2011-03-20]:
> I am interested in the details using cucumber-puppet to verify catalog
> compilation. Could you please elaborate on it?
I setup a catalog policy that applies to all catalogs, by specifying a
scenario for each of our servers like so:

        @node
        Given a node specified by "$nodefact"
        When I compile its catalog
        Then compilation should succeed
        And all resource dependencies should resolve
        And all files should require their owner
        And all services should have a nagios check
        And all websites should have a nagios check
        ...

As the scenarios are almost identical, I generate the policy from a
script. When someone pushes a change, our CI server runs the complete
test suite to verify all server catalogs. We use tagged scenarios, so
that I can run a single relevant scenario (aka verify a specific node''s
catalog) when making changes to a node specific module.

If you have many identical servers, you probably just want to verify the
catalog for each kind of server.

cheers,

Nikolay

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.