Puppet users - Apr 2013 - how do you test and release puppet changes?

We are in the process of evaluating our puppet related test and
release process and interested in knowing what other folks are doing.

We are in a position that is not ideal but is not unique from what I
can tell.   Our current testing process is basically the
responsibility of each person making a change.   Small changes are
committed and pushed to dev/qa/prod in one swoop with the committer
spot checking the results manually.    Larger changes are tested by
running a node against a puppet environment which is pointed to the
change branch and the desired behavior is manually verified.

What we would like to do is start with implementing some basic control
points which require passing tests before the changes move along.
With the goal of being able to increase the test coverage over time to
protect ourselves from ourselves.

One thought we had as an initial step is to just verify catalog
compilation for some number of nodes against the proposed changes and
block the changes if catalog compilation fails.   This raises the next
question around tooling.   We could script up a catalog compiler test
calling the the puppet binaries but should we use this as an
opportunity to get familiar with rspec-puppet?

Are people using catalog diffs at all in their release process?   It
would seem nice to provide an automated catalog diff for people making
''small'' changes so they can make sure their change
didn''t accidentally
drop or change a large number of resources.

So please share what you find works or doesn''t work at your shop.

TIA>

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Derek,

when testing puppet-cleaner I wrote puppet-diff[1], which compiles the 
catalogs for two given manifests (before and after changes) and compares 
their YAML representation, previously removing some irrelevant stuff.

That helped me test that some transformations, like whitespace changes or 
single/double [un]quoting some tokens did nothing, or just what they were 
supposed to change. It will help you notice if something was removed or 
added to the catalog, but it will be difficult to test more complex changes.

I''ve read rspec-puppet code and my first impression is that it also 
compiles the catalog but instead of comparing it with another one, it tests 
for anything you tell it; then you have to practically rewrite your 
manifests in rspec DSL.

In any case, we''re just testing the manifest on an imaginary clean box.
The
real result, as you know for sure, heavily depends on the current state of 
the box.

[1] https://github.com/santana/puppet-cleaner/blob/master/bin/puppet-diff

El viernes, 26 de abril de 2013 12:03:47 UTC-5, Derek Olsen
escribió:
>
>
> We are in the process of evaluating our puppet related test and 
> release process and interested in knowing what other folks are doing. 
>
> We are in a position that is not ideal but is not unique from what I 
> can tell.   Our current testing process is basically the 
> responsibility of each person making a change.   Small changes are 
> committed and pushed to dev/qa/prod in one swoop with the committer 
> spot checking the results manually.    Larger changes are tested by 
> running a node against a puppet environment which is pointed to the 
> change branch and the desired behavior is manually verified. 
>
> What we would like to do is start with implementing some basic control 
> points which require passing tests before the changes move along. 
> With the goal of being able to increase the test coverage over time to 
> protect ourselves from ourselves. 
>
> One thought we had as an initial step is to just verify catalog 
> compilation for some number of nodes against the proposed changes and 
> block the changes if catalog compilation fails.   This raises the next 
> question around tooling.   We could script up a catalog compiler test 
> calling the the puppet binaries but should we use this as an 
> opportunity to get familiar with rspec-puppet? 
>
> Are people using catalog diffs at all in their release process?   It 
> would seem nice to provide an automated catalog diff for people making 
> ''small'' changes so they can make sure their change
didn''t accidentally
> drop or change a large number of resources. 
>
> So please share what you find works or doesn''t work at your shop. 
>
> TIA> 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

I''m basically doing the same.

I have replicated my production environment in Vagrant, that means the 
puppetmaster/puppetdb setup as well as the app,db,cache,api layers are 
identical to production in the vagrant setup.

After all tests are done in Vagrant, destroying and re-creating the VMs 
from scratch to make sure any new change or manifests do not break when 
installing a new machine from scratch we submit a pull request of the 
change to a specific branch, somebody else in the team reviews and merges 
it. The requester pull the changes in the production puppetmaster in a 
specific environment, connect to a few nodes in production and run puppet 
agent --environment to validate it. 

Once we are satisfied, we merge that branch into master and pull it into 
the puppetmaster production environment which all nodes use by default.

Any tips to improve this process is welcome :)

Felipe

On Friday, April 26, 2013 10:03:47 AM UTC-7, Derek Olsen
wrote:
>
>
> We are in the process of evaluating our puppet related test and 
> release process and interested in knowing what other folks are doing. 
>
> We are in a position that is not ideal but is not unique from what I 
> can tell.   Our current testing process is basically the 
> responsibility of each person making a change.   Small changes are 
> committed and pushed to dev/qa/prod in one swoop with the committer 
> spot checking the results manually.    Larger changes are tested by 
> running a node against a puppet environment which is pointed to the 
> change branch and the desired behavior is manually verified. 
>
> What we would like to do is start with implementing some basic control 
> points which require passing tests before the changes move along. 
> With the goal of being able to increase the test coverage over time to 
> protect ourselves from ourselves. 
>
> One thought we had as an initial step is to just verify catalog 
> compilation for some number of nodes against the proposed changes and 
> block the changes if catalog compilation fails.   This raises the next 
> question around tooling.   We could script up a catalog compiler test 
> calling the the puppet binaries but should we use this as an 
> opportunity to get familiar with rspec-puppet? 
>
> Are people using catalog diffs at all in their release process?   It 
> would seem nice to provide an automated catalog diff for people making 
> ''small'' changes so they can make sure their change
didn''t accidentally
> drop or change a large number of resources. 
>
> So please share what you find works or doesn''t work at your shop. 
>
> TIA> 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

I''ve heard of some, who use mcollective to order a test-run on all
nodes,
against an environment - with --noop - and then they watch puppet-dashboard 
for nodes that had changes (which ofcourse weren''t actually done -
because
of --noop).

I was thinking of doing exactly that - and then exploiting the feature of 
puppet-dashboard''s environments - so a test of --environment klavs -
would
make the reports get imported into the klavs environment in 
puppet-dashboard as well - effectively giving each puppet dev their own 
dashboard - in which they can see results of ordered tests (run on ALL 
actual production hosts) - and see if something would have changed - that 
shouldn''t etc.

Where I''d go from there, depends on the experience we gather from that
:)

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.