Puppet users - Jun 2010 - change management workflow for puppet configs

This is kind of a meta-puppet question. If you are keeping your puppet modules
-- manifests + templates + files -- in revision control (and you are, right?),
how do you manage releasing out to the world? Let me start out with our setup
and if other folks can follow up in a similar format perhaps a summary would be
worth putting in the docs. (General "man don''t do that, do this
instead" commentary on my stuff appreciated also)

- Describe your environment (general application, number of clients, number of
puppetmasters)
internet-facing production services with ~2000 clients split roughly
half-and-half between preproduction/lab environments and prod. one set of 2
apache+passenger+puppetmasters for all lab environments and a second set for
prod.

- Describe revision control setup
subversion server with a branch named ''test'' and a branch
named ''prod''. all puppetmasters live on the prod branch -
$moduledir is a working-copy checkout of svn, and one puppetmaster in each env
has a second server running on :8141 that lives on the test branch.  prod branch
commits should only be ''svn merge''s of commits in test but
this isn''t enforced by technological control.  Post-commit hook sends
out mail with the diff to everyone on the team.

- What do you like about it?
Tight control over changes that go to prod, a fairly straightforward way to test
out changes on real clients (stop real puppetd, run manually with
--masterport=8141), Being able ''svn blame'' on configs is super
helpful.
I love my post-commit email hook - one simple but helpful thing was to set the
From address of the mailouts to be a real list so people can just reply to the
mails and start a code review type discussion.

- What do you NOT like about it? 
We have to manually ''svn up'' on all the puppetmasters after a
commit, which people sometimes don''t do and clients get out of sync.
Merging can be complicated.
Flip-flopping clients back and forth between test branch and production is
slightly messy.


- What are you doing to improve/change the setup?
Pre-commit hook to do syntax checking on files to refuse commits on blatant
errors
Contemplating a post-commit push to ''svn up'' the puppetmasters
but not sure if this is really a good idea
Need some auditing between the branches to see what changes are not yet merged
or if prod has somehow diverged (non-merge changes on the branch)
Maybe one/handful of clients per class of host ought to live pointed at the test
server.



 - Eric Sorenson - N37 17.255 W121 55.738  - http://twitter.com/ahpook  -

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Eric,

thanks for starting this interesting thread!

On 6/25/2010 1:02 AM, Eric Sorenson wrote:
> - Describe revision control setup subversion server with a branch
> named ''test'' and a branch named ''prod''.
all puppetmasters live on the
> prod branch - $moduledir is a working-copy checkout of svn, and one
> puppetmaster in each env has a second server running on :8141 that
> lives on the test branch.  prod branch commits should only be ''svn
> merge''s of commits in test but this isn''t enforced by
technological
> control.  Post-commit hook sends out mail with the diff to everyone
> on the team.
Is there a specific reason you''re not using real environments?
> - What are you doing to improve/change the setup? Pre-commit hook to
> do syntax checking on files to refuse commits on blatant errors
> Contemplating a post-commit push to ''svn up'' the
puppetmasters but
> not sure if this is really a good idea
Guess that depends on the quality of your checkins.
> Need some auditing between the
> branches to see what changes are not yet merged or if prod has
> somehow diverged (non-merge changes on the branch) Maybe one/handful
> of clients per class of host ought to live pointed at the test
> server.
This would also reduce the danger of automatically svn ups.



How are you handling "breaking" changes which shall be done in 
maintenance windows and/or "rolling upgrades"?



Best Regards, David
-- 
dasz.at OG              Tel: +43 (0)664 2602670     Web: http://dasz.at
Klosterneuburg                                         UID: ATU64260999

        FB-Nr.: FN 309285 g          FB-Gericht: LG Korneuburg

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jun 24, 4:02 pm, Eric Sorenson <eric.soren...@me.com>
wrote:
> - Describe your environment (general application, number of clients, number
of puppetmasters)
Internet facing private SaaS application (at my former employer),
about 150 nodes, one puppetmaster for production, one for testing
running on a different port.
> - Describe revision control setup
Commits are pushed to a "central" git repository into topic branches.
This is distinctly different than subversion, which doesn''t handle
branches like Git does.  On each of the two puppet master servers,
environments are setup for individual people to work without affecting
other people.  Environments hack at the modulepath, so this
effectively tells the puppet master where to look for the local git
clone.  During testing, the environment in question is checked out to
the branch in question and the nodes are tested using puppetd --test --
environment=jmccune

To promote from development to testing, commits are cherry-picked or
full on merged into another branch named environment/testing, which is
shared and a sort of "staging area"

To promote from testing to production, the testing branch where topic
branches are merged and staged, we would simply merge the environment/
testing branch into the environment/production branch, then pull down
the branch from the "central" git repository onto the puppet master.
> - What do you like about it?
I love that Git allows you to switch branches quickly and easily
without the filesystem path changing.  This allows for a relatively
static --modulepath configuration setting.  I love that Git
facilitates each topic branch being tied to a specific Sales Force
case number (any ticketing system would work just as well).  I love
the clear and easy promotion from development to testing to
production.  Git is fast.  Very fast.  I like being able to hack on
puppet code on an airplane without network access.  I like being able
to rebase a topic branch and consolidate a number of small commits
into a single patch, then apply it to another branch.
> - What do you NOT like about it?
Git is difficult.  It took me some time to wrap my brain around it,
but once I did everything is great.  Git is difficult for junior level
people to approach as well.  The "central" git repository used a
shared "git" unix account with SSH authorized_keys containing a public
key for each person with commit access.  I was surprised to find many
of my colleagues had difficulty even setting up SSH keys for this
purpose.
> - What are you doing to improve/change the setup?
Pre-commit hook to check syntax before accepting a commit to the
environment/production branch.
Post-commit hook to trigger a git pull on the puppet masters after a
commit to the environment/production branch.
Integration with Salesforce case numbers (Ticketing system).

I also wrote a custom report processor for the puppet master to append
the Git commit ID to every log message generated.  This allowed me to
identify the state of the environment/production repository that
caused a resource to change or whatever log message coming from the
client.  My presentation on this is available at:
http://puppetcamp.org/europe-2010-ghent/schedule-and-speakers/
Slides: http://bit.ly/puppetsplunkslides
Video of the talk: http://blip.tv/file/3759813/
Custom Report Processor:
http://github.com/jeffmccune/puppet-demotools/blob/master/lib/puppet/reports/logversion.rb
Finally, the config_version command line option to actually get the
Git SHA.  This could easily use Subversion as well:
http://github.com/jeffmccune/puppet-demotools/blob/master/bin/get-config-version


Let me know if you have any questions about this setup, I''m happy to
help.

Cheers,
--
Jeff McCune

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.