Puppet users - Jul 2010 - Recipes release management : separate test and prod

Hi there,

Is there a canonical way to manage my recipes'' release process ? I mean
test
it before they''re pushed in production.

For the moment we only have 1 puppetmaster here. I noticed that if I have a
syntax error in one of my included .pp, then puppet runs fail on all nodes
(even if they do not use this recipe), which is not acceptable. More
generally, working directly on production recipes seems too riskee, since
people who will manage those recipes are not devs but sysadmins, which means
poor development skills (and no versionning for the moment).

I found the documentation about puppet "environments" (
http://docs.puppetlabs.com/guides/environment.html ) : does anybody use it
here ? I also read some of you have 2 puppetmasters running on different
ports, for test and prod : how do you do that (init scripts available
somewhere? different conf/manifests directories? sync between test and
prod?). Do you have a pool of client machines dedicated to your tests ? Do
you handle this kind of problems directly with your version control system
(post hooks?), or maybe at an other level in your external node provider ?

I was not able to find much information about that, but don''t hesitate
to
redirect me to an article or a thread here if it has already been discussed.

Thank you,
Cheers,
--
Jean-Baptiste

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Jean-Baptiste Barth <jeanbaptiste.barth@gmail.com> writes:
> Is there a canonical way to manage my recipes'' release process? I
mean test
> it before they''re pushed in production.
Not "canonical", as far as I know.
> For the moment we only have 1 puppetmaster here. I noticed that if I have a
> syntax error in one of my included .pp, then puppet runs fail on all nodes
> (even if they do not use this recipe), which is not acceptable.
We work on this by keeping the puppet configuration in Subversion.  We then
have a Makefile at the top level which runs a puppet syntax check on any
modified file before a commit.

While the puppet check option is pretty terrible[1], it does catch the most
gratuitous syntax errors before they get into the repository.
> More generally, working directly on production recipes seems too riskee,
> since people who will manage those recipes are not devs but sysadmins,
which
> means poor development skills (and no versionning for the moment).
You should explain to your current sysadmins that they are not good enough,
and that if they don''t improve you will hire better sysadmins —
seriously.

That said, wrapping up the process in something like a Makefile that will do
the Subversion commit, and using appropriate hooks so that the push to
production is automatic means you can work around almost anything:

All you have to do is document the point-and-drool version of the
instructions, then make sure they are tasked by their manager to follow those
instructions.[2]


Also useful is a culture of code review: when they write puppet code it should
be complete and thorough, not "experimenting on the production
systems", so
they should be able to have a second member of staff review their change
*before* they are allowed to commit it.
> I found the documentation about puppet "environments"
> ( http://docs.puppetlabs.com/ guides/environment.html ) : does anybody use
> it here?
Yes.  Other than the semi-documented limitations about custom types and
providers, we use them to good effect.  They allowed me to set the standard
for my sysadmin staff to have them test everything before they release.
> I also read some of you have 2 puppetmasters running on different ports,
for
> test and prod: how do you do that (init scripts available somewhere?
> different conf/manifests directories? sync between test and prod?).
I wouldn''t bother with this, for all that it would get different types:
it
seems over-kill for the return on the investment.
> Do you have a pool of client machines dedicated to your tests?
Sometimes.  Other times we rely on testing the modified manifest in place.
The former, using a dedicated test machine, is vastly preferable.


Mostly, though, I think your problem will be best solved by getting the other
sysadmins doing the right thing — which might mean getting management involved
to make sure it actually happes.

Regards,
        Daniel

Footnotes: 
[1]  It doesn''t do *ANY* validation of types, so you can have something
     obviously stupid like ''file { "foo": requiref => blah
}'' and it passes.
     It would be really nice if it checked even just that the keys were
     vaguely valid.

[2]  Of course, this will take a few rounds of them finding the flaws in the
     instructions, and you improving the document, but eventually...

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 5 July 2010 11:01, Jean-Baptiste Barth
<jeanbaptiste.barth@gmail.com>wrote:
> For the moment we only have 1 puppetmaster here. I noticed that if I have a
> syntax error in one of my included .pp, then puppet runs fail on all nodes
> (even if they do not use this recipe), which is not acceptable. More
> generally, working directly on production recipes seems too riskee, since
> people who will manage those recipes are not devs but sysadmins, which
means
> poor development skills (and no versionning for the moment).
>
> I found the documentation about puppet "environments" (
> http://docs.puppetlabs.com/guides/environment.html ) : does anybody use it
> here ? I also read some of you have 2 puppetmasters running on different
> ports, for test and prod : how do you do that (init scripts available
> somewhere? different conf/manifests directories? sync between test and
> prod?). Do you have a pool of client machines dedicated to your tests ? Do
> you handle this kind of problems directly with your version control system
> (post hooks?), or maybe at an other level in your external node provider ?
>
Mostly echoing Daniel Pittman''s comments..

 - We use environments a lot. The default production environment is for,
well, production. All other environments are for development use. Each has
their own SVN branch and Puppet `manifest`/`modulepath`/extlookup
directives.

 - In addition to custom types, providers and facts, flapping back and forth
between environments. It''s also not possible to have separate
fileservers
for additional paths between environments. But it''s probably not
something
that will affect you at this stage.

 - We have virtualised dev/staging machines which reflect the roles of our
production machines, only on a slightly smaller scale. Any changes get
pushed here and monitored first. We also have some shell scripts which are a
shorthand for `--test --noop` to quickly review what is going to happen for
a given machine. When we''re happy with the results then the changes get
merged to production.

 - Syntax errors are caught by a pre-commit hook. It won''t guard
against
stupid yet syntactically correct code. But it does vastly reduce the time
spent dealing with simple humanoid typos.

 - Be sure to version control your manifests from day one. Without doing so,
you''ll have no way of knowing who changed what and when. Nor the
ability to
roll back to a point in history. It will also provide you with an easy
mechanism to peer review any code that you''re not confident with, as
Daniel
notes.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

First of all, if your sysadmins aren''t advocates of version control
(even
more so than your developers), fire them now, Now, NOW.  Or at least hire a
competent senior sysadmin to train them properly.

Back to the topic at hand.  We have only two environment (production and
development).  Since the puppet.conf file is itself managed by puppet,
adding "environment = <%= environment %>" to the [puppetd]
stanza means that
switching a system between production and development is as simple as
"puppetd --test --environment development".  That change is
"sticky", in
that it won''t revert without explicitly doing "puppetd --test
--environment
production".

We keep our manifests in Bazaar.  When we''ve got a successfully
tested/reviewed set of changes in development, we do a "bzr commit" on
the
development side, and then a "bzr merge bzr://bzr/puppet/puppet-dev;bzr
commit" on the production side.  The only real down side is that moving
code
to production means that either *everything* in development is now ready for
production, or that you can isolate the interesting changes for the
commit/merge.  Since there''s really only one of us working on manifests
at
the moment, it''s not a big deal.

For future needs, we''ve got a 2nd puppetmaster set up identically that
we
can use if we''re afraid the changes might crash the production
puppetmaster.

On Mon, Jul 5, 2010 at 9:40 AM, Dan Carley <dan.carley@gmail.com> wrote:
> On 5 July 2010 11:01, Jean-Baptiste Barth
<jeanbaptiste.barth@gmail.com>wrote:
>
>> For the moment we only have 1 puppetmaster here. I noticed that if I
have
>> a syntax error in one of my included .pp, then puppet runs fail on all
nodes
>> (even if they do not use this recipe), which is not acceptable. More
>> generally, working directly on production recipes seems too riskee,
since
>> people who will manage those recipes are not devs but sysadmins, which
means
>> poor development skills (and no versionning for the moment).
>>
>> I found the documentation about puppet "environments" (
>> http://docs.puppetlabs.com/guides/environment.html ) : does anybody use
>> it here ? I also read some of you have 2 puppetmasters running on
different
>> ports, for test and prod : how do you do that (init scripts available
>> somewhere? different conf/manifests directories? sync between test and
>> prod?). Do you have a pool of client machines dedicated to your tests ?
Do
>> you handle this kind of problems directly with your version control
system
>> (post hooks?), or maybe at an other level in your external node
provider ?
>>
>
> Mostly echoing Daniel Pittman''s comments..
>
>  - We use environments a lot. The default production environment is for,
> well, production. All other environments are for development use. Each has
> their own SVN branch and Puppet `manifest`/`modulepath`/extlookup
> directives.
>
>  - In addition to custom types, providers and facts, flapping back and
> forth between environments. It''s also not possible to have
separate
> fileservers for additional paths between environments. But it''s
probably not
> something that will affect you at this stage.
>
>  - We have virtualised dev/staging machines which reflect the roles of our
> production machines, only on a slightly smaller scale. Any changes get
> pushed here and monitored first. We also have some shell scripts which are
a
> shorthand for `--test --noop` to quickly review what is going to happen for
> a given machine. When we''re happy with the results then the
changes get
> merged to production.
>
>  - Syntax errors are caught by a pre-commit hook. It won''t guard
against
> stupid yet syntactically correct code. But it does vastly reduce the time
> spent dealing with simple humanoid typos.
>
>  - Be sure to version control your manifests from day one. Without doing
> so, you''ll have no way of knowing who changed what and when. Nor
the ability
> to roll back to a point in history. It will also provide you with an easy
> mechanism to peer review any code that you''re not confident with,
as Daniel
> notes.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Jul 5, 2010 at 3:01 AM, Jean-Baptiste Barth
<jeanbaptiste.barth@gmail.com> wrote:
> Hi there,
> Is there a canonical way to manage my recipes'' release process ? I
mean test
> it before they''re pushed in production.
I recommend adding a "tests" directory in your modules which include
the classes in the manifests directory.

This allows you to quickly "try out" discrete pieces of your
configuration using puppet stand alone.

If you''d like to test complete configurations, I recommend setting up
a "testing" environment for this purpose.  I stage changes into the
testing environment and exercise them before promotion to production
using puppetd --environment=testing

Of course, version control is really nice here.  Specifically, Git''s
ability to switch branches quickly in the same directory works nicely
with environment configurations.

-- 
Jeff McCune
http://www.puppetlabs.com/

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

> We work on this by keeping the puppet configuration in Subversion.  We then
> have a Makefile at the top level which runs a puppet syntax check on any
> modified file before a commit.
>    
Uh, would you mind divulging to someone with a little less experience, 
how this syntax check works? I can read make, if you just want to post 
the contents of it.

Thanks in advance,
Chris

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Jul 5, 2010 at 9:56 PM, christopher floess
<skeptikos@gmail.com>wrote:
>
>  We work on this by keeping the puppet configuration in Subversion.  We
>> then
>> have a Makefile at the top level which runs a puppet syntax check on
any
>> modified file before a commit.
>>
>>
> Uh, would you mind divulging to someone with a little less experience, how
> this syntax check works? I can read make, if you just want to post the
> contents of it.
>
puppet --parseonly can perform syntax check on the file. A detailed SVN
precommit hook example is available at:
http://projects.puppetlabs.com/projects/puppet/wiki/Puppet%20Version%20Control

Thanks,

Nan

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

christopher floess <skeptikos@gmail.com> writes:
>> We work on this by keeping the puppet configuration in Subversion.  We
then
>> have a Makefile at the top level which runs a puppet syntax check on
any
>> modified file before a commit.
>
> Uh, would you mind divulging to someone with a little less experience, how
> this syntax check works? I can read make, if you just want to post the
> contents of it.
Requires GNU Make, and possibly bash as the default shell:

check     := pp erb rb
check_pp  = puppet --confdir=/tmp --vardir=/tmp --parseonly --ignoreimport \
                --color=false --modulepath=./modules $< && echo
"$< syntax OK"
check_erb = echo -n "$<: " && erb -x -T
''-'' $< | ruby -c
check_rb  = echo -n "$<: " && ruby -c $<

define check_conflicts
if /bin/egrep -q -x -e
''^(<<<<<<<|=======|>>>>>>>)''
''$<''; then echo "ERROR: conflict marker found in
$@"; /bin/false; else :; fi
endef

all: pull test
push: all
	@echo "Committing your changes to the upstream server..."
	@svn commit

to_test := $(foreach ext,$(check),$(shell find * -name "*.$(ext)"
-printf ''.check/%p\n'' | sort))

test: $(to_test)
.check/%: ext=$(subst .,,$(suffix $(<F)))
.check/%: %
	@mkdir -p $(dir $@)
	@$(check_conflicts)
	@$(or $(check_$(ext)),$(error syntax rule check_$(ext) is not defined))
	@touch $@

> Thanks in advance,
Don''t thank me until you see the code. ;)

        Daniel

Also, this is edited to excise support for git, and remove extraneous bits and
some other extension-based syntax checks for non-puppet bits we use.

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks for all your advices, will be very very useful.

Just one thing about versionning : I personnally put into git
everything I can. The sysadmins at my new job don''t have this habit,
this is just a fact.
So it''s obvious I''ll spend some afternoons to teach them git,
svn or
any vcs, but sometimes we have to cope with the reality ;)

Thanks again,

Jean-Baptiste

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Dan Carley wrote:
> 
>  - In addition to custom types, providers and facts, flapping back and
> forth between environments. It''s also not possible to have
separate
> fileservers for additional paths between environments. But it''s
probably
> not something that will affect you at this stage.
> 
Dan & Daniel (and others... :))

Just so people know this is fixed in 2.6 and when we''re finished
hammering the alpha into a beta - we''d love you guys to test this.

Regards

James Turnbull

-- 
Puppet Labs - http://www.puppetlabs.com
C: 503-734-8571

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 6 July 2010 21:21, James Turnbull <james@puppetlabs.com> wrote:
> Just so people know this is fixed in 2.6 and when we''re finished
> hammering the alpha into a beta - we''d love you guys to test this.
>
Sure, I''ll try and put some time aside for testing.

It sounds like 2.6 should resolve a few of the bugbears I''ve
accumulated
over time.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.