Puppet users - Sep 2008 - Using git to manage puppet manifests.

I''ve set up puppet and had it running, but never bothered to set up an
version control.  I''ve decided it is time to get it all in version
control as the system is about to become production and I need to keep
track of who is changing things and what is being changed.

I am looking at storing all my configs in Git as that seems to be the
version control of choice here and it''s time I learned it.  I was
wondering what other people do to automatically update their puppet
server with the latest version from git.  I was looking at using some
sort of git hook, but I''m not sure how to implement it efficiently and
securely with a shared repository on another server.  I went looking
on the wiki, but could only find information about doing this with
subversion.

Any examples of what others are doing would be appreciated so I can
figure out the best approach.

Thanks,

Leah
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi,

On Thu, 2008-09-18 at 06:56 -0700, Leah wrote:
> I am looking at storing all my configs in Git as that seems to be the
> version control of choice here and it''s time I learned it.  I was
> wondering what other people do to automatically update their puppet
> server with the latest version from git.  I was looking at using some
> sort of git hook, but I''m not sure how to implement it efficiently
and
> securely with a shared repository on another server.  I went looking
> on the wiki, but could only find information about doing this with
> subversion.
> 
> Any examples of what others are doing would be appreciated so I can
> figure out the best approach.
This is not a complete HOWTO, but I outlined what we''re doing.

Here we have a (group) shared git repository on a server (the
puppetmaster in fact). The repository belongs to the group commiter.
Each commiter has an ssh account and also belongs to the commiter group.
Each commiter has sudo rights to call
"/usr/local/bin/puppet-update" (see below).

The git repository is created with shared mode (git init
--shared=group).
The git repository is bare.

I cloned this repository in /etc/puppet has root.
Then there is a post-update hook in the repository that does:
exec sudo -u root /usr/local/bin/puppet-update

With puppet-update doing basically:
#!/bin/sh

umask 002
( cd /etc/puppet
  git pull --verbose)

Hence, when we commit, and git push to the central
repository, /etc/puppet is automatically updated.

Hope that helps,
-- 
Brice Figureau <brice-puppet@daysofwonder.com>


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

I use a rake task:
desc "Install your manifests"
task :install => [ :update, :test ] do
  sh %{git push}
  sh %{ssh #{MASTER} ''cd /etc/puppet; sudo git pull''}
end

2008/9/19 Leah <leahfist@gmail.com>
>
> I''ve set up puppet and had it running, but never bothered to set
up an
> version control.  I''ve decided it is time to get it all in version
> control as the system is about to become production and I need to keep
> track of who is changing things and what is being changed.
>
> I am looking at storing all my configs in Git as that seems to be the
> version control of choice here and it''s time I learned it.  I was
> wondering what other people do to automatically update their puppet
> server with the latest version from git.  I was looking at using some
> sort of git hook, but I''m not sure how to implement it efficiently
and
> securely with a shared repository on another server.  I went looking
> on the wiki, but could only find information about doing this with
> subversion.
>
> Any examples of what others are doing would be appreciated so I can
> figure out the best approach.
>
> Thanks,
>
> Leah
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

>
> sort of git hook, but I''m not sure how to implement it efficiently
and
> securely with a shared repository on another server.  I went looking
I thought about this, but abandoned it.  Hooks in git have to be  
configured on each repository, since there is no central repository,  
as such.

I keep the three different environments on different branches, and use  
a script to update /etc/puppet from the repository every ten minutes.  
You have to clone the repository manually to do the initial config but  
after that it more or less runs itself. I run it as the puppet user.

Darrell


#!/bin/sh
#
# we do this so we get the right ssh keyfile
HOME=`/usr/bin/getent passwd puppet | /bin/cut -d: -f6`

localrepo=/etc/scripts
repopuppetdir=${localrepo}/puppet

puppetdir=/etc/puppet

gitrepo=''git@github.com:XXXXX.git''

git=/usr/bin/git
rsync=/usr/bin/rsync

puppetmasterpid=/var/run/puppet/puppetmasterd.pid

environments="production staging"

if [ ! -d ${localrepo} ]; then
         if [ -w `dirname ${localrepo}` ]; then
                 $git clone -q $gitrepo $localrepo
                 status=$?
                 if [ $status -ne ''0'' ]; then
                         echo "git clone failed in $0"
                         echo "removing stray ${localrepo}"
                         /bin/rm -rf ${localrepo}
                         exit 1
                 fi
                 #
                 # just to be safe
                 chown -R puppet:puppet $localrepo
                 chmod 750 $localrepo
         else
                 echo "could not greate git clone for puppet in $0"
                 exit 1
         fi

else
         # repository seems to exist, let''s update it
         # apparently -q to git doesn''t mean that it won''t
tell us
         # that nothing''s different. wankers.
         cd $localrepo && $git checkout -q master > /dev/null
         status=$?
         if [ $status -ne ''0'' ]; then
                 echo "$0 : switching to master failed. bailing"
                 exit 1
         fi

         cd $localrepo && $git pull -q -n > /dev/null 2>&1
         status=$?
         if [ $status -ne ''0'' ]; then
                 echo "bailing: git pull failed in $0"
                 exit 1
         fi
fi

# master gets synced to development branch
$rsync --exclude=ssl --exclude=scripts -aq --delete ${repopuppetdir}/ / 
etc/puppet/development

for env in $environments; do
  cd $localrepo && $git checkout -q $env
  cd $localrepo && $git pull -q -n > /dev/null 2>&1
  $rsync --exclude=ssl --exclude=scripts -aq --delete $ 
{repopuppetdir}/ /etc/puppet/${env}

status=$?

if [ $? -ne "0" ]; then
         echo "rsync failed to sync branch $env in puppet repository  
on `hostname` - please investigate"

fi
done


$git checkout -q master

exit 0


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Thu, Sep 18, 2008 at 6:56 AM, Leah <leahfist@gmail.com>
wrote:
>
> I''ve set up puppet and had it running, but never bothered to set
up an
> version control.  I''ve decided it is time to get it all in version
> control as the system is about to become production and I need to keep
> track of who is changing things and what is being changed.
Good plan.
> I am looking at storing all my configs in Git as that seems to be the
> version control of choice here and it''s time I learned it.  I was
> wondering what other people do to automatically update their puppet
> server with the latest version from git.  I was looking at using some
> sort of git hook, but I''m not sure how to implement it efficiently
and
> securely with a shared repository on another server.  I went looking
> on the wiki, but could only find information about doing this with
> subversion.
I can''t even say how bad an idea it is to automatically update from
your version control system. Always make a conscious decision to push
changes to "production".

What we do is we have several environments: one defined for each
person working on puppet (2 of us for now), one "staging" environment,
and one "production" environment. When we bootstrap machines, they get
a puppet.conf that puts them into the "production" environment.

The various environments point to separate directories, each of which
is a working copy (we use SVN, but this is totally about process, not
specific software). The developer directories are usually a checkout
of a branch. The staging directory is the checkout of whatever we
propose to put into production. The production directory is a checkout
of a specific tag.

A developer who wants to add something does development in his own
environment and points his development machine(s) or VM(s) at this
environment. Once he is ready he asks the release manager (me) to
stage the changes. I usually merge the branch to trunk and tag it,
then update the staging environment to this tag. We have a
representative sample of our machines set up with a second instance of
puppet, running with --noop, set to the staging environment. We wait a
couple of hours (currently long enough to verify our setup, this time
may grow in the future) and if we don''t see anything scary in the logs
from these machines we update the production environment to this tag.
Otherwise, we tell the developer to keep trying and revert the change
in the repository.

--Paul

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi there

On 18 Sep, 14:56, Leah <leahf...@gmail.com> wrote:
> I am looking at storing all my configs in Git as that seems to be the
> version control of choice here and it''s time I learned it.  I was
> wondering what other people do to automatically update their puppet
> server with the latest version from git.  I was looking at using some
> sort of git hook, but I''m not sure how to implement it efficiently
and
> securely with a shared repository on another server.  I went looking
> on the wiki, but could only find information about doing this with
> subversion.
I''ve just started working on getting our Puppet configurations under
version control (currently Git). Like Paul (who might''ve suggested
this to me in the first place) we have environments for production,
testing and for each developer/admin. Each of these corresponds to a
branch within Git. Developers develop in their local environment, push
changes into testing to be staged (we''ll have representative machines
running testing at some point) and then to production once we''re sure
it''s okay. I manually pull changes from the repo to the puppetmasters.

It works fairly well but not perfectly. Problems are:

1) It''s fiddly. From my cloned repo I have to switch to my branch,
make the change, commit, push to the central repo, switch to testing,
merge with my branch and then push to the central repo to get a change
live. For most things that''s fine -- I want to be sure -- but when a
developer comes along and wants a rewriterule fixed or some other
simple change or feature request that''s already signed off from above
it''s a bit of a task to get it live. (Perhaps we should consider
storing application configs elsewhere, in a more straightforward repo,
or building them into an RPM or something.)

2) Git''s distributed approach is a bit confusing (read: I
don''t know
how to work it). Local branches tracking remote (central) branches,
pushing a local branch to a remote branch, etc, makes my brain ache a
bit. SVN''s versioned-tree model might work better, especially if you
can just set tags representing testing and production and check those
tags out to the puppetmasters. However, Git''s branching does make it
possible to develop several features in parallel, which a tagged
approach might not give you. Swings and roundabouts. Git has tags but
I haven''t yet figured out how they work.

3) I haven''t worked out how to get the central repo to push to all the
puppetmasters. Like Paul, I don''t want this to happen automatically.
What I really want is a way for admins to push their changes to
testing, have this action recorded and sent to a mailing list (with a
diff!) by running a single script on the central repo. Same for
production. Currently this happens when an admin merges changes into
his local testing/production and then pushes these branches to the
central repo, which I think might be asking for trouble.

Sorry, this is a bit rambly -- early morning and all that. I''m on the
Puppet course next week so I''m hoping I''ll learn The One True
Way
then.

Cheers,

Mark
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

>
>
> It works fairly well but not perfectly. Problems are:
>
> 1) It''s fiddly. From my cloned repo I have to switch to my branch,
> make the change, commit, push to the central repo, switch to testing,
> merge with my branch and then push to the central repo to get a change
> live. For most things that''s fine -- I want to be sure -- but when
a
> developer comes along and wants a rewriterule fixed or some other
> simple change or feature request that''s already signed off from
above
> it''s a bit of a task to get it live. (Perhaps we should consider
> storing application configs elsewhere, in a more straightforward repo,
> or building them into an RPM or something.)
>
I guess the real question is which kind of changes do you have in your
environment.
once you identify them, you could have an action plan for each type of
change.. e.g. changes that really effect system services (must be
coordinated), changes which are minor and doesn''t effect the
application/server productivity, changes which effect only a subset of hosts
etc...

I ended up with having 99% of our modules tagged, and having one module
which is polled from trunk/head.
this allows the admins to push changes immediately if needed by inheriting
the correct class in this module.

the tricky part is that you must audit this changes, and over time inherit
them into your stable environment.

you could break this even deeper to manifests changes/ file changes (which
is simply permissions in your version control - e.g. who can check in this
file).


> 2) Git''s distributed approach is a bit confusing (read: I
don''t know
> how to work it). Local branches tracking remote (central) branches,
> pushing a local branch to a remote branch, etc, makes my brain ache a
> bit. SVN''s versioned-tree model might work better, especially if
you
> can just set tags representing testing and production and check those
> tags out to the puppetmasters. However, Git''s branching does make
it
> possible to develop several features in parallel, which a tagged
> approach might not give you. Swings and roundabouts. Git has tags but
> I haven''t yet figured out how they work.
hmm.. Learn git or move to subversion ;)
>
>
> 3) I haven''t worked out how to get the central repo to push to all
the
> puppetmasters. Like Paul, I don''t want this to happen
automatically.
> What I really want is a way for admins to push their changes to
> testing, have this action recorded and sent to a mailing list (with a
> diff!) by running a single script on the central repo. Same for
> production. Currently this happens when an admin merges changes into
> his local testing/production and then pushes these branches to the
> central repo, which I think might be asking for trouble.
>
If you have a stable pretested environment, why don''t you want it to be
pushed to all of your puppetmasters?
do you really want to allow all of the admins full access to each
puppetmaster?

in my environement, each puppet masters update the
"stable/testing/development/whatever" code automaticily (with cron or
in the
puppet run on the puppetmaster itself).

in general, no one is allowed to change the tags, the worst thing that could
happen is that someone checkin a broken manifest to trunk and then (if its
not reject by a puppet --parseonly hook) you can fix it, but in no way it
effects your running production clients.


>
> Sorry, this is a bit rambly -- early morning and all that. I''m on
the
> Puppet course next week so I''m hoping I''ll learn The One
True Way
> then.
Hmm... I can tell you that it took a loooot of time to get things running
they way we need them to... most of this time was just to graspe the change
managmenet processes..good luck :)

Ohad
>
>
> Cheers,
>
> Mark
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---