Puppet users - Jan 2011 - race conditions in using "svn update" in puppetmaster's /etc/puppet directory?

In multiple places in the Puppet wiki, I see the advice to store the
puppermaster configuration in a VCS (specifically, a Subversion
repository), and then checkout that Subversion repository into /etc/
puppet. That way, the puppermaster''s configuration can be updated by
simply running (on the puppetmaster):

$ cd /etc/puppet
$ svn update

But this approach raises an interesting question: the process of
updating a checked-out copy of a Subversion repository is NOT atomic.
If the puppetmaster were to read files in /etc/puppet while a lengthy
"svn update" operation was updating /etc/puppet to the latest revision
(e.g., from revision 6 to revision 7), it is possible that the
puppetmaster would see a mix of files from revisions 6 and 7. That
could cause problems.

So, here''s my question: if you are currently using the "svn
update"
approach to manage /etc/puppet on the puppetmaster, have you taken
conscious steps to help avoid a race condition? If so, what are they?
And if not, why not?

(Note that you don''t have to worry about individual files containing a
mix of revision 6 versus revision 7 content, because "svn update"
modifies files by first creating a new temporary file, then using the
rename(2) system call (which IS atomic) to move the new file into
place...)

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi James,

2011/1/3 James Ralston <ralston@pobox.com>
> [...]
> So, here''s my question: if you are currently using the "svn
update"
> approach to manage /etc/puppet on the puppetmaster, have you taken
> conscious steps to help avoid a race condition? If so, what are they?
> And if not, why not?
> [...]
>
I use a script (see attachement) which build up a local copy of /etc/puppet
from SVN. When it''s done, the master is stopped, the configuration is
replaced and the master is started again. The script isn''t bullet proof
yet,
but it shouldn''t be a big deal to implement some additional checks to
make
it more robust.

Kind regards,
Thomas

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

> So, here''s my question: if you are currently using the "svn
update"
> approach to manage /etc/puppet on the puppetmaster, have you taken
> conscious steps to help avoid a race condition? If so, what are they?
> And if not, why not?
> 
I do something like this to deploy all web apps, the Puppet master included:


SHA=$(git rev-parse HEAD)
git archive --prefix=puppet-$SHA/ $SHA | gzip >puppet-$SHA.tar.gz
scp puppet-$SHA.tar.gz puppet:/etc
ssh puppet "
tar xf /etc/puppet-$SHA.tar.gz -C /etc
ln -s /etc/puppet-$SHA /etc/puppet+
mv -T /etc/puppet+ /etc/puppet
#find /etc -mindepth 1 -maxdepth 1 \
# -name puppet-\* -and -not -name puppet-$SHA | xargs rm -rf
"



The `mv -T /etc/puppet+ /etc/puppet` ends up calling rename(2).


Rich
> 
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I just use capistrano to do atomic updates of my manifest directory.
For those who haven''t used capistrano, it checks out a new copy to
<dir>/releases/<timestamp>, symlinks it to <dir>/current
(which is
where Puppet is configured to look) and then restarts the unicorns.

On 4 January 2011 13:35, Richard Crowley <r@rcrowley.org>
wrote:
> So, here''s my question: if you are currently using the "svn
update"
> approach to manage /etc/puppet on the puppetmaster, have you taken
> conscious steps to help avoid a race condition? If so, what are they?
> And if not, why not?
>
> I do something like this to deploy all web apps, the Puppet master
included:
> SHA=$(git rev-parse HEAD)
> git archive --prefix=puppet-$SHA/ $SHA | gzip >puppet-$SHA.tar.gz
> scp puppet-$SHA.tar.gz puppet:/etc
> ssh puppet "
> tar xf /etc/puppet-$SHA.tar.gz -C /etc
> ln -s /etc/puppet-$SHA /etc/puppet+
> mv -T /etc/puppet+ /etc/puppet
> #find /etc -mindepth 1 -maxdepth 1 \
> # -name puppet-\* -and -not -name puppet-$SHA | xargs rm -rf
> "
> The `mv -T /etc/puppet+ /etc/puppet` ends up calling rename(2).
> Rich
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, 03 Jan 2011, James Ralston wrote:
> So, here''s my question: if you are currently using the "svn
update"
> approach to manage /etc/puppet on the puppetmaster, have you taken
> conscious steps to help avoid a race condition? If so, what are they?
> And if not, why not?
I made a conscious decision not to worry about this.  I don''t expect
any
consequences worse than compile errors (when some but not all of a set
of mutually-dependent files are updated) or the use of an old version of
a file.

If it proves to be a problem in practice, then I''ll probably
use a symlink to switch atomically between two working copies.

--apb (Alan Barrett)

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hey James,

if you could switch to Git instead of SVN, I have a working script
which will automate this process for you, maybe you could also
adopt it to use SVN as VCS backend.

http://sts.ono.at/blog/2010/12/22/synchronize-puppet-with-git/

Regards, Stefan.

On Jan 3, 2011, at 22:34 , James Ralston wrote:
> In multiple places in the Puppet wiki, I see the advice to store the
> puppermaster configuration in a VCS (specifically, a Subversion
> repository), and then checkout that Subversion repository into /etc/
> puppet. That way, the puppermaster''s configuration can be updated
by
> simply running (on the puppetmaster):
> 
> $ cd /etc/puppet
> $ svn update
> 
> But this approach raises an interesting question: the process of
> updating a checked-out copy of a Subversion repository is NOT atomic.
> If the puppetmaster were to read files in /etc/puppet while a lengthy
> "svn update" operation was updating /etc/puppet to the latest
revision
> (e.g., from revision 6 to revision 7), it is possible that the
> puppetmaster would see a mix of files from revisions 6 and 7. That
> could cause problems.
> 
> So, here''s my question: if you are currently using the "svn
update"
> approach to manage /etc/puppet on the puppetmaster, have you taken
> conscious steps to help avoid a race condition? If so, what are they?
> And if not, why not?
> 
> (Note that you don''t have to worry about individual files
containing a
> mix of revision 6 versus revision 7 content, because "svn update"
> modifies files by first creating a new temporary file, then using the
> rename(2) system call (which IS atomic) to move the new file into
> place...)
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
> 
--
Stefan Schlesinger ////////////////////////////////////////// ///////
sts@ono.at                                            +43.676.4911123

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 4, 2011 5:22 AM, "Alan Barrett" <apb@cequrux.com>
wrote:
> On Mon, 03 Jan 2011, James Ralston wrote:
> > So, here''s my question: if you are currently using the
"svn update"
> > approach to manage /etc/puppet on the puppetmaster, have you taken
> > conscious steps to help avoid a race condition? If so, what are they?
> > And if not, why not?
>
> I made a conscious decision not to worry about this.  I don''t
expect any
> consequences worse than compile errors (when some but not all of a set
> of mutually-dependent files are updated) or the use of an old version of
> a file.
We made this same call: the window was small and the risk that it broke
anything for more than the time between two puppet runs was so low it
wasn''t
worth engineering around until everything else on our network was perfect.
> If it proves to be a problem in practice, then I''ll probably
> use a symlink to switch atomically between two working copies.
That doesn''t solve the problem: if the puppet master is half way
through a
compilation run and you switch the repo under it, you will still see it use
mixed versions - because loading the catalog is not atomic either, and
neither is file serving.

You actually do need to shut down the puppet master (or have it natively
support the VCS like it presently does disk) for this to be truely atomic.

Regards,
    Daniel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 4, 2011, at 9:39 AM, Daniel Pittman wrote:
> On Jan 4, 2011 5:22 AM, "Alan Barrett" <apb@cequrux.com>
wrote:
> > On Mon, 03 Jan 2011, James Ralston wrote:
> > > So, here''s my question: if you are currently using the
"svn update"
> > > approach to manage /etc/puppet on the puppetmaster, have you
taken
> > > conscious steps to help avoid a race condition? If so, what are
they?
> > > And if not, why not?
> >
> > I made a conscious decision not to worry about this.  I don''t
expect any
> > consequences worse than compile errors (when some but not all of a set
> > of mutually-dependent files are updated) or the use of an old version
of
> > a file.
> 
> We made this same call: the window was small and the risk that it broke
anything for more than the time between two puppet runs was so low it
wasn''t worth engineering around until everything else on our network
was perfect.
> 
> > If it proves to be a problem in practice, then I''ll probably
> > use a symlink to switch atomically between two working copies.
> 
> That doesn''t solve the problem: if the puppet master is half way
through a compilation run and you switch the repo under it, you will still see
it use mixed versions - because loading the catalog is not atomic either, and
neither is file serving.
> 
> You actually do need to shut down the puppet master (or have it natively
support the VCS like it presently does disk) for this to be truely atomic.
> 
Even this isn''t enough.  You could be using a file served by File
that''s never than the catalog if you update after the catalog is
compiled, but before the files are served.  Again, we just ignore the problem
here.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 3, 1:34 pm, James Ralston <rals...@pobox.com>
wrote:
> So, here''s my question: if you are currently using the "svn
update"
> approach to manage /etc/puppet on the puppetmaster, have you taken
> conscious steps to help avoid a race condition?
A late vote for Ignore It. At puppet camp SF this came up in two
breakout sessions I was in. As I can recall two large sites had seen
resource/manifest version mismatches occur and ignored the race. Noone
in the room had actually had a serious issue because of this. The
resolution for everyone present was to just let the next run correct
the problem.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Jan 10, 2011 at 11:34 PM, donavan <donavan@desinc.net>
wrote:
> On Jan 3, 1:34 pm, James Ralston <rals...@pobox.com> wrote:
>> So, here''s my question: if you are currently using the
"svn update"
>> approach to manage /etc/puppet on the puppetmaster, have you taken
>> conscious steps to help avoid a race condition?
>
> A late vote for Ignore It. At puppet camp SF this came up in two
> breakout sessions I was in. As I can recall two large sites had seen
> resource/manifest version mismatches occur and ignored the race. Noone
> in the room had actually had a serious issue because of this. The
> resolution for everyone present was to just let the next run correct
> the problem.
Another approach is to have a Big Red Button that controls whether or
not clients update or not. This can be as simple as an URL that you
check before puppet runs, and if the page exists, the clients can
update.

Then your release process becomes:

* Disable all updates
* Push changes to production
* Wait x minutes to ensure atomic updates
* Enable all updates

This also has the advantage of giving you a single location that you
can use to disable all updates if you''ve pushed an undesirable change
to production that wasn''t caught in testing.


>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 4, 12:39 pm, Daniel Pittman <dan...@rimspace.net>
wrote:
> That doesn''t solve the problem: if the puppet master is half way
through a
> compilation run and you switch the repo under it, you will still see it use
> mixed versions - because loading the catalog is not atomic either, and
> neither is file serving.
>
> You actually do need to shut down the puppet master (or have it natively
> support the VCS like it presently does disk) for this to be truely atomic.
I''m fairly new to Puppet, and had been under the impression (based on
the way VCS gets referenced in several docs) that it was natively
supported.

Since it appears that isn''t the case, would it make sense to have the
puppet client which runs on the puppet master be responsible for doing
the svn update on the manifest directory, and have it do a shutdown on
puppetmaster right before doing the update?

I''d be surprised if someone hadn''t already at least tried
this, if not
implemented it in production somewhere.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jan 4, 12:39 pm, Daniel Pittman <dan...@rimspace.net>
wrote:
> That doesn''t solve the problem: if the puppet master is half way
through a
> compilation run and you switch the repo under it, you will still see it use
> mixed versions - because loading the catalog is not atomic either, and
> neither is file serving.
>
> You actually do need to shut down the puppet master (or have it natively
> support the VCS like it presently does disk) for this to be truely atomic.
Based on the way subversion and git get mentioned in much of the
introductory documentation for puppet, I''d assumed they were both
natively supported.  I guess not.

Given that''s the case, wouldn''t it make sense to have the
puppet
client that runs on the puppetmaster server be responsible for
regularly updating the manifest directory from the VCS, and have that
update process stop puppetmaster first, and restart it after?

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Jan 20, 2011 at 07:51, Iconoplasty <iconoplasty@gmail.com> wrote:
> On Jan 4, 12:39 pm, Daniel Pittman <dan...@rimspace.net> wrote:
>
>> That doesn''t solve the problem: if the puppet master is half
way through a
>> compilation run and you switch the repo under it, you will still see it
use
>> mixed versions - because loading the catalog is not atomic either, and
>> neither is file serving.
>>
>> You actually do need to shut down the puppet master (or have it
natively
>> support the VCS like it presently does disk) for this to be truely
atomic.
>
> Based on the way subversion and git get mentioned in much of the
> introductory documentation for puppet, I''d assumed they were both
> natively supported.  I guess not.
Hey there.  So, when I wrote "natively supported" I meant that the
puppet master would need to know how to talk directly to the version
control system rather than the file system, such as talking to an SVN
repository over HTTP to fetch the catalog during compilation.

It doesn''t, but that doesn''t make those tools in any way
useless or
anything like that.  The problem is, as a few people here have said,
not really an issue in the real world, even if there is a theoretical
race and all.  (...and, I should add, it is entirely non-trivial to
define the "correct" behaviour in the face of concurrent updates to a
version control repository as well.)
> Given that''s the case, wouldn''t it make sense to have the
puppet
> client that runs on the puppetmaster server be responsible for
> regularly updating the manifest directory from the VCS, and have that
> update process stop puppetmaster first, and restart it after?
Probably not, especially because that will be more disruptive to the
clients you have out in the wild than the current issue is.

Now, if you really wanted to engineer around it a better approach
might be to deploy mcollective, deploy the puppetcommander extension,
and use that exclusively for scheduling puppet runs on clients.

Then you have a central place (puppetcommander) running on the master
able to control when agents are going to check in or not.  It could be
extended to integrate a "VCS update" phase between triggering puppet
agent runs.

That would let you get rid of the race, and the risk of interrupting a
running puppet agent, by pulling the knowledge about which machines
are doing what into a central place, and that is the *only* way
without substantial changes in the master that would, in effect,
implement the same thing.

However, getting mcollective running, deploying puppetcommander, and
all the stack that they depend on is a non-trivial undertaking.  You
can probably imagine why it isn''t our first port of call for new
users. :)

Regards,
    Daniel
-- 
⎋ Puppet Labs Developer – http://puppetlabs.com
✉ Daniel Pittman <daniel@puppetlabs.com>
✆ Contact me via gtalk, email, or phone: +1 (877) 575-9775
♲ Made with 100 percent post-consumer electrons

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.