Puppet users - Apr 2009 - copying files back to puppet master?

Hi all,

Puppet is used mainly for managing resources on the servers it manages,
and as such files and software may be distributed from the puppet master
to the clients as part of the management or configuration process.

That''s fine.

The puppet installation where I work has evolved starting from a site
with no puppet to a site where most of the servers are puppetised with
the intention being on doing a complete deploy exclusively using puppet.

However, as not all aspects of all resources on the server are managed
by puppet there is room for the administators to make mistakes. As such
we often want to version control configuration information on the server
which protects against mistakes but also allows us to see when changes
were made, either automatically (by puppet) or manually.

If I''m not mistaken it''s not possible to use the file resource
for
copying files back to the puppet master.  Is that correct?

This functionality would be most useful, especially if we combine
this with some revision control system. If we did something like
this the files to be returned back to the puppet master (assuming
they change) would need to be copied back to a location like
/puppet_master_path/site_and_hostname/host_path/filename. The advantage of
doing this in puppet is that you don''t need to ensure that extra
firewall
holes are opened, and that in principal at least the same file resource
could be used. It might be necesssary to additionally qualify/authorise
"back to master" transfers for security reasons.

A feature like this is much easier to manage than doing this outside of
puppet. It is also much easier to compare the configuration of a number of
different "similar" servers if the files are located on the same
server,
especially as the number of managed servers increases.

So is this possible at the moment in puppet, and if not is it a feature
that might be interesting to others?  How are others tackling problems
like this?

Simon

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Tue, 2009-04-21 at 10:50 +0200, Simon J Mudd wrote:
> Hi all,
> 
> Puppet is used mainly for managing resources on the servers it manages,
> and as such files and software may be distributed from the puppet master
> to the clients as part of the management or configuration process.
> 
> That''s fine.
> 
> The puppet installation where I work has evolved starting from a site
> with no puppet to a site where most of the servers are puppetised with
> the intention being on doing a complete deploy exclusively using puppet.
> 
> However, as not all aspects of all resources on the server are managed
> by puppet there is room for the administators to make mistakes. As such
> we often want to version control configuration information on the server
> which protects against mistakes but also allows us to see when changes
> were made, either automatically (by puppet) or manually.
> 
> If I''m not mistaken it''s not possible to use the file
resource for
> copying files back to the puppet master.  Is that correct?
Yes it is correct.
> This functionality would be most useful, especially if we combine
> this with some revision control system. If we did something like
> this the files to be returned back to the puppet master (assuming
> they change) would need to be copied back to a location like
> /puppet_master_path/site_and_hostname/host_path/filename. The advantage of
> doing this in puppet is that you don''t need to ensure that extra
firewall
> holes are opened, and that in principal at least the same file resource
> could be used. It might be necesssary to additionally qualify/authorise
> "back to master" transfers for security reasons.
> 
> A feature like this is much easier to manage than doing this outside of
> puppet. It is also much easier to compare the configuration of a number of
> different "similar" servers if the files are located on the same
server,
> especially as the number of managed servers increases.
> 
> So is this possible at the moment in puppet, and if not is it a feature
> that might be interesting to others?  How are others tackling problems
> like this?
I think you should have a look to the filebucket[1]. It might not be
exactly what you are asking for, but it might still help you.

The filebucket is a way to backup files that are changed by puppet to
the master (or locally). There is also an accompanying application
(called filebucket) that can be used to get back the various
"bucketed"
files.

[1]: http://reductivelabs.com/trac/puppet/wiki/TypeReference#id356
-- 
Brice Figureau
My Blog: http://www.masterzen.fr/


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

brice-puppet@daysofwonder.com (Brice Figureau) writes:

...
> > If I''m not mistaken it''s not possible to use the
file resource for
> > copying files back to the puppet master.  Is that correct?
> 
> Yes it is correct.
ok.

...
> I think you should have a look to the filebucket[1]. It might not be
> exactly what you are asking for, but it might still help you.
No, not really but thanks for the pointer. As I said my idea would be
to use this to "manage" local configuration files, but to manage them
"centrally".

In a previous job using cvs and cfengine I used this to allow us to
maintain local configuration files normally pushed to the central
server, but under configuration control could also be used to push out
the same files on to a different box for example to replace a failed
server.

To be fair I don''t expect puppet to do all of this, but it would be
nice if the current file: resource/protocol could potentially work in
both directions.  This would open up a lot of possibilities.

Simon


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Apr 21, 2009, at 4:25 PM, Simon J Mudd wrote:
>
> brice-puppet@daysofwonder.com (Brice Figureau) writes:
>
> ...
>
>>> If I''m not mistaken it''s not possible to use the
file resource for
>>> copying files back to the puppet master.  Is that correct?
>>
>> Yes it is correct.
>
> ok.
>
> ...
>
>> I think you should have a look to the filebucket[1]. It might not be
>> exactly what you are asking for, but it might still help you.
>
> No, not really but thanks for the pointer. As I said my idea would be
> to use this to "manage" local configuration files, but to manage
them
> "centrally".
>
> In a previous job using cvs and cfengine I used this to allow us to
> maintain local configuration files normally pushed to the central
> server, but under configuration control could also be used to push out
> the same files on to a different box for example to replace a failed
> server.
>
> To be fair I don''t expect puppet to do all of this, but it would
be
> nice if the current file: resource/protocol could potentially work in
> both directions.  This would open up a lot of possibilities.
This will at least be possible internally with the code in 0.25 (which  
looks like it''ll go rc1 this week, I think just one more ticket), but  
I don''t know how it would actually be useful for the client.

I''ve been thinking about what you''re looking for, though, and
I think
it would make more sense to directly integrate filebuckets into a  
version control system - the client would back modified files up to  
the server, the server would automagically check those files into a  
version control repository (into a branch named after the host, I  
assume), and then you could do whatever comparisons you wanted to your  
heart''s content.

I''ve actually got a basic proof of concept of at least replacing the  
filebucket store with git done[1].  It''s just the client-side pieces,  
you''d need to add the server-side pieces that created the branch and  
such, but I don''t think that would be a ton of work, and it''d
provide
everything you want while integrating nicely with how Puppet already  
backs files up to the server.

1 - http://gist.github.com/77811

-- 
When one admits that nothing is certain one must, I think, also admit
that some things are much more nearly certain than others.
     -- Bertrand Russell
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi Luke,

luke@madstop.com (Luke Kanies) writes:
> > ...
> >
> >> I think you should have a look to the filebucket[1]. It might not
be
> >> exactly what you are asking for, but it might still help you.
> >
> > No, not really but thanks for the pointer. As I said my idea would be
> > to use this to "manage" local configuration files, but to
manage them
> > "centrally".
> >
> > In a previous job using cvs and cfengine I used this to allow us to
> > maintain local configuration files normally pushed to the central
> > server, but under configuration control could also be used to push out
> > the same files on to a different box for example to replace a failed
> > server.
> >
> > To be fair I don''t expect puppet to do all of this, but it
would be
> > nice if the current file: resource/protocol could potentially work in
> > both directions.  This would open up a lot of possibilities.
> 
> This will at least be possible internally with the code in 0.25 (which  
> looks like it''ll go rc1 this week, I think just one more ticket),
but
> I don''t know how it would actually be useful for the client.
On the client it''s NOT useful, but that''s not the point.
Perhaps my
vision of a system like puppet is slightly different, but to manage a
whole group of servers, and not just think about a single specific
server. Hence my comment before about using a technique like this to
allow files perhaps not pushed out by puppet to be managed "centrally"
and if needed these older configurations could be later pushed back to
the same or perhaps a different server. I haven''t thought out all the
details but I can see this as an alternative way of "cloning" configs
from a specific server.
> I''ve been thinking about what you''re looking for, though,
and I think
> it would make more sense to directly integrate filebuckets into a  
> version control system - the client would back modified files up to  
> the server, the server would automagically check those files into a  
> version control repository (into a branch named after the host, I  
> assume), and then you could do whatever comparisons you wanted to your  
> heart''s content.
Yes, that''s basically the idea. It allows sysadmins to not be fearful
about changes they make as they would be tracked and thus easier to
revert should the need arise.

I can see this being useful for managing changes in:
- system and user cron jobs
- user ~/.[bash]profile or  ~/.bashrc
- user ~/.ssh/config type files
- ...

While puppet may not be pushing out these changes all of a sudden you
have these files under control. Potentially you could also add some 
sort of extra hooks (on the server) when these changes are collected
to do certain actions such as send out emails or whatever.
> I''ve actually got a basic proof of concept of at least replacing
the
> filebucket store with git done[1].  It''s just the client-side
pieces,
> you''d need to add the server-side pieces that created the branch
and
> such, but I don''t think that would be a ton of work, and
it''d provide
> everything you want while integrating nicely with how Puppet already  
> backs files up to the server.
> 
> 1 - http://gist.github.com/77811
I''ll have a look. Thanks for taking the time to considering what I was
asking.

Simon


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---