Puppet users - Feb 2007 - Recursive home-dir does more than only files in repository

Hi all,

Today I came across something that I consider a bug. Would like to hear your
opinions.

Facts:
- Using 0.22.1 from Debian Unstable on a Debian Testing machine
- Recipes work on other machines

The recipe in question is:

        # Keep home-dirs in sync
        file { "/home/tim":
                recurse => true,
                source => "puppet://puppet/home/tim",
                owner => "tim",
                group => "cidev",
                backup => false,
                require => User["tim"]
        }

It keeps some files in my home-dir in sync on all machines. Mainly stuff
like .bash_profile, .ssh, etc. Now yesterday I built JBoss in my home-dir,
and since then, puppet takes a really really long time to finish:

Feb 13 13:30:45 management puppetd[25958]: Finished configuration run in
12.67 seconds
Feb 13 13:51:51 management puppetd[25958]: Finished configuration run in
12.63 seconds
Feb 13 14:08:05 management puppetd[25958]: Finished configuration run in
13.16 seconds
Feb 13 17:55:45 management puppetd[25958]: Finished configuration run in
12772.65 seconds
Feb 14 03:34:36 management puppetd[25958]: Finished configuration run in
29997.93 seconds

I tried disabling the owner and group directive (just to test if that''s
the
thing), but that didn''t help. Anyone know what puppet is doing with the
files in my home-dir? And why?

I would expect puppet only to concern itself with the files that are
actually in the repository on the puppetmaster, and simply ignore the
rest... Should I file this as a bug?

-- 
Gegroet,
Tim


_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users

Tim Stoop a écrit :
> Hi all,
> 
> Today I came across something that I consider a bug. Would like to hear 
> your opinions.
> 
> Facts:
> - Using 0.22.1 from Debian Unstable on a Debian Testing machine
> - Recipes work on other machines
> 
> The recipe in question is:
> 
>         # Keep home-dirs in sync
>         file { "/home/tim":
>                 recurse => true,
>                 source => "puppet://puppet/home/tim",
>                 owner => "tim",
>                 group => "cidev",
>                 backup => false,
>                 require => User["tim"]
>         }

  i think puppet must look at all the files to see if it need to sync them with
your source. So it goes to all the files
and so it must be quite a lots of files in jboss dir so it makes it slow.
Perhaps there should be a reverse way of
thinking looking at the source list of file and see if they are on the
destination and not the reverse (only thinking
out loud). :)


-- 
Cordialement,
Ghislain



_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users

On Feb 14, 2007, at 7:02 AM, Tim Stoop wrote:
> [...]
> I tried disabling the owner and group directive (just to test if  
> that''s the thing), but that didn''t help. Anyone know what
puppet is
> doing with the files in my home-dir? And why?
Puppet''s file recursion is not what I would consider the smartest  
thing in the world at the moment, and it''s difficult to specify  
subsets of the functionality.  For instance, if you wanted to do that  
recursive copy plus make sure ~tim was owned by time, then your code  
would look the same -- how can Puppet know when you want to manage  
the whole directory vs. just the files with remote sources?

As a result, Puppet currently manages greedily.
> I would expect puppet only to concern itself with the files that  
> are actually in the repository on the puppetmaster, and simply  
> ignore the rest... Should I file this as a bug?
I''m perfectly fine changing it, but I don''t know what to
change it
to.  Can you think of a better way to do it?  How would you prefer to  
differentiate between those cases where you want to manage the whole  
directory vs. the cases where you just want to manage files that have  
remote sources.

  --
  The death rate on Earth is: .... (computing) .... One per person.
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On 2/14/07, Luke Kanies <luke@madstop.com> wrote:
>
> As a result, Puppet currently manages greedily.


Ok, I understand that.


I''m perfectly fine changing it, but I don''t know what to
change it
> to.  Can you think of a better way to do it?  How would you prefer to
> differentiate between those cases where you want to manage the whole
> directory vs. the cases where you just want to manage files that have
> remote sources.
>
Hm... Well, seeing on how I implement it at this time, I see two options
(not thought this through too much, so there might be better):

1) Have a modifier that says "manage the whole directory" or
"only manage
stuff actually put there by puppet".

2) Get the ownership out of File and make another type for it (bad, bad, i
know).

3) Thought of a third while writing, a combination of the two, allow for
another type to control ownership/modes if they aren''t explicitly
mentioned
in the original File.

At the moment, I try to set ownership and modes as best as possible on the
puppetmaster, but sometimes I need to set some modes manually. This results
in some stuff being modified on each machine every puppet run:

- file { "/home/tim" } sees that the .ssh subdir is 700 on disk and
740 on
the server, corrects it
- file { "/home/tim/.ssh" } sets the permissions back to 700.

The 740 on the server is needed because otherwise I wouldn''t be able to
check-in the files of another user into version control (darcs, in our
case).

Using some kind of NFS solution isn''t practical in this case, because
each
home-dir needs to contain different items on each machine.

-- 
Gegroet,
Tim


_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users

On Feb 14, 2007, at 10:18 AM, Tim Stoop wrote:
> Hm... Well, seeing on how I implement it at this time, I see two  
> options (not thought this through too much, so there might be better):
>
> 1) Have a modifier that says "manage the whole directory" or
"only
> manage stuff actually put there by puppet".
Maybe this should just be further valid values for ''recurse''. 
Right
now, there are three types of recursion:  Source-based, local  
recursion, and link recursion (for recursively creating symlink  
trees).  We could list those as valid values for ''ensure'', and
have
"true" be equivalent to all three.

Note that right now recursion can be limited to a certain depth  
(e.g., recurse => 2), and I don''t know how that would work with
these.
> 2) Get the ownership out of File and make another type for it (bad,  
> bad, i know).
Bzzzzt. :)
> 3) Thought of a third while writing, a combination of the two,  
> allow for another type to control ownership/modes if they aren''t  
> explicitly mentioned in the original File.
I think this is generally a bad idea.  The whole goal is encapsulation.


  --
  If a `religion'' is defined to be a system of ideas that contains
  unprovable statements, then Godel taught us that mathematics is not
  only a religion, it is the only religion that can prove itself to be
  one. 					-- John Barrow
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

> > 1) Have a modifier that says "manage the whole directory" or
"only
> > manage stuff actually put there by puppet".
> 
> Maybe this should just be further valid values for
''recurse''.  Right
> now, there are three types of recursion:  Source-based, local  
> recursion, and link recursion (for recursively creating symlink  
> trees).  We could list those as valid values for
''ensure'', and have
> "true" be equivalent to all three.
It seems like matching some of rsync''s symantics might be a useful
option here.  On the plus side, most people are used to them so it
could have better adoption on the user side - but that may also cause
some confusion(match a directory in the source, or append owner/mode
from the File instance?).  I''m also not sure if all of the options
work.  But it''s a possbility to base off of.

--mac

On Feb 14, 2007, at 11:22 AM, Chris McEniry wrote:
>
> It seems like matching some of rsync''s symantics might be a useful
> option here.  On the plus side, most people are used to them so it
> could have better adoption on the user side - but that may also cause
> some confusion(match a directory in the source, or append owner/mode
> from the File instance?).  I''m also not sure if all of the options
> work.  But it''s a possbility to base off of.
What semantics do you mean?

I always seem to get confused when I use rsync, such that I have to  
add ''/.'' on the end of every directory so I know
what''s going to happen.

I''m all for finding more reasonable semantics here; I just
don''t know
what to change them to.

  --
  I think that''s how Chicago got started.  A bunch of people in New
York
  said, ''Gee, I''m enjoying the crime and the poverty, but it
just isn''t
  cold enough. Let''s go west.''         --Richard Jeni
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On 2/14/07, Tim Stoop <tim.stoop@gmail.com> wrote:
> Hm... Well, seeing on how I implement it at this time, I see two options
> (not thought this through too much, so there might be better):
[zap]

I''ve been thinking about this for a little while now, and I figured,
what we really have are the following cases:

a) You want to manage a file.
b) You want to manage a directory, not recursively (for example, set
the correct owner or mode or something)
c) You want to manage a whole directory, recursively (for example, set
the correct owner for the whole directory including all underlying
files)
d) You want to duplicate a structure (directory plus files and maybe
subdirs), leaving the surroundings intact, but maybe change the
owner/group/mode of the files you''re duplicating.

While thinking, I came to the conclusion that although d looks a lot
like a,b and c, it isn''t the same. You can see it by the wording I
used. When reading the above four cases over and over, my (warped?)
logic sense tells me:

- I need an "item" management (a + b + c)
- I need a "group-of-items" management (d)

So maybe it''s possible to write a function the''s more like a
macro
doing something like:

for (each file in specified directory on the server)
  create (file { $name: etc. })

So a meta type that creates the actual desired types based on the
files on the server? Would that logic break some other logic I''m
missing? Are there more cases that should be considered? Love to hear
from you!

-- 
Gegroet,
Tim

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 17 February 2007 00:43, Tim Stoop wrote:
> On 2/14/07, Tim Stoop <tim.stoop@gmail.com> wrote:
> > Hm... Well, seeing on how I implement it at this time, I see two
options
> > (not thought this through too much, so there might be better):
>
> [zap]
>
> I''ve been thinking about this for a little while now, and I
figured,
> what we really have are the following cases:
>
> a) You want to manage a file.
> b) You want to manage a directory, not recursively (for example, set
> the correct owner or mode or something)
> c) You want to manage a whole directory, recursively (for example, set
> the correct owner for the whole directory including all underlying
> files)
> d) You want to duplicate a structure (directory plus files and maybe
> subdirs), leaving the surroundings intact, but maybe change the
> owner/group/mode of the files you''re duplicating.
>
> While thinking, I came to the conclusion that although d looks a lot
> like a,b and c, it isn''t the same. You can see it by the wording I
> used. When reading the above four cases over and over, my (warped?)
> logic sense tells me:
>
> - I need an "item" management (a + b + c)
> - I need a "group-of-items" management (d)
>
> So maybe it''s possible to write a function the''s more
like a macro
> doing something like:
>
> for (each file in specified directory on the server)
>   create (file { $name: etc. })
>
> So a meta type that creates the actual desired types based on the
> files on the server? Would that logic break some other logic I''m
> missing? Are there more cases that should be considered? Love to hear
> from you!
Only a short question: is the difference between c and d not "purge =>
true"
and "purge => false" respectively? See 
http://reductivelabs.com/projects/puppet/documentation/reference/typedocs.html#file
->
purge


Regards, David
- -- 
- - hallo... wie gehts heute?
- - *hust* gut *rotz* *keuch*
- - gott sei dank kommunizieren wir über ein septisches medium ;)
 -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFF2XhB/Pp1N6Uzh0URAtFaAJ9acHssC5GtFMI/shWpQjS44O6ANACfdZO+
1Qz4lW6puNmBIlJc2nRCZOI=AfNw
-----END PGP SIGNATURE-----

On Feb 19, 2007, at 4:13 AM, David Schmitt wrote:
> Only a short question: is the difference between c and d not "purge  
> => true"
> and "purge => false" respectively? See
> http://reductivelabs.com/projects/puppet/documentation/reference/ 
> typedocs.html#file ->
> purge
Purge specifically deletes any unmanaged files, so I don''t think so.

  --
  Nothing is impossible for the man who doesn''t have to do it himself.
  -- A. H. Weiler
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On Feb 16, 2007, at 5:43 PM, Tim Stoop wrote:
> I''ve been thinking about this for a little while now, and I
figured,
> what we really have are the following cases:
>
> a) You want to manage a file.
> b) You want to manage a directory, not recursively (for example, set
> the correct owner or mode or something)
> c) You want to manage a whole directory, recursively (for example, set
> the correct owner for the whole directory including all underlying
> files)
> d) You want to duplicate a structure (directory plus files and maybe
> subdirs), leaving the surroundings intact, but maybe change the
> owner/group/mode of the files you''re duplicating.
e) You want to recursively link one structure to another; i.e.,  
create a link in dirA for every file in dirB.
> While thinking, I came to the conclusion that although d looks a lot
> like a,b and c, it isn''t the same. You can see it by the wording I
> used. When reading the above four cases over and over, my (warped?)
> logic sense tells me:
>
> - I need an "item" management (a + b + c)
> - I need a "group-of-items" management (d)
I''ve been talking with a local developer/friend who thinks that  
recursive file management needs to use a single fileset idea, rather  
than the way it works now.  I frankly don''t know how to do that.

I think file management is one of the most complicated and  
unmaintainable parts of Puppet right now, and I''d love a better  
solution.  I''m completely open to recommendations, but I''m not
confident of my own ability to make things better.  Just note that  
there are always going to be conflicts between these different types  
of management, so any solution will need to handle these conflicts.
> So maybe it''s possible to write a function the''s more
like a macro
> doing something like:
>
> for (each file in specified directory on the server)
>   create (file { $name: etc. })
Hmm.  This is effectively what Puppet does internally, it just does  
it on the client instead of the server.  I''m not sure you can  
effectively do it on the server, because some file copies are local,  
and link recursion is always local.

I agree with you that these are different use cases, but I don''t know  
how to differentiate between the types of file management, really.

  --
  The Number 1 Sign You Have Nothing to Do at Work...
     The 4th Division of Paperclips has overrun the Pushpin Infantry
     and General White-Out has called for a new skirmish.
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

Thanks for taking the time to look at this.

On 2/19/07, Luke Kanies <luke@madstop.com> wrote:
> e) You want to recursively link one structure to another; i.e.,
> create a link in dirA for every file in dirB.
That''s kinda like c, but true, links shouldn''t be forgotten.
> I''ve been talking with a local developer/friend who thinks that
> recursive file management needs to use a single fileset idea, rather
> than the way it works now.  I frankly don''t know how to do that.
The more I think about it (but I''m hardly a developer) the more I
think that Puppet is trying to stuff to much functionality into one
method. If you''re trying to go for "making sure the expected
behaviour
is the actual behaviour" and "see in one glance what we want",
I''d say
it''s time to split things up a bit. Rather the opposite as what your
friend was saying, but from a HCI point of view, it''s more logical...
Isn''t it?

If it were up to me (it isn''t, but I''m only trying to help),
I''d split it up in:

file {}
  Manages a local file resource, can be a template on the server or a
server source. Can do links and permissions.

directory {}
  Manages a local directory, always recursively. Can not be a
template, maybe could be a server source but that should be
discouraged. Can do links and permissions.

remotedirectory {}
  Manages a local directory by filling it with files from an external
source ("puppet://", "file://", NFS, whatever). Has the
"purge"
argument and if that''s false, it only manages the files that are
actually in the external source and leaves all others alone. Can not
do links, can do permissions. Always recursively.
> I think file management is one of the most complicated and
> unmaintainable parts of Puppet right now, and I''d love a better
> solution.  I''m completely open to recommendations, but
I''m not
> confident of my own ability to make things better.  Just note that
> there are always going to be conflicts between these different types
> of management, so any solution will need to handle these conflicts.
You can leave the file {} working the way it works now and gradually
deprecate the directory functions of it. Announce like "1.0 will no
longer support that type of usage". Encourage people to use the
directory {} and remotedirectory {} stuff.

Now that I think about it, maybe remotedirectory should be merged with
directory {}, so it works more like file {}. Yeah, that''s more
obvious.

[ rewriting the stuff to file { "onefile": } ]
> Hmm.  This is effectively what Puppet does internally, it just does
> it on the client instead of the server.  I''m not sure you can
> effectively do it on the server, because some file copies are local,
> and link recursion is always local.
Can''t a client get a listing of the files in the local or remote
repository? The client could break it up into the actual files (and
directories) in the repository, then. If not, how does a client know
now which files it should fetch?

Just my 2 euro-cents. Hope it''s useful :|

-- 
Gegroet,
Tim

A simple workaround to this problem until a proper solution can be
found is to tar up the files you want pushed to home dirs then use
puppet to sync that tar file and untar it into the home dir whenever
it changes refreshed.

On Feb 19, 2007, at 3:41 PM, Tim Stoop wrote:
> Thanks for taking the time to look at this.
Sorry to take so long to respond, but this topic is far more  
complicated than I would like it to b.
> The more I think about it (but I''m hardly a developer) the more I
> think that Puppet is trying to stuff to much functionality into one
> method. If you''re trying to go for "making sure the expected
behaviour
> is the actual behaviour" and "see in one glance what we
want", I''d say
> it''s time to split things up a bit. Rather the opposite as what
your
> friend was saying, but from a HCI point of view, it''s more
logical...
> Isn''t it?
I''d unfortunately have to say that "logic" and "file
copying" seem to
rarely intersect.  There''s a lot of age-old practice around this, and  
there''s a lot of variety.  Advice on best practice from a wider range  
of users would be greatly appreciated.
> If it were up to me (it isn''t, but I''m only trying to
help), I''d
> split it up in:
>
> file {}
>   Manages a local file resource, can be a template on the server or a
> server source. Can do links and permissions.
>
> directory {}
>   Manages a local directory, always recursively. Can not be a
> template, maybe could be a server source but that should be
> discouraged. Can do links and permissions.
>
> remotedirectory {}
>   Manages a local directory by filling it with files from an external
> source ("puppet://", "file://", NFS, whatever). Has the
"purge"
> argument and if that''s false, it only manages the files that are
> actually in the external source and leaves all others alone. Can not
> do links, can do permissions. Always recursively.
This reminds me of what I did in ISconf; I had the equivalent of the  
''remotedirectory'' type called ''fileset'', and
all it could do was copy
files and directories.  I don''t remember how I managed file  
permissions, but I know filesets were very simple and just used rsync  
for transport.

The concern I have about this is that, when talking about  
relationships, you suddenly have to know how a file gets to the  
machine, whether you''re generating it using a template or copying it  
down using a fileset.  My original implementation plan actually  
called for separate file, link, and directory types, but when I went  
to implement it there was too little between them for it to make sense.

The fundamental problem that has to be solved here is how you handle  
cross-over.  What would the following code do?

   file { "/home/tim": user => tim, recurse => true }
   fileset { "/home/tim/.ssh": source => "..." }

If you merge your ''directory'' and
''remotedirectory'' types, then
you''ll just end up with that type being used every time instead of  
''file'', because it''ll be guaranteed to be a superset
of the ''file''
type, and we''ll be back to square one.

The only split that makes sense to me is to separate remote and local  
recursion, because they are pretty different, but they will always  
have significant functional cross-over (e.g., you''ll want to specify  
owner/group/mode on each), and there''ll always be conflicts between  
them which we''ll have to handle smoothly.
> Can''t a client get a listing of the files in the local or remote
> repository? The client could break it up into the actual files (and
> directories) in the repository, then. If not, how does a client know
> now which files it should fetch?
Kind of -- a client can only get a local listing mid-transaction,  
because the files are often being created by a dependency (e.g.,  
create a trac instance, and make sure it''s owned by www-data).

Remote listings could always be done before the transaction starts,  
though.

  --
  Once...in the wilds of Afghanistan, I lost my corkscrew, and we were
  forced to live on nothing but food and water for days.  -- W. C.  
Fields
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Monday 05 March 2007 19:27, Luke Kanies wrote:
> The fundamental problem that has to be solved here is how you handle
> cross-over.  What would the following code do?
>
>    file { "/home/tim": user => tim, recurse => true }
>    fileset { "/home/tim/.ssh": source => "..." }
One - albeit a little complex - approach might be to view a recursive 
remotefile{} as a "macro" that expands server-side to multiple file{} 
definitions which can be overridden/merged by definitions with a more 
specific "basepath" (i.e. the path that is actually specified in the 
manifest).



Regards, David
- -- 
- - hallo... wie gehts heute?
- - *hust* gut *rotz* *keuch*
- - gott sei dank kommunizieren wir über ein septisches medium ;)
 -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFF8FgX/Pp1N6Uzh0URAs/UAKCQk1yT/FgnKslZVys2akowReSUVwCggQHa
++iWaakHFlZZR6qgdiqp1n8=IQrm
-----END PGP SIGNATURE-----

On Mar 8, 2007, at 12:38 PM, David Schmitt wrote:
>
> On Monday 05 March 2007 19:27, Luke Kanies wrote:
>> The fundamental problem that has to be solved here is how you handle
>> cross-over.  What would the following code do?
>>
>>    file { "/home/tim": user => tim, recurse => true }
>>    fileset { "/home/tim/.ssh": source => "..." }
>
> One - albeit a little complex - approach might be to view a recursive
> remotefile{} as a "macro" that expands server-side to multiple
file{}
> definitions which can be overridden/merged by definitions with a more
> specific "basepath" (i.e. the path that is actually specified in
the
> manifest).
This changes the behaviour of filesets -- Puppet rebuilds the remote  
file list on every run, at the moment, but this would only rebuild  
the file list on recompiles.  This might actually be a good thing,  
but it''s definitely different.

A similar difference arises between using templates and using file  
sources to set file contents.

  --
  We either are networking, or we areNT networking...
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com