Tim Stoop
2007-Feb-14 13:02 UTC
Recursive home-dir does more than only files in repository
Hi all, Today I came across something that I consider a bug. Would like to hear your opinions. Facts: - Using 0.22.1 from Debian Unstable on a Debian Testing machine - Recipes work on other machines The recipe in question is: # Keep home-dirs in sync file { "/home/tim": recurse => true, source => "puppet://puppet/home/tim", owner => "tim", group => "cidev", backup => false, require => User["tim"] } It keeps some files in my home-dir in sync on all machines. Mainly stuff like .bash_profile, .ssh, etc. Now yesterday I built JBoss in my home-dir, and since then, puppet takes a really really long time to finish: Feb 13 13:30:45 management puppetd[25958]: Finished configuration run in 12.67 seconds Feb 13 13:51:51 management puppetd[25958]: Finished configuration run in 12.63 seconds Feb 13 14:08:05 management puppetd[25958]: Finished configuration run in 13.16 seconds Feb 13 17:55:45 management puppetd[25958]: Finished configuration run in 12772.65 seconds Feb 14 03:34:36 management puppetd[25958]: Finished configuration run in 29997.93 seconds I tried disabling the owner and group directive (just to test if that''s the thing), but that didn''t help. Anyone know what puppet is doing with the files in my home-dir? And why? I would expect puppet only to concern itself with the files that are actually in the repository on the puppetmaster, and simply ignore the rest... Should I file this as a bug? -- Gegroet, Tim _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
ADNET Ghislain
2007-Feb-14 13:06 UTC
Re: Recursive home-dir does more than only files in repository
Tim Stoop a écrit :> Hi all, > > Today I came across something that I consider a bug. Would like to hear > your opinions. > > Facts: > - Using 0.22.1 from Debian Unstable on a Debian Testing machine > - Recipes work on other machines > > The recipe in question is: > > # Keep home-dirs in sync > file { "/home/tim": > recurse => true, > source => "puppet://puppet/home/tim", > owner => "tim", > group => "cidev", > backup => false, > require => User["tim"] > }i think puppet must look at all the files to see if it need to sync them with your source. So it goes to all the files and so it must be quite a lots of files in jboss dir so it makes it slow. Perhaps there should be a reverse way of thinking looking at the source list of file and see if they are on the destination and not the reverse (only thinking out loud). :) -- Cordialement, Ghislain _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
Luke Kanies
2007-Feb-14 15:59 UTC
Re: Recursive home-dir does more than only files in repository
On Feb 14, 2007, at 7:02 AM, Tim Stoop wrote:> [...] > I tried disabling the owner and group directive (just to test if > that''s the thing), but that didn''t help. Anyone know what puppet is > doing with the files in my home-dir? And why?Puppet''s file recursion is not what I would consider the smartest thing in the world at the moment, and it''s difficult to specify subsets of the functionality. For instance, if you wanted to do that recursive copy plus make sure ~tim was owned by time, then your code would look the same -- how can Puppet know when you want to manage the whole directory vs. just the files with remote sources? As a result, Puppet currently manages greedily.> I would expect puppet only to concern itself with the files that > are actually in the repository on the puppetmaster, and simply > ignore the rest... Should I file this as a bug?I''m perfectly fine changing it, but I don''t know what to change it to. Can you think of a better way to do it? How would you prefer to differentiate between those cases where you want to manage the whole directory vs. the cases where you just want to manage files that have remote sources. -- The death rate on Earth is: .... (computing) .... One per person. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Tim Stoop
2007-Feb-14 16:18 UTC
Re: Recursive home-dir does more than only files in repository
On 2/14/07, Luke Kanies <luke@madstop.com> wrote:> > As a result, Puppet currently manages greedily.Ok, I understand that. I''m perfectly fine changing it, but I don''t know what to change it> to. Can you think of a better way to do it? How would you prefer to > differentiate between those cases where you want to manage the whole > directory vs. the cases where you just want to manage files that have > remote sources. >Hm... Well, seeing on how I implement it at this time, I see two options (not thought this through too much, so there might be better): 1) Have a modifier that says "manage the whole directory" or "only manage stuff actually put there by puppet". 2) Get the ownership out of File and make another type for it (bad, bad, i know). 3) Thought of a third while writing, a combination of the two, allow for another type to control ownership/modes if they aren''t explicitly mentioned in the original File. At the moment, I try to set ownership and modes as best as possible on the puppetmaster, but sometimes I need to set some modes manually. This results in some stuff being modified on each machine every puppet run: - file { "/home/tim" } sees that the .ssh subdir is 700 on disk and 740 on the server, corrects it - file { "/home/tim/.ssh" } sets the permissions back to 700. The 740 on the server is needed because otherwise I wouldn''t be able to check-in the files of another user into version control (darcs, in our case). Using some kind of NFS solution isn''t practical in this case, because each home-dir needs to contain different items on each machine. -- Gegroet, Tim _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
Luke Kanies
2007-Feb-14 16:52 UTC
Re: Recursive home-dir does more than only files in repository
On Feb 14, 2007, at 10:18 AM, Tim Stoop wrote:> Hm... Well, seeing on how I implement it at this time, I see two > options (not thought this through too much, so there might be better): > > 1) Have a modifier that says "manage the whole directory" or "only > manage stuff actually put there by puppet".Maybe this should just be further valid values for ''recurse''. Right now, there are three types of recursion: Source-based, local recursion, and link recursion (for recursively creating symlink trees). We could list those as valid values for ''ensure'', and have "true" be equivalent to all three. Note that right now recursion can be limited to a certain depth (e.g., recurse => 2), and I don''t know how that would work with these.> 2) Get the ownership out of File and make another type for it (bad, > bad, i know).Bzzzzt. :)> 3) Thought of a third while writing, a combination of the two, > allow for another type to control ownership/modes if they aren''t > explicitly mentioned in the original File.I think this is generally a bad idea. The whole goal is encapsulation. -- If a `religion'' is defined to be a system of ideas that contains unprovable statements, then Godel taught us that mathematics is not only a religion, it is the only religion that can prove itself to be one. -- John Barrow --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Chris McEniry
2007-Feb-14 17:22 UTC
Re: Recursive home-dir does more than only files in repository
> > 1) Have a modifier that says "manage the whole directory" or "only > > manage stuff actually put there by puppet". > > Maybe this should just be further valid values for ''recurse''. Right > now, there are three types of recursion: Source-based, local > recursion, and link recursion (for recursively creating symlink > trees). We could list those as valid values for ''ensure'', and have > "true" be equivalent to all three.It seems like matching some of rsync''s symantics might be a useful option here. On the plus side, most people are used to them so it could have better adoption on the user side - but that may also cause some confusion(match a directory in the source, or append owner/mode from the File instance?). I''m also not sure if all of the options work. But it''s a possbility to base off of. --mac
Luke Kanies
2007-Feb-16 03:24 UTC
Re: Recursive home-dir does more than only files in repository
On Feb 14, 2007, at 11:22 AM, Chris McEniry wrote:> > It seems like matching some of rsync''s symantics might be a useful > option here. On the plus side, most people are used to them so it > could have better adoption on the user side - but that may also cause > some confusion(match a directory in the source, or append owner/mode > from the File instance?). I''m also not sure if all of the options > work. But it''s a possbility to base off of.What semantics do you mean? I always seem to get confused when I use rsync, such that I have to add ''/.'' on the end of every directory so I know what''s going to happen. I''m all for finding more reasonable semantics here; I just don''t know what to change them to. -- I think that''s how Chicago got started. A bunch of people in New York said, ''Gee, I''m enjoying the crime and the poverty, but it just isn''t cold enough. Let''s go west.'' --Richard Jeni --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Tim Stoop
2007-Feb-16 23:43 UTC
Re: Recursive home-dir does more than only files in repository
On 2/14/07, Tim Stoop <tim.stoop@gmail.com> wrote:> Hm... Well, seeing on how I implement it at this time, I see two options > (not thought this through too much, so there might be better):[zap] I''ve been thinking about this for a little while now, and I figured, what we really have are the following cases: a) You want to manage a file. b) You want to manage a directory, not recursively (for example, set the correct owner or mode or something) c) You want to manage a whole directory, recursively (for example, set the correct owner for the whole directory including all underlying files) d) You want to duplicate a structure (directory plus files and maybe subdirs), leaving the surroundings intact, but maybe change the owner/group/mode of the files you''re duplicating. While thinking, I came to the conclusion that although d looks a lot like a,b and c, it isn''t the same. You can see it by the wording I used. When reading the above four cases over and over, my (warped?) logic sense tells me: - I need an "item" management (a + b + c) - I need a "group-of-items" management (d) So maybe it''s possible to write a function the''s more like a macro doing something like: for (each file in specified directory on the server) create (file { $name: etc. }) So a meta type that creates the actual desired types based on the files on the server? Would that logic break some other logic I''m missing? Are there more cases that should be considered? Love to hear from you! -- Gegroet, Tim
David Schmitt
2007-Feb-19 10:13 UTC
Re: Recursive home-dir does more than only files in repository
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday 17 February 2007 00:43, Tim Stoop wrote:> On 2/14/07, Tim Stoop <tim.stoop@gmail.com> wrote: > > Hm... Well, seeing on how I implement it at this time, I see two options > > (not thought this through too much, so there might be better): > > [zap] > > I''ve been thinking about this for a little while now, and I figured, > what we really have are the following cases: > > a) You want to manage a file. > b) You want to manage a directory, not recursively (for example, set > the correct owner or mode or something) > c) You want to manage a whole directory, recursively (for example, set > the correct owner for the whole directory including all underlying > files) > d) You want to duplicate a structure (directory plus files and maybe > subdirs), leaving the surroundings intact, but maybe change the > owner/group/mode of the files you''re duplicating. > > While thinking, I came to the conclusion that although d looks a lot > like a,b and c, it isn''t the same. You can see it by the wording I > used. When reading the above four cases over and over, my (warped?) > logic sense tells me: > > - I need an "item" management (a + b + c) > - I need a "group-of-items" management (d) > > So maybe it''s possible to write a function the''s more like a macro > doing something like: > > for (each file in specified directory on the server) > create (file { $name: etc. }) > > So a meta type that creates the actual desired types based on the > files on the server? Would that logic break some other logic I''m > missing? Are there more cases that should be considered? Love to hear > from you!Only a short question: is the difference between c and d not "purge => true" and "purge => false" respectively? See http://reductivelabs.com/projects/puppet/documentation/reference/typedocs.html#file -> purge Regards, David - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFF2XhB/Pp1N6Uzh0URAtFaAJ9acHssC5GtFMI/shWpQjS44O6ANACfdZO+ 1Qz4lW6puNmBIlJc2nRCZOI=AfNw -----END PGP SIGNATURE-----
Luke Kanies
2007-Feb-19 15:15 UTC
Re: Recursive home-dir does more than only files in repository
On Feb 19, 2007, at 4:13 AM, David Schmitt wrote:> Only a short question: is the difference between c and d not "purge > => true" > and "purge => false" respectively? See > http://reductivelabs.com/projects/puppet/documentation/reference/ > typedocs.html#file -> > purgePurge specifically deletes any unmanaged files, so I don''t think so. -- Nothing is impossible for the man who doesn''t have to do it himself. -- A. H. Weiler --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies
2007-Feb-19 17:43 UTC
Re: Recursive home-dir does more than only files in repository
On Feb 16, 2007, at 5:43 PM, Tim Stoop wrote:> I''ve been thinking about this for a little while now, and I figured, > what we really have are the following cases: > > a) You want to manage a file. > b) You want to manage a directory, not recursively (for example, set > the correct owner or mode or something) > c) You want to manage a whole directory, recursively (for example, set > the correct owner for the whole directory including all underlying > files) > d) You want to duplicate a structure (directory plus files and maybe > subdirs), leaving the surroundings intact, but maybe change the > owner/group/mode of the files you''re duplicating.e) You want to recursively link one structure to another; i.e., create a link in dirA for every file in dirB.> While thinking, I came to the conclusion that although d looks a lot > like a,b and c, it isn''t the same. You can see it by the wording I > used. When reading the above four cases over and over, my (warped?) > logic sense tells me: > > - I need an "item" management (a + b + c) > - I need a "group-of-items" management (d)I''ve been talking with a local developer/friend who thinks that recursive file management needs to use a single fileset idea, rather than the way it works now. I frankly don''t know how to do that. I think file management is one of the most complicated and unmaintainable parts of Puppet right now, and I''d love a better solution. I''m completely open to recommendations, but I''m not confident of my own ability to make things better. Just note that there are always going to be conflicts between these different types of management, so any solution will need to handle these conflicts.> So maybe it''s possible to write a function the''s more like a macro > doing something like: > > for (each file in specified directory on the server) > create (file { $name: etc. })Hmm. This is effectively what Puppet does internally, it just does it on the client instead of the server. I''m not sure you can effectively do it on the server, because some file copies are local, and link recursion is always local. I agree with you that these are different use cases, but I don''t know how to differentiate between the types of file management, really. -- The Number 1 Sign You Have Nothing to Do at Work... The 4th Division of Paperclips has overrun the Pushpin Infantry and General White-Out has called for a new skirmish. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Tim Stoop
2007-Feb-19 21:41 UTC
Re: Recursive home-dir does more than only files in repository
Thanks for taking the time to look at this. On 2/19/07, Luke Kanies <luke@madstop.com> wrote:> e) You want to recursively link one structure to another; i.e., > create a link in dirA for every file in dirB.That''s kinda like c, but true, links shouldn''t be forgotten.> I''ve been talking with a local developer/friend who thinks that > recursive file management needs to use a single fileset idea, rather > than the way it works now. I frankly don''t know how to do that.The more I think about it (but I''m hardly a developer) the more I think that Puppet is trying to stuff to much functionality into one method. If you''re trying to go for "making sure the expected behaviour is the actual behaviour" and "see in one glance what we want", I''d say it''s time to split things up a bit. Rather the opposite as what your friend was saying, but from a HCI point of view, it''s more logical... Isn''t it? If it were up to me (it isn''t, but I''m only trying to help), I''d split it up in: file {} Manages a local file resource, can be a template on the server or a server source. Can do links and permissions. directory {} Manages a local directory, always recursively. Can not be a template, maybe could be a server source but that should be discouraged. Can do links and permissions. remotedirectory {} Manages a local directory by filling it with files from an external source ("puppet://", "file://", NFS, whatever). Has the "purge" argument and if that''s false, it only manages the files that are actually in the external source and leaves all others alone. Can not do links, can do permissions. Always recursively.> I think file management is one of the most complicated and > unmaintainable parts of Puppet right now, and I''d love a better > solution. I''m completely open to recommendations, but I''m not > confident of my own ability to make things better. Just note that > there are always going to be conflicts between these different types > of management, so any solution will need to handle these conflicts.You can leave the file {} working the way it works now and gradually deprecate the directory functions of it. Announce like "1.0 will no longer support that type of usage". Encourage people to use the directory {} and remotedirectory {} stuff. Now that I think about it, maybe remotedirectory should be merged with directory {}, so it works more like file {}. Yeah, that''s more obvious. [ rewriting the stuff to file { "onefile": } ]> Hmm. This is effectively what Puppet does internally, it just does > it on the client instead of the server. I''m not sure you can > effectively do it on the server, because some file copies are local, > and link recursion is always local.Can''t a client get a listing of the files in the local or remote repository? The client could break it up into the actual files (and directories) in the repository, then. If not, how does a client know now which files it should fetch? Just my 2 euro-cents. Hope it''s useful :| -- Gegroet, Tim
A simple workaround to this problem until a proper solution can be found is to tar up the files you want pushed to home dirs then use puppet to sync that tar file and untar it into the home dir whenever it changes refreshed.
Luke Kanies
2007-Mar-05 18:27 UTC
Re: Recursive home-dir does more than only files in repository
On Feb 19, 2007, at 3:41 PM, Tim Stoop wrote:> Thanks for taking the time to look at this.Sorry to take so long to respond, but this topic is far more complicated than I would like it to b.> The more I think about it (but I''m hardly a developer) the more I > think that Puppet is trying to stuff to much functionality into one > method. If you''re trying to go for "making sure the expected behaviour > is the actual behaviour" and "see in one glance what we want", I''d say > it''s time to split things up a bit. Rather the opposite as what your > friend was saying, but from a HCI point of view, it''s more logical... > Isn''t it?I''d unfortunately have to say that "logic" and "file copying" seem to rarely intersect. There''s a lot of age-old practice around this, and there''s a lot of variety. Advice on best practice from a wider range of users would be greatly appreciated.> If it were up to me (it isn''t, but I''m only trying to help), I''d > split it up in: > > file {} > Manages a local file resource, can be a template on the server or a > server source. Can do links and permissions. > > directory {} > Manages a local directory, always recursively. Can not be a > template, maybe could be a server source but that should be > discouraged. Can do links and permissions. > > remotedirectory {} > Manages a local directory by filling it with files from an external > source ("puppet://", "file://", NFS, whatever). Has the "purge" > argument and if that''s false, it only manages the files that are > actually in the external source and leaves all others alone. Can not > do links, can do permissions. Always recursively.This reminds me of what I did in ISconf; I had the equivalent of the ''remotedirectory'' type called ''fileset'', and all it could do was copy files and directories. I don''t remember how I managed file permissions, but I know filesets were very simple and just used rsync for transport. The concern I have about this is that, when talking about relationships, you suddenly have to know how a file gets to the machine, whether you''re generating it using a template or copying it down using a fileset. My original implementation plan actually called for separate file, link, and directory types, but when I went to implement it there was too little between them for it to make sense. The fundamental problem that has to be solved here is how you handle cross-over. What would the following code do? file { "/home/tim": user => tim, recurse => true } fileset { "/home/tim/.ssh": source => "..." } If you merge your ''directory'' and ''remotedirectory'' types, then you''ll just end up with that type being used every time instead of ''file'', because it''ll be guaranteed to be a superset of the ''file'' type, and we''ll be back to square one. The only split that makes sense to me is to separate remote and local recursion, because they are pretty different, but they will always have significant functional cross-over (e.g., you''ll want to specify owner/group/mode on each), and there''ll always be conflicts between them which we''ll have to handle smoothly.> Can''t a client get a listing of the files in the local or remote > repository? The client could break it up into the actual files (and > directories) in the repository, then. If not, how does a client know > now which files it should fetch?Kind of -- a client can only get a local listing mid-transaction, because the files are often being created by a dependency (e.g., create a trac instance, and make sure it''s owned by www-data). Remote listings could always be done before the transaction starts, though. -- Once...in the wilds of Afghanistan, I lost my corkscrew, and we were forced to live on nothing but food and water for days. -- W. C. Fields --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
David Schmitt
2007-Mar-08 18:38 UTC
Re: Recursive home-dir does more than only files in repository
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday 05 March 2007 19:27, Luke Kanies wrote:> The fundamental problem that has to be solved here is how you handle > cross-over. What would the following code do? > > file { "/home/tim": user => tim, recurse => true } > fileset { "/home/tim/.ssh": source => "..." }One - albeit a little complex - approach might be to view a recursive remotefile{} as a "macro" that expands server-side to multiple file{} definitions which can be overridden/merged by definitions with a more specific "basepath" (i.e. the path that is actually specified in the manifest). Regards, David - -- - - hallo... wie gehts heute? - - *hust* gut *rotz* *keuch* - - gott sei dank kommunizieren wir über ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFF8FgX/Pp1N6Uzh0URAs/UAKCQk1yT/FgnKslZVys2akowReSUVwCggQHa ++iWaakHFlZZR6qgdiqp1n8=IQrm -----END PGP SIGNATURE-----
Luke Kanies
2007-Mar-08 20:21 UTC
Re: Recursive home-dir does more than only files in repository
On Mar 8, 2007, at 12:38 PM, David Schmitt wrote:> > On Monday 05 March 2007 19:27, Luke Kanies wrote: >> The fundamental problem that has to be solved here is how you handle >> cross-over. What would the following code do? >> >> file { "/home/tim": user => tim, recurse => true } >> fileset { "/home/tim/.ssh": source => "..." } > > One - albeit a little complex - approach might be to view a recursive > remotefile{} as a "macro" that expands server-side to multiple file{} > definitions which can be overridden/merged by definitions with a more > specific "basepath" (i.e. the path that is actually specified in the > manifest).This changes the behaviour of filesets -- Puppet rebuilds the remote file list on every run, at the moment, but this would only rebuild the file list on recompiles. This might actually be a good thing, but it''s definitely different. A similar difference arises between using templates and using file sources to set file contents. -- We either are networking, or we areNT networking... --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com