On 12/06/07, Luke Kanies <luke@madstop.com> wrote:> > On Jun 11, 2007, at 12:01 PM, Thijs Oppermann wrote: > > > But they don''t always work. For example, in the example (a bit > > like) above (which was what I had in my git module definition for > > clientsetup): > > > > file { "/home/${user}/.netrc.d": > > ensure => directory, > > mode => "0700", > > owner => "${user}", > > } > > > > This was included in a defintion that gets called with a $user var. > > However, it possibly gets called multiple times (as I want more > > than one clientsetup for my user). This fails miserably, because > > this directory will already be defined for puppet when the > > definition gets called a second time. However: it is *exactly* the > > same definition. The only thing I want to do here is make sure the > > directory exists before putting stuff in it. > > > > Extracting it out into a class doesn''t work, because it depends on > > the $user var in the definition. > > > > I hacked the puppet code to allow this by allowing completely > > identical file resources to be merged into one. Not sure if that''s > > the way to go, but it''s working for me for now... > > If you can provide a patch that behaves appropriately under all > conditions (esp. when handling overrides), I''ll accept it.Nice one Luke... throw me a challenge that I''m bound to choke on to shut me up.. ;) But this does make me aware that there are some extra problems with what I want. I''m not completely thrown off the scent yet, though. The thing is... what exactly do we want to happen with overrides? I''m actually thinking that we have to deny overriding merged resources. The fact is, actually, that I have not yet had any experience with overrides. Haven''t used them, and I don''t seem to need them (yet)... So I''d like some input from people that do use them, and how they would see this work or not work. What should work and what should not work? Thanks for the input, gr, Thijs --> A cult is a religion with no political power. -- Tom Wolfe > --------------------------------------------------------------------- > Luke Kanies | http://reductivelabs.com | http://madstop.com > > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >_______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
* Thijs Oppermann:> On 12/06/07, Luke Kanies <luke@madstop.com> wrote: >> On Jun 11, 2007, at 12:01 PM, Thijs Oppermann wrote: >> > I hacked the puppet code to allow this by allowing completely >> > identical file resources to be merged into one. Not sure if that''s >> > the way to go, but it''s working for me for now... >> >> If you can provide a patch that behaves appropriately under all >> conditions (esp. when handling overrides), I''ll accept it. > > > Nice one Luke... throw me a challenge that I''m bound to choke on to shut me > up.. ;) > > But this does make me aware that there are some extra problems with what I > want. I''m not completely thrown off the scent yet, though. > > The thing is... what exactly do we want to happen with overrides? I''m > actually thinking that we have to deny overriding merged resources.I have a simple rule of thumb. Merge rule: If attribute values are completely identical, the resources are the same. A resource never conflicts with itself. Conflict rule: If for two different resources, the "primary key" attributes are the same, the resources conflict. For a file, the primary key is the path. Execs have no primary key and can never conflict. For packages, the (name, provider) pair is the primary key. Note that the conflict rule is already what puppet enforces. These rules do not mention overrides at all. Now, if you override resources, they won''t be the same anymore, and the conflict rule is used to decide whether they conflict.> The fact is, actually, that I have not yet had any experience with > overrides. Haven''t used them, and I don''t seem to need them (yet)... So I''d > like some input from people that do use them, and how they would see this > work or not work. What should work and what should not work?
* tobutaz at gmail:> * Thijs Oppermann: >> On 12/06/07, Luke Kanies <luke@madstop.com> wrote: >>> On Jun 11, 2007, at 12:01 PM, Thijs Oppermann wrote: >>>> I hacked the puppet code to allow this by allowing completely >>>> identical file resources to be merged into one. Not sure if that''s >>>> the way to go, but it''s working for me for now... >>> If you can provide a patch that behaves appropriately under all >>> conditions (esp. when handling overrides), I''ll accept it. >> >> Nice one Luke... throw me a challenge that I''m bound to choke on to shut me >> up.. ;) >> >> But this does make me aware that there are some extra problems with what I >> want. I''m not completely thrown off the scent yet, though. >> >> The thing is... what exactly do we want to happen with overrides? I''m >> actually thinking that we have to deny overriding merged resources. > > I have a simple rule of thumb. > Merge rule: > If attribute values are completely identical, the resources are the > same. A resource never conflicts with itself. > Conflict rule: > If for two different resources, the "primary key" attributes are the > same, the resources conflict. For a file, the primary key is the path. > Execs have no primary key and can never conflict. For packages, the > (name, provider) pair is the primary key. > > Note that the conflict rule is already what puppet enforces. > > These rules do not mention overrides at all. Now, if you override > resources, they won''t be the same anymore, and the conflict rule is used > to decide whether they conflict.Oh, this overriding / overriden conflict should hardly ever appear. It is only in case you include both the overriden class and the overriding class that you realize both resources.>> The fact is, actually, that I have not yet had any experience with >> overrides. Haven''t used them, and I don''t seem to need them (yet)... So I''d >> like some input from people that do use them, and how they would see this >> work or not work. What should work and what should not work?
On Jun 13, 2007, at 10:55 AM, Thijs Oppermann wrote:> Nice one Luke... throw me a challenge that I''m bound to choke on to > shut me up.. ;)Frankly, I''m quite struggling to satisfy the demands of the community, and I either need people to start implementing those features they find so critical, or start paying me to implement them, because nothing else is really sustainable. I know that most people can''t pay, so I''m willing to do everything I can to help with the implementations (ask others who have added code, like Jeff McCune or Matt Palmer, how much support I provide in development), but I really do need help getting all of this work done.> But this does make me aware that there are some extra problems with > what I want. I''m not completely thrown off the scent yet, though. > > The thing is... what exactly do we want to happen with overrides? > I''m actually thinking that we have to deny overriding merged > resources.You''re not really talking about merging resources (and that''s a patch I wouldn''t accept anyway) -- you''re talking about not throwing an error on duplication when those duplicates are exactly equivalent. The check for clashes is currently two-fold: When the resource is created, I make sure that no other resource is already defined with that type and title, and when any parameter is set, I make sure that the resource was defined by either the current class or one of its parent classes. The parameter setting then retains a pointer to the class that set it, so that when overrides happen this can be checked again. The important thing here is that resource creation and parameter setting are not done in a single monolithic step -- they''re done piecemeal, because they have to be. Yet, to see if two resources really are equivalent, you have to wait until they''re entirely configured, and only then compare them. So, it seems like the whole process of this validation needs to change to support what you''re looking for -- instead of using a global resource list, as I do now, and failing as soon as a resource is defined (if there''s a conflict), we need to switch to having a separate resource list for every class heirarchy, and only failing at the end, when the configuration is translated for sending to the client. Only then do we know if the resources are completely equivalent. As to defined types, things get even more complicated -- Puppet has to convert them into the contained resources, and you''d need to compare conflicting resources before *and* after that conversion. I won''t say this is a trivial, or even simple, undertaking. It might end up making things somewhat cleaner in the end, but it''s probably a solid couple of days'' worth of work, possibly a week, for me.> The fact is, actually, that I have not yet had any experience with > overrides. Haven''t used them, and I don''t seem to need them > (yet)... So I''d like some input from people that do use them, and > how they would see this work or not work. What should work and what > should not work?The reason that overrides relate to this problem is that they are very late-binding: You might evaluate a base class first in your configuration, and its subclass last, so Puppet is set up to handle overrides in a very late-binding way. This equality check needs to be after the overrides, which means it also needs to be late-binding. -- Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. -- (attributed to) Brian W. Kernighan (unconfirmed) --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jun 13, 2007, at 11:30 AM, tobutaz at gmail wrote:> I have a simple rule of thumb. > Merge rule: > If attribute values are completely identical, the resources are the > same. A resource never conflicts with itself. > Conflict rule: > If for two different resources, the "primary key" attributes are the > same, the resources conflict. For a file, the primary key is the path. > Execs have no primary key and can never conflict. For packages, the > (name, provider) pair is the primary key. > > Note that the conflict rule is already what puppet enforces. > > These rules do not mention overrides at all. Now, if you override > resources, they won''t be the same anymore, and the conflict rule is > used > to decide whether they conflict.Overrides are only valid within a single class heirarchy, and I''ve defined them as not being in conflict. That is, class inheritance exists solely to provide this override ability, and conflict is largely determined based on the relationships between the classes doing the defining. Note Puppet also has two syntaxes in this area: One for initial resource specification, and one for modifying an existing resource. -- Due to circumstances beyond your control, you are master of your fate and captain of your soul. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jun 13, 2007, at 11:54 AM, tobutaz at gmail wrote:> Oh, this overriding / overriden conflict should hardly ever appear. > It is only in case you include both the overriden class and the > overriding class that you realize both resources.Overrides actually are used quite a bit -- I know there are some organizations that rely very heavily on them. And even if it were a rare event, it still has to be covered -- it''s the edge cases taht are really so difficult. -- Today at work an ethernet switch decided to take the ''N'' out of NVRAM -- Richard Letts --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On 13/06/07, Luke Kanies <luke@madstop.com> wrote:> > On Jun 13, 2007, at 10:55 AM, Thijs Oppermann wrote: > > > Nice one Luke... throw me a challenge that I''m bound to choke on to > > shut me up.. ;) > > Frankly, I''m quite struggling to satisfy the demands of the > community, and I either need people to start implementing those > features they find so critical, or start paying me to implement them, > because nothing else is really sustainable. > > I know that most people can''t pay, so I''m willing to do everything I > can to help with the implementations (ask others who have added code, > like Jeff McCune or Matt Palmer, how much support I provide in > development), but I really do need help getting all of this work done.Yes, I know (or at least I got that impression already earlier). I really was just kidding with that remark. I was in no way saying that you were being less than helpful. And, if anything, it''s more that I realised that maybe I was chewing off a bite that was a bit bigger than I had initially thought. I think I said it earlier, but I''ll say it again: one of the *really* good things about puppet is the community that has developed (and is still growing) around it. And I think it''s safe to say a large part of that is thanks to you, Luke. Being able to mouth off in #puppet and often get a helpful response quickly (with a bit of a delay because I live in a different timezone, but.. oh well..) is really great. Add to that the wiki and this mailing list and you have a winner. Ok, enough of this blowing smoke up you-know-whose-you-know-what... ;)> But this does make me aware that there are some extra problems with > > what I want. I''m not completely thrown off the scent yet, though. > > > > The thing is... what exactly do we want to happen with overrides? > > I''m actually thinking that we have to deny overriding merged > > resources. > > You''re not really talking about merging resources (and that''s a patch > I wouldn''t accept anyway) -- you''re talking about not throwing an > error on duplication when those duplicates are exactly equivalent. > > The check for clashes is currently two-fold: When the resource is > created, I make sure that no other resource is already defined with > that type and title, and when any parameter is set, I make sure that > the resource was defined by either the current class or one of its > parent classes. The parameter setting then retains a pointer to the > class that set it, so that when overrides happen this can be checked > again. > > The important thing here is that resource creation and parameter > setting are not done in a single monolithic step -- they''re done > piecemeal, because they have to be. Yet, to see if two resources > really are equivalent, you have to wait until they''re entirely > configured, and only then compare them. > > So, it seems like the whole process of this validation needs to > change to support what you''re looking for -- instead of using a > global resource list, as I do now, and failing as soon as a resource > is defined (if there''s a conflict), we need to switch to having a > separate resource list for every class heirarchy, and only failing at > the end, when the configuration is translated for sending to the > client. Only then do we know if the resources are completely > equivalent. > > As to defined types, things get even more complicated -- Puppet has > to convert them into the contained resources, and you''d need to > compare conflicting resources before *and* after that conversion. > > I won''t say this is a trivial, or even simple, undertaking. It might > end up making things somewhat cleaner in the end, but it''s probably a > solid couple of days'' worth of work, possibly a week, for me. > > > The fact is, actually, that I have not yet had any experience with > > overrides. Haven''t used them, and I don''t seem to need them > > (yet)... So I''d like some input from people that do use them, and > > how they would see this work or not work. What should work and what > > should not work? > > The reason that overrides relate to this problem is that they are > very late-binding: You might evaluate a base class first in your > configuration, and its subclass last, so Puppet is set up to handle > overrides in a very late-binding way. This equality check needs to > be after the overrides, which means it also needs to be late-binding.OK, see... That was really helpful. And at the same time a bit disheartening, because I now realise that my hack really is a *very* ugly hack indeed (even though the few unit tests that I created seem to not fail...). So I''m gonna go look at this some more and see if I really understand what you just said, and if I can think up a way to implement this. Thanks, Thijs --> Debugging is twice as hard as writing the code in the first place. > Therefore, if you write the code as cleverly as possible, you are, by > definition, not smart enough to debug it. > -- (attributed to) Brian W. Kernighan (unconfirmed) > --------------------------------------------------------------------- > Luke Kanies | http://reductivelabs.com | http://madstop.com > > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >_______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
Luke Kanies <luke@madstop.com> writes:> On Jun 13, 2007, at 11:54 AM, tobutaz at gmail wrote:>> Oh, this overriding / overriden conflict should hardly ever appear. It >> is only in case you include both the overriden class and the overriding >> class that you realize both resources.> Overrides actually are used quite a bit -- I know there are some > organizations that rely very heavily on them.Yes, they were a make or break feature for us. We couldn''t really use Puppet without them. We use them *constantly*; I don''t think I could model a single system we have without overrides in the way that we want to capture them. As we get more native types for things like syslog.conf, we''ll rely on them less heavily, but for example we frequently need to do things like put a user on a bunch of different hosts but add them to different supplemental groups depending on the host, and overrides are far and away the cleanest way of expressing that. -- Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/>
* Luke Kanies:> On Jun 13, 2007, at 11:54 AM, tobutaz at gmail wrote: > >> Oh, this overriding / overriden conflict should hardly ever appear. >> It is only in case you include both the overriden class and the >> overriding class that you realize both resources. > > Overrides actually are used quite a bit -- I know there are some > organizations that rely very heavily on them. > > And even if it were a rare event, it still has to be covered -- it''s > the edge cases taht are really so difficult.Good to know. The rules I suggested really have no problem with overrides; they do not mention them because they are not treated specially. Now, here is a test case of the one way I could involve overrides with the problem of merging (identification really). In this test case, the resources //wut/foo/bar/File["foo-dir"] and //wut/bar/File["foo-dir"] can not be identified, because they differ on the "ensure" parameter (one is a file, the other a directory). But they conflict, because they are identical on the "path" parameter. I find it reasonable to throw a conflict in this case, because there really is something wrong here; one cannot ask a resource to be both a file and a directory. So I''m not claiming that overrides are a problem, only that there are ways to misuse overrides that should be considered a conflict. The way to not trigger this would be to comment the "include foo" statement. Test case: #!/usr/bin/env puppet class foo { file { "foo-dir": path => "/tmp/foo", ensure => directory, } } class bar inherits foo { File["foo-dir"] { ensure => file } } class wut { include bar include foo #is /tmp/foo a file, a directory, or both? } include wut Now, maybe this attempt to document or improve puppet semantics was besides the point, because I''m not ready to actually code the stuff. I just want to show that it is a sensible improvement, that breaks nothing, except an edge case that can be fixed in puppet scripts.
On Jun 13, 2007, at 6:10 PM, tobutaz at gmail wrote:> [...] > Test case: > #!/usr/bin/env puppet > > class foo { > file { "foo-dir": > path => "/tmp/foo", > ensure => directory, > } > } > > class bar inherits foo { > File["foo-dir"] { ensure => file } > } > > class wut { > include bar > include foo > #is /tmp/foo a file, a directory, or both? > }It''s a file -- the child class always wins. It doesn''t matter how many times or in what order you ''include'' classes and their parent classes, the subclass *always* wins.> include wut > > > Now, maybe this attempt to document or improve puppet semantics was > besides the point, because I''m not ready to actually code the stuff. > I just want to show that it is a sensible improvement, that breaks > nothing, except an edge case that can be fixed in puppet scripts.Well, this case is actually pretty simple, and is the whole point for the overrides to exist. -- My favorite was a professor at a University I Used To Be Associated With who claimed that our requirement of a non-alphabetic character in our passwords was an abridgement of his freedom of speech. -- Jacob Haller --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jun 13, 2007, at 1:58 PM, Thijs Oppermann wrote:> > Yes, I know (or at least I got that impression already earlier). I > really was just kidding with that remark. I was in no way saying > that you were being less than helpful. And, if anything, it''s more > that I realised that maybe I was chewing off a bite that was a bit > bigger than I had initially thought.I *wish* the override stuff weren''t so complicated, and it''s definitely tons simpler than it used to be (and behaves more correctly), but it''s definitely not easy.> I think I said it earlier, but I''ll say it again: one of the > *really* good things about puppet is the community that has > developed (and is still growing) around it. And I think it''s safe > to say a large part of that is thanks to you, Luke. Being able to > mouth off in #puppet and often get a helpful response quickly (with > a bit of a delay because I live in a different timezone, but.. oh > well..) is really great. Add to that the wiki and this mailing list > and you have a winner.Thanks. *blush*> OK, see... That was really helpful. And at the same time a bit > disheartening, because I now realise that my hack really is a > *very* ugly hack indeed (even though the few unit tests that I > created seem to not fail...). > > So I''m gonna go look at this some more and see if I really > understand what you just said, and if I can think up a way to > implement this.Great, I look forward to it. -- The world tolerates conceit from those who are successful, but not from anybody else. -- John Blake --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Just caught this message while on a short vacation in California. Yosemite is absolutely gorgeous. Here''s my thoughts on hacking Puppet... So, I periodically hack at Puppet when I''ve got a problem with it. Admittedly, it''s not easy, especially since I''m just now getting really comfortable with Ruby, it''s debugger and irb. With that said, Luke is the most involved open source developer I''ve worked with. He''s never turned me away when I come to him with a question regarding the development of Puppet and how it works internally. He''s reachable via every modern form of communication (IRC, IM, Wiki''s, mailing list, blog, etc...) and has never hesitated to let me interrupt whatever he was working on to help me with what I was hacking. Just a plug for an open source model that''s working from the perspective of a Puppet user scratching his own itches and who''s been motivated to hack more. So, please come with development questions! Luke certainly can''t do everything, and frankly I focus on things I''m facing before I worry about writing patches for problems other people are asking about. I also recently discovered the Unroller gem, which is incredibly helpful when I need to figure out what some chunk of puppet''s source is doing behind the scenes. Check it out. Cheers, -- Jeff McCune Systems Manager The Ohio State University Department of Mathematics _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
Possibly Parallel Threads
- Example module for central git repository via http (sort of like a subversion repo) [a bit RFC]
- error "Stack level too deep" on ''exec[ "perl Makefile.PL" ] ?
- New Feature: Graphing
- Aliases not working in a subclass ?
- "server" config option (and cmd-line option) don''t do anything for puppetd?