Puppet users - Oct 2007 - ''Considered Harmful'' Considered Harmful

Because here in Puppet-land we''re all about self-reference and  
recursion, and because I actually have to give a talk at RubyConf  
next weekend about abusing self-reference, and because parent nodes  
aren''t considered harmful in my own world, I give you:

http://www.codinghorror.com/blog/archives/000982.html

I think it manages to somehow be both on- and off-topic at the same  
time.

  --
  The salesman asked me what size I wore, I told him extra-medium.
                  -- Stephen Wright
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Lak, guys!

On Friday 26 October 2007, Luke Kanies wrote:
> Because here in Puppet-land we''re all about self-reference and
> recursion, and because I actually have to give a talk at RubyConf
> next weekend about abusing self-reference, and because parent nodes
> aren''t considered harmful in my own world, I give you:
>
> http://www.codinghorror.com/blog/archives/000982.html
>
> I think it manages to somehow be both on- and off-topic at the same
> time.
I hear you, since you yourself give the best example why my inflammatory 
subject was a bad move: I never spoke out against parent nodes per-se, but 
against "template nodes" as defined in GlossaryOfTerms?version=16 and
used in
PuppetBestPractice?version=18 , which -- as seen often on the IRC -- confuse 
the heck out of some people, due to the nature of scopes. Also the problem 
is -- as far as I am currently  concerned -- solved by the changes I made to 
the wiki.

Please accept my apologies.

Regards, David
- -- 
The primary freedom of open source is not the freedom from cost, but the free-
dom to shape software to do what you want. This freedom is /never/ exercised
without cost, but is available /at all/ only by accepting the very different
costs associated with open source, costs not in money, but in time and effort.
- -- http://www.schierer.org/~luke/log/20070710-1129/on-forks-and-forking
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHIZrV/Pp1N6Uzh0URAioZAJsFXuvYsq9FplIwKYl0TP6nB2JsHQCfdzyf
f0wRO/61FGS2ar3AmeInjMI=Vnjz
-----END PGP SIGNATURE-----

On Oct 26, 2007, at 2:44 AM, David Schmitt wrote:
> I hear you, since you yourself give the best example why my  
> inflammatory
> subject was a bad move: I never spoke out against parent nodes per- 
> se, but
> against "template nodes" as defined in GlossaryOfTerms?version=16
> and used in
> PuppetBestPractice?version=18 , which -- as seen often on the IRC  
> -- confuse
> the heck out of some people, due to the nature of scopes. Also the  
> problem
> is -- as far as I am currently  concerned -- solved by the changes  
> I made to
> the wiki.
>
> Please accept my apologies.
Heh, I wasn''t looking for an apology, just pointing out something  
that was a bit meta and thus both appropriate and funny.

I actually think I''ve found a way to fix the majority of the  
confusion around using node inheritance, but I want to see how hard  
it''s actually going to be before I go sending a complicated email  
about it.  The summary is that it would make all class scopes have  
the node scope as their parent scope, so that classes could look up  
variables defined in node scopes.

  --
  Never interrupt your enemy when he is making a mistake.
                      --Napolean Bonaparte
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Friday 26 October 2007, Luke Kanies wrote:
> I actually think I''ve found a way to fix the majority of the
> confusion around using node inheritance, but I want to see how hard
> it''s actually going to be before I go sending a complicated email
> about it.  The summary is that it would make all class scopes have
> the node scope as their parent scope, so that classes could look up
> variables defined in node scopes.
I don''t think that adding a exception to the way scopes work would make
the
situation better in the long run. For me, the really simplest workable model 
when thinking about nodes is the seperate external node classifier (ENC). The 
ENC defines a node as a set of top level variables and classes. Making that 
distinction really lets me see the information coming from the ENC as "the 
configuration", while modules and other included classes are
"just" libraries
to be used. Using a bit constraint on what is written in the site.pp (e.g. by 
using "template classes") this functionality can be emulated today.

Of course this still leaves open the problem with site- and node-specific 
resources which do not lend themselves to be put into a class like the 
recently mentioned network interfaces or my own hosting_vserver definitions.

In my ideal world though the latter problem would rather be fixed by extending 
the ENC to handle resources too, instead of hacking into an already complex 
area, namely scoping. Also, creating resource in the ENC would have the 
additional benefit of enabling resource creation e.g. from a third party 
database. Like hooking up puppet into your webshop or user management 
interface. 

More pragmatically, the ENC could include $hostname and $fqdn, which could 
then hold additional per-node resources, already inheriting all the 
configuration settings from the ENC.


Regards, David

- -- 
The primary freedom of open source is not the freedom from cost, but the free-
dom to shape software to do what you want. This freedom is /never/ exercised
without cost, but is available /at all/ only by accepting the very different
costs associated with open source, costs not in money, but in time and effort.
- -- http://www.schierer.org/~luke/log/20070710-1129/on-forks-and-forking
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHIkiP/Pp1N6Uzh0URAqEDAJ0eGirA8c7macB9n6rz2ZpmHgOeqACdHLAV
XpPZmTNwbxJ0gwTlPF50KqI=EOVo
-----END PGP SIGNATURE-----

On Oct 26, 2007, at 3:05 PM, David Schmitt wrote:
> I don''t think that adding a exception to the way scopes work would
> make the
> situation better in the long run. For me, the really simplest  
> workable model
> when thinking about nodes is the seperate external node classifier  
> (ENC). The
> ENC defines a node as a set of top level variables and classes.  
> Making that
> distinction really lets me see the information coming from the ENC  
> as "the
> configuration", while modules and other included classes are
"just"
> libraries
> to be used. Using a bit constraint on what is written in the  
> site.pp (e.g. by
> using "template classes") this functionality can be emulated
today.
I completely agree -- I think an ENC, as you put it, is the right  
direction; we just need a couple of good options to replace the  
current system, including something simple.
> Of course this still leaves open the problem with site- and node- 
> specific
> resources which do not lend themselves to be put into a class like the
> recently mentioned network interfaces or my own hosting_vserver  
> definitions.
I don''t really see why; saying that each node has a node-specific  
class is still a classification question, and you can pretty easily  
have that per-node class still actually defined in Puppet.

Fundamentally, you''re going to have to define those resources  
individually somewhere; it makes a lot more sense to me to have all  
resources in Puppet and have just node/class mappings plus variable  
setting in the ENC.
> In my ideal world though the latter problem would rather be fixed  
> by extending
> the ENC to handle resources too, instead of hacking into an already  
> complex
> area, namely scoping. Also, creating resource in the ENC would have  
> the
> additional benefit of enabling resource creation e.g. from a third  
> party
> database. Like hooking up puppet into your webshop or user management
> interface.
I think it''s a very bad idea to move resource specification into your  
node classification tool, and I''m unlikely to directly support that  
kind of integration.  I could see providing a general mechanism for  
this in Puppet, but I''d hope people wouldn''t put both nodes
and
resources in the same tool; otherwise you''re never really going to  
know where nodes end and resources start.
> More pragmatically, the ENC could include $hostname and $fqdn,  
> which could
> then hold additional per-node resources, already inheriting all the
> configuration settings from the ENC.
This is exactly what they should do.  Then you can just define those  
classes if you want, or not if they''re unnecessary.  This is  
definitely what I would recommend.

  --
  I happen to feel that the degree of a person''s intelligence is  
directly
  reflected by the number of conflicting attitudes she can bring to bear
  on the same topic.            -- Lisa Alther
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On 10/29/07, Luke Kanies <luke@madstop.com> wrote:
> On Oct 26, 2007, at 3:05 PM, David Schmitt wrote:
>
> > Of course this still leaves open the problem with site- and node-
> > specific
> > resources which do not lend themselves to be put into a class like the
> > recently mentioned network interfaces or my own hosting_vserver
> > definitions.
>
> I don''t really see why; saying that each node has a node-specific
> class is still a classification question, and you can pretty easily
> have that per-node class still actually defined in Puppet.
Doesn''t needing per-node classes defeat the purpose of external node
classification?

I want to move to ENC in order to consolidate information in a
location accessible to different interfaces and people other than me
that I don''t necessarily want to trust with direct access to manifests
or classes.  So if I have to define an individual class for each node
(such as to set multiple ip addresses) then by definition I still have
raw data that I can not effectively consolidate (new ip addresses
cannot be created with out me modifying the per node class).

The only way I can think of to get around this without being able to
use defines from an ENC would be to expand on arrays to allow
iterating and multi dimensional arrays, so one could do:
interfaces[[eth0,<mac>,<ip>],[eth1,<mac>,<ip>]]
for interface in interfaces{

Freaken tab switching to the send button from a text box...

On 10/29/07, Brian Finney <y0gi636@gmail.com>
wrote:
> On 10/29/07, Luke Kanies <luke@madstop.com> wrote:
> > On Oct 26, 2007, at 3:05 PM, David Schmitt wrote:
> >
> > > Of course this still leaves open the problem with site- and node-
> > > specific
> > > resources which do not lend themselves to be put into a class
like the
> > > recently mentioned network interfaces or my own hosting_vserver
> > > definitions.
> >
> > I don''t really see why; saying that each node has a
node-specific
> > class is still a classification question, and you can pretty easily
> > have that per-node class still actually defined in Puppet.
Doesn''t needing per-node classes defeat the purpose of external node
classification?

I want to move to ENC in order to consolidate information in a
location accessible to different interfaces and people other than me
that I don''t necessarily want to trust with direct access to manifests
or classes.  So if I have to define an individual class for each node
(such as to set multiple ip addresses) then by definition I still have
raw data that I can not effectively consolidate (new ip addresses
cannot be created with out me modifying the per node class).

The only way I can think of to get around this without being able to
use defines from an ENC would be to expand on arrays to allow
iterating and multi dimensional arrays, so one could do:
interfaces[[eth0,<mac>,<ip>],[eth1,<mac>,<ip>]]

for interface in interfaces{
           file{interface[0]:
                          path => /where/ever/local/eth/definitions/go,
                          content => template(/my/eth/template) #uses
interface array
           }
}

Then again I suppose this is basically just one step away from
directly using defines from an ENC.

Although iteration would be really cool if we could also append to an
array in a global fashion, but thats a completely different dream.

Thanks
Brian

On Oct 29, 2007, at 2:35 PM, Brian Finney wrote:
>
> Doesn''t needing per-node classes defeat the purpose of external
node
> classification?
Not at all, according to how I''ve defined things.  The external node  
tool has one simple function:  Provide the list of class memberships  
that a given node has, along with any attributes/variables those  
classes have.

Note that nothing about this is Puppet-specific.  You could use this  
same node classification in your monitoring, logging, reporting,  
billing, auditing, security, and configuration tools.  In fact, this  
idea is exactly what led to thinking of an external tool -- can we  
extract this classification problem into a tool that everyone can use  
and rely on?
> I want to move to ENC in order to consolidate information in a
> location accessible to different interfaces and people other than me
> that I don''t necessarily want to trust with direct access to
manifests
> or classes.  So if I have to define an individual class for each node
> (such as to set multiple ip addresses) then by definition I still have
> raw data that I can not effectively consolidate (new ip addresses
> cannot be created with out me modifying the per node class).
This is a good point, but the solution is to enhance the interface  
between the two tools, not pollute the node tool with functionality  
that doesn''t belong there. (More below.)
> The only way I can think of to get around this without being able to
> use defines from an ENC would be to expand on arrays to allow
> iterating and multi dimensional arrays, so one could do:
> interfaces[[eth0,<mac>,<ip>],[eth1,<mac>,<ip>]]
>
> for interface in interfaces{
>            file{interface[0]:
>                           path => /where/ever/local/eth/definitions/ 
> go,
>                           content => template(/my/eth/template) #uses
> interface array
>            }
> }
>
> Then again I suppose this is basically just one step away from
> directly using defines from an ENC.
I could see having some kind of iteration in Puppet, but I can''t see  
how it would actually look.

However, I can definitely see having the interface between nodes and  
resources support arrays, and possibly hashes, or arbitrarily complex  
data structures.  Then you could have a function in Puppet (at least  
until we can come up with a good syntax) that could convert a hash  
into a resource:

   hash2resource(interface, $myname, $myhash)

This is really ugly, and starts to make it look like we should just  
be writing in Ruby instead of a custom language, since it pushes so  
much functionality into Ruby anyway.  Unfortunately, I can''t think of  
a good syntax for iteration, so I can''t get much better.  Maybe  
something like the following, assuming $interfaces is a hash of hashes:

interface { $interfaces }

That''s pretty heinous, too, because you don''t know wtf is
going on --
you can''t tell that it''s iterating, nor what format
$interfaces needs
to be in.  Worse, in many ways, you''re not providing any kind of  
isolation between the format of the hash and the parameter list for  
the resource.

I''ll have to stew on this some more, but, frankly, without a good  
proposal to decide on, it''s hard to decide whether this will make  
sense as part of the language.  I just know it doesn''t make sense to  
be part of the node classification tool.
> Although iteration would be really cool if we could also append to an
> array in a global fashion, but thats a completely different dream.
What do you mean?

  --
  A censor is a man who knows more than he thinks you ought to.
          -- Granville Hicks
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On 10/29/07, Luke Kanies <luke@madstop.com> wrote:
> On Oct 29, 2007, at 2:35 PM, Brian Finney wrote:
> >
> > Doesn''t needing per-node classes defeat the purpose of
external node
> > classification?
>
> Not at all, according to how I''ve defined things.  The external
node
> tool has one simple function:  Provide the list of class memberships
> that a given node has, along with any attributes/variables those
> classes have.
>
> Note that nothing about this is Puppet-specific.  You could use this
> same node classification in your monitoring, logging, reporting,
> billing, auditing, security, and configuration tools.  In fact, this
> idea is exactly what led to thinking of an external tool -- can we
> extract this classification problem into a tool that everyone can use
> and rely on?
>
That''s the awesome part
> > I want to move to ENC in order to consolidate information in a
> > location accessible to different interfaces and people other than me
> > that I don''t necessarily want to trust with direct access to
manifests
> > or classes.  So if I have to define an individual class for each node
> > (such as to set multiple ip addresses) then by definition I still have
> > raw data that I can not effectively consolidate (new ip addresses
> > cannot be created with out me modifying the per node class).
>
> This is a good point, but the solution is to enhance the interface
> between the two tools, not pollute the node tool with functionality
> that doesn''t belong there. (More below.)
>
> > The only way I can think of to get around this without being able to
> > use defines from an ENC would be to expand on arrays to allow
> > iterating and multi dimensional arrays, so one could do:
> >
interfaces[[eth0,<mac>,<ip>],[eth1,<mac>,<ip>]]
> >
> > for interface in interfaces{
> >            file{interface[0]:
> >                           path =>
/where/ever/local/eth/definitions/
> > go,
> >                           content => template(/my/eth/template)
#uses
> > interface array
> >            }
> > }
> >
> > Then again I suppose this is basically just one step away from
> > directly using defines from an ENC.
>
> I could see having some kind of iteration in Puppet, but I can''t
see
> how it would actually look.
Personally I''m really liking "for each <var> in
<array>", as in "for
each interface in interfaces specify resource file".
>
> However, I can definitely see having the interface between nodes and
> resources support arrays, and possibly hashes, or arbitrarily complex
> data structures.  Then you could have a function in Puppet (at least
> until we can come up with a good syntax) that could convert a hash
> into a resource:
>
>    hash2resource(interface, $myname, $myhash)
>
> This is really ugly, and starts to make it look like we should just
> be writing in Ruby instead of a custom language, since it pushes so
> much functionality into Ruby anyway.
I frequently have to remind myself (as you''ve stated) that I need to
think of the puppet language as providing a definition rather than a
process,  honestly I like it this way.
>Unfortunately, I can''t think of
> a good syntax for iteration, so I can''t get much better.  Maybe
> something like the following, assuming $interfaces is a hash of hashes:
>
> interface { $interfaces }
>
> That''s pretty heinous, too, because you don''t know wtf is
going on --
> you can''t tell that it''s iterating, nor what format
$interfaces needs
> to be in.  Worse, in many ways, you''re not providing any kind of
> isolation between the format of the hash and the parameter list for
> the resource.
The disassociation between variables and what uses them is the main
thing that has been bugging me about not having access to defines in
ENC, but I suppose this is some thing that should be handled by the
ENC and distilled down to variables and classes.   Personally I still
don''t like the disconnect but I hate complaining without at least
being able to offer an initial solution.
>
> I''ll have to stew on this some more, but, frankly, without a good
> proposal to decide on, it''s hard to decide whether this will make
> sense as part of the language.  I just know it doesn''t make sense
to
> be part of the node classification tool.
If you don''t like for each, maybe .iterate

file{$interfaces.iterate:}

for multidimensional we would end up with

file{$interfaces.iterate[0]:  # interface[0]
       path => $interfaces.iterate[1] # interface[1]
}

or perhaps (to borrow a symbol from perl) @interfaces instead of
$interfaces.iterator.  Although this still seems some what counter
intuitive and likely complicated to implement, especially if iterators
are combined, or for deeper levels of multiple dimensions.

Then again I suppose you could say that if you use an array as the
name for a define the define iterates over the array.


I really need to dig deeper into the code base, make it a lot easier
for me to understand what''s possible.
>
> > Although iteration would be really cool if we could also append to an
> > array in a global fashion, but thats a completely different dream.
>
> What do you mean?
I''ll start a new thread

Thanks
Brian

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Monday 29 October 2007, Luke Kanies wrote:
> On Oct 26, 2007, at 3:05 PM, David Schmitt wrote:
> > Of course this still leaves open the problem with site- and node-
> > specific
> > resources which do not lend themselves to be put into a class like the
> > recently mentioned network interfaces or my own hosting_vserver
> > definitions.
>
> I don''t really see why; saying that each node has a node-specific
> class is still a classification question, and you can pretty easily
> have that per-node class still actually defined in Puppet.
>
> Fundamentally, you''re going to have to define those resources
> individually somewhere; it makes a lot more sense to me to have all
> resources in Puppet and have just node/class mappings plus variable
> setting in the ENC.

> > In my ideal world though the latter problem would rather be fixed
> > by extending
> > the ENC to handle resources too, instead of hacking into an already
> > complex
> > area, namely scoping. Also, creating resource in the ENC would have
> > the
> > additional benefit of enabling resource creation e.g. from a third
> > party
> > database. Like hooking up puppet into your webshop or user management
> > interface.
>
> I think it''s a very bad idea to move resource specification into
your
> node classification tool, and I''m unlikely to directly support
that
> kind of integration.  I could see providing a general mechanism for
> this in Puppet, but I''d hope people wouldn''t put both
nodes and
> resources in the same tool; otherwise you''re never really going to
> know where nodes end and resources start.
> > More pragmatically, the ENC could include $hostname and $fqdn,
> > which could
> > then hold additional per-node resources, already inheriting all the
> > configuration settings from the ENC.
>
> This is exactly what they should do.  Then you can just define those
> classes if you want, or not if they''re unnecessary.  This is
> definitely what I would recommend.
After reading the rest of the discussion between Brian and yourself, I have to 
admit that the fundamental processes I was thinking about need 
re-examination. I will explain, perhaps enlightenment hits me (or someone 
hits me with enlightenment, whatever happens earlier).

There are two fundamental concepts currently underlying my work.

1) Create re-usable puppet modules.

This focus leads to trying to seperate completely the generic modules from the 
site specific "configuration". There the delineation is quite clear to
me:
Where to put a apache site on a debian host is generic, to put the roundcube 
vhost on the webmail server is site specific.

The next step in this evolution for me is to pull all "configuration"
data out
of puppet proper and get it into a database my users can edit.


2) Centralize all customer/user- and meta data.

Directly tying into the last sentence from above, but coming from another 
angle, is my drive to put all customer data into one (or more) central 
databases and derive system state from them. This enables _immediate_ 
propagation of customer updates from the webinterface into the system state. 
Especially easy if -- like in mailrouting -- the database can _be_ the system 
state.



On the one hand, puppet enables me to create powerful modules to have a very 
high level of talking about the system state, on the other hand, it does not 
integrate easily into the datastore and it doesn''t lend itself to
immediate
updates.

For me it was natural to extend ECN to cover the complete
"configuration"
data, which could then be fetched from a database. I see now that ECN has -- 
like e.g. facter -- a very tight definition and shouldn''t be extended
too
much lest it''s simplicity and usablity be destroyed. I''m
wondering if the new
indirection stuff might help me, by adding appropriate hooks where I can go 
to the DB instead.

I hope this shows my point of view and explains a bit of my opinions.


Regards, David
- -- 
The primary freedom of open source is not the freedom from cost, but the free-
dom to shape software to do what you want. This freedom is /never/ exercised
without cost, but is available /at all/ only by accepting the very different
costs associated with open source, costs not in money, but in time and effort.
- -- http://www.schierer.org/~luke/log/20070710-1129/on-forks-and-forking
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHJwCo/Pp1N6Uzh0URAks5AJ0Qqw7pMpwlrfZd2+ImWW8BgWetxACgngyd
cqiJwAFVtlA8DzO/L3Vw6jk=V4PO
-----END PGP SIGNATURE-----

> On 10/29/07, Luke Kanies <luke@madstop.com> wrote:
>
> I''ll have to stew on this some more, but, frankly, without a good
> proposal to decide on, it''s hard to decide whether this will make
> sense as part of the language.  I just know it doesn''t make sense
to
> be part of the node classification tool.
>
On The way to work I got another crazy idea for this:

class ip_/((\d{1,3}\.){3}\d{1,3})/{
    interface{ "ip_$1":
        ip => $1
    }
}

then ENC can

include ip_10.10.10.10
include ip_10.10.20.10

Though I could also see this being horribly abused

Thanks
Brian

On Oct 29, 2007, at 11:53 PM, Brian Finney wrote:
>>
>> I could see having some kind of iteration in Puppet, but I
can''t see
>> how it would actually look.
>
> Personally I''m really liking "for each <var> in
<array>", as in "for
> each interface in interfaces specify resource file".
So....  Maybe this is a good time to bring up the Ruby internal DSL I  
wrote more than a year ago that no one has shown any interest in.   
You can see it at lib/puppet/DSL.rb, with some example usages there.

It''s not exactly exciting, and it''s not yet complete (I
haven''t spent
much time on it because no one has shown any interest), but it''s in  
pure ruby, so you''ve got much more language power available to you.

That being said, the problem with this is that you''re really talking  
about creating resources from a list-like structure, but that list  
can''t just be a flat list, else the current implicit iteration would  
suffice.  Instead, you''re probably hoping to use a hash of hashes or  
maybe an array of hashes, which case you''ll need, at the least,  
nested ''for'' loops.  I basically think nested
''for'' loops are off the
reservation and I''m very unlikely to support them.

Puppet''s language doesn''t even support hashes yet (although,  
probably, it could pretty easily).  Once we added hashes, I think I''d  
much prefer a special syntax that only accepted, say, a hash of  
hashes or an array of hashes and turned them into a list of resources.

After all, you still can''t modify strings or arrays, nor do any kind  
of boolean matching, nor, really, do much of interest other than  
specify resources.  I agree that much of that would be nice, but it''s  
not there yet, and it seems unlikely it will be any time soon, which  
means that iteration for any purpose other than specifying resources  
is kind of pointless.
> The disassociation between variables and what uses them is the main
> thing that has been bugging me about not having access to defines in
> ENC, but I suppose this is some thing that should be handled by the
> ENC and distilled down to variables and classes.   Personally I still
> don''t like the disconnect but I hate complaining without at least
> being able to offer an initial solution.
It seems to me that you''re specifically looking at dynamic resource  
generation from outside the Puppet language, which is in neither  
Puppet''s nor the ENC''s territory.  I don''t exactly
have a published
API for this right now, but you could do it today.  If you wanted to  
have complex data structures in the same db as your ENC data, then  
you could write a function that knew how to turn those complex data  
structures into resources.

I really think this is the best option, because anything else is  
shoehorning functionality into a place you don''t really want it.   
You''ve got resource specified outside of Puppet but you want those  
resources in Puppet''s configuration.  The node API is never going to  
support this, but there are other ways to add resources to the  
configuration.

I think if people started doing this then we could start to  
characterize the effects it would have on the system and maybe how we  
really want it to behave.
>> I''ll have to stew on this some more, but, frankly, without a
good
>> proposal to decide on, it''s hard to decide whether this will
make
>> sense as part of the language.  I just know it doesn''t make
sense to
>> be part of the node classification tool.
>
> If you don''t like for each, maybe .iterate
>
> file{$interfaces.iterate:}
>
> for multidimensional we would end up with
>
> file{$interfaces.iterate[0]:  # interface[0]
>        path => $interfaces.iterate[1] # interface[1]
> }
>
> or perhaps (to borrow a symbol from perl) @interfaces instead of
> $interfaces.iterator.  Although this still seems some what counter
> intuitive and likely complicated to implement, especially if iterators
> are combined, or for deeper levels of multiple dimensions.
>
> Then again I suppose you could say that if you use an array as the
> name for a define the define iterates over the array.
Exactly.  Puppet already behaves different for list types than for  
normal strings.

Heck, maybe we just make it behave differently if that list contains  
hashes vs. strings, I dunno.

There are currently no ''object.method'' calls in Puppet,
though, nor
any ability to retrieve values by index from an array, so this would  
be at least a couple new forms of syntax.
> I really need to dig deeper into the code base, make it a lot easier
> for me to understand what''s possible.
It''s all there for the readin''. :)

  --
  One of the keys to happiness is a bad memory.   -- Rita Mae Brown
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On Oct 30, 2007, at 5:00 AM, David Schmitt wrote:
>
> After reading the rest of the discussion between Brian and  
> yourself, I have to
> admit that the fundamental processes I was thinking about need
> re-examination. I will explain, perhaps enlightenment hits me (or  
> someone
> hits me with enlightenment, whatever happens earlier).
>
> There are two fundamental concepts currently underlying my work.
>
> 1) Create re-usable puppet modules.
>
> This focus leads to trying to seperate completely the generic  
> modules from the
> site specific "configuration". There the delineation is quite
clear
> to me:
> Where to put a apache site on a debian host is generic, to put the  
> roundcube
> vhost on the webmail server is site specific.
>
> The next step in this evolution for me is to pull all  
> "configuration" data out
> of puppet proper and get it into a database my users can edit.
I guess this is the step that confuses me.  I completely understand  
wanting to separate system stuff from site stuff, but we''re already  
used to doing that in all of our other languages, why not Puppet?   
The module path can have multiple entries, so why not a standard  
module dir and a site module dir?  Your users can modify the site dir  
but not the standard dir.

Maybe the difference is that your users aren''t comfortable editing  
text files (which, um, would be strange); if that''s the case, then  
it''s just a question of writing a GUI interface to Puppet''s
language,
which shouldn''t be all that difficult, either.  (Brian -- this brings  
up one reason why I don''t like ''for'' loops: 
they''re very hard to
model with GUIs, whereas the rest of Puppet is pretty easy to model  
that way right now.)
> 2) Centralize all customer/user- and meta data.
>
> Directly tying into the last sentence from above, but coming from  
> another
> angle, is my drive to put all customer data into one (or more) central
> databases and derive system state from them. This enables _immediate_
> propagation of customer updates from the webinterface into the  
> system state.
> Especially easy if -- like in mailrouting -- the database can _be_  
> the system
> state.
It''s hard to talk about this much, because the state of affairs for  
most applications is so abysmal.

I liken it to people wanting to use Puppet to manage users and then  
complaining that Puppet resets their passwords.  There are already  
tools that do a good job of centralizing user information, and the  
application model for managing that information is completely  
different than Puppet''s application model, and always will be.  Use  
those other tools, and leave Puppet to specifying and managing the  
resources you don''t have good centralization for.

In some ways this provides a path for some amount of obsoletion of  
Puppet, in that if all of your resources have their own good  
centralization model then you might not need Puppet.  I''m fine with  
that, since 1) it''s decades away and 2) it means a lot of the  
problems Puppet is meant to solve have gone away.

For instance, rPath seems to have largely made the package management  
problem go away.  You could hook it up to an ENC for classing  
information but never have to deal with Puppet for packaging.  You''d  
still need to do so for configurations, though, since rPath doesn''t  
do that well.

And really, that''s where I see Puppet for the foreseeable future --  
it provides a way to specify and automate the bits that don''t have  
their own solutions.  A general application model is never going to  
work as well as a specific application model, and it''s always going  
to be difficult to have all of your specifications exactly right on  
all of your systems all of the time.

There''s obviously a lot of opportunity still here, though; even  
having rPath solve the packaging problem doesn''t mean that you
don''t
have change control issues, so as I (hopefully) build up the  
ecosystem here, you might have my change management tool talking to  
rPath and Puppet, triggering changes, or you might use Puppet to  
perform the actual triggers, I dunno.

There''s still so much to think about and experiment with, and so  
little time.  All I really know is that Puppet''s compiler''s
job is to
produce a list of resources for a given host, the ENC''s job is to  
produce a list of classes and attributes for a given node, and  
Puppet''s client''s job is to make sure all of the provided
resources
are in their appropriate state.  I really want to iterate and enhance  
these separately, and I think over time you''ll see a deeper  
separation between the compiler and the client (at the least, I hope  
to disconnect the configuration updates from the configuration  
evaluation).

For now, though, I''m just trying to be cautious while providing what  
functionality I can.  And, of course, working at my limits already,  
so this kind of "new stuff" is heavily dependent on either getting  
more development help from the community or growing my company (or,  
preferably, both).
> On the one hand, puppet enables me to create powerful modules to  
> have a very
> high level of talking about the system state, on the other hand, it  
> does not
> integrate easily into the datastore and it doesn''t lend itself to
> immediate
> updates.
Yeah, the immediate updates are a ways off, based on how Puppet''s  
world works right now.  I''m hoping to make some changes next year  
that would at least enable them, but it''s a very different  
application model to Puppet''s current model.

Integrating with data stores is something I''ve often thought about,  
though, and I could see providing a fixed API for that, supporting  
plugins or something, so if you wanted to generate resources from a  
database or LDAP or whatever you could do so.
> For me it was natural to extend ECN to cover the complete  
> "configuration"
> data, which could then be fetched from a database. I see now that  
> ECN has --
> like e.g. facter -- a very tight definition and shouldn''t be  
> extended too
> much lest it''s simplicity and usablity be destroyed. I''m
wondering
> if the new
> indirection stuff might help me, by adding appropriate hooks where  
> I can go
> to the DB instead.
Maybe and maybe not.  I''ll be using the indirection stuff for  
resource collection, and you could certainly use the resource  
collection interface (today!) to get resources from a database, but  
it''s not currently very dynamically configurable in that it
doesn''t
really allow you to say things like "get user resources from ldap but  
filesystem resources from the database".
> I hope this shows my point of view and explains a bit of my opinions.
Definitely.  Hopefully my painfully long-winded response has tired  
you out too much to respond. :)

Really, though, I hope it''s given you some insight into why things  
are moving slowly and why I''m trying to keep each tool''s
domain tight
and fixed.

  --
  I take my children everywhere, but they always find their way back  
home.
              --Robert Orben
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On Oct 30, 2007, at 3:06 PM, Brian Finney wrote:
>
> On The way to work I got another crazy idea for this:
>
> class ip_/((\d{1,3}\.){3}\d{1,3})/{
>     interface{ "ip_$1":
>         ip => $1
>     }
> }
>
> then ENC can
>
> include ip_10.10.10.10
> include ip_10.10.20.10
>
> Though I could also see this being horribly abused
Huh.  That''s definitely a really interesting idea.  I don''t
see
anything fundamentally wrong with it, and it exactly meets my general  
desire for avoiding explicit iteration.

I think you''d want a different keyword, "dynamic_class" or
something,
and of course you''d have to quote the IP address, but overall...  I  
really like it.

Patches accepted. :)

  --
  The difference between scientists and engineers is that when
  engineers screw up, people die.    -- Professor Orthlieb
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wednesday 31 October 2007, Luke Kanies wrote:
> On Oct 30, 2007, at 5:00 AM, David Schmitt wrote:
> > After reading the rest of the discussion between Brian and
> > yourself, I have to
> > admit that the fundamental processes I was thinking about need
> > re-examination. I will explain, perhaps enlightenment hits me (or
> > someone
> > hits me with enlightenment, whatever happens earlier).
> >
> > There are two fundamental concepts currently underlying my work.
> >
> > 1) Create re-usable puppet modules.
> >
> > This focus leads to trying to seperate completely the generic
> > modules from the
> > site specific "configuration". There the delineation is
quite clear
> > to me:
> > Where to put a apache site on a debian host is generic, to put the
> > roundcube
> > vhost on the webmail server is site specific.
> >
> > The next step in this evolution for me is to pull all
> > "configuration" data out
> > of puppet proper and get it into a database my users can edit.
>
> I guess this is the step that confuses me.  I completely understand
> wanting to separate system stuff from site stuff, but we''re
already
> used to doing that in all of our other languages, why not Puppet?
> The module path can have multiple entries, so why not a standard
> module dir and a site module dir?  Your users can modify the site dir
> but not the standard dir.
>
> Maybe the difference is that your users aren''t comfortable editing
> text files (which, um, would be strange); if that''s the case, then
> it''s just a question of writing a GUI interface to
Puppet''s language,
> which shouldn''t be all that difficult, either. 
Because a) when I sey "edit a database" I mean "use a
webinterface". My users
are customers of a shared webhosting solution, which totally excludes editing 
anything even coming near to my puppet modules. And b) because you always 
said that people who generate manifests do something wrong.
> > I hope this shows my point of view and explains a bit of my opinions.
>
> Definitely.  Hopefully my painfully long-winded response has tired
> you out too much to respond. :)
> Really, though, I hope it''s given you some insight into why things
> are moving slowly and why I''m trying to keep each tool''s
domain tight
> and fixed.
The mail gave a good overview of the bigger picture. I think I understand now 
where puppet is going under your lead.



- -- 
The primary freedom of open source is not the freedom from cost, but the free-
dom to shape software to do what you want. This freedom is /never/ exercised
without cost, but is available /at all/ only by accepting the very different
costs associated with open source, costs not in money, but in time and effort.
- -- http://www.schierer.org/~luke/log/20070710-1129/on-forks-and-forking
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHKPBX/Pp1N6Uzh0URAsylAJ4gQqjBoQGZkjyIS/UfBlN/5afvMwCfatXo
CGpZdqFjqlbx5et5W1Xa3mg=uGf3
-----END PGP SIGNATURE-----