Puppet users - May 2011 - Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/

Hi,

I have been trying to improve the coding of some of
my puppet recipes and had some trouble so wrote this:
http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/

Comments on the web seem to indicate that extlookup() solves "all
problems" but I don''t really see that and hence have proposed a
possible
way to keep the data closer together and make the extlookup()
behaviour more explicit and thus IMO clearer.

What do you think?

Simon

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On May 13, 1:25 am, Simon J Mudd <sjm...@pobox.com>
wrote:
> Hi,
>
> I have been trying to improve the coding of some of
> my puppet recipes and had some trouble so wrote
this:http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
>
> Comments on the web seem to indicate that extlookup() solves "all
> problems" but I don''t really see that and hence have proposed
a possible
> way to keep the data closer together and make the extlookup()
> behaviour more explicit and thus IMO clearer.
>
> What do you think?
>
> Simon
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On May 13, 8:07 am, jcbollinger <John.Bollin...@stJude.org> wrote:

My apologies for the empty post.

I agree that it is silly to suppose that extlookup() is a universal
solution, but personally, I don''t take that impression from web
comments in the first place.  In fact, Puppet Labs''s own recently
updated style guide recommends against using extlookup(), though that
position is controversial.

I also agree that for some sites, a richer / more powerful lookup
feature might be needed.

On the other hand, extlookup() is easy to set up and use, and it is
powerful enough for most needs.  If you need a more powerful data
lookup function then you would be lucky to find a canned one that
meets all your needs.

Moreover, I disagree with several of the opinions and conclusions in
your post:

1) You write: "The extlookup() functionality only allows [...
specifying implicitly ...] where to look for this value."  That is
false.  Extlookup does provide for configuration of a standard set of
CSV files to search (which can be parameterized by nodes'' facts), but
the function also has an optional parameter to specify a specific file
to be searched first on any given invocation.

2) You would prefer looking up data via a compound key (config_item,
lookup_value) rather than via a simple key (config_item).  You rightly
observe that this would sometimes allow fewer files to be used than
are needed in the current implementation, because different values for
the same item could be distinguished by the "lookup_value" instead of
by the file in which they reside.  The trade-off, however, is that
every lookup is more expensive, because on average the function would
need to read more lines, and to perform a more complex analysis on
each one.  In your blog you suggest a DB backend, but that''s not
really relevant because by that point you have a completely different
function than extlookup().  And even a DB performs multi-key queries
slower than single-key searches.

3) You argue that your suggested formulation of extlookup would be
"clearer as the configuration is more explicit then the current
extlookup() definition."  I think you''re missing the point.  It
would
indeed be clearer from which file the data would come, but the
objective of extlookup is to separate the *definition* of the data as
much as possible from the *use* of the data.  And I like that.  I
prefer that my manifests _not_ specify a bunch of details to every
extlookup() call, because that would obscure the details that are
important at the manifest level.

4) You seem to want to use an arbitrary-length list of values as your
lookup key, but I don''t see how that can work while retaining the
apparent objective of reducing the number of data files that must be
associated with your configuration and retaining the use of CSV.

5) You write "Using the current (puppet 2.6) mechanism would require
me, if I understand it correctly, to configure different .csv files to
be searched for each ''configuration parameter''." 
Perhaps, but I think
that''s a bit of a straw man in that you are supposing a usage mode for
extlookup that isn''t well attuned to its design.  Although it can
support them, extlookup is not designed for data domains that are
parameterized along many different dimensions.  More to the point,
such extensive parameterization is rarely needed in practice.

6) You write "I may also need in several places to override the
$extlookup_precedence. In any case this can’t be overridden several
times within the same module which is what I would need if I want to
lookup different variables."  I don''t follow.  In the first place,
you
can always use extlookup''s third parameter to specify a particular
file to check first.  In the second place, your data are not well-
defined if it seems to make sense to alter the lookup precedence of
files that are already in the search path.


Let me not be taken as an extlookup() zealot.  I do like the design as
it is, but that doesn''t mean it is ideal -- or even good -- for every
purpose.  One of the advantages of Puppet, however, is that it is
fairly easy to modify and extend.  If you would like an alternative
lookup facility then by all means, write one.  If you are community-
minded then you could consider sharing it; the Puppet Forge (http://
forge.puppetlabs.com/) makes it easy to do that.

You may also be interested in some of the extlookup() author''s
comments about it (http://www.devco.net/archives/2009/08/31/
complex_data_and_puppet.php).  He hangs out here, so you may also hear
from him directly.


Best,

John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

----- Original Message -----
> 
> Let me not be taken as an extlookup() zealot.  I do like the design
> as it is, but that doesn''t mean it is ideal -- or even good -- for
every
> purpose.  One of the advantages of Puppet, however, is that it is
> fairly easy to modify and extend.  If you would like an alternative
> lookup facility then by all means, write one.  If you are community-
> minded then you could consider sharing it; the Puppet Forge (http://
> forge.puppetlabs.com/) makes it easy to do that.
> 
> You may also be interested in some of the extlookup() author''s
> comments about it (http://www.devco.net/archives/2009/08/31/
> complex_data_and_puppet.php).  He hangs out here, so you may also
> hear from him directly.
> 
So extlookup wasnt written as a be-all and end-all of data sources.

I recognized we want data and code separation and of course I could write
an ENC but I also recognized that I could do something generic that would
solve a large % of cases for people to push out the point at which they 
need an ENC a bit later or in many cases entirely.  I hoped I could solve
it in a way that adds some value like make it easier to write configurable
modules and to do so in a way that people without programming backgrounds
can grasp and use and make their manifests a bit better.

With this in mind I wrote extlookup aiming to solve the most common problems.

I am not against ENCs I think big/complex shops absolutely must have them and I
think a lot of the extlookup bashers missed this point - if you are big
enough or complex enough where extlookup doesnt work for you then
you''ve
probably reached a legitimate point where investing time in writing an 
ENC is time well spent.  And you''re also not any more the the target
audience
extlookup was created for.

You should though just look at extlookups code, its extremely simple and easy
to understand (especially since Jesse rewrote my ancient crappy code) and this 
should let you hack up solutions that solves your problems.

Puppet is a framework, its very hackable, its easy to write functions and types,
easy to copy them out to all your servers and make it work in new and
interesting
ways.  So I totally encourage you to use extlookup as a example and solve the 
problem in your own way or even to evaluate writing your own ENC.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Are there some sample codes for ENC for complex environments? You
mentioned before to code ENC would need a lot of efforts, it would be
nice if there are some sample codes for reference ..

On 5/13/11, R.I.Pienaar <rip@devco.net> wrote:
>
>
> ----- Original Message -----
>
>>
>> Let me not be taken as an extlookup() zealot.  I do like the design
>> as it is, but that doesn''t mean it is ideal -- or even good --
for every
>> purpose.  One of the advantages of Puppet, however, is that it is
>> fairly easy to modify and extend.  If you would like an alternative
>> lookup facility then by all means, write one.  If you are community-
>> minded then you could consider sharing it; the Puppet Forge (http://
>> forge.puppetlabs.com/) makes it easy to do that.
>>
>> You may also be interested in some of the extlookup() author''s
>> comments about it (http://www.devco.net/archives/2009/08/31/
>> complex_data_and_puppet.php).  He hangs out here, so you may also
>> hear from him directly.
>>
>
> So extlookup wasnt written as a be-all and end-all of data sources.
>
> I recognized we want data and code separation and of course I could write
> an ENC but I also recognized that I could do something generic that would
> solve a large % of cases for people to push out the point at which they
> need an ENC a bit later or in many cases entirely.  I hoped I could solve
> it in a way that adds some value like make it easier to write configurable
> modules and to do so in a way that people without programming backgrounds
> can grasp and use and make their manifests a bit better.
>
> With this in mind I wrote extlookup aiming to solve the most common
> problems.
>
> I am not against ENCs I think big/complex shops absolutely must have them
> and I
> think a lot of the extlookup bashers missed this point - if you are big
> enough or complex enough where extlookup doesnt work for you then
you''ve
> probably reached a legitimate point where investing time in writing an
> ENC is time well spent.  And you''re also not any more the the
target
> audience
> extlookup was created for.
>
> You should though just look at extlookups code, its extremely simple and
> easy
> to understand (especially since Jesse rewrote my ancient crappy code) and
> this
> should let you hack up solutions that solves your problems.
>
> Puppet is a framework, its very hackable, its easy to write functions and
> types,
> easy to copy them out to all your servers and make it work in new and
> interesting
> ways.  So I totally encourage you to use extlookup as a example and solve
> the
> problem in your own way or even to evaluate writing your own ENC.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
Sent from my mobile device

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

----- Original Message -----
> Are there some sample codes for ENC for complex environments? You
> mentioned before to code ENC would need a lot of efforts, it would be
> nice if there are some sample codes for reference ..
writing an ENC isnt hard, writing one thats generic, has a nice UI
and fits the need of a lot of people is really hard.

Tools like Foreman and Puppet Dashboards are both good examples that
have been under development for years and still progress is being made.

Dan Bode has some good example of ones that are purely YAML based and 
there are a few other ones around if you search - but mostly people 
have special needs and write their own I think.

> 
> On 5/13/11, R.I.Pienaar <rip@devco.net> wrote:
> >
> >
> > ----- Original Message -----
> >
> >>
> >> Let me not be taken as an extlookup() zealot.  I do like the
> >> design
> >> as it is, but that doesn''t mean it is ideal -- or even
good -- for
> >> every
> >> purpose.  One of the advantages of Puppet, however, is that it is
> >> fairly easy to modify and extend.  If you would like an
> >> alternative
> >> lookup facility then by all means, write one.  If you are
> >> community-
> >> minded then you could consider sharing it; the Puppet Forge
> >> (http://
> >> forge.puppetlabs.com/) makes it easy to do that.
> >>
> >> You may also be interested in some of the extlookup()
author''s
> >> comments about it (http://www.devco.net/archives/2009/08/31/
> >> complex_data_and_puppet.php).  He hangs out here, so you may also
> >> hear from him directly.
> >>
> >
> > So extlookup wasnt written as a be-all and end-all of data sources.
> >
> > I recognized we want data and code separation and of course I could
> > write
> > an ENC but I also recognized that I could do something generic that
> > would
> > solve a large % of cases for people to push out the point at which
> > they
> > need an ENC a bit later or in many cases entirely.  I hoped I could
> > solve
> > it in a way that adds some value like make it easier to write
> > configurable
> > modules and to do so in a way that people without programming
> > backgrounds
> > can grasp and use and make their manifests a bit better.
> >
> > With this in mind I wrote extlookup aiming to solve the most common
> > problems.
> >
> > I am not against ENCs I think big/complex shops absolutely must
> > have them
> > and I
> > think a lot of the extlookup bashers missed this point - if you are
> > big
> > enough or complex enough where extlookup doesnt work for you then
> > you''ve
> > probably reached a legitimate point where investing time in writing
> > an
> > ENC is time well spent.  And you''re also not any more the the
> > target
> > audience
> > extlookup was created for.
> >
> > You should though just look at extlookups code, its extremely
> > simple and
> > easy
> > to understand (especially since Jesse rewrote my ancient crappy
> > code) and
> > this
> > should let you hack up solutions that solves your problems.
> >
> > Puppet is a framework, its very hackable, its easy to write
> > functions and
> > types,
> > easy to copy them out to all your servers and make it work in new
> > and
> > interesting
> > ways.  So I totally encourage you to use extlookup as a example and
> > solve
> > the
> > problem in your own way or even to evaluate writing your own ENC.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups
> > "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> > puppet-users+unsubscribe@googlegroups.com.
> > For more options, visit this group at
> > http://groups.google.com/group/puppet-users?hl=en.
> >
> >
> 
> --
> Sent from my mobile device
> 
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
> 
> 
-- 
R.I.Pienaar

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi John,

While you obviously disagree with me, thanks for taking the time to
reply. I may be looking at the problem the wrong way which is why
I''m trying to figure out if that''s the case and why.

John.Bollinger@stJude.org (jcbollinger) writes:
> On May 13, 8:07 am, jcbollinger <John.Bollin...@stJude.org> wrote:
> 
> I agree that it is silly to suppose that extlookup() is a universal
> solution, but personally, I don''t take that impression from web
> comments in the first place.  In fact, Puppet Labs''s own recently
> updated style guide recommends against using extlookup(), though that
> position is controversial.
Interesting. I wasn''t aware of that. A colleague is starting to 
use extlookup() for other functionality and if this may not be
ideal we may need to reconsider.
> I also agree that for some sites, a richer / more powerful lookup
> feature might be needed.
> 
> On the other hand, extlookup() is easy to set up and use, and it is
> powerful enough for most needs.  If you need a more powerful data
> lookup function then you would be lucky to find a canned one that
> meets all your needs.
Perhaps but I don''t see it as being that bad, and it seems to be
the closest thing to a solution I could find at the moment. My current
recipes (perhaps incorrectly) are heavily customised in a way
which often is not just "host specific" or "domain"
specific. Hence
the need to override the default settings. Other settings though
may need this thus requiring this to be more dynamic.
> Moreover, I disagree with several of the opinions and conclusions in
> your post:
> 
> 1) You write: "The extlookup() functionality only allows [...
> specifying implicitly ...] where to look for this value."  That is
> false.  Extlookup does provide for configuration of a standard set of
> CSV files to search (which can be parameterized by nodes'' facts),
but
> the function also has an optional parameter to specify a specific file
> to be searched first on any given invocation.
Perhaps coming from a database background, I''d like to mirror what
seems more _natural_ and having values spread around over potentially
a large number of files seems non-intuitive.
> 2) You would prefer looking up data via a compound key (config_item,
> lookup_value) rather than via a simple key (config_item).  You rightly
> observe that this would sometimes allow fewer files to be used than
> are needed in the current implementation, because different values for
> the same item could be distinguished by the "lookup_value"
instead of
> by the file in which they reside.  The trade-off, however, is that
> every lookup is more expensive, because on average the function would
> need to read more lines, and to perform a more complex analysis on
> each one.  In your blog you suggest a DB backend, but that''s not
> really relevant because by that point you have a completely different
> function than extlookup().  And even a DB performs multi-key queries
> slower than single-key searches.
Not if they are part of the primary key. That''s part of the point.
Any modern database can sustain thousands of queries a second doing this.
However, you are right small sites may not want to go this way to 
start with unless configuration using a database is trivial.
> 3) You argue that your suggested formulation of extlookup would be
> "clearer as the configuration is more explicit then the current
> extlookup() definition."  I think you''re missing the point. 
It would
> indeed be clearer from which file the data would come, but the
> objective of extlookup is to separate the *definition* of the data as
> much as possible from the *use* of the data.  And I like that.  I
> prefer that my manifests _not_ specify a bunch of details to every
> extlookup() call, because that would obscure the details that are
> important at the manifest level.
ok. This perhaps is a different point of view we have. extlookup()
looks like a function. As such it''s nice if the function given the same
input provides the same output. Otherwise confusion is easy.
As written it''s clear if you do
extlookup(''dns_server'') that you are
getting a "dns server" result. In many of the examples provided
the expected use case is: check first for the host, then the domain,
then ....., but that''s not specific, _and_ may be important.
Where the dns_server info is coming from is IMO important, as is how
you are trying to look it up (the order of the parameters in the search).
_How_ it gets retrieved (.csv file, db backend, ...) is _not_
something which I care about as probably that mechanism is likely to
be static and stable amongst all recipes.
> 4) You seem to want to use an arbitrary-length list of values as your
> lookup key, but I don''t see how that can work while retaining the
> apparent objective of reducing the number of data files that must be
> associated with your configuration and retaining the use of CSV.
The .csv files are currently lookup_key,lookup_value. AFAIK there is
no limit on either parameter''s size. I''m only suggesting a
third
grouping/column item_type, and that is unlikely to change the width
of the .csv files much. My suggestion is the structure would be:

dns_server,host1.example.com,ns1.example.com
dns_server,host2.example.com,ns1.example.com
dns_server,all_domains,ns1.maindomain.com
...

The suggestion (though maybe it is not possible to implement this in
ruby or the puppet recipes) was that the keys that are looked up: 
($hostname, $domain, ....) are provided as a single parameter (a list)
to the extlookup() call. Currently this is hardcoded in a list assigned to a
variable.
> 5) You write "Using the current (puppet 2.6) mechanism would require
> me, if I understand it correctly, to configure different .csv files to
> be searched for each ''configuration parameter''." 
Perhaps, but I think
> that''s a bit of a straw man in that you are supposing a usage mode
for
> extlookup that isn''t well attuned to its design.  Although it can
> support them, extlookup is not designed for data domains that are
> parameterized along many different dimensions.  More to the point,
> such extensive parameterization is rarely needed in practice.
ok. That''s perhaps the main "misunderstanding on my side". If
I want to
configure a variable to be looked up, _I_ would like that information
to be kept mainly in the same place.  Depending on the variable
I may want to lookup the result based on $hostname, $domain, ...
but I may want to lookup based on other variables. So having this 
list not visibly part of the extlookup() call appears to make this
harder.
> 6) You write "I may also need in several places to override the
> $extlookup_precedence. In any case this can’t be overridden several
> times within the same module which is what I would need if I want to
> lookup different variables."  I don''t follow.  In the first
place, you
> can always use extlookup''s third parameter to specify a particular
> file to check first.
No, not the file to check, but the parameters to lookup. that is I''d 
like the extlookup() to be a generic lookup function which allows me
to choose the values I lookup, and depending on each call.
> Let me not be taken as an extlookup() zealot.  I do like the design as
> it is, but that doesn''t mean it is ideal -- or even good -- for
every
> purpose.  One of the advantages of Puppet, however, is that it is
> fairly easy to modify and extend.  If you would like an alternative
> lookup facility then by all means, write one.  If you are community-
> minded then you could consider sharing it; the Puppet Forge (http://
> forge.puppetlabs.com/) makes it easy to do that.
I partly wrote this to try to get an idea as to whether I''m missing the
point, something which often happens when you don''t use a tool the way
it''s designed, or whether perhaps more flexibility is desirable.

So far I''ve seen many of the extlookup() usage examples show the simple
setup a DNS server, setup ...., but not application related where the
configuration gets more complicated because the required parameters 
depend less on the hostname/domain/etc but on groups of boxes setup
a certain way.  Maybe I''m just missing examples to indicate
how this can be done. Nevertheless the "lack of transparency" of how
the extlookup() makes it''s calls is what troubles me somewhat.
I guess I need to see why Puppet Labs recommend against using
extlookup() even though they have added it recently in 2.6.

In any case thanks for sharing your thoughts with me. Hopefully 
this will help me get a better understanding of how to configure what
I want.

Simon

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

rip@devco.net ("R.I.Pienaar") writes:
> > Let me not be taken as an extlookup() zealot.  I do like the design
> > as it is, but that doesn''t mean it is ideal -- or even good
-- for every
> > purpose.  One of the advantages of Puppet, however, is that it is
> > fairly easy to modify and extend.  If you would like an alternative
> > lookup facility then by all means, write one.  If you are community-
> > minded then you could consider sharing it; the Puppet Forge (http://
> > forge.puppetlabs.com/) makes it easy to do that.
> > 
> > You may also be interested in some of the extlookup()
author''s
> > comments about it (http://www.devco.net/archives/2009/08/31/
> > complex_data_and_puppet.php).  He hangs out here, so you may also
> > hear from him directly.
> > 
> 
> So extlookup wasnt written as a be-all and end-all of data sources.
> 
> I recognized we want data and code separation and of course I could write
> an ENC but I also recognized that I could do something generic that would
> solve a large % of cases for people to push out the point at which they 
> need an ENC a bit later or in many cases entirely.  I hoped I could solve
> it in a way that adds some value like make it easier to write configurable
> modules and to do so in a way that people without programming backgrounds
> can grasp and use and make their manifests a bit better.
> 
> With this in mind I wrote extlookup aiming to solve the most common
problems.
> 
> I am not against ENCs I think big/complex shops absolutely must have them
and I
> think a lot of the extlookup bashers missed this point - if you are big
> enough or complex enough where extlookup doesnt work for you then
you''ve
> probably reached a legitimate point where investing time in writing an 
> ENC is time well spent.  And you''re also not any more the the
target audience
> extlookup was created for.
That''s an interesting thought and also something I''m not sure
of the answer to.
Personally I''m responsible for managing quite a few database servers,
but the
company of course has several more severs of different types what we manage
besides that.

When I looked at the ENCs it seems to approach (if I understand how
they should be written) the problem from a different point of
view. That is provide: a list of global variables, and a list of
classes which should be defined for each node. That requires thinking
about the problem in quite a different way to the extlookup()
functionality which basically says: give me this value, and now based
on that I can decide what to do...  That''s my understanding of the view
"views" of how to configure a system. If that''s right one
thing I don''t like
with the ENCs is the fact that it appears to use and rely on a large number
of global variables which are then used by the classes.  As systems get more
complex that strikes me as not being good as the class behaviour may
change depending on the scope of where something comes from and how
it''s included
and that''s not clear. I''ve already been tripped up by not
setting a global variable
correctly (or using the wrong name) and not noticing and of course behaviour
changes.  So I still think it''s clearer if you can do:

$ip_address = extlookup( parameters to get an ip address )
$interface  = extlookup( parameters to get an interface name )
interface::check{ $interface: ip_address => $ip_address }

than

inteface::check{ $global_interface1: ip_address => $global_ip_address1 }

as in the latter case it''s not clear where these 2 values come
from. Their names are also less intuitive and if you have a number of
these to set then it gets worse.
> You should though just look at extlookups code, its extremely simple and
easy
> to understand (especially since Jesse rewrote my ancient crappy code) and
this
> should let you hack up solutions that solves your problems.
Ive tried to keep away from Ruby. Perhaps that''s a real issue but yes,
given
the idea is certainly sound, extending in a way that''s more appropriate
to my
perceived needs seems a sane approach.
> Puppet is a framework, its very hackable, its easy to write functions and
types,
> easy to copy them out to all your servers and make it work in new and
interesting
> ways.  So I totally encourage you to use extlookup as a example and solve
the
> problem in your own way or even to evaluate writing your own ENC.
One way or another I am probably going to go down that path. For the reasons
stated earlier building a complete ENC means that everything depends on it.
using a extlookup() type functionaliy initially at least makes the change seem
less imposing. However, if used heavily in the end it''s doing just the
same thing,
but a different way.

Thanks for your comments.

Simon

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On May 14, 2011, at 12:55 AM, Simon J Mudd wrote:
> Hi John,
> 
> While you obviously disagree with me, thanks for taking the time to
> reply. I may be looking at the problem the wrong way which is why
> I''m trying to figure out if that''s the case and why.
> 
> John.Bollinger@stJude.org (jcbollinger) writes:
> 
>> On May 13, 8:07 am, jcbollinger <John.Bollin...@stJude.org>
wrote:
>> 2) You would prefer looking up data via a compound key (config_item,
>> lookup_value) rather than via a simple key (config_item).  You rightly
>> observe that this would sometimes allow fewer files to be used than
>> are needed in the current implementation, because different values for
>> the same item could be distinguished by the "lookup_value"
instead of
>> by the file in which they reside.  The trade-off, however, is that
>> every lookup is more expensive, because on average the function would
>> need to read more lines, and to perform a more complex analysis on
>> each one.  In your blog you suggest a DB backend, but that''s
not
>> really relevant because by that point you have a completely different
>> function than extlookup().  And even a DB performs multi-key queries
>> slower than single-key searches.
> 
> Not if they are part of the primary key. That''s part of the point.
> Any modern database can sustain thousands of queries a second doing this.
> However, you are right small sites may not want to go this way to 
> start with unless configuration using a database is trivial.
As a side note, putting the information in a csv file should make it much easier
to version along with the rest of your files.  I assume, though I could easily
be wrong, that most DBs aren''t easy to version control using GIT and
Subversion.  That means (using csv files) you can keep your data in lockstep
with the version of the config files.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

kc7zzv@gmail.com (Patrick) writes:
> On May 14, 2011, at 12:55 AM, Simon J Mudd wrote:
...
> > Not if they are part of the primary key. That''s part of the
point.
> > Any modern database can sustain thousands of queries a second doing
this.
> > However, you are right small sites may not want to go this way to 
> > start with unless configuration using a database is trivial.
> 
> As a side note, putting the information in a csv file should make it
> much easier to version along with the rest of your files.  I assume,
> though I could easily be wrong, that most DBs aren''t easy to
version
> control using GIT and Subversion.
Using GIT or subversion, no. But versioning the configuration in the
database is hardly difficult to do. (If you got that far you have
enough infrastructure and expertise to solve this).
> That means (using csv files) you can keep your data in lockstep with
> the version of the config files.
Indeed and I think that''s nice. So the only time this becomes an issue
is when the load on the puppet master to provide the lookup based on
the extlookup() calls is sufficiently high to be a problem. When it''s
a problem you override the "csv" lookup to work the same based on some
sort of "db lookup" alternative.

Simon

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

John.Bollinger@stJude.org (jcbollinger) writes:
> In fact, Puppet Labs''s own recently updated style guide recommends
> against using extlookup(), though that position is controversial.
I found the URL: http://docs.puppetlabs.com/guides/style_guide.html

"The extlookup() Function

Modules should avoid the use of extlookup() in favor of ENCs or other
alternatives."

The statement is clear, but it would be nice to know the reasoning.
That is if there''s a problem with extlookup() then it would be good to
know what it is. This may have been discussed elsewhere but it
shouldn''t be necessary to have to go and search for this.

Simon

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

sjmudd@pobox.com (Simon J Mudd) writes:
> John.Bollinger@stJude.org (jcbollinger) writes:
> 
> > In fact, Puppet Labs''s own recently updated style guide
recommends
> > against using extlookup(), though that position is controversial.
> 
> I found the URL: http://docs.puppetlabs.com/guides/style_guide.html
> 
> "The extlookup() Function
> 
> Modules should avoid the use of extlookup() in favor of ENCs or other
alternatives."
> 
> The statement is clear, but it would be nice to know the reasoning.
> That is if there''s a problem with extlookup() then it would be
good to
> know what it is. This may have been discussed elsewhere but it
> shouldn''t be necessary to have to go and search for this.
In the end I did.
http://groups.google.com/group/puppet-users/browse_thread/thread/34e4dfb4828a2835/d9f6d8d6c7bba841?show_docid=d9f6d8d6c7bba841
It is a good read. However, from the discussion a few things strike me:

1. the use of parameterised classes is recommended heavily. I''ve just
found out about this "new feature" inspite of using puppet for a long
time. Hence many recipes that I''m using are not paremeterised and I
have a large number of similar classes. This is certainly worth fixing
but is quite a painful transition to make given the number of systems
in use and the chance of creating heavy breakage during that change.
So if you haven''t used parameterised classes much that''s the
first
thing that needs looking at.

2. Does puppet provide an ENC? The dashboard? Again if you''re not
using one switching over is a heavy tasks as you have to define
_everything_ first before you can use it. (that''s how it seems to
me). I recently attended a puppet master course and it''s content was
interesting, but despite there was little emphasis on this sort of
thing "getting the high level design right" and more on the nitty
gritty of the usage of the lower-level resource types. To push the use
of an ENC it seems that this higher level planning would be most helpful.
Also a real-life overview of how this would be implemented.  Saying how to
configure an ssh server doesn''t count. I''d be interested in
the "deployment"
of a series of database servers, their configuration, setup, and how 
to take into account differences between one set of boxes and another.
If I can relate to a use case that fits my own (using an ENC) and
there''s
a way to slowly bring in more systems into using this ENC then it makes
it easier to follow that path.

The alternative of using an extlookup() behaviour does indeed have the
inconvenience of hiding some behaviour depending on external aspects
which are picked up by the extlookups, but if you already have working
recipes perhaps and aren''t using an ENC this gives you a way to
clearly remove the conditional logic the recipes currently have. It''s
then a smaller step "up" to using an ENC if you want to go that
way. For those that don''t it still becomes reasonably clear where the
configuration is parameterised and how/why that''s done, so
it''s still
a more flexible approach.

Judging from people I''ve listened to at the last puppet camp there are
many ways to use puppet. Sites vary from small to very large and
there''s no clear "one solution" fits all. So I''m
inclined to agree
that a extlookup() [ or get_config_by_key lookup] mechanism may work
fine for a lot of people.  As long as you know and understand the
limitations of the approach you are taking that''s fine.

Again the ENC setup described to me at the course I went on did not
mention parameterised classes, but simple variable settings and
classes.  I guess that will change as perhaps if you use parameterised
classes the enc should do nothing other than provide a definition of
the parameterised classes to use and their initial setup valus. Removing the 
global variables would ensure that each class''s behaviour is
encapsulated
only by it''s input parameters and that does sound much more reasonable.

Simon

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On May 14, 8:52 am, Simon J Mudd <sjm...@pobox.com>
wrote:
> sjm...@pobox.com (Simon J Mudd) writes:
>
> > John.Bollin...@stJude.org (jcbollinger) writes:
>
> > > In fact, Puppet Labs''s own recently updated style guide
recommends
> > > against using extlookup(), though that position is controversial.
>
> > I found the URL:http://docs.puppetlabs.com/guides/style_guide.html
>
> > "The extlookup() Function
>
> > Modules should avoid the use of extlookup() in favor of ENCs or other
alternatives."
>
> > The statement is clear, but it would be nice to know the reasoning.
> > That is if there''s a problem with extlookup() then it would
be good to
> > know what it is. This may have been discussed elsewhere but it
> > shouldn''t be necessary to have to go and search for this.
>
> In the end I
did.http://groups.google.com/group/puppet-users/browse_thread/thread/34e4...
> It is a good read.

Now you know what I meant when I called Puppet Labs''s style
recommendation "controversial."  Personally, I remain unpersuaded that
the recommended style is a good technical choice for most people, but
I will not rehash the arguments that you have just read.

> However, from the discussion a few things strike me:
>
> 1. the use of parameterised classes is recommended heavily. I''ve
just
> found out about this "new feature" inspite of using puppet for a
long
> time. Hence many recipes that I''m using are not paremeterised and
I
> have a large number of similar classes. This is certainly worth fixing
> but is quite a painful transition to make given the number of systems
> in use and the chance of creating heavy breakage during that change.
> So if you haven''t used parameterised classes much that''s
the first
> thing that needs looking at.

If you have a complex set of manifests that do not use parameterized
classes, then I think it much safer to coalesce similar classes with
the help of extlookup() than by attempting to switch to parameterized
classes.  There are a couple of important ways (other than
parameterization itself) by which parameterized classes differ from
non-parameterized ones, and the more complex your current manifests,
the more likely these are to bite you if you parameterize existing
classes.

Note, however, that it is the use of extlookup() *as an alternative to
class parameterization* that the style guide recommends against.  Do
not take it as a blanket recommendation against using the feature.
Note also that another part of the Style Guide''s approach is that
classes should avoid including other classes.  Although it may not be
immediately obvious, that goes hand-in-hand with extensive use of
parameterized classes.  Do take that into consideration as you
evaluate the Style Guide''s recommendations.

> [...] if you already have working recipes
> perhaps and aren''t using an ENC [, extlookup] gives you a way to
> clearly remove the conditional logic the recipes currently have.
It''s
> then a smaller step "up" to using an ENC if you want to go that
> way. For those that don''t it still becomes reasonably clear where
the
> configuration is parameterised and how/why that''s done, so
it''s still
> a more flexible approach.

I agree.  Furthermore, I maintain that if you are not using an ENC,
then parameterized classes offer few advantages to offset their
disadvantages.  Therefore, if you are not already using an ENC and do
not want to switch to one right now, then extlookup() is the best
generally available way to externalize your configuration''s data.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Simon,

On May 14, 2:55 am, Simon J Mudd <sjm...@pobox.com>
wrote:
> Hi John,
>
> While you obviously disagree with me, thanks for taking the time to
> reply. I may be looking at the problem the wrong way which is why
> I''m trying to figure out if that''s the case and why.

Fair enough.

> John.Bollin...@stJude.org (jcbollinger) writes:
> > On May 13, 8:07 am, jcbollinger <John.Bollin...@stJude.org>
wrote:
> > On the other hand, extlookup() is easy to set up and use, and it is
> > powerful enough for most needs.  If you need a more powerful data
> > lookup function then you would be lucky to find a canned one that
> > meets all your needs.
>
> Perhaps but I don''t see it as being that bad, and it seems to be
> the closest thing to a solution I could find at the moment. My current
> recipes (perhaps incorrectly) are heavily customised in a way
> which often is not just "host specific" or "domain"
specific. Hence
> the need to override the default settings. Other settings though
> may need this thus requiring this to be more dynamic.

Let''s not cast things in terms of "correctness," except
insomuch as
whether they reliably produce the desired effect on clients.

Depending on what you''re trying to do with Puppet, however, it is
certainly true that some approaches to structuring your manifests will
make them easier to write and maintain than will others.  Personally,
when writing specializations into my manifests, I find it useful to
keep in mind the question of *why* a particular host or group of hosts
is different from others.  I have yet to run into an answer that
didn''t fall into one of these categories, or a combination of them:

1) It has a functional role that some other systems do not
2) Its network environment requires differences from some other
systems.
3) It''s just special

It has worked well for me so far to model roles via (unparameterized)
classes, assigned to nodes via their node declarations.  That leaves
only one level between "common" and "node-specific" where I
might need
to customize data.  (That intermediate level could in principal be
parameterized by network domain, but in my case it is by subnet.)
Most of my nodes do not require per-node customization, so I don''t end
up with many data files for extlookup.

> > Moreover, I disagree with several of the opinions and conclusions in
> > your post:
>
> > 1) You write: "The extlookup() functionality only allows [...
> > specifying implicitly ...] where to look for this value."  That
is
> > false.  Extlookup does provide for configuration of a standard set of
> > CSV files to search (which can be parameterized by nodes''
facts), but
> > the function also has an optional parameter to specify a specific file
> > to be searched first on any given invocation.
>
> Perhaps coming from a database background, I''d like to mirror what
> seems more _natural_ and having values spread around over potentially
> a large number of files seems non-intuitive.

If extlookup use would indeed require you to maintain a large number
of separate files then that might be a good reason to find something
better, but in all likelihood you can avoid that, or else structure it
in a sane way.  Consider also:

When you work with a database, do you normally focus on how it maps
data to the host filesystem?

Given the diverse data parameterization you described, if you created
a database for your configuration data, would you really organize
everything into a single table?  (And what would be its key?)

If you wanted to allow customization of your data with varying scopes
(e.g. supporting both per-node and per-network customization of the
same parameters) then how would you design the DB?

> > 2) You would prefer looking up data via a compound key (config_item,
> > lookup_value) rather than via a simple key (config_item).  
> > [...]  And even a DB performs multi-key queries
> > slower than single-key searches.
>
> Not if they are part of the primary key. That''s part of the point.

Well that''s a possible answer for you then.  Extlookup performs
lookups based on a single key, and nothing prevents you from using
keys that allow you to flatten your data.  For example, you can
structure your data so that instead of extlookup("interface"), you can
extlookup("interface-$subnet-$is_master").  In other words, the
distinction in your examples between "config_item" and
"lookup_value(s)" is artificial and unnecessary.  There is no reason
why you could not combine them all into a single field in the CSV.

> > 3) You argue that your suggested formulation of extlookup would be
> > "clearer as the configuration is more explicit then the current
> > extlookup() definition."  I think you''re missing the
point.  It would
> > indeed be clearer from which file the data would come, but the
> > objective of extlookup is to separate the *definition* of the data as
> > much as possible from the *use* of the data.  And I like that.  I
> > prefer that my manifests _not_ specify a bunch of details to every
> > extlookup() call, because that would obscure the details that are
> > important at the manifest level.
>
> ok. This perhaps is a different point of view we have. extlookup()
> looks like a function. As such it''s nice if the function given the
same
> input provides the same output. Otherwise confusion is easy.

Personally, I don''t find it at all surprising when the output of a
function depends on factors other than its arguments, as long as the
dependencies are well documented.  That''s common in the programming
and database worlds.  YMMV, of course.

There are things you could do with extlookup() that would contribute
to confusion, such as overriding the lookup precedence variable below
global scope.  You could also write put ''exec { "cd /$mydir; rm -f
*":  }'' into one of your manifests.  Avoid these things (even if
you''re *sure* $mydir will always be defined).

> As written it''s clear if you do
extlookup(''dns_server'') that you are
> getting a "dns server" result. In many of the examples provided
> the expected use case is: check first for the host, then the domain,
> then ....., but that''s not specific, _and_ may be important.

When you say it''s not "specific," do you mean that you
can''t tell from
the extlookup() call itself?  The typical use of extlookup() involves
defining the lookup precedence once, globally.  For the exceptional
case where you need to override the lookup order, you specify the
first-resort lookup file explicitly in the function call.  If you do
that, then you always know how the lookup will proceed.  To the extent
that your concern may be premised on the notion that you''ll override
the lookup precedence via its controlling variable, I suggest you
construe your point as a good reason to not do that.

> > 4) You seem to want to use an arbitrary-length list of values as your
> > lookup key, but I don''t see how that can work while retaining
the
> > apparent objective of reducing the number of data files that must be
> > associated with your configuration and retaining the use of CSV.
>
> The .csv files are currently lookup_key,lookup_value. AFAIK there is
> no limit on either parameter''s size. I''m only suggesting
a third
> grouping/column item_type, and that is unlikely to change the width
> of the .csv files much. My suggestion is the structure would be:
>
> dns_server,host1.example.com,ns1.example.com
> dns_server,host2.example.com,ns1.example.com
> dns_server,all_domains,ns1.maindomain.com

Ah.  I thought you were suggesting a variable number of additional
columns between lookup_key and lookup_value.  Nevertheless, and
contrary to my own (4), I just offered a way to approach the problem
that works with extlookup() as currently written.  That is, put all
key information into the lookup_key field, delimited by some character
other than a comma.  Not only will that allow you to flatten the file
to an arbitrary degree, it will also allow (force) you to explicitly
include all the key parts in your extlookup() calls.


Cheers,

John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

John.Bollinger@stJude.org (jcbollinger) writes:
> On May 14, 8:52 am, Simon J Mudd <sjm...@pobox.com> wrote:
> 
> > However, from the discussion a few things strike me:
> >
> > 1. the use of parameterised classes is recommended heavily.
I''ve just
> > found out about this "new feature" inspite of using puppet
for a long
> > time. Hence many recipes that I''m using are not paremeterised
and I
> > have a large number of similar classes. This is certainly worth fixing
> > but is quite a painful transition to make given the number of systems
> > in use and the chance of creating heavy breakage during that change.
> > So if you haven''t used parameterised classes much
that''s the first
> > thing that needs looking at.
> 
> If you have a complex set of manifests that do not use parameterized
> classes, then I think it much safer to coalesce similar classes with
> the help of extlookup() than by attempting to switch to parameterized
> classes.
Perhaps. To some extend my non-parameterised classes are _very_ similar
in many ways except for various parameters (creation of logical volumes
and filesystems, version of mysql to use, creation of certain cron jobs
for importing informtation into a configuration database, and other crons
for doing standard jobs, etc....) Most of the "completely common"
stuff
has been taken out. However, using extlookup() doesn''t really work for
me "right now" for a simple reason I haven''t yet mentioned.
I''d further
like to have a lookup_by_key() function [let''s say similar to the
extlookup()
one provided now] but also a lookup_by_regexp() function where I can match
groups of boxes, in my case mainly based on regexps of their hostnames
rather than on specific $hostname, or $domain, etc.... Without that
I need to add more entries to match by lots of hostnames, which doesn''t
work.

So I''d like something like 

$mount_point = lookup_by_regexp( ''mount_point'', $hostname,
''regexp_file'', '''' ) # parameters:
lookup_item, lookup_value, regexp file to lookup, default
$lvsize      = lookup_by_regexp( ''lvsize'',      $hostname,
''regexp_file'', '''' )
class db_lvconfig { db_lvconfig: mount_point => $mount_point, lvsize =>
$lvsize } # note: other default parameters not shown like: owner, group, mode,
vgname, lvname

and have in the .csv file something like 

mount_point,/dbservera/,/mount_point/for/servera
mount_point,/dbserverb/,/mount_point/for/serverb
mount_point,/dbserverc/,/mount_point/for/serverc
lvsize,/dbservera/,100G
lvsize,/dbserverb/,200G
lvsize,/dbserverc/,300G

Then if $hostname = ''dbservera-001''
it would get $lvsize = ''100G'', and $mount_point =
''/mount_point/for/servera''

If I have 10 dbservera-xxx boxes then this keeps the config simple and the
.csv files much smaller.
> There are a couple of important ways (other than
> parameterization itself) by which parameterized classes differ from
> non-parameterized ones, and the more complex your current manifests,
> the more likely these are to bite you if you parameterize existing
> classes.
Perhaps. for most of my cases I don''t think that''s the case.
I''m
slowly adjusting the configuration of several classes to use
parameterised classes but even these still work nicely with
extlookup() alternatives such as the currently unimplemented
lookup_by_key() or lookup_by_regexp() alternatives I''m thinking of
using.
> Note, however, that it is the use of extlookup() *as an alternative to
> class parameterization* that the style guide recommends against.  Do
> not take it as a blanket recommendation against using the feature.
I''m not.  You need to have a very considered mindset to build
_everything_ top down using an ENC and classes or globalvariables.  If
using global variables as heavily as puppet seem to imply then ensuing
that you have a clear naming policy for this is quite important to
avoid unexpected name clashes (or mistakes) which might go unnoticed.
So I''d certainly like to see real world examples of complete
configurations to be fully convinced this works well. Perhaps after
moving a configuration to use extlookup() type mechanisms it then
becomes easier to make a further step "up" to get to the ENCs style
Puppetlabs reccommends.  However, with system configuration being a
continual ongoing process I tend to think that a 100% ENC-only style
build using only parameterised classes is going to be a hard thing to
achieve in practice. And the global variables just don''t convince me
yet. Perhaps I still have to see the light.
> Note also that another part of the Style Guide''s approach is that
> classes should avoid including other classes.  Although it may not be
> immediately obvious, that goes hand-in-hand with extensive use of
> parameterized classes.  Do take that into consideration as you
> evaluate the Style Guide''s recommendations.
Hm.. well building bigger blocks from smaller ones tends to be a good
practice to follow, so this recommendation seems counterintuitive.
I''ll have to lookup the section and see if they explain why, but guess
it''s because the intention is you build your server in terms of
smallish components.

I still tend to like the idea of:

class db_lvconfig { db_lvconfig: mount_point => $mount_point, lvsize =>
$lvsize }

You shouldn''t have to understand or worry about what''s
underneath this definition
as long as the interface is clearly defined.
> > [...] if you already have working recipes
> > perhaps and aren''t using an ENC [, extlookup] gives you a way
to
> > clearly remove the conditional logic the recipes currently have.
It''s
> > then a smaller step "up" to using an ENC if you want to go
that
> > way. For those that don''t it still becomes reasonably clear
where the
> > configuration is parameterised and how/why that''s done, so
it''s still
> > a more flexible approach.
> 
> I agree.  Furthermore, I maintain that if you are not using an ENC,
> then parameterized classes offer few advantages to offset their
> disadvantages.  Therefore, if you are not already using an ENC and do
> not want to switch to one right now, then extlookup() is the best
> generally available way to externalize your configuration''s data.
I tend to agree.

Perhaps it''s also a matter of scale and I don''t work in a big
enough
environment to see the need for an ENC, at least the way it seems to
be envisaged by Puppetlabs. It may well be that I''m not fully
understanding the message or doing things the right way, so if someone
can shed more light on this I''d be more than happy to learn some more.

Simon

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

----- Original Message -----
> John.Bollinger@stJude.org (jcbollinger) writes:
> 
> Perhaps. To some extend my non-parameterised classes are _very_ similar
> in many ways except for various parameters (creation of logical volumes
> and filesystems, version of mysql to use, creation of certain cron jobs
> for importing informtation into a configuration database, and other crons
> for doing standard jobs, etc....) Most of the "completely common"
stuff
> has been taken out. However, using extlookup() doesn''t really work
for
> me "right now" for a simple reason I haven''t yet
mentioned. I''d further
> like to have a lookup_by_key() function [let''s say similar to the
extlookup()
> one provided now] but also a lookup_by_regexp() function where I can match
> groups of boxes, in my case mainly based on regexps of their hostnames
> rather than on specific $hostname, or $domain, etc.... Without that
> I need to add more entries to match by lots of hostnames, which
> doesn''t work.
I think you''re really trying to just avoid thinking in the puppet way
instead
trying to force some previous ideas on how things should work on it rather
than figure out how it works.

For example, you could just define a fact on your machines that indicate what
role they have - perhaps based on your regex hostnames - and then use that in
extlookup to select csv file to use.  That would achieve your goal and it would
be the puppet way of solving it.


> Puppetlabs reccommends.  However, with system configuration being a
> continual ongoing process I tend to think that a 100% ENC-only style
> build using only parameterised classes is going to be a hard thing to
> achieve in practice. And the global variables just don''t convince
me
> yet. Perhaps I still have to see the light.
ENCs can (now) pass parameters to param classes.  So you dont have to do
it all with global vars.
> > Note also that another part of the Style Guide''s approach is
that
> > classes should avoid including other classes.  Although it may not
> > be
> > immediately obvious, that goes hand-in-hand with extensive use of
> > parameterized classes.  Do take that into consideration as you
> > evaluate the Style Guide''s recommendations.
> 
> Hm.. well building bigger blocks from smaller ones tends to be a good
> practice to follow, so this recommendation seems counterintuitive.
> I''ll have to lookup the section and see if they explain why, but
> guess it''s because the intention is you build your server in terms
of
> smallish components.
+1, I think this is mostly down to the bad design in param classes - they
really do not promote the use of many small classes that creates a larger
module.  Making modules harder to read and making dependencies harder to
specify.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi John,

John.Bollinger@stJude.org (jcbollinger) writes:

...
> Let''s not cast things in terms of "correctness," except
insomuch as
> whether they reliably produce the desired effect on clients.
Indeed. As always there is more than one way to solve a problem.
> Depending on what you''re trying to do with Puppet, however, it is
> certainly true that some approaches to structuring your manifests will
> make them easier to write and maintain than will others.  Personally,
> when writing specializations into my manifests, I find it useful to
> keep in mind the question of *why* a particular host or group of hosts
> is different from others.  I have yet to run into an answer that
> didn''t fall into one of these categories, or a combination of
them:
> 
> 1) It has a functional role that some other systems do not
In my case 90% of the differences are functional differences, often
expressed as part of the hostname.
> 2) Its network environment requires differences from some other
> systems.
There may be some minor differences here but they are the exceptions
rather than the rule.
> 3) It''s just special
I have a few of these too.
> It has worked well for me so far to model roles via (unparameterized)
> classes, assigned to nodes via their node declarations.  That leaves
> only one level between "common" and "node-specific"
where I might need
> to customize data.
Yes, I have db_server::type_a and db_server::type_b, ...
> (That intermediate level could in principal be
> parameterized by network domain, but in my case it is by subnet.)
> Most of my nodes do not require per-node customization, so I don''t
end
> up with many data files for extlookup.
In my case one db_server is pretty much similar to another one. Things
vary such as mysql version to use (normally pretty constant), partitions
to mount and their location(s), cron jobs to setup.

All of these are lower level classes which are almost the same except
for a few parameters. I also started with non-parameterised classes so
created a large number of nearly duplicate classes and now I''m
beginning to parameterise some of these the total number of classes
should drop much more easily. Of course if I can lookup the parameters
from a "data source" (extlookup or similar) then I can pretty much
make many classes identical: all they do is pickup the required
parameters and apply them.
> 
> 
> > > Moreover, I disagree with several of the opinions and conclusions
in
> > > your post:
> >
> > > 1) You write: "The extlookup() functionality only allows
[...
> > > specifying implicitly ...] where to look for this value."
 That is
> > > false.  Extlookup does provide for configuration of a standard
set of
> > > CSV files to search (which can be parameterized by
nodes'' facts), but
> > > the function also has an optional parameter to specify a specific
file
> > > to be searched first on any given invocation.
> >
> > Perhaps coming from a database background, I''d like to mirror
what
> > seems more _natural_ and having values spread around over potentially
> > a large number of files seems non-intuitive.
> 
> 
> If extlookup use would indeed require you to maintain a large number
> of separate files then that might be a good reason to find something
> better, but in all likelihood you can avoid that, or else structure it
> in a sane way.  Consider also:
> 
> When you work with a database, do you normally focus on how it maps
> data to the host filesystem?
Focus no. Again as mentioned I try to simplify the structure by
"normalising" it.  Here I see I want to lookup a parameter ( say
"partition size" ) from somewhere and I need to find it for a server
called $hostname. If I can''t find a parameter for $hostname, I may be
happy if I find the same parameter for $domain, or if not I may be
happy with some default value. _My_ preference is to look that up in
one place. Also as mentioned in a different message, doing this lookup
via a regexp might nicely enable me to keep the list of entries short.
> Given the diverse data parameterization you described, if you created
> a database for your configuration data, would you really organize
> everything into a single table?  (And what would be its key?)
yes:

CREATE TABLE lookup_table ( 
        config_item VARCHAR(200) NOT NULL,
        lookup_value VARCHAR(200) NOT NULL,
        return_value VARCHAR(200) NOT NULL
        PRIMARY KEY ( config_item, lookup_value )
)

SELECT return_value FROM lookup_table WHERE config_item =
''lvsize'' AND lookup_value = ''myhostname001''

provides a fast lookup of the value. If that fails you can do

SELECT return_value FROM lookup_table WHERE config_item =
''lvsize'' AND lookup_value = ''example.com''

for a more generic response. If that fails you can do:

SELECT return_value FROM lookup_table WHERE config_item =
''lvsize'' AND lookup_value = ''DEFAULT''

Yes, I''m fully aware this could be normalised better but given the
limited number of entries it''s trivial to understand, trivial to
maintain fast to access. The 3 SELECTs could be optimised into a
_single_ more complex SELECT adding a priority value to the lookup
values and a LIMIT 1.

A db server holding a trivial table like this can probably serve 
thousands of queries a second without breaking into a sweat.
> If you wanted to allow customization of your data with varying scopes
> (e.g. supporting both per-node and per-network customization of the
> same parameters) then how would you design the DB?
See above.  In fact I''ve given the "typical example" of
ordering by
$hostname, $domain, ''DEFAULT'' but one of the reasons I wanted
to have
the flexibilty in providing the input values was so that I can choose
whatever "lookup values" I want.

So I could do something like:

$mount_point = lookup_by_key( ''mount_point'', $hostname,   
''config_file'', '''' ) # no default
$vgname      = lookup_by_key( ''vgname''      
"$hostname,$domain", ''config_file'',
''vg00'' ) # first lookup $hostname, then $domain
$lvname      = lookup_by_key( ''lvname'',      $hostname,   
''config_file'', ''logical_volume_00'' )
$lvsize      = lookup_by_key( ''lvsize'',      $mount_point,
''config_file'', '''' ) # no default
$fstype      = lookup_by_key( ''fstype'',      $mount_point,
''config_file'', '''' )
$owner       = lookup_by_key( ''owner'',       $mount_point,
''config_file'', '''' )
$group       = lookup_by_key( ''group'',       $mount_point,
''config_file'', '''' )

Now I have all the parameters I need to:
1. create a directory
2. create a logical volume of the required size in the volume group
3. mkfs the filesystem on the logical volume
4. mount it on the directory
5. change the owner, group, and mode as required

All the configs fit nicely in the same config file.
> > > 2) You would prefer looking up data via a compound key
> > > (config_item, lookup_value) rather than via a simple key
> > > (config_item).  
> 
> > > [...]  And even a DB performs multi-key queries
> > > slower than single-key searches.
> >
> > Not if they are part of the primary key. That''s part of the
point.
> 
> 
> Well that''s a possible answer for you then.  Extlookup performs
> lookups based on a single key, and nothing prevents you from using
> keys that allow you to flatten your data.  For example, you can
> structure your data so that instead of extlookup("interface"),
you can
> extlookup("interface-$subnet-$is_master").  In other words, the
> distinction in your examples between "config_item" and
> "lookup_value(s)" is artificial and unnecessary.  There is no
reason
> why you could not combine them all into a single field in the CSV.
That''s true and to be honest something I hadn''t considered
before.
It certainly works but feels a big ugly to me.

...
> > As written it''s clear if you do
extlookup(''dns_server'') that you are
> > getting a "dns server" result. In many of the examples
provided
> > the expected use case is: check first for the host, then the domain,
> > then ....., but that''s not specific, _and_ may be important.
> 
> When you say it''s not "specific," do you mean that you
can''t tell from
> the extlookup() call itself?  The typical use of extlookup() involves
> defining the lookup precedence once, globally.
Indeed and that is what doesn''t feel natural in the way I expect to
use extlookup(). Perhaps that''s the problem. I was expecting this to
be a general lookup() function but while it is it assumes a certain
constant usage which I wasn''t planing on following :-(

Just to summarise your comments have helped me think out a little bit
more what it is I want and what I''m trying to achieve. I''m
beginning
to think that in the end extlookup() isn''t quite what I''m
looking
for. I was pleasantly suprised to find the code is actually only a few
lines and it _looks_ reasonably easy to adapt it to give me a
lookup_by_key() function with the behaviour I''m more comfortable with.

So thanks for taking the time to explain. I now need to take a look at
the Ruby code for extlookup.rb and adjust it to my needs and see if
that works as wanted.

Simon

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

rip@devco.net ("R.I.Pienaar") writes:
> ----- Original Message -----
> > John.Bollinger@stJude.org (jcbollinger) writes:
> > 
> > Perhaps. To some extend my non-parameterised classes are _very_
similar
> > in many ways except for various parameters (creation of logical
volumes
> > and filesystems, version of mysql to use, creation of certain cron
jobs
> > for importing informtation into a configuration database, and other
crons
> > for doing standard jobs, etc....) Most of the "completely
common" stuff
> > has been taken out. However, using extlookup() doesn''t really
work for
> > me "right now" for a simple reason I haven''t yet
mentioned. I''d further
> > like to have a lookup_by_key() function [let''s say similar to
the extlookup()
> > one provided now] but also a lookup_by_regexp() function where I can
match
> > groups of boxes, in my case mainly based on regexps of their hostnames
> > rather than on specific $hostname, or $domain, etc.... Without that
> > I need to add more entries to match by lots of hostnames, which
> > doesn''t work.
> 
> I think you''re really trying to just avoid thinking in the puppet
way instead
> trying to force some previous ideas on how things should work on it rather
> than figure out how it works.
> 
> For example, you could just define a fact on your machines that indicate
what
> role they have - perhaps based on your regex hostnames - and then use that
in
> extlookup to select csv file to use.  That would achieve your goal and it
would
> be the puppet way of solving it.
Currently I build nodes.pp from an external source and that provides
the node with it''s basic class (and hence role). So that''s not
needed.

One of the irritating things I see when we discuss puppet is the
inevitable "I can''t describe my complete problem" as
it''s
[confidential] my site specific, so you can''t have a full
understanding of what I''m doing or why I''m doing it wrong, or
provide
me a precise, clear explanation of how better to solve my original
problem.

We talk in generics and that means we never fully understand each
other.  I guess the reasons are obvious but it''s probably frustrating
to us all.

Nevertheless this thread I started has helped me clarify several
things so I appreciate the time you''ve spent discussing them with me.

Simon

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

----- Original Message -----
> rip@devco.net ("R.I.Pienaar") writes:
> 
> > ----- Original Message -----
> > > John.Bollinger@stJude.org (jcbollinger) writes:
> > > 
> > > Perhaps. To some extend my non-parameterised classes are _very_
> > > similar
> > > in many ways except for various parameters (creation of logical
> > > volumes
> > > and filesystems, version of mysql to use, creation of certain
> > > cron jobs
> > > for importing informtation into a configuration database, and
> > > other crons
> > > for doing standard jobs, etc....) Most of the "completely
common"
> > > stuff
> > > has been taken out. However, using extlookup() doesn''t
really
> > > work for
> > > me "right now" for a simple reason I haven''t
yet mentioned. I''d
> > > further
> > > like to have a lookup_by_key() function [let''s say
similar to the
> > > extlookup()
> > > one provided now] but also a lookup_by_regexp() function where I
> > > can match
> > > groups of boxes, in my case mainly based on regexps of their
> > > hostnames
> > > rather than on specific $hostname, or $domain, etc.... Without
> > > that
> > > I need to add more entries to match by lots of hostnames, which
> > > doesn''t work.
> > 
> > I think you''re really trying to just avoid thinking in the
puppet
> > way instead
> > trying to force some previous ideas on how things should work on it
> > rather
> > than figure out how it works.
> > 
> > For example, you could just define a fact on your machines that
> > indicate what
> > role they have - perhaps based on your regex hostnames - and then
> > use that in
> > extlookup to select csv file to use.  That would achieve your goal
> > and it would
> > be the puppet way of solving it.
> 
> Currently I build nodes.pp from an external source and that provides
> the node with it''s basic class (and hence role). So
that''s not
> needed.
so if you just set a variable in the node scope you can use those in 
extlookup as I described - much simpler than writing an entire new type
of function.

And combine this with the optional paramter to extlookup that let you 
specify a search-first file and you get additional key/dimension you
wish.

Still, just write the functions you need - sounds like you know exactly
what you want and its quite specific so just impliment it :)

If its good and all, you can share it on forge.puppetlabs.com

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.