Puppet users - May 2012 - defaults for virtual defined resources from hiera.

Hi,
This follows on a bit from the previous thread ''trouble with hiera and 
puppet defines'' [1]

Up to now I''ve had a large file of virtual resources and then enabled
them
on demand
on various services. The very standard.

@metric{''1234:
      one => 1
      two => [1,2]
}
@metric{''abcd'':
      one => a, 
      two => [b,c]
}

and then somewhere realize potentially with an override <| title == 1234
|>
{ one => 2  }

metric is something like 
define metric ($one, $two) {
   # do stuff.
}

Attempting to move the data of virtual resources above into hiera now  and 
a few questions.

metric1234:
     one: 1
     two:
           -''1''
           -''2''

metricabcd:
     one: a
     two:
          -''b''
          -''c''

I accept  to have to use that hierarchy rather than the perhaps more obvious
metric:
   1234: ...
    abcd: ...

since hiera can only select on the first level as the key?, i.e 
hiera(metic[1234]) or something is not posible is it?

To  realize in a manifest the following works:

$h = hiera_hash(''metric4102'')
$g = { ''4102'' => $h }
create_resources(''lemon::metric'',$g)

however what I would like to do is get the defaults from hiera within the 
defined resource so it becomes 
like the following which is incorrect but hopefully explains it.

metric{4102: one=> ''override''}

where metric is defined as 

define metric (hiera("lemon${title}") {
  # do stuff.
}

I can understand why that does not make sense, the define is only going to 
evaluated once and badly.

Only option I can think of is to override the paramters inside the define 
if they are non-sensical
defaults.

define metric(one=>''UNSET'', two=>''UNSET) {

  $h = hiera("lemon${title}")
  if $one == ''UNSET'' {
      $one = $h[''one'']
  }
  if $two == ''UNSET'' {
      $two = $[''two'']
  }
}

but this is it a bit bizarre.

I am looking for a better way for a defined resource to get its defaults 
dynamically from hiera?

The motivation is that I probably only realize 20 of a potential > 1000 
resources 
on a particular host so dropping that yaml file data in a database backend 
to hiera would
make it easier.

I realize I can override with an earlier priority yaml file and hiera_hash 
but would like
the paramatized declaration to work also.

  Steve.

[1] https://groups.google.com/forum/?fromgroups#!topic/puppet-users/Ou9F3KdzqeE

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/dtpmuIJ-9h8J.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On May 14, 2:27 pm, Steve Traylen <steve.tray...@cern.ch>
wrote:
> Hi,
> This follows on a bit from the previous thread ''trouble with hiera
and
> puppet defines'' [1]
>
> Up to now I''ve had a large file of virtual resources and then
enabled them
> on demand
> on various services. The very standard.
>
> @metric{''1234:
>       one => 1
>       two => [1,2]}
>
> @metric{''abcd'':
>       one => a,
>       two => [b,c]
>
> }
>
> and then somewhere realize potentially with an override <| title == 1234
|>
> { one => 2  }

Overriding parameter values at realization time is mildly evil.  You
need at least to understand that doing this prevents you from
realizing the overridden resource in more than one place, which
otherwise is safe and sometimes useful.

> metric is something like
> define metric ($one, $two) {
>    # do stuff.
>
> }
>
> Attempting to move the data of virtual resources above into hiera now  and
> a few questions.
>
> metric1234:
>      one: 1
>      two:
>            -''1''
>            -''2''
>
> metricabcd:
>      one: a
>      two:
>           -''b''
>           -''c''
>
> I accept  to have to use that hierarchy rather than the perhaps more
obvious
> metric:
>    1234: ...
>     abcd: ...
>
> since hiera can only select on the first level as the key?, i.e
> hiera(metic[1234]) or something is not posible is it?

Not exactly, but similar should be possible.  Hiera can return hashes
as values, so you ought to be able to do something like

$metric_data = hiera(''metric'')
$metric_1234 = $metric_data[''1234'']

That sort of structure might also play well with Puppet''s
create_resources() function, which I see you already know about.

> To  realize in a manifest the following works:
>
> $h = hiera_hash(''metric4102'')
> $g = { ''4102'' => $h }
> create_resources(''lemon::metric'',$g)

Ok.  You need to know, if you don''t already, that the hiera() function
will return a hash if that''s the type of value associated with the
specified key.  Hiera_hash() also returns a hash, but its main purpose
is to collect and merge data from your whole hierarchy, instead of
from just the highest-priority level where the key is found.  I don''t
see any evidence from your example that the latter is what you''re
really after in that case.

> however what I would like to do is get the defaults from hiera within the
> defined resource so it becomes
> like the following which is incorrect but hopefully explains it.
>
> metric{4102: one=> ''override''}
>
> where metric is defined as
>
> define metric (hiera("lemon${title}") {
>   # do stuff.
>
> }
>
> I can understand why that does not make sense, the define is only going to
> evaluated once and badly.

What does not make sense is "default values" that vary with the
resource instance you are declaring.  At least, that doesn''t match up
with what Puppet means by that term.

> Only option I can think of is to override the paramters inside the define
> if they are non-sensical
> defaults.
>
> define metric(one=>''UNSET'', two=>''UNSET) {
>
>   $h = hiera("lemon${title}")
>   if $one == ''UNSET'' {
>       $one = $h[''one'']
>   }
>   if $two == ''UNSET'' {
>       $two = $[''two'']
>   }
>
> }
>
> but this is it a bit bizarre.

And it wouldn''t work as written, since you cannot redefine variables
in Puppet, but something similar could be done.  You could do
something a little cleaner and clearer with the aid of a couple of
functions.  For example,

define metric($one => ''UNSET'', $two =>
''UNSET'') {
  $passed_params = remove_value({ ''one'' => $one,
''two'' => $two },
''UNSET'')
  $standard_params = hiera("metric_${title}")
  $resolved_params = merge($standard_params, $passed_params)
  # do something with $resolved_params[''one'']
  # do something with $resolved_params[''two'']
}

There is just such a merge() function in the Puppetlabs stdlib
module.  You would have to write remove_values() yourself, but it
ought to be pretty simple.

> I am looking for a better way for a defined resource to get its defaults
> dynamically from hiera?

If the "default" values were associated with top-level keys in your
data store, then perhaps you could do something like this:

define metric($one => hiera("metric${title}_one"), $two =>
hiera("metric${title}_one")) {
   # ...
}

Again, however, part of the problem is that you are trying to use
Puppet''s default value facility in a way that was not intended.
Generally speaking, default values are defaults for the *type*, not
separate defaults for each instance.

> The motivation is that I probably only realize 20 of a potential > 1000
> resources
> on a particular host so dropping that yaml file data in a database backend
> to hiera would
> make it easier.
>
> I realize I can override with an earlier priority yaml file and hiera_hash
> but would like
> the paramatized declaration to work also.

I think you''re creating a mess for yourself, especially with parameter
overrides at realization time.  You would be better off moving all the
way to hiera.  In fact, you could leverage hiera further to pull those
1000 virtual resource declarations out of your manifests altogether.
For example:

class metric::instances {
  # Retrieve an array of the names of the metric
  # instances wanted for this node
  $metrics = hiera(''metrics'')

  # Declare all the wanted ''metric'' instances (concretely).
  # The definition relies on hiera for all needed data,
  # likely by merging data from two or more levels via
  # hiera_hash()
  metric::metric { $metrics: }
}

Isn''t that better than thousands of lines of virtual resource
declarations?  It will be a lot lighter on your puppetmaster, too.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tuesday, 15 May 2012 16:48:48 UTC+2, jcbollinger
wrote:
>
> On May 14, 2:27 pm, Steve Traylen <steve.tray...@cern.ch> wrote: 
> > Up to now I''ve had a large file of virtual resources and then
enabled
> them 
> > on demand 
> > on various services. The very standard. 
> > 
> > @metric{''1234: 
> >       one => 1 
> >       two => [1,2]} 
> > 
> > @metric{''abcd'': 
> >       one => a, 
> >       two => [b,c] 
> > } 
> > and then somewhere realize potentially with an override <| title ==
1234
> |> 
> > { one => 2  } 
>
> Overriding parameter values at realization time is mildly evil.  You 
> need at least to understand that doing this prevents you from 
> realizing the overridden resource in more than one place, which 
> otherwise is safe and sometimes useful. 
>
>
Agreed, i need to set paramaters at creation time, the parameters just may
differ in place to place at creation time sometimes.
>
> > metric is something like 
> > define metric ($one, $two) { 
> >    # do stuff. 
> > 
> > } 
> > 
> > Attempting to move the data of virtual resources above into hiera now 
>  and 
> > a few questions. 
> > 
> > metric1234: 
> >      one: 1 
> >      two: 
> >            -''1'' 
> >            -''2'' 
> > 
> > metricabcd: 
> >      one: a 
> >      two: 
> >           -''b'' 
> >           -''c'' 
> > 
> > I accept  to have to use that hierarchy rather than the perhaps more 
> obvious 
> > metric: 
> >    1234: ... 
> >     abcd: ... 
> > 
> > since hiera can only select on the first level as the key?, i.e 
> > hiera(metic[1234]) or something is not posible is it? 
>
> Not exactly, but similar should be possible.  Hiera can return hashes 
> as values, so you ought to be able to do something like 
>
> $metric_data = hiera(''metric'') 
> $metric_1234 = $metric_data[''1234''] 
>
> That sort of structure might also play well with Puppet''s 
> create_resources() function, which I see you already know about. 
>
> > To  realize in a manifest the following works: 
> > 
> > $h = hiera_hash(''metric4102'') 
> > $g = { ''4102'' => $h } 
> > create_resources(''lemon::metric'',$g) 
>
> Ok.  You need to know, if you don''t already, that the hiera()
function
> will return a hash if that''s the type of value associated with the
> specified key.  Hiera_hash() also returns a hash, but its main purpose 
> is to collect and merge data from your whole hierarchy, instead of 
> from just the highest-priority level where the key is found.  I
don''t
> see any evidence from your example that the latter is what you''re 
> really after in that case. 
>
> > however what I would like to do is get the defaults from hiera within 
> the 
> > defined resource so it becomes 
> > like the following which is incorrect but hopefully explains it. 
> > 
> > metric{4102: one=> ''override''} 
> > 
> > where metric is defined as 
> > 
> > define metric (hiera("lemon${title}") { 
> >   # do stuff. 
> > 
> > } 
> > 
> > I can understand why that does not make sense, the define is only
going
> to 
> > evaluated once and badly. 
>
> What does not make sense is "default values" that vary with the 
> resource instance you are declaring.  At least, that doesn''t match
up
> with what Puppet means by that term. 
>
>
Okay, I guess i am just using hiera as a data store because its easy to 
access that
and let it deal with the complexity.
>
> > Only option I can think of is to override the paramters inside the 
> define 
> > if they are non-sensical 
> > defaults. 
> > 
> > define metric(one=>''UNSET'',
two=>''UNSET) {
> > 
> >   $h = hiera("lemon${title}") 
> >   if $one == ''UNSET'' { 
> >       $one = $h[''one''] 
> >   } 
> >   if $two == ''UNSET'' { 
> >       $two = $[''two''] 
> >   } 
> > 
> > } 
> > 
> > but this is it a bit bizarre. 
>
> And it wouldn''t work as written, since you cannot redefine
variables
> in Puppet, but something similar could be done.  You could do 
> something a little cleaner and clearer with the aid of a couple of 
> functions.  For example, 
>
> define metric($one => ''UNSET'', $two =>
''UNSET'') {
>   $passed_params = remove_value({ ''one'' => $one,
''two'' => $two },
> ''UNSET'') 
>   $standard_params = hiera("metric_${title}") 
>   $resolved_params = merge($standard_params, $passed_params) 
>   # do something with $resolved_params[''one''] 
>   # do something with $resolved_params[''two''] 
> } 
>
> There is just such a merge() function in the Puppetlabs stdlib 
> module.  You would have to write remove_values() yourself, but it 
> ought to be pretty simple. 
>  
> I am looking for a better way for a defined resource to get its defaults 
> > dynamically from hiera? 
>
> If the "default" values were associated with top-level keys in
your
> data store, then perhaps you could do something like this: 
>
> define metric($one => hiera("metric${title}_one"), $two => 
> hiera("metric${title}_one")) { 
>    # ... 
> } 
>
> Again, however, part of the problem is that you are trying to use 
> Puppet''s default value facility in a way that was not intended. 
> Generally speaking, default values are defaults for the *type*, not 
> separate defaults for each instance. 
>
>
What I went for in end which was a merge of some of your comments was

define metric( $param => undef ) {
    $metricid = $title
    $metrichash = hiera("lemon_${metricid}")

    # Set $my_param to first passed parameter, then hiera value then 
default value.
    if ( $param != undef ) 
      $my_param = $param 
    } elsif ( has_key($metrichas,''param'') 
       $my_param = $metrichash[''hash'']
    } else {
       $my_param = undef
    }
    
   And then use $my_param in the erb template.       
 
> > The motivation is that I probably only realize 20 of a potential >
1000
> > resources 
> > on a particular host so dropping that yaml file data in a database 
> backend 
> > to hiera would 
> > make it easier. 
> > 
> > I realize I can override with an earlier priority yaml file and 
> hiera_hash 
> > but would like 
> > the paramatized declaration to work also. 
>
>
> I think you''re creating a mess for yourself, especially with
parameter
> overrides at realization time.  You would be better off moving all the 
> way to hiera.  In fact, you could leverage hiera further to pull those 
> 1000 virtual resource declarations out of your manifests altogether. 
> For example: 
>
>
I now have no virtual resources, I just load the bits of data from hiera as 
and when needed.

Thanks for the input , it helped me a lot with my understanding of the 
finer points.

I am at the moment fairly happy with the result simply because it allows 
one person to maintain
the library of metrics in YAML and other people to enable what they want. 
.. And they can still
do override of the data which is especially useful when you want to 
override based
on a fact for instance which can only be set in a manifest.

For reference: https://github.com/CERN-Ops/puppet-lemon

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/2m8so2NH2lkJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.