thr3ads.net - Puppet users - [Puppet Users] "Best" way to relate keys, or stuff multiple values into a single key in hiera? [Jun 2012]

If this information is useful, please help other people find it:
Share via:

Wolf Noble

2012-Jun-12 18:48 UTC

[Puppet Users] "Best" way to relate keys, or stuff multiple values into a single key in hiera?

So… I''m coming across a situation where I foresee myself having a
collection of related k/v pairs in hiera. I want to create a thing… lets say a
yumrepo for example''s sake, but this same idea translates to  a vhost,
a web app configuration , a virtual domain, etc.

lets assume a hierarchy of:

  - %{region}/%{dc}/%{env}/%{function}/%{fqdn}
  - %{region}/%{dc}/%{env}/%{function}/common
  - %{region}/%{dc}/%{env}/common
  - %{region}/%{dc}/common
  - %{region}/common

in the us-east region common yaml file, should I create the following two repos
values:

repo_descr: eastcoast-pub
repo_baseurl: http://myfoo.com/eastcoastgoodtimes
repo_gpgkey: http://myfoo.com/eastcoastgoodtimes/key
repo_gpgcheck: 1
repo_enabled: 1
repo_enablegroups: 0

repo_descr: eastcoast-internal
repo_baseurl: http://myfoo.com/eastcoast
repo_gpgkey: http://myfoo.com/eastcoast/key
repo_gpgcheck: 1
repo_enabled: 1
repo_enablegroups: 0


in the foo DC common file I have a repo for that datacenter:

repo_descr: foo
repo_baseurl: http://myfoo.com/goodtimes
repo_gpgkey: http://myfoo.com/goodtimes/key
repo_gpgcheck: 1
repo_enabled: 1
repo_enablegroups: 0



in the ''qa'' env common yaml file

repo_descr: fooqarepo
repo_baseurl: http://myfoo.com/qa_crispybacon/goodtimes
repo_gpgkey: http://myfoo.com/qa_crispybacon/goodtimes/key
repo_gpgcheck: 1
repo_enabled: 1
repo_enablegroups: 0



in the webservers function common yaml file, I have another repo:

repo_descr: webservers
repo_baseurl: http://myfoo.com/bacon/goodtimes
repo_gpgkey: http://myfoo.com/bacon/goodtimes/key
repo_gpgcheck: 1
repo_enabled: 1
repo_enablegroups: 0






so I want all the qa web servers in the foo DC in the east coast to have all
five of those repo files defined.. except for web02, of course, because
he''s special… looking at it this way makes my head hurt. I
don''t see a good way to actualize these repos onto a node, or exclude
them from a unique snowflake if necessary, which sort-of defeats the purpose of
hiera altogether.

Do I not create them as separate keys, but as a single ''repo''
array, and in my manifest use a template to iterate through the array and
associate the values contained within  the array with values needed for the
addition of a repo?

What have people found works "best"?





________________________________

This message may contain confidential or privileged information. If you are not
the intended recipient, please advise us immediately and delete this message.
See http://www.datapipe.com/legal/email_disclaimer/ for further information on
confidentiality and the risks of non-secure electronic communication. If you
cannot access these links, please notify us by reply message and we will send
the contents to you.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

jcbollinger

2012-Jun-12 22:18 UTC

head link

[Puppet Users] Re: "Best" way to relate keys, or stuff multiple values into a single key in hiera?

On Tuesday, June 12, 2012 1:48:09 PM UTC-5, Wolf Noble
wrote:>
> So… I''m coming across a situation where I foresee myself having a 
> collection of related k/v pairs in hiera. I want to create a thing… lets 
> say a yumrepo for example''s sake, but this same idea translates to
a
> vhost, a web app configuration , a virtual domain, etc. 
>
> lets assume a hierarchy of: 
>
>   - %{region}/%{dc}/%{env}/%{function}/%{fqdn} 
>   - %{region}/%{dc}/%{env}/%{function}/common 
>   - %{region}/%{dc}/%{env}/common 
>   - %{region}/%{dc}/common 
>   - %{region}/common 
>
> in the us-east region common yaml file, should I create the following two 
> repos values: 
>
> repo_descr: eastcoast-pub 
> repo_baseurl: http://myfoo.com/eastcoastgoodtimes 
> repo_gpgkey: http://myfoo.com/eastcoastgoodtimes/key 
> repo_gpgcheck: 1 
> repo_enabled: 1 
> repo_enablegroups: 0 
>
> repo_descr: eastcoast-internal 
> repo_baseurl: http://myfoo.com/eastcoast 
> repo_gpgkey: http://myfoo.com/eastcoast/key 
> repo_gpgcheck: 1 
> repo_enabled: 1 
> repo_enablegroups: 0 
>
>
> in the foo DC common file I have a repo for that datacenter: 
>
> repo_descr: foo 
> repo_baseurl: http://myfoo.com/goodtimes 
> repo_gpgkey: http://myfoo.com/goodtimes/key 
> repo_gpgcheck: 1 
> repo_enabled: 1 
> repo_enablegroups: 0 
>
>
>
> in the ''qa'' env common yaml file 
>
> repo_descr: fooqarepo 
> repo_baseurl: http://myfoo.com/qa_crispybacon/goodtimes 
> repo_gpgkey: http://myfoo.com/qa_crispybacon/goodtimes/key 
> repo_gpgcheck: 1 
> repo_enabled: 1 
> repo_enablegroups: 0 
>
>
>
> in the webservers function common yaml file, I have another repo: 
>
> repo_descr: webservers 
> repo_baseurl: http://myfoo.com/bacon/goodtimes 
> repo_gpgkey: http://myfoo.com/bacon/goodtimes/key 
> repo_gpgcheck: 1 
> repo_enabled: 1 
> repo_enablegroups: 0 
>
>
>
>
>
>
> so I want all the qa web servers in the foo DC in the east coast to have 
> all five of those repo files defined.. except for web02, of course, because
> he''s special… looking at it this way makes my head hurt. I
don''t see a good
> way to actualize these repos onto a node, or exclude them from a unique 
> snowflake if necessary, which sort-of defeats the purpose of hiera 
> altogether. 
>
> Do I not create them as separate keys, but as a single
''repo'' array, and
> in my manifest use a template to iterate through the array and associate 
> the values contained within  the array with values needed for the addition 
> of a repo? 
>
> What have people found works "best"? 
>
For complex data you need to put some structure in your YAML and process it 
accordingly in your manifests.  In particular, in this case it would be to 
your advantage to use hiera''s (and YAML''s) support for arrays
and / or
hashes as values.  For example:

in us-east/common.yaml:
=====region_repos:
  eastcoast_pub_repo: {
      descr: eastcoast-pub,
      baseurl: http://myfoo.com/eastcoastgoodtimes,
      gpgkey: http://myfoo.com/eastcoastgoodtimes/key,
      gpgcheck: 1,
      enabled: 1,
      enablegroups: 0
    },
  eastcoast_internal_repo: {
      descr: eastcoast-internal,
      baseurl: http://myfoo.com/eastcoast,
      gpgkey: http://myfoo.com/eastcoast/key,
      gpgcheck: 1,
      enabled: 1,
      enablegroups: 0
    }
=====
That''s a hash of hashes, associated with the key
"region_repos".  You could
use that in your manifests like so:

mymodule::repos {
    create_resources(''yumrepo'',
hiera(''region_repos''))
    # perhaps also create_resources(''yumrepo'',
hiera(''dc_repos'')), etc.
}

If that looks simple, that''s because I tailored the YAML data structure
to
the expectations of the create_resources() function.

It''s also because the structure assumes that you will explicitly load
the
data for each hierarchy level you want, instead of, for example, using 
hiera_array() or hiera_hash() to assemble an even more complex aggregate 
data structure from various levels of your hierarchy.  I think that''s 
reasonable for your use case, especially given that you anticipate certain 
machines will need exceptions.  You''re still getting indirection out of
it
because hiera is choosing the appropriate region-, dc-, and 
environment-specific data files.

There are many other ways to do it, but that one has the advantage of being 
simple (relatively speaking) and fairly easy to understand.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/Rug-iffTkLkJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Wolf Noble

2012-Jun-12 22:51 UTC

head link

Re: [Puppet Users] Re: "Best" way to relate keys, or stuff multiple values into a single key in hiera?

Hi John,

Thanks a bunch. This seems incredibly sensible.

If I''m understanding your suggestion properly, I interpret the best
thing to do as something like:


mymodule::repos {
$region_repos = hiera(''region_repos'' , '''' )
$dc_repos = hiera(''dc_repos'' , '''' )

  if $region_repos {
    create_resources(''yumrepo'', $region_repos))
  }
  if $dc_repos {
    create_resources(''yumrepo'', $dc_repos))
  }
#...
}

and then I could override them like

us-east/foo/qa/webservers/web02.qa.myfoo.com.yaml

region_repos: ""
dc_repos:
  foodc_qa_repo: {
    descr: foodc-qa,
    baseurl: http://myfoo.com/foodc-repo,
    gpgkey: http://myfoo.com/foodcrepo/key,
    gpgcheck: 1,
    enabled: 1,
    enablegroups: 0
}


with the known caveat that I''d have to explicitly redeclare the entire
hash at the most relevant tier if I wanted to change any element of the hash
already defined.


Did I miss anything?


On Jun 12, 2012, at 5:18 PM, jcbollinger wrote:
>
>
> On Tuesday, June 12, 2012 1:48:09 PM UTC-5, Wolf Noble wrote:
> So… I''m coming across a situation where I foresee myself having a
collection of related k/v pairs in hiera. I want to create a thing… lets say a
yumrepo for example''s sake, but this same idea translates to  a vhost,
a web app configuration , a virtual domain, etc.
>
> lets assume a hierarchy of:
>
>   - %{region}/%{dc}/%{env}/%{function}/%{fqdn}
>   - %{region}/%{dc}/%{env}/%{function}/common
>   - %{region}/%{dc}/%{env}/common
>   - %{region}/%{dc}/common
>   - %{region}/common
>
> in the us-east region common yaml file, should I create the following two
repos values:
>
> repo_descr: eastcoast-pub
> repo_baseurl: http://myfoo.com/eastcoastgoodtimes
> repo_gpgkey: http://myfoo.com/eastcoastgoodtimes/key
> repo_gpgcheck: 1
> repo_enabled: 1
> repo_enablegroups: 0
>
> repo_descr: eastcoast-internal
> repo_baseurl: http://myfoo.com/eastcoast
> repo_gpgkey: http://myfoo.com/eastcoast/key
> repo_gpgcheck: 1
> repo_enabled: 1
> repo_enablegroups: 0
>
>
> in the foo DC common file I have a repo for that datacenter:
>
> repo_descr: foo
> repo_baseurl: http://myfoo.com/goodtimes
> repo_gpgkey: http://myfoo.com/goodtimes/key
> repo_gpgcheck: 1
> repo_enabled: 1
> repo_enablegroups: 0
>
>
>
> in the ''qa'' env common yaml file
>
> repo_descr: fooqarepo
> repo_baseurl: http://myfoo.com/qa_crispybacon/goodtimes
> repo_gpgkey: http://myfoo.com/qa_crispybacon/goodtimes/key
> repo_gpgcheck: 1
> repo_enabled: 1
> repo_enablegroups: 0
>
>
>
> in the webservers function common yaml file, I have another repo:
>
> repo_descr: webservers
> repo_baseurl: http://myfoo.com/bacon/goodtimes
> repo_gpgkey: http://myfoo.com/bacon/goodtimes/key
> repo_gpgcheck: 1
> repo_enabled: 1
> repo_enablegroups: 0
>
>
>
>
>
>
> so I want all the qa web servers in the foo DC in the east coast to have
all five of those repo files defined.. except for web02, of course, because
he''s special… looking at it this way makes my head hurt. I
don''t see a good way to actualize these repos onto a node, or exclude
them from a unique snowflake if necessary, which sort-of defeats the purpose of
hiera altogether.
>
> Do I not create them as separate keys, but as a single
''repo'' array, and in my manifest use a template to iterate
through the array and associate the values contained within  the array with
values needed for the addition of a repo?
>
> What have people found works "best"?
>
> For complex data you need to put some structure in your YAML and process it
accordingly in your manifests.  In particular, in this case it would be to your
advantage to use hiera''s (and YAML''s) support for arrays and /
or hashes as values.  For example:
>
> in us-east/common.yaml:
> =====> region_repos:
>   eastcoast_pub_repo: {
>       descr: eastcoast-pub,
>       baseurl: http://myfoo.com/eastcoastgoodtimes,
>       gpgkey: http://myfoo.com/eastcoastgoodtimes/key,
>       gpgcheck: 1,
>       enabled: 1,
>       enablegroups: 0
>     },
>   eastcoast_internal_repo: {
>       descr: eastcoast-internal,
>       baseurl: http://myfoo.com/eastcoast,
>       gpgkey: http://myfoo.com/eastcoast/key,
>       gpgcheck: 1,
>       enabled: 1,
>       enablegroups: 0
>     }
> =====>
> That''s a hash of hashes, associated with the key
"region_repos".  You could use that in your manifests like so:
>
> mymodule::repos {
>     create_resources(''yumrepo'',
hiera(''region_repos''))
>     # perhaps also create_resources(''yumrepo'',
hiera(''dc_repos'')), etc.
> }
>
> If that looks simple, that''s because I tailored the YAML data
structure to the expectations of the create_resources() function.
>
> It''s also because the structure assumes that you will explicitly
load the data for each hierarchy level you want, instead of, for example, using
hiera_array() or hiera_hash() to assemble an even more complex aggregate data
structure from various levels of your hierarchy.  I think that''s
reasonable for your use case, especially given that you anticipate certain
machines will need exceptions.  You''re still getting indirection out of
it because hiera is choosing the appropriate region-, dc-, and
environment-specific data files.
>
> There are many other ways to do it, but that one has the advantage of being
simple (relatively speaking) and fairly easy to understand.
>
>
> John
>
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/Rug-iffTkLkJ.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

________________________________

This message may contain confidential or privileged information. If you are not
the intended recipient, please advise us immediately and delete this message.
See http://www.datapipe.com/legal/email_disclaimer/ for further information on
confidentiality and the risks of non-secure electronic communication. If you
cannot access these links, please notify us by reply message and we will send
the contents to you.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

jcbollinger

2012-Jun-13 13:24 UTC

head link

Re: [Puppet Users] Re: "Best" way to relate keys, or stuff multiple values into a single key in hiera?

On Tuesday, June 12, 2012 5:51:06 PM UTC-5, Wolf Noble
wrote:>
> Hi John, 
>
> Thanks a bunch. This seems incredibly sensible. 
>
> If I''m understanding your suggestion properly, I interpret the
best thing
> to do as something like: 
>
>
> mymodule::repos { 
> $region_repos = hiera(''region_repos'' ,
'''' )
> $dc_repos = hiera(''dc_repos'' , '''' ) 
>
>   if $region_repos { 
>     create_resources(''yumrepo'', $region_repos)) 
>   } 
>   if $dc_repos { 
>     create_resources(''yumrepo'', $dc_repos)) 
>   } 
> #... 
> } 
>
> and then I could override them like 
>
> us-east/foo/qa/webservers/web02.qa.myfoo.com.yaml 
>
> region_repos: "" 
> dc_repos: 
>   foodc_qa_repo: { 
>     descr: foodc-qa, 
>     baseurl: http://myfoo.com/foodc-repo, 
>     gpgkey: http://myfoo.com/foodcrepo/key, 
>     gpgcheck: 1, 
>     enabled: 1, 
>     enablegroups: 0 
> } 
>
>
> with the known caveat that I''d have to explicitly redeclare the
entire
> hash at the most relevant tier if I wanted to change any element of the 
> hash already defined. 
>
>
> Did I miss anything? 
>
I don''t see any obvious errors or omissions.  In particular, you are 
correct that overriding repos declared at a lower level will involve 
declaring an entirely new hash (of hashes) at the appropriate higher 
level.  That''s one of the costs attending complex data.  All in all, it
looks nicely fleshed out.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/Ts3RcypLci0J.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Wolf Noble

2012-Jun-13 15:21 UTC

head link

Re: [Puppet Users] Re: "Best" way to relate keys, or stuff multiple values into a single key in hiera?

Awesome!

I wonder about one other thing though,  Is create_resources likely to balk if
there are values in the hash that are irrelevant for the resource type being
fed?

ie: if I have a region_repos hash that has both yum values, and apt values…
(assuming that a debian sibling to the yumrepo type comes along at some point in
the future) and feed them both via the same key, is that likely going to cause
me pain? or should the key be geared more to one or the other, i.e.:

region_yumrepo:
  foo_repo: {
    ..
  }

region_aptsource
  foo_aptsource: {
    …
  }

just trying to think about where the pointy/sharp parts of this might be.


On Jun 13, 2012, at 8:24 AM, jcbollinger wrote:
>
>
> On Tuesday, June 12, 2012 5:51:06 PM UTC-5, Wolf Noble wrote:
> Hi John,
>
> Thanks a bunch. This seems incredibly sensible.
>
> If I''m understanding your suggestion properly, I interpret the
best thing to do as something like:
>
>
> mymodule::repos {
> $region_repos = hiera(''region_repos'' ,
'''' )
> $dc_repos = hiera(''dc_repos'' , '''' )
>
>   if $region_repos {
>     create_resources(''yumrepo'', $region_repos))
>   }
>   if $dc_repos {
>     create_resources(''yumrepo'', $dc_repos))
>   }
> #...
> }
>
> and then I could override them like
>
> us-east/foo/qa/webservers/web02.qa.myfoo.com.yaml
>
> region_repos: ""
> dc_repos:
>   foodc_qa_repo: {
>     descr: foodc-qa,
>     baseurl: http://myfoo.com/foodc-repo,
>     gpgkey: http://myfoo.com/foodcrepo/key,
>     gpgcheck: 1,
>     enabled: 1,
>     enablegroups: 0
> }
>
>
> with the known caveat that I''d have to explicitly redeclare the
entire hash at the most relevant tier if I wanted to change any element of the
hash already defined.
>
>
> Did I miss anything?
>
> I don''t see any obvious errors or omissions.  In particular, you
are correct that overriding repos declared at a lower level will involve
declaring an entirely new hash (of hashes) at the appropriate higher level. 
That''s one of the costs attending complex data.  All in all, it looks
nicely fleshed out.
>
>
> John
>
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/Ts3RcypLci0J.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

________________________________

This message may contain confidential or privileged information. If you are not
the intended recipient, please advise us immediately and delete this message.
See http://www.datapipe.com/legal/email_disclaimer/ for further information on
confidentiality and the risks of non-secure electronic communication. If you
cannot access these links, please notify us by reply message and we will send
the contents to you.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

jcbollinger

2012-Jun-14 13:02 UTC

head link

Re: [Puppet Users] Re: "Best" way to relate keys, or stuff multiple values into a single key in hiera?

On Wednesday, June 13, 2012 10:21:19 AM UTC-5, Wolf Noble
wrote:>
> I wonder about one other thing though,  Is create_resources likely to balk 
> if there are values in the hash that are irrelevant for the resource type 
> being fed? 
> 
I don''t know off-hand, but I wouldn''t be at all surprised if 
create_resources() fails when its input contains mappings for properties 
that the target resource type does not actually have.  If you put such 
additional data into your repository structure then you''ll probably
need to
process the result to select the desired portions before feeding it to 
create_resources().  You might need to write a custom function for that 
purpose.  Alternatively, you could associate data for different resource 
types with different hiera keys.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/sZuRX4_F_BYJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Puppet users - Jun 2012 - "Best" way to relate keys, or stuff multiple values into a single key in hiera?

[Puppet Users] "Best" way to relate keys, or stuff multiple values into a single key in hiera?

[Puppet Users] Re: "Best" way to relate keys, or stuff multiple values into a single key in hiera?

Re: [Puppet Users] Re: "Best" way to relate keys, or stuff multiple values into a single key in hiera?

Re: [Puppet Users] Re: "Best" way to relate keys, or stuff multiple values into a single key in hiera?

Re: [Puppet Users] Re: "Best" way to relate keys, or stuff multiple values into a single key in hiera?

Re: [Puppet Users] Re: "Best" way to relate keys, or stuff multiple values into a single key in hiera?