Puppet users - Jun 2013 - Accessing other parts of the Hiera tree

This seems to come up for me a lot.  As an example, my Hiera data
includes both hypervisors and the VMs that they contain.  It would
be very useful to have the VMs say "I''m a VM on host X" and,
in
templates in host X, be able to say "What are all my VMs?".

Doing this via storeconfigs rather defeats the purpose of my Hiera
tree, which is to be the single source of truth.

The way I''m currently doing it is to have a script that collates all
my Hiera host data into a special YAML file, and then have templates
read that YAML.

I''m wondering if there''s a better/more correct/more idiomatic
way to
engage in this sort of referential linking across the Hiera tree?

-Robin

-- 
http://intelligence.org/ :  Our last, best hope for a fantastic future.
.i ko na cpedu lo nu stidi vau loi jbopre .i danfu lu na go''i
li''u .e
lu go''i li''u .i ji''a go''i lu na''e
go''i li''u .e lu go''i na''i li''u .e
lu no''e go''i li''u .e lu to''e go''i
li''u .e lu lo mamta be do cu sofybakni li''u

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Monday, June 17, 2013 12:28:18 AM UTC-5, Robin Powell
wrote:
>
> This seems to come up for me a lot.  As an example, my Hiera data 
> includes both hypervisors and the VMs that they contain.  It would 
> be very useful to have the VMs say "I''m a VM on host X"
and, in
> templates in host X, be able to say "What are all my VMs?". 
>

It is inherently non-idiomatic for your templates (or manifests) to pose 
such a question, or really any question.  Rather than asking a question, 
the templates may need to say "my VMs are [...]".  I know I''m
being
nit-picky with the wording, but people really need to get a firm grip on 
the concept in order to master Puppet.

 
>
> Doing this via storeconfigs rather defeats the purpose of my Hiera 
> tree, which is to be the single source of truth. 
>

I think you''re focusing on the wrong thing when you say that.  To the 
extent that it''s applicable to the problem, storeconfigs is simply the 
infrastructure that makes exported resources work.  It is in no way an 
independent source of truth, because everything in it comes from 
declarations made in one manifest or another, based on node facts and 
possibly on data from some other source.  Put another way, in its 
application to problems of this kind, storeconfigs serves merely as an 
asynchronous communication channel.

Focusing on exported resources, then, these are simply a mechanism for 
making declarations during the compilation of one node''s catalog that
can
later be incorporated by reference into any node''s catalog.  Such 
declarations draw on whatever source(s) of truth you choose, and no others, 
just like any other declarations your manifests make.  Collecting exported 
resources later does not introduce any new sources of truth.
 
>
> The way I''m currently doing it is to have a script that collates
all
> my Hiera host data into a special YAML file, and then have templates 
> read that YAML. 
>
>
So you in fact *don''t* have a single source of truth currently.  Not
any
more so than you would have with exported resources, at least.  Your 
collation script and special YAML file simply take the place of 
master-mediated resource export and the storeconfigs database, 
respectively.  Perhaps you have other reasons to want to avoid 
storeconfigs, but otherwise I cannot imagine why you think what you''ve
come
up with is superior to using exported resources.

 
> I''m wondering if there''s a better/more correct/more
idiomatic way to
> engage in this sort of referential linking across the Hiera tree? 
>
>
It depends in part on whether you intend to extend your 
single-source-of-truth requirement to include DRY requirements on your 
hiera data.  One possibility would be to provide indices in your hiera data 
-- for example, for each VM host, an array of the names of its VMs. 
That''s
not DRY, however, if it duplicates information in VMs'' individual data 
files.

If you want all needed data stored in hiera, with no redundancy and no use 
of intermediate collation, then all the data need to be provided to (at 
least) all the associated nodes; that is, to the VM host and all its VMs.  
They all need to get all the data, else there is no escaping a need for 
some kind of data duplication.

You might find it convenient and logical to structure it as one large, 
complex, nested value, from which the individual components would select 
the pieces they need.  For example, a hash with VM hostnames as keys, and 
hashes of VM names to VM parameter hashes as values (i.e. a trebly-nested 
hash).  Alternatively, there are ways you could spread out the data over 
multiple Hiera keys, but the overall logical structure is going to be 
similar.

It should be obvious at this point is that with this kind of data 
structure, you are not going to be able to use automated class parameter 
binding to assist in declaring most of the various interrelated bits and 
pieces.

Given those choices, I would probably opt for exported resources, myself.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

[snipped heavily, hopefully nothing important]

On Mon, Jun 17, 2013 at 09:48:48AM -0700, jcbollinger
wrote:
> 
> On Monday, June 17, 2013 12:28:18 AM UTC-5, Robin Powell wrote:
> >
> > Doing this via storeconfigs rather defeats the purpose of my
> > Hiera tree, which is to be the single source of truth. 
> >
> 
> 
> I think you''re focusing on the wrong thing when you say that.  To
> the extent that it''s applicable to the problem, storeconfigs is
> simply the infrastructure that makes exported resources work.  It
> is in no way an independent source of truth, because everything in
> it comes from declarations made in one manifest or another, based
> on node facts and possibly on data from some other source.
I didn''t see the point in getting into this before, but you raise
valid issues, so:

That would be true, if facts automatically expired when not renewed,
but they don''t.  I''ve had *no end* of trouble with lies
sticking
around because they were in storeconfigs.  I don''t mean to propose
that they *should* auto-expire, as that doesn''t fit the model at
all, but it''s still a very real problem.
> You might find it convenient and logical to structure it as one
> large, complex, nested value, from which the individual components
> would select the pieces they need.  For example, a hash with VM
> hostnames as keys, and hashes of VM names to VM parameter hashes
> as values (i.e. a trebly-nested hash).  Alternatively, there are
> ways you could spread out the data over multiple Hiera keys, but
> the overall logical structure is going to be similar.
*nodnod*  OK, that seems sane, thanks.
> It should be obvious at this point is that with this kind of data
> structure, you are not going to be able to use automated class
> parameter binding to assist in declaring most of the various
> interrelated bits and pieces.
Enh; I''ve already gotten around that with my users setup by using
create_resources(...)
> Given those choices, I would probably opt for exported resources,
> myself.
It''s been so bad for me.  ;_;  After the 4th time of "Why
won''t
nagios stop alerting on this host I turned off and removed from all
my configs!?! ... Oh.", I got really sick of it.  The *main goal*
for me in using Hiera is to never use storeconfigs ever again.

-Robin

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

> > You might find it convenient and logical to structure it as one
> > large, complex, nested value, from which the individual
> > components would select the pieces they need.  For example, a
> > hash with VM hostnames as keys, and hashes of VM names to VM
> > parameter hashes as values (i.e. a trebly-nested hash).
> > Alternatively, there are ways you could spread out the data over
> > multiple Hiera keys, but the overall logical structure is going
> > to be similar.
> 
> *nodnod*  OK, that seems sane, thanks.
Thanks again for the push there; it really helped.

FWIW, what I''ve got is actually a hybrid, combining a structured
giant hiera data tree with hierarchical class overrides; this seems
useful so I''ll show y''all.

Here''s a snippet of the hosts structure, which lives in hosts.yaml:

- ----------------

    hosts:
      dev01.[snip]:
        # type, environment, and site are all used to *find* what we
        # inherit from in hiera, and hence cannot themselves be inherited,
        # or at least not in a reasonable fashion
        fqdn: dev01.[snip]
        type: cytoweb
        type_version: 3
        environment: dev
        site: dev01
        ipv6: false
        ipv4: true
        ipv4_ip: [snip]
        openvzid: 6222
        openvzhost: ds518
      qa01.[snip]:
        # type, environment, and site are all used to *find* what we
        # inherit from in hiera, and hence cannot themselves be inherited,
        # or at least not in a reasonable fashion
        fqdn: qa01.[snip]
        type: cytoweb
        type_version: 2
        environment: qa
        site: qa01
        ipv6: false
        ipv4: true
        ipv4_ip: [snip]
        openvzid: 8589
        openvzhost: ds988

- ----------------

Here''s the relevant bit of site.pp; *all* other puppet decisions are
hiera driven:

- -------------

$hosts = hiera_hash(''hosts'')
$host_type = $hosts[$fqdn][''type'']
$host_type_version = $hosts[$fqdn][''type_version'']

# Make sure our type can be loaded
$type_test = hiera(''type_loaded'')
notify{ "sphtf": message => "site.pp: host type file:
hiera/types/${::host_type}_v$host_type_version" }

$host_environment = $hosts[$fqdn][''environment'']
$host_site = $hosts[$fqdn][''site'']

node default {
  notify{ "spht": message => "site.pp: host type:
$::host_type" }
  notify{ "sphtv": message => "site.pp: host type version:
$::host_type_version" }

  hiera_include(''classes'')
}

- -------------

So we grab some specific bits out of the host structure for the
current host, but then, and this is the fun part, we *turn them back
into hiera selectors*; this is my hiera.yaml:

- -------------

---
:backends:
  - yaml
:yaml:
  :datadir: /opt/puppet3/etc/hiera
:hierarchy:
  - node/%{::fqdn}
  - types/%{::host_type}_v%{::host_type_version}
  - site/%{::host_site}
  - environments/%{::host_environment}
  - os/%{::operatingsystem}
  - osfamily/%{::osfamily}
  - hosts
  - sites
  - users
  - common
# options are native, deep, deeper
:merge_behavior: deeper

- -------------

So this means that qa01.[snip] can have behaviour driven by
information in the global hosts structure, which can be accessed by
everybody for global things like generating an /etc/hosts file,
*and* it can have hiera/node/qa01.[snip] , which can have class
overrides like:

sudo::dev_sudo: true

or whatever.

Note in addition that with "deeper" merging setup, if you use
hiera_hash as I''ve done, you can actually override bits of the
structure.

We don''t use this for the hosts structure, but we *do* use it for
users:

  create_resources( users::user, hiera_hash(''users::users'') )

is how that structure gets used, and individual hosts can have
things like:

    ---
    users::users:
      stu:
        ensure: present
        until: ''Mon Jun 24 12:59:12 PDT 2013''
      andrew:
        ensure: present
        until: ''Mon Jun 24 12:59:12 PDT 2013''

in their node/hostname.[snip] files, which will override those parts
of the users::user structure, in which those users default to
"enable: false".  users::user, like hosts, is a big giant nested
hash.

Hopefully this all is of use to someone.

-Robin

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.