Puppet users - Sep 2012 - Puppet 2.7, hiera 1.0 and hiera as an ENC

This is the situation I have:

All my hosts are the* same OS.*
All my host are in the* same puppet environment,* so I cannot use 
%{environment}

I have a module that sets all the *basic* functionality for the OS, 
resolution, authentication, security, packages, etc
I have a module for each application hosted.

At the moment all the ''data'' is in Puppet, mostly in
parametrised classes
in site.pp.

What I want to get is a hiera set-up that allows me to use this structure:

:hierarchy:
  - global     # source of application names (classes? modules?) and 
environments list
  - basic     # data for the basic class
  - prod/%{application}/%{hostname}        # hostname files for specific 
data
  - prod/%{application}/%{env}                 # environmental data for 
each application (module)
  - prod/%{application}/default                 # default data for an 
application
  - nonprod/%{sysclass}/%{hostname}
  - nonprod/%{sysclass}/%{env}
  - nonprod/%{sysclass}/default             
  

Then to have something like this under the datadir:


#├── global.yaml
#├── basic.yaml
#├── nonprod
#│   ├── app1
#│   │   ├── common-integration.yaml           << Alfresco common 
integration
#│   │   ├── continuous-integration.yaml       
#│   │   ├── dev.yaml
#│   │   ├── default.yaml
#│   │   ├── host1.yaml                       
#│   │   ├── host2.yaml
#│   │   ├── performance.yaml
#│   │   ├── qa.yaml
#│   │   ├── test.yaml
#│   │   └── uat.yaml
#│   └── app2
#└── prod
#    ├── app1
#    └── app2
# 
# etc.

In global.yaml

---
:classes:
  basic:
  app1:
  app2:
  app3:
  app4:
:env:
  test:
  dev:
  commonint:
  continuousint:
  dev:
  performance:
  qa:
  test:
  uat:


in app1 default.yaml:

---
classes:
  app1:

app1_version:  ''latest''



in app1 dev.yaml:
---
app1_version:  ''3.0''

If I wanted host1 and host2 to be part of dev for app1:


host1.yaml:
---
classes:
  basic:
  app1:
env:
  dev:

maybe in host2 I want to override version too:

host2.yaml
---
classes:
  basic:
  app1:
env:
  dev:
app1_version: ''3.1''


So in short, I would like hiera to be a source of facts, where I can get 
information that feeds Puppet in order to classify the nodes and to feed 
the parametrised classes.

I recently found this blog entry:

http://garyhetzel.com/2012/04/12/hiera_as_a_puppet_enc

Gary has been very helpful and I have got an idea of what needs doing. I 
can query all the data the way I want using the hiera command. Something 
like these:

hiera app1_version sysclass=app1 env=dev

Returns the expected ''3.0'' and if I query by adding teh host:

hiera app1_version sysclass=app1 env=dev hostname=host1

I get 3.1. Cool!


Example using Gary''s approach:

/opt/puppet-data/nonprod/hieratest/default.yaml 
---
classes: hieratest
env: hieratest_default

/opt/puppet-data/nonprod/hieratest/host1.yaml 
---
classes: 
  hieratest:
env: ''hieratest_performance''


# hiera env sysclass=hieratest --debug
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Hiera YAML backend starting
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking up type in YAML backend
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source global
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source basic
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source 
nonprod/hieratest/default
DEBUG: Wed Sep 26 16:40:46 +0100 2012: Found env in 
nonprod/hieratest/default
hieratest_default

# hiera type sysclass=hieratest hostname=host1 --debug
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Hiera YAML backend starting
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking up type in YAML backend
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source global
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source basic
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source 
nonprod/hieratest/host1
DEBUG: Wed Sep 26 16:40:57 +0100 2012: Found env in nonprod/hieratest/host1
hieratest_performance


But when it comes to use this in Puppet the results are not as I expect, 
nothing happens, it just does a run no classes are used. I see that the 
basic class custom facts are loaded, but nothing gets executed, as if the 
catalogue for host1 would not include it.


In Puppet  I expect to just have:

in site.pp:
node default {}

And then in each application’s init.pp:

$env = hiera("env")  << this allows me to get the right config 
files (with
are maintained in a git repo)
$app1_version = hiera("app1_version") << this allows me to set
the right
RPM version (from satellite/spacewalk/RHN)

As per Gary''s post, I can use hiera as node terminus, and so it is set
in
puppet.conf.

I would like to make emphasis  in this: Gary''s hiera as an ENC works,
but
for a more simple scenario than the one I am proposing, if I only wanted to 
classify Classes and Hosts, it does work fine. Where I have not been able 
to succeed is in adding an ''env'' layer after the application
(classes,
organised in modules).

I can get to organise things solely by host-name too (I could have a huge 
list of hundreds of hosts and manage that), but what I need is to be able 
to split them into directories as these are managed by each team 
responsible for the applications. 

I a a beginner in ruby, and I am willing to help as much as I can to get 
the use of hiera as an ENC as much as possible.

Do I have the right approach given the requirements? 
Is there a better approach to this one?


Kind regards,

Guillem

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/kFpoqN97hncJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Guillem, 

I''m a little unsure of what you are trying to do with hiera. I think
you''re saying you want hiera to provide facts that you can use to then
do lookups in hiera. I think this is a little bit of a chicken and egg style
issue.

I worked with Gary Hetzel on creating this ENC originally. I''d be happy
to help you understand how to work his ENC, in case he has not been very
available.

Would you mind trying to explain what you''re trying to achieve in more
detail, I''d be happy to help!

Thanks,

-Tom Linkin

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Tom,

Thanks for you reply. I have already shared this with Gary, but i think it 
is worth to make it here too.

Gary  says that ''sysclass'' and ''classes'' are
facts he sets, so this
explains why in my case they are being ignored. In any case, what I really 
want is to control that from hiera completely.

As I understand it, the node in-director hiera.rb that is then later used 
as a node_terminus = hiera in teh puppet.conf, does a preliminary lookup in 
hiera to discover hostnames, classes, etc?

Is that correct? 
If not, how can I make it happen?

I am going to paste part of teh email I wrote to Gary:


_____

 I understand the facts are retrieved  from the hosts. I do that to query 
states of applications, sudoers, versions of certain software. Including 
''tags'' in the system you want to manage is always something
tempting, but
you would have to add that information in the first place.  

The question is:  Where do I say that host1 should subscribe to classes 
''basic'' and ''app1'' and to environment
''dev''?

Do you set that in a plain text file in the same host1?

What I want to do is to store that information in hiera:

host1.yaml
---
classes:
  - basic
  - app1
env: dev

param1: valuea
param2: valueb

By looking at your hiera.rb, you put together the facts from facter and add 
the ones you discover form hiera, is that correct?

What do I need to read to start making changes in Gary''s hiera.rb so I
can
classify nodes in hiera, and not from external facts?


What I have at the moment is:

site.pp   which contains things like:
stage {"basic": before => Stage[main] }      # this lets me make
sure that
basic is applied before any other
node default {}  # this is to let ENCs as an entry point, as I understand it
node ''host1.domain.com''  { class {basic: stage => basic}
class { app1: env
=> "dev"}}   # here we pass env as a parameter, we could pass other
parameters too.

this allows me to have structures to control what files get delivered such 
as:

Production and development are Puppet file service locations:

[production]
...
path /opt/puppet-prod-files         # under version control, owned by 
operations

[development]
...
path /opt/puppet-prod-files       # under version control, owned by 
development


file { ''/path/to/file/filename'':
        source => [  
"puppet:///production/app1/$env/filename-$hostname",

                            
"puppet:///production/app1/$env/filename-$env",
                             
"puppet:///development/app1/$env/filename-$hostname",
                             
"puppet:///development/app1/$env/filename-$env",
                             
"puppet:///development/app1/$env/filename",
                          ],
      }

Form top to bottom granularity, giving production precedence. This allows a 
directory tree that is under a couple of git repos, development is RW for 
developers of each app and production is owned by operations.

This works well, but now developers want to control the RPM version that 
gets installed to each environment (and maybe control it by host) and also 
choose what hosts are members of each environment. One option would be to 
give them access to site.pp or use includes. But that means giving them too 
much and the possibility that it breaks often. Also I think it is a good 
idea to separate code and data.

My final goal is something like this:

Puppet manifest:

 
$env = hiera("env")
$app1_version = hiera("app1_version")


app1/host1.yaml    # granular settings for hosts
---
classes:
    - basic
    - app1
env: dev
app1_version: 3.1    # for env override

app1/dev.yaml      # settings for env
---
app1_version = 3.0

There is nothing that stops me using this as a combination with hiera. I 
can declare nodes like that and assign classes and parameters, and do the 
rest with hiera. I crossed my main where to store the classification 
information, with tag files in the hosts or centrally. Both have 
advantages, but then a bigger reason came:

I am required to give development groups control of the environment in this 
fashion:

By manipulating a plain text files, stored in version control,  developers 
should be able to manipulate:

- environment    (hope to do with hiera)
- application versions   (hope to do with hiera too)
- make changes to files   (I already do by setting the env manually)
- decide configuration files granularity   (the source array takes care of 
that)

Bottom line: do you think it would be possible to retrieve
''env'' from hiera
and populate it as a fact before it compiles the catalogue? If hosts are 
looked up first in the hierarchy, can it use it to fall to the next  node 
in the hierarchy depending on that?


hostname = host1

and hierarchy:

:hierarchy:
  - basic
  - prod/%{application}/%{hostaname}
  - prod/%{application}/%{env}
  - prod/%{application}/default
  - nonprod/%{application}/%{hostaname}
  - nonprod/%{application}/%{env}
  - nonprod/%{application}/default

*Chicken and egg problem*: how do I tell what %{application} is, if we are 
not there yet?


Puppet discoveres facts on:
hostname = host1

Start.
Read   data/prod/app1/*                            << it does NOT  finds 
matching hostname so it continues
Read   data/nonprod/app1/host1.yaml     << finds matching hostname and 
gets   classes and env

Read  data/{nonprod,prod}/app{N}/host{N}.yaml   <<< no more host1.yaml 
files are to be found, no more data gathered

At this point hiera should know:  classes = basic, app1   env = dev

Read data/nonprod/app1/dev.yaml             <<< finds matching env, so
it
gathers environmental settings if not overridden by host. app1_version = 3.0
Read data/{nonprod,prod}/app{N}/{ENV}.yaml   <<  no more data gathered, as
there are no more matches.

hostname: host1
classes: basic, app1
env: dev
app1_version:  3.0  

END

It seems that the problem basically is that with hiera, the classification 
would happen AFTER the facts have been gathered, and the functions are 
called to retrieve values. 

I understand Gary''s  node indirector hiera.rb  should get those lookups
done before the catalogue is compiled, giving then the desired effect.

What I need help with is:

- How do I shuffle my hierarchy so the lookups are done in the correct 
order? 
- Do I have to modify the node indirector created by you to give the extra 
flexibility I need. 
- And if so, where do I start? 



Any ideas?


On Thursday, 27 September 2012 00:58:11 UTC+1, Thomas Linkin
wrote:
>
>  Guillem, 
>
> I''m a little unsure of what you are trying to do with hiera. I
think
> you''re saying you want hiera to provide facts that you can use to
then do
> lookups in hiera. I think this is a little bit of a chicken and egg style 
> issue. 
>
> I worked with Gary Hetzel on creating this ENC originally. I''d be
happy to
> help you understand how to work his ENC, in case he has not been very 
> available.
>
> Would you mind trying to explain what you''re trying to achieve in
more
> detail, I''d be happy to help!
>
> Thanks,
>
> -Tom Linkin
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/6qRv60YeMMkJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

All,

Do I understand I have hit a dead-end? 

On Wednesday, 26 September 2012 16:52:19 UTC+1, Guillem Liarte
wrote:
>
> This is the situation I have:
>
> All my hosts are the* same OS.*
> All my host are in the* same puppet environment,* so I cannot use 
> %{environment}
>
> I have a module that sets all the *basic* functionality for the OS, 
> resolution, authentication, security, packages, etc
> I have a module for each application hosted.
>
> At the moment all the ''data'' is in Puppet, mostly in
parametrised classes
> in site.pp.
>
> What I want to get is a hiera set-up that allows me to use this structure:
>
> :hierarchy:
>   - global     # source of application names (classes? modules?) and 
> environments list
>   - basic     # data for the basic class
>   - prod/%{application}/%{hostname}        # hostname files for specific 
> data
>   - prod/%{application}/%{env}                 # environmental data for 
> each application (module)
>   - prod/%{application}/default                 # default data for an 
> application
>   - nonprod/%{sysclass}/%{hostname}
>   - nonprod/%{sysclass}/%{env}
>   - nonprod/%{sysclass}/default             
>   
>
> Then to have something like this under the datadir:
>
>
> #├── global.yaml
> #├── basic.yaml
> #├── nonprod
> #│   ├── app1
> #│   │   ├── common-integration.yaml           << Alfresco common 
> integration
> #│   │   ├── continuous-integration.yaml       
> #│   │   ├── dev.yaml
> #│   │   ├── default.yaml
> #│   │   ├── host1.yaml                       
> #│   │   ├── host2.yaml
> #│   │   ├── performance.yaml
> #│   │   ├── qa.yaml
> #│   │   ├── test.yaml
> #│   │   └── uat.yaml
> #│   └── app2
> #└── prod
> #    ├── app1
> #    └── app2
> # 
> # etc.
>
> In global.yaml
>
> ---
> :classes:
>   basic:
>   app1:
>   app2:
>   app3:
>   app4:
> :env:
>   test:
>   dev:
>   commonint:
>   continuousint:
>   dev:
>   performance:
>   qa:
>   test:
>   uat:
>
>
> in app1 default.yaml:
>
> ---
> classes:
>   app1:
>
> app1_version:  ''latest''
>
>
>
> in app1 dev.yaml:
> ---
> app1_version:  ''3.0''
>
> If I wanted host1 and host2 to be part of dev for app1:
>
>
> host1.yaml:
> ---
> classes:
>   basic:
>   app1:
> env:
>   dev:
>
> maybe in host2 I want to override version too:
>
> host2.yaml
> ---
> classes:
>   basic:
>   app1:
> env:
>   dev:
> app1_version: ''3.1''
>
>
> So in short, I would like hiera to be a source of facts, where I can get 
> information that feeds Puppet in order to classify the nodes and to feed 
> the parametrised classes.
>
> I recently found this blog entry:
>
> http://garyhetzel.com/2012/04/12/hiera_as_a_puppet_enc
>
> Gary has been very helpful and I have got an idea of what needs doing. I 
> can query all the data the way I want using the hiera command. Something 
> like these:
>
> hiera app1_version sysclass=app1 env=dev
>
> Returns the expected ''3.0'' and if I query by adding teh
host:
>
> hiera app1_version sysclass=app1 env=dev hostname=host1
>
> I get 3.1. Cool!
>
>
> Example using Gary''s approach:
>
> /opt/puppet-data/nonprod/hieratest/default.yaml 
> ---
> classes: hieratest
> env: hieratest_default
>
> /opt/puppet-data/nonprod/hieratest/host1.yaml 
> ---
> classes: 
>   hieratest:
> env: ''hieratest_performance''
>
>
> # hiera env sysclass=hieratest --debug
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Hiera YAML backend starting
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking up type in YAML backend
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source global
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source basic
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source 
> nonprod/hieratest/default
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Found env in 
> nonprod/hieratest/default
> hieratest_default
>
> # hiera type sysclass=hieratest hostname=host1 --debug
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Hiera YAML backend starting
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking up type in YAML backend
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source global
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source basic
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source 
> nonprod/hieratest/host1
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Found env in nonprod/hieratest/host1
> hieratest_performance
>
>
> But when it comes to use this in Puppet the results are not as I expect, 
> nothing happens, it just does a run no classes are used. I see that the 
> basic class custom facts are loaded, but nothing gets executed, as if the 
> catalogue for host1 would not include it.
>
>
> In Puppet  I expect to just have:
>
> in site.pp:
> node default {}
>
> And then in each application’s init.pp:
>
> $env = hiera("env")  << this allows me to get the right
config  files
> (with are maintained in a git repo)
> $app1_version = hiera("app1_version") << this allows me to
set the right
> RPM version (from satellite/spacewalk/RHN)
>
> As per Gary''s post, I can use hiera as node terminus, and so it is
set in
> puppet.conf.
>
> I would like to make emphasis  in this: Gary''s hiera as an ENC
works, but
> for a more simple scenario than the one I am proposing, if I only wanted to
> classify Classes and Hosts, it does work fine. Where I have not been able 
> to succeed is in adding an ''env'' layer after the
application (classes,
> organised in modules).
>
> I can get to organise things solely by host-name too (I could have a huge 
> list of hundreds of hosts and manage that), but what I need is to be able 
> to split them into directories as these are managed by each team 
> responsible for the applications. 
>
> I a a beginner in ruby, and I am willing to help as much as I can to get 
> the use of hiera as an ENC as much as possible.
>
> Do I have the right approach given the requirements? 
> Is there a better approach to this one?
>
>
> Kind regards,
>
> Guillem
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/jKRheSEbgXIJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wednesday, September 26, 2012 10:52:19 AM UTC-5, Guillem Liarte
wrote:
>
> This is the situation I have:
>
> All my hosts are the* same OS.*
> All my host are in the* same puppet environment,* so I cannot use 
> %{environment}
>
> I have a module that sets all the *basic* functionality for the OS, 
> resolution, authentication, security, packages, etc
> I have a module for each application hosted.
>
> At the moment all the ''data'' is in Puppet, mostly in
parametrised classes
> in site.pp.
>
> What I want to get is a hiera set-up that allows me to use this structure:
>

I suspect that one of the reasons you are having trouble is that you are 
trying to employ a usage paradigm that is inconsistent with hiera''s
design
(more below).
 
>
> :hierarchy:
>   - global     # source of application names (classes? modules?) and 
> environments list
>   - basic     # data for the basic class
>
There''s nothing wrong with those levels.
 
>   - prod/%{application}/%{hostname}        # hostname files for specific 
> data
>   - prod/%{application}/%{env}                 # environmental data for 
> each application (module)
>   - prod/%{application}/default                 # default data for an 
> application
>

But there *is* a problem with those.  It may be possible to make it work, 
but it''s shaky to use variable hierarchy levels for data *selection*.  
That''s what keys are for.  With that said, recent Puppet releases
provide
automatic $calling_module and $calling_class variables, one of which you 
could probably use in place of $application.  As I understand it,
that''s
intended to provide (better) support for module-specific data, which might 
be a good way to cast that part of your problem.

 
>   - nonprod/%{sysclass}/%{hostname}
>   - nonprod/%{sysclass}/%{env}
>   - nonprod/%{sysclass}/default
>

You additionally have a fundamental problem with %{env}.  Hiera will 
attempt to resolve that as a *Puppet* variable, to which the presence of a 
matching key somewhere in the Hiera hierarchy is irrelevant.  Hiera needs 
to know the value to resolve the hierarchy (as you have defined it), and it 
would need, in principle, to resolve the hierarchy before it could look up 
the value in your data store.

What actually happens, I''m sure, is that hiera uses the value of $::env
that it looks up in Puppet at function entry.  You might be able to work 
around that by setting that variable in Puppet before looking up other 
data, such as by putting

$env = hiera(''env'')

at top scope near the beginning of your site.pp.


> So in short, I would like hiera to be a source of facts, where I can get 
> information that feeds Puppet in order to classify the nodes and to feed 
> the parametrised classes.
>

As an aside, throwing parametrized classes into this mix has only downside 
as far as I am concerned, except inasmuch as you may want to use 
parametrized classes that are (unwisely) provided by modules written by 
others.  Since you want to rely on hiera (which is good), it is superior to 
write your classes like this wherever you are in control of module 
interfaces:

class mymodule::class1 {
  $param1 = hiera(''mymodule::class1::param1'')
  $param2 = hiera(''mymodule::class1::param2'')
  # or with simpler keys enabled by use of
  # %{calling_module} and/or %{calling_class}
}

There are several advantages, among them that you can encode interclass 
parse-order dependencies via the built-in ''include'' function,
and that you
can use hiera''s ''hiera_include()'' function to assign
such classes to nodes.
 
>
> I recently found this blog entry:
>
> http://garyhetzel.com/2012/04/12/hiera_as_a_puppet_enc
>
>
Gary appears to have done some cool work there, but as you have discovered, 
it''s not going to overcome the inherent problem with self-referrential 
data.  It might be possible to work around this by augmenting Gary''s
hiera
additions/modifications with a separate pre-lookup of needed extra 
variables, but you''re then talking about a distinctly non-trivial
effort
and a substantial branch away from stock hiera.
 

I would like to make emphasis  in this: Gary''s hiera as an ENC works,
but 
> for a more simple scenario than the one I am proposing, if I only wanted to
> classify Classes and Hosts, it does work fine. Where I have not been able 
> to succeed is in adding an ''env'' layer after the
application (classes,
> organised in modules).
>

You are classifying based only on hostname, because that''s the only
data
you want to consider that actually originates from the node being 
classified.  Everything else is logic and structure of the ENC you are 
trying to build.

The problem is that you are trying to implement a data structure that Hiera 
does not natively support.  You can make it work, but you will need either 
significant changes in hiera, or a different usage mode.  I would suggest 
the latter.

Were I you, I would consider writing a separate, hiera-based ENC instead of 
trying to build all the ENC features you want directly into hiera itself.  
Among other things, you would not then need to worry about maintaining a 
private hiera fork.  It looks like the basic idea would need to involve the 
ENC making multiple (at least two) lookups in hiera, using possibly 
different subsets of the overall data hierarchy for your site, to retrieve 
the variables to and classes (optionally with parameters) that should be 
assigned.  Anywhere the assigned classes perform their own hiera lookups, 
they would use the full hierarchy.

Although Gary''s work doesn''t do everything you want, it would
probably
serve as a good guide to much of what is needed.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/IT2oIrddP-QJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Guillem, 

Sorry for the delayed response. Anyway, John is correct about what
you''re trying to do with Hiera. I can say as far as the ENC Gary has
written follows all the rules of what an ENC ''should do'' as
per the documentation.

http://docs.puppetlabs.com/guides/external_nodes.html

Keep in mind also, when the ENC is run, the only information it has access to is
the Facts from the node. The manifests are not compiled until after the ENC
returns the classes it has determined should be declared. This ENC does it in
one call to hiera for classes. When this Hiera ENC processes, it never adds the
discovered parameters and other variables to the current running context. While
that could achieve what you want, it could also complicate things in unexpected
ways. So as I said above, when the ENC runs, you only have just the facts from
the host as your current context.

John''s suggestion of a new usage model that aligns better with
Hiera''s design is probably the best answer. That being a case, I would
suggest custom facts to help you navigate your hiera tree in a more
controlled/granular manner.

-- 
Tom Linkin
Professional Services Engineer
http://puppetlabs.com/
twitter: @trlinkin


On Monday, October 1, 2012 at 10:47 AM, jcbollinger wrote:
> 
> 
> On Wednesday, September 26, 2012 10:52:19 AM UTC-5, Guillem Liarte wrote:
> > This is the situation I have:
> > 
> > All my hosts are the same OS.
> > All my host are in the same puppet environment, so I cannot use
%{environment}
> > 
> > I have a module that sets all the basic functionality for the OS,
resolution, authentication, security, packages, etc
> > I have a module for each application hosted.
> > 
> > At the moment all the ''data'' is in Puppet, mostly in
parametrised classes in site.pp.
> > 
> > What I want to get is a hiera set-up that allows me to use this
structure:
> 
> 
> I suspect that one of the reasons you are having trouble is that you are
trying to employ a usage paradigm that is inconsistent with hiera''s
design (more below).
>  
> > 
> > :hierarchy:
> >   - global     # source of application names (classes? modules?) and
environments list
> >   - basic     # data for the basic class
> 
> There''s nothing wrong with those levels.
>  
> >   - prod/%{application}/%{hostname}        # hostname files for
specific data
> >   - prod/%{application}/%{env}                 # environmental data
for each application (module)
> >   - prod/%{application}/default                 # default data for an
application
> 
> 
> But there is a problem with those.  It may be possible to make it work, but
it''s shaky to use variable hierarchy levels for data selection. 
That''s what keys are for.  With that said, recent Puppet releases
provide automatic $calling_module and $calling_class variables, one of which you
could probably use in place of $application.  As I understand it,
that''s intended to provide (better) support for module-specific data,
which might be a good way to cast that part of your problem.
> 
>  
> >   - nonprod/%{sysclass}/%{hostname}
> >   - nonprod/%{sysclass}/%{env}
> >   - nonprod/%{sysclass}/default
> 
> 
> You additionally have a fundamental problem with %{env}.  Hiera will
attempt to resolve that as a Puppet variable, to which the presence of a
matching key somewhere in the Hiera hierarchy is irrelevant.  Hiera needs to
know the value to resolve the hierarchy (as you have defined it), and it would
need, in principle, to resolve the hierarchy before it could look up the value
in your data store.
> 
> What actually happens, I''m sure, is that hiera uses the value of
$::env that it looks up in Puppet at function entry.  You might be able to work
around that by setting that variable in Puppet before looking up other data,
such as by putting
> 
> $env = hiera(''env'')
> 
> at top scope near the beginning of your site.pp.
> 
> 
> > 
> > So in short, I would like hiera to be a source of facts, where I can
get information that feeds Puppet in order to classify the nodes and to feed the
parametrised classes.
> 
> 
> As an aside, throwing parametrized classes into this mix has only downside
as far as I am concerned, except inasmuch as you may want to use parametrized
classes that are (unwisely) provided by modules written by others.  Since you
want to rely on hiera (which is good), it is superior to write your classes like
this wherever you are in control of module interfaces:
> 
> class mymodule::class1 {
>   $param1 = hiera(''mymodule::class1::param1'')
>   $param2 = hiera(''mymodule::class1::param2'')
>   # or with simpler keys enabled by use of
>   # %{calling_module} and/or %{calling_class}
> }
> 
> There are several advantages, among them that you can encode interclass
parse-order dependencies via the built-in ''include'' function,
and that you can use hiera''s ''hiera_include()''
function to assign such classes to nodes.
>  
> > 
> > I recently found this blog entry:
> > 
> > http://garyhetzel.com/2012/04/12/hiera_as_a_puppet_enc
> > 
> 
> 
> Gary appears to have done some cool work there, but as you have discovered,
it''s not going to overcome the inherent problem with self-referrential
data.  It might be possible to work around this by augmenting Gary''s
hiera additions/modifications with a separate pre-lookup of needed extra
variables, but you''re then talking about a distinctly non-trivial
effort and a substantial branch away from stock hiera.
>  
> 
> > I would like to make emphasis  in this: Gary''s hiera as an
ENC works, but for a more simple scenario than the one I am proposing, if I only
wanted to classify Classes and Hosts, it does work fine. Where I have not been
able to succeed is in adding an ''env'' layer after the
application (classes, organised in modules).
> 
> 
> You are classifying based only on hostname, because that''s the
only data you want to consider that actually originates from the node being
classified.  Everything else is logic and structure of the ENC you are trying to
build.
> 
> The problem is that you are trying to implement a data structure that Hiera
does not natively support.  You can make it work, but you will need either
significant changes in hiera, or a different usage mode.  I would suggest the
latter.
> 
> Were I you, I would consider writing a separate, hiera-based ENC instead of
trying to build all the ENC features you want directly into hiera itself.  Among
other things, you would not then need to worry about maintaining a private hiera
fork.  It looks like the basic idea would need to involve the ENC making
multiple (at least two) lookups in hiera, using possibly different subsets of
the overall data hierarchy for your site, to retrieve the variables to and
classes (optionally with parameters) that should be assigned.  Anywhere the
assigned classes perform their own hiera lookups, they would use the full
hierarchy.
> 
> Although Gary''s work doesn''t do everything you want, it
would probably serve as a good guide to much of what is needed.
> 
> 
> John
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/IT2oIrddP-QJ.
> To post to this group, send email to puppet-users@googlegroups.com
(mailto:puppet-users@googlegroups.com).
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
(mailto:puppet-users+unsubscribe@googlegroups.com).
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

John,

Many thanks for your long reply, it is much appreciated. You have assured 
me in teh conclusions and course of action and i am glad that my 
appreciation of teh problem I have was not so far from the reality. 

Thanks again.

Guillem 

On Wednesday, 26 September 2012 16:52:19 UTC+1, Guillem Liarte
wrote:
>
> This is the situation I have:
>
> All my hosts are the* same OS.*
> All my host are in the* same puppet environment,* so I cannot use 
> %{environment}
>
> I have a module that sets all the *basic* functionality for the OS, 
> resolution, authentication, security, packages, etc
> I have a module for each application hosted.
>
> At the moment all the ''data'' is in Puppet, mostly in
parametrised classes
> in site.pp.
>
> What I want to get is a hiera set-up that allows me to use this structure:
>
> :hierarchy:
>   - global     # source of application names (classes? modules?) and 
> environments list
>   - basic     # data for the basic class
>   - prod/%{application}/%{hostname}        # hostname files for specific 
> data
>   - prod/%{application}/%{env}                 # environmental data for 
> each application (module)
>   - prod/%{application}/default                 # default data for an 
> application
>   - nonprod/%{sysclass}/%{hostname}
>   - nonprod/%{sysclass}/%{env}
>   - nonprod/%{sysclass}/default             
>   
>
> Then to have something like this under the datadir:
>
>
> #├── global.yaml
> #├── basic.yaml
> #├── nonprod
> #│   ├── app1
> #│   │   ├── common-integration.yaml           << Alfresco common 
> integration
> #│   │   ├── continuous-integration.yaml       
> #│   │   ├── dev.yaml
> #│   │   ├── default.yaml
> #│   │   ├── host1.yaml                       
> #│   │   ├── host2.yaml
> #│   │   ├── performance.yaml
> #│   │   ├── qa.yaml
> #│   │   ├── test.yaml
> #│   │   └── uat.yaml
> #│   └── app2
> #└── prod
> #    ├── app1
> #    └── app2
> # 
> # etc.
>
> In global.yaml
>
> ---
> :classes:
>   basic:
>   app1:
>   app2:
>   app3:
>   app4:
> :env:
>   test:
>   dev:
>   commonint:
>   continuousint:
>   dev:
>   performance:
>   qa:
>   test:
>   uat:
>
>
> in app1 default.yaml:
>
> ---
> classes:
>   app1:
>
> app1_version:  ''latest''
>
>
>
> in app1 dev.yaml:
> ---
> app1_version:  ''3.0''
>
> If I wanted host1 and host2 to be part of dev for app1:
>
>
> host1.yaml:
> ---
> classes:
>   basic:
>   app1:
> env:
>   dev:
>
> maybe in host2 I want to override version too:
>
> host2.yaml
> ---
> classes:
>   basic:
>   app1:
> env:
>   dev:
> app1_version: ''3.1''
>
>
> So in short, I would like hiera to be a source of facts, where I can get 
> information that feeds Puppet in order to classify the nodes and to feed 
> the parametrised classes.
>
> I recently found this blog entry:
>
> http://garyhetzel.com/2012/04/12/hiera_as_a_puppet_enc
>
> Gary has been very helpful and I have got an idea of what needs doing. I 
> can query all the data the way I want using the hiera command. Something 
> like these:
>
> hiera app1_version sysclass=app1 env=dev
>
> Returns the expected ''3.0'' and if I query by adding teh
host:
>
> hiera app1_version sysclass=app1 env=dev hostname=host1
>
> I get 3.1. Cool!
>
>
> Example using Gary''s approach:
>
> /opt/puppet-data/nonprod/hieratest/default.yaml 
> ---
> classes: hieratest
> env: hieratest_default
>
> /opt/puppet-data/nonprod/hieratest/host1.yaml 
> ---
> classes: 
>   hieratest:
> env: ''hieratest_performance''
>
>
> # hiera env sysclass=hieratest --debug
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Hiera YAML backend starting
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking up type in YAML backend
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source global
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source basic
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Looking for data source 
> nonprod/hieratest/default
> DEBUG: Wed Sep 26 16:40:46 +0100 2012: Found env in 
> nonprod/hieratest/default
> hieratest_default
>
> # hiera type sysclass=hieratest hostname=host1 --debug
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Hiera YAML backend starting
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking up type in YAML backend
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source global
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source basic
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Looking for data source 
> nonprod/hieratest/host1
> DEBUG: Wed Sep 26 16:40:57 +0100 2012: Found env in nonprod/hieratest/host1
> hieratest_performance
>
>
> But when it comes to use this in Puppet the results are not as I expect, 
> nothing happens, it just does a run no classes are used. I see that the 
> basic class custom facts are loaded, but nothing gets executed, as if the 
> catalogue for host1 would not include it.
>
>
> In Puppet  I expect to just have:
>
> in site.pp:
> node default {}
>
> And then in each application’s init.pp:
>
> $env = hiera("env")  << this allows me to get the right
config  files
> (with are maintained in a git repo)
> $app1_version = hiera("app1_version") << this allows me to
set the right
> RPM version (from satellite/spacewalk/RHN)
>
> As per Gary''s post, I can use hiera as node terminus, and so it is
set in
> puppet.conf.
>
> I would like to make emphasis  in this: Gary''s hiera as an ENC
works, but
> for a more simple scenario than the one I am proposing, if I only wanted to
> classify Classes and Hosts, it does work fine. Where I have not been able 
> to succeed is in adding an ''env'' layer after the
application (classes,
> organised in modules).
>
> I can get to organise things solely by host-name too (I could have a huge 
> list of hundreds of hosts and manage that), but what I need is to be able 
> to split them into directories as these are managed by each team 
> responsible for the applications. 
>
> I a a beginner in ruby, and I am willing to help as much as I can to get 
> the use of hiera as an ENC as much as possible.
>
> Do I have the right approach given the requirements? 
> Is there a better approach to this one?
>
>
> Kind regards,
>
> Guillem
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/QhOEBYWgOCcJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.