In a posting a few days ago was this hiera.yaml source listing:
---------------------------------
:backends: - json
:hierarchy: - %{fqdn}
- %{lmax_role}_role
- %{lmax_env}_server
- %{pop}.tradefair
- common
:json:
:datadir: - /etc/puppet/private/
- /etc/puppet/environments/%{environment}/hiera_data_store/
- /etc/puppet/environments/%{environment}/rebirth_data_store/
- /etc/puppet/environments/%{environment}/satellite_system_groups/
---------------------------------
I am curious how one might utilize this configuration.
Can the same key value appear in more than one datadir ?
Or does one use the multiple datadir''s to group keys in some fashion ?
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Luke Bigum
2012-May-18 08:24 UTC
Re: [Puppet Users] Hiera Questions: An array of :datadir: ?
On 18/05/12 04:04, Dan White wrote:> In a posting a few days ago was this hiera.yaml source listing: > --------------------------------- > :backends: - json > :hierarchy: - %{fqdn} > - %{lmax_role}_role > - %{lmax_env}_server > - %{pop}.tradefair > - common > :json: > :datadir: - /etc/puppet/private/ > - /etc/puppet/environments/%{environment}/hiera_data_store/ > - /etc/puppet/environments/%{environment}/rebirth_data_store/ > - /etc/puppet/environments/%{environment}/satellite_system_groups/ > --------------------------------- > > I am curious how one might utilize this configuration.Hi Dan, That''s my own terrible modifications to Hiera to support multiple data stores of the same type. Code is here, but I warn you, I hacked in the quickest way to get the functionality I wanted. I only modify hiera-json, would be simple to do any others though: https://github.com/lukebigum/hiera https://github.com/lukebigum/hiera-json As for use cases, a more elegant description is in this feature request: http://projects.puppetlabs.com/issues/13954> Can the same key value appear in more than one datadir ?Yes, but precedence is taken just like Hiera''s natural hierarchy, so data stores at the start of the array have higher priority. In my case for example, "hiera_data_store" overrides "satellite_system_groups".> Or does one use the multiple datadir''s to group keys in some fashion ? >That was the the original idea of the hack - I had data coming from several places - some JSON written by hand, some generated by scripts (the ''satellite_system_groups'' store is a dump of our old Red Hat Satellite System Groups and ''rebirth_data_store'' is procedurally generated). I could have modified the generating scripts to read any existing JSON and write out a merged file, but I was rushing and I didn''t think of that at the time ;-) It has managed to keep a nice logical separation of types of Hiera keys though and you can put your data stores in different locations, like having a ''private'' dir for password hashes that''s not under revision control. This leads to giving our Development team a Git repo that''s the lowest priority data store and allow them to specify their own keys to configure their Dev servers. Any Admin still has the ability to overwrite any key in an data store above it, so it stops them doing anything silly like changing passwords, LDAP servers, etc, as these keys will already be set by us (Admins). -Luke -- Luke Bigum Information Systems Ph: +44 (0) 20 3192 2520 luke.bigum@lmax.com | http://www.lmax.com LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved. The information in this email is not directed at residents of the United States of America or any other jurisdiction where trading in CFDs and/or FX is restricted or prohibited by local laws or regulations. The information in this email and any attachment is confidential and is intended only for the named recipient(s). The email may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not the intended recipient please notify the sender immediately and delete any copies of this message. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. LMAX operates a multilateral trading facility. Authorised and regulated by the Financial Services Authority (firm registration number 509778) and is registered in England and Wales (number 06505809). Our registered address is Yellow Building, 1A Nicholas Road, London, W11 4AN. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Dan White
2012-May-18 13:00 UTC
Re: [Puppet Users] Hiera Questions: An array of :datadir: ?
----- Luke Bigum <Luke.Bigum@lmax.com> wrote:> On 18/05/12 04:04, Dan White wrote: > > In a posting a few days ago was this hiera.yaml source listing: > > --------------------------------- > > :backends: - json > > :hierarchy: - %{fqdn} > > - %{lmax_role}_role > > - %{lmax_env}_server > > - %{pop}.tradefair > > - common > > :json: > > :datadir: - /etc/puppet/private/ > > - /etc/puppet/environments/%{environment}/hiera_data_store/ > > - /etc/puppet/environments/%{environment}/rebirth_data_store/ > > - /etc/puppet/environments/%{environment}/satellite_system_groups/ > > --------------------------------- > > > > I am curious how one might utilize this configuration. > > Hi Dan, > > That''s my own terrible modifications to Hiera to support multiple data > stores of the same type. Code is here, but I warn you, I hacked in the > quickest way to get the functionality I wanted. I only modify > hiera-json, would be simple to do any others though: > > https://github.com/lukebigum/hiera > https://github.com/lukebigum/hiera-json > > As for use cases, a more elegant description is in this feature request: > http://projects.puppetlabs.com/issues/13954 > > > Can the same key value appear in more than one datadir ? > > Yes, but precedence is taken just like Hiera''s natural hierarchy, so > data stores at the start of the array have higher priority. In my case > for example, "hiera_data_store" overrides "satellite_system_groups". > > Or does one use the multiple datadir''s to group keys in some fashion ? > > > > That was the the original idea of the hack - I had data coming from > several places - some JSON written by hand, some generated by scripts > (the ''satellite_system_groups'' store is a dump of our old Red Hat > Satellite System Groups and ''rebirth_data_store'' is procedurally > generated). I could have modified the generating scripts to read any > existing JSON and write out a merged file, but I was rushing and I > didn''t think of that at the time ;-) > > It has managed to keep a nice logical separation of types of Hiera keys > though and you can put your data stores in different locations, like > having a ''private'' dir for password hashes that''s not under revision > control. This leads to giving our Development team a Git repo that''s the > lowest priority data store and allow them to specify their own keys to > configure their Dev servers. Any Admin still has the ability to > overwrite any key in an data store above it, so it stops them doing > anything silly like changing passwords, LDAP servers, etc, as these keys > will already be set by us (Admins). > > -LukeThanks for responding, Luke. This looks like a useful expansion of the hiera back-end, but as I am still fairly new to Puppet (and Python and Ruby for that matter), I think I will wait for this to be formally accepted and incorporated before trying it myself. One parting question: Is the :hierarchy: repeated in each mamber of the :datadir: ? Logically, it should repeat, but I''d rather ask and be certain. “Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.” Bill Waterson (Calvin & Hobbes) -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Luke Bigum
2012-May-21 08:16 UTC
Re: [Puppet Users] Hiera Questions: An array of :datadir: ?
On 18/05/12 14:00, Dan White wrote:> Thanks for responding, Luke. > > This looks like a useful expansion of the hiera back-end, but as I am still fairly new to Puppet (and Python and Ruby for that matter), I think I will wait for this to be formally accepted and incorporated before trying it myself. > > One parting question: Is the :hierarchy: repeated in each mamber of the :datadir: ? Logically, it should repeat, but I''d rather ask and be certain. > > “Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.” > Bill Waterson (Calvin& Hobbes) >Hi Dan, Last week I was going to reply with "Sure does, here''s an example debug..." and I released that my code didn''t work they way I thought it did, I had the inner and outer loops around the wrong way ;-) So after much more butchering of RIP''s code I can happily report that the :hierarchy: is searched as the outer loop, the :datadir: Array as the inner loop. This means that first the %{fqdn} files are searched in each member of :datadir:, then if nothing is found it moves on to ''%{lmax_env}_role'' and so on down the hierarchy. A (working!) example: [root@gs2puppet01 ~]# /opt/ruby-enterprise/bin/hiera -c /etc/puppet/hiera.yaml -y /var/lib/puppet/yaml/facts/test.example.com.yaml -a collective --debug DEBUG: Mon May 21 07:59:39 +0000 2012: Hiera JSON backend starting DEBUG: Mon May 21 07:59:39 +0000 2012: Looking up collective in JSON backend DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source test.example.com DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an Array, multiple data dirs to search DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/private/test.example.com.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile /etc/puppet/environments/production/hiera_data_store/test.example.com.json DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/environments/production/rebirth_data_store/test.example.com.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/environments/production/satellite_system_groups/test.example.com.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from /etc/puppet/environments/production/hiera_data_store/test.example.com.json DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source _role DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an Array, multiple data dirs to search DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/private/_role.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/environments/production/hiera_data_store/_role.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/environments/production/rebirth_data_store/_role.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/environments/production/satellite_system_groups/_role.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source _server DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an Array, multiple data dirs to search DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/private/_server.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/environments/production/hiera_data_store/_server.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/environments/production/rebirth_data_store/_server.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/environments/production/satellite_system_groups/_server.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source example.com DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an Array, multiple data dirs to search DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile /etc/puppet/private/example.com.json DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile /etc/puppet/environments/production/hiera_data_store/example.com.json DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/environments/production/rebirth_data_store/example.com.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/environments/production/satellite_system_groups/example.com.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from /etc/puppet/private/example.com.json DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from /etc/puppet/environments/production/hiera_data_store/example.com.json DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source common DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an Array, multiple data dirs to search DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/private/common.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile /etc/puppet/environments/production/hiera_data_store/common.json DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile /etc/puppet/environments/production/rebirth_data_store/common.json, skipping DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile /etc/puppet/environments/production/satellite_system_groups/common.json DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from /etc/puppet/environments/production/hiera_data_store/common.json DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from /etc/puppet/environments/production/satellite_system_groups/common.json DEBUG: Mon May 21 07:59:39 +0000 2012: Found key collective in file /etc/puppet/environments/production/satellite_system_groups/common.json, appending to Array DEBUG: Mon May 21 07:59:39 +0000 2012: Lookup found answer '''' [""] -- Luke Bigum Information Systems Ph: +44 (0) 20 3192 2520 luke.bigum@lmax.com | http://www.lmax.com LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved. The information in this email is not directed at residents of the United States of America or any other jurisdiction where trading in CFDs and/or FX is restricted or prohibited by local laws or regulations. The information in this email and any attachment is confidential and is intended only for the named recipient(s). The email may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not the intended recipient please notify the sender immediately and delete any copies of this message. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. LMAX operates a multilateral trading facility. Authorised and regulated by the Financial Services Authority (firm registration number 509778) and is registered in England and Wales (number 06505809). Our registered address is Yellow Building, 1A Nicholas Road, London, W11 4AN. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.