Sai_Emc
2013-Feb-21 18:55 UTC
[Puppet Users] Puppet-Hiera: hiera_include and hiera_hash behaviour
I am looking for few clarifications on puppet-hiera integration. Trying to move away completly from using site.pp. For that I started exploring Hiera. End of the day I want an external system prepare yaml files automatucally based on user requests. Environment: Ubuntu12.04, PE 2.7, Hiera 1.1.2 EX: class classA { notice("ClassA") } class classB ($param1 = '''') { notice ("ClassB: ParamValue $param1") } Now I am want to include these two classes into two different nodes, so prepared two yaml files hiera.yaml --- :hierarchy: - %{::clientcert} - common :backends: - yaml :yaml: :datadir: ''/etc/puppetlabs/puppet/hieradata'' NodeA.yaml --- emcutil::a: - a NodeB.yaml --- classesH: emcutil::b: param1: ''Puppet-Hiera'' As I have classes as array and hash, so added below code in site.pp default section so that I can include hash and arrays. node default { hiera_include(''classesA'','''') $param_packagesH = hiera_hash(''classesH'') create_resources(''class'',$param_classesH) } } This approach does not work. On NodeA following error: err: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `empty?'' for nil:NilClass at /etc/puppetlabs/puppet/environments/development/manifests/site.pp:48 on node On NodeB following error: err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item classesA in any Hiera data file and no default supplied at /etc/puppetlabs/puppet/environments/development/manifests/site.pp:46 on node What I am doing here is valid? If we are using hiera_include or hiera_hash all the node yaml files should have all the targets and the target returns not null values? Do we have alternatives other than using hiera, hiera_include and hiera_hash? PE 2.7 do we need to do any configuration for Hiera, apart from puppet-hiera package. Specifically related to these two attributes node_terminus, external_nodes? With Hiera we can move away from site.pp completly. My understanding is correct on Hiera? Any help on this greatly appreciated. Sai. EMC Corporation. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ellison Marks
2013-Feb-21 19:32 UTC
[Puppet Users] Re: Puppet-Hiera: hiera_include and hiera_hash behaviour
quick definitions first: hiera() gets a value from the backend. stops searching after it finds a value. the value can be any data type. hiera_array() gets multiple values from the backend, combining them into one large array as possible. hiera_hash() gets multiple values from the backend, combining them into one large hash as possible. hiera_include() calls hiera_array() and then includes each classname returned. Ok, so hiera_include will look in the backend for a variable named in the call, in your case ''classesA''. this should contain an array of class names to include. As your hierarchy apparently doesn''t contain the variable ''classesA'', this explains the second error you are getting. You also didn''t include the variable ''classesH'' in nodeA.yaml, so that might be the cause of the failure there. create_resources is also not used on classes, just types or defines. Generally, for hiera_include, it should look sort of like this. class classA { notice("ClassA") } class classB ($param1 = '''') { #This uses the auto lookup of parameters. It will perform a hiera search for ''classB::param1''. notice ("ClassB: ParamValue $param1") } NodeA.yaml --- classes: - a NodeB.yaml --- classes: - b ''classB::param1'': ''Puppet-Hiera'' node default { hiera_include(''classes'', []) #empty array, not empty string, which might also have been messing things up. } } On Thursday, February 21, 2013 10:55:59 AM UTC-8, Sai_Emc wrote:> > I am looking for few clarifications on puppet-hiera integration. > > Trying to move away completly from using site.pp. For that I started > exploring Hiera. End of the day I want an external system prepare yaml > files automatucally based on user requests. > > Environment: Ubuntu12.04, PE 2.7, Hiera 1.1.2 > > EX: > class classA { > notice("ClassA") > } > > class classB ($param1 = '''') { > notice ("ClassB: ParamValue $param1") > } > > > Now I am want to include these two classes into two different nodes, so > prepared two yaml files > > hiera.yaml > --- > :hierarchy: > - %{::clientcert} > - common > :backends: > - yaml > :yaml: > :datadir: ''/etc/puppetlabs/puppet/hieradata'' > > NodeA.yaml > --- > emcutil::a: > - a > > NodeB.yaml > --- > classesH: > emcutil::b: > param1: ''Puppet-Hiera'' > > > As I have classes as array and hash, so added below code in site.pp > default section so that I can include hash and arrays. > > node default { > hiera_include(''classesA'','''') > $param_packagesH = hiera_hash(''classesH'') > create_resources(''class'',$param_classesH) > } > } > > This approach does not work. > > On NodeA following error: > err: Could not retrieve catalog from remote server: Error 400 on SERVER: > undefined method `empty?'' for nil:NilClass at > /etc/puppetlabs/puppet/environments/development/manifests/site.pp:48 on node > On NodeB following error: > err: Could not retrieve catalog from remote server: Error 400 on SERVER: > Could not find data item classesA in any Hiera data file and no default > supplied at > /etc/puppetlabs/puppet/environments/development/manifests/site.pp:46 on node > > What I am doing here is valid? > If we are using hiera_include or hiera_hash all the node yaml files should > have all the targets and the target returns not null values? > Do we have alternatives other than using hiera, hiera_include and > hiera_hash? > PE 2.7 do we need to do any configuration for Hiera, apart from > puppet-hiera package. Specifically related to these two attributes > node_terminus, external_nodes? > With Hiera we can move away from site.pp completly. My understanding is > correct on Hiera? > > Any help on this greatly appreciated. > > Sai. > EMC Corporation. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Sai_Emc
2013-Feb-21 19:57 UTC
[Puppet Users] Re: Puppet-Hiera: hiera_include and hiera_hash behaviour
Mark, Thanks for the quick reply. "This uses the auto lookup of parameters. It will perform a hiera search for ''classB::param1''." --> Auto lookup feature what you mentioned is in PE 2.7? I remember reading as 3.0 feature. Regards Sai. EMC On Thursday, February 21, 2013 2:32:10 PM UTC-5, Ellison Marks wrote:> quick definitions first: > > hiera() gets a value from the backend. stops searching after it finds a > value. the value can be any data type. > hiera_array() gets multiple values from the backend, combining them into > one large array as possible. > hiera_hash() gets multiple values from the backend, combining them into > one large hash as possible. > hiera_include() calls hiera_array() and then includes each classname > returned. > > Ok, so hiera_include will look in the backend for a variable named in the > call, in your case ''classesA''. this should contain an array of class names > to include. As your hierarchy apparently doesn''t contain the variable > ''classesA'', this explains the second error you are getting. You also didn''t > include the variable ''classesH'' in nodeA.yaml, so that might be the cause > of the failure there. create_resources is also not used on classes, just > types or defines. Generally, for hiera_include, it should look sort of like > this. > > class classA { > notice("ClassA") > } > > class classB ($param1 = '''') { #This uses the auto lookup of parameters. It > will perform a hiera search for ''classB::param1''. > notice ("ClassB: ParamValue $param1") > } > > > NodeA.yaml > --- > classes: > - a > > NodeB.yaml > --- > classes: > - b > > ''classB::param1'': ''Puppet-Hiera'' > > node default { > hiera_include(''classes'', []) #empty array, not empty string, which > might also have been messing things up. > > } > } > > > On Thursday, February 21, 2013 10:55:59 AM UTC-8, Sai_Emc wrote: >> >> I am looking for few clarifications on puppet-hiera integration. >> >> Trying to move away completly from using site.pp. For that I started >> exploring Hiera. End of the day I want an external system prepare yaml >> files automatucally based on user requests. >> >> Environment: Ubuntu12.04, PE 2.7, Hiera 1.1.2 >> >> EX: >> class classA { >> notice("ClassA") >> } >> >> class classB ($param1 = '''') { >> notice ("ClassB: ParamValue $param1") >> } >> >> >> Now I am want to include these two classes into two different nodes, so >> prepared two yaml files >> >> hiera.yaml >> --- >> :hierarchy: >> - %{::clientcert} >> - common >> :backends: >> - yaml >> :yaml: >> :datadir: ''/etc/puppetlabs/puppet/hieradata'' >> >> NodeA.yaml >> --- >> emcutil::a: >> - a >> >> NodeB.yaml >> --- >> classesH: >> emcutil::b: >> param1: ''Puppet-Hiera'' >> >> >> As I have classes as array and hash, so added below code in site.pp >> default section so that I can include hash and arrays. >> >> node default { >> hiera_include(''classesA'','''') >> $param_packagesH = hiera_hash(''classesH'') >> create_resources(''class'',$param_classesH) >> } >> } >> >> This approach does not work. >> >> On NodeA following error: >> err: Could not retrieve catalog from remote server: Error 400 on SERVER: >> undefined method `empty?'' for nil:NilClass at >> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:48 on node >> On NodeB following error: >> err: Could not retrieve catalog from remote server: Error 400 on SERVER: >> Could not find data item classesA in any Hiera data file and no default >> supplied at >> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:46 on node >> >> What I am doing here is valid? >> If we are using hiera_include or hiera_hash all the node yaml files >> should have all the targets and the target returns not null values? >> Do we have alternatives other than using hiera, hiera_include and >> hiera_hash? >> PE 2.7 do we need to do any configuration for Hiera, apart from >> puppet-hiera package. Specifically related to these two attributes >> node_terminus, external_nodes? >> With Hiera we can move away from site.pp completly. My understanding is >> correct on Hiera? >> >> Any help on this greatly appreciated. >> >> Sai. >> EMC Corporation. >> >> >> >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ellison Marks
2013-Feb-21 20:01 UTC
[Puppet Users] Re: Puppet-Hiera: hiera_include and hiera_hash behaviour
Ah, sorry, my mistake, mised the version. That line would have to be class classB ($param1 = hiera(''classB::param1'', '''')) { with the hiera call in the definition. Sorry about that :P On Thursday, February 21, 2013 11:57:24 AM UTC-8, Sai_Emc wrote:> > Mark, > > Thanks for the quick reply. > > "This uses the auto lookup of parameters. It will perform a hiera search > for ''classB::param1''." --> Auto lookup feature what you mentioned is in PE > 2.7? I remember reading as 3.0 feature. > > Regards > Sai. > EMC > > On Thursday, February 21, 2013 2:32:10 PM UTC-5, Ellison Marks wrote: > >> quick definitions first: >> >> hiera() gets a value from the backend. stops searching after it finds a >> value. the value can be any data type. >> hiera_array() gets multiple values from the backend, combining them into >> one large array as possible. >> hiera_hash() gets multiple values from the backend, combining them into >> one large hash as possible. >> hiera_include() calls hiera_array() and then includes each classname >> returned. >> >> Ok, so hiera_include will look in the backend for a variable named in the >> call, in your case ''classesA''. this should contain an array of class names >> to include. As your hierarchy apparently doesn''t contain the variable >> ''classesA'', this explains the second error you are getting. You also didn''t >> include the variable ''classesH'' in nodeA.yaml, so that might be the cause >> of the failure there. create_resources is also not used on classes, just >> types or defines. Generally, for hiera_include, it should look sort of like >> this. >> >> class classA { >> notice("ClassA") >> } >> >> class classB ($param1 = '''') { #This uses the auto lookup of parameters. >> It will perform a hiera search for ''classB::param1''. >> notice ("ClassB: ParamValue $param1") >> } >> >> >> NodeA.yaml >> --- >> classes: >> - a >> >> NodeB.yaml >> --- >> classes: >> - b >> >> ''classB::param1'': ''Puppet-Hiera'' >> >> node default { >> hiera_include(''classes'', []) #empty array, not empty string, which >> might also have been messing things up. >> >> } >> } >> >> >> On Thursday, February 21, 2013 10:55:59 AM UTC-8, Sai_Emc wrote: >>> >>> I am looking for few clarifications on puppet-hiera integration. >>> >>> Trying to move away completly from using site.pp. For that I started >>> exploring Hiera. End of the day I want an external system prepare yaml >>> files automatucally based on user requests. >>> >>> Environment: Ubuntu12.04, PE 2.7, Hiera 1.1.2 >>> >>> EX: >>> class classA { >>> notice("ClassA") >>> } >>> >>> class classB ($param1 = '''') { >>> notice ("ClassB: ParamValue $param1") >>> } >>> >>> >>> Now I am want to include these two classes into two different nodes, so >>> prepared two yaml files >>> >>> hiera.yaml >>> --- >>> :hierarchy: >>> - %{::clientcert} >>> - common >>> :backends: >>> - yaml >>> :yaml: >>> :datadir: ''/etc/puppetlabs/puppet/hieradata'' >>> >>> NodeA.yaml >>> --- >>> emcutil::a: >>> - a >>> >>> NodeB.yaml >>> --- >>> classesH: >>> emcutil::b: >>> param1: ''Puppet-Hiera'' >>> >>> >>> As I have classes as array and hash, so added below code in site.pp >>> default section so that I can include hash and arrays. >>> >>> node default { >>> hiera_include(''classesA'','''') >>> $param_packagesH = hiera_hash(''classesH'') >>> create_resources(''class'',$param_classesH) >>> } >>> } >>> >>> This approach does not work. >>> >>> On NodeA following error: >>> err: Could not retrieve catalog from remote server: Error 400 on >>> SERVER: undefined method `empty?'' for nil:NilClass at >>> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:48 on node >>> On NodeB following error: >>> err: Could not retrieve catalog from remote server: Error 400 on >>> SERVER: Could not find data item classesA in any Hiera data file and no >>> default supplied at >>> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:46 on node >>> >>> What I am doing here is valid? >>> If we are using hiera_include or hiera_hash all the node yaml files >>> should have all the targets and the target returns not null values? >>> Do we have alternatives other than using hiera, hiera_include and >>> hiera_hash? >>> PE 2.7 do we need to do any configuration for Hiera, apart from >>> puppet-hiera package. Specifically related to these two attributes >>> node_terminus, external_nodes? >>> With Hiera we can move away from site.pp completly. My understanding is >>> correct on Hiera? >>> >>> Any help on this greatly appreciated. >>> >>> Sai. >>> EMC Corporation. >>> >>> >>> >>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Sai_Emc
2013-Feb-21 20:27 UTC
[Puppet Users] Re: Puppet-Hiera: hiera_include and hiera_hash behaviour
That''s ok Mark. I kind of remember that from documentation. When we use above approach puppet need to make multiple hiera() calls to get resources specific to a class. I mean a class with say 4 arguments, puppet need to make a multiple calls to get all the parameters. NodeB.yaml --- classes: - b ''classB::param1'': ''Puppet-Hiera'' ''classB::param2'': ''Puppet-Hiera'' ''classB::param3'': ''Puppet-Hiera'' ''classB::param4'': ''Puppet-Hiera'' If we have thousands of nodes we might see any performance issues with array approach? I might be totally wrong on this. The same if I have declared as Hash then from puppet that is just one call, but it might look all defined hierarchies. So if we do use hiera_hash then without using create_resources() is there any other method I can include class definitions. Thank you. Regards Sai. EMC On Thursday, February 21, 2013 3:01:54 PM UTC-5, Ellison Marks wrote:> Ah, sorry, my mistake, mised the version. That line would have to be > > class classB ($param1 = hiera(''classB::param1'', '''')) { > > with the hiera call in the definition. Sorry about that :P > > On Thursday, February 21, 2013 11:57:24 AM UTC-8, Sai_Emc wrote: >> >> Mark, >> >> Thanks for the quick reply. >> >> "This uses the auto lookup of parameters. It will perform a hiera search >> for ''classB::param1''." --> Auto lookup feature what you mentioned is in PE >> 2.7? I remember reading as 3.0 feature. >> >> Regards >> Sai. >> EMC >> >> On Thursday, February 21, 2013 2:32:10 PM UTC-5, Ellison Marks wrote: >> >>> quick definitions first: >>> >>> hiera() gets a value from the backend. stops searching after it finds a >>> value. the value can be any data type. >>> hiera_array() gets multiple values from the backend, combining them into >>> one large array as possible. >>> hiera_hash() gets multiple values from the backend, combining them into >>> one large hash as possible. >>> hiera_include() calls hiera_array() and then includes each classname >>> returned. >>> >>> Ok, so hiera_include will look in the backend for a variable named in >>> the call, in your case ''classesA''. this should contain an array of class >>> names to include. As your hierarchy apparently doesn''t contain the variable >>> ''classesA'', this explains the second error you are getting. You also didn''t >>> include the variable ''classesH'' in nodeA.yaml, so that might be the cause >>> of the failure there. create_resources is also not used on classes, just >>> types or defines. Generally, for hiera_include, it should look sort of like >>> this. >>> >>> class classA { >>> notice("ClassA") >>> } >>> >>> class classB ($param1 = '''') { #This uses the auto lookup of parameters. >>> It will perform a hiera search for ''classB::param1''. >>> notice ("ClassB: ParamValue $param1") >>> } >>> >>> >>> NodeA.yaml >>> --- >>> classes: >>> - a >>> >>> NodeB.yaml >>> --- >>> classes: >>> - b >>> >>> ''classB::param1'': ''Puppet-Hiera'' >>> >>> node default { >>> hiera_include(''classes'', []) #empty array, not empty string, which >>> might also have been messing things up. >>> >>> } >>> } >>> >>> >>> On Thursday, February 21, 2013 10:55:59 AM UTC-8, Sai_Emc wrote: >>>> >>>> I am looking for few clarifications on puppet-hiera integration. >>>> >>>> Trying to move away completly from using site.pp. For that I started >>>> exploring Hiera. End of the day I want an external system prepare yaml >>>> files automatucally based on user requests. >>>> >>>> Environment: Ubuntu12.04, PE 2.7, Hiera 1.1.2 >>>> >>>> EX: >>>> class classA { >>>> notice("ClassA") >>>> } >>>> >>>> class classB ($param1 = '''') { >>>> notice ("ClassB: ParamValue $param1") >>>> } >>>> >>>> >>>> Now I am want to include these two classes into two different nodes, so >>>> prepared two yaml files >>>> >>>> hiera.yaml >>>> --- >>>> :hierarchy: >>>> - %{::clientcert} >>>> - common >>>> :backends: >>>> - yaml >>>> :yaml: >>>> :datadir: ''/etc/puppetlabs/puppet/hieradata'' >>>> >>>> NodeA.yaml >>>> --- >>>> emcutil::a: >>>> - a >>>> >>>> NodeB.yaml >>>> --- >>>> classesH: >>>> emcutil::b: >>>> param1: ''Puppet-Hiera'' >>>> >>>> >>>> As I have classes as array and hash, so added below code in site.pp >>>> default section so that I can include hash and arrays. >>>> >>>> node default { >>>> hiera_include(''classesA'','''') >>>> $param_packagesH = hiera_hash(''classesH'') >>>> create_resources(''class'',$param_classesH) >>>> } >>>> } >>>> >>>> This approach does not work. >>>> >>>> On NodeA following error: >>>> err: Could not retrieve catalog from remote server: Error 400 on >>>> SERVER: undefined method `empty?'' for nil:NilClass at >>>> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:48 on node >>>> On NodeB following error: >>>> err: Could not retrieve catalog from remote server: Error 400 on >>>> SERVER: Could not find data item classesA in any Hiera data file and no >>>> default supplied at >>>> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:46 on node >>>> >>>> What I am doing here is valid? >>>> If we are using hiera_include or hiera_hash all the node yaml files >>>> should have all the targets and the target returns not null values? >>>> Do we have alternatives other than using hiera, hiera_include and >>>> hiera_hash? >>>> PE 2.7 do we need to do any configuration for Hiera, apart from >>>> puppet-hiera package. Specifically related to these two attributes >>>> node_terminus, external_nodes? >>>> With Hiera we can move away from site.pp completly. My understanding is >>>> correct on Hiera? >>>> >>>> Any help on this greatly appreciated. >>>> >>>> Sai. >>>> EMC Corporation. >>>> >>>> >>>> >>>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ellison Marks
2013-Feb-21 22:32 UTC
[Puppet Users] Re: Puppet-Hiera: hiera_include and hiera_hash behaviour
It''s not that bad performance wise. Hiera should only lode the applicable yaml files. If (god forbid) you had one yaml file per host and had thousands of hosts, hiera should still only load two files, the host''s yaml file and the common yaml file. The only thing that would be bad is manually maintaining all the yaml files :) As to the multiple parameters, it''s also not that bad. You could format the parameters as an array and seperate them later, like this: NodeB.yaml --- classes: - b ''classB::params'': - ''Puppet-Hiera'' - ''Puppet-Hiera'' - ''Puppet-Hiera'' - ''Puppet-Hiera'' class classB ($params = hiera(''classB::params'', '''')) { $param1=$params[0] $param2=$params[1] $param3=$params[2] $param4=$params[3] but it''s not that much better. The real trick is in using create resources correctly. You want to have a 3 depth array in hiera. top is the name you will look up with the hiera() call. Second level is the names of the resources you want to create. For example, if you were creating users, this level could have ''john'' and ''jane''. Third level is the parameters for the individual resource. In this example, below john could be {''uid''=>501, ''group''=>john} and below jane could be {''uid''=>502, ''group''=>jane} userhash: john: uid: 501 group: john jane: uid: 502 group: jane then you would say $userhash=hiera(''userhash'') create_resources(user, $userhash) So if you can structure your class in a way that supports that flow, it can be really good. On Thursday, February 21, 2013 12:27:30 PM UTC-8, Sai_Emc wrote:> > That''s ok Mark. I kind of remember that from documentation. > > When we use above approach puppet need to make multiple hiera() calls to > get resources specific to a class. > > I mean a class with say 4 arguments, puppet need to make a multiple calls > to get all the parameters. > > NodeB.yaml > --- > classes: > - b > > ''classB::param1'': ''Puppet-Hiera'' > ''classB::param2'': ''Puppet-Hiera'' > ''classB::param3'': ''Puppet-Hiera'' > ''classB::param4'': ''Puppet-Hiera'' > > If we have thousands of nodes we might see any performance issues with > array approach? I might be totally wrong on this. > > The same if I have declared as Hash then from puppet that is just one > call, but it might look all defined hierarchies. So if we do use hiera_hash > then without using create_resources() is there any other method I can > include class definitions. > > Thank you. > > Regards > Sai. > EMC > > On Thursday, February 21, 2013 3:01:54 PM UTC-5, Ellison Marks wrote: > >> Ah, sorry, my mistake, mised the version. That line would have to be >> >> class classB ($param1 = hiera(''classB::param1'', '''')) { >> >> with the hiera call in the definition. Sorry about that :P >> >> On Thursday, February 21, 2013 11:57:24 AM UTC-8, Sai_Emc wrote: >>> >>> Mark, >>> >>> Thanks for the quick reply. >>> >>> "This uses the auto lookup of parameters. It will perform a hiera search >>> for ''classB::param1''." --> Auto lookup feature what you mentioned is in PE >>> 2.7? I remember reading as 3.0 feature. >>> >>> Regards >>> Sai. >>> EMC >>> >>> On Thursday, February 21, 2013 2:32:10 PM UTC-5, Ellison Marks wrote: >>> >>>> quick definitions first: >>>> >>>> hiera() gets a value from the backend. stops searching after it finds a >>>> value. the value can be any data type. >>>> hiera_array() gets multiple values from the backend, combining them >>>> into one large array as possible. >>>> hiera_hash() gets multiple values from the backend, combining them into >>>> one large hash as possible. >>>> hiera_include() calls hiera_array() and then includes each classname >>>> returned. >>>> >>>> Ok, so hiera_include will look in the backend for a variable named in >>>> the call, in your case ''classesA''. this should contain an array of class >>>> names to include. As your hierarchy apparently doesn''t contain the variable >>>> ''classesA'', this explains the second error you are getting. You also didn''t >>>> include the variable ''classesH'' in nodeA.yaml, so that might be the cause >>>> of the failure there. create_resources is also not used on classes, just >>>> types or defines. Generally, for hiera_include, it should look sort of like >>>> this. >>>> >>>> class classA { >>>> notice("ClassA") >>>> } >>>> >>>> class classB ($param1 = '''') { #This uses the auto lookup of parameters. >>>> It will perform a hiera search for ''classB::param1''. >>>> notice ("ClassB: ParamValue $param1") >>>> } >>>> >>>> >>>> NodeA.yaml >>>> --- >>>> classes: >>>> - a >>>> >>>> NodeB.yaml >>>> --- >>>> classes: >>>> - b >>>> >>>> ''classB::param1'': ''Puppet-Hiera'' >>>> >>>> node default { >>>> hiera_include(''classes'', []) #empty array, not empty string, which >>>> might also have been messing things up. >>>> >>>> } >>>> } >>>> >>>> >>>> On Thursday, February 21, 2013 10:55:59 AM UTC-8, Sai_Emc wrote: >>>>> >>>>> I am looking for few clarifications on puppet-hiera integration. >>>>> >>>>> Trying to move away completly from using site.pp. For that I started >>>>> exploring Hiera. End of the day I want an external system prepare yaml >>>>> files automatucally based on user requests. >>>>> >>>>> Environment: Ubuntu12.04, PE 2.7, Hiera 1.1.2 >>>>> >>>>> EX: >>>>> class classA { >>>>> notice("ClassA") >>>>> } >>>>> >>>>> class classB ($param1 = '''') { >>>>> notice ("ClassB: ParamValue $param1") >>>>> } >>>>> >>>>> >>>>> Now I am want to include these two classes into two different nodes, >>>>> so prepared two yaml files >>>>> >>>>> hiera.yaml >>>>> --- >>>>> :hierarchy: >>>>> - %{::clientcert} >>>>> - common >>>>> :backends: >>>>> - yaml >>>>> :yaml: >>>>> :datadir: ''/etc/puppetlabs/puppet/hieradata'' >>>>> >>>>> NodeA.yaml >>>>> --- >>>>> emcutil::a: >>>>> - a >>>>> >>>>> NodeB.yaml >>>>> --- >>>>> classesH: >>>>> emcutil::b: >>>>> param1: ''Puppet-Hiera'' >>>>> >>>>> >>>>> As I have classes as array and hash, so added below code in site.pp >>>>> default section so that I can include hash and arrays. >>>>> >>>>> node default { >>>>> hiera_include(''classesA'','''') >>>>> $param_packagesH = hiera_hash(''classesH'') >>>>> create_resources(''class'',$param_classesH) >>>>> } >>>>> } >>>>> >>>>> This approach does not work. >>>>> >>>>> On NodeA following error: >>>>> err: Could not retrieve catalog from remote server: Error 400 on >>>>> SERVER: undefined method `empty?'' for nil:NilClass at >>>>> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:48 on node >>>>> On NodeB following error: >>>>> err: Could not retrieve catalog from remote server: Error 400 on >>>>> SERVER: Could not find data item classesA in any Hiera data file and no >>>>> default supplied at >>>>> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:46 on node >>>>> >>>>> What I am doing here is valid? >>>>> If we are using hiera_include or hiera_hash all the node yaml files >>>>> should have all the targets and the target returns not null values? >>>>> Do we have alternatives other than using hiera, hiera_include and >>>>> hiera_hash? >>>>> PE 2.7 do we need to do any configuration for Hiera, apart from >>>>> puppet-hiera package. Specifically related to these two attributes >>>>> node_terminus, external_nodes? >>>>> With Hiera we can move away from site.pp completly. My understanding >>>>> is correct on Hiera? >>>>> >>>>> Any help on this greatly appreciated. >>>>> >>>>> Sai. >>>>> EMC Corporation. >>>>> >>>>> >>>>> >>>>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Sai_Emc
2013-Feb-21 22:50 UTC
[Puppet Users] Re: Puppet-Hiera: hiera_include and hiera_hash behaviour
Wonderful...This is cool. As you said, maintaining thousands yaml files manually might be difficlut. I have to automate this process to accept requests from an external system about which node needs which resources. So I might use json over yaml, because at this point I dont know my external system can do yaml. Any idea json backend supported in 2.7?. I saw in the documentation but not sure supported in 2.7 or not. Once again thank you very much. Regards Sai. EMC On Thursday, February 21, 2013 5:32:55 PM UTC-5, Ellison Marks wrote:> It''s not that bad performance wise. Hiera should only lode the applicable > yaml files. If (god forbid) you had one yaml file per host and had > thousands of hosts, hiera should still only load two files, the host''s yaml > file and the common yaml file. The only thing that would be bad is manually > maintaining all the yaml files :) > > As to the multiple parameters, it''s also not that bad. You could format > the parameters as an array and seperate them later, like this: > > NodeB.yaml > --- > classes: > - b > > ''classB::params'': > - ''Puppet-Hiera'' > - ''Puppet-Hiera'' > - ''Puppet-Hiera'' > - ''Puppet-Hiera'' > > class classB ($params = hiera(''classB::params'', '''')) { > $param1=$params[0] > $param2=$params[1] > $param3=$params[2] > $param4=$params[3] > > but it''s not that much better. The real trick is in using create resources > correctly. You want to have a 3 depth array in hiera. top is the name you > will look up with the hiera() call. Second level is the names of the > resources you want to create. For example, if you were creating users, this > level could have ''john'' and ''jane''. Third level is the parameters for the > individual resource. In this example, below john could be {''uid''=>501, > ''group''=>john} and below jane could be {''uid''=>502, ''group''=>jane} > > userhash: > john: > uid: 501 > group: john > jane: > uid: 502 > group: jane > > then you would say > > $userhash=hiera(''userhash'') > create_resources(user, $userhash) > > So if you can structure your class in a way that supports that flow, it > can be really good. > > On Thursday, February 21, 2013 12:27:30 PM UTC-8, Sai_Emc wrote: >> >> That''s ok Mark. I kind of remember that from documentation. >> >> When we use above approach puppet need to make multiple hiera() calls to >> get resources specific to a class. >> >> I mean a class with say 4 arguments, puppet need to make a multiple calls >> to get all the parameters. >> >> NodeB.yaml >> --- >> classes: >> - b >> >> ''classB::param1'': ''Puppet-Hiera'' >> ''classB::param2'': ''Puppet-Hiera'' >> ''classB::param3'': ''Puppet-Hiera'' >> ''classB::param4'': ''Puppet-Hiera'' >> >> If we have thousands of nodes we might see any performance issues with >> array approach? I might be totally wrong on this. >> >> The same if I have declared as Hash then from puppet that is just one >> call, but it might look all defined hierarchies. So if we do use hiera_hash >> then without using create_resources() is there any other method I can >> include class definitions. >> >> Thank you. >> >> Regards >> Sai. >> EMC >> >> On Thursday, February 21, 2013 3:01:54 PM UTC-5, Ellison Marks wrote: >> >>> Ah, sorry, my mistake, mised the version. That line would have to be >>> >>> class classB ($param1 = hiera(''classB::param1'', '''')) { >>> >>> with the hiera call in the definition. Sorry about that :P >>> >>> On Thursday, February 21, 2013 11:57:24 AM UTC-8, Sai_Emc wrote: >>>> >>>> Mark, >>>> >>>> Thanks for the quick reply. >>>> >>>> "This uses the auto lookup of parameters. It will perform a hiera >>>> search for ''classB::param1''." --> Auto lookup feature what you mentioned >>>> is in PE 2.7? I remember reading as 3.0 feature. >>>> >>>> Regards >>>> Sai. >>>> EMC >>>> >>>> On Thursday, February 21, 2013 2:32:10 PM UTC-5, Ellison Marks wrote: >>>> >>>>> quick definitions first: >>>>> >>>>> hiera() gets a value from the backend. stops searching after it finds >>>>> a value. the value can be any data type. >>>>> hiera_array() gets multiple values from the backend, combining them >>>>> into one large array as possible. >>>>> hiera_hash() gets multiple values from the backend, combining them >>>>> into one large hash as possible. >>>>> hiera_include() calls hiera_array() and then includes each classname >>>>> returned. >>>>> >>>>> Ok, so hiera_include will look in the backend for a variable named in >>>>> the call, in your case ''classesA''. this should contain an array of class >>>>> names to include. As your hierarchy apparently doesn''t contain the variable >>>>> ''classesA'', this explains the second error you are getting. You also didn''t >>>>> include the variable ''classesH'' in nodeA.yaml, so that might be the cause >>>>> of the failure there. create_resources is also not used on classes, just >>>>> types or defines. Generally, for hiera_include, it should look sort of like >>>>> this. >>>>> >>>>> class classA { >>>>> notice("ClassA") >>>>> } >>>>> >>>>> class classB ($param1 = '''') { #This uses the auto lookup of >>>>> parameters. It will perform a hiera search for ''classB::param1''. >>>>> notice ("ClassB: ParamValue $param1") >>>>> } >>>>> >>>>> >>>>> NodeA.yaml >>>>> --- >>>>> classes: >>>>> - a >>>>> >>>>> NodeB.yaml >>>>> --- >>>>> classes: >>>>> - b >>>>> >>>>> ''classB::param1'': ''Puppet-Hiera'' >>>>> >>>>> node default { >>>>> hiera_include(''classes'', []) #empty array, not empty string, which >>>>> might also have been messing things up. >>>>> >>>>> } >>>>> } >>>>> >>>>> >>>>> On Thursday, February 21, 2013 10:55:59 AM UTC-8, Sai_Emc wrote: >>>>>> >>>>>> I am looking for few clarifications on puppet-hiera integration. >>>>>> >>>>>> Trying to move away completly from using site.pp. For that I started >>>>>> exploring Hiera. End of the day I want an external system prepare yaml >>>>>> files automatucally based on user requests. >>>>>> >>>>>> Environment: Ubuntu12.04, PE 2.7, Hiera 1.1.2 >>>>>> >>>>>> EX: >>>>>> class classA { >>>>>> notice("ClassA") >>>>>> } >>>>>> >>>>>> class classB ($param1 = '''') { >>>>>> notice ("ClassB: ParamValue $param1") >>>>>> } >>>>>> >>>>>> >>>>>> Now I am want to include these two classes into two different nodes, >>>>>> so prepared two yaml files >>>>>> >>>>>> hiera.yaml >>>>>> --- >>>>>> :hierarchy: >>>>>> - %{::clientcert} >>>>>> - common >>>>>> :backends: >>>>>> - yaml >>>>>> :yaml: >>>>>> :datadir: ''/etc/puppetlabs/puppet/hieradata'' >>>>>> >>>>>> NodeA.yaml >>>>>> --- >>>>>> emcutil::a: >>>>>> - a >>>>>> >>>>>> NodeB.yaml >>>>>> --- >>>>>> classesH: >>>>>> emcutil::b: >>>>>> param1: ''Puppet-Hiera'' >>>>>> >>>>>> >>>>>> As I have classes as array and hash, so added below code in site.pp >>>>>> default section so that I can include hash and arrays. >>>>>> >>>>>> node default { >>>>>> hiera_include(''classesA'','''') >>>>>> $param_packagesH = hiera_hash(''classesH'') >>>>>> create_resources(''class'',$param_classesH) >>>>>> } >>>>>> } >>>>>> >>>>>> This approach does not work. >>>>>> >>>>>> On NodeA following error: >>>>>> err: Could not retrieve catalog from remote server: Error 400 on >>>>>> SERVER: undefined method `empty?'' for nil:NilClass at >>>>>> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:48 on node >>>>>> On NodeB following error: >>>>>> err: Could not retrieve catalog from remote server: Error 400 on >>>>>> SERVER: Could not find data item classesA in any Hiera data file and no >>>>>> default supplied at >>>>>> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:46 on node >>>>>> >>>>>> What I am doing here is valid? >>>>>> If we are using hiera_include or hiera_hash all the node yaml files >>>>>> should have all the targets and the target returns not null values? >>>>>> Do we have alternatives other than using hiera, hiera_include and >>>>>> hiera_hash? >>>>>> PE 2.7 do we need to do any configuration for Hiera, apart from >>>>>> puppet-hiera package. Specifically related to these two attributes >>>>>> node_terminus, external_nodes? >>>>>> With Hiera we can move away from site.pp completly. My understanding >>>>>> is correct on Hiera? >>>>>> >>>>>> Any help on this greatly appreciated. >>>>>> >>>>>> Sai. >>>>>> EMC Corporation. >>>>>> >>>>>> >>>>>> >>>>>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Sai_Emc
2013-Feb-22 20:05 UTC
[Puppet Users] Re: Puppet-Hiera: hiera_include and hiera_hash behaviour
I can see json_backend working in latest hiera. Tried from command line its working . puppetmaster was looking at old hiera-0.3.0 gem even whem latest hiera is installed. Posted a different topic for that. Thanks. Sai EMC On Thursday, February 21, 2013 5:50:32 PM UTC-5, Sai_Emc wrote:> Wonderful...This is cool. > > As you said, maintaining thousands yaml files manually might be difficlut. > I have to automate this process to accept requests from an external system > about which node needs which resources. > > So I might use json over yaml, because at this point I dont know my > external system can do yaml. Any idea json backend supported in 2.7?. I saw > in the documentation but not sure supported in 2.7 or not. > > Once again thank you very much. > > Regards > Sai. > EMC > > On Thursday, February 21, 2013 5:32:55 PM UTC-5, Ellison Marks wrote: > >> It''s not that bad performance wise. Hiera should only lode the applicable >> yaml files. If (god forbid) you had one yaml file per host and had >> thousands of hosts, hiera should still only load two files, the host''s yaml >> file and the common yaml file. The only thing that would be bad is manually >> maintaining all the yaml files :) >> >> As to the multiple parameters, it''s also not that bad. You could format >> the parameters as an array and seperate them later, like this: >> >> NodeB.yaml >> --- >> classes: >> - b >> >> ''classB::params'': >> - ''Puppet-Hiera'' >> - ''Puppet-Hiera'' >> - ''Puppet-Hiera'' >> - ''Puppet-Hiera'' >> >> class classB ($params = hiera(''classB::params'', '''')) { >> $param1=$params[0] >> $param2=$params[1] >> $param3=$params[2] >> $param4=$params[3] >> >> but it''s not that much better. The real trick is in using create >> resources correctly. You want to have a 3 depth array in hiera. top is the >> name you will look up with the hiera() call. Second level is the names of >> the resources you want to create. For example, if you were creating users, >> this level could have ''john'' and ''jane''. Third level is the parameters for >> the individual resource. In this example, below john could be {''uid''=>501, >> ''group''=>john} and below jane could be {''uid''=>502, ''group''=>jane} >> >> userhash: >> john: >> uid: 501 >> group: john >> jane: >> uid: 502 >> group: jane >> >> then you would say >> >> $userhash=hiera(''userhash'') >> create_resources(user, $userhash) >> >> So if you can structure your class in a way that supports that flow, it >> can be really good. >> >> On Thursday, February 21, 2013 12:27:30 PM UTC-8, Sai_Emc wrote: >>> >>> That''s ok Mark. I kind of remember that from documentation. >>> >>> When we use above approach puppet need to make multiple hiera() calls to >>> get resources specific to a class. >>> >>> I mean a class with say 4 arguments, puppet need to make a multiple >>> calls to get all the parameters. >>> >>> NodeB.yaml >>> --- >>> classes: >>> - b >>> >>> ''classB::param1'': ''Puppet-Hiera'' >>> ''classB::param2'': ''Puppet-Hiera'' >>> ''classB::param3'': ''Puppet-Hiera'' >>> ''classB::param4'': ''Puppet-Hiera'' >>> >>> If we have thousands of nodes we might see any performance issues with >>> array approach? I might be totally wrong on this. >>> >>> The same if I have declared as Hash then from puppet that is just one >>> call, but it might look all defined hierarchies. So if we do use hiera_hash >>> then without using create_resources() is there any other method I can >>> include class definitions. >>> >>> Thank you. >>> >>> Regards >>> Sai. >>> EMC >>> >>> On Thursday, February 21, 2013 3:01:54 PM UTC-5, Ellison Marks wrote: >>> >>>> Ah, sorry, my mistake, mised the version. That line would have to be >>>> >>>> class classB ($param1 = hiera(''classB::param1'', '''')) { >>>> >>>> with the hiera call in the definition. Sorry about that :P >>>> >>>> On Thursday, February 21, 2013 11:57:24 AM UTC-8, Sai_Emc wrote: >>>>> >>>>> Mark, >>>>> >>>>> Thanks for the quick reply. >>>>> >>>>> "This uses the auto lookup of parameters. It will perform a hiera >>>>> search for ''classB::param1''." --> Auto lookup feature what you mentioned >>>>> is in PE 2.7? I remember reading as 3.0 feature. >>>>> >>>>> Regards >>>>> Sai. >>>>> EMC >>>>> >>>>> On Thursday, February 21, 2013 2:32:10 PM UTC-5, Ellison Marks wrote: >>>>> >>>>>> quick definitions first: >>>>>> >>>>>> hiera() gets a value from the backend. stops searching after it finds >>>>>> a value. the value can be any data type. >>>>>> hiera_array() gets multiple values from the backend, combining them >>>>>> into one large array as possible. >>>>>> hiera_hash() gets multiple values from the backend, combining them >>>>>> into one large hash as possible. >>>>>> hiera_include() calls hiera_array() and then includes each classname >>>>>> returned. >>>>>> >>>>>> Ok, so hiera_include will look in the backend for a variable named in >>>>>> the call, in your case ''classesA''. this should contain an array of class >>>>>> names to include. As your hierarchy apparently doesn''t contain the variable >>>>>> ''classesA'', this explains the second error you are getting. You also didn''t >>>>>> include the variable ''classesH'' in nodeA.yaml, so that might be the cause >>>>>> of the failure there. create_resources is also not used on classes, just >>>>>> types or defines. Generally, for hiera_include, it should look sort of like >>>>>> this. >>>>>> >>>>>> class classA { >>>>>> notice("ClassA") >>>>>> } >>>>>> >>>>>> class classB ($param1 = '''') { #This uses the auto lookup of >>>>>> parameters. It will perform a hiera search for ''classB::param1''. >>>>>> notice ("ClassB: ParamValue $param1") >>>>>> } >>>>>> >>>>>> >>>>>> NodeA.yaml >>>>>> --- >>>>>> classes: >>>>>> - a >>>>>> >>>>>> NodeB.yaml >>>>>> --- >>>>>> classes: >>>>>> - b >>>>>> >>>>>> ''classB::param1'': ''Puppet-Hiera'' >>>>>> >>>>>> node default { >>>>>> hiera_include(''classes'', []) #empty array, not empty string, which >>>>>> might also have been messing things up. >>>>>> >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> On Thursday, February 21, 2013 10:55:59 AM UTC-8, Sai_Emc wrote: >>>>>>> >>>>>>> I am looking for few clarifications on puppet-hiera integration. >>>>>>> >>>>>>> Trying to move away completly from using site.pp. For that I started >>>>>>> exploring Hiera. End of the day I want an external system prepare yaml >>>>>>> files automatucally based on user requests. >>>>>>> >>>>>>> Environment: Ubuntu12.04, PE 2.7, Hiera 1.1.2 >>>>>>> >>>>>>> EX: >>>>>>> class classA { >>>>>>> notice("ClassA") >>>>>>> } >>>>>>> >>>>>>> class classB ($param1 = '''') { >>>>>>> notice ("ClassB: ParamValue $param1") >>>>>>> } >>>>>>> >>>>>>> >>>>>>> Now I am want to include these two classes into two different nodes, >>>>>>> so prepared two yaml files >>>>>>> >>>>>>> hiera.yaml >>>>>>> --- >>>>>>> :hierarchy: >>>>>>> - %{::clientcert} >>>>>>> - common >>>>>>> :backends: >>>>>>> - yaml >>>>>>> :yaml: >>>>>>> :datadir: ''/etc/puppetlabs/puppet/hieradata'' >>>>>>> >>>>>>> NodeA.yaml >>>>>>> --- >>>>>>> emcutil::a: >>>>>>> - a >>>>>>> >>>>>>> NodeB.yaml >>>>>>> --- >>>>>>> classesH: >>>>>>> emcutil::b: >>>>>>> param1: ''Puppet-Hiera'' >>>>>>> >>>>>>> >>>>>>> As I have classes as array and hash, so added below code in site.pp >>>>>>> default section so that I can include hash and arrays. >>>>>>> >>>>>>> node default { >>>>>>> hiera_include(''classesA'','''') >>>>>>> $param_packagesH = hiera_hash(''classesH'') >>>>>>> create_resources(''class'',$param_classesH) >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> This approach does not work. >>>>>>> >>>>>>> On NodeA following error: >>>>>>> err: Could not retrieve catalog from remote server: Error 400 on >>>>>>> SERVER: undefined method `empty?'' for nil:NilClass at >>>>>>> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:48 on node >>>>>>> On NodeB following error: >>>>>>> err: Could not retrieve catalog from remote server: Error 400 on >>>>>>> SERVER: Could not find data item classesA in any Hiera data file and no >>>>>>> default supplied at >>>>>>> /etc/puppetlabs/puppet/environments/development/manifests/site.pp:46 on node >>>>>>> >>>>>>> What I am doing here is valid? >>>>>>> If we are using hiera_include or hiera_hash all the node yaml files >>>>>>> should have all the targets and the target returns not null values? >>>>>>> Do we have alternatives other than using hiera, hiera_include and >>>>>>> hiera_hash? >>>>>>> PE 2.7 do we need to do any configuration for Hiera, apart from >>>>>>> puppet-hiera package. Specifically related to these two attributes >>>>>>> node_terminus, external_nodes? >>>>>>> With Hiera we can move away from site.pp completly. My understanding >>>>>>> is correct on Hiera? >>>>>>> >>>>>>> Any help on this greatly appreciated. >>>>>>> >>>>>>> Sai. >>>>>>> EMC Corporation. >>>>>>> >>>>>>> >>>>>>> >>>>>>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.