Puppet users - Aug 2013 - Share data between classes/modules, best practices

Hi,

I use Puppet 3.2 with Hiera. I would like to know the best practices
when you want to share one data between 2 classes or modules.

I take an example. I have:

- One "snmp" module which installs and configures snmpd in the hosts.
In this module, I must define the community password.

- One "monitoring" module which installs and configures any monitoring
software (by example Nagios if you want, it doesn''t matter). In this
module,
I must define too the community password in order to the software monitoring
runs valid snmp requests.

So, the "snmp" module and the "monitoring" module must share
the same
data: the community password. Which are the best practice to do this?


Firstly, I see this way.

In the /etc/puppet/hieradata/default.yaml, I have this:

---
snmp:
 community: abcd1234

monitoring:
 community: abcd1234

Then, in my "snmp" class, I have this:

class snmp {

    $snmp      = hiera_hash(''snmp'')
    $community = $snmp[''community'']
    
    # and the rest of the class...

}

And in my "monitoring" class, I have this:

class monitoring {

    $monitoring = hiera_hash(''monitoring'')
    $community  = $monitoring[''community'']

    # and the rest of the class...

}

Like this, my modules are independent but the same "community
password"
data is duplicated in 2 different places. Ok, in my example, the data
is in the same file (default.yaml) but we can imagine more complex cases
where the same data is duplicated in 2 different files (because the data
hierarchy isn''t limited to one "default.yaml" file etc).


Secondly, I see this way.

In the /etc/puppet/hieradata/default.yaml, I have this:

---
# The "community" key is present just in one place and that''s
all.
snmp:
 community: abcd1234

# No "community" key in monitoring.
monitoring:
 # some keys but not the "community" key.
 contacts:
  - john
  - herbert

Then, in my "snmp" module, I have the snmp::params class which
just contains variables assignments:

class snmp::params {

    $snmp      = hiera_hash(''snmp'')
    $community = $snmp[''community'']

    # and the rest of the class...

}

class snmp {

    include snmp::params

    # and the rest of the class...
    # we can use the $snmp::params::community variable.

}

And in my "monitoring" class, I have this:

class monitoring {

    include snmp::params
    $community  = $snmp::params::community

    # and the rest of the class...

}

Now, the "community password" data isn''t duplicated, but my
modules
aren''t independent because the "monitoring" module depends on
the
"snmp" module. And I suppose that''s better to have
independent modules.

So, what are the best practices to share the "community password"
between the "snmp" module and the "monitoring" module?

Thanks in advance for your help.

--
François Lafont

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Friday, August 23, 2013 7:08:21 AM UTC-5, François Lafont
wrote:
>
> Hi, 
>
> I use Puppet 3.2 with Hiera. I would like to know the best practices 
> when you want to share one data between 2 classes or modules. 
>
> I take an example. I have: 
>
> - One "snmp" module which installs and configures snmpd in the
hosts.
> In this module, I must define the community password. 
>
> - One "monitoring" module which installs and configures any
monitoring
> software (by example Nagios if you want, it doesn''t matter). In
this
> module, 
> I must define too the community password in order to the software 
> monitoring 
> runs valid snmp requests. 
>
> So, the "snmp" module and the "monitoring" module must
share the same
> data: the community password. Which are the best practice to do this? 
>
>
[...]
 
>
> Like this, my modules are independent but the same "community
password"
> data is duplicated in 2 different places. Ok, in my example, the data 
> is in the same file (default.yaml) but we can imagine more complex cases 
> where the same data is duplicated in 2 different files (because the data 
> hierarchy isn''t limited to one "default.yaml" file etc).
>
>
[...]
 
>
> Now, the "community password" data isn''t duplicated, but
my modules
> aren''t independent because the "monitoring" module
depends on the
> "snmp" module. And I suppose that''s better to have
independent modules.
>
>
Your modules are not independent in any case.  The data dependency is a 
symptom, not the root issue.

To answer more generally, since you started by posing a rather general 
problem, where you have a bona fide configuration dependency, it is not 
wrong for one module to depend on another.  Real systems are composed of 
multiple interdependent pieces, so it is natural for your models of them to 
contain dependencies.  You can shift them around to various places and 
forms, but you cannot omit them and still have anything of much use.

One place you did not mention, to which you could shift the dependency 
modeling, is your data.  For example, lift your ''community''
key to the top
level of your YAML data, consolidating all copies of it.  Then have any 
class that needs that particular key just load it.

I don''t think there is a clear, universal right answer here.  My best 
advice is to find where each datum belongs, and put it there.  Coincidental 
data duplication should not concern you overmuch, but do avoid duplicating 
data for the purpose of presenting a false semblance of independence.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Le 27/08/2013 22:49, jcbollinger a écrit :
 
> Your modules are not independent in any case.  The data dependency is a 
> symptom, not the root issue.
Yes, you are right.
> To answer more generally, since you started by posing a rather general 
> problem, where you have a bona fide configuration dependency, it is not 
> wrong for one module to depend on another.  Real systems are composed of 
> multiple interdependent pieces, so it is natural for your models of them to
> contain dependencies.  You can shift them around to various places and 
> forms, but you cannot omit them and still have anything of much use.
Ok, I understand, impossible to escape real dependencies, that''s life.
> One place you did not mention, to which you could shift the dependency 
> modeling, is your data.  For example, lift your
''community'' key to the top
> level of your YAML data, consolidating all copies of it.  Then have any 
> class that needs that particular key just load it.
Very good idea. In fact, I saw the things with an oriented object approach: a
class must know its data.
> I don''t think there is a clear, universal right answer here.  My
best
> advice is to find where each datum belongs, and put it there.  Coincidental
> data duplication should not concern you overmuch, but do avoid duplicating 
> data for the purpose of presenting a false semblance of independence.
Ok, I keep theses advices in my mind:
- no duplication of data
- define clearly where each datum belongs and put it there.

I think it''s a good idea to border the dependency on the data and not
in
the puppet code.

I thought about another way with the extlookup function.

---
snmp:
 community: extvalue_community

monitoring:
 community: extvalue_community

And in a common.csv file:

extvalue_community,abcd1234

Then:

class snmp {

    $snmp      = hiera_hash(''snmp'')
    $community = extlookup($snmp[''community''])
    
    # and the rest of the class...

}
class monitoring {

    $monitoring = hiera_hash(''monitoring'')
    $community  = extlookup($monitoring[''community''])

    # and the rest of the class...

}

I don''t know if it''s good method. 

Thanks for your help John.

--
Francois Lafont

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

It seems a very clever approach!!!
If it works could be the approach for not having dependency.
I do not agree that the two module are dependend, they just depend on the 
same data, but given the data should be able to work on their own.
This problem is also mine in trying to design modules that do not require 
people to have knowledge of other modules.

This approach enables to place all possible common data in the external 
file and using hiera to decouple the stuff.
So on the module usage description people will just be asked to set the 
files accordingly and the possible common data is referenced by hiera.

could the data be protected in some way? just not to have maby common 
password written in clear?

Luca

Il giorno mercoledì 28 agosto 2013 02:26:41 UTC+2, François Lafont ha 
scritto:
>
> Le 27/08/2013 22:49, jcbollinger a écrit : 
>   
> > Your modules are not independent in any case.  The data dependency is
a
> > symptom, not the root issue. 
>
> Yes, you are right. 
>
> > To answer more generally, since you started by posing a rather general
> > problem, where you have a bona fide configuration dependency, it is
not
> > wrong for one module to depend on another.  Real systems are composed
of
> > multiple interdependent pieces, so it is natural for your models of
them
> to 
> > contain dependencies.  You can shift them around to various places and
> > forms, but you cannot omit them and still have anything of much use. 
>
> Ok, I understand, impossible to escape real dependencies, that''s
life.
>
> > One place you did not mention, to which you could shift the dependency
> > modeling, is your data.  For example, lift your
''community'' key to the
> top 
> > level of your YAML data, consolidating all copies of it.  Then have
any
> > class that needs that particular key just load it. 
>
> Very good idea. In fact, I saw the things with an oriented object 
> approach: a 
> class must know its data. 
>
> > I don''t think there is a clear, universal right answer here. 
My best
> > advice is to find where each datum belongs, and put it there. 
>  Coincidental 
> > data duplication should not concern you overmuch, but do avoid 
> duplicating 
> > data for the purpose of presenting a false semblance of independence. 
>
> Ok, I keep theses advices in my mind: 
> - no duplication of data 
> - define clearly where each datum belongs and put it there. 
>
> I think it''s a good idea to border the dependency on the data and
not in
> the puppet code. 
>
> I thought about another way with the extlookup function. 
>
> --- 
> snmp: 
>  community: extvalue_community 
>
> monitoring: 
>  community: extvalue_community 
>
> And in a common.csv file: 
>
> extvalue_community,abcd1234 
>
> Then: 
>
> class snmp { 
>
>     $snmp      = hiera_hash(''snmp'') 
>     $community = extlookup($snmp[''community'']) 
>     
>     # and the rest of the class... 
>
> } 
> class monitoring { 
>
>     $monitoring = hiera_hash(''monitoring'') 
>     $community  = extlookup($monitoring[''community'']) 
>
>     # and the rest of the class... 
>
> } 
>
> I don''t know if it''s good method. 
>
> Thanks for your help John. 
>
> -- 
> Francois Lafont 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Le 28/08/2013 11:57, Luca Gioppo a écrit :
> It seems a very clever approach!!!
Are you talking about the "extlookup" approach?

There is one thing which toubles me with extlookup. For me, when I put:

     $snmp      = hiera_hash(''snmp'')
     $community = extlookup($snmp[''community''])

instead of :

     $snmp      = hiera_hash(''snmp'')
     $community = $snmp[''community'']


I show an implementation detail ("the data is not in a yaml file but in 
a external source") which should not be appear in the module. In fact, I 
would like to have a new "hiera_hash"function which behaves exactly
like
the current "hiera_hash" function (in Puppet 3), except when a value 
matchs this regex /^extkey_/ (by example) where, in this case, the 
function apply automatically extlookup("the-value") in a transparent
way
for the user.
> If it works could be the approach for not having dependency.
> I do not agree that the two module are dependend, they just depend on 
> the same data, but given the data should be able to work on their own.
Ok, the data are dependent, not the modules. Why not.
> could the data be protected in some way? just not to have maby common 
> password written in clear?
This is another question. Personally, I have no solution, but it''s a 
interesting question.

--
François Lafont

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Wednesday, August 28, 2013 4:57:36 AM UTC-5, Luca Gioppo
wrote:
>
> It seems a very clever approach!!!
> If it works could be the approach for not having dependency.
> I do not agree that the two module are dependend, they just depend on the 
> same data, but given the data should be able to work on their own.
> This problem is also mine in trying to design modules that do not require 
> people to have knowledge of other modules.
>
>
At minimum, modules that depend on the same hiera key are coupled via that 
key.  I choose to consider that a form of dependency, but you can certainly 
take a narrower view of that term if you wish.  The fact remains that even 
if module authors are not aware of such coupling, module *users* need to 
be.  Moreover, although module authors may not be aware of other modules 
depending on the same data, they do need to be mindful which data are 
shared, for those belong to the overall site, not to any individual 
module.  The technique can certainly be useful and appropriate, but like 
any other technique, you need to understand the implications of what you do 
with it.

 
> This approach enables to place all possible common data in the external 
> file and using hiera to decouple the stuff.
> So on the module usage description people will just be asked to set the 
> files accordingly and the possible common data is referenced by hiera.
>
> could the data be protected in some way? just not to have maby common 
> password written in clear?
>
>
Hiera supports pluggable back-ends, and there are some available that 
provide for encrypted storage of the data.  I am inclined to think, 
however, that it is sufficient for most sites to limit access to the 
Puppetmaster and rely on appropriate access controls for the data files.  
In fact, no encryption solution can do more than slow a knowledgeable and 
determined assailant attempting to obtaining data that the master itself 
can decrypt, for in the end there has to be a cleartext encryption key or 
comparable credential that Puppet can read.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On dependency:
My view is that many modules require you to install another module for 
working (let''s take concat for example).

I like as a module developer to release modules that "depend" on
official
modules from puppetlabs if possible and if able to contribute to them 
becouse there ale lots of different flavors and I think that a user could 
be happyer to get just a single module and not a tree of dependant stuff so 
... agree that user has to be avware that if he want to use the two modules 
together he has to comply to a naming convention on data but no module 
dependancy around.
But this is just my approach and very personal.

On the the hiera stuff I also vould like to have a hiera_hash_lookup where 
the function behaves in a single shot getting the value fron the hash and 
instead of returning it it looks-up for the value to the external lookpu.
Maybe it could be an idea for a feature enancement this way it could be 
cleaner for user.
But in any way user will have to be instructed to create a hash in the yaml 
file with the reference and the real value in the csv file so maybe the 
fact of having the cose in the module is not a big problem user should just 
"use" it is more a developer "cleaness need".

In any case I''ll use that approach.

Obviously the security stuff can be a problem, I agree that you cannot 
protect all, but a level of protection could be useful if there could be a 
extlookup-gpg like there is a hiera-gpg maybe it could be nice.

Thanks
Luca

Il giorno mercoledì 28 agosto 2013 15:18:01 UTC+2, François Lafont ha 
scritto:
>
> Le 28/08/2013 11:57, Luca Gioppo a �crit : 
> > It seems a very clever approach!!! 
>
> Are you talking about the "extlookup" approach? 
>
> There is one thing which toubles me with extlookup. For me, when I put: 
>
>      $snmp      = hiera_hash(''snmp'') 
>      $community = extlookup($snmp[''community'']) 
>
> instead of : 
>
>      $snmp      = hiera_hash(''snmp'') 
>      $community = $snmp[''community''] 
>
>
> I show an implementation detail ("the data is not in a yaml file but
in
> a external source") which should not be appear in the module. In fact,
I
> would like to have a new "hiera_hash"function which behaves
exactly like
> the current "hiera_hash" function (in Puppet 3), except when a
value
> matchs this regex /^extkey_/ (by example) where, in this case, the 
> function apply automatically extlookup("the-value") in a
transparent way
> for the user. 
>
> > If it works could be the approach for not having dependency. 
> > I do not agree that the two module are dependend, they just depend on 
> > the same data, but given the data should be able to work on their own.
>
> Ok, the data are dependent, not the modules. Why not. 
>
> > could the data be protected in some way? just not to have maby common 
> > password written in clear? 
>
> This is another question. Personally, I have no solution, but it''s
a
> interesting question. 
>
> -- 
> Fran�ois Lafont 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

On Tuesday, August 27, 2013 7:26:41 PM UTC-5, François Lafont wrote:
>
> I thought about another way with the extlookup function. 
>
> --- 
> snmp: 
>  community: extvalue_community 
>
> monitoring: 
>  community: extvalue_community 
>
> And in a common.csv file: 
>
> extvalue_community,abcd1234 
>
> Then: 
>
> class snmp { 
>
>     $snmp      = hiera_hash(''snmp'') 
>     $community = extlookup($snmp[''community'']) 
>     
>     # and the rest of the class... 
>
> } 
> class monitoring { 
>
>     $monitoring = hiera_hash(''monitoring'') 
>     $community  = extlookup($monitoring[''community'']) 
>
>     # and the rest of the class... 
>
> } 
>
> I don''t know if it''s good method. 
>
> Thanks for your help John. 
>
> -- 
> Francois Lafont 
>
I think you''re taking an extra step and arriving at the same solution. 
 Eliminate the extlookup step and instead have both modules look at the 
same variable in hiera. Make it some arbitrary name that doesn''t
conflict
with any module (and hopefully won''t in the future either).  Then just 
lookup that value explicitly with the hiera function in any module that 
needs it.

--- 
snmp_community: abcd1234

class snmp { 
    $snmp_community      = hiera(''snmp_community'') 
}

class monitoring {
    $snmp_community      = hiera(''snmp_community'')
}

If you do use a hash like you were using in Hiera, please note that you do 
not need to use hiera_hash() to get the data unless you''re
"merging" that
hash up your hierarchy (i.e. Setting part of the hash data in global and 
setting some more pieces of the hash in another part of the hierarchy that 
you want to merge together for the final data).  hiera() will get the hash 
just fine but will not merge it up the tree.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Since I need to store things like Db passwords (used on DB node to create 
stuff and on the app_server node to establish the connection) I could use 
either a hash dedicated to generic DB stuff or a hash for passwords and in 
this way I could also use hiera-gpg

All too simple.
Thanks
Luca

Il giorno venerdì 30 agosto 2013 14:15:14 UTC+2, Drew Blessing ha
scritto:
>
>
> On Tuesday, August 27, 2013 7:26:41 PM UTC-5, François Lafont wrote:
>
>>
>> I thought about another way with the extlookup function. 
>>
>> --- 
>> snmp: 
>>  community: extvalue_community 
>>
>> monitoring: 
>>  community: extvalue_community 
>>
>> And in a common.csv file: 
>>
>> extvalue_community,abcd1234 
>>
>> Then: 
>>
>> class snmp { 
>>
>>     $snmp      = hiera_hash(''snmp'') 
>>     $community = extlookup($snmp[''community'']) 
>>     
>>     # and the rest of the class... 
>>
>> } 
>> class monitoring { 
>>
>>     $monitoring = hiera_hash(''monitoring'') 
>>     $community  = extlookup($monitoring[''community''])
>>
>>     # and the rest of the class... 
>>
>> } 
>>
>> I don''t know if it''s good method. 
>>
>> Thanks for your help John. 
>>
>> -- 
>> Francois Lafont 
>>
>
> I think you''re taking an extra step and arriving at the same
solution.
>  Eliminate the extlookup step and instead have both modules look at the 
> same variable in hiera. Make it some arbitrary name that doesn''t
conflict
> with any module (and hopefully won''t in the future either).  Then
just
> lookup that value explicitly with the hiera function in any module that 
> needs it.
>
> --- 
> snmp_community: abcd1234
>
> class snmp { 
>     $snmp_community      = hiera(''snmp_community'') 
> }
>
> class monitoring {
>     $snmp_community      = hiera(''snmp_community'')
> }
>
> If you do use a hash like you were using in Hiera, please note that you do 
> not need to use hiera_hash() to get the data unless you''re
"merging" that
> hash up your hierarchy (i.e. Setting part of the hash data in global and 
> setting some more pieces of the hash in another part of the hierarchy that 
> you want to merge together for the final data).  hiera() will get the hash 
> just fine but will not merge it up the tree.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.