Puppet users - Dec 2012 - Question on modeling multiple services sharing configuration

Hi!

I would appreciate any advice on the best practices
on how to model a collection of services that each
has its own configuration file, but also share a common
one.

Now, the trouble is, that the common configuration file
is not *really* just a place for the common configuration
to reside, but also may have sections dedicated to holding
properties of a particular service (just like /etc/puppet/puppet.conf
has [main] [agent] and [master] sections).

Thus, in reality, the configuration properties are
per service and whether they need to go into a common
configuration file or a service-specific one is NOT cleanly
partitioned and is pretty awkward to remember. Which
means I really don''t want to expose the common config
explicitly to the end user.

Instead, I''d like to expose the natural hierarchy of:

class service1($foo, $bar, $baz) {
}
.....
class serviceN($qoo, $zoo) {
}

but the question then becomes -- how can I model a
common configuration file behind user''s back so that
$foo, $bar and $qoo end up there.

At first I was thinking that I could utilize a singleton
non-parameterized class and include it multiple
times in all of the serviceX class definitions, but it
seems I can''t pass any values into it.

I suppose I can still do that and create a skeleton
of the common config in that singleton class
that would later be manipulated either by augeas or
concat patterns, but this seems to be pretty heavyweight.

What would you, guys, recommend?

Thanks,
Roman.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 12/16/2012 01:09 AM, Roman Shaposhnik wrote:
> Hi!
>
> I would appreciate any advice on the best practices
> on how to model a collection of services that each
> has its own configuration file, but also share a common
> one.
>
> Now, the trouble is, that the common configuration file
> is not *really* just a place for the common configuration
> to reside, but also may have sections dedicated to holding
> properties of a particular service (just like /etc/puppet/puppet.conf
> has [main] [agent] and [master] sections).
>
> Thus, in reality, the configuration properties are
> per service and whether they need to go into a common
> configuration file or a service-specific one is NOT cleanly
> partitioned and is pretty awkward to remember. Which
> means I really don''t want to expose the common config
> explicitly to the end user.
>
> Instead, I''d like to expose the natural hierarchy of:
>
> class service1($foo, $bar, $baz) {
> }
> .....
> class serviceN($qoo, $zoo) {
> }
>
> but the question then becomes -- how can I model a
> common configuration file behind user''s back so that
> $foo, $bar and $qoo end up there.
>
> At first I was thinking that I could utilize a singleton
> non-parameterized class and include it multiple
> times in all of the serviceX class definitions, but it
> seems I can''t pass any values into it.
>
> I suppose I can still do that and create a skeleton
> of the common config in that singleton class
> that would later be manipulated either by augeas or
> concat patterns, but this seems to be pretty heavyweight.
>
> What would you, guys, recommend?
This is a pattern I feel augeas is awesome at.  I just did a similar 
thing for puppet.conf on my end.

Make the common configuration file virtual.  Give it a default "dummy"
configuration file with replace = false (I do this because creating a 
file with augeas seems a bit shaky to me).

Then have each of the other services realize the common configuration 
file and use augeas to add their own specific configuration elements.

If the file is an inifile style, creating a lens is pretty trivial - you 
can see the puppet file lens (Which is in 
/usr/share/augeas/lenses/dist/puppet.aug on ubuntu) for an example.

Jason

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

You may also want to consider looking at the concat module that R.I. Pienaar has
on github. It should allow you to easily do the fragments on a common file like
you described.
https://github.com/ripienaar/puppet-concat

As for the way you expose the configuration properties as class parameters for
the services, are you saying the actual names of $foo/$bar/$baz, for instance,
would inform as to what information they need except it is not consistent
between services which values go in which file? I suppose you could hide the
detail of which property to which file if needed, however, parameterized classes
do not work like functions. The parameters, at declaration time, can be declared
in any order. Additionally, even if you could enforce the order in which they
were passed, simply ordering them does not seem very intuitive still. Your best
bet is to expose parameters with meaningful names, and document if it still is
not very clear.

-- 
Tom Linkin
Professional Services Engineer
http://puppetlabs.com/
twitter: @trlinkin



On Sunday, December 16, 2012 at 9:41 AM, Jason Slagle wrote:
> 
> 
> On 12/16/2012 01:09 AM, Roman Shaposhnik wrote:
> > Hi!
> > 
> > I would appreciate any advice on the best practices
> > on how to model a collection of services that each
> > has its own configuration file, but also share a common
> > one.
> > 
> > Now, the trouble is, that the common configuration file
> > is not *really* just a place for the common configuration
> > to reside, but also may have sections dedicated to holding
> > properties of a particular service (just like /etc/puppet/puppet.conf
> > has [main] [agent] and [master] sections).
> > 
> > Thus, in reality, the configuration properties are
> > per service and whether they need to go into a common
> > configuration file or a service-specific one is NOT cleanly
> > partitioned and is pretty awkward to remember. Which
> > means I really don''t want to expose the common config
> > explicitly to the end user.
> > 
> > Instead, I''d like to expose the natural hierarchy of:
> > 
> > class service1($foo, $bar, $baz) {
> > }
> > .....
> > class serviceN($qoo, $zoo) {
> > }
> > 
> > but the question then becomes -- how can I model a
> > common configuration file behind user''s back so that
> > $foo, $bar and $qoo end up there.
> > 
> > At first I was thinking that I could utilize a singleton
> > non-parameterized class and include it multiple
> > times in all of the serviceX class definitions, but it
> > seems I can''t pass any values into it.
> > 
> > I suppose I can still do that and create a skeleton
> > of the common config in that singleton class
> > that would later be manipulated either by augeas or
> > concat patterns, but this seems to be pretty heavyweight.
> > 
> > What would you, guys, recommend?
> 
> This is a pattern I feel augeas is awesome at. I just did a similar 
> thing for puppet.conf on my end.
> 
> Make the common configuration file virtual. Give it a default
"dummy"
> configuration file with replace = false (I do this because creating a 
> file with augeas seems a bit shaky to me).
> 
> Then have each of the other services realize the common configuration 
> file and use augeas to add their own specific configuration elements.
> 
> If the file is an inifile style, creating a lens is pretty trivial - you 
> can see the puppet file lens (Which is in 
> /usr/share/augeas/lenses/dist/puppet.aug on ubuntu) for an example.
> 
> Jason
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com
(mailto:puppet-users@googlegroups.com).
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
(mailto:puppet-users+unsubscribe@googlegroups.com).
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
> 
> 

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Sun, Dec 16, 2012 at 1:39 PM, Tom Linkin <thomas@puppetlabs.com>
wrote:
> You may also want to consider looking at the concat module that R.I.
Pienaar
> has on github. It should allow you to easily do the fragments on a common
> file like you described.
> https://github.com/ripienaar/puppet-concat
Yup. This is the one I was looking at when I mentioned that I could''ve
gone
a concat route. It just feels a bit heavy. In fact, I''d love for Puppet
to
have a sort of a in-memory store that can be manipulated same way
that concat manipulates files in the actual filesystem. Something like
tmpfile resource. Has anything like that ever been considered?
> As for the way you expose the configuration properties as class parameters
> for the services, are you saying the actual names of $foo/$bar/$baz, for
> instance, would inform as to what information they need except it is not
> consistent between services which values go in which file?
Only that it is not consistent (and you can''t guess from a name)
whether
each one needs to go into the common or service-specific configuration
file.
> The parameters, at declaration time, can be declared in any order.
> Additionally, even if you could enforce the order in which they were
passed,
> simply ordering them does not seem very intuitive still. Your best bet is
to expose parameters with
> meaningful names, and document if it still is not very clear.
I don''t think this would be a problem. The names themselves are
descriptive (and they are NOT order sensitive). The only problem
with them is that you can''t tell by just looking at a name where
the property should go.

Thanks,
Roman.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Sun, Dec 16, 2012 at 6:41 AM, Jason Slagle <raistlin@tacorp.net>
wrote:
> This is a pattern I feel augeas is awesome at.  I just did a similar thing
> for puppet.conf on my end.
Yup. Using augeas is my #1 design choice ATM. Btw, have you had
any experience with the XML lens? Those config files are XML :-(
> Make the common configuration file virtual.  Give it a default
"dummy"
> configuration file with replace = false (I do this because creating a file
> with augeas seems a bit shaky to me)
replace = false ? Could you elaborate, please?

Thanks,
Roman.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

You have a bunch of options for this if I understand you well. You can
one of the following:

* use augeas with virtual resources
* use the concat module 
* use the standard template function with multiple arguments; look at
  http://docs.puppetlabs.com/guides/templating.html and scroll down to
  "Combining templates"

Best,  Nikola

On Sat, Dec 15, 2012 at 10:09:51PM -0800, Roman Shaposhnik
wrote:
> Hi!
> 
> I would appreciate any advice on the best practices
> on how to model a collection of services that each
> has its own configuration file, but also share a common
> one.
> 
> Now, the trouble is, that the common configuration file
> is not *really* just a place for the common configuration
> to reside, but also may have sections dedicated to holding
> properties of a particular service (just like /etc/puppet/puppet.conf
> has [main] [agent] and [master] sections).
> 
> Thus, in reality, the configuration properties are
> per service and whether they need to go into a common
> configuration file or a service-specific one is NOT cleanly
> partitioned and is pretty awkward to remember. Which
> means I really don''t want to expose the common config
> explicitly to the end user.
> 
> Instead, I''d like to expose the natural hierarchy of:
> 
> class service1($foo, $bar, $baz) {
> }
> .....
> class serviceN($qoo, $zoo) {
> }
> 
> but the question then becomes -- how can I model a
> common configuration file behind user''s back so that
> $foo, $bar and $qoo end up there.
> 
> At first I was thinking that I could utilize a singleton
> non-parameterized class and include it multiple
> times in all of the serviceX class definitions, but it
> seems I can''t pass any values into it.
> 
> I suppose I can still do that and create a skeleton
> of the common config in that singleton class
> that would later be manipulated either by augeas or
> concat patterns, but this seems to be pretty heavyweight.
> 
> What would you, guys, recommend?
> 
> Thanks,
> Roman.
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Dec 17, 2012 at 6:46 AM, Nikola Petrov <nikolavp@gmail.com>
wrote:
> You have a bunch of options for this if I understand you well. You can
> one of the following:
>
> * use augeas with virtual resources
> * use the concat module
Understood. I''m leaning heavily towards augeas at this point
(although I''m a bit scared of throwing various XML at it).

If stock augeas doesn''t pan out I think I''m going to
go with the most basic solution of all -- having a template
with easy markers and doing a bunch of exec''s of sed -i
(wrapped in a nice custom resource of course ;-)).
> * use the standard template function with multiple arguments; look at
>   http://docs.puppetlabs.com/guides/templating.html and scroll down to
>   "Combining templates"
You lost me there -- would you mind elaborating how these
type of ''Combining templates'' could be useful in my case?

Thanks,
Roman.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Dec 17, 2012 at 10:47:36PM -0800, Roman Shaposhnik
wrote:
> On Mon, Dec 17, 2012 at 6:46 AM, Nikola Petrov <nikolavp@gmail.com>
wrote:
> > You have a bunch of options for this if I understand you well. You can
> > one of the following:
> >
> > * use augeas with virtual resources
> > * use the concat module
> 
> Understood. I''m leaning heavily towards augeas at this point
> (although I''m a bit scared of throwing various XML at it).
> 
> If stock augeas doesn''t pan out I think I''m going to
> go with the most basic solution of all -- having a template
> with easy markers and doing a bunch of exec''s of sed -i
> (wrapped in a nice custom resource of course ;-)).
> 
> > * use the standard template function with multiple arguments; look at
> >   http://docs.puppetlabs.com/guides/templating.html and scroll down to
> >   "Combining templates"
> 
> You lost me there -- would you mind elaborating how these
> type of ''Combining templates'' could be useful in my case?
I hope that I understood you correctly in the first place - otherwise
ignore my advice :). You want to aggregate a common configuration file
for *all* of your services so you want one of the following:

* combine all config files to look them in one place - this is easily
done with templates if you have a bunch of service classes 

    class service1(...) {...}
    class service2(...) {...}

each of these has a template for it''s configuration(with params and
what
not). You then just call the template function with all of them and they
will be concatenated. In the case of the XML you might get clever and
include a pre and post templates to get a proper tree

* you want to provide a common config file for sane defaults and then
override some of the settings in each service. This can be done again
with templates - it is a bit harder and it might be slower for your
taste but it is still doable - you define some kind of virtual resource
for sub templates and then realize only those that you see fit. 

Note that both are almost the same(if not the same) as the concat
plugin. So you might just better use the already working implementation
from there.

-- 
Nikola

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.