Puppet users - Apr 2012 - Parametrized classes, definitions and structure

Hi,

I''ve written a set of Puppet modules to manage servers, but am
slightly concerned about the structure I''ve used as it seems to differ
significantly from anything else I''ve seen in Puppet Forge/elsewhere
on the internet.

I''ve made extensive use of definitions so that defaults for modules
can be overridden in the node manifest for a server. For example:

node examplenode.example.com {

include apache
include mysql

define apache::config {
 listenport => 8080
}

define mysql::config
}

So in each module, I have a config definition which must be called in
each node manifest, even if you don''t want to override any defaults
(as in the case of MySQL above). I haven''t used parametrized classes
at all.

It all works fine, but are there any disadvantages to the sort of
structure I''m using? Should I be using more parametrized classes
instead of definitions?

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Apr 24, 9:13 am, Andy Taylor <andytaylo...@gmail.com>
wrote:
> Hi,
>
> I''ve written a set of Puppet modules to manage servers, but am
> slightly concerned about the structure I''ve used as it seems to
differ
> significantly from anything else I''ve seen in Puppet
Forge/elsewhere
> on the internet.
>
> I''ve made extensive use of definitions so that defaults for
modules
> can be overridden in the node manifest for a server. For example:
>
> node examplenode.example.com {
>
> include apache
> include mysql
>
> define apache::config {
>  listenport => 8080
>
> }
>
> define mysql::config
>
> }
>
> So in each module, I have a config definition which must be called in
> each node manifest, even if you don''t want to override any
defaults
> (as in the case of MySQL above). I haven''t used parametrized
classes
> at all.
>
> It all works fine,

That it works well is the most important criterion.  With that
said, ...

> but are there any disadvantages to the sort of
> structure I''m using? Should I be using more parametrized classes
> instead of definitions?

One problem with using definitions, especially definitions that are
required to be instantiated for multiple classes to work, is that each
instantiation of a definition is a separate resource.  Each must
therefore have a unique name among the resources of its type, and each
will be applied separately.  This presents either a management problem
(who is responsible for instantiating the definition in any given set
of circumstances?) or an efficiency problem (equivalent definition
instances are redundantly applied) or both.

Parameterized classes do not present that efficiency problem because
all classes are singletons.  That intensifies the management problem,
however, because unlike ordinary classes, each parameterized class can
be declared only once for any given node.  In general, few problems
are best solved by parameterized classes.

You should use defined types only when you need *resources* for which
there is no native type available.  In that case you should know that
you want resources rather than classes.  That might be because nodes
can have more than one instance of your ''thing'', because you
want
support for virtual or exported ''things'', or (best) simply
because
your ''thing'' is better characterized as something nodes *have*
than as
something that they *are*.

On the other hand, you should use (ordinary) classes where you want
their singleton nature -- especially idempotency, which is a far more
useful property than many appreciate.  You must also use classes where
you need your ''thing'' to be declared via an ENC (the C is for
CLASSifier, after all).  You should generally use classes where they
represent an aspect of nodes'' type / kind / role / nature, as opposed
to a tangible thing you can actually manage.

As for default values and value overriding, if your ''things''
are a
better fit for classes than for definitions, then you should be
looking at external data and Hiera instead of parameterization.  Even
for any definitions you retain, it might be to your advantage to
define certain default values via external data than to code them into
your manifests, or even to remove parameters in favor of external
data.  Really, it is best if your manifests contain no node- or site-
specific data.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thank you very much for your detailed reply John. I hadn''t looked at
Hiera before, it looks very interesting. On the point of it''s not good
to include node/site data in manifests, it''s kind of essential in my
eyes to the setup I am trying to implement.

At my company, one server may serve a site which has very different
requirements to another, so I really needed something that was very
flexible. To expand on the structure I outlined in my previous post:
each server has it''s own node.pp file. If you need to change something
(e.g. MySQL settings, which will probably the most frequent one that
needs changing), you can simply edit that server''s node manifest file.
The great advantage here is that each site''s node manifest serves as
documentation of what changes have been made to it. It''s also very
easy to read and understand for someone new to Puppet.

From some quick reading of the Hiera posts on the Puppet Labs blog, I
guess I could simply have a yaml file for each server with overrides
instead of doing it via the node manifest. However, I don''t really see
the advantage of this approach, except for it would cut out some of
the clutter in my Puppet modules. The idea of a hierarchical set of
data being considered in line with Facter facts is very interesting
though, especially if I combined it with some custom facts of my own;
I will do some more reading on that.

With the class/definition point, I am wherever possible using classes,
as usually I''m dealing with the nature of a node (e.g. you are a
webserver, database server etc.) The only case I have used definitions
is for these configuration file overrides. So if I''m understanding
your comments on class/definition use properly, I think I''m sort of
taking the right approach to it...

The key concern for me at the moment is flexibility and scalability...
none of my stuff is live yet, and I want to make sure I''m using best
practices before I put it all in place. :)

On Apr 24, 5:13 pm, jcbollinger <John.Bollin...@stJude.org>
wrote:
> On Apr 24, 9:13 am, Andy Taylor <andytaylo...@gmail.com> wrote:
>
>
>
>
>
>
>
>
>
> > Hi,
>
> > I''ve written a set of Puppet modules to manage servers, but
am
> > slightly concerned about the structure I''ve used as it seems
to differ
> > significantly from anything else I''ve seen in Puppet
Forge/elsewhere
> > on the internet.
>
> > I''ve made extensive use of definitions so that defaults for
modules
> > can be overridden in the node manifest for a server. For example:
>
> > node examplenode.example.com {
>
> > include apache
> > include mysql
>
> > define apache::config {
> >  listenport => 8080
>
> > }
>
> > define mysql::config
>
> > }
>
> > So in each module, I have a config definition which must be called in
> > each node manifest, even if you don''t want to override any
defaults
> > (as in the case of MySQL above). I haven''t used parametrized
classes
> > at all.
>
> > It all works fine,
>
> That it works well is the most important criterion.  With that
> said, ...
>
> > but are there any disadvantages to the sort of
> > structure I''m using? Should I be using more parametrized
classes
> > instead of definitions?
>
> One problem with using definitions, especially definitions that are
> required to be instantiated for multiple classes to work, is that each
> instantiation of a definition is a separate resource.  Each must
> therefore have a unique name among the resources of its type, and each
> will be applied separately.  This presents either a management problem
> (who is responsible for instantiating the definition in any given set
> of circumstances?) or an efficiency problem (equivalent definition
> instances are redundantly applied) or both.
>
> Parameterized classes do not present that efficiency problem because
> all classes are singletons.  That intensifies the management problem,
> however, because unlike ordinary classes, each parameterized class can
> be declared only once for any given node.  In general, few problems
> are best solved by parameterized classes.
>
> You should use defined types only when you need *resources* for which
> there is no native type available.  In that case you should know that
> you want resources rather than classes.  That might be because nodes
> can have more than one instance of your ''thing'', because
you want
> support for virtual or exported ''things'', or (best)
simply because
> your ''thing'' is better characterized as something nodes
*have* than as
> something that they *are*.
>
> On the other hand, you should use (ordinary) classes where you want
> their singleton nature -- especially idempotency, which is a far more
> useful property than many appreciate.  You must also use classes where
> you need your ''thing'' to be declared via an ENC (the C is
for
> CLASSifier, after all).  You should generally use classes where they
> represent an aspect of nodes'' type / kind / role / nature, as
opposed
> to a tangible thing you can actually manage.
>
> As for default values and value overriding, if your
''things'' are a
> better fit for classes than for definitions, then you should be
> looking at external data and Hiera instead of parameterization.  Even
> for any definitions you retain, it might be to your advantage to
> define certain default values via external data than to code them into
> your manifests, or even to remove parameters in favor of external
> data.  Really, it is best if your manifests contain no node- or site-
> specific data.
>
> John
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Apr 24, 12:31 pm, Andy Taylor <andytaylo...@gmail.com>
wrote:
> Thank you very much for your detailed reply John. I hadn''t looked
at
> Hiera before, it looks very interesting. On the point of it''s not
good
> to include node/site data in manifests, it''s kind of essential in
my
> eyes to the setup I am trying to implement.

Unless the setup you''re trying to implement is "one with
node-specific
data in its manifests", you are mistaken.  Or perhaps you
misunderstand me.  I am not at all saying that your manifests should
not rely on node- or site-specific data, only that those data do not
belong in the manifest files themselves.  Instead, put them in an
external data source, such as one accessed via Hiera.  That''s why I
mentioned the principle in the first place.

> At my company, one server may serve a site which has very different
> requirements to another, so I really needed something that was very
> flexible. To expand on the structure I outlined in my previous post:
> each server has it''s own node.pp file. If you need to change
something
> (e.g. MySQL settings, which will probably the most frequent one that
> needs changing), you can simply edit that server''s node manifest
file.
> The great advantage here is that each site''s node manifest serves
as
> documentation of what changes have been made to it. It''s also very
> easy to read and understand for someone new to Puppet.

It is somewhat better for all node-specific data to be associated with
nodes'' declarations, and for all site-specific data to be in site.pp,
than for such data to be spread throughout your manifests.  It still
presents problems, however.

In particular, it''s brittle to put the node-specific data into global
variable declarations.  The only way that can work is to use
conditional ''import''ing of the appropriate node file (else
different
nodes'' declarations will clash).  ''Import'' itself is
best avoided as
much as possible, though you''d be on the borderline of one of the few
reasonable use cases.

On the other hand, putting node-specific data into node-scoped
variables limits you to either relying on dynamic scoping (which is
being phased out) or using those variables only directly inside your
node declaration.

Using literal node-specific values directly as function, class, or
defined type parameters is a bit better, but it requires you to
forward those values throughout the rest of your manifests wherever
they are needed.  That complicates everything, requiring extra
parameters, and perhaps even preventing you from using non-
parameterized classes in some places.

> From some quick reading of the Hiera posts on the Puppet Labs blog, I
> guess I could simply have a yaml file for each server with overrides
> instead of doing it via the node manifest.

Pretty much, yes.

> However, I don''t really see
> the advantage of this approach, except for it would cut out some of
> the clutter in my Puppet modules.

More than that:

1) It could greatly simplify and clarify some of your manifests.  Your
node declarations especially would benefit, but reducing
parameterization elsewhere clarifies your manifests by focusing
attention on the big picture -- roles, features, capabilities -- and
away from the details.

2) It can help you obtain the benefits of using classes (where they
are appropriate) without requiring class parameterization (which would
lose you some of those same benefits).

3) From a change management perspective, separating the data allows
you to more easily recognize the nature of any given change.

> The idea of a hierarchical set of
> data being considered in line with Facter facts is very interesting
> though, especially if I combined it with some custom facts of my own;
> I will do some more reading on that.
>
> With the class/definition point, I am wherever possible using classes,
> as usually I''m dealing with the nature of a node (e.g. you are a
> webserver, database server etc.) The only case I have used definitions
> is for these configuration file overrides. So if I''m understanding
> your comments on class/definition use properly, I think I''m sort
of
> taking the right approach to it...

It sounds like you are generally doing the right thing, but you did
ask about those definitions, and I don''t think they''re good
style.  As
I said before, however, the most important criterion is that it works
well.

> The key concern for me at the moment is flexibility and scalability...
> none of my stuff is live yet, and I want to make sure I''m using
best
> practices before I put it all in place. :)

I think making your manifests as generic as possible, separating data
from manifests, and using a flexible external data source such as
Hiera is a win on both flexibility and scalability.  I doubt you''ll
find a consensus on best practices, but I am confident that you will
find others who also consider those to be Puppet best-practices
principles.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 4/24/2012 10:31 AM, Andy Taylor wrote:
> Thank you very much for your detailed reply John. I hadn''t looked
at
> Hiera before, it looks very interesting. On the point of it''s not
good
> to include node/site data in manifests, it''s kind of essential in
my
> eyes to the setup I am trying to implement.
>
> At my company, one server may serve a site which has very different
> requirements to another, so I really needed something that was very
> flexible. To expand on the structure I outlined in my previous post:
> each server has it''s own node.pp file. If you need to change
something
> (e.g. MySQL settings, which will probably the most frequent one that
> needs changing), you can simply edit that server''s node manifest
file.
> The great advantage here is that each site''s node manifest serves
as
> documentation of what changes have been made to it. It''s also very
> easy to read and understand for someone new to Puppet.
>
>  From some quick reading of the Hiera posts on the Puppet Labs blog, I
> guess I could simply have a yaml file for each server with overrides
> instead of doing it via the node manifest. However, I don''t really
see
> the advantage of this approach, except for it would cut out some of
> the clutter in my Puppet modules. The idea of a hierarchical set of
> data being considered in line with Facter facts is very interesting
> though, especially if I combined it with some custom facts of my own;
> I will do some more reading on that.
>
> With the class/definition point, I am wherever possible using classes,
> as usually I''m dealing with the nature of a node (e.g. you are a
> webserver, database server etc.) The only case I have used definitions
> is for these configuration file overrides. So if I''m understanding
> your comments on class/definition use properly, I think I''m sort
of
> taking the right approach to it...
>
> The key concern for me at the moment is flexibility and scalability...
> none of my stuff is live yet, and I want to make sure I''m using
best
> practices before I put it all in place. :)
	Mr Bollinger did his usual excellent job of thoroughly explaining the 
problem space in other parts of this thread. I''d simply add that using
a
richer data store, such as Hiera, can keep you from doing complicated 
gymnastics in your module code.

I''m using a hiera.yaml file similar to the following with fqdn at the 
top and common at the bottom. I''ve added a role fact which is reading 
/etc/role on the client. You can use any Facter fact such as %{domain} 
or write your own. %{location} or %{datacenter} seems to be a common one 
for admins with multiple sites.

:hierarchy:
     - %{fqdn}
     - %{role}_%{environment}
     - %{role}
     - %{environment}
     - common

In my mysql::data class I add a variable and give it a default of 256M.

$innodb_buffer_pool_size =
hiera(''mysql_innodb_buffer_pool_size'',
''256M'')

I can then set the following to deal with the specifics of my environment.
   512M in my prod environment, production.yaml
   4G in role db, db.yaml
   1.5G on role db in environment stage, db_stage.yaml
   8G on server db01.main.sfo, db01.main.sfo.mydomain.com.yaml

Going one step further you can use hiera_array to merge data. For 
example you might allow a set of servers to monitor all servers and then 
allow the local monitors as well. Assuming a location fact and the 
following data files

common.yaml
monitoring_hosts:
    - ''mon01.ord''
    - ''mon02.ord''

lax.yaml
monitoring_hosts:
    - ''mon01.lax''

class someclass::data {
$hosts = hiera_array(''monitoring_hosts'')
}

someclass::data::hosts =
[''mon01.lax'',''mon01.ord'',''mon02.ord'']

Hopefully this should give you some ideas of how to use Hiera in your 
new system.

Ramin

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks John for yet another very useful reply, it is much appreciated.
I''m going to do some more reading on Hiera and look at reworking my
modules.

And thank you Ramin for some examples of Hiera in practice, looks very
interesting! Going to have a play around with it next week :)

Thanks!

On Apr 25, 7:01 pm, Ramin K <ramin-l...@badapple.net>
wrote:
> On 4/24/2012 10:31 AM,AndyTaylorwrote:
>
>
>
>
>
>
>
>
>
> > Thank you very much for your detailed reply John. I hadn''t
looked at
> > Hiera before, it looks very interesting. On the point of it''s
not good
> > to include node/site data in manifests, it''s kind of
essential in my
> > eyes to the setup I am trying to implement.
>
> > At my company, one server may serve a site which has very different
> > requirements to another, so I really needed something that was very
> > flexible. To expand on the structure I outlined in my previous post:
> > each server has it''s own node.pp file. If you need to change
something
> > (e.g. MySQL settings, which will probably the most frequent one that
> > needs changing), you can simply edit that server''s node
manifest file.
> > The great advantage here is that each site''s node manifest
serves as
> > documentation of what changes have been made to it. It''s also
very
> > easy to read and understand for someone new to Puppet.
>
> >  From some quick reading of the Hiera posts on the Puppet Labs blog, I
> > guess I could simply have a yaml file for each server with overrides
> > instead of doing it via the node manifest. However, I don''t
really see
> > the advantage of this approach, except for it would cut out some of
> > the clutter in my Puppet modules. The idea of a hierarchical set of
> > data being considered in line with Facter facts is very interesting
> > though, especially if I combined it with some custom facts of my own;
> > I will do some more reading on that.
>
> > With the class/definition point, I am wherever possible using classes,
> > as usually I''m dealing with the nature of a node (e.g. you
are a
> > webserver, database server etc.) The only case I have used definitions
> > is for these configuration file overrides. So if I''m
understanding
> > your comments on class/definition use properly, I think I''m
sort of
> > taking the right approach to it...
>
> > The key concern for me at the moment is flexibility and scalability...
> > none of my stuff is live yet, and I want to make sure I''m
using best
> > practices before I put it all in place. :)
>
>         Mr Bollinger did his usual excellent job of thoroughly explaining
the
> problem space in other parts of this thread. I''d simply add that
using a
> richer data store, such as Hiera, can keep you from doing complicated
> gymnastics in your module code.
>
> I''m using a hiera.yaml file similar to the following with fqdn at
the
> top and common at the bottom. I''ve added a role fact which is
reading
> /etc/role on the client. You can use any Facter fact such as %{domain}
> or write your own. %{location} or %{datacenter} seems to be a common one
> for admins with multiple sites.
>
> :hierarchy:
>      - %{fqdn}
>      - %{role}_%{environment}
>      - %{role}
>      - %{environment}
>      - common
>
> In my mysql::data class I add a variable and give it a default of 256M.
>
> $innodb_buffer_pool_size =
hiera(''mysql_innodb_buffer_pool_size'',
''256M'')
>
> I can then set the following to deal with the specifics of my environment.
>    512M in my prod environment, production.yaml
>    4G in role db, db.yaml
>    1.5G on role db in environment stage, db_stage.yaml
>    8G on server db01.main.sfo, db01.main.sfo.mydomain.com.yaml
>
> Going one step further you can use hiera_array to merge data. For
> example you might allow a set of servers to monitor all servers and then
> allow the local monitors as well. Assuming a location fact and the
> following data files
>
> common.yaml
> monitoring_hosts:
>     - ''mon01.ord''
>     - ''mon02.ord''
>
> lax.yaml
> monitoring_hosts:
>     - ''mon01.lax''
>
> class someclass::data {
> $hosts = hiera_array(''monitoring_hosts'')
>
> }
>
> someclass::data::hosts =
[''mon01.lax'',''mon01.ord'',''mon02.ord'']
>
> Hopefully this should give you some ideas of how to use Hiera in your
> new system.
>
> Ramin
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.