Puppet users - Mar 2011 - Why is it so hard to make a sane nagios server config?

I''ve tried to achieve my overall goals with several different features
of Puppet, but I''ve hit a bit of a wall here.  I think it''s
time for me
to explain what I''m trying to accomplish:

	I want the enabling of a service in my manifests to configure
	the monitoring of that service by a nagios server, without
	needless repetition.

Let me explain how my non-automated nagios3 server is configured:

	Each service is declared once per type (or type+role, but there
	are very few types distinguished by role, so we''ll ignore that
	for now).  Services are members of hostgroups and servicegroups,
	and hosts join hostgroups in order to activate monitoring.
	Servicegroups are mostly used for sorting in the Web UI.

Now let''s look at how Puppet "wants" to configure Nagios:

	Each host exports a unique service for each of the checks it
	should be subject to.

Holy cow!  So if I have 25 checks per machine, and a thousand nodes,
that''s 25,000 entries!  We''re talking about a
nagios_services.cfg
measured in the tens of *megabytes*!  I''m a little stunned by this
pattern.

I had created a system whereby nodes exported concat::fragments
expressing desired membership in a hostgroup, and defines that created
services, hostgroups, and servicegroups only if they hadn''t already
been
made.  All this ground to a halt and with only 50 test nodes the
compilation of the nagios server''s catalog took 3 minutes on a
reasonably spec''d machine.  I have to assume that the AST''s
mass of
defines-of-defines was partly to blame.

I''d like to make it as simple as possible for my module writers:
"Just
use the nagios::nrpe_check define.  That''ll make the nrpe config
locally
and export everything needed to ensure this host is checked for your
module''s behaviors!" Forcing them to make a service check *here*
and a
hostgroup *there* and don''t forget to add the hostgroup to the node
that
used the class *there*… well it just opens up too many opportunities for
human fallibility.

So what are my options here?  I don''t want to probe the hosts to
determine what should be monitored, because the whole *point* of
monitoring is to alert you when your live system state isn''t what you
asked for.  I also worry that this Cartesian product of everything by
everyone will not scale acceptably into the future.

Fingers crossed that I''m just being dense and missed something big!

-- 
"Ill-informed qmail-bashing is better than no
qmail-bashing at all."
        --Don Marti

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Nick,
> I''ve tried to achieve my overall goals with several different
features
> of Puppet, but I''ve hit a bit of a wall here.  I think
it''s time for me
> to explain what I''m trying to accomplish:
> 
> 	I want the enabling of a service in my manifests to configure
> 	the monitoring of that service by a nagios server, without
> 	needless repetition.
> 
> Let me explain how my non-automated nagios3 server is configured:
> 
> 	Each service is declared once per type (or type+role, but there
> 	are very few types distinguished by role, so we''ll ignore that
> 	for now).  Services are members of hostgroups and servicegroups,
> 	and hosts join hostgroups in order to activate monitoring.
> 	Servicegroups are mostly used for sorting in the Web UI.
> 
> Now let''s look at how Puppet "wants" to configure
Nagios:
> 
> 	Each host exports a unique service for each of the checks it
> 	should be subject to.
> 
> Holy cow!  So if I have 25 checks per machine, and a thousand nodes,
> that''s 25,000 entries!  We''re talking about a
nagios_services.cfg
> measured in the tens of *megabytes*!  I''m a little stunned by this
> pattern.
> 
> I had created a system whereby nodes exported concat::fragments
> expressing desired membership in a hostgroup, and defines that created
> services, hostgroups, and servicegroups only if they hadn''t
already been
> made.  All this ground to a halt and with only 50 test nodes the
> compilation of the nagios server''s catalog took 3 minutes on a
> reasonably spec''d machine.  I have to assume that the
AST''s mass of
> defines-of-defines was partly to blame.
> 
> I''d like to make it as simple as possible for my module writers:
"Just
> use the nagios::nrpe_check define.  That''ll make the nrpe config
locally
> and export everything needed to ensure this host is checked for your
> module''s behaviors!" Forcing them to make a service check
*here* and a
> hostgroup *there* and don''t forget to add the hostgroup to the
node that
> used the class *there*… well it just opens up too many opportunities for
> human fallibility.
> 
> So what are my options here?  I don''t want to probe the hosts to
> determine what should be monitored, because the whole *point* of
> monitoring is to alert you when your live system state isn''t what
you
> asked for.  I also worry that this Cartesian product of everything by
> everyone will not scale acceptably into the future.
> 
> Fingers crossed that I''m just being dense and missed something
big!
I''m afraid not.

Most of this past week I spent building my Puppet-induced Nagios
configuration, and after exploring my options, I decided that the
hostgroup-based configuration you describe is the only sensible way.

I did exactly what you did: use exported concat-fragments to collect the
hostgroups on the puppetmaster and then use generate() to provision the
hostgroups parameter of the nagios_host.

Catalog runs on the Nagios servers typically take less than a minute to
run, but I only have about 50 hosts defined so far. I hope that this
scheme will be sustainable.

The biggest downside of my system so far, is the fact that, in a
worst-case scenario, it takes 90 minutes for a configuration change to
propagate to Nagios.

Regards,
Martijn.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 11-03-11 12:46, Martijn Grendelman wrote:
[snip]
> I did exactly what you did: use exported concat-fragments to collect the
> hostgroups on the puppetmaster and then use generate() to provision the
> hostgroups parameter of the nagios_host.
[snip]
> The biggest downside of my system so far, is the fact that, in a
> worst-case scenario, it takes 90 minutes for a configuration change to
> propagate to Nagios.
I now replaced the exported concat fragments with local concat fragments
and a custom fact, which reads the result of the concat and joins it into
a comma-separated string.

Dependencies make sure that the concats are done before the nagios_hosts
are exported.

This removes the need to collect anything on the puppetmaster and reduces
the time needed to propagate a configuration change to Nagios in the worst
case to 30 minutes.

Best regards,
Martijn.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Martijn Grendelman:
> I''m afraid not.
This is distressing news.

I feel like there are two things conspiring to defeat me, here:  one is
that nagios resource types like to enumerate *children* instead of
*parent* relationships.  Having to maintain these static lists of
"members" is part of the crisis.  My kingdom for a cons cell!

The second problem is that Puppet doesn''t provide me with any efficient
mechanism for *deduplicating* collected resources.  I appreciate that
there are race conditions that make it impossible to determine which
"version" of a resource is the one that gets instantiated.  

I''d hope, though, that when all parameters are equal, that there could
be some way to say "uniq these, thanks."  Maybe something like
puppet::concat that removes instead of adds.

-- 
"N''aimez pas votre voiture?
Alor, l''heure est arrive pour la brulé!"
        -- Mark Jaroski

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

My setup also has a worst-case propagation delay of 90 minutes.  I have a
custom fact that collects all of the information in classes.txt on the
client.  That, in turn, is used (for Nagios) by a custom parser function
that produces the hostgroup list for when the nagios_host resource is
exported.  It''s not optimal, but it''s fully automatic.  In the
rare case
where I want/need the info to propagate immediately, I can run "puppet
agent
--test" twice on the client and then once more on the Nagios server.

All of the services, as in your case, are generated as members of
hostgroups.

On Fri, Mar 11, 2011 at 6:14 AM, Martijn Grendelman
<martijn@iphion.nl>wrote:
> On 11-03-11 12:46, Martijn Grendelman wrote:
> [snip]
>
> > I did exactly what you did: use exported concat-fragments to collect
the
> > hostgroups on the puppetmaster and then use generate() to provision
the
> > hostgroups parameter of the nagios_host.
>
> [snip]
>
> > The biggest downside of my system so far, is the fact that, in a
> > worst-case scenario, it takes 90 minutes for a configuration change to
> > propagate to Nagios.
>
> I now replaced the exported concat fragments with local concat fragments
> and a custom fact, which reads the result of the concat and joins it into
> a comma-separated string.
>
> Dependencies make sure that the concats are done before the nagios_hosts
> are exported.
>
> This removes the need to collect anything on the puppetmaster and reduces
> the time needed to propagate a configuration change to Nagios in the worst
> case to 30 minutes.
>
> Best regards,
> Martijn.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

A cons cell isn''t actually that hard to produce in Puppet: you can use
a
hash.  Each key is the member of the cell you want to keep, while the value
is a throwaway.  But.  Puppet''s non-determinate evaluation order is
pretty
much guaranteed to bite you in the butt.  You can really only get the
"full"
information on the puppet client after the catalog is fully generated, which
means that all your templated or variable-substituted strings are already
fully formed.  You have to have a custom fact on the child which gets its
information from /var/lib/puppet/state.

I''m not sure what you mean by deduplicating.  Puppet won''t let
you have
duplicate *anything*, exported or not.  If you mean "filtering",
that''s
doable, but any non-trivial filtering rule will require to you write evil
Puppet functions that hook into the internals.  If you could provide us a
reasonably detailed example, perhaps we could be of more help.

On Fri, Mar 11, 2011 at 6:29 AM, Nick Moffitt <nick@zork.net> wrote:
> Martijn Grendelman:
> > I''m afraid not.
>
> This is distressing news.
>
> I feel like there are two things conspiring to defeat me, here:  one is
> that nagios resource types like to enumerate *children* instead of
> *parent* relationships.  Having to maintain these static lists of
> "members" is part of the crisis.  My kingdom for a cons cell!
>
> The second problem is that Puppet doesn''t provide me with any
efficient
> mechanism for *deduplicating* collected resources.  I appreciate that
> there are race conditions that make it impossible to determine which
> "version" of a resource is the one that gets instantiated.
>
> I''d hope, though, that when all parameters are equal, that there
could
> be some way to say "uniq these, thanks."  Maybe something like
> puppet::concat that removes instead of adds.
>
> --
> "N''aimez pas votre voiture?
> Alor, l''heure est arrive pour la brulé!"
>        -- Mark Jaroski
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Brian Gallew:
> A cons cell isn''t actually that hard to produce in Puppet: you can
use
> a hash.  Each key is the member of the cell you want to keep, while
> the value is a throwaway.  
Sure.  I was actually talking about Nagios there, not Puppet.
> I''m not sure what you mean by deduplicating.  Puppet
won''t let you
> have duplicate *anything*, exported or not.  If you mean
"filtering",
> that''s doable, but any non-trivial filtering rule will require to
you
> write evil Puppet functions that hook into the internals.  If you
> could provide us a reasonably detailed example, perhaps we could be of
> more help.
All right, let''s say I have a nagios check that has a new hostgroup. 
My
nagios configs fall out as a consequence of *use* to ensure that they
match the manifests 100%.  So there''s a define in the foo::monitoring
class that updates the local NRPE configs and then exports:

	1. A request that a service exist, in a hostgroup provided, to
	   probe this check over NRPE.
	2. A request that the provided hostgroup exist.
	3. A request that the host''s server-side definition include
	   membership in the provided hostgroup.

1. and 2. need to be unique globally.  3. only needs to be unique
per-host.  All would benefit from behaving like the invocation of a
class, or the resolution of notifies down to a single event.  Having
the ability to export "ensure X is Y" rather than "here is an
X+Z=Y just
like thousands of others" would be helpful to me.

-- 
"Ill-informed qmail-bashing is better than no
qmail-bashing at all."
        --Don Marti

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mar 11, 2011, at 7:52 AM, Nick Moffitt wrote:
>> I''m not sure what you mean by deduplicating.  Puppet
won''t let you
>> have duplicate *anything*, exported or not.  If you mean
"filtering",
>> that''s doable, but any non-trivial filtering rule will require
to you
>> write evil Puppet functions that hook into the internals.  If you
>> could provide us a reasonably detailed example, perhaps we could be of
>> more help.
> 
> All right, let''s say I have a nagios check that has a new
hostgroup.  My
> nagios configs fall out as a consequence of *use* to ensure that they
> match the manifests 100%.  So there''s a define in the
foo::monitoring
> class that updates the local NRPE configs and then exports:
> 
> 	1. A request that a service exist, in a hostgroup provided, to
> 	   probe this check over NRPE.
> 	2. A request that the provided hostgroup exist.
> 	3. A request that the host''s server-side definition include
> 	   membership in the provided hostgroup.
> 
> 1. and 2. need to be unique globally.  3. only needs to be unique
> per-host.  All would benefit from behaving like the invocation of a
> class, or the resolution of notifies down to a single event.  Having
> the ability to export "ensure X is Y" rather than "here is
an X+Z=Y just
> like thousands of others" would be helpful to me.
Oooooh, I see.  You want to define the Nagios service in the same place the
service is declared, and the hostgroup in the class.  Yeah, that would be a
sweet way to do it, but Puppet''s manifest methodology is completely
incapable of doing it.  I''ve always wanted a reasonable way to collect
the hostgroups from the various service/host definitions, but that''s
never been possible without deep magic.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I gave up and stopped trying to obfuscate the nagios configuration in
puppet. I simply have puppet pushing out anything I drop into
/etc/nagios/conf.d, and then doing a nagios restart. This makes it very easy
to add hosts and services because a script can read the node manifest and
drop the necessary files into modules/nagios/etc/nagios/conf.d.

Doug.

On Fri, Mar 11, 2011 at 7:22 AM, Brian Gallew <geek@gallew.org> wrote:
> My setup also has a worst-case propagation delay of 90 minutes.  I have a
> custom fact that collects all of the information in classes.txt on the
> client.  That, in turn, is used (for Nagios) by a custom parser function
> that produces the hostgroup list for when the nagios_host resource is
> exported.  It''s not optimal, but it''s fully automatic. 
In the rare case
> where I want/need the info to propagate immediately, I can run "puppet
agent
> --test" twice on the client and then once more on the Nagios server.
>
> All of the services, as in your case, are generated as members of
> hostgroups.
>
>
> On Fri, Mar 11, 2011 at 6:14 AM, Martijn Grendelman
<martijn@iphion.nl>wrote:
>
>> On 11-03-11 12:46, Martijn Grendelman wrote:
>> [snip]
>>
>> > I did exactly what you did: use exported concat-fragments to
collect the
>> > hostgroups on the puppetmaster and then use generate() to
provision the
>> > hostgroups parameter of the nagios_host.
>>
>> [snip]
>>
>> > The biggest downside of my system so far, is the fact that, in a
>> > worst-case scenario, it takes 90 minutes for a configuration
change to
>> > propagate to Nagios.
>>
>> I now replaced the exported concat fragments with local concat
fragments
>> and a custom fact, which reads the result of the concat and joins it
into
>> a comma-separated string.
>>
>> Dependencies make sure that the concats are done before the
nagios_hosts
>> are exported.
>>
>> This removes the need to collect anything on the puppetmaster and
reduces
>> the time needed to propagate a configuration change to Nagios in the
worst
>> case to 30 minutes.
>>
>> Best regards,
>> Martijn.
>>
>> --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> puppet-users+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>


-- 
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mar 11, 10:29 am, Brian Gallew <g...@gallew.org>
wrote:
> I''m not sure what you mean by deduplicating.  Puppet
won''t let you have
> duplicate *anything*, exported or not.
To be more precise, Puppet won''t let you declare two or more resources
having the same name or title for the same node.  That could actually
help you, depending on what exactly you mean by "deduplicating".
>  If you mean "filtering", that''s
> doable, but any non-trivial filtering rule will require to you write evil
> Puppet functions that hook into the internals.
Maybe.  There is no supported way to remove a resource declaration
from a node''s catalog during the compilation of that catalog, nor, at
a higher level, to "uninclude" a class that you have included.  BUT,
you can include the same class any number of times without inducing
duplication of the class or its contents, and you can realize /
collect the same resource in multiple places or from multiple sources
without causing it to be duplicated.

Basically, Puppet does not provide "deduplication" in part because it
does not provide duplication in the first place.  The trick, of
course, is all in what Puppet considers to be duplicate, and to the
best of my knowledge, that''s entirely based on resource names and
titles.

In addition, Puppet does provide some filtering capability for
resource collection and realization.  Filter criteria based on
resource properties can be used to select specific resources to
collect / realize, e.g.:

File<<| tag == ''''web-file" |>>

What Puppet does not offer is any ability to filter resources after
realization / collection, or based on what *other* resources are being
realized / collected.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.