Puppet users - Dec 2013 - testing and exported (nagios) resources

Hello,

I have 3 puppet stacks (master, puppetdb, enc) - dev, test/qa and prod.
Dev is used for initial development and testing of code (including
puppet), which is then promoted to test and then prod.

I''d like to start using the nagios types to configure monitoring, via
exported resources (yes I''m aware of the issues with the builtins, but
they''ll have to do for now). I only have one Nagios server, and
I''d like
to reliably monitor at least some stuff on the dev and test puppet
nodes. So... setting up all 3 puppet stacks to export resources that are
realized somehow on the Nagios server isn''t a possibility, as bad
manifests/modules could affect the monitoring of one of the dev or test
hosts.

What''s the safe way to "freeze" exported resources, or
prevent them from
being changed? The best that I can come up with so far is to have the
nagios server connected to the production puppetmaster, and when I want
to update the (exported resource) monitoring configuration for one of
the dev or test nodes, have to do a one-time run on each node in
question against the prod puppet master.

Any other thoughts or theories?

Thanks,
Jason Antman

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/529D363C.4030202%40jasonantman.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

I must be missing an essential piece here.

All three of your puppet stack nodes must be present in each instance,
no? The production master manages all three masters, normally. To change
monitoring of either of them, you update the production manifests.

Of course, if you implement some new monitoring feature on the dev
master, you must have that node run puppet against its local dev master
to export resources, then the nagios server also against the dev master
to import them. But that is just the usual dev workflow, I assume.

So I suspect things aren''t so simple for you. I just don''t see
in what
manner.

Thanks in advance for any clarification,
Felix

On 12/03/2013 02:39 AM, Jason Antman wrote:
> Hello,
> 
> I have 3 puppet stacks (master, puppetdb, enc) - dev, test/qa and prod.
> Dev is used for initial development and testing of code (including
> puppet), which is then promoted to test and then prod.
> 
> I''d like to start using the nagios types to configure monitoring,
via
> exported resources (yes I''m aware of the issues with the builtins,
but
> they''ll have to do for now). I only have one Nagios server, and
I''d like
> to reliably monitor at least some stuff on the dev and test puppet
> nodes. So... setting up all 3 puppet stacks to export resources that are
> realized somehow on the Nagios server isn''t a possibility, as bad
> manifests/modules could affect the monitoring of one of the dev or test
> hosts.
> 
> What''s the safe way to "freeze" exported resources, or
prevent them from
> being changed? The best that I can come up with so far is to have the
> nagios server connected to the production puppetmaster, and when I want
> to update the (exported resource) monitoring configuration for one of
> the dev or test nodes, have to do a one-time run on each node in
> question against the prod puppet master.
> 
> Any other thoughts or theories?
> 
> Thanks,
> Jason Antman
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/529F2CA4.2040203%40alumni.tu-berlin.de.
For more options, visit https://groups.google.com/groups/opt_out.

On 12/04/2013 08:22 AM, Felix Frank wrote:
> Hi,
>
> I must be missing an essential piece here.
>
> All three of your puppet stack nodes must be present in each instance,
> no? The production master manages all three masters, normally. To change
> monitoring of either of them, you update the production manifests.
What do you mean by "present in each instance"? Each stack is 
self-contained - it has its own ENC, PuppetDB, and Master. Each stack''s
master manages itself and its stack, out of a "production" environment
(git master). The logic behind this is that if the production stack 
suffers an outage (i.e. hardware failure) the ENC data is imported to 
the test stack, and nodes are seamlessly moved over. Yes, I''m aware of 
the tradeoff that certain things aren''t constrained by environment, and
bad code in one environment on the dev or test masters could bring down 
that stack.

I''m not just worried about monitoring the puppet stack itself,
I''m also
worried about monitoring the nodes. I.e. the dev stack has a bunch of 
VMs that exist to test code, and they need to be monitored in
Nagios.
>
> Of course, if you implement some new monitoring feature on the dev
> master, you must have that node run puppet against its local dev master
> to export resources, then the nagios server also against the dev master
> to import them. But that is just the usual dev workflow, I assume.
Yeah, that''s understood. But what about the production monitoring?
I''d
need to run all of the nodes in the environment against the production 
master to actually export the nagios configs to the nagios server... or 
else I''d need (what I''m asking about) some way of exporting
the Nagios
configs from the dev and test masters to the Nagios server, but only for 
one environment (broken in puppet... exported resources don''t work this
way) or only when manually requested...
>
> So I suspect things aren''t so simple for you. I just
don''t see in what
> manner.
>
> Thanks in advance for any clarification,
> Felix
>
> On 12/03/2013 02:39 AM, Jason Antman wrote:
>> Hello,
>>
>> I have 3 puppet stacks (master, puppetdb, enc) - dev, test/qa and prod.
>> Dev is used for initial development and testing of code (including
>> puppet), which is then promoted to test and then prod.
>>
>> I''d like to start using the nagios types to configure
monitoring, via
>> exported resources (yes I''m aware of the issues with the
builtins, but
>> they''ll have to do for now). I only have one Nagios server,
and I''d like
>> to reliably monitor at least some stuff on the dev and test puppet
>> nodes. So... setting up all 3 puppet stacks to export resources that
are
>> realized somehow on the Nagios server isn''t a possibility, as
bad
>> manifests/modules could affect the monitoring of one of the dev or test
>> hosts.
>>
>> What''s the safe way to "freeze" exported resources,
or prevent them from
>> being changed? The best that I can come up with so far is to have the
>> nagios server connected to the production puppetmaster, and when I want
>> to update the (exported resource) monitoring configuration for one of
>> the dev or test nodes, have to do a one-time run on each node in
>> question against the prod puppet master.
>>
>> Any other thoughts or theories?
>>
>> Thanks,
>> Jason Antman
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/52A0A207.8080406%40jasonantman.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ah, so this is the meat of your problem. I think I get it now.

Tough call. Seeing as running multiple puppetdbs in parallel is not
really an expected scenario, there is quite possibly no way to share the
required resources between them.

Are nagios resources being actively removed from your nagios server when
they have not been exported to the puppet master being queried? I would
have thought that they just cease to be in management and left alone by
puppet.

On 12/05/2013 04:55 PM, Jason Antman wrote:
> Yeah, that''s understood. But what about the production monitoring?
I''d
> need to run all of the nodes in the environment against the production
> master to actually export the nagios configs to the nagios server...
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/52A0AC91.4010603%40alumni.tu-berlin.de.
For more options, visit https://groups.google.com/groups/opt_out.

Three thoughts. The first would be to have node definitions for monitor on
both dev and test environments, doing a minimal amount of work to generate
nagios host definitions in a disjoint directory that you include in your
nagios config.

So:
/var/lib/{puppet,puppet-dev,puppet-test}
/etc/{puppet,puppet-dev,puppet-test}
/etc/nagios/object/{dev,test}

To further isolate your nagios box from harm, dev and test environment runs
can be done from an unprivileged user and puppet agent runs can be tied to
host/service additions/removals in dev/test. I know this idea directly
violates you saying "So... setting up all 3 puppet stacks to export
resources that are realized somehow on the Nagios server isn''t a
possibility, as bad manifests/modules could affect the monitoring of one of
the dev or test hosts." but it seems the least-harm least-gross way to do
this.

The other way that came to mind was an aperiodic dump/insert of relevant
postgresql tables relating to exported resources from dev/test into the
production postgresql puppetdb. This would require investigating the schema
in use, and cleanup could get tricky.

The third way that came to mind was to use the inventory service
http://docs.puppetlabs.com/guides/inventory_service.html to loop over
hostnames, GET''ing yaml from dev/test and PUT''ing it onto the
production
server. I don''t know how deletions would be handled, there, or even
what
you''d want your failure mode to be.

Jeff



On Mon, Dec 2, 2013 at 8:39 PM, Jason Antman <jason@jasonantman.com>
wrote:
> Hello,
>
> I have 3 puppet stacks (master, puppetdb, enc) - dev, test/qa and prod.
> Dev is used for initial development and testing of code (including
> puppet), which is then promoted to test and then prod.
>
> I''d like to start using the nagios types to configure monitoring,
via
> exported resources (yes I''m aware of the issues with the builtins,
but
> they''ll have to do for now). I only have one Nagios server, and
I''d like
> to reliably monitor at least some stuff on the dev and test puppet
> nodes. So... setting up all 3 puppet stacks to export resources that are
> realized somehow on the Nagios server isn''t a possibility, as bad
> manifests/modules could affect the monitoring of one of the dev or test
> hosts.
>
> What''s the safe way to "freeze" exported resources, or
prevent them from
> being changed? The best that I can come up with so far is to have the
> nagios server connected to the production puppetmaster, and when I want
> to update the (exported resource) monitoring configuration for one of
> the dev or test nodes, have to do a one-time run on each node in
> question against the prod puppet master.
>
> Any other thoughts or theories?
>
> Thanks,
> Jason Antman
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/puppet-users/529D363C.4030202%40jasonantman.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAHahqg0BadZAtitdE75A0QLqT7VDU5U_2mm5yHCWNxurwEVSxw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.