Puppet users - Feb 2011 - Trigger an event after a puppet run

Hi,

We''d like to run puppet regularly in noop then trigger an event after
the run depending on the results, namely to hook puppet in to our
Nagios implementation so that if anything changes for any reason we
receive an alert.

We already have puppet configured to send out puppet reports via email
but don''t want to completely rely on that.

Is it possible to craft a trigger of some sort so that after a run we
can run a shell script which goes ''ok, nothing has changed, send
nagios an OK'' or ''omg changes,
send nagios a CRITICAL and wake everyone up'' ?

Thanks,

Rus

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Feb 15, 2011 at 6:55 AM, Rus Hughes <russell.hughes@gmail.com>
wrote:
> Hi,
>
> We''d like to run puppet regularly in noop then trigger an event
after
> the run depending on the results, namely to hook puppet in to our
> Nagios implementation so that if anything changes for any reason we
> receive an alert.
>
> We already have puppet configured to send out puppet reports via email
> but don''t want to completely rely on that.
>
> Is it possible to craft a trigger of some sort so that after a run we
> can run a shell script which goes ''ok, nothing has changed, send
> nagios an OK'' or ''omg changes,
> send nagios a CRITICAL and wake everyone up'' ?
It sounds like you''re looking for a postrun_command ?

# A command to run after every agent run.  If this command returns a non-zero
# return code, the entire Puppet run will be considered to have
failed, even though it might have
# performed work during the normal run.
# The default value is ''''.
# postrun_command 
Or perhaps even more simply, set up reporting so you can mine the
structured report data for status information.

>
> Thanks,
>
> Rus
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Feb 15, 2011 at 4:51 PM, Nigel Kersten <nigel@puppetlabs.com>
wrote:
> On Tue, Feb 15, 2011 at 6:55 AM, Rus Hughes
<russell.hughes@gmail.com> wrote:
>> Hi,
>>
>> We''d like to run puppet regularly in noop then trigger an
event after
>> the run depending on the results, namely to hook puppet in to our
>> Nagios implementation so that if anything changes for any reason we
>> receive an alert.
>>
>> We already have puppet configured to send out puppet reports via email
>> but don''t want to completely rely on that.
>>
>> Is it possible to craft a trigger of some sort so that after a run we
>> can run a shell script which goes ''ok, nothing has changed,
send
>> nagios an OK'' or ''omg changes,
>> send nagios a CRITICAL and wake everyone up'' ?
>
> It sounds like you''re looking for a postrun_command ?
>
> # A command to run after every agent run.  If this command returns a
non-zero
> # return code, the entire Puppet run will be considered to have
> failed, even though it might have
> # performed work during the normal run.
> # The default value is ''''.
> # postrun_command >
> Or perhaps even more simply, set up reporting so you can mine the
> structured report data for status information.
Yeah I twigged just after posting that a report is probably the best
way to do this so i can analyze the results

at the moment I have this :

require ''puppet''

Puppet::Reports.register_report(:nagios) do
    Puppet.settings.use(:reporting)
    desc "Sends reports directly to Nagios"

    def process
      begin
        exec "/usr/lib/nagios/eventhandlers/submit_puppet_result
10.30.0.101 puppet-report 0 \"There has been no change\""
      end
    end
end

http://pastie.org/1567064 in case email formatting munges things

and i''ve configured puppetmasterd like so

reports = log, store, rrdgraph, smt, tagmail, nagios

but when I do a puppet run I get

2011-02-15T17:00:24.00+00:00 vfantmon1 puppetmasterd[28860]: Report
nagios failed: no block given

any idea what i''ve done wrong?

Thanks,

Rus

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Feb 15, 2011 at 11:03 AM, Rus Hughes <russell.hughes@gmail.com>
wrote:
[snip]
>
> require ''puppet''
>
> Puppet::Reports.register_report(:nagios) do
>    Puppet.settings.use(:reporting)
>    desc "Sends reports directly to Nagios"
>
>    def process
>      begin
>        exec "/usr/lib/nagios/eventhandlers/submit_puppet_result
> 10.30.0.101 puppet-report 0 \"There has been no change\""
>      end
>    end
> end
>
> http://pastie.org/1567064 in case email formatting munges things
>
You may want to avoid the use of exec, if Kernel.exec is the receiver
of this message, then the running Puppet Master process itself will be
replaced with your nagios check.

Check out the documentation for Kernel#exec:

699 puppet:2.6.next % /usr/bin/ri Kernel\#exec
------------------------------------------------------------ Kernel#exec
     exec(command [, arg, ...])
------------------------------------------------------------------------
     Replaces the current process by running the given external
     _command_. If +exec+ is given a single argument, that argument is
     taken as a line that is subject to shell expansion before being
     executed. If multiple arguments are given, the second and
     subsequent arguments are passed as parameters to _command_ with no
     shell expansion. If the first argument is a two-element array, the
     first element is the command to be executed, and the second
     argument is used as the +argv[0]+ value, which may show up in
     process listings. In MSDOS environments, the command is executed in
     a subshell; otherwise, one of the +exec(2)+ system calls is used,
     so the running command may inherit some of the environment of the
     original program (including open file descriptors).

        exec "echo *"       # echoes list of files in current
directory
        # never get here

        exec "echo", "*"    # echoes an asterisk
        # never get here


-- 
Jeff McCune
http://www.puppetlabs.com/

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks for the help guys, I''ve now finished it and got a working
solution. For anyone replicating this you need to configure a passive
service check on each host called puppet-report which then will get
updated.

http://pastie.org/private/812ijdrd7gwmwgxfmc9rdq

require ''puppet''
require ''socket''

Puppet::Reports.register_report(:nagios) do
    Puppet.settings.use(:reporting)
    desc "Sends reports directly to Nagios"

    def process
        tree = YAML::parse(to_yaml)
        host = tree.select(''/host'')[0].value
        ip = IPSocket::getaddress(host)

        size = self.logs.size
        if size == 0 then
            system "/usr/lib/nagios/eventhandlers/submit_puppet_result
" + ip + " puppet-report 0 \"There has been no
change\""
        else
            system "/usr/lib/nagios/eventhandlers/submit_puppet_result
" + ip + " puppet-report 2 \"There have been " + size.to_s +
"
changes\""
        end
    end
end

which calls the following bash script

http://pastie.org/private/drn1jj65tu5v8ljsdfkvwa

#!/bin/sh

# SUBMIT_CHECK_RESULT
# Written by Ethan Galstad (egalstad@nagios.org)
# Last Modified: 02-18-2002
#
# This script will write a command to the Nagios command
# file to cause Nagios to process a passive service check
# result.  Note: This script is intended to be run on the
# same host that is running Nagios.  If you want to
# submit passive check results from a remote machine, look
# at using the nsca addon.
#
# Arguments:
#  $1 = host_name (Short name of host that the service is
#       associated with)
#  $2 = svc_description (Description of the service)
#  $3 = return_code (An integer that determines the state
#       of the service check, 0=OK, 1=WARNING, 2=CRITICAL,
#       3=UNKNOWN).
#  $4 = plugin_output (A text string that should be used
#       as the plugin output for the service check)
#

echocmd="/bin/echo"

CommandFile="/var/log/nagios/rw/nagios.cmd"

# get the current date/time in seconds since UNIX epoch
datetime=`date +%s`

# create the command line to add to the command file
IP=$1
SERVICE=$2
RESULT=$3
MSG=$4
cmdline="[$datetime]
PROCESS_SERVICE_CHECK_RESULT;$IP;$SERVICE;$RESULT;$MSG"

# append the command to the end of the command file
`$echocmd $cmdline >> $CommandFile`

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>        size = self.logs.size
>        if size == 0 then
>            system "/usr/lib/nagios/eventhandlers/submit_puppet_result
> " + ip + " puppet-report 0 \"There has been no
change\""
>        else
>            system "/usr/lib/nagios/eventhandlers/submit_puppet_result
> " + ip + " puppet-report 2 \"There have been " +
size.to_s + "
> changes\""
>        end
>    end
> end
This seems, at face value, to only be reporting upward
"change/no-change" type of thing.

I''m not at all "ruby-ready" as it were, in terms of
programming skill, but it occurs to me that I can think of other conditions
I''d be interested in reporting upward (syntax or other errors in the
catalog received, or anything really, which causes a cached catalog to be used).

Is there some easy way to add that functionality into this script?

D


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Feb 16, 2011 at 11:38 AM, Derek J. Balling <dredd@megacity.org>
wrote:
>
>>        size = self.logs.size
>>        if size == 0 then
>>            system
"/usr/lib/nagios/eventhandlers/submit_puppet_result
>> " + ip + " puppet-report 0 \"There has been no
change\""
>>        else
>>            system
"/usr/lib/nagios/eventhandlers/submit_puppet_result
>> " + ip + " puppet-report 2 \"There have been " +
size.to_s + "
>> changes\""
>>        end
>>    end
>> end
>
> This seems, at face value, to only be reporting upward
"change/no-change" type of thing.
>
> I''m not at all "ruby-ready" as it were, in terms of
programming skill, but it occurs to me that I can think of other conditions
I''d be interested in reporting upward (syntax or other errors in the
catalog received, or anything really, which causes a cached catalog to be used).
>
> Is there some easy way to add that functionality into this script?
>
> D
Correct, it''ll only do change or no change, and if you run puppet with
--test it will also report a change if there isn''t any. The yaml
contains a complete report of what''s going on though, so you can parse
that for errors and change the message string accordingly and probably
even parse it a little more thoroughly so even --test doesn''t report
any changes.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 16, 3:38 am, "Derek J. Balling" <dr...@megacity.org>
wrote:
> >        size = self.logs.size
> >        if size == 0 then
> >            system
"/usr/lib/nagios/eventhandlers/submit_puppet_result
> > " + ip + " puppet-report 0 \"There has been no
change\""
> >        else
> >            system
"/usr/lib/nagios/eventhandlers/submit_puppet_result
> > " + ip + " puppet-report 2 \"There have been " +
size.to_s + "
> > changes\""
> >        end
> >    end
> > end
>
> This seems, at face value, to only be reporting upward
"change/no-change" type of thing.
>
> I''m not at all "ruby-ready" as it were, in terms of
programming skill, but it occurs to me that I can think of other conditions
I''d be interested in reporting upward (syntax or other errors in the
catalog received, or anything really, which causes a cached catalog to be used).
>
> Is there some easy way to add that functionality into this script?
Off hand you may be interested in the metrics portion of the report.
Something like metrics["changes"][:failed] would give you access to
any resources that failed to apply.
I don''t recall how/if actual catalog failures are show in reports. A
syntax error, for example, happens on the master and is presented to
the client as a server error.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 16, 2011, at 2:49 PM, donavan wrote:
> Off hand you may be interested in the metrics portion of the report.
> Something like metrics["changes"][:failed] would give you access
to
> any resources that failed to apply.
> I don''t recall how/if actual catalog failures are show in reports.
A
> syntax error, for example, happens on the master and is presented to
> the client as a server error.
One thing I have seen is where the client gives up waiting for the master and
uses the cached catalog, but the master doesn''t recognize that the
client went away, and doesn''t think anything went wrong, or anything
like that (or at least doesn''t create any sort of
"failure/error" report).

D

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.