We want to use Nagios to monitor out puppet server so we can be notified if it goes down. We are using Fusion Passenger and Apache on Red Hat. Any suggestion for what and how to monitor? -- Thanks, Allan Marcus 505-667-5666 Allan@lanl.gov -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
I would just use your typical template for a Linux server (or individual services, which is how I tend to manage Nagios), and make sure to add service-specific service checks for puppetd/puppetmaster, Apache and any other important services. I don''t think Puppet has any specific needs here... Sam On Tue, Oct 4, 2011 at 6:47 AM, Marcus, Allan B <allan@lanl.gov> wrote:> We want to use Nagios to monitor out puppet server so we can be notified > if it goes down. We are using Fusion Passenger and Apache on Red Hat. > > Any suggestion for what and how to monitor? > > -- > Thanks, > > Allan Marcus > 505-667-5666 > Allan@lanl.gov > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Oct 4, 2011, at 9:47 AM, Marcus, Allan B wrote:> Any suggestion for what and how to monitor?We’re not using Nagios, but on the server we just check to see if port 8140 is listening. On the clients, we check that at least one puppetd process is running and we check for errors in the logs with this regex: puppetd\[.*failed:.* Seems to work pretty well. -- Rob McBroom <http://www.skurfer.com/> -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
We use the basic checks for any Unix machine along with special checks for running Puppet master and client process where appropriate. A service which uses NRPE and a check_procs call on the Puppet boxes like these two examples works fine for us: Command[check_puppetmaster]=/usr/lib64/nagios/plugins/check_procs -w 1:1 -c 1:1 -C puppetmasterd Command[check_puppetclient]=/usr/lib64/nagios/plugins/check_procs -w 1:1 -c 1:1 -C puppet Bernd> -----Ursprüngliche Nachricht----- > Von: puppet-users@googlegroups.com [mailto:puppet- > users@googlegroups.com] Im Auftrag von Marcus, Allan B > Gesendet: Dienstag, 4. Oktober 2011 15:47 > An: puppet-users@googlegroups.com > Betreff: [Puppet Users] How best to monitor puppet? > > We want to use Nagios to monitor out puppet server so we can be > notified > if it goes down. We are using Fusion Passenger and Apache on Red Hat. > > Any suggestion for what and how to monitor? > > -- > Thanks, > > Allan Marcus > 505-667-5666 > Allan@lanl.gov > > > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet- > users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Tue, Oct 04, 2011 at 01:47:17PM +0000, Marcus, Allan B wrote:> We want to use Nagios to monitor out puppet server so we can be notified > if it goes down. We are using Fusion Passenger and Apache on Red Hat. > > Any suggestion for what and how to monitor?Not so far using Nagios or anything. I have monit checking my puppetmaster and init supervises monit (and a chaos monkey in the lab to keep them on their toes). Of course the puppetmaster/monit/monkey setups are puppetized.> -- > Thanks, > > Allan Marcus > 505-667-5666 > Allan@lanl.gov > > > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Tue, 4 Oct 2011, Bernd Adamowicz wrote:> > -----Ursprüngliche Nachricht----- > > Von: puppet-users@googlegroups.com [mailto:puppet- > > users@googlegroups.com] Im Auftrag von Marcus, Allan B > > Gesendet: Dienstag, 4. Oktober 2011 15:47 > > An: puppet-users@googlegroups.com > > Betreff: [Puppet Users] How best to monitor puppet? > > > > We want to use Nagios to monitor out puppet server so we can be > > notified > > if it goes down. We are using Fusion Passenger and Apache on Red Hat. > > > > Any suggestion for what and how to monitor? > > We use the basic checks for any Unix machine along with special checks for running Puppet master and client process where appropriate. A service which uses NRPE and a check_procs call on the Puppet boxes like these two examples works fine for us: > > Command[check_puppetmaster]=/usr/lib64/nagios/plugins/check_procs -w 1:1 -c 1:1 -C puppetmasterd > Command[check_puppetclient]=/usr/lib64/nagios/plugins/check_procs -w 1:1 -c 1:1 -C puppet > > Bernd >What about checking the logfile on the master to make sure that everything is checking in? Theoretically, the client daemons could be running and accepting port 8140, but the daemon could be locked up: http://cafuego.net/2011/09/24/keeping-eye-puppet-updated -- Tim Connors -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
It doesn''t matter if puppetmasterd or puppetd are running and working if your clients are failing catalog runs. Send reports, write a check that alerts on N hosts with failed reports over X timeframe or something. On Tue, Oct 4, 2011 at 8:09 PM, Tim Connors <tim.w.connors@gmail.com> wrote:> On Tue, 4 Oct 2011, Bernd Adamowicz wrote: > > > > -----Ursprüngliche Nachricht----- > > > Von: puppet-users@googlegroups.com [mailto:puppet- > > > users@googlegroups.com] Im Auftrag von Marcus, Allan B > > > Gesendet: Dienstag, 4. Oktober 2011 15:47 > > > An: puppet-users@googlegroups.com > > > Betreff: [Puppet Users] How best to monitor puppet? > > > > > > We want to use Nagios to monitor out puppet server so we can be > > > notified > > > if it goes down. We are using Fusion Passenger and Apache on Red Hat. > > > > > > Any suggestion for what and how to monitor? > > > > We use the basic checks for any Unix machine along with special checks > for running Puppet master and client process where appropriate. A service > which uses NRPE and a check_procs call on the Puppet boxes like these two > examples works fine for us: > > > > Command[check_puppetmaster]=/usr/lib64/nagios/plugins/check_procs -w 1:1 > -c 1:1 -C puppetmasterd > > Command[check_puppetclient]=/usr/lib64/nagios/plugins/check_procs -w 1:1 > -c 1:1 -C puppet > > > > Bernd > > > > What about checking the logfile on the master to make sure that everything > is checking in? Theoretically, the client daemons could be running and > accepting port 8140, but the daemon could be locked up: > > http://cafuego.net/2011/09/24/keeping-eye-puppet-updated > > > -- > Tim Connors > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Our approach is to combine the monitoring and the execution. Totally ditch the puppet client service and use nagios to run a "check_puppet" script which does the puppet run and reports the exit codes. This also covers things like having a defined retry interval, so if puppet runs and there are no changes, it''ll come back and run again in an hour, but if there are changes, it will run in 10 minutes etc... works really well. On 5 October 2011 04:36, Scott Smith <scott@ohlol.net> wrote:> It doesn''t matter if puppetmasterd or puppetd are running and working if > your clients are failing catalog runs. > > Send reports, write a check that alerts on N hosts with failed reports over > X timeframe or something. > > > On Tue, Oct 4, 2011 at 8:09 PM, Tim Connors <tim.w.connors@gmail.com>wrote: > >> On Tue, 4 Oct 2011, Bernd Adamowicz wrote: >> >> > > -----Ursprüngliche Nachricht----- >> > > Von: puppet-users@googlegroups.com [mailto:puppet- >> > > users@googlegroups.com] Im Auftrag von Marcus, Allan B >> > > Gesendet: Dienstag, 4. Oktober 2011 15:47 >> > > An: puppet-users@googlegroups.com >> > > Betreff: [Puppet Users] How best to monitor puppet? >> > > >> > > We want to use Nagios to monitor out puppet server so we can be >> > > notified >> > > if it goes down. We are using Fusion Passenger and Apache on Red Hat. >> > > >> > > Any suggestion for what and how to monitor? >> > >> > We use the basic checks for any Unix machine along with special checks >> for running Puppet master and client process where appropriate. A service >> which uses NRPE and a check_procs call on the Puppet boxes like these two >> examples works fine for us: >> > >> > Command[check_puppetmaster]=/usr/lib64/nagios/plugins/check_procs -w 1:1 >> -c 1:1 -C puppetmasterd >> > Command[check_puppetclient]=/usr/lib64/nagios/plugins/check_procs -w 1:1 >> -c 1:1 -C puppet >> > >> > Bernd >> > >> >> What about checking the logfile on the master to make sure that everything >> is checking in? Theoretically, the client daemons could be running and >> accepting port 8140, but the daemon could be locked up: >> >> http://cafuego.net/2011/09/24/keeping-eye-puppet-updated >> >> >> -- >> Tim Connors >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Oct 5, 4:36 am, Scott Smith <sc...@ohlol.net> wrote:> It doesn''t matter if puppetmasterd or puppetd are running and working if > your clients are failing catalog runs. > > Send reports, write a check that alerts on N hosts with failed reports over > X timeframe or something.I favor monitoring each node rather than try to figure out if the master works or not as this is ultimately the outcome you care for - can my nodes do puppet runs. Puppet now saves its last run status in the libdir: /var/lib/puppet/state/last_run_summary.yaml: --- time: group: 0.001692 last_run: 1317804488 class: 0.003929 yumrepo: 0.020103 service: 9.017434 schedule: 0.004151 cron: 0.010546 config_retrieval: 15.0572321414948 total: 34.9742621414947 package: 0.588751 filebucket: 0.000687 file: 8.895422 exec: 1.361625 user: 0.01269 resources: total: 414 events: total: 0 changes: total: 0 I wrote a nagios check to check this and report on any problems: When last did puppet run: # check_puppet.rb -c 3600 -w 1800 OK: Puppet is currently enabled, last run 712 seconds ago with 0 failures how many resources failed: # check_puppet.rb -c 5 -w 2 --check-failures OK: Puppet is currently enabled, last run 742 seconds ago with 0 failures and you can also add --only-enabled to have machines undergoing maintenance not show up as critical. I run this under nrpe etc. The last run summary yaml file is a bit odd in that it doesn''t always contain all the information for instance a syntax error in catalogs will have it only include 1 piece of data and other differences during execution, hopefully this will improve in time (I should probably file tickets) The code for the check is @ https://github.com/ripienaar/monitoring-scripts/blob/master/puppet/check_puppet.rb and it is compatible with all the current odd behaviors of the file as of 2.6.9 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
All good ideas. Once issue I se with the ''have nagios run the puppet client'' approach is that we want to monitor puppet more closely than once per hour. I''d like to come up with a intensive than a full puppet run, even if the run is inventory only. I think we will monitor the MySQL server (runs on a different machine), Apache port 80 on the puppetmaster machine, and a puppet run hourly. We will also monitor disk, cpu, and IO on the server. All of those should give us a good view. Thanks for your ideas. -- Thanks, Allan Marcus 505-667-5666 Allan@lanl.gov From: Chris Phillips <chris@untrepid.com<mailto:chris@untrepid.com>> Reply-To: <puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>> Date: Wed, 5 Oct 2011 06:13:53 +0100 To: <puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>> Subject: Re: [Puppet Users] AW: How best to monitor puppet? Our approach is to combine the monitoring and the execution. Totally ditch the puppet client service and use nagios to run a "check_puppet" script which does the puppet run and reports the exit codes. This also covers things like having a defined retry interval, so if puppet runs and there are no changes, it''ll come back and run again in an hour, but if there are changes, it will run in 10 minutes etc... works really well. On 5 October 2011 04:36, Scott Smith <scott@ohlol.net<mailto:scott@ohlol.net>> wrote: It doesn''t matter if puppetmasterd or puppetd are running and working if your clients are failing catalog runs. Send reports, write a check that alerts on N hosts with failed reports over X timeframe or something. On Tue, Oct 4, 2011 at 8:09 PM, Tim Connors <tim.w.connors@gmail.com<mailto:tim.w.connors@gmail.com>> wrote: On Tue, 4 Oct 2011, Bernd Adamowicz wrote:> > -----Ursprüngliche Nachricht----- > > Von: puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com> [mailto:puppet-<mailto:puppet-> > > users@googlegroups.com<mailto:users@googlegroups.com>] Im Auftrag von Marcus, Allan B > > Gesendet: Dienstag, 4. Oktober 2011 15:47 > > An: puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com> > > Betreff: [Puppet Users] How best to monitor puppet? > > > > We want to use Nagios to monitor out puppet server so we can be > > notified > > if it goes down. We are using Fusion Passenger and Apache on Red Hat. > > > > Any suggestion for what and how to monitor? > > We use the basic checks for any Unix machine along with special checks for running Puppet master and client process where appropriate. A service which uses NRPE and a check_procs call on the Puppet boxes like these two examples works fine for us: > > Command[check_puppetmaster]=/usr/lib64/nagios/plugins/check_procs -w 1:1 -c 1:1 -C puppetmasterd > Command[check_puppetclient]=/usr/lib64/nagios/plugins/check_procs -w 1:1 -c 1:1 -C puppet > > Bernd >What about checking the logfile on the master to make sure that everything is checking in? Theoretically, the client daemons could be running and accepting port 8140, but the daemon could be locked up: http://cafuego.net/2011/09/24/keeping-eye-puppet-updated -- Tim Connors -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users%2Bunsubscribe@googlegroups.com>. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users%2Bunsubscribe@googlegroups.com>. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com<mailto:puppet-users@googlegroups.com>. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users+unsubscribe@googlegroups.com>. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Mon, Oct 17, 2011 at 11:02 AM, Marcus, Allan B <allan@lanl.gov> wrote:> All good ideas. > > Once issue I se with the ''have nagios run the puppet client'' approach is > that we want to monitor puppet more closely than once per hour. I''d like to > come up with a intensive than a full puppet run, even if the run is > inventory only. >I''m a big fan of setting up a dedicated environment for monitoring. I usually put several file transfers into it with source parameter specified files resources and keep that list static. Then I monitor how long it takes for a client to retrieve and to apply the monitoring environment catalog. I used to run environments like this far more frequently than once an hour, as the load itself is small, but it''s a reliable set of metrics for observing variation.> > I think we will monitor the MySQL server (runs on a different machine), > Apache port 80 on the puppetmaster machine, and a puppet run hourly. We will > also monitor disk, cpu, and IO on the server. All of those should give us a > good view. > > Thanks for your ideas. > > > > -- > Thanks, > > Allan Marcus > 505-667-5666 > Allan@lanl.gov > > From: Chris Phillips <chris@untrepid.com> > Reply-To: <puppet-users@googlegroups.com> > Date: Wed, 5 Oct 2011 06:13:53 +0100 > To: <puppet-users@googlegroups.com> > Subject: Re: [Puppet Users] AW: How best to monitor puppet? > > Our approach is to combine the monitoring and the execution. Totally ditch > the puppet client service and use nagios to run a "check_puppet" script > which does the puppet run and reports the exit codes. This also covers > things like having a defined retry interval, so if puppet runs and there are > no changes, it''ll come back and run again in an hour, but if there are > changes, it will run in 10 minutes etc... works really well. > > On 5 October 2011 04:36, Scott Smith <scott@ohlol.net> wrote: > >> It doesn''t matter if puppetmasterd or puppetd are running and working if >> your clients are failing catalog runs. >> >> Send reports, write a check that alerts on N hosts with failed reports >> over X timeframe or something. >> >> >> On Tue, Oct 4, 2011 at 8:09 PM, Tim Connors <tim.w.connors@gmail.com>wrote: >> >>> On Tue, 4 Oct 2011, Bernd Adamowicz wrote: >>> >>> > > -----Ursprüngliche Nachricht----- >>> > > Von: puppet-users@googlegroups.com [mailto:puppet- >>> > > users@googlegroups.com] Im Auftrag von Marcus, Allan B >>> > > Gesendet: Dienstag, 4. Oktober 2011 15:47 >>> > > An: puppet-users@googlegroups.com >>> > > Betreff: [Puppet Users] How best to monitor puppet? >>> > > >>> > > We want to use Nagios to monitor out puppet server so we can be >>> > > notified >>> > > if it goes down. We are using Fusion Passenger and Apache on Red Hat. >>> > > >>> > > Any suggestion for what and how to monitor? >>> > >>> > We use the basic checks for any Unix machine along with special checks >>> for running Puppet master and client process where appropriate. A service >>> which uses NRPE and a check_procs call on the Puppet boxes like these two >>> examples works fine for us: >>> > >>> > Command[check_puppetmaster]=/usr/lib64/nagios/plugins/check_procs -w >>> 1:1 -c 1:1 -C puppetmasterd >>> > Command[check_puppetclient]=/usr/lib64/nagios/plugins/check_procs -w >>> 1:1 -c 1:1 -C puppet >>> > >>> > Bernd >>> > >>> >>> What about checking the logfile on the master to make sure that >>> everything >>> is checking in? Theoretically, the client daemons could be running and >>> accepting port 8140, but the daemon could be locked up: >>> >>> http://cafuego.net/2011/09/24/keeping-eye-puppet-updated >>> >>> >>> -- >>> Tim Connors >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Puppet Users" group. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> To unsubscribe from this group, send email to >>> puppet-users+unsubscribe@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/puppet-users?hl=en. >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- Nigel Kersten Product Manager, Puppet Labs -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.