Does anybody have experience setting up monitoring of puppet itself? We have had problems with the puppetmasterd or puppetd failing (or not being started) and machines not being updated. I want to make sure that all of puppet managed machines are up-to-date. The tagmail reports are good for seeing errors with applying changes. But it doesn''t help when a daemon stops (probably a log rotation issue). Or when there is a error in the manifests that causes the puppetmaster not to load it or the client to use the cached configuration. I think we need three kinds of monitoring: - Check connections to puppetmasterd on the server - Check connections to puppetd for puppetrun on clients - Check machine has been updated successfully from a fresh configuration - Ian --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ian Burrell wrote: | Does anybody have experience setting up monitoring of puppet itself? | We have had problems with the puppetmasterd or puppetd failing (or not | being started) and machines not being updated. I want to make sure | that all of puppet managed machines are up-to-date. The tagmail | reports are good for seeing errors with applying changes. But it | doesn''t help when a daemon stops (probably a log rotation issue). Or | when there is a error in the manifests that causes the puppetmaster | not to load it or the client to use the cached configuration. | | I think we need three kinds of monitoring: | | - Check connections to puppetmasterd on the server | - Check connections to puppetd for puppetrun on clients | - Check machine has been updated successfully from a fresh configuration | I''d recommend either Nagios or god (http://god.rubyforge.org/) for this - - I use both for different purposes. With either you can do log monitoring, confirm daemons running and issue remedial actions if something is broken. Regards James Turnbull - -- James Turnbull (james@lovedthanlost.net) - -- Author of: - - Pulling Strings with Puppet (http://www.amazon.com/gp/product/1590599780/) - - Pro Nagios 2.0 (http://www.amazon.com/gp/product/1590596099/) - - Hardening Linux (http://www.amazon.com/gp/product/1590594444/) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH6sf69hTGvAxC30ARAiHMAKDTLCQf8zl0PDhXaUWiaeBsIpqqzACfVPxG TW8+4IpLNhWNOClARw1hGl0=Ok1g -----END PGP SIGNATURE----- --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
We have puppetd setup with ''listen = true'' and just make sure the port is open. another option would be to make sure that puppetd is in your process list if you have snmp running on the boxes. Hope this helps out. -j On Wed, Mar 26, 2008 at 2:55 PM, Ian Burrell <ianburrell@gmail.com> wrote:> > Does anybody have experience setting up monitoring of puppet itself? > We have had problems with the puppetmasterd or puppetd failing (or not > being started) and machines not being updated. I want to make sure > that all of puppet managed machines are up-to-date. The tagmail > reports are good for seeing errors with applying changes. But it > doesn''t help when a daemon stops (probably a log rotation issue). Or > when there is a error in the manifests that causes the puppetmaster > not to load it or the client to use the cached configuration. > > I think we need three kinds of monitoring: > > - Check connections to puppetmasterd on the server > - Check connections to puppetd for puppetrun on clients > - Check machine has been updated successfully from a fresh configuration > > - Ian > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I my self use monit. I have monit monitor puppet, and puppet monitor monit. If puppet goes down, monit restarts it and send me an email. Works pretty well. Of course it only makes sure that puppet is running, and not whether its actualy doing anything. -Joel On Wed, 26 Mar 2008, Ian Burrell wrote:> > Does anybody have experience setting up monitoring of puppet itself? > We have had problems with the puppetmasterd or puppetd failing (or not > being started) and machines not being updated. I want to make sure > that all of puppet managed machines are up-to-date. The tagmail > reports are good for seeing errors with applying changes. But it > doesn''t help when a daemon stops (probably a log rotation issue). Or > when there is a error in the manifests that causes the puppetmaster > not to load it or the client to use the cached configuration. > > I think we need three kinds of monitoring: > > - Check connections to puppetmasterd on the server > - Check connections to puppetd for puppetrun on clients > - Check machine has been updated successfully from a fresh configuration > > - Ian > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
--On Wednesday, March 26, 2008 2:55 PM -0700 Ian Burrell <ianburrell@gmail.com> wrote:> Does anybody have experience setting up monitoring of puppet itself? > We have had problems with the puppetmasterd or puppetd failing (or not > being started) and machines not being updated. I want to make sure > that all of puppet managed machines are up-to-date. The tagmail > reports are good for seeing errors with applying changes. But it > doesn''t help when a daemon stops (probably a log rotation issue). Or > when there is a error in the manifests that causes the puppetmaster > not to load it or the client to use the cached configuration. > > I think we need three kinds of monitoring: > > - Check connections to puppetmasterd on the server > - Check connections to puppetd for puppetrun on clients > - Check machine has been updated successfully from a fresh configuration > > - IanLots of options here: - You can setup Nagios to monitor the Puppetmaster. - Since we''ve noticed that puppetd dies a lot if the puppetmaster is being bad, we''ve added a simple cron job that starts puppet if it isn''t running. - You can use the lastcheck report to see when a particular server last checked in to puppet. We run a nightly cron that sends out a list of servers that haven''t checked in for over a day. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Launch puppetd out of cron. I also don''t use puppet.conf on our nodes.
0 */1 * * * /usr/sbin/runpuppet-noop
aj@crmtest:~$ cat `which runpuppet`
#!/bin/bash
# this file is installed via Puppet, edits will be overwritten
hostname=`hostname -f`
echo "running puppet on $hostname, standby."
/usr/sbin/puppetd --test --pluginsync --vardir /var/lib/puppet --ssldir
/var/lib/puppet/ssl --factpath /var/lib/puppet/lib/facter
aj@crmtest:~$ cat `which runpuppet-noop`
#!/bin/bash
# this file is installed via Puppet, edits will be overwritten
hostname=`hostname -f`
echo "running puppet on $hostname, standby."
/usr/sbin/puppetd -t --pluginsync --puppetdlockfile
/var/lib/puppet/state/puppetdnooplock --vardir /var/lib/puppet --ssldir
/var/lib/puppet/ssl --factpath /var/lib/puppet/lib/facter --noop --color false
|grep noop|egrep -v ''(checksum|Git)''|mail -s "puppet noop
run on $hostname" -e puppet
(''puppet'' is a mailalias for myself and my colleagues email
address, edit as necessary)
This is lame, but ensures that puppetd does not rely on a constant connection to
the puppetmaster, thus resolving the automatic shutdown of puppetd in the event
of loss of connectivity.
Arjuna Christensen | Systems Engineer
Maximum Internet Ltd
7a Parkhead Pl, Albany, North Shore, 0632 | PO Box 8006, Auckland, 1150, NZ
DDI: + 64 9 913 9683 | Ph: +64 9 915 1825 | Fax:: +64 9 300 7227
arjuna.christensen@maxnet.co.nz| www.maxnet.co.nz
________________________________
Maxnet | mission critical internet
________________________________
This email (including any attachments) is confidential and intended only for the
person to whom it is addressed.
If you have received this email in error, please notify the sender immediately
and erase all copies of this message
and attachments. The views expressed in this email do not necessarily reflect
those held by Maxnet.
-----Original Message-----
From: puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] On
Behalf Of Digant C Kasundra
Sent: Thursday, 27 March 2008 11:23 a.m.
To: puppet-users@googlegroups.com
Subject: [Puppet Users] Re: Monitoring puppet
--On Wednesday, March 26, 2008 2:55 PM -0700 Ian Burrell
<ianburrell@gmail.com> wrote:
> Does anybody have experience setting up monitoring of puppet itself?
> We have had problems with the puppetmasterd or puppetd failing (or not
> being started) and machines not being updated. I want to make sure
> that all of puppet managed machines are up-to-date. The tagmail
> reports are good for seeing errors with applying changes. But it
> doesn''t help when a daemon stops (probably a log rotation issue).
Or
> when there is a error in the manifests that causes the puppetmaster
> not to load it or the client to use the cached configuration.
>
> I think we need three kinds of monitoring:
>
> - Check connections to puppetmasterd on the server
> - Check connections to puppetd for puppetrun on clients
> - Check machine has been updated successfully from a fresh configuration
>
> - Ian
Lots of options here:
- You can setup Nagios to monitor the Puppetmaster.
- Since we''ve noticed that puppetd dies a lot if the puppetmaster is
being
bad, we''ve added a simple cron job that starts puppet if it
isn''t running.
- You can use the lastcheck report to see when a particular server last
checked in to puppet. We run a nightly cron that sends out a list of
servers that haven''t checked in for over a day.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
On 26/03/2008, Ian Burrell <ianburrell@gmail.com> wrote:> > I think we need three kinds of monitoring: > > - Check connections to puppetmasterd on the server > - Check connections to puppetd for puppetrun on clients > - Check machine has been updated successfully from a fresh configuration > > - Ian >puppetd: $LIBDIR/state/state.yaml is updated when puppet runs. what I would do is something similar to the check_nagios plugin, I''d look to see if that file has been modified in the last XXXX seconds, and make sure that puppet is itself running. puppetmasterd: There should be a port open (8140 by default) that is running https. You could just check that and make sure it responds with a reasonable reply (I got a "not found" when I did a ''curl -k https://puppet:8140''). .r'' --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I also prefer monit, it supports includes which makes it easy use a definition. Also excellent for watching the memory usage, etc. I switched to God for a while but it suffered from massive memory leaks, and when puppet also had memory leaks, that wasn''t a good combination. :) I think God is definitely a great way to go though, and it has a lot of very interesting integration possibilities. -Blake On Mar 26, 2008, at 3:17 PM, Joel Wood wrote:> > I my self use monit. I have monit monitor puppet, and puppet monitor > monit. If puppet goes down, monit restarts it and send me an email. > Works pretty well. > > Of course it only makes sure that puppet is running, and not whether > its > actualy doing anything. > > -Joel > > On Wed, 26 Mar 2008, Ian Burrell wrote: > >> >> Does anybody have experience setting up monitoring of puppet itself? >> We have had problems with the puppetmasterd or puppetd failing (or >> not >> being started) and machines not being updated. I want to make sure >> that all of puppet managed machines are up-to-date. The tagmail >> reports are good for seeing errors with applying changes. But it >> doesn''t help when a daemon stops (probably a log rotation issue). Or >> when there is a error in the manifests that causes the puppetmaster >> not to load it or the client to use the cached configuration. >> >> I think we need three kinds of monitoring: >> >> - Check connections to puppetmasterd on the server >> - Check connections to puppetd for puppetrun on clients >> - Check machine has been updated successfully from a fresh >> configuration >> >> - Ian >> >>> > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Mar 26, 2008, at 5:57 PM, RijilV wrote:> puppetmasterd: > There should be a port open (8140 by default) that is running > https. You could just check that and make sure it responds with a > reasonable reply (I got a "not found" when I did a ''curl -khttps:// > puppet:8140'').Note that there''s also a ''status'' namespace, and you should be able to follow some of the existing examples (e.g., the ''file'' example in ext/ puppet-test) in how to run a status command. It should be as simple as something like: @client = Puppet::Network::Client.status.new(:Server => Puppet[:server]) unless @client.read_cert fail "Could not read client certificate" end @client.status ------- This''ll have to run as a user who can read a cert (but you can use puppetca to set up certs for any user, with a bit of effort), and the ''status'' method always returns 1, but you can at least track how long it takes to respond and such. If you do write such a script, please post it. -- There are three social classes in America: upper middle class, middle class, and lower middle class. --Judith Martin --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Mar 26, 2008, at 5:05 PM, Jason Aras wrote:> We have puppetd setup with ''listen = true'' and just make sure the > port is open. another option would be to make sure that puppetd is > in your process list if you have snmp running on the boxes.Note that my bit about ''status'' works on the client, too, although there''s currently no way to set this up in the config file. If you start puppetd like this: sudo puppetd --serve Runner --serve Status --listen You should have the same ''status'' namespace available. Thus, you could do a more complete check on all clients and the server. The client listens on a different port, of course, so you''ll have to specify the other port in the arguments to the status client. -- Man is the only animal that can remain on friendly terms with the victims he intends to eat until he eats them. -- Samuel Butler (1835-1902) --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 RijilV wrote: | On 26/03/2008, *Ian Burrell* <ianburrell@gmail.com | <mailto:ianburrell@gmail.com>> wrote: | | | I think we need three kinds of monitoring: | | - Check connections to puppetmasterd on the server | - Check connections to puppetd for puppetrun on clients | - Check machine has been updated successfully from a fresh configuration | | - Ian | | | puppetd: | $LIBDIR/state/state.yaml is updated when puppet runs. what I would do | is something similar to the check_nagios plugin, I''d look to see if that | file has been modified in the last XXXX seconds, and make sure that | puppet is itself running. In fact it''d be fairly easy to clone the check_nagios plugin and create a check_puppet plugin if anyone is game. | puppetmasterd: | There should be a port open (8140 by default) that is running https. | You could just check that and make sure it responds with a reasonable | reply (I got a "not found" when I did a ''curl -k https://puppet:8140'' | <https://puppet:8140''>). Ditto for check_nagios/check_puppet plugin. Regards James Turnbull - -- James Turnbull (james@lovedthanlost.net) - -- Author of: - - Pulling Strings with Puppet (http://www.amazon.com/gp/product/1590599780/) - - Pro Nagios 2.0 (http://www.amazon.com/gp/product/1590596099/) - - Hardening Linux (http://www.amazon.com/gp/product/1590594444/) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH6uUF9hTGvAxC30ARAiImAJ4q7HF02YwlA6Z/aZ6KgqegswzVtwCdEPD3 s/Jhni3JNlZdmAU/5SC/yx0=d88Z -----END PGP SIGNATURE----- --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
We go the way currently with Nagios. On the puppetmaster a file is regularly (0:00, 12:00) written with a timestamp. This file is getting managed by puppet and so transfered to all the clients. And there nagios checks if the file is in position and its age (check_file_age). If older then 18 hours or the puppetd process is no longer available, eventhandlers of Nagios restart puppet on the client. Cheers, Unki Ian Burrell wrote:> Does anybody have experience setting up monitoring of puppet itself? > We have had problems with the puppetmasterd or puppetd failing (or not > being started) and machines not being updated. I want to make sure > that all of puppet managed machines are up-to-date. The tagmail > reports are good for seeing errors with applying changes. But it > doesn''t help when a daemon stops (probably a log rotation issue). Or > when there is a error in the manifests that causes the puppetmaster > not to load it or the client to use the cached configuration. > > I think we need three kinds of monitoring: > > - Check connections to puppetmasterd on the server > - Check connections to puppetd for puppetrun on clients > - Check machine has been updated successfully from a fresh configuration > > - Ian > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On this topic I''ve just added a basic Nagios plug-in for checking Puppet to ext/nagios/check_puppet.rb. It''s written in Ruby (probably badly) and mimics the behaviour of the check_nagios plug-in. You can see some details at http://reductivelabs.com/trac/puppet/ticket/1162. Until Luke pushes it you can pull it from my repo. Hope that helps someone and feel free to improve on my initial take. Regards James Turnbull - -- James Turnbull (james@lovedthanlost.net) - -- Author of: - - Pulling Strings with Puppet (http://www.amazon.com/gp/product/1590599780/) - - Pro Nagios 2.0 (http://www.amazon.com/gp/product/1590596099/) - - Hardening Linux (http://www.amazon.com/gp/product/1590594444/) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH628m9hTGvAxC30ARAtU8AJ96o1MDsiFe/UUyy7xnlirJ35Hz+ACcCgRc kHcNDFQ3uz12hS3AtjBquh8=BOqb -----END PGP SIGNATURE----- --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
--On Thursday, March 27, 2008 11:28 AM +1300 Arjuna Christensen <arjuna.christensen@maxnet.co.nz> wrote:> This is lame, but ensures that puppetd does not rely on a constant > connection to the puppetmaster, thus resolving the automatic shutdown of > puppetd in the event of loss of connectivity.puppetd doesn''t actually stay connected to the puppetmaster, it just dies when it tries to find the puppetmaster and can''t, which I think is a bug that might be fixed in 0.24.4. Anyway, cron does work fine, though. But it does mean you can''t use the puppet.conf to determine how often it should run -- you''d have to put that in the cron setup. Other than that, it is pretty similar, I think. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I''ve a very similar solution to this as well. I to run puppet from cron one an hour randomly. I have another ''watcher'' if you will that will connect to a host and stat the state.yaml file. Some other logic to let me know when a host hasn''t checked in for 8 hours mails me. Cheers, Ryan On Wed, Mar 26, 2008 at 3:28 PM, Arjuna Christensen < arjuna.christensen@maxnet.co.nz> wrote:> > Launch puppetd out of cron. I also don''t use puppet.conf on our nodes. > > 0 */1 * * * /usr/sbin/runpuppet-noop > > aj@crmtest:~$ cat `which runpuppet` > #!/bin/bash > # this file is installed via Puppet, edits will be overwritten > hostname=`hostname -f` > echo "running puppet on $hostname, standby." > /usr/sbin/puppetd --test --pluginsync --vardir /var/lib/puppet --ssldir > /var/lib/puppet/ssl --factpath /var/lib/puppet/lib/facter > > aj@crmtest:~$ cat `which runpuppet-noop` > #!/bin/bash > # this file is installed via Puppet, edits will be overwritten > hostname=`hostname -f` > echo "running puppet on $hostname, standby." > /usr/sbin/puppetd -t --pluginsync --puppetdlockfile > /var/lib/puppet/state/puppetdnooplock --vardir /var/lib/puppet --ssldir > /var/lib/puppet/ssl --factpath /var/lib/puppet/lib/facter --noop --color > false |grep noop|egrep -v ''(checksum|Git)''|mail -s "puppet noop run on > $hostname" -e puppet > > (''puppet'' is a mailalias for myself and my colleagues email address, edit > as necessary) > > This is lame, but ensures that puppetd does not rely on a constant > connection to the puppetmaster, thus resolving the automatic shutdown of > puppetd in the event of loss of connectivity. > > Arjuna Christensen | Systems Engineer > Maximum Internet Ltd > 7a Parkhead Pl, Albany, North Shore, 0632 | PO Box 8006, Auckland, 1150, > NZ > DDI: + 64 9 913 9683 | Ph: +64 9 915 1825 | Fax:: +64 9 300 7227 > arjuna.christensen@maxnet.co.nz| www.maxnet.co.nz > ________________________________ > Maxnet | mission critical internet > ________________________________ > This email (including any attachments) is confidential and intended only > for the person to whom it is addressed. > If you have received this email in error, please notify the sender > immediately and erase all copies of this message > and attachments. The views expressed in this email do not necessarily > reflect those held by Maxnet. > > -----Original Message----- > From: puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] > On Behalf Of Digant C Kasundra > Sent: Thursday, 27 March 2008 11:23 a.m. > To: puppet-users@googlegroups.com > Subject: [Puppet Users] Re: Monitoring puppet > > > --On Wednesday, March 26, 2008 2:55 PM -0700 Ian Burrell > <ianburrell@gmail.com> wrote: > > > Does anybody have experience setting up monitoring of puppet itself? > > We have had problems with the puppetmasterd or puppetd failing (or not > > being started) and machines not being updated. I want to make sure > > that all of puppet managed machines are up-to-date. The tagmail > > reports are good for seeing errors with applying changes. But it > > doesn''t help when a daemon stops (probably a log rotation issue). Or > > when there is a error in the manifests that causes the puppetmaster > > not to load it or the client to use the cached configuration. > > > > I think we need three kinds of monitoring: > > > > - Check connections to puppetmasterd on the server > > - Check connections to puppetd for puppetrun on clients > > - Check machine has been updated successfully from a fresh configuration > > > > - Ian > > Lots of options here: > - You can setup Nagios to monitor the Puppetmaster. > - Since we''ve noticed that puppetd dies a lot if the puppetmaster is being > bad, we''ve added a simple cron job that starts puppet if it isn''t running. > - You can use the lastcheck report to see when a particular server last > checked in to puppet. We run a nightly cron that sends out a list of > servers that haven''t checked in for over a day. > > > > > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
--On Thursday, March 27, 2008 10:29:18 AM -0700 Ryan Dooley <ryan.dooley@gmail.com> wrote:> I have another ''watcher'' if you will that will connect to > a host and stat the state.yaml file. Some other logic to let me know > when a host hasn''t checked in for 8 hours mails me.If you want to just query the puppetmaster instead of each host, this si something the lastcheck report can help with. I''m not sure if that made it into upstream so if not, let me know and I can send you the code. It basically creates a file under the reports directory with the timestamp of the last checkin of a client ($VARDIR/$fqdn/lastcheck). Makes it easier to audit from one place without the need to allow an incoming connection to the individual clients to ping the state.yaml file. Either way works fine but this is an alternative that you might be interested in. I''ve also got a version that stores that information in a mysql database. -- Digant C Kasundra <digant@stanford.edu> Technical Lead, ITS Unix Systems and Applications, Stanford University --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I''m very interested. Tim -----Original Message----- From: puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] On Behalf Of Digant C Kasundra Sent: Thursday, March 27, 2008 4:06 PM To: puppet-users@googlegroups.com Subject: [Puppet Users] Re: Monitoring puppet --On Thursday, March 27, 2008 10:29:18 AM -0700 Ryan Dooley <ryan.dooley@gmail.com> wrote:> I have another ''watcher'' if you will that will connect to > a host and stat the state.yaml file. Some other logic to let me know > when a host hasn''t checked in for 8 hours mails me.If you want to just query the puppetmaster instead of each host, this si something the lastcheck report can help with. I''m not sure if that made it into upstream so if not, let me know and I can send you the code. It basically creates a file under the reports directory with the timestamp of the last checkin of a client ($VARDIR/$fqdn/lastcheck). Makes it easier to audit from one place without the need to allow an incoming connection to the individual clients to ping the state.yaml file. Either way works fine but this is an alternative that you might be interested in. I''ve also got a version that stores that information in a mysql database. -- Digant C Kasundra <digant@stanford.edu> Technical Lead, ITS Unix Systems and Applications, Stanford University --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Sounds like a good upstream patch to me. On Thu, Mar 27, 2008 at 1:05 PM, Digant C Kasundra <digant@stanford.edu> wrote:> > --On Thursday, March 27, 2008 10:29:18 AM -0700 Ryan Dooley > <ryan.dooley@gmail.com> wrote: > > > I have another ''watcher'' if you will that will connect to > > a host and stat the state.yaml file. Some other logic to let me know > > when a host hasn''t checked in for 8 hours mails me. > > If you want to just query the puppetmaster instead of each host, this si > something the lastcheck report can help with. I''m not sure if that made > it > into upstream so if not, let me know and I can send you the code. It > basically creates a file under the reports directory with the timestamp of > the last checkin of a client ($VARDIR/$fqdn/lastcheck). Makes it easier > to > audit from one place without the need to allow an incoming connection to > the individual clients to ping the state.yaml file. Either way works fine > but this is an alternative that you might be interested in. I''ve also got > a version that stores that information in a mysql database. > > -- > Digant C Kasundra <digant@stanford.edu> > Technical Lead, ITS Unix Systems and Applications, Stanford University > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Mar 27, 2008, at 1:23 PM, Ryan Dooley wrote:> Sounds like a good upstream patch to me. > > On Thu, Mar 27, 2008 at 1:05 PM, Digant C Kasundra <digant@stanford.edu > > wrote: > > --On Thursday, March 27, 2008 10:29:18 AM -0700 Ryan Dooley > <ryan.dooley@gmail.com> wrote: > > > I have another ''watcher'' if you will that will connect to > > a host and stat the state.yaml file. Some other logic to let me > know > > when a host hasn''t checked in for 8 hours mails me. > > If you want to just query the puppetmaster instead of each host, > this si > something the lastcheck report can help with. I''m not sure if that > made it > into upstream so if not, let me know and I can send you the code. It > basically creates a file under the reports directory with the > timestamp of > the last checkin of a client ($VARDIR/$fqdn/lastcheck). Makes it > easier to > audit from one place without the need to allow an incoming > connection to > the individual clients to ping the state.yaml file. Either way > works fine > but this is an alternative that you might be interested in. I''ve > also got > a version that stores that information in a mysql database.You may also want to try storeconfigs, it stores "last_compile" and "last_freshcheck" for each client. We''re also thinking of integrating the report database storage with storeconfigs, since there''s obviously a lot of overlap. -Blake --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I have puppetmaster with postgres for storeconfigs I then just use a script that checks the "last_compile" field and if its older than an hour it send an email with a list of hosts that have dragged behind the window, this helps me quickly see if theres a logic problem with changes that have been recently been made I also then use monit to make sure that the puppetd daemon is running (as the daemon can still be running but in a dead state sometimes, the method above combats this) Cheers Brendan Tim.Metz@cox.com wrote:> I''m very interested. > > Tim > > -----Original Message----- > From: puppet-users@googlegroups.com > [mailto:puppet-users@googlegroups.com] On Behalf Of Digant C Kasundra > Sent: Thursday, March 27, 2008 4:06 PM > To: puppet-users@googlegroups.com > Subject: [Puppet Users] Re: Monitoring puppet > > > --On Thursday, March 27, 2008 10:29:18 AM -0700 Ryan Dooley > <ryan.dooley@gmail.com> wrote: > > >> I have another ''watcher'' if you will that will connect to >> a host and stat the state.yaml file. Some other logic to let me know >> when a host hasn''t checked in for 8 hours mails me. >> > > If you want to just query the puppetmaster instead of each host, this si > > something the lastcheck report can help with. I''m not sure if that made > it > into upstream so if not, let me know and I can send you the code. It > basically creates a file under the reports directory with the timestamp > of > the last checkin of a client ($VARDIR/$fqdn/lastcheck). Makes it easier > to > audit from one place without the need to allow an incoming connection to > > the individual clients to ping the state.yaml file. Either way works > fine > but this is an alternative that you might be interested in. I''ve also > got > a version that stores that information in a mysql database. > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---