Puppet users - Feb 2009 - Best method to monitor puppetmasterd?

Hi All,

I have puppet monitoring down pat, what''s the best way to monitor
puppetmasterd?  We currently monitor the puppetmaster port but would
like to monitor something to make sure not only if the port is alive
but the puppetmaster is returning results.

I assume this will become easier when REST is implemented.

-L
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

I use monit to monitor the puppet masters (e.g. mongrels), for memory leaks
etc but I dont think what you had in mind...

Cant we just use one of the load simulators to check if you get a config?


On Fri, Feb 13, 2009 at 7:06 AM, Larry Ludwig <larrylud@gmail.com> wrote:
>
> Hi All,
>
> I have puppet monitoring down pat, what''s the best way to monitor
> puppetmasterd?  We currently monitor the puppetmaster port but would
> like to monitor something to make sure not only if the port is alive
> but the puppetmaster is returning results.
>
> I assume this will become easier when REST is implemented.
>
> -L
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Yea I want to make sure puppetmaster returns some sort of valid
result.  Though the monit for memory leaks is not a bad idea.

More than just pinging a port.

For example in our case we monitor PHP.  We''ve had PHP crap out for a
few customers, so we monitor a phpinfo page to make sure php returns a
valid result.

I also mentioned to Luke about getting some metrics out of
puppetmaster at least via command line.  I would LOVE to be able to
monitor it via Cacti to see how performance is going.  Not sure what
the metrics would be, but something to help with when to scale to
bigger hardware.  At least from the baseline graphs we do (CPU, disk
IO, memory, bandwidth) the Puppetmaster appears to be primarily CPU
bound first, memory second.

-L
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Thu, Feb 12, 2009 at 6:19 PM, Larry Ludwig <larrylud@gmail.com>
wrote:
>
> Yea I want to make sure puppetmaster returns some sort of valid
> result.  Though the monit for memory leaks is not a bad idea.
>
> More than just pinging a port.
We have some python I''ve been meaning to try and get published that do
a slightly better job that this.

Basically I''m faking out the required XMLRPC headers for an
authenticated client on the machines themselves and issuing a listing
of the facts mountpoint for each mongrel instance (because I''m on the
localhost we can do this) so we can then publish this data via our
local monitoring system daemon.

I''ve got a background job of working on trying to fake out a proper
getconfig request for a catalog to check timing data, but that''s not
functional yet.

If you sit down and look at the traffic, either de-encrypting with
ssldump or simply monitoring the unencrypted traffic between your SSL
frontend and the puppet backends (assuming you''re using that sort of a
setup) you can see the actual conversation is reasonably simple to
mimic.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Fri, Feb 13, 2009 at 10:19 AM, Larry Ludwig <larrylud@gmail.com> wrote:
> I also mentioned to Luke about getting some metrics out of
> puppetmaster at least via command line.  I would LOVE to be able to
> monitor it via Cacti to see how performance is going.  Not sure what
> the metrics would be, but something to help with when to scale to
> bigger hardware.  At least from the baseline graphs we do (CPU, disk
> IO, memory, bandwidth) the Puppetmaster appears to be primarily CPU
> bound first, memory second.
>
I wrote a simple munin plugin, probably not to hard to convert to cacti...

but I like your approach better, if we could get some details (or maybe via
a signal) to the puppetmaster to return so valid results.. (e.g. up time,
number of compiled / failed catalogs etc)

> -L
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

> If you sit down and look at the traffic, either de-encrypting with
> ssldump or simply monitoring the unencrypted traffic between your SSL
> frontend and the puppet backends (assuming you''re using that sort
of a
> setup) you can see the actual conversation is reasonably simple to
> mimic.
I noticed someone had done this for performance testing you might be
able to use this info as a starting point for creating a test client.

http://www.forwardcamegrendel.org/benchmarking-puppet-jmeter

Paul

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Just out of curiosity, is there a problem with good old fashioned log
monitoring and cron?

I.e. logwatch -> email -> poke puppetmaster combined with cron to
check if puppetmaster is still rolling along.

In the case of PHP, it makes sense to monitor the page, however the
puppetmaster is a system daemon that should have things checking into
it and should be regularly generating logs like "compiled manifest for
blah.foo".  If you wanted to tie that into cacti, you could watch the
heuristics graph and see if there is a drop in trending.

Of course, this could also mean that your client networks all just
blew up but, either way, you have a serious problem.

Trevor

On Thu, Feb 12, 2009 at 21:19, Larry Ludwig <larrylud@gmail.com>
wrote:
>
> Yea I want to make sure puppetmaster returns some sort of valid
> result.  Though the monit for memory leaks is not a bad idea.
>
> More than just pinging a port.
>
> For example in our case we monitor PHP.  We''ve had PHP crap out
for a
> few customers, so we monitor a phpinfo page to make sure php returns a
> valid result.
>
> I also mentioned to Luke about getting some metrics out of
> puppetmaster at least via command line.  I would LOVE to be able to
> monitor it via Cacti to see how performance is going.  Not sure what
> the metrics would be, but something to help with when to scale to
> bigger hardware.  At least from the baseline graphs we do (CPU, disk
> IO, memory, bandwidth) the Puppetmaster appears to be primarily CPU
> bound first, memory second.
>
> -L
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Feb 13, 6:45 am, Trevor Vaughan <peiriann...@gmail.com>
wrote:
> Just out of curiosity, is there a problem with good old fashioned log
> monitoring and cron?
>
> I.e. logwatch -> email -> poke puppetmaster combined with cron to
> check if puppetmaster is still rolling along.
That is definitely an option and easier to implement. Though I do like
monitoring off server, via nagios in this case.  Also looking at the
logs won''t tell ya if some stupid change was made like blocking the
port via a firewall rule, or other network related problems.  I look
at it from the angle checks should occur at the same level you need
the service to operate.   Other checks like memory leaks I do consider
important if the daemon has a known issue otherwise overall memory and
swap usage monitoring should be enough.
>
> In the case of PHP, it makes sense to monitor the page, however the
> puppetmaster is a system daemon that should have things checking into
> it and should be regularly generating logs like "compiled manifest for
> blah.foo".
I could actually do via log files also, but choose to do it this way
for the same reasons above.

 If you wanted to tie that into cacti, you could watch
the
> heuristics graph and see if there is a drop in trending.
Yea I''m wondering out loud to the group what metrics should be graphed
with Puppetmaster.  Maybe per time period (ie 5min)
- Avg time to store  acatalog
- Avg time to compile a catalog
- amount of nodes connected
- Amount of modules

Maybe some other things related to the internals of puppetmaster
itself?
> Of course, this could also mean that your client networks all just
> blew up but, either way, you have a serious problem.
>
> Trevor
>
> On Thu, Feb 12, 2009 at 21:19, Larry Ludwig <larry...@gmail.com>
wrote:
>
> > Yea I want to make sure puppetmaster returns some sort of valid
> > result.  Though the monit for memory leaks is not a bad idea.
>
> > More than just pinging a port.
>
> > For example in our case we monitor PHP.  We''ve had PHP crap
out for a
> > few customers, so we monitor a phpinfo page to make sure php returns a
> > valid result.
>
> > I also mentioned to Luke about getting some metrics out of
> > puppetmaster at least via command line.  I would LOVE to be able to
> > monitor it via Cacti to see how performance is going.  Not sure what
> > the metrics would be, but something to help with when to scale to
> > bigger hardware.  At least from the baseline graphs we do (CPU, disk
> > IO, memory, bandwidth) the Puppetmaster appears to be primarily CPU
> > bound first, memory second.
>
> > -L
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Larry Ludwig wrote:
> I have puppet monitoring down pat, what''s the best way to monitor
> puppetmasterd?  We currently monitor the puppetmaster port but would
> like to monitor something to make sure not only if the port is alive
> but the puppetmaster is returning results.
Have you seen the puppet-test tool that comes packaged with puppet? It
seems like it fits what you''re looking for.

James Bellenger.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFJlbaP8TYLVDRcCYURAgdAAJ9FwfvDNJ6wAD2hx91zqtUk0XPlbACfRBoB
B8uatKmG0NOWJE5zIODHBGU=vEnH
-----END PGP SIGNATURE-----

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Nigel Kersten wrote:
> I''ve got a background job of working on trying to fake out a
proper
> getconfig request for a catalog to check timing data, but that''s
not
> functional yet.
I have a tool that does this in javascript here:
http://www.forwardcamegrendel.org/puppet-request-generator

It''s pretty handy as is and I make use of it pretty often, though
it''s
simple enough to rip out the javascript and reimplement in a different
language.

James Bellenger.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFJlbbI8TYLVDRcCYURAgGmAJoDt4PKbhENSEpzF7WUYZvczKbUKwCeP/RY
ImZg1Rkkln8zrkIuZVZmkgg=Y8Zg
-----END PGP SIGNATURE-----

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hmm interesting... I''ll check it out.

-L

On Feb 13, 1:07 pm, James Bellenger <ja...@forwardcamegrendel.org>
wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Nigel Kersten wrote:
> > I''ve got a background job of working on trying to fake out a
proper
> > getconfig request for a catalog to check timing data, but
that''s not
> > functional yet.
>
> I have a tool that does this in javascript
here:http://www.forwardcamegrendel.org/puppet-request-generator
>
> It''s pretty handy as is and I make use of it pretty often, though
it''s
> simple enough to rip out the javascript and reimplement in a different
> language.
>
> James Bellenger.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.7 (GNU/Linux)
>
> iD8DBQFJlbbI8TYLVDRcCYURAgGmAJoDt4PKbhENSEpzF7WUYZvczKbUKwCeP/RY
> ImZg1Rkkln8zrkIuZVZmkgg> =Y8Zg
> -----END PGP SIGNATURE-----
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---