Puppet users - Feb 2009 - Could not request certificate -> big logs, full /var, failure

I had this problem affect at least one server overnight; there might be 
more.

I put in a new version of puppet and puppetmaster, and had to step away 
before getting things all the way working.  I left it in a state where 
the puppetmaster was not running, but some puppetd were running.

(version 0.24.7)

Apparently, puppetd tries quite vigorously to connect.  It generated 
10GB of syslog and daemon.log overnight, full of this:



Feb 26 07:45:10 tr11 puppetd[14683]: : Certificate retrieval failed: 
Could not connect to puppet on port 8140
Feb 26 07:45:10 tr11 puppetd[14749]: Could not request certificate: 
Certificate retrieval failed: Could not connect to puppet on port 8140
Feb 26 07:45:10 tr11 puppetd[14683]: Could not request certificate: 
Certificate retrieval failed: Could not connect to puppet on port 8140
Feb 26 07:45:10 tr11 puppetd[14749]: Could not request certificate: 
Certificate retrieval failed: Could not connect to puppet on port 8140
Feb 26 07:45:10 tr11 puppetd[14683]: Could not request certificate: 
Certificate retrieval failed: Could not connect to puppet on port 8140
Feb 26 07:45:10 tr11 puppetd[14749]: Could not request certificate: 
Certificate retrieval failed: Could not connect to puppet on port 8140
Feb 26 07:45:10 tr11 puppetd[14683]: Could not request certificate: 
Certificate retrieval failed: Could not connect to puppet on port 8140
Feb 26 07:45:10 tr11 puppetd[14749]: Could not request certificate: 
Certificate retrieval failed: Could not connect to puppet on port 8140
Feb 26 07:45:10 tr11 puppetd[14683]: Could not request certificate: 
Certificate retrieval failed: Could not connect to puppet on port 8140



Then /var filled up and various things broke.


Needless to say, this is a Bad Thing.

I suggest that puppetd should refuse to try more than once every N 
seconds or minutes or whatever, regardless of:

* network failures, regardless of failure modu

* other errors, regardless of what error

* certificate problems of any nature

* stupid configuration

* stunningly idiotic configuration

* ruby / library / OS / etc versions

... because killing servers by filling /var, is not a good path to 
popularity  :-)


-- 
Kyle Cordes
http://kylecordes.com

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Kyle Cordes wrote:
> Apparently, puppetd tries quite vigorously to connect.  It generated 
> 10GB of syslog and daemon.log overnight, full of this:
> Feb 26 07:45:10 tr11 puppetd[14683]: : Certificate retrieval failed: 
> Could not connect to puppet on port 8140

A more enlightened syslogd on another machine reported on the situation 
numerically:


Feb 26 07:45:01 tr10 puppetd[24101]: Could not request certificate: 
Certificate retrieval failed: Could not connect to puppet on port
  8140
Feb 26 07:45:32 tr10 last message repeated 35255 times
Feb 26 07:46:33 tr10 last message repeated 71679 times
Feb 26 07:47:34 tr10 last message repeated 71426 times
Feb 26 07:48:35 tr10 last message repeated 71472 times
Feb 26 07:49:36 tr10 last message repeated 71349 times
Feb 26 07:50:37 tr10 last message repeated 71453 times
Feb 26 07:51:38 tr10 last message repeated 71495 times
Feb 26 07:52:39 tr10 last message repeated 71407 times
Feb 26 07:53:40 tr10 last message repeated 71657 times
Feb 26 07:54:01 tr10 last message repeated 25631 times



70,000 times per minute, is a bit too often to attempt to connect  ;-)


-- 
Kyle Cordes
http://kylecordes.com

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Kyle Cordes wrote:
> 70,000 times per minute, is a bit too often to attempt to connect  ;-)
Look for "-w 0" in the puppetd arguments, common in Debian and Ubuntu 
packages. Change it to something like "-w 120" to make it only check
for
certificates every 2 minutes. I''ve had the exact same thing happen in 
the past.

-- 
Mike Renfro  / R&D Engineer, Center for Manufacturing Research,
931 372-3601 / Tennessee Technological University -- renfro@tntech.edu

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Mike Renfro wrote:
> Look for "-w 0" in the puppetd arguments, common in Debian and
Ubuntu
> packages. Change it to something like "-w 120" to make it only
check for
> certificates every 2 minutes. I''ve had the exact same thing happen
in

Ouch. This seems like:

a) a really awful choice for the Debian / Ub packages

and also

b) a feature that Puppet would be much better off without.


I can''t imagine a compelling use case for a "pound on the server 
continuously" setting; and I''m not surprised, with some years of 
software development experience, to find that given the existence of 
such a capability, that someone would set it as the *default*.


By the way, how do you install puppetd, on Deb systems?

a) use the Debian-provided package, then go back and edit the settings

b) make your own package

c) some other means, not using a package

?

-- 
Kyle Cordes
http://kylecordes.com

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Thu, Feb 26, 2009 at 6:58 AM, Kyle Cordes <kyle@kylecordes.com>
wrote:
> Ouch. This seems like:
>
> a) a really awful choice for the Debian / Ub packages
Yep.
> b) a feature that Puppet would be much better off without.
>
>
> I can''t imagine a compelling use case for a "pound on the
server
> continuously" setting; and I''m not surprised, with some years
of
> software development experience, to find that given the existence of
> such a capability, that someone would set it as the *default*.
Just because you can''t imagine it, doesn''t mean it
doesn''t exist. I
use -w 0 intentionally and happily, and there is a compelling use case
for my environment. I can''t go into it without getting into internal
details I can''t discuss, but, like most settings in Puppet, I believe
it exists for a reason.
> By the way, how do you install puppetd, on Deb systems?
>
> a) use the Debian-provided package, then go back and edit the settings
>
> b) make your own package
I definitely recommend this route over all the others. We maintain our
own repository of Debian packages for things the Debian packagers have
screwed up, or things we really need to be newer, etc.
> c) some other means, not using a package
Don''t do this on Debian machines, it will only end in tears ;-)

--Paul

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Paul Lathrop wrote:
>> I can''t imagine a compelling use case for a "pound on the
server
>> continuously" setting; and I''m not surprised, with some
years of
> Just because you can''t imagine it, doesn''t mean it
doesn''t exist. I
> use -w 0 intentionally and happily, and there is a compelling use case
> for my environment. I can''t go into it without getting into
internal
> details I can''t discuss, but, like most settings in Puppet, I
believe
> it exists for a reason.

I am sure there is a good use for some aspect of the -w 0 feature; the 
part I don''t think there is a good use for, is writing 70,000 entries 
per minute to syslog, or for trying to connect in a tight loop for 12 
hours.  I am sure there is a way for it to do what you need it to do, 
without doing those other things.


 >> b) make your own package
 >
 > I definitely recommend this route over all the others. We maintain our


This is the path I''m on.  My comment above is just a hope to make life 
slightly better for the next person who types "apt-get install puppet"
with Debian or Ubuntu out of the box.


-- 
Kyle Cordes
http://kylecordes.com

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Thu, Feb 26, 2009 at 8:49 AM, Kyle Cordes <kyle@kylecordes.com>
wrote:
> I am sure there is a good use for some aspect of the -w 0 feature; the
> part I don''t think there is a good use for, is writing 70,000
entries
> per minute to syslog, or for trying to connect in a tight loop for 12
> hours.  I am sure there is a way for it to do what you need it to do,
> without doing those other things.
You may be right. On the other hand, powerful tools *often* provide
ways of shooting oneself in the foot. One might make a similar
argument to yours for the ability to purge resources. The following
puppet snippet will probably cause you no end of headaches:

resources { "package": purge => true; }

I have yet to meet the daemon that couldn''t be mis-configured to fill
up as much log as you allow it to; I don''t think that is a good
argument for removing functionality. I fight "dumbing down" tools
wherever I can, and this seems like one of those cases. The correct
way to handle this case is to remember that if you don''t rotate your
logs correctly, a rogue daemon will fill up your filesystem, and act
accordingly.

Your complaint is valid; this should not be the default behavior, and
it *isn''t* -- your beef should be with the package maintainer who
misconfigured the tool, not with the tool itself.

--Paul

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On 2/26/2009 11:14 AM, Paul Lathrop wrote:
> On Thu, Feb 26, 2009 at 8:49 AM, Kyle Cordes <kyle@kylecordes.com>
wrote:
>> I am sure there is a good use for some aspect of the -w 0 feature; the
>> part I don''t think there is a good use for, is writing 70,000
entries
>> per minute to syslog, or for trying to connect in a tight loop for 12
>> hours.  I am sure there is a way for it to do what you need it to do,
>> without doing those other things.
> Your complaint is valid; this should not be the default behavior, and
> it *isn''t* -- your beef should be with the package maintainer who
> misconfigured the tool, not with the tool itself.
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=509566 is probably tbe 
most relevant discussion, and matches my dim recollections. Back when I 
first started using puppet, "-w 0" caused the first run of puppetd to 
ask for a cert, and if it didn''t get a response, it would exit out and 
wait for you to restart it. But any other argument to -w would cause 
puppetd to hang the boot process until its certificate got signed. Both 
were annoying, but at least "-w 0" let you log into the system to 
restart puppetd.

Sometime later, "-w 0" was no longer a special case, and caused
puppetd
to behave like you''re seeing now: tight loops checking for certificates
and writing errors into the logs. If "-w 120" doesn''t cause a
puppetd
hang any more, then I''d guess that ought to be the default again. I can
afford to burn 2 minutes on reinstalling a new system (I''ve got base 
preseed installations down into the 5 minute range, and can get a 
cluster node tied to the Active Directory and install all my software 
within an hour or so).

-- 
Mike Renfro  / R&D Engineer, Center for Manufacturing Research,
931 372-3601 / Tennessee Technological University

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Paul Lathrop wrote:
> You may be right. On the other hand, powerful tools *often* provide
> ways of shooting oneself in the foot. One might make a similar

I heartily agree; what I am looking at is the difference between:

a) it is possible to type "sudo rm -rf /"

b) making rm default to a path of "/" and default -r -f options ON,
and
include it in sudoers by default.


Obviously B is an awful idea, whether it''s done by the tool or by the 
distribution.  If I was the author of rm, and I saw a distribution do 
such a thing, I''d recoil in horror and go find a way to make it harder 
to use my tool badly, while also begging any distro doing that to please 
stop.


-- 
Kyle Cordes
http://kylecordes.com

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Thu, Feb 26, 2009 at 9:38 AM, Mike Renfro <renfro@tntech.edu>
wrote:
>
> On 2/26/2009 11:14 AM, Paul Lathrop wrote:
>> On Thu, Feb 26, 2009 at 8:49 AM, Kyle Cordes
<kyle@kylecordes.com> wrote:
>>> I am sure there is a good use for some aspect of the -w 0 feature;
the
>>> part I don''t think there is a good use for, is writing
70,000 entries
>>> per minute to syslog, or for trying to connect in a tight loop for
12
>>> hours.  I am sure there is a way for it to do what you need it to
do,
>>> without doing those other things.
>
>> Your complaint is valid; this should not be the default behavior, and
>> it *isn''t* -- your beef should be with the package maintainer
who
>> misconfigured the tool, not with the tool itself.
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=509566 is probably tbe
> most relevant discussion, and matches my dim recollections. Back when I
> first started using puppet, "-w 0" caused the first run of
puppetd to
> ask for a cert, and if it didn''t get a response, it would exit out
and
> wait for you to restart it. But any other argument to -w would cause
> puppetd to hang the boot process until its certificate got signed. Both
> were annoying, but at least "-w 0" let you log into the system to
> restart puppetd.
>
> Sometime later, "-w 0" was no longer a special case, and caused
puppetd
> to behave like you''re seeing now: tight loops checking for
certificates
> and writing errors into the logs. If "-w 120" doesn''t
cause a puppetd
> hang any more, then I''d guess that ought to be the default again.
I can
> afford to burn 2 minutes on reinstalling a new system (I''ve got
base
> preseed installations down into the 5 minute range, and can get a
> cluster node tied to the Active Directory and install all my software
> within an hour or so).
Thanks for the link. There *is* a Puppet bug here; the documentation
doesn''t match the behavior. I''ll take a look.

--Paul

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Thu, Feb 26, 2009 at 9:58 AM, Kyle Cordes <kyle@kylecordes.com>
wrote:
> I heartily agree; what I am looking at is the difference between:
>
> a) it is possible to type "sudo rm -rf /"
>
> b) making rm default to a path of "/" and default -r -f options
ON, and
> include it in sudoers by default.
>
>
> Obviously B is an awful idea, whether it''s done by the tool or by
the
> distribution.  If I was the author of rm, and I saw a distribution do
> such a thing, I''d recoil in horror and go find a way to make it
harder
> to use my tool badly, while also begging any distro doing that to please
> stop.
Ah, there''s the key difference. What I would do is file a bug report
calling them out for their horrible use of ''rm'' while not
trying to
make it hard to use intentionally-added functionality.

Cool, we can agree to disagree on that :-P

--Paul

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---