Puppet users - Mar 2011 - failed to retrieve certificate on Amazon EC2

Hi,

I''m using puppet on EC2 to setup my VMs with the following
configuration:

# puppetd --version
0.25.5
# uname -a
Linux hostname.domain 2.6.16-xenU #1 SMP Mon May 28 03:41:49 SAST 2007 i686
i686 i386 GNU/Linux

But I keep facing some timeout from puppetd:

warning: peer certificate won''t be verified in this SSL session
Exiting; failed to retrieve certificate and waitforcert is disabled

Puppetmaster is running on autosign, and I can see in the puppetmaster logs
that the puppets are actually able to connect and request a certificate:

info: Could not find certificate for ''hostname.domain''

But, nothing else seems to happens on puppetmaster side and the puppetd
finally timeout.

Did anybody runned into this situation ? Any idea on what could be
triggering this ?

-- 
Romain PELISSE,
*"The trouble with having an open mind, of course, is that people will
insist on coming along and trying to put things in it" -- Terry Pratchett*
http://belaran.eu/wordpress/belaran

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Try puppetd --test --waitforcert 15 

I think it''s complaining about waitforcert because you need that flag
to retrieve the cert from the server.

But it would have been nice to see the actual command you used to produce that
error. :)

The number is the interval that the puppet client will pause between requests to
the server for it''s cert and is a required parameter.

At that point, back on the server you should see what facter considers to be the
fqdn of the requesting machine when you issue

puppetca --list

Then sign

puppetca --sign host.domain.com

I''d recommend turning off autosign on the puppet master as
it''s a security risk.



Hope that is helpful.

Sent from my iPhone

On Mar 11, 2011, at 8:04 AM, Romain Pelisse <belaran@gmail.com> wrote:
> Hi,
> 
> I''m using puppet on EC2 to setup my VMs with the following
configuration:
> 
> # puppetd --version
> 0.25.5
> # uname -a
> Linux hostname.domain 2.6.16-xenU #1 SMP Mon May 28 03:41:49 SAST 2007 i686
i686 i386 GNU/Linux
> 
> But I keep facing some timeout from puppetd: 
> 
> warning: peer certificate won''t be verified in this SSL session
> Exiting; failed to retrieve certificate and waitforcert is disabled
> 
> Puppetmaster is running on autosign, and I can see in the puppetmaster logs
that the puppets are actually able to connect and request a certificate:
> 
> info: Could not find certificate for ''hostname.domain''
> 
> But, nothing else seems to happens on puppetmaster side and the puppetd
finally timeout.
> 
> Did anybody runned into this situation ? Any idea on what could be
triggering this ?
> 
> -- 
> Romain PELISSE,
> "The trouble with having an open mind, of course, is that people will
insist on coming along and trying to put things in it" -- Terry Pratchett
> http://belaran.eu/wordpress/belaran
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

Thanks for you quick reply.

Try puppetd --test --waitforcert 15
>
I actually tried --waitforcert 480000 ! But it still hang and end up into a
certificate timeout : puppetd[2078]: Could not request certificate:
Connection refused - connect(2)

> I think it''s complaining about waitforcert because you need that
flag to
> retrieve the cert from the server.
>
> But it would have been nice to see the actual command you used to produce
> that error. :)
>
puppet is running as "regular" red hat service, so there is no
"command"
(well there is but I just let the service runs). I have nothing fancy in my
[puppetd] configuration:

[puppetd]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuratiion.  Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is ''$confdir/classes.txt''.
    classfile = $vardir/classes.txt

    # Where puppetd caches the local configuration.  An
    # extension indicating the cache format is added automatically.
    # The default value is ''$confdir/localconfig''.
    localconfig = $vardir/localconfig

    server = puppetmaster.domain

The number is the interval that the puppet client will pause
between
> requests to the server for it''s cert and is a required parameter.
>
Ah, ok , so maybe having a big number here ( --waitforcert 480000) is a bad
idea....

> At that point, back on the server you should see what facter considers to
> be the fqdn of the requesting machine when you issue
>
> puppetca --list
>
> Then sign
>
> puppetca --sign <http://host.domain.com>host.domain.com
>
> I''d recommend turning off autosign on the puppet master as
it''s a security
> risk.
>
For now I need autosign, because I''m creating instance "on the
fly" - so I
don''t want to "manually" accept each one. (but I''m
aware of the security
risk of this setup).


>
>
>
> Hope that is helpful.
>
> Sent from my iPhone
>
> On Mar 11, 2011, at 8:04 AM, Romain Pelisse <belaran@gmail.com>
wrote:
>
> Hi,
>
> I''m using puppet on EC2 to setup my VMs with the following
configuration:
>
> # puppetd --version
> 0.25.5
> # uname -a
> Linux hostname.domain 2.6.16-xenU #1 SMP Mon May 28 03:41:49 SAST 2007 i686
> i686 i386 GNU/Linux
>
> But I keep facing some timeout from puppetd:
>
> warning: peer certificate won''t be verified in this SSL session
> Exiting; failed to retrieve certificate and waitforcert is disabled
>
> Puppetmaster is running on autosign, and I can see in the puppetmaster logs
> that the puppets are actually able to connect and request a certificate:
>
> info: Could not find certificate for ''hostname.domain''
>
> But, nothing else seems to happens on puppetmaster side and the puppetd
> finally timeout.
>
> Did anybody runned into this situation ? Any idea on what could be
> triggering this ?
>
> --
> Romain PELISSE,
> *"The trouble with having an open mind, of course, is that people will
> insist on coming along and trying to put things in it" -- Terry
Pratchett*
>
<http://belaran.eu/wordpress/belaran>http://belaran.eu/wordpress/belaran
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>


-- 
Romain PELISSE,
*"The trouble with having an open mind, of course, is that people will
insist on coming along and trying to put things in it" -- Terry Pratchett*
http://belaran.eu/wordpress/belaran

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

> I actually tried --waitforcert 480000 ! But it still hang and end up into a
> certificate timeout : puppetd[2078]: Could not request certificate:
> Connection refused - connect(2)
"Connection refused" sounds like networking or a firewall is in your
way. On your new Puppet client, can you "telnet puppetmaster.domain
8140"? Assuming your Puppet Master is running on the default port of
8140.

What''s the output of "puppetd --test" ?
> > I think it''s complaining about waitforcert because you need
that flag to
> > retrieve the cert from the server.
>
> > But it would have been nice to see the actual command you used to
produce
> > that error. :)
>
> puppet is running as "regular" red hat service, so there is no
"command"
> (well there is but I just let the service runs). I have nothing fancy in my
> [puppetd] configuration:
>
> [puppetd]
>     # The file in which puppetd stores a list of the classes
>     # associated with the retrieved configuratiion.  Can be loaded in
>     # the separate ``puppet`` executable using the ``--loadclasses``
>     # option.
>     # The default value is ''$confdir/classes.txt''.
>     classfile = $vardir/classes.txt
>
>     # Where puppetd caches the local configuration.  An
>     # extension indicating the cache format is added automatically.
>     # The default value is ''$confdir/localconfig''.
>     localconfig = $vardir/localconfig
>
>     server = puppetmaster.domain
>
> The number is the interval that the puppet client will pause between
>
> > requests to the server for it''s cert and is a required
parameter.
>
> Ah, ok , so maybe having a big number here ( --waitforcert 480000) is a bad
> idea....
>
> > At that point, back on the server you should see what facter considers
to
> > be the fqdn of the requesting machine when you issue
>
> > puppetca --list
>
> > Then sign
>
> > puppetca --sign <http://host.domain.com>host.domain.com
>
> > I''d recommend turning off autosign on the puppet master as
it''s a security
> > risk.
>
> For now I need autosign, because I''m creating instance "on
the fly" - so I
> don''t want to "manually" accept each one. (but
I''m aware of the security
> risk of this setup).
>
>
>
>
>
> > Hope that is helpful.
>
> > Sent from my iPhone
>
> > On Mar 11, 2011, at 8:04 AM, Romain Pelisse <bela...@gmail.com>
wrote:
>
> > Hi,
>
> > I''m using puppet on EC2 to setup my VMs with the following
configuration:
>
> > # puppetd --version
> > 0.25.5
> > # uname -a
> > Linux hostname.domain 2.6.16-xenU #1 SMP Mon May 28 03:41:49 SAST 2007
i686
> > i686 i386 GNU/Linux
>
> > But I keep facing some timeout from puppetd:
>
> > warning: peer certificate won''t be verified in this SSL
session
> > Exiting; failed to retrieve certificate and waitforcert is disabled
>
> > Puppetmaster is running on autosign, and I can see in the puppetmaster
logs
> > that the puppets are actually able to connect and request a
certificate:
>
> > info: Could not find certificate for
''hostname.domain''
>
> > But, nothing else seems to happens on puppetmaster side and the
puppetd
> > finally timeout.
>
> > Did anybody runned into this situation ? Any idea on what could be
> > triggering this ?
>
> > --
> > Romain PELISSE,
> > *"The trouble with having an open mind, of course, is that people
will
> > insist on coming along and trying to put things in it" -- Terry
Pratchett*
> >
<http://belaran.eu/wordpress/belaran>http://belaran.eu/wordpress/belaran
>
> > --
> > You received this message because you are subscribed to the Google
Groups
> > "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> > puppet-users+unsubscribe@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/puppet-users?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google
Groups
> > "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> > puppet-users+unsubscribe@googlegroups.com.
> > For more options, visit this group at
> >http://groups.google.com/group/puppet-users?hl=en.
>
> --
> Romain PELISSE,
> *"The trouble with having an open mind, of course, is that people will
> insist on coming along and trying to put things in it" -- Terry
Pratchett*http://belaran.eu/wordpress/belaran
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 11 March 2011 14:41, Luke Bigum <Luke.Bigum@lmax.com> wrote:
> > I actually tried --waitforcert 480000 ! But it still hang and end up
into
> a
> > certificate timeout : puppetd[2078]: Could not request certificate:
> > Connection refused - connect(2)
>
> "Connection refused" sounds like networking or a firewall is in
your
> way. On your new Puppet client, can you "telnet puppetmaster.domain
> 8140"? Assuming your Puppet Master is running on the default port of
> 8140.
>

Yep it does sounds like that, but as I can see the notice with the new
instance name appears on the puppetmaster side I assume it wasn''t the
firewall.

However, if I telnet on port 8140 I can''t connect...

I''m starting to wonder if my puppetmaster can''t just handle
the 4 instances
starting up at the same time. It''s far from being a big scale but
I''m still
using only on instance of puppetmasterd running with webbrick... Can this
default setup supports more than one client at the time ?

> What''s the output of "puppetd --test" ?
>

puppetd --test
warning: peer certificate won''t be verified in this SSL session

and then the certificate error.

-- 
Romain PELISSE,
*"The trouble with having an open mind, of course, is that people will
insist on coming along and trying to put things in it" -- Terry Pratchett*
http://belaran.eu/wordpress/belaran

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mar 11, 1:49 pm, Romain Pelisse <bela...@gmail.com>
wrote:
> On 11 March 2011 14:41, Luke Bigum <Luke.Bi...@lmax.com> wrote:
> > "Connection refused" sounds like networking or a firewall is
in your
> > way. On your new Puppet client, can you "telnet
puppetmaster.domain
> > 8140"? Assuming your Puppet Master is running on the default port
of
> > 8140.
>
> Yep it does sounds like that, but as I can see the notice with the new
> instance name appears on the puppetmaster side I assume it wasn''t
the
> firewall.
>
> However, if I telnet on port 8140 I can''t connect...
>
> I''m starting to wonder if my puppetmaster can''t just
handle the 4 instances
> starting up at the same time. It''s far from being a big scale but
I''m still
> using only on instance of puppetmasterd running with webbrick... Can this
> default setup supports more than one client at the time ?
I don''t know webrick much at all to be honest, but I''m sure it
can
support more than one. Forgive the stupid question but you don''t have
another host with the same name somewhere that you''re confusing log
messages with?

You could always try spin up another puppetmaster process on a
different port and try connect to that - check the command line
options, I''m going by memory:

puppetmaster --no-daemonize --verbose --server_port=8141

Which should print to stdout, and only be your new server (trying) to
connect. That''ll confirm/deny a network problem.
> > What''s the output of "puppetd --test" ?
>
> puppetd --test
> warning: peer certificate won''t be verified in this SSL session
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>
> I don''t know webrick much at all to be honest, but I''m
sure it can
> support more than one. Forgive the stupid question but you don''t
have
> another host with the same name somewhere that you''re confusing
log
> messages with?
>
No need to apologies, it''s always good to check this kind of thing
twice.

Ok, I''ve found one of the root cause of the problem. My puppetmaster is
not
able to DNS resolve the name - Amazon Route DNS 53 is not responding quickly
enough.

Of course, by the time the DNS name is available, I run into an other issue
with puppet:

 puppetd --test --verbose --waitforcert 15
err: Could not request certificate: Retrieved certificate does not match
private key; please remove certificate from server and regenerate it with
the current key
err: Could not retrieve catalog from remote server: Retrieved certificate
does not match private key; please remove certificate from server and
regenerate it with the current key
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

This is because - I assume, a first certificate associated with the
unresolved DNS name, has been created...



-- 
Romain PELISSE,
*"The trouble with having an open mind, of course, is that people will
insist on coming along and trying to put things in it" -- Terry Pratchett*
http://belaran.eu/wordpress/belaran

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.