Puppet users - Jan 2009 - Certificates were not trusted: certificate verify failed

Hello,

We are installing some Xen guests using puppet 0.24.5-1.el5 (from
http://people.redhat.com/dlutter/yum/rhel/5/x86_64/) on CentOS 5. The
Xen host is also a CentOS 5 running the same version for a puppet
master.

We have two such identical Xen hosts (running puppet master each). The
first one works perfectly for a while and we are not trying to deploy
the second one for redundancy.

The first Xen guest which tries to use puppet hits this apparently
familiar problem. Here is a sample output:

Wed Jan 14 11:25:32 +1100 2009 //Node[portal2-prod-
ascent.threatmetrix.com]/portal-prod-ascent/portal-ks/common-ks/File[/
etc/ssh/sshd_config] (err): Failed to retrieve current state of
resource: Certificates were not trusted: certificate verify failed
Could not describe /files/common/sshd_config: Certificates were not
trusted: certificate verify failed at /etc/puppet/svn/manifests/common-
ks.pp:78

We''ve been googl''ing this for two days now, we found both old
and
recent threads about this error as well as the page at
http://reductivelabs.com/trac/puppet/wiki/RubySSL-2007-006 but even
though we follow all the advise there and see the expected output (the
certificate verifies well using "openssl verify ...") we
can''t
convince puppet to accept the certificate.

One thing where our output doesn''t match the one in the instructions
on Wiki page are that the wiki page says:

"Look for subject=/C=US/ST=Ohio/O=The Ohio State University/
OU=Department of Mathematics/CN=puppet.math.ohio-state.edu" but
I''m
not sure whether this is just an example or we should really have this
specific CN in the certificate. We receive identical output on the
working server.

Instead, we have output as follows:

# openssl s_client -connect ds502.blueboxgrid.com:8140
CONNECTED(00000003)
depth=0 /CN=ds502.blueboxgrid.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /CN=ds502.blueboxgrid.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=ds502.blueboxgrid.com
   i:/CN=ds502.blueboxgrid.com
---
Server certificate
-----BEGIN CERTIFICATE-----
[....deleted....]
-----END CERTIFICATE-----
subject=/CN=ds502.blueboxgrid.com
issuer=/CN=ds502.blueboxgrid.com
---
No client certificate CA names sent
---
SSL handshake has read 1244 bytes and written 343 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID:
0F117816A195A5791AC317D30F6489E4874815B83DF734933A2B5B58DB9FC6F5
Session-ID-ctx:
    Master-Key:
E1DF12E889C1D3C5215EF451FD229BC29864666EF247789FE5179758C8018EF84D45AA6B9B552890110765BD71B65E64
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1231893176
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---


What else can we do? We are stuck in deployment of production system
because of this and can''t find what makes the first host tick while
the second one won''t accept anything.

I''ve tried also to completely remove and re-install puppet and puppet-
master (and remove the /var/lib/puppet and /etc/puppet directories)
but still get the same results.

Thanks.,

--Amos
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Oops. Replying to my own post, after re-reading one of the messages in
this group a few more times (http://groups.google.com/group/puppet-
users/msg/559819ffc956337e) while waiting for my experiments to run I
finally realised that it''s relevant to my too.

It turned out that the $fileserver and $urlbase were still pointing to
the other server (ds501). So I think what happened is that ds502 got
the certificate request, I signed it, then the puppet clients accessed
it, got hold of the manifests and even the templates, but they tried
to fetch the files from ds501 (the "working" server) which
didn''t
recognise the client certificate and refused access. Once we fixed
$fileserver to point to the right server things started dancing again.

TAKE AWAY from this (and other tackles by puppet problems): PLEASE
make it clear in the log messages where they are coming from and what
they complain about - is this error printed bythe puppet master? the
puppet client? Which host name? What string did it see in the
certificate vs. what did it expect?

Thanks.

--Amos

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

This error is printed from the client - e.g. the client doesn''t trust
the
server..

It is possible to make it work with your setup, however I would not
recommend to work this way, either have an external CA that signs for all
puppetmasters, or use certificate chain, it simplifies the setup and
troubleshooting...

Cheers,
Ohad

On Wed, Jan 14, 2009 at 10:04 AM, Amos Shapira
<amos.shapira@gmail.com>wrote:
>
> Oops. Replying to my own post, after re-reading one of the messages in
> this group a few more times (http://groups.google.com/group/puppet-
>
users/msg/559819ffc956337e<http://groups.google.com/group/puppet-users/msg/559819ffc956337e>)
> while waiting for my experiments to run I
> finally realised that it''s relevant to my too.
>
> It turned out that the $fileserver and $urlbase were still pointing to
> the other server (ds501). So I think what happened is that ds502 got
> the certificate request, I signed it, then the puppet clients accessed
> it, got hold of the manifests and even the templates, but they tried
> to fetch the files from ds501 (the "working" server) which
didn''t
> recognise the client certificate and refused access. Once we fixed
> $fileserver to point to the right server things started dancing again.
>
> TAKE AWAY from this (and other tackles by puppet problems): PLEASE
> make it clear in the log messages where they are coming from and what
> they complain about - is this error printed bythe puppet master? the
> puppet client? Which host name? What string did it see in the
> certificate vs. what did it expect?
>
> Thanks.
>
> --Amos
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

I ran into a similar issue today and I thought I would share.

My co-woker kickstarted a server, puppet gets installed in the
kickstart with the appropriate configs.
Puppet is responsible for installing and configuring numerous packages
and services.
It would try to get the cert, and puppetmaster would sign it perfectly
fine.
However, the client would not trust it.
I read around a bunch and nothing came to mind, I was moving the ssl
dir and deleting it on the client. Nothing.
Finally I checked the date.
The client-servers'' date (newly kickstarted) was way out of sync.
Way before the certificate on the puppetmaster was created even.
I set the time appropriately and everything worked as expected.

To remedy this I added a line in the kickstart file that is generated
from a template to set a date, (at least the clock will be somewhat
closer than two years behind)
-jason

On Jan 13, 7:30 pm, "Ohad Levy" <ohadl...@gmail.com>
wrote:
> This error is printed from the client - e.g. the client doesn''t
trust the
> server..
>
> It is possible to make it work with your setup, however I would not
> recommend to work this way, either have an external CA that signs for all
> puppetmasters, or use certificate chain, it simplifies the setup and
> troubleshooting...
>
> Cheers,
> Ohad
>
> On Wed, Jan 14, 2009 at 10:04 AM, Amos Shapira
<amos.shap...@gmail.com>wrote:
>
>
>
> > Oops. Replying to my own post, after re-reading one of the messages in
> > this group a few more times (http://groups.google.com/group/puppet-
> >
users/msg/559819ffc956337e<http://groups.google.com/group/puppet-users/msg/559819ffc956337e>)
> > while waiting for my experiments to run I
> > finally realised that it''s relevant to my too.
>
> > It turned out that the $fileserver and $urlbase were still pointing to
> > the other server (ds501). So I think what happened is that ds502 got
> > the certificate request, I signed it, then the puppet clients accessed
> > it, got hold of the manifests and even the templates, but they tried
> > to fetch the files from ds501 (the "working" server) which
didn''t
> > recognise the client certificate and refused access. Once we fixed
> > $fileserver to point to the right server things started dancing again.
>
> > TAKE AWAY from this (and other tackles by puppet problems): PLEASE
> > make it clear in the log messages where they are coming from and what
> > they complain about - is this error printed bythe puppet master? the
> > puppet client? Which host name? What string did it see in the
> > certificate vs. what did it expect?
>
> > Thanks.
>
> > --Amos
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Jan 13, 2009, at 8:04 PM, Amos Shapira wrote:
> TAKE AWAY from this (and other tackles by puppet problems): PLEASE
> make it clear in the log messages where they are coming from and what
> they complain about - is this error printed bythe puppet master? the
> puppet client? Which host name? What string did it see in the
> certificate vs. what did it expect?

Can you please make sure this is filed as a bug?

That''s the only way for us to make sure of anything. :)

-- 
The Number 1 Sign You Have Nothing to Do at Work...
     The 4th Division of Paperclips has overrun the Pushpin Infantry
     and General White-Out has called for a new skirmish.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---