Puppet users - Sep 2010 - certificate problem ; puppetca can't find cert request ?

Hello,

We recently re-deployed puppet certificates in our environment.  I 
removed and regenerated the certificates for all of the clients save for 
one : the puppetmaster server itself.

As one might expect, when i run puppetd --test on the puppetmaster 
server, i get :

err: Could not request certificate: Retrieved certificate does not match 
private key; please remove certificate from server and regenerate it 
with the current key

I removed /var/lib/puppet/ssl/certs/<puppetmaster_fqdn>.pem , then ran 
puppetd with --waitforcert <puppetmaster_fqdn>.  Unfortunately, when i 
run a puppetca --list --all , <puppetmaster_fqdn> is not listed, even 
though there is very clearly a request pem in 
/var/lib/puppet/ssl/certificate_requests .

Executing puppetca --clean <puppetmaster_fqdn> removes the private key 
(as expected), but does not change the error condition.  I also tried 
puppetca --revoke <puppetmaster_fqdn> ; no change.

I also tried removing every instance <puppetmaster>.pem from 
/var/lib/puppet/ssl/* ; this also did nothing.  Finally, i saw that 
<puppetmaster_fqdn> was listed in only one spot : 
/var/lib/puppet/ssl/ca/inventory.txt .  Removing the line from this file 
also does nothing (as expected).

In the archives, one solution proposed for this problem is to rm -rf 
/var/lib/puppet/ssl and let puppet regenerate it all ; this is fine on 
the clients, i suppose, but i hesitate to do it on the puppetmaster, as 
i''d rather not have to start from scratch with the certificates of all 
the clients again.

I''m running puppet 0.25.5 on CentOS 5.5 x86_64.

Any ideas ?

Thank you all.


-- 
Daniel Maher <dma AT witbe DOT net>
"The Internet is completely over." -- Prince

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Sep 30, 2010 at 6:20 AM, Daniel Maher <dma@witbe.net>
wrote:
> I removed /var/lib/puppet/ssl/certs/<puppetmaster_fqdn>.pem , then
ran
> puppetd with --waitforcert <puppetmaster_fqdn>.  Unfortunately, when
i run a
> puppetca --list --all , <puppetmaster_fqdn> is not listed, even
though there
> is very clearly a request pem in /var/lib/puppet/ssl/certificate_requests .
So first bbackup you ssl dir, then try the following command:

puppetca --clean <puppetmaster_fqdn>
puppetca --generate <puppetmaster_fqdn>
--certdnsname="puppet;puppetmaster"

In certdnsname, provide a list of DNS cname to puppet master, and
include puppet for convenience.
> Executing puppetca --clean <puppetmaster_fqdn> removes the private
key (as
> expected), but does not change the error condition.  I also tried puppetca
> --revoke <puppetmaster_fqdn> ; no change.
This will add a few more wrinkles, once the certificate is generated,
it will list as revoked:
# puppet cert --list --all
- <puppetmaster_fqdn>
(D7:B1:1B:33:80:51:2C:11:24:C5:EF:CE:92:04:4A:24) (certificate
revoked)

If you have a backup of the certificate revocation list restore it. I
don''t know the command to undo a single revoke off hand.

Thanks,

Nan

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 09/30/2010 05:49 PM, Nan Liu wrote:
> On Thu, Sep 30, 2010 at 6:20 AM, Daniel Maher<dma@witbe.net>  wrote:
>> I removed /var/lib/puppet/ssl/certs/<puppetmaster_fqdn>.pem ,
then ran
>> puppetd with --waitforcert<puppetmaster_fqdn>.  Unfortunately,
when i run a
>> puppetca --list --all ,<puppetmaster_fqdn>  is not listed, even
though there
>> is very clearly a request pem in
/var/lib/puppet/ssl/certificate_requests .
>
> So first bbackup you ssl dir, then try the following command:
>
> puppetca --clean<puppetmaster_fqdn>
> puppetca --generate<puppetmaster_fqdn> 
--certdnsname="puppet;puppetmaster"
>
> In certdnsname, provide a list of DNS cname to puppet master, and
> include puppet for convenience.
>
Thank you for the advice ; unfortunately, as i had already revoked the 
certificate, cleaning and re-issuing was not a possibility.  I ended up 
biting the bullet and just wiping out and re-initialising the 
certificates across the board.  Thank god for clusterssh.

The moral of the story here, i suppose, is that /var/lib/puppet/ssl/ 
should be backed up and set aside for every client (including the 
puppetmaster), and that if certificates need to be re-issued from the 
ground-up, it''s going to be trouble.  This, i suppose, is why some 
people opt to move to an external certificate provider within their 
organisation - it''s really not a bad idea.

Finally, is "--certdnsname" documented anywhere ?  In 0.25.5, at
least,
puppetca --help doesn''t mention it, and neither does the manpage.  On 
the puppet website, a search for "certdnsname" only leads to a
reference
in Release_Notes.  If you already know to search for it, google will 
give you some hits from the mailing list, and some blogs, but you have 
to know to look for it in the first place. :P

Thanks again.

-- 
Daniel Maher <dma AT witbe DOT net>
"The Internet is completely over." -- Prince

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Oct 1, 2010, at 2:50 AM, Daniel Maher wrote:
> On 09/30/2010 05:49 PM, Nan Liu wrote:
>> On Thu, Sep 30, 2010 at 6:20 AM, Daniel Maher<dma@witbe.net> 
wrote:
>>> I removed /var/lib/puppet/ssl/certs/<puppetmaster_fqdn>.pem ,
then ran
>>> puppetd with --waitforcert<puppetmaster_fqdn>. 
Unfortunately, when i run a
>>> puppetca --list --all ,<puppetmaster_fqdn>  is not listed,
even though there
>>> is very clearly a request pem in
/var/lib/puppet/ssl/certificate_requests .
>> 
>> So first bbackup you ssl dir, then try the following command:
>> 
>> puppetca --clean<puppetmaster_fqdn>
>> puppetca --generate<puppetmaster_fqdn> 
--certdnsname="puppet;puppetmaster"
>> 
>> In certdnsname, provide a list of DNS cname to puppet master, and
>> include puppet for convenience.
>> 
> 
> Thank you for the advice ; unfortunately, as i had already revoked the
certificate, cleaning and re-issuing was not a possibility.  I ended up biting
the bullet and just wiping out and re-initialising the certificates across the
board.  Thank god for clusterssh.
You can re-issue a certificate with the same name as a revoked certificate
without any issues.  You should never, ever have to wipe out certificates across
the board.  If you do this, there''s a solution to the problem
you''ve likely overlooked.
> The moral of the story here, i suppose, is that /var/lib/puppet/ssl/ should
be backed up and set aside for every client (including the puppetmaster), and
that if certificates need to be re-issued from the ground-up, it''s
going to be trouble.  This, i suppose, is why some people opt to move to an
external certificate provider within their organisation - it''s really
not a bad idea.
Again, you don''t need to backup /var/lib/puppet/ssl on the clients, and
in fact it''s "best practice" from an SSL key management point
of view not to do so because it fundamentally means copying the private keys
somewhere else.

Consider host "jeff"  If I do puppetca --revoke jeff on the puppet ca,
then ensure $cadir/certs/jeff.pem is not present on the ca, then wipe out
$ssldir on the node "jeff" only (it''s certificate is revoked
and invalid), you''ll be able to go through the certificate signing
process just as if "jeff" were a brand new node unknown to puppet.
> Finally, is "--certdnsname" documented anywhere ?  In 0.25.5, at
least, puppetca --help doesn''t mention it, and neither does the
manpage.  On the puppet website, a search for "certdnsname" only leads
to a reference in Release_Notes.  If you already know to search for it, google
will give you some hits from the mailing list, and some blogs, but you have to
know to look for it in the first place. :P
certdnsnames (Note the plural form) is documented in the Configuration Reference
at:

http://docs.puppetlabs.com/references/stable/configuration.html#certdnsnames

Hope this helps,
-Jeff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Fri, Oct 1, 2010 at 2:50 AM, Daniel Maher <dma@witbe.net>
wrote:
>
> Thank you for the advice ; unfortunately, as i had already revoked the
> certificate, cleaning and re-issuing was not a possibility.  I ended up
> biting the bullet and just wiping out and re-initialising the certificates
> across the board.  Thank god for clusterssh.
>
Ah, I''m a step too late for you. I suppose this would be helpful if
someone run across this problem again.

Here''s the steps to recover an accidental revocation:

List revoked certs in CRL to find the serial number:
# openssl crl -in /var/lib/puppet/ca/ca_crl.pem -noout -text
openssl crl -in ca/ca_crl.pem -noout -text
Certificate Revocation List (CRL):
        Version 2 (0x1)
...

Revoked Certificates:
    Serial Number: 11
        Revocation Date: Sep 25 14:50:35 2010 GMT
...
                Key Compromise
    Serial Number: 19
        Revocation Date: Sep 24 20:51:22 2010 GMT
...
                Key Compromise

This matches with the history in inventory.txt:
# Inventory of signed certificates
# SERIAL NOT_BEFORE NOT_AFTER SUBJECT
...
0x0011 2010-09-22T11:03:55GMT 2015-09-21T11:03:55GMT /CN=foo
...
0x0019 2010-09-23T20:50:28GMT 2015-09-22T20:50:28GMT /CN=puppetmaster.training

In this case 0x0019 was a mistake:
Remove both the ca_crl.pem and crl.pem, revoke all compromised certs
and skip the ones that was a mistake and you are set:
# rm /var/lib/puppet/ca/ca_crl.pem
# rm /var/lib/puppet/crl.pem
# puppet cert --revoke foo
# openssl crl -in /var/lib/puppet/ca/ca_crl.pem -noout -text

At this point 0x0019 will no longer be in the list.

Thanks,

Nan

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.