Puppet users - Feb 2012 - SSL certificates issues with some of the nodes

Hi,

I am using puppet open source & their are above 2000 nodes in my
network & getting SSL certificates issues with some of the remote
nodes & they are not in sync with puppet server where others are in
sync with server & date of the nodes systems are in sync with ntp
server, also tried the following as,

   1. Deleted the nodes certificate on the puppet server
   2. Deleted the SSL directory of the agent.
   3. Run puppet agent --test on that agent to generate a new
certificate request.

but I have to regularly repeat the above process within a timeframe
(weekly or monthly) so is there any other way other than this so that
all of my agent should be in sync.

Also how can I find the sync & unsync count    ?

Thanks
mukulm

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 10, 8:59 am, mukulm <smilemukul2...@gmail.com>
wrote:
> Hi,
>
> I am using puppet open source & their are above 2000 nodes in my
> network & getting SSL certificates issues with some of the remote
> nodes & they are not in sync with puppet server where others are in
> sync with server & date of the nodes systems are in sync with ntp
> server, also tried the following as,
>
>    1. Deleted the nodes certificate on the puppet server
>    2. Deleted the SSL directory of the agent.
>    3. Run puppet agent --test on that agent to generate a new
> certificate request.
>
> but I have to regularly repeat the above process within a timeframe
> (weekly or monthly) so is there any other way other than this so that
> all of my agent should be in sync.
>
> Also how can I find the sync & unsync count    ?
>
> Thanks
> mukulm
To help clarify things, are all your nodes and masters in sync with
the same NTP server? This is a common source of the issues you are
describing.

If you are still running into issues can you post (pastbin) the errors
so
others can help you troubleshoot?

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

I am getting the following error message as after removing the SSL
certificates from the node & server,

*err: Could not request certificate: Retrieved certificate does not match
private key; please remove certificate from server and regenerate it with
the current key*

Thanks
mukulm


On Fri, Feb 10, 2012 at 7:56 PM, Kelsey Hightower
<kelsey@puppetlabs.com>wrote:
> On Feb 10, 8:59 am, mukulm <smilemukul2...@gmail.com> wrote:
> > Hi,
> >
> > I am using puppet open source & their are above 2000 nodes in my
> > network & getting SSL certificates issues with some of the remote
> > nodes & they are not in sync with puppet server where others are
in
> > sync with server & date of the nodes systems are in sync with ntp
> > server, also tried the following as,
> >
> >    1. Deleted the nodes certificate on the puppet server
> >    2. Deleted the SSL directory of the agent.
> >    3. Run puppet agent --test on that agent to generate a new
> > certificate request.
> >
> > but I have to regularly repeat the above process within a timeframe
> > (weekly or monthly) so is there any other way other than this so that
> > all of my agent should be in sync.
> >
> > Also how can I find the sync & unsync count    ?
> >
> > Thanks
> > mukulm
>
> To help clarify things, are all your nodes and masters in sync with
> the same NTP server? This is a common source of the issues you are
> describing.
>
> If you are still running into issues can you post (pastbin) the errors
> so
> others can help you troubleshoot?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 13, 10:58 am, Mukul Malhotra <smilemukul2...@gmail.com>
wrote:
> Hi,
>
> I am getting the following error message as after removing the SSL
> certificates from the node & server,
>
> *err: Could not request certificate: Retrieved certificate does not match
> private key; please remove certificate from server and regenerate it with
> the current key*

It looks like the master is returning a signed certificate to the
client that doesn''t match its signing request.  Specifically, it
doesn''t match the client''s private key.  I can think of a
couple of
ways that might happen:

A)
  (i) Node A issues a certificate request to the master using certname
"a.mydomain.com", then
  (ii) the CA signs the request but node A''s Puppet SSL private key is
changed, before
  (iii) node A''s Puppet agent requests a catalog

B)
  (i) Node B1 issues a certificate request to the master using
certname "b.mydomain.com", then
  (ii) node B2 also issues a certificate request to the master using
certname "b.mydomain.com", then
  (iii) the CA signs either B1''s or B2''s certificate (but not
both,
because of the certname clash), then
  (iv) nodes B1 and B2 request catalogs (one of them should give an
error similar to the one you present)

Basically, the error message is pretty clear as these things go: the
signed certificate returned by the master to the agent does not
correspond to a certificate request signed by the agent''s (current)
private key.  The agent doesn''t need to track specific certificate
requests to know this; the private key is sufficient.

My best guess about your overall problem would be that either you are
triggering (A) by rebuilding client nodes without preserving their SSL
private keys, or else that you are triggering (B) by having multiple
nodes using the same certname (which by default is based on their
hostname).


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks for the valuable info but my scenario is option (A) but my
question is if once SSL certificates are signed for a node then how &
due to what reason
the SSL private key changes & what does preserving SSL private keys
means  ?

Thanks
mukulm


On Feb 14, 7:14 pm, jcbollinger <John.Bollin...@stJude.org>
wrote:
> On Feb 13, 10:58 am, Mukul Malhotra <smilemukul2...@gmail.com> wrote:
>
> > Hi,
>
> > I am getting the following error message as after removing the SSL
> > certificates from the node & server,
>
> > *err: Could not request certificate: Retrieved certificate does not
match
> > private key; please remove certificate from server and regenerate it
with
> > the current key*
>
> It looks like the master is returning a signed certificate to the
> client that doesn''t match its signing request.  Specifically, it
> doesn''t match the client''s private key.  I can think of a
couple of
> ways that might happen:
>
> A)
>   (i) Node A issues a certificate request to the master using certname
> "a.mydomain.com", then
>   (ii) the CA signs the request but node A''s Puppet SSL private
key is
> changed, before
>   (iii) node A''s Puppet agent requests a catalog
>
> B)
>   (i) Node B1 issues a certificate request to the master using
> certname "b.mydomain.com", then
>   (ii) node B2 also issues a certificate request to the master using
> certname "b.mydomain.com", then
>   (iii) the CA signs either B1''s or B2''s certificate (but
not both,
> because of the certname clash), then
>   (iv) nodes B1 and B2 request catalogs (one of them should give an
> error similar to the one you present)
>
> Basically, the error message is pretty clear as these things go: the
> signed certificate returned by the master to the agent does not
> correspond to a certificate request signed by the agent''s
(current)
> private key.  The agent doesn''t need to track specific certificate
> requests to know this; the private key is sufficient.
>
> My best guess about your overall problem would be that either you are
> triggering (A) by rebuilding client nodes without preserving their SSL
> private keys, or else that you are triggering (B) by having multiple
> nodes using the same certname (which by default is based on their
> hostname).
>
> John
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 14, 8:49 am, mukulm <smilemukul2...@gmail.com>
wrote:
> Thanks for the valuable info but my scenario is option (A) but my
> question is if once SSL certificates are signed for a node then how &
> due to what reason
> the SSL private key changes & what does preserving SSL private keys
> means  ?

If you can be certain that your nodes all have unique names and if you
are not using the "certname" parameter in your puppet.conf or on the
command line, then you can rule out scenario B.  If you have to ask
that question, however, then you cannot be certain that your problem
matches scenario A.  Perhaps it arises instead by some path that I
haven''t thought of.

Anyway, it sounds like some more information will help you.  First
off, read
http://projects.puppetlabs.com/projects/1/wiki/Certificates_And_Security.
It''s a bit dated (for example, the program puppetca is now rolled into
the main "puppet" driver as "puppet cert"), but for the most
part it''s
still accurate.

The bottom line, however, is that nodes'' SSL private keys are stored
in files on the node''s file system.  They will be lost if the file is
deleted (such as if the node is wiped and rebuilt), and they can be
changed or restored (as can their signed certificates) by changing or
restoring the file in which they are recorded.

If you frequently rebuild nodes without changing their names, then
rather than preserving their keys, it may be easier to revoke nodes''
certificates and delete them from the master whenever you rebuild
those nodes.  Then they should request new certificates, and the
master should be willing to sign them.  To achieve that, you would run
a command like this _on the master_: "puppet cert revoke <certname>;
puppet cert clean <certname>" to allow the node to request a new
certificate.

Also, BE CERTAIN that you do not install a common private key on your
nodes, as might happen if you rebuild nodes by applying a pre-made
filesystem image that has a Puppet private key in it.  Every node
should have a distinct private key (which it will generate for itself
at first need).


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I had build a new client & sync the client with ntp server for the
first time but i am getting a certificate error as "err: Could not
request certificate: Retrieved certificate does not match private key;
please remove certificate from server and regenerate it with the
current key"   ?

what is the reason for the above error ?

Thanks



On Feb 14, 7:14 pm, jcbollinger <John.Bollin...@stJude.org>
wrote:
> On Feb 13, 10:58 am, Mukul Malhotra <smilemukul2...@gmail.com> wrote:
>
> > Hi,
>
> > I am getting the following error message as after removing the SSL
> > certificates from the node & server,
>
> > *err: Could not request certificate: Retrieved certificate does not
match
> > private key; please remove certificate from server and regenerate it
with
> > the current key*
>
> It looks like the master is returning a signed certificate to the
> client that doesn''t match its signing request.  Specifically, it
> doesn''t match the client''s private key.  I can think of a
couple of
> ways that might happen:
>
> A)
>   (i) Node A issues a certificate request to the master using certname
> "a.mydomain.com", then
>   (ii) the CA signs the request but node A''s Puppet SSL private
key is
> changed, before
>   (iii) node A''s Puppet agent requests a catalog
>
> B)
>   (i) Node B1 issues a certificate request to the master using
> certname "b.mydomain.com", then
>   (ii) node B2 also issues a certificate request to the master using
> certname "b.mydomain.com", then
>   (iii) the CA signs either B1''s or B2''s certificate (but
not both,
> because of the certname clash), then
>   (iv) nodes B1 and B2 request catalogs (one of them should give an
> error similar to the one you present)
>
> Basically, the error message is pretty clear as these things go: the
> signed certificate returned by the master to the agent does not
> correspond to a certificate request signed by the agent''s
(current)
> private key.  The agent doesn''t need to track specific certificate
> requests to know this; the private key is sufficient.
>
> My best guess about your overall problem would be that either you are
> triggering (A) by rebuilding client nodes without preserving their SSL
> private keys, or else that you are triggering (B) by having multiple
> nodes using the same certname (which by default is based on their
> hostname).
>
> John
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Feb 29, 10:21 am, mukulm <smilemukul2...@gmail.com>
wrote:
> I had build a new client & sync the client with ntp server for the
> first time but i am getting a certificate error as "err: Could not
> request certificate: Retrieved certificate does not match private key;
> please remove certificate from server and regenerate it with the
> current key"   ?
>
> what is the reason for the above error ?

The same as I told you before: the master is returning a signed
certificate to the client that doesn''t match the client''s
signing
request / private key.  The possible reasons are the same as before,
too.

You should verify that the client''s unqualified host name is unique
among your Puppet clients, else it may be contending with another
client for the same SSL certname.  If the host name is unique then
probably you are re-using the name that a previous Puppet client used
(perhaps the same hardware, but definitely with a different private
key).  In that case, you should do exactly as the message tells you to
do.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.