Puppet users - Mar 2010 - Best practises for managing secret keys with puppet?

G''day.

One of the current problems we face with our puppet network, and about which
I would like to solicit opinions, is the distribution of secret keys for
(mostly) SSL secured services.

The most pressing example of this is that for reasons of availability want to
process SSL on multiple machines for a single certificate.  The same issue
applies to a wide range of internal and public services, though, where we need
to distribute key material.[1]


The prospect of putting the secret key into our revision control system has
... well, little appeal is probably being fair: we could certainly do it, but
it suddenly means that a whole bunch of extra data has to be treated as high
security rather than low security.[2]


On the other hand, it would be good if a trusted agent like puppet can be
configured to automatically bring up nodes that run secure services from
scratch with no human intervention involved.  That helps with availability,
as well as security issues like key rollover.


On the gripping hand, I mistrust human involvement: if I have to give *people*
(including me) access to the key material, rather than software, I have a
whole set of less trustworthy and reliable agents handling it.


So, on the whole my feeling is that an automatic "key distribution
service"
that was accessible to puppet but (mostly) not to people would be ideal.

I also feel that it would be ideal to keep that service distinct from our
standard puppet Subversion repository, to keep the need for security
boundaries small.


I am interested here in other views on my assessment of the problem, and in
pointers to practical tools used to solve the problem.

        Daniel

Oh, and at the moment I am considering the puppet client/server protocol
trustworthy enough to enforce access control to the key material. :)


Footnotes: 
[1]  Generally, key material where we don''t require a passphrase to
unlock
     before use, because we have made the availability/security decision based
     on our business needs.

[2]  ...well, technically "medium grade" security, since at present
reading
     our puppet manifests is worth little to an attacker, but writing to them
     is valuable.  So, we protect writes heavily but don''t worry too
much
     about, for example, a backup of the VCS repository they live in.

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Fri, Mar 26, 2010 at 3:22 AM, Daniel Pittman <daniel@rimspace.net>
wrote:
> G''day.
>
> One of the current problems we face with our puppet network, and about
> which
> I would like to solicit opinions, is the distribution of secret keys for
> (mostly) SSL secured services.
>
> The most pressing example of this is that for reasons of availability want
> to
> process SSL on multiple machines for a single certificate.  The same issue
> applies to a wide range of internal and public services, though, where we
> need
> to distribute key material.[1]
>
>
> The prospect of putting the secret key into our revision control system has
> ... well, little appeal is probably being fair: we could certainly do it,
> but
> it suddenly means that a whole bunch of extra data has to be treated as
> high
> security rather than low security.[2]
>
>
One way to handle this would be by keeping confidential information in a
seperate version control repository (not public), rather than in your main
one.   Puppet has a system of module paths so you could keep your
confidential info seperate from the content you would want to give to
everyone who would normally be working with Puppet, and check *both* of
these out on the Puppet server.     For development systems/testing, you
could just check out a copy of a different repo, with testing/stage
credentials in the modules instead.

You could also use a custom function to pull this information from other
sources for accessing a keystore server side, though I''d be curious to
what
those other services might be.

How is everyone else handling this?

--Michael

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Fri, Mar 26, 2010 at 12:22 AM, Daniel Pittman <daniel@rimspace.net>
wrote:
> G''day.
>
> One of the current problems we face with our puppet network, and about
which
> I would like to solicit opinions, is the distribution of secret keys for
> (mostly) SSL secured services.
>
> The most pressing example of this is that for reasons of availability want
to
> process SSL on multiple machines for a single certificate.  The same issue
> applies to a wide range of internal and public services, though, where we
need
> to distribute key material.[1]
>
>
> The prospect of putting the secret key into our revision control system has
> ... well, little appeal is probably being fair: we could certainly do it,
but
> it suddenly means that a whole bunch of extra data has to be treated as
high
> security rather than low security.[2]
What about keeping encrypted versions of the key in your revision
control system, and then writing a function for the puppet servers
that decrypts them using a key that you need to distribute out of band
to your puppet servers?

You''d still be sending the unencrypted data across, but this should be
a secured connection, and it will end up in your client catalogs, but
they should be protected just as much as the data on the client disk
would be.


>
>
> On the other hand, it would be good if a trusted agent like puppet can be
> configured to automatically bring up nodes that run secure services from
> scratch with no human intervention involved.  That helps with availability,
> as well as security issues like key rollover.
>
>
> On the gripping hand, I mistrust human involvement: if I have to give
*people*
> (including me) access to the key material, rather than software, I have a
> whole set of less trustworthy and reliable agents handling it.
>
>
> So, on the whole my feeling is that an automatic "key distribution
service"
> that was accessible to puppet but (mostly) not to people would be ideal.
>
> I also feel that it would be ideal to keep that service distinct from our
> standard puppet Subversion repository, to keep the need for security
> boundaries small.
>
>
> I am interested here in other views on my assessment of the problem, and in
> pointers to practical tools used to solve the problem.
>
>        Daniel
>
> Oh, and at the moment I am considering the puppet client/server protocol
> trustworthy enough to enforce access control to the key material. :)
>
>
> Footnotes:
> [1]  Generally, key material where we don''t require a passphrase
to unlock
>     before use, because we have made the availability/security decision
based
>     on our business needs.
>
> [2]  ...well, technically "medium grade" security, since at
present reading
>     our puppet manifests is worth little to an attacker, but writing to
them
>     is valuable.  So, we protect writes heavily but don''t worry
too much
>     about, for example, a backup of the VCS repository they live in.
>
> --
> ✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155
707
>               ♽ made with 100 percent post-consumer electrons
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>


-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Fri, 26 Mar 2010, Daniel Pittman wrote:
> The prospect of putting the secret key into our revision control
> system has ... well, little appeal is probably being fair: we could
> certainly do it, but it suddenly means that a whole bunch of extra
> data has to be treated as high security rather than low security.[2]
I configure puppet to print an error message that explains the
situation:

# util::manually_copied_file -- set permissions on a manually-copied
# file, and print an error message if the file is missing.
#
# usage:
#     util::manually_copied_file { "/dirname/filename":
#         message => "where to copy it from, or why it''s not in
puppet",
#         owner => root,
#         group => bin,
#         mode => 0400,
#     }
#
define util::manually_copied_file($message, $owner, $group, $mode)
{
    # If the file exists and has a size > 0, then do nothing.
    # Otherwise, print an error message and fail.
    exec { "util::manually_copied_file check $name":
        unless => "/bin/test -s $name",
        command => "/bin/cat <<''EOF''; /bin/false
Please copy ${name} manually - ${message}
EOF
",
        logoutput => true,
        before => File[$name],
        require => [],
    }

    # Set the ownership and permissions, but do not modify the content
    file { $name:
        ensure => file,
        replace => false,
        owner => $owner,
        group => $group,
        mode => $mode,
    }
}
> So, on the whole my feeling is that an automatic "key distribution
> service" that was accessible to puppet but (mostly) not to people
> would be ideal.
That would be nice.

--apb (Alan Barrett)

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Michael DeHaan wrote:
> One way to handle this would be by keeping confidential information in a 
> seperate version control repository (not public), rather than in your 
> main one.   Puppet has a system of module paths so you could keep your 
> confidential info seperate from the content you would want to give to 
> everyone who would normally be working with Puppet, and check *both* of 
> these out on the Puppet server.     For development systems/testing, you 
> could just check out a copy of a different repo, with testing/stage 
> credentials in the modules instead. 
> 
> You could also use a custom function to pull this information from other 
> sources for accessing a keystore server side, though I''d be
curious to
> what those other services might be.
> 
> How is everyone else handling this?
We have a filerserving module named "private" that is defined like

     [private]
         path    /config/private/%h
         allow   *

in fileserver.conf.  Each client gets its own subdirectory
"/config/private/client1", "/config/private/client2", and so
on.
In there we stuff things like SSH host keys, X.509 host certificates,
license keys, and other secret data, and let Puppet install them via

     file {
         "/etc/ssh/ssh_host_rsa_key":
             source => "puppet:///private/ssh_host_rsa_key", ...;
     }

We currently don''t have /config/private under version control, but
if we did it would definitely be in a separate repository from
our manifests.


	/Bellman

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Daniel Pittman wrote:
> G''day.

Hey Daniel, your puppet SSL keys can be used for other services as well. 
I successfully used them as authentication for Splunk''s SSL receiver 
when I was piloting the software. IDK if this helps you, but I feel like 
this tidbit may get overlooked sometimes. How do you currently manage 
your puppet keys?

-- 
Joe McDonagh
AIM: YoosingYoonickz
IRC: joe-mac on freenode
L''ennui est contre-révolutionnaire

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Joe McDonagh <joseph.e.mcdonagh@gmail.com> writes:
> Daniel Pittman wrote:
>
> Hey Daniel, your puppet SSL keys can be used for other services as well.
*nod*  Sadly, we need a whole bunch of different public SSL services,
including SSL services on host names and domains that we operate on the behalf
of clients.

It was, in fact, mostly services like that which I was considering when
I asked the initial question.  So, while reuse of the puppet keys would be
occasionally convenient for low value stuff, it doesn''t solve my bigger
problem.

[...]
> How do you currently manage your puppet keys?
Entirely manually, using the puppet CA, because it just works, and because we
don''t have any significant cross-over between the internal CA we use
and the
places that we run puppet.

        Daniel

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.