Gluster users - Aug 2014 - Transparent encryption in GlusterFS: Implications on manageability

Hello everyone,

In this thread we'll consider in details all issues related to
the management aspects of the feature "Transparent encryption in
GlusterFS".

We assume that the reader is familiar with the basic concepts of the
feature "Transparent encryption in GlusterFS". See, for example,
slides 1-7 of the design document [2].

Comments, questions, any your experience are welcome.

Thanks,
Edward.


             Transparent encryption in GlusterFS:
                Implications on manageability

 

              I. Managing master volume keys


Any user of the feature "Transparent encryption in GlusterFS" will
need to manage master volume keys (see man pages [1], section 6.2).


            I.1 Generating the master volume key


Master volume key should be generated by user on the trusted machine.
Recommendations on master key generation provided at section 6.2
of the manpages [1]. Generating of master volume key is in user's
competence.


           I.2 Location of the master volume key
                  when mounting a volume


At mount time the crypt translator searches for a master volume key
on the client machine at the location specified by the respective
translator option. If there is no any key at the specified location,
or the key at specified location is in improper format, then mount
will fail. Otherwise, the crypt translator loads the key to its
private memory data structures.

Location of the master volume key can be specified at volume creation
time (see option "master-key", section 6.7 of the man pages [1]).
However, this option can be overridden by user at mount time to specify
another location, see section 7 of manpages [1], steps 6, 7, 8.


             I.3 Retention of master volume keys
                    between mount sessions


After successful mount the file with master volume key can be removed
from the client machine: the crypt translator don't need it for _this_
mount session anymore: it will use the key installed in the private
memory data structures of the crypt translator. IMPORTANT: next mount
session will require the master volume key again, so if you remove the
file with master volume key after the mount command, make sure you
have stored it in the safe place for the next mount session.

Master volume key is a private secret key, so it is highly important
to provide its proper storage (on trusted machines, trusted media,
trusted network, etc). Otherwise it can be easily compromised.

Keeping the master volume key between mount sessions is in user's
competence. If user has more than one encrypted volume, he should
maintain the mapping volume <-> master-volume-key. Maintenance of
such mapping is in user's competence.


          I.4 Don't use the same master volume key
               for different Gluster volumes!


Different volumes should be encrypted with different master volume
keys. Using the same master volume key for different volumes is
highly not recommended because of security reasons.


   I.5 Using different master volume keys for the same volume


It is possible to mount a volume with a different master volume key
(see FAQ #4, section 11 of manpages [1]). In such mount session
content of files created with different master volume key will be
inaccessible. In particular, the crypt translator will refuse to open
files encrypted with different master volume key. However, it will be
possible to create new files on this volume and access them in the
mount sessions with respective master volume key.

In this case instead of simple mapping volume <-> master-volume-key
user will need to maintain mapping file <-> master-volume-key, which
is more complicated. Maintenance of this mapping is in user's
competence.


            I.6 Compromised master volume key


If your master volume key has been compromised because of some
reasons, or you suspect, that it has been compromised, then the whole
volume should be re-encrypted with the newly generated master volume
key. The common way is to create a new empty encrypted GlusterFS
volume, mount it with the new master volume key (see the manpages [1],
section 7), copy the old volume, mounted with compromised key to the
new one, mounted with the new key (for example, using cp(1) command.
After successful copy overwrite the content of all regular files of
your old volume with zeros. After this delete the old volume as usual.
Don't use the compromised master volume key anymore.


               II. Check graph of translators
             on your client machine after mount!


During mount your client machine receives configuration info from
the non-trusted server. In particular, this info contains the graph of
translators, which can be subjected to tampering, so that encryption
won't be invoked for your volume at all. So it is highly important to
verify this graph. After successful mount make sure that the graph of
translators contains the crypt translator with proper options (see
FAQ#1, section 11 of the manpages [1]).


         III. Changing the encryption status of the volume


Encryption status (enable/disable encryption) can be changed only
for empty volumes (with no files or directories). Changing encryption
status for not empty volumes is unacceptable: it will lead to IO
errors, data corruption and security problems. We highly recommend
to decide once and forever at volume creation time, whether your
volume has to be encrypted.

[1]
http://www.gluster.org/community/documentation/index.php/Features/disk-encryption
[2]
http://www.gluster.org/community/documentation/images/e/e2/GlusterFS_transparent_encryption.pdf

>             I.1 Generating the master volume key
>
>
> Master volume key should be generated by user on the trusted machine.
> Recommendations on master key generation provided at section 6.2 of
> the manpages [1]. Generating of master volume key is in user's
> competence.
That was fine for an initial implementation, but it's still the single
largest obstacle to adoption of this feature.  Looking forward, we need
to provide full CLI support for generating keys in the necessary format,
specifying their location, etc.
>            I.2 Location of the master volume key when mounting a
>            volume
>
>
> At mount time the crypt translator searches for a master volume key on
> the client machine at the location specified by the respective
> translator option. If there is no any key at the specified location,
> or the key at specified location is in improper format, then mount
> will fail. Otherwise, the crypt translator loads the key to its
> private memory data structures.
>
> Location of the master volume key can be specified at volume creation
> time (see option "master-key", section 6.7 of the man pages [1]).
> However, this option can be overridden by user at mount time to
> specify another location, see section 7 of manpages [1], steps 6, 7,
> 8.
Again, we need to improve on this.  We should support this as a volume
or mount option in its own right, not rely on the generic
--xlator-option mechanism.  Adding options to mount.glusterfs isn't
hard.  Alternatively, we could make this look like a volume option
settable once through the CLI, even though the path is stored locally on
the client.  Or we could provide a separate special-purpose
command/script, which again only needs to be run once.  It would even be
acceptable to treat the path to the key file (not its contents!) as a
true volume option, stored on the servers.  Any of these would be better
than requiring the user to understand our volfile format and
construction so that they can add the necessary option by hand.
>                II. Check graph of translators on your client machine
>                after mount!
>
>
> During mount your client machine receives configuration info from the
> non-trusted server. In particular, this info contains the graph of
> translators, which can be subjected to tampering, so that encryption
> won't be invoked for your volume at all. So it is highly important to
> verify this graph. After successful mount make sure that the graph of
> translators contains the crypt translator with proper options (see
> FAQ#1, section 11 of the manpages [1]).
It is important to verify the graph, but not by poking through log files
and not without more information about what to look for.  So we got a
volfile that includes the crypt translator, with some options.  The
*code* should ensure that the master-key option has the value from the
command line or local config, and not some other.  If we have to add
special support for this in otherwise-generic graph initialization code,
that's fine.