Gluster users - Dec 2018 - 'No data available' when using disk encryption on volume

Hello,

New Here. I am experimenting with Disk Encryption [1] and getting a 'No
data available when trying to access data.

I followed these steps:

* Create a new volume:

```
# gluster volume create str01-cl-vol disperse 3
str0{1..6}:/data/glusterfs/brick1/str01-cl-vol
```

* Enable encryption on it:

```
# gluster volume set str01-cl-vol encryption on
```

* Disable performance indicators:

```
# for OPT in quick-read write-behind open-behind; do gluster volume set
str01-cl-vol performance.${OPT} off; done
volume set: success
volume set: success
volume set: success
```

* Set the location of the master encryption key:

```
# gluster volume set str01-cl-vol encryption.master-key
/data/glusterfs/private/str01-cl-vol/encryption.master-key
```

* Generate the master encryption key:

```
#  openssl rand -hex 32 >
/data/glusterfs/private/str01-cl-vol/encryption.master-key
```

* Set the key size to 512 bits (default is 256):

```
# gluster volume set str01-cl-vol encryption.data-key-size 512
```

* Start the volume:

```
# gluster volume start str01-cl-vol
```

* Verify:

```
# gluster volume info str01-cl-vol

Volume Name: str01-cl-vol
Type: Distributed-Disperse
Volume ID: 85ed34b2-9057-4f27-9594-168cb8343e25
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: str01:/data/glusterfs/brick1/str01-cl-vol
Brick2: str02:/data/glusterfs/brick1/str01-cl-vol
Brick3: str03:/data/glusterfs/brick1/str01-cl-vol
Brick4: str04:/data/glusterfs/brick1/str01-cl-vol
Brick5: str05:/data/glusterfs/brick1/str01-cl-vol
Brick6: str06:/data/glusterfs/brick1/str01-cl-vol
Options Reconfigured:
encryption.data-key-size: 512
encryption.master-key:
/data/glusterfs/private/str01-cl-vol/encryption.master-key
performance.open-behind: off
performance.write-behind: off
performance.quick-read: off
features.encryption: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
```

* Mount the volume (on localhost):

```
mount -t glusterfs -o
xlator-option=str01-cl-vol.master-key=/data/glusterfs/private/str01-cl.vol-encryption.master-key,log-file=gluster.log
localhost:/str01-cl-vol /mnt
```

Log says:

https://pastebin.com/fxQaGjF1

* Listing the files:

```
root at str01:/mnt# ls -la
ls: reading directory '.': Input/output error
total 4
-rw-r--r-- 1 root root    0 Dec 10 13:28 test123
-rw-r--r-- 1 root root    0 Dec 10 13:33 test543
-rw-r--r-- 1 root root    0 Dec 10 13:27 test.txt
drwxr-xr-x 3 root root 4096 Dec 10 13:27 .trashcan
```

* Creating a file:

```
root at str01:/mnt# echo test > test987
bash: test987: Invalid argument
```

* Readding a file fails (and is  very slow):

```
root at str01:/mnt# time cat test987
cat: test987: No data available

real    0m2.018s
user    0m0.000s
sys     0m0.000s
```

Glusterfs version is (installed from Debian stretch repos):

```
root at str01:/mnt# gluster --version
glusterfs 3.8.8 built on Jan 11 2017 14:07:11
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU
General Public License.
```

Can you see something wrong in my workflow?

References:
-----------
[1]
https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.5/Disk%20Encryption.md
[2]
https://keithseahus.github.io/sphinx/build/html/technology/glusterfs/new_features/3.5/disk_encryption.html

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20181212/bbdb0897/attachment.sig>