Stefan Kania
2024-Jan-26 09:44 UTC
[Gluster-users] Gluster communication via TLS client problem
Hi to all, The system is running Debian 12 with Gluster 10. All systems are using the same versions. I try to encrypt the communication between the peers and the clients via TLS. The encryption between the peers works, but when I try to mount the volume on the client I always get an error. What have I done? 1. all hosts and clients can resolve the name of all systems involved. 2. the volume is running and all hosts and clients can mount the volume, when TLS is not activated. To activate TLS I did in /usr/lib/ssl on all participating systems with openssl genrsa -out glusterfs.key 2048 openssl req -new -x509 -key glusterfs.key -subj "/CN=c01.gluster" -out glusterfs.pem Keys and certificates created (CN customised) Then combine all certificates into one and copy them to /usr/lib/ssl/ as glusterfs.ca to all hosts. Create the file /var/lib/glusterd/secure-access on the gluster peers. Gluster volume stopped and glusterd restarted. Then set the following parameters: gluster volume set gv1 auth.ssl-allow '*' gluster volume set gv1 client.ssl on gluster volume set gv1 server.ssl on When mounting the volume on the peers, I get the following messages: ------------------- _64-linux-gnu/libglusterfs.so.0(runner_log+0x100) [0x7ffa11782640] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=gv1 --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.42:49147 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.43:49147 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.41:49151 ------------------- Looks good to me Now trying to mount on the client with: --------------- mount -t glusterfs c01.gluster:/gv1 /mnt --------------- Then I get the following messages: On the gluster node in /var/log/gluster/glusterd ------ [2024-01-26 09:27:34.987837 +0000] I [socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.51:49151 [2024-01-26 09:27:34.991908 +0000] E [socket.c:224:ssl_dump_error_stack] 0-socket.management: error:0A00010B:SSL routines::wrong version number ------ On the client in /var/log/gluster/mnt.log ------- [2024-01-26 09:30:06.673990 +0000] I [MSGID: 100030] [glusterfsd.c:2767:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=10.5}, {cmdlinestr=/usr/sbin/glusterfs --process-name fuse --volfile-server=c01.gluster --volfile-id=/gv1 /mnt}] [2024-01-26 09:30:06.677184 +0000] I [glusterfsd.c:2447:daemonize] 0-glusterfs: Pid of current running process is 931 [2024-01-26 09:30:06.685814 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] [2024-01-26 09:30:06.686116 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] [2024-01-26 09:30:06.690443 +0000] I [glusterfsd-mgmt.c:2681:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: c01.gluster [2024-01-26 09:30:06.690512 +0000] I [glusterfsd-mgmt.c:2720:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers [2024-01-26 09:30:06.691618 +0000] W [glusterfsd.c:1458:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfa35) [0x7f83ace13a35] -->/usr/sbin/glusterfs(+0x14769) [0x55650549b769] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x556505492447] ) 0-: received signum (1), shutting down [2024-01-26 09:30:06.691699 +0000] I [fuse-bridge.c:7065:fini] 0-fuse: Unmounting '/mnt'. [2024-01-26 09:30:06.694246 +0000] I [fuse-bridge.c:7069:fini] 0-fuse: Closing fuse connection to '/mnt'. [2024-01-26 09:30:06.694431 +0000] W [glusterfsd.c:1458:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(+0x89044) [0x7f83acc98044] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x556505499e05] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x556505492447] ) 0-: received signum (15), shutting down ------- Testing with openssl on the client show: root at cluster-client:~# openssl s_client -CAfile /usr/lib/ssl/glusterfs.ca -connect c01.gluster:24007 CONNECTED(00000003) depth=0 CN = c01.gluster verify return:1 --- Certificate chain 0 s:CN = c01.gluster i:CN = c01.gluster a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Jan 26 08:27:12 2024 GMT; NotAfter: Feb 25 08:27:12 2024 GMT --- Server certificate -----BEGIN CERTIFICATE----- MIIDDTCCAfWgAwIBAgIULCwcIV9jWFzeZoeO1Xs5TJ9J5rkwDQYJKoZIhvcNAQEL BQAwFjEUMBIGA1UEAwwLYzAxLmdsdXN0ZXIwHhcNMjQwMTI2MDgyNzEyWhcNMjQw MjI1MDgyNzEyWjAWMRQwEgYDVQQDDAtjMDEuZ2x1c3RlcjCCASIwDQYJKoZIhvcN AQEBBQADggEPADCCAQoCggEBANPQ+fSk2V+hAjSOViZJxDWEgkjO1g8r3JH47QmI D8mhEAVoeUhzDdbDV2gWw26pgU1Z22cCQr72rnZaK9vV1xzvGVjdzbKwQU8NhqhR XWGJVlHdc5LxcOXfU7FpY6XMDzDLvRuNTMzsc685vJ8hjMxMAZJSLMAXNmLPMPnW NuaudBE+1f7oc9sdGWhUqmPcWXT6xUeEUEJCDbOrccH8nhUwBMbQFiU7S4pV3smB bbYNHFtw7Liz9B/vMoX1HckUKAsWcaWqPlWYr1rFHHPneyuG2evVcfRDhGsA1Fmo v7kamrGtXgEAdgXC6HdENFBJzdSSb77A89d8OSHOYNyEV5UCAwEAAaNTMFEwHQYD VR0OBBYEFCFjInacsKnR6TuPf+BI30b8qWPtMB8GA1UdIwQYMBaAFCFjInacsKnR 6TuPf+BI30b8qWPtMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEB AKBZNCRxKO5rv4yezGZRa/SDdpEc/vrGD5jKbHxQjBP+0YX/hToOGt04oh48iNFT A2vqUVby4JXml9FjPCNktHlRk/NYXIlQiTm//TBeG2D+HrAQRyLR6TXF62/4H3Pb Yktzr+vNk/znd5AKv3g8kMMpAB0UGxjZ9CtMDTuAYrQPtFCgCy1rf6fvP3cKZwaK kk/QjJyc9u6zTvL0ptOHdOdQbhrHjZHiQ1D6QvJu6LouMsY3gGlVXfh0rQHUzSvT 7MmDRb/l4jTs2sn/nexh9bpHUv/m3vzDWBbrWcwGzenKXR+lg1hvAZAP3Ds33S/+ W7sfZVptCwBXbYK0bSh+KiU-----END CERTIFICATE----- subject=CN = c01.gluster issuer=CN = c01.gluster --- No client certificate CA names sent Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224 Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512 Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: ECDH, prime256v1, 256 bits --- SSL handshake has read 1534 bytes and written 777 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: A9CA3DA57FDA9BF9D9EFBBD0E5CE5D8F7A5DE091C10E54310D52A23DCB7DA95B Session-ID-ctx: Resumption PSK: C7BA79D9FB045352371121AC97F891FBD4C2578AA48A7CD57747A941C6864CCE5163D5AF94BE01D75233148BD75E755E PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 7200 (seconds) TLS session ticket: 0000 - 6e fd 36 f6 0f 16 dc d0-f1 9f 02 4b 32 20 5e 4b n.6........K2 ^K 0010 - e4 98 1e 6f 4c 8d b3 71-c8 12 40 ed 75 3f f7 ce ...oL..q.. at .u?.. Start Time: 1706261953 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Max Early Data: 0 --- read R BLOCK --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: 42BA7A7BFC9B64C030DB99E2D12B060052F53B5A771826199868A6AE913ED245 Session-ID-ctx: Resumption PSK: 3E66E04230CDFDF569A87764318B3C0C67FEA910742784CBC31E0221C44DB4EB91C2EBCB471AEB31FFFD5AB452C899F3 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 7200 (seconds) TLS session ticket: 0000 - 79 2a c8 0c 4c c7 2b f1-2d 3c 01 cf dd b3 e0 68 y*..L.+.-<.....h 0010 - 7c 19 e7 e3 96 d9 5d 77-19 a3 e1 a8 9e 6c 3a 37 |.....]w.....l:7 Start Time: 1706261953 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Max Early Data: 0 --- read R BLOCK 40D7F609527F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:303: Any help? Thank's Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3477 bytes Desc: Kryptografische S/MIME-Signatur URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240126/e7c34743/attachment.p7s>
Aravinda
2024-Jan-26 16:01 UTC
[Gluster-users] Gluster communication via TLS client problem
Hi Stefan, Does the combined?glusterfs.ca includes client nodes pem? Also this file need to be placed in Client node as well. -- Aravinda Kadalu Technologies ---- On Fri, 26 Jan 2024 15:14:39 +0530 Stefan Kania <stefan at kania-online.de> wrote --- Hi to all, The system is running Debian 12 with Gluster 10. All systems are using the same versions. I try to encrypt the communication between the peers and the clients via TLS. The encryption between the peers works, but when I try to mount the volume on the client I always get an error. What have I done? 1. all hosts and clients can resolve the name of all systems involved. 2. the volume is running and all hosts and clients can mount the volume, when TLS is not activated. To activate TLS I did in /usr/lib/ssl on all participating systems with openssl genrsa -out glusterfs.key 2048 openssl req -new -x509 -key glusterfs.key -subj "/CN=c01.gluster" -out glusterfs.pem Keys and certificates created (CN customised) Then combine all certificates into one and copy them to /usr/lib/ssl/ as glusterfs.ca to all hosts. Create the file /var/lib/glusterd/secure-access on the gluster peers. Gluster volume stopped and glusterd restarted. Then set the following parameters: gluster volume set gv1 auth.ssl-allow '*' gluster volume set gv1 client.ssl on gluster volume set gv1 server.ssl on When mounting the volume on the peers, I get the following messages: ------------------- _64-linux-gnu/libglusterfs.so.0(runner_log+0x100) [0x7ffa11782640] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=gv1 --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.42:49147 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.43:49147 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.41:49151 ------------------- Looks good to me Now trying to mount on the client with: --------------- mount -t glusterfs c01.gluster:/gv1 /mnt --------------- Then I get the following messages: On the gluster node in /var/log/gluster/glusterd ------ [2024-01-26 09:27:34.987837 +0000] I [socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.51:49151 [2024-01-26 09:27:34.991908 +0000] E [socket.c:224:ssl_dump_error_stack] 0-socket.management: error:0A00010B:SSL routines::wrong version number ------ On the client in /var/log/gluster/mnt.log ------- [2024-01-26 09:30:06.673990 +0000] I [MSGID: 100030] [glusterfsd.c:2767:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=10.5}, {cmdlinestr=/usr/sbin/glusterfs --process-name fuse --volfile-server=c01.gluster --volfile-id=/gv1 /mnt}] [2024-01-26 09:30:06.677184 +0000] I [glusterfsd.c:2447:daemonize] 0-glusterfs: Pid of current running process is 931 [2024-01-26 09:30:06.685814 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] [2024-01-26 09:30:06.686116 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] [2024-01-26 09:30:06.690443 +0000] I [glusterfsd-mgmt.c:2681:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: c01.gluster [2024-01-26 09:30:06.690512 +0000] I [glusterfsd-mgmt.c:2720:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers [2024-01-26 09:30:06.691618 +0000] W [glusterfsd.c:1458:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfa35) [0x7f83ace13a35] -->/usr/sbin/glusterfs(+0x14769) [0x55650549b769] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x556505492447] ) 0-: received signum (1), shutting down [2024-01-26 09:30:06.691699 +0000] I [fuse-bridge.c:7065:fini] 0-fuse: Unmounting '/mnt'. [2024-01-26 09:30:06.694246 +0000] I [fuse-bridge.c:7069:fini] 0-fuse: Closing fuse connection to '/mnt'. [2024-01-26 09:30:06.694431 +0000] W [glusterfsd.c:1458:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(+0x89044) [0x7f83acc98044] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x556505499e05] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x556505492447] ) 0-: received signum (15), shutting down ------- Testing with openssl on the client show: root at cluster-client:~# openssl s_client -CAfile /usr/lib/ssl/glusterfs.ca -connect c01.gluster:24007 CONNECTED(00000003) depth=0 CN = c01.gluster verify return:1 --- Certificate chain 0 s:CN = c01.gluster i:CN = c01.gluster a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Jan 26 08:27:12 2024 GMT; NotAfter: Feb 25 08:27:12 2024 GMT --- Server certificate -----BEGIN CERTIFICATE----- MIIDDTCCAfWgAwIBAgIULCwcIV9jWFzeZoeO1Xs5TJ9J5rkwDQYJKoZIhvcNAQEL BQAwFjEUMBIGA1UEAwwLYzAxLmdsdXN0ZXIwHhcNMjQwMTI2MDgyNzEyWhcNMjQw MjI1MDgyNzEyWjAWMRQwEgYDVQQDDAtjMDEuZ2x1c3RlcjCCASIwDQYJKoZIhvcN AQEBBQADggEPADCCAQoCggEBANPQ+fSk2V+hAjSOViZJxDWEgkjO1g8r3JH47QmI D8mhEAVoeUhzDdbDV2gWw26pgU1Z22cCQr72rnZaK9vV1xzvGVjdzbKwQU8NhqhR XWGJVlHdc5LxcOXfU7FpY6XMDzDLvRuNTMzsc685vJ8hjMxMAZJSLMAXNmLPMPnW NuaudBE+1f7oc9sdGWhUqmPcWXT6xUeEUEJCDbOrccH8nhUwBMbQFiU7S4pV3smB bbYNHFtw7Liz9B/vMoX1HckUKAsWcaWqPlWYr1rFHHPneyuG2evVcfRDhGsA1Fmo v7kamrGtXgEAdgXC6HdENFBJzdSSb77A89d8OSHOYNyEV5UCAwEAAaNTMFEwHQYD VR0OBBYEFCFjInacsKnR6TuPf+BI30b8qWPtMB8GA1UdIwQYMBaAFCFjInacsKnR 6TuPf+BI30b8qWPtMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEB AKBZNCRxKO5rv4yezGZRa/SDdpEc/vrGD5jKbHxQjBP+0YX/hToOGt04oh48iNFT A2vqUVby4JXml9FjPCNktHlRk/NYXIlQiTm//TBeG2D+HrAQRyLR6TXF62/4H3Pb Yktzr+vNk/znd5AKv3g8kMMpAB0UGxjZ9CtMDTuAYrQPtFCgCy1rf6fvP3cKZwaK kk/QjJyc9u6zTvL0ptOHdOdQbhrHjZHiQ1D6QvJu6LouMsY3gGlVXfh0rQHUzSvT 7MmDRb/l4jTs2sn/nexh9bpHUv/m3vzDWBbrWcwGzenKXR+lg1hvAZAP3Ds33S/+ W7sfZVptCwBXbYK0bSh+KiU= -----END CERTIFICATE----- subject=CN = c01.gluster issuer=CN = c01.gluster --- No client certificate CA names sent Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224 Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512 Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: ECDH, prime256v1, 256 bits --- SSL handshake has read 1534 bytes and written 777 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: A9CA3DA57FDA9BF9D9EFBBD0E5CE5D8F7A5DE091C10E54310D52A23DCB7DA95B Session-ID-ctx: Resumption PSK: C7BA79D9FB045352371121AC97F891FBD4C2578AA48A7CD57747A941C6864CCE5163D5AF94BE01D75233148BD75E755E PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 7200 (seconds) TLS session ticket: 0000 - 6e fd 36 f6 0f 16 dc d0-f1 9f 02 4b 32 20 5e 4b n.6........K2 ^K 0010 - e4 98 1e 6f 4c 8d b3 71-c8 12 40 ed 75 3f f7 ce ...oL..q.. at .u?.. Start Time: 1706261953 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Max Early Data: 0 --- read R BLOCK --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: 42BA7A7BFC9B64C030DB99E2D12B060052F53B5A771826199868A6AE913ED245 Session-ID-ctx: Resumption PSK: 3E66E04230CDFDF569A87764318B3C0C67FEA910742784CBC31E0221C44DB4EB91C2EBCB471AEB31FFFD5AB452C899F3 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 7200 (seconds) TLS session ticket: 0000 - 79 2a c8 0c 4c c7 2b f1-2d 3c 01 cf dd b3 e0 68 y*..L.+.-<.....h 0010 - 7c 19 e7 e3 96 d9 5d 77-19 a3 e1 a8 9e 6c 3a 37 |.....]w.....l:7 Start Time: 1706261953 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Max Early Data: 0 --- read R BLOCK 40D7F609527F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:303: Any help? Thank's Stefan ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list mailto:Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240126/da1da092/attachment.html>