Aravinda
2024-Jan-31 08:22 UTC
[Gluster-users] Gluster communication via TLS client problem
Hi Stefan, I reproduced this in our lab and it is working without any issues. Lab setup: Debian 12 and Gluster version 10.5 Three servers and one client: c01.gluster, c02.gluster, c03.gluster and cluster-client.gluster I used RSA key length as 4096 instead of 2048 and used the below volume option gluster volume set gv1 ssl.cipher-list 'HIGH:!SSLv2'? I used Kadalu Binnacle (https://github.com/kadalu/binnacle) to setup container based three nodes cluster. The details and the test file are available in the below Github repository. https://github.com/aravindavk/gluster-tests?tab=readme-ov-file#gluster-tls-tests -- Aravinda Kadalu Technologies ---- On Mon, 29 Jan 2024 22:10:50 +0530 Stefan Kania <stefan at kania-online.de> wrote --- Hi Strahil, hi Aravinda Am 28.01.24 um 23:03 schrieb Strahil Nikolov:> You didn't specify correctly the IP in the SANS but I'm not sure if that's the root cause. > In the SANs section Specify all hosts + their IPs: IP.1=1.2.3.4IP.2=2.3.4.5DNS.1=c01.glusterDNS.2=c02.glusterThat's what I did now: I took the commands from the article you recommended and added all the alternative names and IPs into the certificate: ------------- openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=`hostname -f`" -addext "subjectAltName = IP:192.168.57.41,IP:192.168.57.42,IP:192.168.57.43,IP:192.168.57.51,DNS:c01.gluster,DNS:c02.gluster,DNS:c03.gluster,DNS:cluster-client.gluster" -out /etc/ssl/glusterfs.pem ------------- Stille getting on the server: ------------- [2024-01-29 16:32:08.877499 +0000] I [socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.51:49151 [2024-01-29 16:32:08.881842 +0000] E [socket.c:224:ssl_dump_error_stack] 0-socket.management: error:0A00010B:SSL routines::wrong version number ------------- And on the client: ------------- [2024-01-29 16:32:08.865731 +0000] I [MSGID: 100030] [glusterfsd.c:2767:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=10.5}, {cmdlinestr=/usr/sbin/glusterfs --process-name fuse --volfile-server=c02.gluster --volfile-id=/gv1 /mnt}] [2024-01-29 16:32:08.870129 +0000] I [glusterfsd.c:2447:daemonize] 0-glusterfs: Pid of current running process is 664 [2024-01-29 16:32:08.880528 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] [2024-01-29 16:32:08.880935 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] [2024-01-29 16:32:08.885755 +0000] I [glusterfsd-mgmt.c:2681:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: c02.gluster [2024-01-29 16:32:08.885879 +0000] I [glusterfsd-mgmt.c:2720:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers [2024-01-29 16:32:08.887116 +0000] W [glusterfsd.c:1458:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfa35) [0x7fd18d185a35] -->/usr/sbin/glusterfs(+0x14769) [0x55d4f8d5d769] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d4f8d54447] ) 0-: received signum (1), shutting down [2024-01-29 16:32:08.887209 +0000] I [fuse-bridge.c:7065:fini] 0-fuse: Unmounting '/mnt'. [2024-01-29 16:32:08.889719 +0000] I [fuse-bridge.c:7069:fini] 0-fuse: Closing fuse connection to '/mnt'. [2024-01-29 16:32:08.889909 +0000] W [glusterfsd.c:1458:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(+0x89044) [0x7fd18d00a044] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x55d4f8d5be05] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d4f8d54447] ) 0-: received signum (15), shutting down ------------- executing the connect command on the client: -------------- openssl s_client -showcerts -connect c02.gluster:24007 -------------- shows on the sever: -------------- [2024-01-29 16:37:08.747123 +0000] I [socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.51:58060 [2024-01-29 16:37:08.767715 +0000] E [socket.c:426:ssl_setup_connection_postfix] 0-socket.management: SSL connect error (client: 192.168.57.51:58060) (server: 192.168.57.42:24007) -------------- So still the same, no changes :-( Stefan -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240131/aa5b3414/attachment.html>
Stefan Kania
2024-Jan-31 11:19 UTC
[Gluster-users] Gluster communication via TLS client problem
Hello Aravinda, wow, that's a lot of work to help me find my problem. I use virtualbox and my VM having 3 IPs one is the nat-network to install packages. The next is an host-only-network (192.168.56.0) for the productiv network. The next is also a host-only-network (192.168.57.0) for the communication between the gluster-nodes abd the client. Bind9 is running and resolving both networks in both directions. I think I will redo it with just one interface. The thing is, I did it before the same way and it was running :-(. I will look at the git-repository Stefan Am 31.01.24 um 09:22 schrieb Aravinda:> Hi Stefan, > > I reproduced this in our lab and it is working without any issues. > > Lab setup: Debian 12 and Gluster version 10.5 > Three servers and one client: c01.gluster, c02.gluster, c03.gluster and > cluster-client.gluster > > I used RSA key length as 4096 instead of 2048 and used the below volume > option > > gluster volume set gv1 ssl.cipher-list 'HIGH:!SSLv2' > > I used Kadalu Binnacle (https://github.com/kadalu/binnacle > <https://github.com/kadalu/binnacle>) to setup container based three > nodes cluster. The details and the test file are available in the below > Github repository. > > https://github.com/aravindavk/gluster-tests?tab=readme-ov-file#gluster-tls-tests <https://github.com/aravindavk/gluster-tests?tab=readme-ov-file#gluster-tls-tests> > > -- > Aravinda > Kadalu Technologies > > > > ---- On Mon, 29 Jan 2024 22:10:50 +0530 *Stefan Kania > <stefan at kania-online.de>* wrote --- > > Hi Strahil, hi Aravinda > > Am 28.01.24 um 23:03 schrieb Strahil Nikolov: > > You didn't specify correctly the IP in the SANS but I'm not sure > if that's the root cause. > > In the SANs section Specify all hosts + their IPs: > IP.1=1.2.3.4IP.2=2.3.4.5DNS.1=c01.glusterDNS.2=c02.gluster > > That's what I did now: > > I took the commands from the article you recommended and added all the > alternative names and IPs into the certificate: > ------------- > openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=`hostname > -f`" -addext "subjectAltName > IP:192.168.57.41,IP:192.168.57.42,IP:192.168.57.43,IP:192.168.57.51,DNS:c01.gluster,DNS:c02.gluster,DNS:c03.gluster,DNS:cluster-client.gluster" > -out /etc/ssl/glusterfs.pem > ------------- > Stille getting on the server: > ------------- > [2024-01-29 16:32:08.877499 +0000] I > [socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL > support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 > for peer 192.168.57.51:49151 > [2024-01-29 16:32:08.881842 +0000] E > [socket.c:224:ssl_dump_error_stack] > 0-socket.management: error:0A00010B:SSL routines::wrong version number > > ------------- > > And on the client: > ------------- > [2024-01-29 16:32:08.865731 +0000] I [MSGID: 100030] > [glusterfsd.c:2767:main] 0-/usr/sbin/glusterfs: Started running version > [{arg=/usr/sbin/glusterfs}, {version=10.5}, > {cmdlinestr=/usr/sbin/glusterfs --process-name fuse > --volfile-server=c02.gluster --volfile-id=/gv1 /mnt}] > [2024-01-29 16:32:08.870129 +0000] I [glusterfsd.c:2447:daemonize] > 0-glusterfs: Pid of current running process is 664 > [2024-01-29 16:32:08.880528 +0000] I [MSGID: 101190] > [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread > with index [{index=1}] > [2024-01-29 16:32:08.880935 +0000] I [MSGID: 101190] > [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread > with index [{index=0}] > [2024-01-29 16:32:08.885755 +0000] I > [glusterfsd-mgmt.c:2681:mgmt_rpc_notify] 0-glusterfsd-mgmt: > disconnected > from remote-host: c02.gluster > [2024-01-29 16:32:08.885879 +0000] I > [glusterfsd-mgmt.c:2720:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted > all volfile servers > [2024-01-29 16:32:08.887116 +0000] W > [glusterfsd.c:1458:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfa35) [0x7fd18d185a35] > -->/usr/sbin/glusterfs(+0x14769) [0x55d4f8d5d769] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d4f8d54447] ) 0-: > received signum (1), shutting down > [2024-01-29 16:32:08.887209 +0000] I [fuse-bridge.c:7065:fini] 0-fuse: > Unmounting '/mnt'. > [2024-01-29 16:32:08.889719 +0000] I [fuse-bridge.c:7069:fini] 0-fuse: > Closing fuse connection to '/mnt'. > [2024-01-29 16:32:08.889909 +0000] W > [glusterfsd.c:1458:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libc.so.6(+0x89044) [0x7fd18d00a044] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x55d4f8d5be05] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d4f8d54447] ) 0-: > received signum (15), shutting down > ------------- > > executing the connect command on the client: > -------------- > openssl s_client -showcerts -connect c02.gluster:24007 > -------------- > > shows on the sever: > -------------- > [2024-01-29 16:37:08.747123 +0000] I > [socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL > support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 > for peer 192.168.57.51:58060 > [2024-01-29 16:37:08.767715 +0000] E > [socket.c:426:ssl_setup_connection_postfix] 0-socket.management: SSL > connect error (client: 192.168.57.51:58060) (server: > 192.168.57.42:24007) > -------------- > > So still the same, no changes :-( > > Stefan > > >-- Stefan Kania Landweg 13 25693 St. Michaelisdonn Signieren jeder E-Mail hilft Spam zu reduzieren und sch?tzt Ihre Privatsph?re. Ein kostenfreies Zertifikat erhalten Sie unter https://www.dgn.de/dgncert/index.html Download der root-Zertifikate: https://www.dgn.de/dgncert/downloads.html Neuer GPG-Key der public key befindet sich im Anhang -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3477 bytes Desc: Kryptografische S/MIME-Signatur URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240131/d4be351c/attachment.p7s>
Stefan Kania
2024-Jan-31 16:31 UTC
[Gluster-users] Gluster communication via TLS client problem
Hi Aravinda, im not so into Docker :-( So I just looked at your commands and I saw that you did exacly the same I did. I even removed all TLS configuration and all certificates and then copied your commands (as far as it was possible) to create the certificates and compared it with my commands. Everything is exacly the same. But my setup is not working :-(. At this point I think it's not a Gluster problem but a problem of my Debian installation and configuration. So I will start from scratch and do it all again. One reason I don't want to use docker is, I need to install it on VMs in the future. If everything is working, I will put the hole setup into ansible. But first step first. And the first step is, geting TLS running. Stefan Am 31.01.24 um 09:22 schrieb Aravinda:> Hi Stefan, > > > > I reproduced this in our lab and it is working without any issues. > > > > Lab setup: Debian 12 and Gluster version 10.5 > > Three servers and one client: c01.gluster, c02.gluster, c03.gluster and cluster-client.gluster > > > > I used RSA key length as 4096 instead of 2048 and used the below volume option > > > > gluster volume set gv1 ssl.cipher-list 'HIGH:!SSLv2' > > > > I used Kadalu Binnacle (https://github.com/kadalu/binnacle) to setup container based three nodes cluster. The details and the test file are available in the below Github repository. > > > > https://github.com/aravindavk/gluster-tests?tab=readme-ov-file#gluster-tls-tests > > > > -- > Aravinda > > Kadalu Technologies > > > > > > > > > ---- On Mon, 29 Jan 2024 22:10:50 +0530 Stefan Kania <stefan at kania-online.de> wrote --- > > > > Hi Strahil, hi Aravinda > > Am 28.01.24 um 23:03 schrieb Strahil Nikolov: >> You didn't specify correctly the IP in the SANS but I'm not sure if that's the root cause. >> In the SANs section Specify all hosts + their IPs: IP.1=1.2.3.4IP.2=2.3.4.5DNS.1=c01.glusterDNS.2=c02.gluster > > That's what I did now: > > I took the commands from the article you recommended and added all the > alternative names and IPs into the certificate: > ------------- > openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=`hostname > -f`" -addext "subjectAltName > IP:192.168.57.41,IP:192.168.57.42,IP:192.168.57.43,IP:192.168.57.51,DNS:c01.gluster,DNS:c02.gluster,DNS:c03.gluster,DNS:cluster-client.gluster" > -out /etc/ssl/glusterfs.pem > ------------- > Stille getting on the server: > ------------- > [2024-01-29 16:32:08.877499 +0000] I > [socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL > support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 > for peer 192.168.57.51:49151 > [2024-01-29 16:32:08.881842 +0000] E [socket.c:224:ssl_dump_error_stack] > 0-socket.management: error:0A00010B:SSL routines::wrong version number > > ------------- > > And on the client: > ------------- > [2024-01-29 16:32:08.865731 +0000] I [MSGID: 100030] > [glusterfsd.c:2767:main] 0-/usr/sbin/glusterfs: Started running version > [{arg=/usr/sbin/glusterfs}, {version=10.5}, > {cmdlinestr=/usr/sbin/glusterfs --process-name fuse > --volfile-server=c02.gluster --volfile-id=/gv1 /mnt}] > [2024-01-29 16:32:08.870129 +0000] I [glusterfsd.c:2447:daemonize] > 0-glusterfs: Pid of current running process is 664 > [2024-01-29 16:32:08.880528 +0000] I [MSGID: 101190] > [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread > with index [{index=1}] > [2024-01-29 16:32:08.880935 +0000] I [MSGID: 101190] > [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread > with index [{index=0}] > [2024-01-29 16:32:08.885755 +0000] I > [glusterfsd-mgmt.c:2681:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected > from remote-host: c02.gluster > [2024-01-29 16:32:08.885879 +0000] I > [glusterfsd-mgmt.c:2720:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted > all volfile servers > [2024-01-29 16:32:08.887116 +0000] W > [glusterfsd.c:1458:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfa35) [0x7fd18d185a35] > -->/usr/sbin/glusterfs(+0x14769) [0x55d4f8d5d769] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d4f8d54447] ) 0-: > received signum (1), shutting down > [2024-01-29 16:32:08.887209 +0000] I [fuse-bridge.c:7065:fini] 0-fuse: > Unmounting '/mnt'. > [2024-01-29 16:32:08.889719 +0000] I [fuse-bridge.c:7069:fini] 0-fuse: > Closing fuse connection to '/mnt'. > [2024-01-29 16:32:08.889909 +0000] W > [glusterfsd.c:1458:cleanup_and_exit] > (-->/lib/x86_64-linux-gnu/libc.so.6(+0x89044) [0x7fd18d00a044] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x55d4f8d5be05] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d4f8d54447] ) 0-: > received signum (15), shutting down > ------------- > > executing the connect command on the client: > -------------- > openssl s_client -showcerts -connect c02.gluster:24007 > -------------- > > shows on the sever: > -------------- > [2024-01-29 16:37:08.747123 +0000] I > [socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL > support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 > for peer 192.168.57.51:58060 > [2024-01-29 16:37:08.767715 +0000] E > [socket.c:426:ssl_setup_connection_postfix] 0-socket.management: SSL > connect error (client: 192.168.57.51:58060) (server: 192.168.57.42:24007) > -------------- > > So still the same, no changes :-( > > Stefan-- Stefan Kania Landweg 13 25693 St. Michaelisdonn Signieren jeder E-Mail hilft Spam zu reduzieren und sch?tzt Ihre Privatsph?re. Ein kostenfreies Zertifikat erhalten Sie unter https://www.dgn.de/dgncert/index.html -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3477 bytes Desc: Kryptografische S/MIME-Signatur URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240131/37390614/attachment.p7s>