Strahil Nikolov
2024-Jan-28 22:03 UTC
[Gluster-users] Gluster communication via TLS client problem
You didn't specify correctly the IP in the SANS but I'm not sure if
that's the root cause.
In the SANs section Specify all hosts + their IPs:
IP.1=1.2.3.4IP.2=2.3.4.5DNS.1=c01.glusterDNS.2=c02.gluster
What is the output from the client:openssl s_client -showcerts -connect
c02.gluster:24007?
There is a very good article on the
topic:https://www.redhat.com/en/blog/hardening-gluster-installations-tls
Can you check it for a missed step ?Can you share the volume settings ?
Best Regards,Strahil Nikolov
On Sun, Jan 28, 2024 at 11:38, Stefan Kania<stefan at kania-online.de>
wrote: Hi Strahil,
ok, that's what I did now to create the certificate:
---------------------
openssl req -x509 -sha256 -key glusterfs.key? -out "glusterfs.pem"
-days
365 -subj "/C=de/ST=SH/L=St.
Michel/O=stka/OU=gluster-nodes/CN=c01.gluster" -addext "subjectAltName
=
DNS:192.168.56.41"
--------------------
still the same. The communication between the gluster-nodes is working
with TLS, but the client can't mount the volume anymore. I now try to
mount the volume with? log-level=trace
mount -t glusterfs -o log-level=trace? c02.gluster:/gv1 /mnt
and got the following:
---------------
[2024-01-28 09:22:38.348905 +0000] I [MSGID: 100030]
[glusterfsd.c:2767:main] 0-/usr/sbin/glusterfs: Started running version
[{arg=/usr/sbin/glusterfs}, {version=10.5},
{cmdlinestr=/usr/sbin/glusterfs --log-level=TRACE --process-name fuse
--volfile-server=c02.gluster --volfile-id=/gv1 /mnt}]
[2024-01-28 09:22:38.349095 +0000] T [MSGID: 0]
[xlator.c:388:xlator_dynload] 0-xlator: attempt to load file
/usr/lib/x86_64-linux-gnu/glusterfs/10.5/xlator/mount/fuse.so
[2024-01-28 09:22:38.349650 +0000] T [MSGID: 0]
[xlator.c:301:xlator_dynload_apis] 0-xlator: fuse: method missing
(reconfigure)
[2024-01-28 09:22:38.349728 +0000] T [MSGID: 0]
[xlator.c:319:xlator_dynload_apis] 0-xlator: fuse: method missing
(dump_metrics)
[2024-01-28 09:22:38.349854 +0000] T [MSGID: 0]
[xlator.c:325:xlator_dynload_apis] 0-xlator: fuse: method missing
(pass_through_fops), falling back to default
[2024-01-28 09:22:38.349979 +0000] D [MSGID: 0]
[glusterfsd.c:421:set_fuse_mount_options] 0-glusterfsd: fopen-keep-cache
mode 2
[2024-01-28 09:22:38.350111 +0000] D [MSGID: 0]
[glusterfsd.c:465:set_fuse_mount_options] 0-glusterfsd: fuse direct io
type 2
[2024-01-28 09:22:38.350222 +0000] D [MSGID: 0]
[glusterfsd.c:478:set_fuse_mount_options] 0-glusterfsd: fuse
no-root-squash mode 0
[2024-01-28 09:22:38.350347 +0000] D [MSGID: 0]
[glusterfsd.c:519:set_fuse_mount_options] 0-glusterfsd:
kernel-writeback-cache mode 2
[2024-01-28 09:22:38.350458 +0000] D [MSGID: 0]
[glusterfsd.c:537:set_fuse_mount_options] 0-glusterfsd:
fuse-flush-handle-interrupt mode 2
[2024-01-28 09:22:38.350674 +0000] T [MSGID: 0]
[options.c:1239:xlator_option_init_double] 0-fuse: option
attribute-timeout using default value 1.0
[2024-01-28 09:22:38.350792 +0000] T [MSGID: 0]
[options.c:513:xlator_option_validate_double] 0-fuse: no range check
required for 'option attribute-timeout 1.0'
[2024-01-28 09:22:38.350925 +0000] T [MSGID: 0]
[options.c:1230:xlator_option_init_uint32] 0-fuse: option
reader-thread-count using default value 1
[2024-01-28 09:22:38.351133 +0000] D [dict.c:2503:dict_get_str]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/10.5/xlator/mount/fuse.so(+0x1ee10)
[0x7ff51324ce10]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_option_init_bool+0x60)
[0x7ff513e88bf0]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_str+0xdf)
[0x7ff513e358df] ) 0-dict: key auto-invalidation, string type asked, has
unsigned integer type [Das Argument ist ung?ltig]
[2024-01-28 09:22:38.351262 +0000] D [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option
auto-invalidation using set value 0
[2024-01-28 09:22:38.351514 +0000] T [MSGID: 0]
[options.c:1239:xlator_option_init_double] 0-fuse: option entry-timeout
using default value 1.0
[2024-01-28 09:22:38.351661 +0000] T [MSGID: 0]
[options.c:513:xlator_option_validate_double] 0-fuse: no range check
required for 'option entry-timeout 1.0'
[2024-01-28 09:22:38.351894 +0000] D [dict.c:2503:dict_get_str]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/10.5/xlator/mount/fuse.so(+0x1ee6e)
[0x7ff51324ce6e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_option_init_double+0x60)
[0x7ff513e89080]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_str+0xdf)
[0x7ff513e358df] ) 0-dict: key negative-timeout, string type asked, has
float type [Das Argument ist ung?ltig]
[2024-01-28 09:22:38.351970 +0000] D [MSGID: 0]
[options.c:1239:xlator_option_init_double] 0-fuse: option
negative-timeout using set value 0.000000
[2024-01-28 09:22:38.352092 +0000] T [MSGID: 0]
[options.c:513:xlator_option_validate_double] 0-fuse: no range check
required for 'option negative-timeout 0.000000'
[2024-01-28 09:22:38.352283 +0000] T [MSGID: 0]
[options.c:1231:xlator_option_init_int32] 0-fuse: option client-pid not set
[2024-01-28 09:22:38.352402 +0000] T [MSGID: 0]
[options.c:1230:xlator_option_init_uint32] 0-fuse: option uid-map-root
not set
[2024-01-28 09:22:38.352527 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option
strict-volfile-check using default value false
[2024-01-28 09:22:38.352649 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option acl using
default value false
[2024-01-28 09:22:38.352826 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option selinux using
default value false
[2024-01-28 09:22:38.352947 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option capability using
default value false
[2024-01-28 09:22:38.353065 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option read-only not set
[2024-01-28 09:22:38.353169 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option enable-ino32
using default value false
[2024-01-28 09:22:38.353311 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option use-readdirp
using default value yes
[2024-01-28 09:22:38.353518 +0000] D [dict.c:2503:dict_get_str]
(-->/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_init+0xc5)
[0x7ff513e38c45]
-->/usr/lib/x86_64-linux-gnu/glusterfs/10.5/xlator/mount/fuse.so(+0x1f0fd)
[0x7ff51324d0fd]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_str+0xdf)
[0x7ff513e358df] ) 0-dict: key sync-to-mount, string type asked, has
pointer type [Das Argument ist ung?ltig]
[2024-01-28 09:22:38.353644 +0000] T [MSGID: 0]
[options.c:1240:xlator_option_init_time] 0-fuse: option gid-timeout
using default value 300
[2024-01-28 09:22:38.353766 +0000] T [MSGID: 0]
[options.c:80:xlator_option_validate_int] 0-fuse: no range check
required for 'option gid-timeout 300'
[2024-01-28 09:22:38.353887 +0000] T [MSGID: 0]
[options.c:1227:xlator_option_init_str] 0-fuse: option fuse-mountopts
not set
[2024-01-28 09:22:38.354028 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option resolve-gids
using default value false
[2024-01-28 09:22:38.354140 +0000] T [MSGID: 0]
[options.c:1231:xlator_option_init_int32] 0-fuse: option background-qlen
using default value 64
[2024-01-28 09:22:38.354311 +0000] T [MSGID: 0]
[options.c:1231:xlator_option_init_int32] 0-fuse: option
congestion-threshold using default value 48
[2024-01-28 09:22:38.354452 +0000] D [dict.c:2503:dict_get_str]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/10.5/xlator/mount/fuse.so(+0x1f2a1)
[0x7ff51324d2a1]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_option_init_bool+0x60)
[0x7ff513e88bf0]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_str+0xdf)
[0x7ff513e358df] ) 0-dict: key no-root-squash, string type asked, has
pointer type [Das Argument ist ung?ltig]
[2024-01-28 09:22:38.354519 +0000] D [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option no-root-squash
using set value disable
[2024-01-28 09:22:38.354671 +0000] T [MSGID: 0]
[options.c:1230:xlator_option_init_uint32] 0-fuse: option lru-limit
using default value 65536
[2024-01-28 09:22:38.354769 +0000] T [MSGID: 0]
[options.c:80:xlator_option_validate_int] 0-fuse: no range check
required for 'option lru-limit 65536'
[2024-01-28 09:22:38.354974 +0000] D [dict.c:2503:dict_get_str]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/10.5/xlator/mount/fuse.so(+0x1f312)
[0x7ff51324d312]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_option_init_uint32+0x60)
[0x7ff513e89900]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_str+0xdf)
[0x7ff513e358df] ) 0-dict: key invalidate-limit, string type asked, has
integer type [Das Argument ist ung?ltig]
[2024-01-28 09:22:38.355042 +0000] D [MSGID: 0]
[options.c:1230:xlator_option_init_uint32] 0-fuse: option
invalidate-limit using set value 0
[2024-01-28 09:22:38.355255 +0000] T [MSGID: 0]
[options.c:80:xlator_option_validate_int] 0-fuse: no range check
required for 'option invalidate-limit 0'
[2024-01-28 09:22:38.355366 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option event-history
using default value false
[2024-01-28 09:22:38.355480 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option thin-client
using default value false
[2024-01-28 09:22:38.355581 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option
kernel-writeback-cache using default value false
[2024-01-28 09:22:38.355679 +0000] T [MSGID: 0]
[options.c:1231:xlator_option_init_int32] 0-fuse: option
attr-times-granularity using default value 0
[2024-01-28 09:22:38.355873 +0000] T [MSGID: 0]
[options.c:1236:xlator_option_init_bool] 0-fuse: option
flush-handle-interrupt using default value false
[2024-01-28 09:22:38.356004 +0000] T [MSGID: 0]
[options.c:1230:xlator_option_init_uint32] 0-fuse: option
fuse-dev-eperm-ratelimit-ns using default value 10000000
[2024-01-28 09:22:38.358563 +0000] I [glusterfsd.c:2447:daemonize]
0-glusterfs: Pid of current running process is 792
[2024-01-28 09:22:38.358647 +0000] D
[logging.c:1705:__gf_log_inject_timer_event] 0-logging-infra: Starting
timer now. Timeout = 120, current buf size = 5
[2024-01-28 09:22:38.362664 +0000] D [MSGID: 0] [gf-io.c:513:gf_io_run]
0-io: Trying I/O engine 'io_uring'
[2024-01-28 09:22:38.363447 +0000] D [MSGID: 0]
[gf-io-uring.c:191:gf_io_uring_dump_params] 0-io: I/O URing: SQEs=32768,
CQEs=65536, CPU=0, Idle=0
[2024-01-28 09:22:38.363523 +0000] D [MSGID: 0]
[gf-io-uring.c:196:gf_io_uring_dump_params] 0-io: I/O URing: Flags:
CLAMP(10)
[2024-01-28 09:22:38.363663 +0000] D [MSGID: 0]
[gf-io-uring.c:199:gf_io_uring_dump_params] 0-io: I/O URing: Features:
SINGLE_MMAP(1) NODROP(2) SUBMIT_STABLE(4) RW_CUR_POS(8)
CUR_PERSONALITY(10) FAST_POLL(20) POLL_32BITS(40) SQPOLL_NONFIXED(80)
EXT_ARG(100) NATIVE_WORKERS(200) ?(1c00)
[2024-01-28 09:22:38.363800 +0000] D [MSGID: 0]
[gf-io-uring.c:251:gf_io_uring_dump_ops] 0-io: I/O URing: Max opcode = 48
[2024-01-28 09:22:38.363932 +0000] D [MSGID: 0]
[gf-io-uring.c:270:gf_io_uring_dump_ops] 0-io: I/O URing: Ops: NOP(0)
READV(1) WRITEV(2) FSYNC(3) READ_FIXED(4) WRITE_FIXED(5) POLL_ADD(6)
POLL_REMOVE(7) SYNC_FILE_RANGE(8) SENDMSG(9) RECVMSG(10) TIMEOUT(11)
TIMEOUT_REMOVE(12) ACCEPT(13) ASYNC_CANCEL(14) LINK_TIMEOUT(15)
CONNECT(16) FALLOCATE(17) OPENAT(18) CLOSE(19) FILES_UPDATE(20)
STATX(21) READ(22) WRITE(23) FADVISE(24) MADVISE(25) SEND(26) RECV(27)
OPENAT2(28) EPOLL_CTL(29) SPLICE(30) PROVIDE_BUFFERS(31)
REMOVE_BUFFERS(32) TEE(33) SHUTDOWN(34) RENAMEAT(35) UNLINKAT(36) ?(37)
?(38) ?(39) ?(40) ?(41) ?(42) ?(43) ?(44) ?(45) ?(46) ?(47) ?(48)
[2024-01-28 09:22:38.364281 +0000] D [MSGID: 0] [gf-io.c:517:gf_io_run]
0-io: I/O engine 'io_uring' is ready
[2024-01-28 09:22:38.365254 +0000] D
[rpc-clnt.c:1018:rpc_clnt_connection_init] 0-glusterfs: defaulting
frame-timeout to 30mins
[2024-01-28 09:22:38.365410 +0000] D
[rpc-clnt.c:1030:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout
[2024-01-28 09:22:38.365501 +0000] D
[rpc-transport.c:278:rpc_transport_load] 0-rpc-transport: attempt to
load file /usr/lib/x86_64-linux-gnu/glusterfs/10.5/rpc-transport/socket.so
[2024-01-28 09:22:38.366186 +0000] D [MSGID: 101233]
[options.c:973:xl_opt_validate] 0-glusterfs: option is deprecated,
continuing with correction [{key=address-family},
{preferred=transport.address-family}]
[2024-01-28 09:22:38.366264 +0000] T [MSGID: 0]
[options.c:80:xlator_option_validate_int] 0-glusterfs: no range check
required for 'option remote-port 24007'
[2024-01-28 09:22:38.367351 +0000] D [socket.c:4561:socket_init]
0-glusterfs: Configured transport.tcp-user-timeout=42
[2024-01-28 09:22:38.367566 +0000] D [socket.c:4581:socket_init]
0-glusterfs: Reconfigured transport.keepalivecnt=9
[2024-01-28 09:22:38.367629 +0000] D
[rpc-clnt.c:1591:rpcclnt_cbk_program_register] 0-glusterfs: New program
registered: GlusterFS Callback, Num: 52743234, Ver: 1
[2024-01-28 09:22:38.367748 +0000] T [rpc-clnt.c:396:rpc_clnt_reconnect]
0-glusterfs: attempting reconnect
[2024-01-28 09:22:38.367865 +0000] T [socket.c:3377:socket_connect]
0-glusterfs: connecting 0x7ff50c007768, sock=-1
[2024-01-28 09:22:38.368068 +0000] D [dict.c:1290:data_to_uint16]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/10.5/rpc-transport/socket.so(+0x98dd)
[0x7ff5123438dd]
-->/usr/lib/x86_64-linux-gnu/glusterfs/10.5/rpc-transport/socket.so(socket_client_get_remote_sockaddr+0x2c8)
[0x7ff512343448]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(data_to_uint16+0x146)
[0x7ff513e32676] ) 0-dict: key null, unsigned integer type asked, has
integer type [Das Argument ist ung?ltig] # --> the argument is invalide
[2024-01-28 09:22:38.368170 +0000] T [MSGID: 0]
[common-utils.c:504:gf_resolve_ip6] 0-resolver: DNS cache not present,
freshly probing hostname: c02.gluster
[2024-01-28 09:22:38.368577 +0000] D [MSGID: 0]
[common-utils.c:542:gf_resolve_ip6] 0-resolver: returning
ip-192.168.57.42 (port-24007) for hostname: c02.gluster and port: 24007
[2024-01-28 09:22:38.368641 +0000] D [socket.c:3294:socket_fix_ssl_opts]
0-glusterfs: disabling SSL for portmapper connection
[2024-01-28 09:22:38.368739 +0000] T [socket.c:1021:__socket_nodelay]
0-glusterfs: NODELAY enabled for socket 11
[2024-01-28 09:22:38.368846 +0000] T [socket.c:1107:__socket_keepalive]
0-glusterfs: Keep-alive enabled for socket: 11, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2024-01-28 09:22:38.368954 +0000] T [socket.c:3495:socket_connect]
0-glusterfs: >>> connect() with non-blocking IO for ALL
[2024-01-28 09:22:38.369071 +0000] T [socket.c:206:socket_dump_info]
0-glusterfs: $$$ client: connecting to (af:2,sock:11) 192.168.57.42
non-SSL (errno:0:Erfolg)
[2024-01-28 09:22:38.369501 +0000] I [MSGID: 101190]
[event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread
with index [{index=1}]
[2024-01-28 09:22:38.369565 +0000] T
[socket.c:2897:socket_event_handler] 0-glusterfs: client (sock:11) in:0,
out:4, err:0
[2024-01-28 09:22:38.369682 +0000] T
[socket.c:2903:socket_event_handler] 0-glusterfs: client (sock:11)
socket is not connected, completing connection
[2024-01-28 09:22:38.369872 +0000] T
[rpc-clnt.c:1443:rpc_clnt_record_build_header] 0-rpc-clnt: Request
fraglen 140, payload: 76, rpc hdr: 64
[2024-01-28 09:22:38.369990 +0000] T [rpc-clnt.c:1729:rpc_clnt_submit]
0-rpc-clnt: submitted request (unique: 0, XID: 0x2, Program: GlusterFS
Handshake, ProgVers: 2, Proc: 2) to rpc-transport (glusterfs)
[2024-01-28 09:22:38.370068 +0000] D
[rpc-clnt-ping.c:296:rpc_clnt_start_ping] 0-glusterfs: ping timeout is
0, returning
[2024-01-28 09:22:38.370207 +0000] T
[socket.c:2803:socket_handle_client_connection_attempt] 0-glusterfs:
socket_connect_finish() returned 0
[2024-01-28 09:22:38.370301 +0000] T
[socket.c:2910:socket_event_handler] 0-glusterfs: (sock:11)
socket_complete_connection() returned 1
[2024-01-28 09:22:38.370471 +0000] T
[socket.c:2916:socket_event_handler] 0-glusterfs: (sock:11) returning to
wait on socket
[2024-01-28 09:22:38.370529 +0000] T
[socket.c:2897:socket_event_handler] 0-glusterfs: client (sock:11) in:0,
out:4, err:0
[2024-01-28 09:22:38.370668 +0000] T
[socket.c:2923:socket_event_handler] 0-glusterfs: Client socket (11) is
already connected
[2024-01-28 09:22:38.370733 +0000] T
[socket.c:2932:socket_event_handler] 0-glusterfs: (sock:11)
socket_event_poll_out returned 0
[2024-01-28 09:22:38.370837 +0000] I [MSGID: 101190]
[event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread
with index [{index=0}]
[2024-01-28 09:22:38.374114 +0000] T
[socket.c:2897:socket_event_handler] 0-glusterfs: client (sock:11) in:1,
out:0, err:24
[2024-01-28 09:22:38.374179 +0000] T [socket.c:206:socket_dump_info]
0-glusterfs: $$$ client: disconnecting from (af:2,sock:11) 192.168.57.42
non-SSL (errno:104:Die Verbindung wurde vom Kommunikationspartner
zur?ckgesetzt) # --> connection reseted by peer
[2024-01-28 09:22:38.374326 +0000] D
[socket.c:2966:socket_event_handler] 0-transport: EPOLLERR -
disconnecting (sock:11) (non-SSL)
[2024-01-28 09:22:38.374447 +0000] I
[glusterfsd-mgmt.c:2681:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected
from remote-host: c02.gluster
[2024-01-28 09:22:38.374529 +0000] I
[glusterfsd-mgmt.c:2720:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted
all volfile servers
[2024-01-28 09:22:38.375599 +0000] D
[logging.c:1675:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer
size reduced. About to flush 5 extra log messages
[2024-01-28 09:22:38.375716 +0000] D
[logging.c:1681:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 5
extra log messages
[2024-01-28 09:22:38.375878 +0000] W
[glusterfsd.c:1458:cleanup_and_exit]
(-->/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfa35) [0x7ff513de8a35]
-->/usr/sbin/glusterfs(+0x14769) [0x564f61e2c769]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x564f61e23447] ) 0-:
received signum (1), shutting down
[2024-01-28 09:22:38.375999 +0000] D
[mgmt-pmap.c:90:rpc_clnt_mgmt_pmap_signout] 0-fsd-mgmt: portmapper
signout arguments not given
[2024-01-28 09:22:38.376093 +0000] I [fuse-bridge.c:7065:fini] 0-fuse:
Unmounting '/mnt'.
[2024-01-28 09:22:38.378550 +0000] I [fuse-bridge.c:7069:fini] 0-fuse:
Closing fuse connection to '/mnt'.
[2024-01-28 09:22:38.378765 +0000] W
[glusterfsd.c:1458:cleanup_and_exit]
(-->/lib/x86_64-linux-gnu/libc.so.6(+0x89044) [0x7ff513c6d044]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x564f61e2ae05]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x564f61e23447] ) 0-:
received signum (15), shutting down
---------------
So the client can resolve the hostname aand it fits to the neme of the
gluster-host and it's fqdn.
Still the same on the gluster-host:
----------------
==> /var/log/glusterfs/glusterd.log <=[2024-01-28 09:32:47.673142 +0000] I
[socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL
support for MGMT is ENABLED IO path is ENABLED certificate depth is 1
for peer 192.168.57.51:49151
[2024-01-28 09:32:47.677804 +0000] E [socket.c:224:ssl_dump_error_stack]
0-socket.management:? error:0A00010B:SSL routines::wrong version number
----------------
? :-( still not working. What I don't anderstand: why is it working
between the gluster-host but not between the gluster-client and any of
the gluster-hosts? Atre they manage the TLS connection a different way?
Stefan
Am 28.01.24 um 08:44 schrieb Strahil Nikolov:> Usually with Certificates it's always a pain.I would ask you to
regenerate the certificates but by adding the FQDN of the system and the IP used
by the clients to reach the brick in 'SANS' section of the cert. Also,
set the validity to 365 days for the test.
> Best Regards,Strahil Nikolov
>?
>?
>? ? On Fri, Jan 26, 2024 at 21:37, Stefan Kania<stefan at
kania-online.de> wrote:? Hi Aravinda
>
> Am 26.01.24 um 17:01 schrieb Aravinda:
>> Does the combined glusterfs.ca includes client nodes pem? Also this
file
>> need to be placed in Client node as well.
>
> Yes, I put all the Gluster-node Certificates AND the client certificate
> into the glusterfs.ca file. And I put the file to all gluster-nodes and
> clients. I did it twice (delete all certificate and restart all over)the
> result was always the same.
>
> Stefan
>
>
>
>? ?
>
--
Stefan Kania
Landweg 13
25693 St. Michaelisdonn
Signieren jeder E-Mail hilft Spam zu reduzieren und sch?tzt Ihre
Privatsph?re. Ein kostenfreies Zertifikat erhalten Sie unter
https://www.dgn.de/dgncert/index.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20240128/2db8df6f/attachment.html>
Stefan Kania
2024-Jan-29 13:07 UTC
[Gluster-users] Gluster communication via TLS client problem
Am 28.01.24 um 23:03 schrieb Strahil Nikolov:> You didn't specify correctly the IP in the SANS but I'm not sure if that's the root cause. > In the SANs section Specify all hosts + their IPs: IP.1=1.2.3.4IP.2=2.3.4.5DNS.1=c01.glusterDNS.2=c02.glusterahh ok, I can try it, but I don't think that's my problem :-(> > What is the output from the client:openssl s_client -showcerts -connect c02.gluster:24007Here is the result connecting from client to server: ------------------- root at cluster-client:~# openssl s_client -showcerts -connect c02.gluster:24007 CONNECTED(00000003) depth=0 CN = c02.gluster verify error:num=18:self-signed certificate verify return:1 depth=0 CN = c02.gluster verify return:1 --- Certificate chain 0 s:CN = c02.gluster i:CN = c02.gluster a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Jan 28 15:04:34 2024 GMT; NotAfter: Feb 27 15:04:34 2024 GMT -----BEGIN CERTIFICATE----- MIIDDTCCAfWgAwIBAgIUZeKYuE2vfouJdoZmqyjQQfKSNzMwDQYJKoZIhvcNAQEL BQAwFjEUMBIGA1UEAwwLYzAyLmdsdXN0ZXIwHhcNMjQwMTI4MTUwNDM0WhcNMjQw MjI3MTUwNDM0WjAWMRQwEgYDVQQDDAtjMDIuZ2x1c3RlcjCCASIwDQYJKoZIhvcN AQEBBQADggEPADCCAQoCggEBALqvFOHIz3AEmwTxEE826NU2InRVogAPgdZxfNon OC9ydY87L5mfJcdDTrOLNOODxtKsd9IaZ2Y2Y7gSNYT+pEq0SfylN2Fq3OxIAkF0 dXRgroQQo/sV4UKaiMEcZ1Z38PUgjDomnXclMZc6tPMo2nVSbTQCdOcgI3bf0qbS tDVVeKAbgEtVV/+6HBZQJPEuQiT/Gy88sVmS3kdioyOAus+K31nBx2L4jhCot+Vl 8Sw6G+TIIUvKcYJ1P5dOz9dgZ4/gs9/PwP2AlvzAM1DGHsq2lmsBaPgqCGEpAn+5 asYgKwwPYQEeT/MEypA7pNXPdhvtgkjzNEQXMXWpgt/8iD0CAwEAAaNTMFEwHQYD VR0OBBYEFIy62thrmXQc71J4Re1txCszlvjVMB8GA1UdIwQYMBaAFIy62thrmXQc 71J4Re1txCszlvjVMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEB ACShUM7RYCVjIVgNJQvZ+eSknBTdV8sub0EFM0oD6nkkt8DDVkdaE7E83ykzcQSZ cNDWEMJdt1yKcaCtbOaHnE4BPsL4AIFkHVAq3hJNMDkZQY7aslHTnWgYJBqj3fHR K95jEyAv1C9Eo7pRj2WX5C6FlpQ/FhNWZd5IxM0J+/TL3qC/y41+v9EZZ+e3DDYp LQ0z/qLbDjebvjSRgudVaTR5TVCZXydkpY6kMCBAnYhgqkcWA0FhalpMcZ8qzPRD NW/SvZDmZH7SbUjuxHNDnwFI8iJ43gzgFoFrUOXuThROkqn2uOaXaPfClW0Z3quX l1fRx/Tjnth2y8hor+EDsDM-----END CERTIFICATE----- --- Server certificate subject=CN = c02.gluster issuer=CN = c02.gluster --- No client certificate CA names sent Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224 Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512 Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: ECDH, prime256v1, 256 bits --- SSL handshake has read 1534 bytes and written 777 bytes Verification error: self-signed certificate --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 18 (self-signed certificate) --- --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: EC49FFB5EB73CC773A4D6BF322644B69450452ECA5D6CEC813505C98301DB277 Session-ID-ctx: Resumption PSK: 957A3A01436961C058515E8E5F74C817E1CEE574234DF6071E78117565CC6D579EBF6423DF94D7CDAD122F515EA03631 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 7200 (seconds) TLS session ticket: 0000 - 09 7c c3 b4 fd 39 18 ae-3a e4 ee 74 64 ed cc d7 .|...9..:..td... 0010 - c1 90 39 48 7d 00 69 a5-82 1c 0d 15 42 77 7a 31 ..9H}.i.....Bwz1 Start Time: 1706527955 Timeout : 7200 (sec) Verify return code: 18 (self-signed certificate) Extended master secret: no Max Early Data: 0 --- read R BLOCK --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Session-ID: 80A42FC0DF4F92F3118474F1CEFC4FB8A12344E74E190EE9E9161884C482E2B1 Session-ID-ctx: Resumption PSK: C18CC9CD3BFDF0701B46255049802F5BAA8D36DA3EAC2BD7C2350DDEC71EDB2E622DDF8CD926B6174E1EFE09E72479C7 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 7200 (seconds) TLS session ticket: 0000 - da 99 d0 e7 2d 1d 1a 73-8b 98 62 a6 43 34 b8 72 ....-..s..b.C4.r 0010 - bc 84 12 b5 6e 37 19 d9-b2 b5 ff 48 98 f7 e6 07 ....n7.....H.... Start Time: 1706527955 Timeout : 7200 (sec) Verify return code: 18 (self-signed certificate) Extended master secret: no Max Early Data: 0 --- read R BLOCK 4037D3DD357F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:303 ------------------- I think, the last line can be ignored. Testing same command from server to sever the result is the same. The management-encryption is working. The connection between the three nodes is not showing any error glusterd.log is showing: ----------------- 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.41:49148 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.42:49149 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.43:49148 ----------------- ?> There is a very good article on the topic:https://www.redhat.com/en/blog/hardening-gluster-installations-tlsNice article ;-) Maybe the only one I did note read up to now :-) I did everything the same way :-(> Can you check it for a missed step ?Can you share the volume settings ?Yes, here are the result from "gluster v info" and "gluster v status" ---------------- root at c01:~# gluster v info Volume Name: gv1 Type: Replicate Volume ID: fe89dc61-3ee5-4507-8025-22c19f248d53 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: c01.gluster:/gluster/brick Brick2: c02.gluster:/gluster/brick Brick3: c03.gluster:/gluster/brick Options Reconfigured: performance.client-io-threads: off nfs.disable: on transport.address-family: inet storage.fips-mode-rchecksum: on cluster.granular-entry-heal: on auth.ssl-allow: * client.ssl: on server.ssl: on root at c01:~# gluster v status Status of volume: gv1 Gluster process TCP Port RDMA Port Online Pid -------------------------------------------------------------------------- Brick c01.gluster:/gluster/brick 59287 0 Y 866 Brick c02.gluster:/gluster/brick 51998 0 Y 850 Brick c03.gluster:/gluster/brick 60291 0 Y 807 Self-heal Daemon on localhost N/A N/A Y 1216 Self-heal Daemon on c03.gluster N/A N/A Y 883 Self-heal Daemon on c02.gluster N/A N/A Y 883 Task Status of Volume gv1 ------------------------------------------------------------------------------ There are no active volume tasks ---------------- There is only on thing I set "auth.ssl-allow: *" instead of all the hostnames. But with all fqdn set it's the same. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3477 bytes Desc: Kryptografische S/MIME-Signatur URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240129/cbecd269/attachment.p7s>
Stefan Kania
2024-Jan-29 16:40 UTC
[Gluster-users] Gluster communication via TLS client problem
Hi Strahil, hi Aravinda Am 28.01.24 um 23:03 schrieb Strahil Nikolov:> You didn't specify correctly the IP in the SANS but I'm not sure if that's the root cause. > In the SANs section Specify all hosts + their IPs: IP.1=1.2.3.4IP.2=2.3.4.5DNS.1=c01.glusterDNS.2=c02.glusterThat's what I did now: I took the commands from the article you recommended and added all the alternative names and IPs into the certificate: ------------- openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=`hostname -f`" -addext "subjectAltName = IP:192.168.57.41,IP:192.168.57.42,IP:192.168.57.43,IP:192.168.57.51,DNS:c01.gluster,DNS:c02.gluster,DNS:c03.gluster,DNS:cluster-client.gluster" -out /etc/ssl/glusterfs.pem ------------- Stille getting on the server: ------------- [2024-01-29 16:32:08.877499 +0000] I [socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.51:49151 [2024-01-29 16:32:08.881842 +0000] E [socket.c:224:ssl_dump_error_stack] 0-socket.management: error:0A00010B:SSL routines::wrong version number ------------- And on the client: ------------- [2024-01-29 16:32:08.865731 +0000] I [MSGID: 100030] [glusterfsd.c:2767:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=10.5}, {cmdlinestr=/usr/sbin/glusterfs --process-name fuse --volfile-server=c02.gluster --volfile-id=/gv1 /mnt}] [2024-01-29 16:32:08.870129 +0000] I [glusterfsd.c:2447:daemonize] 0-glusterfs: Pid of current running process is 664 [2024-01-29 16:32:08.880528 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}] [2024-01-29 16:32:08.880935 +0000] I [MSGID: 101190] [event-epoll.c:667:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}] [2024-01-29 16:32:08.885755 +0000] I [glusterfsd-mgmt.c:2681:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: c02.gluster [2024-01-29 16:32:08.885879 +0000] I [glusterfsd-mgmt.c:2720:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers [2024-01-29 16:32:08.887116 +0000] W [glusterfsd.c:1458:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfa35) [0x7fd18d185a35] -->/usr/sbin/glusterfs(+0x14769) [0x55d4f8d5d769] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d4f8d54447] ) 0-: received signum (1), shutting down [2024-01-29 16:32:08.887209 +0000] I [fuse-bridge.c:7065:fini] 0-fuse: Unmounting '/mnt'. [2024-01-29 16:32:08.889719 +0000] I [fuse-bridge.c:7069:fini] 0-fuse: Closing fuse connection to '/mnt'. [2024-01-29 16:32:08.889909 +0000] W [glusterfsd.c:1458:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(+0x89044) [0x7fd18d00a044] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x55d4f8d5be05] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x55d4f8d54447] ) 0-: received signum (15), shutting down ------------- executing the connect command on the client: -------------- openssl s_client -showcerts -connect c02.gluster:24007 -------------- shows on the sever: -------------- [2024-01-29 16:37:08.747123 +0000] I [socket.c:4288:ssl_setup_connection_params] 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 192.168.57.51:58060 [2024-01-29 16:37:08.767715 +0000] E [socket.c:426:ssl_setup_connection_postfix] 0-socket.management: SSL connect error (client: 192.168.57.51:58060) (server: 192.168.57.42:24007) -------------- So still the same, no changes :-( Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3477 bytes Desc: Kryptografische S/MIME-Signatur URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240129/db217275/attachment.p7s>