Marco Fais
2021-May-17 10:51 UTC
[Gluster-users] Issues with glustershd with release 8.4 and 9.1
Hi, I am having significant issues with glustershd with releases 8.4 and 9.1. My oVirt clusters are using gluster storage backends, and were running fine with Gluster 7.x (shipped with earlier versions of oVirt Node 4.4.x). Recently the oVirt project moved to Gluster 8.4 for the nodes, and hence I have moved to this release when upgrading my clusters. Since then I am having issues whenever one of the nodes is brought down; when the nodes come back up online the bricks are typically back up and working, but some (random) glustershd processes in the various nodes seem to have issues connecting to some of them. Typically when this happens the files are not getting healed VM_Storage_1 Distributed_replicate Started (UP) - 27/27 Bricks Up Capacity: (27.10% used) 2.00 TiB/8.00 TiB (used/total) Self-Heal: lab-cnvirt-h01-storage:/bricks/vm_b1_vol/brick (8 File(s) to heal). lab-cnvirt-h02-storage:/bricks/vm_b1_vol/brick (8 File(s) to heal). lab-cnvirt-h01-storage:/bricks/vm_b2_vol/brick (4 File(s) to heal). lab-cnvirt-h02-storage:/bricks/vm_b2_arb/brick (4 File(s) to heal). lab-cnvirt-h02-storage:/bricks/vm_b2_vol/brick (5 File(s) to heal). lab-cnvirt-h01-storage:/bricks/vm_b1_arb/brick (5 File(s) to heal). lab-cnvirt-h01-storage:/bricks/vm_b3_vol/brick (9 File(s) to heal). lab-cnvirt-h02-storage:/bricks/vm_b3_vol/brick (9 File(s) to heal). lab-cnvirt-h01-storage:/bricks/vm_b4_vol/brick (4 File(s) to heal). lab-cnvirt-h02-storage:/bricks/vm_b4_arb/brick (4 File(s) to heal). lab-cnvirt-h02-storage:/bricks/vm_b4_vol/brick (10 File(s) to heal). lab-cnvirt-h01-storage:/bricks/vm_b3_arb/brick (10 File(s) to heal). lab-cnvirt-h01-storage:/bricks/vm_b5_vol/brick (3 File(s) to heal). lab-cnvirt-h02-storage:/bricks/vm_b5_vol/brick (3 File(s) to heal). lab-cnvirt-h01-storage:/bricks/vm_b6_vol/brick (4 File(s) to heal). lab-cnvirt-h02-storage:/bricks/vm_b6_arb/brick (4 File(s) to heal). lab-cnvirt-h02-storage:/bricks/vm_b6_vol/brick (9 File(s) to heal). lab-cnvirt-h01-storage:/bricks/vm_b5_arb/brick (9 File(s) to heal). (They will never heal; the number of files to heal however changes). In the glustershd.log files, I can see the following continuously: [2021-05-17 10:27:30.531561 +0000] I [rpc-clnt.c:1968:rpc_clnt_reconfig] 2-VM_Storage_1-client-3: changing port to 49154 (from 0) [2021-05-17 10:27:30.533709 +0000] I [rpc-clnt.c:1968:rpc_clnt_reconfig] 2-VM_Storage_1-client-7: changing port to 49155 (from 0) [2021-05-17 10:27:30.534211 +0000] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 2-VM_Storage_1-client-3: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}] [2021-05-17 10:27:30.534514 +0000] W [MSGID: 114043] [client-handshake.c:727:client_setvolume_cbk] 2-VM_Storage_1-client-3: failed to set the volume [{errno=2}, {error=No such file or directory}] The message "I [MSGID: 114018] [client.c:2229:client_rpc_notify] 2-VM_Storage_1-client-3: disconnected from client, process will keep trying to connect glusterd until brick's port is available [{conn-name=VM_Storage_1-client-3}]" repeated 4 times between [2021-05-17 10:27:18.510668 +0000] and [2021-05-17 10:27:30.534569 +0000] [2021-05-17 10:27:30.536254 +0000] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 2-VM_Storage_1-client-7: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}] [2021-05-17 10:27:30.536620 +0000] W [MSGID: 114043] [client-handshake.c:727:client_setvolume_cbk] 2-VM_Storage_1-client-7: failed to set the volume [{errno=2}, {error=No such file or directory}] [2021-05-17 10:27:30.536638 +0000] W [MSGID: 114007] [client-handshake.c:752:client_setvolume_cbk] 2-VM_Storage_1-client-7: failed to get from reply dict [{process-uuid}, {errno=22}, {error=Invalid argument}] [2021-05-17 10:27:30.536651 +0000] E [MSGID: 114044] [client-handshake.c:757:client_setvolume_cbk] 2-VM_Storage_1-client-7: SETVOLUME on remote-host failed [{remote-error=Brick not found}, {errno=2}, {error=No such file or directory}] [2021-05-17 10:27:30.536660 +0000] I [MSGID: 114051] [client-handshake.c:879:client_setvolume_cbk] 2-VM_Storage_1-client-7: sending CHILD_CONNECTING event [] [2021-05-17 10:27:30.536686 +0000] I [MSGID: 114018] [client.c:2229:client_rpc_notify] 2-VM_Storage_1-client-7: disconnected from client, process will keep trying to connect glusterd until brick's port is available [{conn-name=VM_Storage_1-client-7}] [2021-05-17 10:27:33.537589 +0000] I [rpc-clnt.c:1968:rpc_clnt_reconfig] 2-VM_Storage_1-client-3: changing port to 49154 (from 0) [2021-05-17 10:27:33.539554 +0000] I [rpc-clnt.c:1968:rpc_clnt_reconfig] 2-VM_Storage_1-client-7: changing port to 49155 (from 0)>From my understanding the process is trying to connect to the brick on thewrong port lab-cnvirt-h03-storage:-bricks-vm_b2_vol-brick:8:brick-id=VM_Storage_1-client-7 Brick lab-cnvirt-h03-storage:/bricks/vm_b2_vol/brick *49169 *0 Y 1600469 lab-cnvirt-h03-storage:-bricks-vm_b1_vol-brick:8:brick-id=VM_Storage_1-client-3 Brick lab-cnvirt-h03-storage:/bricks/vm_b1_vol/brick *49168 *0 Y 1600460 Typically to resolve this I have to manually kill the affected glusterfsd process (in this case the two processes above) and then issue a gluster volume start VM_Storage_1 force to restart them. As soon as I do that, the process is able to re-connect and start the healing: [2021-05-17 10:46:12.513706 +0000] I [MSGID: 100041] [glusterfsd-mgmt.c:1035:glusterfs_handle_svc_attach] 0-glusterfs: received attach request for volfile [{volfile-id=shd/VM_Storage_1}] [2021-05-17 10:46:12.513847 +0000] I [MSGID: 100040] [glusterfsd-mgmt.c:109:mgmt_process_volfile] 0-glusterfs: No change in volfile, countinuing [] [2021-05-17 10:46:14.626397 +0000] I [rpc-clnt.c:1968:rpc_clnt_reconfig] 2-VM_Storage_1-client-7: changing port to 49157 (from 0) [2021-05-17 10:46:14.628468 +0000] I [rpc-clnt.c:1968:rpc_clnt_reconfig] 2-VM_Storage_1-client-3: changing port to 49156 (from 0) [2021-05-17 10:46:14.628927 +0000] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 2-VM_Storage_1-client-7: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}] [2021-05-17 10:46:14.629633 +0000] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 2-VM_Storage_1-client-7: Connected, attached to remote volume [{conn-name=VM_Storage_1-client-7}, {remote_subvol=/bricks/vm_b2_vol/brick}] [2021-05-17 10:46:14.631212 +0000] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 2-VM_Storage_1-client-3: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}] [2021-05-17 10:46:14.631949 +0000] I [MSGID: 114046] [client-handshake.c:857:client_setvolume_cbk] 2-VM_Storage_1-client-3: Connected, attached to remote volume [{conn-name=VM_Storage_1-client-3}, {remote_subvol=/bricks/vm_b1_vol/brick}] [2021-05-17 10:46:14.705116 +0000] I [MSGID: 108026] [afr-self-heal-data.c:347:afr_selfheal_data_do] 2-VM_Storage_1-replicate-2: performing data selfheal on 399bdfe5-01b7-46f9-902b-9351420debc9 [2021-05-17 10:46:14.705214 +0000] I [MSGID: 108026] [afr-self-heal-data.c:347:afr_selfheal_data_do] 2-VM_Storage_1-replicate-2: performing data selfheal on 3543a4c7-4a68-4193-928a-c9f7ef08ce4e Am I doing something wrong here? I have also tried to upgrade to 9.1 in my test cluster (logs are from the 9.1) -- but I have the exact same issue. Do you need any specific information? It is happening with all my volumes, but info for the one above is listed below: Volume Name: VM_Storage_1 Type: Distributed-Replicate Volume ID: 1a4e23db-1c98-4d89-b888-b4ae2e0ad5fc Status: Started Snapshot Count: 0 Number of Bricks: 9 x (2 + 1) = 27 Transport-type: tcp Bricks: Brick1: lab-cnvirt-h01-storage:/bricks/vm_b1_vol/brick Brick2: lab-cnvirt-h02-storage:/bricks/vm_b1_vol/brick Brick3: lab-cnvirt-h03-storage:/bricks/vm_b1_arb/brick (arbiter) Brick4: lab-cnvirt-h03-storage:/bricks/vm_b1_vol/brick Brick5: lab-cnvirt-h01-storage:/bricks/vm_b2_vol/brick Brick6: lab-cnvirt-h02-storage:/bricks/vm_b2_arb/brick (arbiter) Brick7: lab-cnvirt-h02-storage:/bricks/vm_b2_vol/brick Brick8: lab-cnvirt-h03-storage:/bricks/vm_b2_vol/brick Brick9: lab-cnvirt-h01-storage:/bricks/vm_b1_arb/brick (arbiter) Brick10: lab-cnvirt-h01-storage:/bricks/vm_b3_vol/brick Brick11: lab-cnvirt-h02-storage:/bricks/vm_b3_vol/brick Brick12: lab-cnvirt-h03-storage:/bricks/vm_b3_arb/brick (arbiter) Brick13: lab-cnvirt-h03-storage:/bricks/vm_b3_vol/brick Brick14: lab-cnvirt-h01-storage:/bricks/vm_b4_vol/brick Brick15: lab-cnvirt-h02-storage:/bricks/vm_b4_arb/brick (arbiter) Brick16: lab-cnvirt-h02-storage:/bricks/vm_b4_vol/brick Brick17: lab-cnvirt-h03-storage:/bricks/vm_b4_vol/brick Brick18: lab-cnvirt-h01-storage:/bricks/vm_b3_arb/brick (arbiter) Brick19: lab-cnvirt-h01-storage:/bricks/vm_b5_vol/brick Brick20: lab-cnvirt-h02-storage:/bricks/vm_b5_vol/brick Brick21: lab-cnvirt-h03-storage:/bricks/vm_b5_arb/brick (arbiter) Brick22: lab-cnvirt-h03-storage:/bricks/vm_b5_vol/brick Brick23: lab-cnvirt-h01-storage:/bricks/vm_b6_vol/brick Brick24: lab-cnvirt-h02-storage:/bricks/vm_b6_arb/brick (arbiter) Brick25: lab-cnvirt-h02-storage:/bricks/vm_b6_vol/brick Brick26: lab-cnvirt-h03-storage:/bricks/vm_b6_vol/brick Brick27: lab-cnvirt-h01-storage:/bricks/vm_b5_arb/brick (arbiter) Options Reconfigured: storage.owner-uid: 36 storage.owner-gid: 36 performance.strict-o-direct: on cluster.read-hash-mode: 3 performance.client-io-threads: on server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off storage.fips-mode-rchecksum: on nfs.disable: on transport.address-family: inet cluster.self-heal-daemon: enable Regards, Marco -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210517/3dfe6c3b/attachment.html>
Ravishankar N
2021-May-19 05:30 UTC
[Gluster-users] Issues with glustershd with release 8.4 and 9.1
On Mon, May 17, 2021 at 4:22 PM Marco Fais <evilmf at gmail.com> wrote:> Hi, > > I am having significant issues with glustershd with releases 8.4 and 9.1. > > My oVirt clusters are using gluster storage backends, and were running > fine with Gluster 7.x (shipped with earlier versions of oVirt Node 4.4.x). > Recently the oVirt project moved to Gluster 8.4 for the nodes, and hence I > have moved to this release when upgrading my clusters. > > Since then I am having issues whenever one of the nodes is brought down; > when the nodes come back up online the bricks are typically back up and > working, but some (random) glustershd processes in the various nodes seem > to have issues connecting to some of them. > >When the issue happens, can you check if the TCP port number of the brick (glusterfsd) processes displayed in `gluster volume status` matches with that of the actual port numbers observed (i.e. the --brick-port argument) when you run `ps aux | grep glusterfsd` ? If they don't match, then glusterd has incorrect brick port information in its memory and serving it to glustershd. Restarting glusterd instead of (killing the bricks + `volume start force`) should fix it, although we need to find why glusterd serves incorrect port numbers. If they do match, then can you take a statedump of glustershd to check that it is indeed disconnected from the bricks? You will need to verify that 'connected=1' in the statedump. See "Self-heal is stuck/ not getting completed." section in https://docs.gluster.org/en/latest/Troubleshooting/troubleshooting-afr/. Statedump can be taken by `kill -SIGUSR1 $pid-of-glustershd`. It will be generated in the /var/run/gluster/ directory. Regards, Ravi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210519/63d9466b/attachment.html>