Chris Drescher
2018-Dec-10 08:22 UTC
[Gluster-users] glusterd keeps resyncing shards over and over again
Hello everybody We are experiencing an urgent issue with glusterd! After an upgrade from centos7.5 to 7.6 our grusterfs keeps resyncing specific shards over and over again! Is this a known problem? This is very urgent! Please help! Thanks in advance! Kind regards. Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181210/bf93052d/attachment.html>
Chris Drescher
2018-Dec-10 10:12 UTC
[Gluster-users] glusterd keeps resyncing shards over and over again
Let me provide more information. We have 3 gluster nodes running with sharding activated. Node1: CentOS 7.5 - Glusterfs 3.12.6Node2: CentOS 7.5 - Glusterfs 3.12.6Node3: CentOS 7.5 - Glusterfs 3.12.6 Now we updated Node 3 from CentOS 7.5 to 7.6 which caused a reboot.Glusterd Version changed from 3.12.6 to 3.12.15 Node1: CentOS 7.5 - Glusterfs 3.12.6Node2: CentOS 7.5 - Glusterfs 3.12.6Node3: CentOS 7.6 - Glusterfs 3.12.15 Afterwards gluster heal daemon keeps resyncing specific shards on bricks on Node1 and Node2. Always the same shards. LOGS: On upgraded NODE3: /var/log/glusterfs/glusterd.log[2018-12-10 09:24:42.314624] E [MSGID: 106062] [glusterd-utils.c:10112:glusterd_max_opversion_use_rsp_dict] 0- management: Maximum supported op-version not set in destination dictionary tail -f /var/log/glusterfs/glustershd.log[2018-12-09 04:28:05.687127] I [MSGID: 108026] [afr-self-heal-common.c:1726:afr_log_selfheal] 0-data- replicate-0: Completed data selfheal on 3f1711c2-de8c-4e8e-be10- a252f5b1b4ad. sources=[0] 2 sinks=1 On NODE1: tail -f /var/log/glusterfs/glfsheal-data.log[2018-12-10 10:00:01.898139] I [MSGID: 114035] [client- handshake.c:202:client_set_lk_version_cbk] 0-data-client-16: Server lk version = 1[2018-12-10 10:00:01.898487] I [MSGID: 114057] [client- handshake.c:1478:select_server_supported_programs] 0-data-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-12-10 10:00:01.898892] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-data- client-17: Connected to data-client-17, attached to remote volume '/gluster/arb2/data'. [2018-12-10 10:00:01.898900] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0- data-client-17: Server and Client lk-version numbers are not same, reopening the fds [2018-12-10 10:00:01.899007] I [MSGID: 114035] [client- handshake.c:202:client_set_lk_version_cbk] 0-data-client-17: Server lk version = 1[2018-12-10 10:00:01.901528] I [MSGID: 108031] [afr- common.c:2376:afr_local_discovery_cbk] 0-data-replicate-3: selecting local read_child data-client-9[2018-12-10 10:00:01.901876] I [MSGID: 108031] [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-5: selecting local read_child data-client-15[2018-12-10 10:00:01.901978] I [MSGID: 108031] [afr-common.c:2376:afr_local_discovery_cbk] 0-data- replicate-4: selecting local read_child data-client-12[2018-12-10 10:00:01.902708] I [MSGID: 108031] [afr- common.c:2376:afr_local_discovery_cbk] 0-data-replicate-2: selecting local read_child data-client-6[2018-12-10 10:00:01.902750] I [MSGID: 104041] [glfs-resolve.c:971:__glfs_active_subvol] 0-data: switched to graph 70312d70-6f64-3031-2e6c-696e75786661 (0) Hope that helps! On Mon, 2018-12-10 at 09:22 +0100, Chris Drescher wrote:> Hello everybody > > We are experiencing an urgent issue with glusterd! > After an upgrade from centos7.5 to 7.6 our grusterfs keeps resyncing > specific shards over and over again! > > Is this a known problem? > > This is very urgent! Please help! > > Thanks in advance! > > Kind regards. > > Chris > > _______________________________________________Gluster-users mailing > listGluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181210/6c4dfd81/attachment.html>