Atin Mukherjee
2018-Dec-10 11:36 UTC
[Gluster-users] glusterd keeps resyncing shards over and over again
Even though the subject says the issue is with glusterd, I think the question is more applicable on heal/shards. Added the relevant folks to help out. On Mon, Dec 10, 2018 at 3:43 PM Chris Drescher <info at linuxfabrik.ch> wrote:> Let me provide more information. > > We have 3 gluster nodes running with sharding activated. > > Node1: CentOS 7.5 - Glusterfs 3.12.6 > Node2: CentOS 7.5 - Glusterfs 3.12.6 > Node3: CentOS 7.5 - Glusterfs 3.12.6 > > Now we updated Node 3 from CentOS 7.5 to 7.6 which caused a reboot. > Glusterd Version changed from 3.12.6 to 3.12.15 > > Node1: CentOS 7.5 - Glusterfs 3.12.6 > Node2: CentOS 7.5 - Glusterfs 3.12.6 > Node3: CentOS 7.6 - Glusterfs 3.12.15 > > Afterwards gluster heal daemon keeps resyncing specific shards on bricks > on Node1 and Node2. Always the same shards. > > LOGS: > > On upgraded NODE3: > > /var/log/glusterfs/glusterd.log > [2018-12-10 09:24:42.314624] E [MSGID: 106062] > [glusterd-utils.c:10112:glusterd_max_opversion_use_rsp_dict] 0-management: > Maximum supported op-version not set in destination dictionary > > tail -f /var/log/glusterfs/glustershd.log > [2018-12-09 04:28:05.687127] I [MSGID: 108026] > [afr-self-heal-common.c:1726:afr_log_selfheal] 0-data-replicate-0: > Completed data selfheal on 3f1711c2-de8c-4e8e-be10-a252f5b1b4ad. > sources=[0] 2 sinks=1 > > > On NODE1: > > tail -f /var/log/glusterfs/glfsheal-data.log > [2018-12-10 10:00:01.898139] I [MSGID: 114035] > [client-handshake.c:202:client_set_lk_version_cbk] 0-data-client-16: Server > lk version = 1 > [2018-12-10 10:00:01.898487] I [MSGID: 114057] > [client-handshake.c:1478:select_server_supported_programs] > 0-data-client-17: Using Program GlusterFS 3.3, Num (1298437), Version > (330) > [2018-12-10 10:00:01.898892] I [MSGID: 114046] > [client-handshake.c:1231:client_setvolume_cbk] 0-data-client-17: Connected > to data-client-17, attached to remote volume > '/gluster/arb2/data'. > [2018-12-10 10:00:01.898900] I [MSGID: 114047] > [client-handshake.c:1242:client_setvolume_cbk] 0-data-client-17: Server and > Client lk-version numbers are not same, reopening the > fds > [2018-12-10 10:00:01.899007] I [MSGID: 114035] > [client-handshake.c:202:client_set_lk_version_cbk] 0-data-client-17: Server > lk version = 1 > [2018-12-10 10:00:01.901528] I [MSGID: 108031] > [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-3: selecting > local read_child data-client-9 > [2018-12-10 10:00:01.901876] I [MSGID: 108031] > [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-5: selecting > local read_child data-client-15 > [2018-12-10 10:00:01.901978] I [MSGID: 108031] > [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-4: selecting > local read_child data-client-12 > [2018-12-10 10:00:01.902708] I [MSGID: 108031] > [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-2: selecting > local read_child data-client-6 > [2018-12-10 10:00:01.902750] I [MSGID: 104041] > [glfs-resolve.c:971:__glfs_active_subvol] 0-data: switched to graph > 70312d70-6f64-3031-2e6c-696e75786661 (0) > > Hope that helps! > > On Mon, 2018-12-10 at 09:22 +0100, Chris Drescher wrote: > > Hello everybody > > We are experiencing an urgent issue with glusterd! > After an upgrade from centos7.5 to 7.6 our grusterfs keeps resyncing > specific shards over and over again! > > Is this a known problem? > > This is very urgent! Please help! > > Thanks in advance! > > Kind regards. > > Chris > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181210/15105364/attachment.html>
Ravishankar N
2018-Dec-10 12:03 UTC
[Gluster-users] glusterd keeps resyncing shards over and over again
On 12/10/2018 05:06 PM, Atin Mukherjee wrote:> Even though the subject says the issue is with glusterd, I think the > question is more applicable on heal/shards. Added the relevant folks > to help out. > > On Mon, Dec 10, 2018 at 3:43 PM Chris Drescher <info at linuxfabrik.ch > <mailto:info at linuxfabrik.ch>> wrote: > > Let me provide more information. > > We have 3 gluster nodes running with sharding activated. > > Node1: CentOS 7.5 - Glusterfs 3.12.6 > Node2: CentOS 7.5 - Glusterfs 3.12.6 > Node3: CentOS 7.5 - Glusterfs 3.12.6 > > Now we updated Node 3 from CentOS 7.5 to 7.6 which caused a reboot. > Glusterd Version changed from 3.12.6?to 3.12.15 > > Node1: CentOS 7.5 - Glusterfs 3.12.6 > Node2: CentOS 7.5 - Glusterfs 3.12.6 > Node3: CentOS 7.6 - Glusterfs 3.12.15 > > Afterwards gluster heal daemon keeps resyncing specific shards on > bricks on Node1 and Node2. Always the same shards. >Your clients (mounts)? might be experiencing disconnects from the brick process(es) while the same set of shards are being written to. Possibly to the second brick, going from the "sinks=1 " log below. Check if that is the case. -Ravi> > > LOGS: > > On upgraded NODE3: > > /var/log/glusterfs/glusterd.log > [2018-12-10 09:24:42.314624] E [MSGID: 106062] > [glusterd-utils.c:10112:glusterd_max_opversion_use_rsp_dict] > 0-management: Maximum supported op-version not set in destination > dictionary > > tail -f /var/log/glusterfs/glustershd.log > [2018-12-09 04:28:05.687127] I [MSGID: 108026] > [afr-self-heal-common.c:1726:afr_log_selfheal] 0-data-replicate-0: > Completed data selfheal on 3f1711c2-de8c-4e8e-be10-a252f5b1b4ad. > sources=[0] 2??sinks=1 > > > On NODE1: > > tail -f /var/log/glusterfs/glfsheal-data.log > [2018-12-10 10:00:01.898139] I [MSGID: 114035] > [client-handshake.c:202:client_set_lk_version_cbk] > 0-data-client-16: Server lk version = 1 > [2018-12-10 10:00:01.898487] I [MSGID: 114057] > [client-handshake.c:1478:select_server_supported_programs] > 0-data-client-17: Using Program GlusterFS 3.3, Num (1298437), > Version (330) > [2018-12-10 10:00:01.898892] I [MSGID: 114046] > [client-handshake.c:1231:client_setvolume_cbk] 0-data-client-17: > Connected to data-client-17, attached to remote volume > '/gluster/arb2/data'. > [2018-12-10 10:00:01.898900] I [MSGID: 114047] > [client-handshake.c:1242:client_setvolume_cbk] 0-data-client-17: > Server and Client lk-version numbers are not same, reopening the fds > [2018-12-10 10:00:01.899007] I [MSGID: 114035] > [client-handshake.c:202:client_set_lk_version_cbk] > 0-data-client-17: Server lk version = 1 > [2018-12-10 10:00:01.901528] I [MSGID: 108031] > [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-3: > selecting local read_child data-client-9 > [2018-12-10 10:00:01.901876] I [MSGID: 108031] > [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-5: > selecting local read_child data-client-15 > [2018-12-10 10:00:01.901978] I [MSGID: 108031] > [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-4: > selecting local read_child data-client-12 > [2018-12-10 10:00:01.902708] I [MSGID: 108031] > [afr-common.c:2376:afr_local_discovery_cbk] 0-data-replicate-2: > selecting local read_child data-client-6 > [2018-12-10 10:00:01.902750] I [MSGID: 104041] > [glfs-resolve.c:971:__glfs_active_subvol] 0-data: switched to > graph 70312d70-6f64-3031-2e6c-696e75786661 (0) > > Hope that helps! > > On Mon, 2018-12-10 at 09:22 +0100, Chris Drescher wrote: >> Hello everybody >> >> We are experiencing an urgent issue with glusterd! >> After an upgrade from centos7.5 to 7.6 our grusterfs keeps >> resyncing specific shards over and over again! >> >> Is this a known problem? >> >> This is very urgent! Please help! >> >> Thanks in advance! >> >> Kind regards. >> >> Chris >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >> https://lists.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > https://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181210/3a0d65ae/attachment.html>