lejeczek
2020-May-29 22:09 UTC
[Gluster-users] Gfid mismatch detected - but no split brain - how to solve?
hi Guys I'm seeing "Gfid mismatch detected" in the logs but no split brain indicated (4-way replica) Brick swir-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME Status: Connected Total Number of entries: 22 Number of entries in heal pending: 22 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick whale-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME Status: Connected Total Number of entries: 22 Number of entries in heal pending: 22 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick rider-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick dzien:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME Status: Connected Total Number of entries: 10 Number of entries in heal pending: 10 Number of entries in split-brain: 0 Number of entries possibly healing: 0 On swir-ring8: ... The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:bea05f85-a20e-4cbf-9e41-5e351f99b396>/lock_file>, 37b2456f-5216-4679-ac5c-4908b24f895a on USER-HOME-client-15 and ba8f87ed-9bf3-404e-8d67-2631923e1645 on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:47:49.034935] and [2020-05-29 21:47:49.079480] The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:a8a084b9-6484-48be-bec7-e704cf5ab880>/t>, d7a4ed01-139b-4df3-8070-31bd620a6f15 on USER-HOME-client-15 and d794b6ba-2a1d-4043-bb31-b98b22692763 on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:47:49.126173] and [2020-05-29 21:47:49.155432] The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:1f686fba-e325-4539-bf48-94b06325d2f4>/Tables.docx>, 344febd8-c89c-4bf3-8ad8-6494c2189c43 on USER-HOME-client-15 and 48d5b12b-03f4-46bf-bed1-9f8f88815615 on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:47:49.194061] and [2020-05-29 21:47:49.239896] The message "E [MSGID: 108008] [afr-self-heal-entry.c:257:afr_selfheal_detect_gfid_and_type_mismatch] 0-USER-HOME-replicate-0: Skipping conservative merge on the file." repeated 8 times between [2020-05-29 21:47:49.037812] and [2020-05-29 21:47:49.240423] ... On whale-ring8: ... The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/pcs>, a83d0e5f-ef3a-40ab-be7b-784538d150be on USER-HOME-client-15 and 89af3d31-81fa-4242-b8f7-0f49fd5fe57b on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:45:46.152052] and [2020-05-29 21:45:46.422393] The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/history_database>, 81ebb0d5-264a-4eba-984a-e18673b43826 on USER-HOME-client-15 and 2498a303-8937-43c3-939e-5e1d786b07fa on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:45:46.167704] and [2020-05-29 21:45:46.437702] The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/client-state>, fe86c057-c74d-417f-9c2c-6e6eb9778851 on USER-HOME-client-15 and a66f2714-c2a0-4bdc-8786-ad5b93e0e988 on USER-HOME-client-13." repeated 2 times between [2020-05-29 21:45:46.144242] and [2020-05-29 21:45:46.442526] The message "E [MSGID: 108008] [afr-self-heal-common.c:384:afr_gfid_split_brain_source] 0-USER-HOME-replicate-0: Gfid mismatch detected for <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/history_database.1>, 9826d8ad-fecc-4dd7-bc1f-87d0eff23d73 on USER-HOME-client-15 and 81ebb0d5-264a-4eba-984a-e18673b43826 on USER-HOME-client-13." repeated 3 times between [2020-05-29 21:45:46.162016] and [2020-05-29 21:45:46.476935] ... On rider-ring8: ... 2020-05-29 21:46:53.122929] E [MSGID: 114031] [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] 0-QEMU_VMs-client-3: remote operation failed. Path: <gfid:6f01098f-e8db-4f63-a661-86b4d02d937f> (6f01098f-e8db-4f63-a661-86b4d02d937f) [Permission denied] [2020-05-29 21:46:53.124148] E [MSGID: 114031] [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] 0-QEMU_VMs-client-4: remote operation failed. Path: <gfid:6f01098f-e8db-4f63-a661-86b4d02d937f> (6f01098f-e8db-4f63-a661-86b4d02d937f) [Permission denied] [2020-05-29 21:46:53.133566] I [MSGID: 108026] [afr-self-heal-entry.c:898:afr_selfheal_entry_do] 0-QEMU_VMs-replicate-0: performing entry selfheal on e0121f76-2452-44dc-b1a6-82b46cc9ec79 [2020-05-29 21:46:53.145991] E [MSGID: 114031] [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] 0-QEMU_VMs-client-3: remote operation failed. Path: <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1> (3f0239ac-e027-4a0c-b271-431e76ad97b1) [Permission denied] [2020-05-29 21:46:53.147110] E [MSGID: 114031] [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] 0-QEMU_VMs-client-4: remote operation failed. Path: <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1> (3f0239ac-e027-4a0c-b271-431e76ad97b1) [Permission denied] The most recent data I'm 100% certain is on rider-ring8. Any expert could probably quickly suggest how to go about the problem. Would something like manual healing with 'source-brick' or 'latest-mtime' work or is too naive? many thanks, L.
Karthik Subrahmanya
2020-Jun-01 06:24 UTC
[Gluster-users] Gfid mismatch detected - but no split brain - how to solve?
Hi, I am assuming that you are using one of the maintained versions of gluster. GFID split-brains can be resolved using one of the methods in the split-brain resolution CLI as explained in the section "3. Resolution of split-brain using gluster CLI" of https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/. The things to be noted here while using this CLI for resolving gfid split-brains are: - You can not use the GFID of the file as an argument with any of the CLI options to resolve GFID split-brain. It should be the absolute path as seen from the mount point to the file considered as source. - With source-brick option there is no way to resolve all the GFID split-brain in one shot by not specifying any file-path in the CLI as done while resolving data or metadata split-brain. For each file in GFID split-brain, run the CLI with the policy you want to use. - Resolving directory GFID split-brain using CLI with the "source-brick" option in a "distributed-replicated" volume needs to be done on all the volumes explicitly if the file is in gfid split-brain on multiple subvolumes. Since directories get created on all the subvolumes, using one particular brick as source for directory GFID split-brain, heal the directories for that subvolume. In this case, other subvolumes must be healed using the brick which has the same GFID as that of the previous brick which was used as source for healing other subvolume. Regards, Karthik On Sat, May 30, 2020 at 3:39 AM lejeczek <peljasz at yahoo.co.uk> wrote:> hi Guys > > I'm seeing "Gfid mismatch detected" in the logs but no split > brain indicated (4-way replica) > > Brick > swir-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME > Status: Connected > Total Number of entries: 22 > Number of entries in heal pending: 22 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick > whale-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME > Status: Connected > Total Number of entries: 22 > Number of entries in heal pending: 22 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick > rider-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick dzien:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME > Status: Connected > Total Number of entries: 10 > Number of entries in heal pending: 10 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > On swir-ring8: > ... > The message "E [MSGID: 108008] > [afr-self-heal-common.c:384:afr_gfid_split_brain_source] > 0-USER-HOME-replicate-0: Gfid mismatch detected for > <gfid:bea05f85-a20e-4cbf-9e41-5e351f99b396>/lock_file>, > 37b2456f-5216-4679-ac5c-4908b24f895a on USER-HOME-client-15 > and ba8f87ed-9bf3-404e-8d67-2631923e1645 on > USER-HOME-client-13." repeated 2 times between [2020-05-29 > 21:47:49.034935] and [2020-05-29 21:47:49.079480] > The message "E [MSGID: 108008] > [afr-self-heal-common.c:384:afr_gfid_split_brain_source] > 0-USER-HOME-replicate-0: Gfid mismatch detected for > <gfid:a8a084b9-6484-48be-bec7-e704cf5ab880>/t>, > d7a4ed01-139b-4df3-8070-31bd620a6f15 on USER-HOME-client-15 > and d794b6ba-2a1d-4043-bb31-b98b22692763 on > USER-HOME-client-13." repeated 2 times between [2020-05-29 > 21:47:49.126173] and [2020-05-29 21:47:49.155432] > The message "E [MSGID: 108008] > [afr-self-heal-common.c:384:afr_gfid_split_brain_source] > 0-USER-HOME-replicate-0: Gfid mismatch detected for > <gfid:1f686fba-e325-4539-bf48-94b06325d2f4>/Tables.docx>, > 344febd8-c89c-4bf3-8ad8-6494c2189c43 on USER-HOME-client-15 > and 48d5b12b-03f4-46bf-bed1-9f8f88815615 on > USER-HOME-client-13." repeated 2 times between [2020-05-29 > 21:47:49.194061] and [2020-05-29 21:47:49.239896] > The message "E [MSGID: 108008] > [afr-self-heal-entry.c:257:afr_selfheal_detect_gfid_and_type_mismatch] > 0-USER-HOME-replicate-0: Skipping conservative merge on the > file." repeated 8 times between [2020-05-29 21:47:49.037812] > and [2020-05-29 21:47:49.240423] > ... > > On whale-ring8: > ... > The message "E [MSGID: 108008] > [afr-self-heal-common.c:384:afr_gfid_split_brain_source] > 0-USER-HOME-replicate-0: Gfid mismatch detected for > <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/pcs>, > a83d0e5f-ef3a-40ab-be7b-784538d150be on USER-HOME-client-15 > and 89af3d31-81fa-4242-b8f7-0f49fd5fe57b on > USER-HOME-client-13." repeated 2 times between [2020-05-29 > 21:45:46.152052] and [2020-05-29 21:45:46.422393] > The message "E [MSGID: 108008] > [afr-self-heal-common.c:384:afr_gfid_split_brain_source] > 0-USER-HOME-replicate-0: Gfid mismatch detected for > <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/history_database>, > 81ebb0d5-264a-4eba-984a-e18673b43826 on USER-HOME-client-15 > and 2498a303-8937-43c3-939e-5e1d786b07fa on > USER-HOME-client-13." repeated 2 times between [2020-05-29 > 21:45:46.167704] and [2020-05-29 21:45:46.437702] > The message "E [MSGID: 108008] > [afr-self-heal-common.c:384:afr_gfid_split_brain_source] > 0-USER-HOME-replicate-0: Gfid mismatch detected for > <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/client-state>, > fe86c057-c74d-417f-9c2c-6e6eb9778851 on USER-HOME-client-15 > and a66f2714-c2a0-4bdc-8786-ad5b93e0e988 on > USER-HOME-client-13." repeated 2 times between [2020-05-29 > 21:45:46.144242] and [2020-05-29 21:45:46.442526] > The message "E [MSGID: 108008] > [afr-self-heal-common.c:384:afr_gfid_split_brain_source] > 0-USER-HOME-replicate-0: Gfid mismatch detected for > <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/history_database.1>, > 9826d8ad-fecc-4dd7-bc1f-87d0eff23d73 on USER-HOME-client-15 > and 81ebb0d5-264a-4eba-984a-e18673b43826 on > USER-HOME-client-13." repeated 3 times between [2020-05-29 > 21:45:46.162016] and [2020-05-29 21:45:46.476935] > ... > > On rider-ring8: > ... > 2020-05-29 21:46:53.122929] E [MSGID: 114031] > [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] > 0-QEMU_VMs-client-3: remote operation failed. Path: > <gfid:6f01098f-e8db-4f63-a661-86b4d02d937f> > (6f01098f-e8db-4f63-a661-86b4d02d937f) [Permission denied] > [2020-05-29 21:46:53.124148] E [MSGID: 114031] > [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] > 0-QEMU_VMs-client-4: remote operation failed. Path: > <gfid:6f01098f-e8db-4f63-a661-86b4d02d937f> > (6f01098f-e8db-4f63-a661-86b4d02d937f) [Permission denied] > [2020-05-29 21:46:53.133566] I [MSGID: 108026] > [afr-self-heal-entry.c:898:afr_selfheal_entry_do] > 0-QEMU_VMs-replicate-0: performing entry selfheal on > e0121f76-2452-44dc-b1a6-82b46cc9ec79 > [2020-05-29 21:46:53.145991] E [MSGID: 114031] > [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] > 0-QEMU_VMs-client-3: remote operation failed. Path: > <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1> > (3f0239ac-e027-4a0c-b271-431e76ad97b1) [Permission denied] > [2020-05-29 21:46:53.147110] E [MSGID: 114031] > [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk] > 0-QEMU_VMs-client-4: remote operation failed. Path: > <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1> > (3f0239ac-e027-4a0c-b271-431e76ad97b1) [Permission denied] > > The most recent data I'm 100% certain is on rider-ring8. > Any expert could probably quickly suggest how to go about > the problem. > Would something like manual healing with 'source-brick' or > 'latest-mtime' work or is too naive? > > many thanks, L. > > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200601/6319800d/attachment.html>