thr3ads.net - Gluster users - [Gluster-users] Gfid mismatch detected - but no split brain

If this information is useful, please help other people find it:
Share via:

lejeczek

2020-May-29 22:09 UTC

[Gluster-users] Gfid mismatch detected - but no split brain - how to solve?

hi Guys

I'm seeing "Gfid mismatch detected" in the logs but no split
brain indicated (4-way replica)

Brick
swir-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME
Status: Connected
Total Number of entries: 22
Number of entries in heal pending: 22
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick
whale-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME
Status: Connected
Total Number of entries: 22
Number of entries in heal pending: 22
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick
rider-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick dzien:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME
Status: Connected
Total Number of entries: 10
Number of entries in heal pending: 10
Number of entries in split-brain: 0
Number of entries possibly healing: 0

On swir-ring8:
...
The message "E [MSGID: 108008]
[afr-self-heal-common.c:384:afr_gfid_split_brain_source]
0-USER-HOME-replicate-0: Gfid mismatch detected for
<gfid:bea05f85-a20e-4cbf-9e41-5e351f99b396>/lock_file>,
37b2456f-5216-4679-ac5c-4908b24f895a on USER-HOME-client-15
and ba8f87ed-9bf3-404e-8d67-2631923e1645 on
USER-HOME-client-13." repeated 2 times between [2020-05-29
21:47:49.034935] and [2020-05-29 21:47:49.079480]
The message "E [MSGID: 108008]
[afr-self-heal-common.c:384:afr_gfid_split_brain_source]
0-USER-HOME-replicate-0: Gfid mismatch detected for
<gfid:a8a084b9-6484-48be-bec7-e704cf5ab880>/t>,
d7a4ed01-139b-4df3-8070-31bd620a6f15 on USER-HOME-client-15
and d794b6ba-2a1d-4043-bb31-b98b22692763 on
USER-HOME-client-13." repeated 2 times between [2020-05-29
21:47:49.126173] and [2020-05-29 21:47:49.155432]
The message "E [MSGID: 108008]
[afr-self-heal-common.c:384:afr_gfid_split_brain_source]
0-USER-HOME-replicate-0: Gfid mismatch detected for
<gfid:1f686fba-e325-4539-bf48-94b06325d2f4>/Tables.docx>,
344febd8-c89c-4bf3-8ad8-6494c2189c43 on USER-HOME-client-15
and 48d5b12b-03f4-46bf-bed1-9f8f88815615 on
USER-HOME-client-13." repeated 2 times between [2020-05-29
21:47:49.194061] and [2020-05-29 21:47:49.239896]
The message "E [MSGID: 108008]
[afr-self-heal-entry.c:257:afr_selfheal_detect_gfid_and_type_mismatch]
0-USER-HOME-replicate-0: Skipping conservative merge on the
file." repeated 8 times between [2020-05-29 21:47:49.037812]
and [2020-05-29 21:47:49.240423]
...

On whale-ring8:
...
The message "E [MSGID: 108008]
[afr-self-heal-common.c:384:afr_gfid_split_brain_source]
0-USER-HOME-replicate-0: Gfid mismatch detected for
<gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/pcs>,
a83d0e5f-ef3a-40ab-be7b-784538d150be on USER-HOME-client-15
and 89af3d31-81fa-4242-b8f7-0f49fd5fe57b on
USER-HOME-client-13." repeated 2 times between [2020-05-29
21:45:46.152052] and [2020-05-29 21:45:46.422393]
The message "E [MSGID: 108008]
[afr-self-heal-common.c:384:afr_gfid_split_brain_source]
0-USER-HOME-replicate-0: Gfid mismatch detected for
<gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/history_database>,
81ebb0d5-264a-4eba-984a-e18673b43826 on USER-HOME-client-15
and 2498a303-8937-43c3-939e-5e1d786b07fa on
USER-HOME-client-13." repeated 2 times between [2020-05-29
21:45:46.167704] and [2020-05-29 21:45:46.437702]
The message "E [MSGID: 108008]
[afr-self-heal-common.c:384:afr_gfid_split_brain_source]
0-USER-HOME-replicate-0: Gfid mismatch detected for
<gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/client-state>,
fe86c057-c74d-417f-9c2c-6e6eb9778851 on USER-HOME-client-15
and a66f2714-c2a0-4bdc-8786-ad5b93e0e988 on
USER-HOME-client-13." repeated 2 times between [2020-05-29
21:45:46.144242] and [2020-05-29 21:45:46.442526]
The message "E [MSGID: 108008]
[afr-self-heal-common.c:384:afr_gfid_split_brain_source]
0-USER-HOME-replicate-0: Gfid mismatch detected for
<gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/history_database.1>,
9826d8ad-fecc-4dd7-bc1f-87d0eff23d73 on USER-HOME-client-15
and 81ebb0d5-264a-4eba-984a-e18673b43826 on
USER-HOME-client-13." repeated 3 times between [2020-05-29
21:45:46.162016] and [2020-05-29 21:45:46.476935]
...

On rider-ring8:
...
2020-05-29 21:46:53.122929] E [MSGID: 114031]
[client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk]
0-QEMU_VMs-client-3: remote operation failed. Path:
<gfid:6f01098f-e8db-4f63-a661-86b4d02d937f>
(6f01098f-e8db-4f63-a661-86b4d02d937f) [Permission denied]
[2020-05-29 21:46:53.124148] E [MSGID: 114031]
[client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk]
0-QEMU_VMs-client-4: remote operation failed. Path:
<gfid:6f01098f-e8db-4f63-a661-86b4d02d937f>
(6f01098f-e8db-4f63-a661-86b4d02d937f) [Permission denied]
[2020-05-29 21:46:53.133566] I [MSGID: 108026]
[afr-self-heal-entry.c:898:afr_selfheal_entry_do]
0-QEMU_VMs-replicate-0: performing entry selfheal on
e0121f76-2452-44dc-b1a6-82b46cc9ec79
[2020-05-29 21:46:53.145991] E [MSGID: 114031]
[client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk]
0-QEMU_VMs-client-3: remote operation failed. Path:
<gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1>
(3f0239ac-e027-4a0c-b271-431e76ad97b1) [Permission denied]
[2020-05-29 21:46:53.147110] E [MSGID: 114031]
[client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk]
0-QEMU_VMs-client-4: remote operation failed. Path:
<gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1>
(3f0239ac-e027-4a0c-b271-431e76ad97b1) [Permission denied]

The most recent data I'm 100% certain is on rider-ring8.
Any expert could probably quickly suggest how to go about
the problem.
Would something like manual healing with 'source-brick' or
'latest-mtime' work or is too naive?

many thanks, L.

Karthik Subrahmanya

2020-Jun-01 06:24 UTC

head link

[Gluster-users] Gfid mismatch detected - but no split brain - how to solve?

Hi,

I am assuming that you are using one of the maintained versions of gluster.

GFID split-brains can be resolved using one of the methods in the
split-brain resolution CLI as explained in the section "3. Resolution of
split-brain using gluster CLI" of
https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/.

The things to be noted here while using this CLI for resolving gfid
split-brains are:
- You can not use the GFID of the file as an argument with any of the CLI
options to resolve GFID split-brain. It should be the absolute path as seen
from the mount point to the file considered as source.
- With source-brick option there is no way to resolve all the GFID
split-brain in one shot by not specifying any file-path in the CLI as done
while resolving data or metadata split-brain. For each file in GFID
split-brain, run the CLI with the policy you want to use.
- Resolving directory GFID split-brain using CLI with the
"source-brick"
option in a "distributed-replicated" volume needs to be done on all
the
volumes explicitly if the file is in gfid split-brain on multiple
subvolumes. Since directories get created on all the subvolumes, using one
particular brick as source for directory GFID split-brain, heal the
directories for that subvolume. In this case, other subvolumes must be
healed using the brick which has the same GFID as that of the previous
brick which was used as source for healing other subvolume.
Regards,
Karthik

On Sat, May 30, 2020 at 3:39 AM lejeczek <peljasz at yahoo.co.uk> wrote:
> hi Guys
>
> I'm seeing "Gfid mismatch detected" in the logs but no split
> brain indicated (4-way replica)
>
> Brick
> swir-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME
> Status: Connected
> Total Number of entries: 22
> Number of entries in heal pending: 22
> Number of entries in split-brain: 0
> Number of entries possibly healing: 0
>
> Brick
> whale-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME
> Status: Connected
> Total Number of entries: 22
> Number of entries in heal pending: 22
> Number of entries in split-brain: 0
> Number of entries possibly healing: 0
>
> Brick
> rider-ring8:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME
> Status: Connected
> Total Number of entries: 0
> Number of entries in heal pending: 0
> Number of entries in split-brain: 0
> Number of entries possibly healing: 0
>
> Brick dzien:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.USER-HOME
> Status: Connected
> Total Number of entries: 10
> Number of entries in heal pending: 10
> Number of entries in split-brain: 0
> Number of entries possibly healing: 0
>
> On swir-ring8:
> ...
> The message "E [MSGID: 108008]
> [afr-self-heal-common.c:384:afr_gfid_split_brain_source]
> 0-USER-HOME-replicate-0: Gfid mismatch detected for
> <gfid:bea05f85-a20e-4cbf-9e41-5e351f99b396>/lock_file>,
> 37b2456f-5216-4679-ac5c-4908b24f895a on USER-HOME-client-15
> and ba8f87ed-9bf3-404e-8d67-2631923e1645 on
> USER-HOME-client-13." repeated 2 times between [2020-05-29
> 21:47:49.034935] and [2020-05-29 21:47:49.079480]
> The message "E [MSGID: 108008]
> [afr-self-heal-common.c:384:afr_gfid_split_brain_source]
> 0-USER-HOME-replicate-0: Gfid mismatch detected for
> <gfid:a8a084b9-6484-48be-bec7-e704cf5ab880>/t>,
> d7a4ed01-139b-4df3-8070-31bd620a6f15 on USER-HOME-client-15
> and d794b6ba-2a1d-4043-bb31-b98b22692763 on
> USER-HOME-client-13." repeated 2 times between [2020-05-29
> 21:47:49.126173] and [2020-05-29 21:47:49.155432]
> The message "E [MSGID: 108008]
> [afr-self-heal-common.c:384:afr_gfid_split_brain_source]
> 0-USER-HOME-replicate-0: Gfid mismatch detected for
> <gfid:1f686fba-e325-4539-bf48-94b06325d2f4>/Tables.docx>,
> 344febd8-c89c-4bf3-8ad8-6494c2189c43 on USER-HOME-client-15
> and 48d5b12b-03f4-46bf-bed1-9f8f88815615 on
> USER-HOME-client-13." repeated 2 times between [2020-05-29
> 21:47:49.194061] and [2020-05-29 21:47:49.239896]
> The message "E [MSGID: 108008]
> [afr-self-heal-entry.c:257:afr_selfheal_detect_gfid_and_type_mismatch]
> 0-USER-HOME-replicate-0: Skipping conservative merge on the
> file." repeated 8 times between [2020-05-29 21:47:49.037812]
> and [2020-05-29 21:47:49.240423]
> ...
>
> On whale-ring8:
> ...
> The message "E [MSGID: 108008]
> [afr-self-heal-common.c:384:afr_gfid_split_brain_source]
> 0-USER-HOME-replicate-0: Gfid mismatch detected for
> <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/pcs>,
> a83d0e5f-ef3a-40ab-be7b-784538d150be on USER-HOME-client-15
> and 89af3d31-81fa-4242-b8f7-0f49fd5fe57b on
> USER-HOME-client-13." repeated 2 times between [2020-05-29
> 21:45:46.152052] and [2020-05-29 21:45:46.422393]
> The message "E [MSGID: 108008]
> [afr-self-heal-common.c:384:afr_gfid_split_brain_source]
> 0-USER-HOME-replicate-0: Gfid mismatch detected for
> <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/history_database>,
> 81ebb0d5-264a-4eba-984a-e18673b43826 on USER-HOME-client-15
> and 2498a303-8937-43c3-939e-5e1d786b07fa on
> USER-HOME-client-13." repeated 2 times between [2020-05-29
> 21:45:46.167704] and [2020-05-29 21:45:46.437702]
> The message "E [MSGID: 108008]
> [afr-self-heal-common.c:384:afr_gfid_split_brain_source]
> 0-USER-HOME-replicate-0: Gfid mismatch detected for
> <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/client-state>,
> fe86c057-c74d-417f-9c2c-6e6eb9778851 on USER-HOME-client-15
> and a66f2714-c2a0-4bdc-8786-ad5b93e0e988 on
> USER-HOME-client-13." repeated 2 times between [2020-05-29
> 21:45:46.144242] and [2020-05-29 21:45:46.442526]
> The message "E [MSGID: 108008]
> [afr-self-heal-common.c:384:afr_gfid_split_brain_source]
> 0-USER-HOME-replicate-0: Gfid mismatch detected for
> <gfid:e57771ac-c18d-44fe-81ff-6e9072d9c9ef>/history_database.1>,
> 9826d8ad-fecc-4dd7-bc1f-87d0eff23d73 on USER-HOME-client-15
> and 81ebb0d5-264a-4eba-984a-e18673b43826 on
> USER-HOME-client-13." repeated 3 times between [2020-05-29
> 21:45:46.162016] and [2020-05-29 21:45:46.476935]
> ...
>
> On rider-ring8:
> ...
> 2020-05-29 21:46:53.122929] E [MSGID: 114031]
> [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk]
> 0-QEMU_VMs-client-3: remote operation failed. Path:
> <gfid:6f01098f-e8db-4f63-a661-86b4d02d937f>
> (6f01098f-e8db-4f63-a661-86b4d02d937f) [Permission denied]
> [2020-05-29 21:46:53.124148] E [MSGID: 114031]
> [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk]
> 0-QEMU_VMs-client-4: remote operation failed. Path:
> <gfid:6f01098f-e8db-4f63-a661-86b4d02d937f>
> (6f01098f-e8db-4f63-a661-86b4d02d937f) [Permission denied]
> [2020-05-29 21:46:53.133566] I [MSGID: 108026]
> [afr-self-heal-entry.c:898:afr_selfheal_entry_do]
> 0-QEMU_VMs-replicate-0: performing entry selfheal on
> e0121f76-2452-44dc-b1a6-82b46cc9ec79
> [2020-05-29 21:46:53.145991] E [MSGID: 114031]
> [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk]
> 0-QEMU_VMs-client-3: remote operation failed. Path:
> <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1>
> (3f0239ac-e027-4a0c-b271-431e76ad97b1) [Permission denied]
> [2020-05-29 21:46:53.147110] E [MSGID: 114031]
> [client-rpc-fops_v2.c:1548:client4_0_xattrop_cbk]
> 0-QEMU_VMs-client-4: remote operation failed. Path:
> <gfid:3f0239ac-e027-4a0c-b271-431e76ad97b1>
> (3f0239ac-e027-4a0c-b271-431e76ad97b1) [Permission denied]
>
> The most recent data I'm 100% certain is on rider-ring8.
> Any expert could probably quickly suggest how to go about
> the problem.
> Would something like manual healing with 'source-brick' or
> 'latest-mtime' work or is too naive?
>
> many thanks, L.
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200601/6319800d/attachment.html>

Gluster users - May 2020 - Gfid mismatch detected - but no split brain - how to solve?

[Gluster-users] Gfid mismatch detected - but no split brain - how to solve?

[Gluster-users] Gfid mismatch detected - but no split brain - how to solve?