Amudhan P
2019-Aug-26 10:12 UTC
[Gluster-users] same gfid for two directories and dht_layout_mismatch and failed to combine inode error msgs
Hi, I am facing an issue in my gluster setup. glusterfs version 3.10.1 distributed disperse volume glusterfs-fuse client. One of node SAS BackPlane had some problem so my drives are ejected from service, this caused the brick process to terminate but glusterd daemon was still running. gluster peer status showing faulty node as online but bricks in the faulty node is offline. In the meantime, client upload process was going on, cpu load and memory usage on other nodes keep on increasing. so, I restarted the problem node and started glusterd service after mounting all disks. now gluster status showing all brick is online. now the problem I am facing here is folders which are uploaded or moved between one path to another path when bringing back problem node to live is not listing files originally they had and I am not able to delete the folder even though I have permission, throwing error "permission denied" and "folder not empty" but doing "ls" of the showing empty folder. error msg in fuse mount log. [2019-08-26 07:33:13.005165] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644) [2019-08-26 07:33:13.005211] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 07:33:21.020529] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) [2019-08-26 07:34:59.005318] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 23 times between [2019-08-26 07:33:13.005165] and [2019-08-26 07:35:11.005147] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 25 times between [2019-08-26 07:33:13.005211] and [2019-08-26 07:35:11.005149] [2019-08-26 07:35:13.006967] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644) [2019-08-26 07:35:13.007028] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 07:35:37.005121] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)" repeated 2 times between [2019-08-26 07:35:37.005121] and [2019-08-26 07:36:21.012016] The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 21 times between [2019-08-26 07:35:13.006967] and [2019-08-26 07:37:11.005302] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 24 times between [2019-08-26 07:35:13.007028] and [2019-08-26 07:37:11.005305] [2019-08-26 07:37:13.006618] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) [2019-08-26 07:37:13.006721] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 07:37:21.024033] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644) [2019-08-26 07:38:22.026108] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-139: Failed to combine iatt (inode: 11541463450251411113-11541463450251411113, links: 1-1, uid: 1000-1000, gid: 1000-1000, rdev: 0-0, size: 517816320-517799936, mode: 100600-100600) [2019-08-26 07:38:22.026144] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-139: Mismatching iatt in answers of 'GF_FOP_LOOKUP' The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 19 times between [2019-08-26 07:37:21.024033] and [2019-08-26 07:39:10.005761] The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)" repeated 3 times between [2019-08-26 07:37:13.006618] and [2019-08-26 07:39:11.004587] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 23 times between [2019-08-26 07:37:13.006721] and [2019-08-26 07:39:11.004591] [2019-08-26 07:39:13.007257] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644) [2019-08-26 07:39:13.007335] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 07:39:39.004926] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)" repeated 2 times between [2019-08-26 07:39:39.004926] and [2019-08-26 07:40:11.004852] The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 23 times between [2019-08-26 07:39:13.007257] and [2019-08-26 07:41:13.006759] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 26 times between [2019-08-26 07:39:13.007335] and [2019-08-26 07:41:13.006761] [2019-08-26 07:41:21.010830] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) [2019-08-26 07:41:21.010861] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 4 times between [2019-08-26 07:18:22.003323] and [2019-08-26 07:18:42.019306] [2019-08-26 07:18:47.003349] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) [2019-08-26 07:18:47.003392] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-137: Failed to combine iatt (inode: 12704371770406674620-12704371770406674620, links: 1-1, uid: 1000-1000, gid: 1000-1000, rdev: 0-0, size: 696369152-696352768, mode: 100600-100600)" repeated 2 times between [2019-08-26 07:17:11.019084] and [2019-08-26 07:17:11.019201] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-137: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 2 times between [2019-08-26 07:17:11.019111] and [2019-08-26 07:17:11.019202] The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 19 times between [2019-08-26 07:17:08.003143] and [2019-08-26 07:18:50.003418] The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'" repeated 4 times between [2019-08-26 07:18:47.003392] and [2019-08-26 07:18:50.003420] [2019-08-26 07:19:08.002986] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644) [2019-08-26 07:19:08.003012] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 07:19:39.011760] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644) [2019-08-26 09:21:53.015705] W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine] 0-qubevaultdr-disperse-139: Failed to combine iatt (inode: 11196349995399397487-11196349995399397487, links: 1-1, uid: 1000-1000, gid: 1000-1000, rdev: 0-0, size: 1177141248-1177124864, mode: 100600-100600) [2019-08-26 09:21:53.015747] N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup] 0-qubevaultdr-disperse-139: Mismatching iatt in answers of 'GF_FOP_LOOKUP' [2019-08-26 04:49:11.580839] I [MSGID: 109064] [dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol: glusterdr-disperse-135; inode layout - 3251903722 - 3282582058 - 1; disk layout - 0 - 0 - 0 [2019-08-26 04:49:11.580850] I [MSGID: 109018] [dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000 [2019-08-26 04:49:11.580986] I [MSGID: 109064] [dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol: glusterdr-disperse-136; inode layout - 3282582059 - 3313260395 - 1; disk layout - 0 - 1073741822 - 1 [2019-08-26 04:49:11.580997] I [MSGID: 109018] [dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000 [2019-08-26 04:49:11.581026] I [MSGID: 109064] [dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol: glusterdr-disperse-137; inode layout - 3313260396 - 3343938732 - 1; disk layout - 1073741823 - 2147483645 - 1 [2019-08-26 04:49:11.581038] I [MSGID: 109018] [dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000 [2019-08-26 04:49:11.581232] I [MSGID: 109064] [dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol: glusterdr-disperse-138; inode layout - 3343938733 - 3374617069 - 1; disk layout - 2147483646 - 3221225468 - 1 [2019-08-26 04:49:11.581243] I [MSGID: 109018] [dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000 [2019-08-26 04:49:11.581333] I [MSGID: 109064] [dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol: glusterdr-disperse-139; inode layout - 3374617070 - 3405295406 - 1; disk layout - 3221225469 - 4294967295 - 1 [2019-08-26 04:49:11.581350] I [MSGID: 109018] [dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000 error message in brick log. [2019-08-24 13:12:54.041605] W [MSGID: 113026] [posix.c:1542:posix_mkdir] 0-glusterdr-posix: mkdir (/Incoming/TempUpload/Weekly UPload/23-.8-19/1-2-19/client_folder2): gfid (b2bf1c5b-2be4-4074-be37-8118f1279d04) is already associated with directory (/media/disk13/brick13/.glusterfs/61/9a/619a5949-9001-4457-94c1-8f02c45996ba/client_folder2). Hence, both directories will share same gfid and this can lead to inconsistencies. regards Amudhan P -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190826/a28a6926/attachment.html>