thr3ads.net - Gluster users - [Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance [Jul 2020]

If this information is useful, please help other people find it:
Share via:

Shreyansh Shah

2020-Jul-02 14:39 UTC

[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance

Hi All,

*We are facing "Mismatching layouts for <directory>,gfid =
<id>" errors.*

We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each)
on each node, 7 nodes in total. We added new bricks yesterday to the
existing setup.
Post that we did a rebalance fix-layout and then a rebalance (which is
currently still in progress). The status shows "failed" on certain
bricks
but "in progress" for others. Adding output for gluster rebalance
status
below.

The glusterfs client logs are flooded with "Mismatching layouts for
<directory>,gfid = <id>"
The performance too seems to have degraded due to this, even basic commands
like `cd` and `ls` are taking more than a minute compared to sub-second
number before brick addition.
Apart from that we also experienced many binaries and files giving error
stale file handle error even though the files were present.


*gluster rebalance status :*

Node Rebalanced-files          size       scanned      failures
skipped               status  run time in h:m:s
---------      -----------   -----------   -----------   -----------
-----------         ------------     --------------
localhost              176         3.5GB         12790             0
   8552          in progress       21:36:01
10.132.0.72             8232       394.8GB         19995            21
       26               failed       14:50:30
10.132.0.44            12625         1.0TB         50023             1
    10202          in progress       21:36:00
10.132.0.3            21982       956.8GB         79145             1
  34571          in progress       21:36:00
10.132.0.9             7975       355.8GB         20157             6
   1522               failed       14:51:45
10.132.0.73             6293       394.5GB         26414           151
     8085               failed       14:51:45
10.132.0.70             6480       477.1GB         21058            27
     1787               failed       14:50:32
Estimated time left for rebalance to complete :      130:56:28


*Logs from one of the clients below:*

[2020-07-02 12:30:14.971916] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
layout - 2761060380 - 3067813815 - 4159036738
[2020-07-02 12:30:14.971935] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.032013] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
layout - 3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:15.032059] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.032107] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:15.032153] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.093329] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
layout - 2454306944 - 2761060379 - 4159036738
[2020-07-02 12:30:15.093373] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.093460] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
layout - 2761060380 - 3067813815 - 4159036738
[2020-07-02 12:30:15.093515] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.151063] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
layout - 3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:15.151108] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.151149] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:15.151162] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.424321] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
layout - 920400036 - 1227153471 - 4159036738
[2020-07-02 12:30:15.424380] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424456] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
layout - 1840730208 - 2147483643 - 4159036738
[2020-07-02 12:30:15.424484] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424525] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
layout - 1533976772 - 1840730207 - 4159036738
[2020-07-02 12:30:15.424542] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424596] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
layout - 613646600 - 920400035 - 4159036738
[2020-07-02 12:30:15.424607] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:16.004482] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
data-client-7 (hashed subvol is data-client-17)
[2020-07-02 12:30:16.005523] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_1_DATA.dat
[2020-07-02 12:30:16.531047] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
on data-client-9 (hashed subvol is data-client-19)
[2020-07-02 12:30:16.532086] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_1_METADATA.dat
[2020-07-02 12:30:18.733229] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
data-client-17 (hashed subvol is data-client-9)
[2020-07-02 12:30:18.734421] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_2_DATA.dat
[2020-07-02 12:30:19.171930] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
on data-client-9 (hashed subvol is data-client-18)
[2020-07-02 12:30:19.172901] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_2_METADATA.dat
[2020-07-02 12:30:21.028495] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
data-client-6 (hashed subvol is data-client-15)
[2020-07-02 12:30:21.029836] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_2_DATA.dat
[2020-07-02 12:30:21.127648] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
on data-client-11 (hashed subvol is data-client-3)
[2020-07-02 12:30:21.128713] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_2_METADATA.dat
[2020-07-02 12:30:21.201126] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
data-client-15 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.201928] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_3_DATA.dat
[2020-07-02 12:30:21.566158] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
on data-client-7 (hashed subvol is data-client-16)
[2020-07-02 12:30:21.567123] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_3_METADATA.dat
[2020-07-02 12:30:21.649357] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:21.661381] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_4_DATA.dat
[2020-07-02 12:30:21.748937] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
on data-client-15 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.749481] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_4_METADATA.dat
[2020-07-02 12:30:21.898593] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
data-client-14 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.899442] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_6_DATA.dat
[2020-07-02 12:30:22.039337] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
on data-client-10 (hashed subvol is data-client-2)
[2020-07-02 12:30:22.040086] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_6_METADATA.dat
[2020-07-02 12:30:22.501877] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
on data-client-15 (hashed subvol is data-client-8)
[2020-07-02 12:30:22.502712] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECDS1_DATA.dat
[2020-07-02 12:30:22.782577] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
(hashed subvol is data-client-6)
[2020-07-02 12:30:22.783777] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECDS1_METADATA.dat
[2020-07-02 12:30:23.146847] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
data-client-17 (hashed subvol is data-client-9)
[2020-07-02 12:30:23.148009] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM1_DATA.dat
[2020-07-02 12:30:23.229290] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
subvol is data-client-6)
[2020-07-02 12:30:23.230151] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM1_METADATA.dat
[2020-07-02 12:30:23.889520] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:23.896618] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM2_DATA.dat
[2020-07-02 12:30:24.093017] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
subvol is data-client-15)
[2020-07-02 12:30:24.094117] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM2_METADATA.dat
[2020-07-02 12:30:24.345257] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
data-client-17 (hashed subvol is data-client-10)
[2020-07-02 12:30:24.346234] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM3_DATA.dat
[2020-07-02 12:30:24.425835] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
subvol is data-client-15)
[2020-07-02 12:30:24.426880] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM3_METADATA.dat
[2020-07-02 12:30:25.158718] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
on data-client-9 (hashed subvol is data-client-19)
[2020-07-02 12:30:25.159619] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO1_DATA.dat
[2020-07-02 12:30:25.531479] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
subvol is data-client-10)
[2020-07-02 12:30:25.540569] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO1_METADATA.dat
[2020-07-02 12:30:25.771692] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
on data-client-11 (hashed subvol is data-client-3)
[2020-07-02 12:30:25.772610] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO2_DATA.dat
[2020-07-02 12:30:25.866118] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
(hashed subvol is data-client-8)
[2020-07-02 12:30:25.866917] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO2_METADATA.dat
[2020-07-02 12:30:26.424386] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
on data-client-9 (hashed subvol is data-client-18)
[2020-07-02 12:30:26.425309] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO3_DATA.dat
[2020-07-02 12:30:26.818852] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
(hashed subvol is data-client-2)
[2020-07-02 12:30:26.819890] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO3_METADATA.dat
[2020-07-02 12:30:27.352405] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
on data-client-10 (hashed subvol is data-client-2)
[2020-07-02 12:30:27.352914] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO4_DATA.dat
[2020-07-02 12:30:27.521286] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
subvol is data-client-18)
[2020-07-02 12:30:27.522325] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO4_METADATA.dat
[2020-07-02 12:30:28.566634] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
on data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:28.579295] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO5_DATA.dat
[2020-07-02 12:30:28.958028] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
on data-client-7 (hashed subvol is data-client-16)
[2020-07-02 12:30:28.959102] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO6_DATA.dat
[2020-07-02 12:30:29.012429] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
subvol is data-client-15)
[2020-07-02 12:30:29.013416] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO6_METADATA.dat
[2020-07-02 12:30:29.396716] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
data-client-17 (hashed subvol is data-client-10)
[2020-07-02 12:30:29.397740] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSEFO_BSE_TSDATA.dat
[2020-07-02 12:30:29.556312] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
subvol is data-client-18)
[2020-07-02 12:30:29.557197] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
[2020-07-02 12:30:30.605354] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
(hashed subvol is data-client-19)
[2020-07-02 12:30:30.606117] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
[2020-07-02 12:30:31.559206] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
613576736 - 920330171 - 4159036738
[2020-07-02 12:30:31.559255] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
[2020-07-02 12:30:31.569025] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
920330172 - 1227083607 - 4159036738
[2020-07-02 12:30:31.569067] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
[2020-07-02 12:30:31.701849] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:31.701895] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX, gfid fff324f2-f855-4881-b77c-81e856522373
[2020-07-02 12:30:31.738464] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:31.738507] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX, gfid fff324f2-f855-4881-b77c-81e856522373
[2020-07-02 12:30:31.857102] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
layout - 3067883680 - 3374637115 - 4159036738
[2020-07-02 12:30:31.857147] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.857180] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:31.857197] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.917705] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
- 306753435 - 4159036738
[2020-07-02 12:30:31.917781] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.917855] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
layout - 3988213852 - 4294967295 - 4159036738
[2020-07-02 12:30:31.917874] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:32.390945] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
3681460416 - 3988213851 - 4159036738
[2020-07-02 12:30:32.390998] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/NIFTY, gfid
b2d4deb7-c58c-4046-b6f2-7c7f44d71311
[2020-07-02 12:30:32.391056] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
3988213852 - 4294967295 - 4159036738
[2020-07-02 12:30:32.391075] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/NIFTY, gfid
b2d4deb7-c58c-4046-b6f2-7c7f44d71311
[2020-07-02 12:33:50.915279] I [MSGID: 109066]
[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
/raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
(2cb54500-814d-4e85-83e7-e33d9440b18d)
(hash=data-client-6/cache=data-client-18) =>
/raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
(hash=data-client-6/cache=<nul>)
[2020-07-02 12:34:09.799586] I [MSGID: 109066]
[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
/raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
(99938ee6-6986-4123-9d72-ec09e2310b4f)
(hash=data-client-17/cache=data-client-18) =>
/raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
(hash=data-client-17/cache=<nul>)
....


Please look into this at top-priority if possible.
Let me know if anything else is required.


-- 
Regards,
Shreyansh Shah
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200702/350ba536/attachment.html>

Strahil Nikolov

2020-Jul-02 20:21 UTC

head link

[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance

Hi Shreyansh,

have you checked  the gluster logs on the failed nodes for any clues -> for
example 10.132.0.9 ?

Sadly I don't have much experience with pure distributed  volumes, but I do
think that everything will be back to normal when the rebalance  is complete.
Yet, based on the output - it won't be soon.

Best Regards,
Strahil Nikolov

?? 2 ??? 2020 ?. 17:39:25 GMT+03:00, Shreyansh Shah <shreyansh.shah at
alpha-grep.com> ??????:>Hi All,
>
>*We are facing "Mismatching layouts for <directory>,gfid =
<id>"
>errors.*
>
>We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB
>each)
>on each node, 7 nodes in total. We added new bricks yesterday to the
>existing setup.
>Post that we did a rebalance fix-layout and then a rebalance (which is
>currently still in progress). The status shows "failed" on certain
>bricks
>but "in progress" for others. Adding output for gluster rebalance
>status
>below.
>
>The glusterfs client logs are flooded with "Mismatching layouts for
><directory>,gfid = <id>"
>The performance too seems to have degraded due to this, even basic
>commands
>like `cd` and `ls` are taking more than a minute compared to sub-second
>number before brick addition.
>Apart from that we also experienced many binaries and files giving
>error
>stale file handle error even though the files were present.
>
>
>*gluster rebalance status :*
>
>Node Rebalanced-files          size       scanned      failures
>skipped               status  run time in h:m:s
>---------      -----------   -----------   -----------   -----------
>-----------         ------------     --------------
>localhost              176         3.5GB         12790             0
>   8552          in progress       21:36:01
>10.132.0.72             8232       394.8GB         19995            21
>       26               failed       14:50:30
>10.132.0.44            12625         1.0TB         50023             1
>    10202          in progress       21:36:00
>10.132.0.3            21982       956.8GB         79145             1
>  34571          in progress       21:36:00
>10.132.0.9             7975       355.8GB         20157             6
>   1522               failed       14:51:45
>10.132.0.73             6293       394.5GB         26414           151
>     8085               failed       14:51:45
>10.132.0.70             6480       477.1GB         21058            27
>     1787               failed       14:50:32
>Estimated time left for rebalance to complete :      130:56:28
>
>
>*Logs from one of the clients below:*
>
>[2020-07-02 12:30:14.971916] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641;
>disk
>layout - 2761060380 - 3067813815 - 4159036738
>[2020-07-02 12:30:14.971935] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>[2020-07-02 12:30:15.032013] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641;
>disk
>layout - 3681390552 - 3988143987 - 4159036738
>[2020-07-02 12:30:15.032059] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>[2020-07-02 12:30:15.032107] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641;
>disk
>layout - 3374637116 - 3681390551 - 4159036738
>[2020-07-02 12:30:15.032153] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>[2020-07-02 12:30:15.093329] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794;
>disk
>layout - 2454306944 - 2761060379 - 4159036738
>[2020-07-02 12:30:15.093373] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI/20200630, gfid >42a506b3-7aff-4935-8ef7-ecb8877c8222
>[2020-07-02 12:30:15.093460] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794;
>disk
>layout - 2761060380 - 3067813815 - 4159036738
>[2020-07-02 12:30:15.093515] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI/20200630, gfid >42a506b3-7aff-4935-8ef7-ecb8877c8222
>[2020-07-02 12:30:15.151063] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794;
>disk
>layout - 3681390552 - 3988143987 - 4159036738
>[2020-07-02 12:30:15.151108] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI/20200630, gfid >42a506b3-7aff-4935-8ef7-ecb8877c8222
>[2020-07-02 12:30:15.151149] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794;
>disk
>layout - 3374637116 - 3681390551 - 4159036738
>[2020-07-02 12:30:15.151162] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI/20200630, gfid >42a506b3-7aff-4935-8ef7-ecb8877c8222
>[2020-07-02 12:30:15.424321] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-11; inode layout - 920400036 - 1227153471 - 3997647794;
>disk
>layout - 920400036 - 1227153471 - 4159036738
>[2020-07-02 12:30:15.424380] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>[2020-07-02 12:30:15.424456] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794;
>disk
>layout - 1840730208 - 2147483643 - 4159036738
>[2020-07-02 12:30:15.424484] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>[2020-07-02 12:30:15.424525] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794;
>disk
>layout - 1533976772 - 1840730207 - 4159036738
>[2020-07-02 12:30:15.424542] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>[2020-07-02 12:30:15.424596] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
>layout - 613646600 - 920400035 - 4159036738
>[2020-07-02 12:30:15.424607] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>[2020-07-02 12:30:16.004482] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat
>on
>data-client-7 (hashed subvol is data-client-17)
>[2020-07-02 12:30:16.005523] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_CDS_1_DATA.dat
>[2020-07-02 12:30:16.531047] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_CDS_1_METADATA.dat
>on data-client-9 (hashed subvol is data-client-19)
>[2020-07-02 12:30:16.532086] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_CDS_1_METADATA.dat
>[2020-07-02 12:30:18.733229] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat
>on
>data-client-17 (hashed subvol is data-client-9)
>[2020-07-02 12:30:18.734421] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_CDS_2_DATA.dat
>[2020-07-02 12:30:19.171930] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_CDS_2_METADATA.dat
>on data-client-9 (hashed subvol is data-client-18)
>[2020-07-02 12:30:19.172901] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_CDS_2_METADATA.dat
>[2020-07-02 12:30:21.028495] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat
>on
>data-client-6 (hashed subvol is data-client-15)
>[2020-07-02 12:30:21.029836] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_2_DATA.dat
>[2020-07-02 12:30:21.127648] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_2_METADATA.dat
>on data-client-11 (hashed subvol is data-client-3)
>[2020-07-02 12:30:21.128713] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_2_METADATA.dat
>[2020-07-02 12:30:21.201126] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat
>on
>data-client-15 (hashed subvol is data-client-7)
>[2020-07-02 12:30:21.201928] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_3_DATA.dat
>[2020-07-02 12:30:21.566158] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_3_METADATA.dat
>on data-client-7 (hashed subvol is data-client-16)
>[2020-07-02 12:30:21.567123] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_3_METADATA.dat
>[2020-07-02 12:30:21.649357] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat
>on
>data-client-2 (hashed subvol is data-client-11)
>[2020-07-02 12:30:21.661381] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_4_DATA.dat
>[2020-07-02 12:30:21.748937] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_4_METADATA.dat
>on data-client-15 (hashed subvol is data-client-7)
>[2020-07-02 12:30:21.749481] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_4_METADATA.dat
>[2020-07-02 12:30:21.898593] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat
>on
>data-client-14 (hashed subvol is data-client-7)
>[2020-07-02 12:30:21.899442] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_6_DATA.dat
>[2020-07-02 12:30:22.039337] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_6_METADATA.dat
>on data-client-10 (hashed subvol is data-client-2)
>[2020-07-02 12:30:22.040086] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_6_METADATA.dat
>[2020-07-02 12:30:22.501877] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECDS1_DATA.dat
>on data-client-15 (hashed subvol is data-client-8)
>[2020-07-02 12:30:22.502712] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECDS1_DATA.dat
>[2020-07-02 12:30:22.782577] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
>(hashed subvol is data-client-6)
>[2020-07-02 12:30:22.783777] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECDS1_METADATA.dat
>[2020-07-02 12:30:23.146847] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM1_DATA.dat on
>data-client-17 (hashed subvol is data-client-9)
>[2020-07-02 12:30:23.148009] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM1_DATA.dat
>[2020-07-02 12:30:23.229290] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14
>(hashed
>subvol is data-client-6)
>[2020-07-02 12:30:23.230151] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM1_METADATA.dat
>[2020-07-02 12:30:23.889520] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM2_DATA.dat on
>data-client-2 (hashed subvol is data-client-11)
>[2020-07-02 12:30:23.896618] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM2_DATA.dat
>[2020-07-02 12:30:24.093017] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6
>(hashed
>subvol is data-client-15)
>[2020-07-02 12:30:24.094117] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM2_METADATA.dat
>[2020-07-02 12:30:24.345257] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM3_DATA.dat on
>data-client-17 (hashed subvol is data-client-10)
>[2020-07-02 12:30:24.346234] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM3_DATA.dat
>[2020-07-02 12:30:24.425835] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6
>(hashed
>subvol is data-client-15)
>[2020-07-02 12:30:24.426880] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM3_METADATA.dat
>[2020-07-02 12:30:25.158718] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO1_DATA.dat
>on data-client-9 (hashed subvol is data-client-19)
>[2020-07-02 12:30:25.159619] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO1_DATA.dat
>[2020-07-02 12:30:25.531479] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2
>(hashed
>subvol is data-client-10)
>[2020-07-02 12:30:25.540569] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO1_METADATA.dat
>[2020-07-02 12:30:25.771692] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO2_DATA.dat
>on data-client-11 (hashed subvol is data-client-3)
>[2020-07-02 12:30:25.772610] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO2_DATA.dat
>[2020-07-02 12:30:25.866118] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
>(hashed subvol is data-client-8)
>[2020-07-02 12:30:25.866917] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO2_METADATA.dat
>[2020-07-02 12:30:26.424386] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO3_DATA.dat
>on data-client-9 (hashed subvol is data-client-18)
>[2020-07-02 12:30:26.425309] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO3_DATA.dat
>[2020-07-02 12:30:26.818852] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
>(hashed subvol is data-client-2)
>[2020-07-02 12:30:26.819890] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO3_METADATA.dat
>[2020-07-02 12:30:27.352405] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO4_DATA.dat
>on data-client-10 (hashed subvol is data-client-2)
>[2020-07-02 12:30:27.352914] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO4_DATA.dat
>[2020-07-02 12:30:27.521286] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8
>(hashed
>subvol is data-client-18)
>[2020-07-02 12:30:27.522325] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO4_METADATA.dat
>[2020-07-02 12:30:28.566634] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO5_DATA.dat
>on data-client-2 (hashed subvol is data-client-11)
>[2020-07-02 12:30:28.579295] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO5_DATA.dat
>[2020-07-02 12:30:28.958028] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO6_DATA.dat
>on data-client-7 (hashed subvol is data-client-16)
>[2020-07-02 12:30:28.959102] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO6_DATA.dat
>[2020-07-02 12:30:29.012429] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6
>(hashed
>subvol is data-client-15)
>[2020-07-02 12:30:29.013416] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO6_METADATA.dat
>[2020-07-02 12:30:29.396716] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/NSEFO_BSE_TSDATA.dat on
>data-client-17 (hashed subvol is data-client-10)
>[2020-07-02 12:30:29.397740] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/NSEFO_BSE_TSDATA.dat
>[2020-07-02 12:30:29.556312] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9
>(hashed
>subvol is data-client-18)
>[2020-07-02 12:30:29.557197] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
>[2020-07-02 12:30:30.605354] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on
>data-client-9
>(hashed subvol is data-client-19)
>[2020-07-02 12:30:30.606117] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
>[2020-07-02 12:30:31.559206] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
>613576736 - 920330171 - 4159036738
>[2020-07-02 12:30:31.559255] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>[2020-07-02 12:30:31.569025] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout
>-
>920330172 - 1227083607 - 4159036738
>[2020-07-02 12:30:31.569067] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>[2020-07-02 12:30:31.701849] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout
>-
>3374637116 - 3681390551 - 4159036738
>[2020-07-02 12:30:31.701895] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX, gfid
>fff324f2-f855-4881-b77c-81e856522373
>[2020-07-02 12:30:31.738464] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout
>-
>3681390552 - 3988143987 - 4159036738
>[2020-07-02 12:30:31.738507] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX, gfid
>fff324f2-f855-4881-b77c-81e856522373
>[2020-07-02 12:30:31.857102] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641;
>disk
>layout - 3067883680 - 3374637115 - 4159036738
>[2020-07-02 12:30:31.857147] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid
>f8447150-4801-4188-add9-ea295bb88729
>[2020-07-02 12:30:31.857180] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641;
>disk
>layout - 3374637116 - 3681390551 - 4159036738
>[2020-07-02 12:30:31.857197] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid
>f8447150-4801-4188-add9-ea295bb88729
>[2020-07-02 12:30:31.917705] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout
>- 0
>- 306753435 - 4159036738
>[2020-07-02 12:30:31.917781] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid
>f8447150-4801-4188-add9-ea295bb88729
>[2020-07-02 12:30:31.917855] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641;
>disk
>layout - 3988213852 - 4294967295 - 4159036738
>[2020-07-02 12:30:31.917874] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid
>f8447150-4801-4188-add9-ea295bb88729
>[2020-07-02 12:30:32.390945] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout
>-
>3681460416 - 3988213851 - 4159036738
>[2020-07-02 12:30:32.390998] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/NIFTY, gfid
>b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>[2020-07-02 12:30:32.391056] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout
>-
>3988213852 - 4294967295 - 4159036738
>[2020-07-02 12:30:32.391075] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/NIFTY, gfid
>b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>[2020-07-02 12:33:50.915279] I [MSGID: 109066]
>[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>/raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
>(2cb54500-814d-4e85-83e7-e33d9440b18d)
>(hash=data-client-6/cache=data-client-18) =>
>/raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
>(hash=data-client-6/cache=<nul>)
>[2020-07-02 12:34:09.799586] I [MSGID: 109066]
>[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>/raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
>(99938ee6-6986-4123-9d72-ec09e2310b4f)
>(hash=data-client-17/cache=data-client-18) =>
>/raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
>(hash=data-client-17/cache=<nul>)
>....
>
>
>Please look into this at top-priority if possible.
>Let me know if anything else is required.

Shreyansh Shah

2020-Jul-06 08:41 UTC

head link

[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance

Hi,
Did anyone get a chance to look into this?

On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <shreyansh.shah at
alpha-grep.com>
wrote:
> Hi All,
>
> *We are facing "Mismatching layouts for <directory>,gfid =
<id>" errors.*
>
> We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each)
> on each node, 7 nodes in total. We added new bricks yesterday to the
> existing setup.
> Post that we did a rebalance fix-layout and then a rebalance (which is
> currently still in progress). The status shows "failed" on
certain bricks
> but "in progress" for others. Adding output for gluster rebalance
status
> below.
>
> The glusterfs client logs are flooded with "Mismatching layouts for
> <directory>,gfid = <id>"
> The performance too seems to have degraded due to this, even basic
> commands like `cd` and `ls` are taking more than a minute compared to
> sub-second number before brick addition.
> Apart from that we also experienced many binaries and files giving error
> stale file handle error even though the files were present.
>
>
> *gluster rebalance status :*
>
> Node Rebalanced-files          size       scanned      failures
> skipped               status  run time in h:m:s
> ---------      -----------   -----------   -----------   -----------
> -----------         ------------     --------------
> localhost              176         3.5GB         12790             0
>    8552          in progress       21:36:01
> 10.132.0.72             8232       394.8GB         19995            21
>        26               failed       14:50:30
> 10.132.0.44            12625         1.0TB         50023             1
>     10202          in progress       21:36:00
> 10.132.0.3            21982       956.8GB         79145             1
>     34571          in progress       21:36:00
> 10.132.0.9             7975       355.8GB         20157             6
>      1522               failed       14:51:45
> 10.132.0.73             6293       394.5GB         26414           151
>      8085               failed       14:51:45
> 10.132.0.70             6480       477.1GB         21058            27
>      1787               failed       14:50:32
> Estimated time left for rebalance to complete :      130:56:28
>
>
> *Logs from one of the clients below:*
>
> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
> layout - 2761060380 - 3067813815 - 4159036738
> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
> layout - 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
> layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
> layout - 2454306944 - 2761060379 - 4159036738
> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
> layout - 2761060380 - 3067813815 - 4159036738
> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
> layout - 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
> layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
> layout - 920400036 - 1227153471 - 4159036738
> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
> layout - 1840730208 - 2147483643 - 4159036738
> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
> layout - 1533976772 - 1840730207 - 4159036738
> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
> layout - 613646600 - 920400035 - 4159036738
> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
> data-client-7 (hashed subvol is data-client-17)
> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_1_DATA.dat
> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
> on data-client-9 (hashed subvol is data-client-19)
> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_1_METADATA.dat
> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
> data-client-17 (hashed subvol is data-client-9)
> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_2_DATA.dat
> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
> on data-client-9 (hashed subvol is data-client-18)
> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_2_METADATA.dat
> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
> data-client-6 (hashed subvol is data-client-15)
> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_2_DATA.dat
> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
> on data-client-11 (hashed subvol is data-client-3)
> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_2_METADATA.dat
> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
> data-client-15 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_3_DATA.dat
> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
> on data-client-7 (hashed subvol is data-client-16)
> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_3_METADATA.dat
> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
> data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_4_DATA.dat
> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
> on data-client-15 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_4_METADATA.dat
> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
> data-client-14 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_6_DATA.dat
> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
> on data-client-10 (hashed subvol is data-client-2)
> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_6_METADATA.dat
> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
> on data-client-15 (hashed subvol is data-client-8)
> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECDS1_DATA.dat
> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
> (hashed subvol is data-client-6)
> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECDS1_METADATA.dat
> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
> data-client-17 (hashed subvol is data-client-9)
> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM1_DATA.dat
> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
> subvol is data-client-6)
> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM1_METADATA.dat
> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
> data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM2_DATA.dat
> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
> subvol is data-client-15)
> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM2_METADATA.dat
> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
> data-client-17 (hashed subvol is data-client-10)
> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM3_DATA.dat
> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
> subvol is data-client-15)
> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM3_METADATA.dat
> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
> on data-client-9 (hashed subvol is data-client-19)
> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO1_DATA.dat
> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
> subvol is data-client-10)
> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
> on data-client-11 (hashed subvol is data-client-3)
> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO2_DATA.dat
> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
> (hashed subvol is data-client-8)
> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
> on data-client-9 (hashed subvol is data-client-18)
> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO3_DATA.dat
> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
> (hashed subvol is data-client-2)
> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
> on data-client-10 (hashed subvol is data-client-2)
> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO4_DATA.dat
> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
> subvol is data-client-18)
> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
> on data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO5_DATA.dat
> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
> on data-client-7 (hashed subvol is data-client-16)
> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO6_DATA.dat
> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
> subvol is data-client-15)
> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
> data-client-17 (hashed subvol is data-client-10)
> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/NSEFO_BSE_TSDATA.dat
> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
> subvol is data-client-18)
> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
> (hashed subvol is data-client-19)
> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
> 613576736 - 920330171 - 4159036738
> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
> 920330172 - 1227083607 - 4159036738
> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
> 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX, gfid >
fff324f2-f855-4881-b77c-81e856522373
> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
> 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX, gfid >
fff324f2-f855-4881-b77c-81e856522373
> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
> layout - 3067883680 - 3374637115 - 4159036738
> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >
f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
> layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >
f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
> - 306753435 - 4159036738
> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >
f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
> layout - 3988213852 - 4294967295 - 4159036738
> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >
f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
> 3681460416 - 3988213851 - 4159036738
> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/NIFTY, gfid >
b2d4deb7-c58c-4046-b6f2-7c7f44d71311
> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
> 3988213852 - 4294967295 - 4159036738
> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/NIFTY, gfid >
b2d4deb7-c58c-4046-b6f2-7c7f44d71311
> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
> (2cb54500-814d-4e85-83e7-e33d9440b18d)
> (hash=data-client-6/cache=data-client-18) =>
> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
> (hash=data-client-6/cache=<nul>)
> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
> (99938ee6-6986-4123-9d72-ec09e2310b4f)
> (hash=data-client-17/cache=data-client-18) =>
> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
> (hash=data-client-17/cache=<nul>)
> ....
>
>
> Please look into this at top-priority if possible.
> Let me know if anything else is required.
>
>
> --
> Regards,
> Shreyansh Shah
>

-- 
Regards,
Shreyansh Shah
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200706/3f8adf21/attachment.html>

Susant Palai

2020-Jul-13 08:05 UTC

head link

[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance

The log messages are fine. Since you added a new brick, the client is responding
to that by syncing its in-memory layout with latest server layout. The
performance drop could be because of locks taken during this layout sync.

> On 02-Jul-2020, at 20:09, Shreyansh Shah <shreyansh.shah at
alpha-grep.com> wrote:
> 
> Hi All,
> 
> We are facing "Mismatching layouts for <directory>,gfid =
<id>" errors.
> 
> We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each)
on each node, 7 nodes in total. We added new bricks yesterday to the existing
setup.
> Post that we did a rebalance fix-layout and then a rebalance (which is
currently still in progress). The status shows "failed" on certain
bricks but "in progress" for others. Adding output for gluster
rebalance status below.
> 
> The glusterfs client logs are flooded with "Mismatching layouts for
<directory>,gfid = <id>"
> The performance too seems to have degraded due to this, even basic commands
like `cd` and `ls` are taking more than a minute compared to sub-second number
before brick addition.
> Apart from that we also experienced many binaries and files giving error
stale file handle error even though the files were present.
> 
> 
> gluster rebalance status :
> 
> Node Rebalanced-files          size       scanned      failures      
skipped               status  run time in h:m:s
> ---------      -----------   -----------   -----------   -----------  
-----------         ------------     --------------
> localhost              176         3.5GB         12790             0       
8552          in progress       21:36:01
> 10.132.0.72             8232       394.8GB         19995            21     
26               failed       14:50:30
> 10.132.0.44            12625         1.0TB         50023             1     
10202          in progress       21:36:00
> 10.132.0.3            21982       956.8GB         79145             1      
34571          in progress       21:36:00
> 10.132.0.9             7975       355.8GB         20157             6      
1522               failed       14:51:45
> 10.132.0.73             6293       394.5GB         26414           151     
8085               failed       14:51:45
> 10.132.0.70             6480       477.1GB         21058            27     
1787               failed       14:50:32
> Estimated time left for rebalance to complete :      130:56:28
> 
> 
> Logs from one of the clients below:
> 
> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16;
inode layout - 2761060380 - 3067813815 - 3995747641; disk layout - 2761060380 -
3067813815 - 4159036738
> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19;
inode layout - 3681390552 - 3988143987 - 3995747641; disk layout - 3681390552 -
3988143987 - 4159036738
> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18;
inode layout - 3374637116 - 3681390551 - 3995747641; disk layout - 3374637116 -
3681390551 - 4159036738
> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15;
inode layout - 2454306944 - 2761060379 - 3997647794; disk layout - 2454306944 -
2761060379 - 4159036738
> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16;
inode layout - 2761060380 - 3067813815 - 3997647794; disk layout - 2761060380 -
3067813815 - 4159036738
> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19;
inode layout - 3681390552 - 3988143987 - 3997647794; disk layout - 3681390552 -
3988143987 - 4159036738
> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18;
inode layout - 3374637116 - 3681390551 - 3997647794; disk layout - 3374637116 -
3681390551 - 4159036738
> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-11;
inode layout - 920400036 - 1227153471 - 3997647794; disk layout - 920400036 -
1227153471 - 4159036738
> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16;
inode layout - 1840730208 - 2147483643 - 3997647794; disk layout - 1840730208 -
2147483643 - 4159036738
> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15;
inode layout - 1533976772 - 1840730207 - 3997647794; disk layout - 1533976772 -
1840730207 - 4159036738
> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-10;
inode layout - 613646600 - 920400035 - 3997647794; disk layout - 613646600 -
920400035 - 4159036738
> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on data-client-7
(hashed subvol is data-client-17)
> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_1_DATA.dat
> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat on data-client-9
(hashed subvol is data-client-19)
> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_1_METADATA.dat
> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on data-client-17
(hashed subvol is data-client-9)
> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_2_DATA.dat
> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat on data-client-9
(hashed subvol is data-client-18)
> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_2_METADATA.dat
> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on data-client-6
(hashed subvol is data-client-15)
> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_2_DATA.dat
> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat on data-client-11
(hashed subvol is data-client-3)
> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_2_METADATA.dat
> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on data-client-15
(hashed subvol is data-client-7)
> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_3_DATA.dat
> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat on data-client-7
(hashed subvol is data-client-16)
> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_3_METADATA.dat
> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on data-client-2
(hashed subvol is data-client-11)
> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_4_DATA.dat
> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat on data-client-15
(hashed subvol is data-client-7)
> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_4_METADATA.dat
> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on data-client-14
(hashed subvol is data-client-7)
> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_6_DATA.dat
> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat on data-client-10
(hashed subvol is data-client-2)
> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_6_METADATA.dat
> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat on data-client-15
(hashed subvol is data-client-8)
> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECDS1_DATA.dat
> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSECDS1_METADATA.dat on
data-client-11 (hashed subvol is data-client-6)
> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECDS1_METADATA.dat
> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on data-client-17
(hashed subvol is data-client-9)
> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM1_DATA.dat
> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSECM1_METADATA.dat on
data-client-14 (hashed subvol is data-client-6)
> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM1_METADATA.dat
> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on data-client-2
(hashed subvol is data-client-11)
> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM2_DATA.dat
> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSECM2_METADATA.dat on
data-client-6 (hashed subvol is data-client-15)
> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM2_METADATA.dat
> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on data-client-17
(hashed subvol is data-client-10)
> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM3_DATA.dat
> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSECM3_METADATA.dat on
data-client-6 (hashed subvol is data-client-15)
> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM3_METADATA.dat
> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat on data-client-9
(hashed subvol is data-client-19)
> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO1_DATA.dat
> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on
data-client-2 (hashed subvol is data-client-10)
> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO1_METADATA.dat
> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat on data-client-11
(hashed subvol is data-client-3)
> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO2_DATA.dat
> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on
data-client-15 (hashed subvol is data-client-8)
> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO2_METADATA.dat
> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat on data-client-9
(hashed subvol is data-client-18)
> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO3_DATA.dat
> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on
data-client-10 (hashed subvol is data-client-2)
> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO3_METADATA.dat
> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat on data-client-10
(hashed subvol is data-client-2)
> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO4_DATA.dat
> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on
data-client-8 (hashed subvol is data-client-18)
> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO4_METADATA.dat
> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat on data-client-2
(hashed subvol is data-client-11)
> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO5_DATA.dat
> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat on data-client-7
(hashed subvol is data-client-16)
> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO6_DATA.dat
> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on
data-client-6 (hashed subvol is data-client-15)
> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO6_METADATA.dat
> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on data-client-17
(hashed subvol is data-client-10)
> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSEFO_BSE_TSDATA.dat
> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on
data-client-9 (hashed subvol is data-client-18)
> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of
stale linkfile /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on
data-client-9 (hashed subvol is data-client-19)
> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned
with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18;
inode layout - 613576736 - 920330171 - 1; disk layout - 613576736 - 920330171 -
4159036738
> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19;
inode layout - 920330172 - 1227083607 - 1; disk layout - 920330172 - 1227083607
- 4159036738
> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18;
inode layout - 3374637116 - 3681390551 - 1; disk layout - 3374637116 -
3681390551 - 4159036738
> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX, gfid = fff324f2-f855-4881-b77c-81e856522373
> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19;
inode layout - 3681390552 - 3988143987 - 1; disk layout - 3681390552 -
3988143987 - 4159036738
> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX, gfid = fff324f2-f855-4881-b77c-81e856522373
> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15;
inode layout - 3067883680 - 3374637115 - 3995747641; disk layout - 3067883680 -
3374637115 - 4159036738
> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16;
inode layout - 3374637116 - 3681390551 - 3995747641; disk layout - 3374637116 -
3681390551 - 4159036738
> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19;
inode layout - 0 - 306753435 - 3995747641; disk layout - 0 - 306753435 -
4159036738
> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18;
inode layout - 3988213852 - 4294967295 - 3995747641; disk layout - 3988213852 -
4294967295 - 4159036738
> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18;
inode layout - 3681460416 - 3988213851 - 1; disk layout - 3681460416 -
3988213851 - 4159036738
> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/NIFTY, gfid =
b2d4deb7-c58c-4046-b6f2-7c7f44d71311
> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19;
inode layout - 3988213852 - 4294967295 - 1; disk layout - 3988213852 -
4294967295 - 4159036738
> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/NIFTY, gfid =
b2d4deb7-c58c-4046-b6f2-7c7f44d71311
> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
/raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
(2cb54500-814d-4e85-83e7-e33d9440b18d) (hash=data-client-6/cache=data-client-18)
=> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
(hash=data-client-6/cache=<nul>)
> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
/raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
(99938ee6-6986-4123-9d72-ec09e2310b4f)
(hash=data-client-17/cache=data-client-18) =>
/raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
(hash=data-client-17/cache=<nul>)
> ....
> 
> 
> Please look into this at top-priority if possible.
> Let me know if anything else is required.
> 
> 
> -- 
> Regards,
> Shreyansh Shah
> ________
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
> 
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200713/e1bea7ca/attachment.html>

Gluster users - Jul 2020 - "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance

[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance

[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance

[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance

[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance