Shreyansh Shah
2020-Jul-07 08:46 UTC
[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance
Sounds good, thank you. On Tue, Jul 7, 2020 at 2:12 PM Barak Sason Rofman <bsasonro at redhat.com> wrote:> Thanks Shreyansh, > > I'll look into it, however I'll likely need some help from more senior > team members to perform RCA. > I'll update once I have new insights. > > My regards, > > On Tue, Jul 7, 2020 at 11:40 AM Shreyansh Shah < > shreyansh.shah at alpha-grep.com> wrote: > >> Hi Barak, >> Thanks for looking into this and helping me out, >> The fix-layout was successful, and I ran a rebalance after completion of >> fix-layout. >> The rebalance status though did show failure for 3 nodes. >> >> On Tue, Jul 7, 2020 at 2:07 PM Barak Sason Rofman <bsasonro at redhat.com> >> wrote: >> >>> Greetings again Shreyansh, >>> >>> I'm indeed seeing a lot of errors in the log file - still unsure about >>> the RC. >>> You mentioned that prior to running rebalance you ran fix-layout, was >>> the fix-layout successful? >>> Another question - did you wait until fix-layout was completed before >>> running rebalance? >>> >>> My thanks, >>> >>> On Mon, Jul 6, 2020 at 9:33 PM Shreyansh Shah < >>> shreyansh.shah at alpha-grep.com> wrote: >>> >>>> Hi, >>>> Attaching rebalance logs >>>> FYI, we ran "gluster rebalance fix-layout" followed by "gluster >>>> rebalance" on 20200701 and today we again ran "gluster rebalance fix-layout" >>>> >>>> >>>> PFA >>>> >>>> On Mon, Jul 6, 2020 at 11:08 PM Barak Sason Rofman <bsasonro at redhat.com> >>>> wrote: >>>> >>>>> I think it would be best. >>>>> As I can't say at this point where the problem is originating from, >>>>> brick logs might also be necessary (I assume I would have a better picture >>>>> once I have the rebalance logs). >>>>> >>>>> Cheers, >>>>> >>>>> On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah < >>>>> shreyansh.shah at alpha-grep.com> wrote: >>>>> >>>>>> Hi Barak, >>>>>> Can provide the rebalance logs. Do you require all the brick logs (14 >>>>>> in total)? >>>>>> >>>>>> On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman < >>>>>> bsasonro at redhat.com> wrote: >>>>>> >>>>>>> Greetings Shreyansh, >>>>>>> >>>>>>> Off-hand I can't come up with a reason for these failures. >>>>>>> In order to start looking into this, access to the full rebalance >>>>>>> logs is required (possibly brick logs as well). >>>>>>> Can you provide those? >>>>>>> >>>>>>> My regards, >>>>>>> >>>>>>> >>>>>>> On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah < >>>>>>> shreyansh.shah at alpha-grep.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> Did anyone get a chance to look into this? >>>>>>>> >>>>>>>> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah < >>>>>>>> shreyansh.shah at alpha-grep.com> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> *We are facing "Mismatching layouts for <directory>,gfid = <id>" >>>>>>>>> errors.* >>>>>>>>> >>>>>>>>> We have a distributed glusterfs 5.10, no replication, 2 bricks >>>>>>>>> (4TB each) on each node, 7 nodes in total. We added new bricks yesterday to >>>>>>>>> the existing setup. >>>>>>>>> Post that we did a rebalance fix-layout and then a rebalance >>>>>>>>> (which is currently still in progress). The status shows "failed" on >>>>>>>>> certain bricks but "in progress" for others. Adding output for gluster >>>>>>>>> rebalance status below. >>>>>>>>> >>>>>>>>> The glusterfs client logs are flooded with "Mismatching layouts >>>>>>>>> for <directory>,gfid = <id>" >>>>>>>>> The performance too seems to have degraded due to this, even basic >>>>>>>>> commands like `cd` and `ls` are taking more than a minute compared to >>>>>>>>> sub-second number before brick addition. >>>>>>>>> Apart from that we also experienced many binaries and files giving >>>>>>>>> error stale file handle error even though the files were present. >>>>>>>>> >>>>>>>>> >>>>>>>>> *gluster rebalance status :* >>>>>>>>> >>>>>>>>> Node Rebalanced-files size scanned failures >>>>>>>>> skipped status run time in h:m:s >>>>>>>>> --------- ----------- ----------- ----------- >>>>>>>>> ----------- ----------- ------------ -------------- >>>>>>>>> localhost 176 3.5GB 12790 >>>>>>>>> 0 8552 in progress 21:36:01 >>>>>>>>> 10.132.0.72 8232 394.8GB 19995 >>>>>>>>> 21 26 failed 14:50:30 >>>>>>>>> 10.132.0.44 12625 1.0TB 50023 >>>>>>>>> 1 10202 in progress 21:36:00 >>>>>>>>> 10.132.0.3 21982 956.8GB 79145 >>>>>>>>> 1 34571 in progress 21:36:00 >>>>>>>>> 10.132.0.9 7975 355.8GB 20157 >>>>>>>>> 6 1522 failed 14:51:45 >>>>>>>>> 10.132.0.73 6293 394.5GB 26414 >>>>>>>>> 151 8085 failed 14:51:45 >>>>>>>>> 10.132.0.70 6480 477.1GB 21058 >>>>>>>>> 27 1787 failed 14:50:32 >>>>>>>>> Estimated time left for rebalance to complete : 130:56:28 >>>>>>>>> >>>>>>>>> >>>>>>>>> *Logs from one of the clients below:* >>>>>>>>> >>>>>>>>> [2020-07-02 12:30:14.971916] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk >>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738 >>>>>>>>> [2020-07-02 12:30:14.971935] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>>> [2020-07-02 12:30:15.032013] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk >>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.032059] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>>> [2020-07-02 12:30:15.032107] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk >>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.032153] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>>> [2020-07-02 12:30:15.093329] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk >>>>>>>>> layout - 2454306944 - 2761060379 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.093373] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>> [2020-07-02 12:30:15.093460] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk >>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.093515] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>> [2020-07-02 12:30:15.151063] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk >>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.151108] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>> [2020-07-02 12:30:15.151149] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk >>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.151162] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>> [2020-07-02 12:30:15.424321] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk >>>>>>>>> layout - 920400036 - 1227153471 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.424380] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>> [2020-07-02 12:30:15.424456] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk >>>>>>>>> layout - 1840730208 - 2147483643 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.424484] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>> [2020-07-02 12:30:15.424525] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk >>>>>>>>> layout - 1533976772 - 1840730207 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.424542] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>> [2020-07-02 12:30:15.424596] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk >>>>>>>>> layout - 613646600 - 920400035 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.424607] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>> [2020-07-02 12:30:16.004482] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on >>>>>>>>> data-client-7 (hashed subvol is data-client-17) >>>>>>>>> [2020-07-02 12:30:16.005523] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_CDS_1_DATA.dat >>>>>>>>> [2020-07-02 12:30:16.531047] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat >>>>>>>>> on data-client-9 (hashed subvol is data-client-19) >>>>>>>>> [2020-07-02 12:30:16.532086] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_CDS_1_METADATA.dat >>>>>>>>> [2020-07-02 12:30:18.733229] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on >>>>>>>>> data-client-17 (hashed subvol is data-client-9) >>>>>>>>> [2020-07-02 12:30:18.734421] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_CDS_2_DATA.dat >>>>>>>>> [2020-07-02 12:30:19.171930] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat >>>>>>>>> on data-client-9 (hashed subvol is data-client-18) >>>>>>>>> [2020-07-02 12:30:19.172901] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_CDS_2_METADATA.dat >>>>>>>>> [2020-07-02 12:30:21.028495] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on >>>>>>>>> data-client-6 (hashed subvol is data-client-15) >>>>>>>>> [2020-07-02 12:30:21.029836] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_2_DATA.dat >>>>>>>>> [2020-07-02 12:30:21.127648] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat >>>>>>>>> on data-client-11 (hashed subvol is data-client-3) >>>>>>>>> [2020-07-02 12:30:21.128713] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_2_METADATA.dat >>>>>>>>> [2020-07-02 12:30:21.201126] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on >>>>>>>>> data-client-15 (hashed subvol is data-client-7) >>>>>>>>> [2020-07-02 12:30:21.201928] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_3_DATA.dat >>>>>>>>> [2020-07-02 12:30:21.566158] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat >>>>>>>>> on data-client-7 (hashed subvol is data-client-16) >>>>>>>>> [2020-07-02 12:30:21.567123] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_3_METADATA.dat >>>>>>>>> [2020-07-02 12:30:21.649357] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on >>>>>>>>> data-client-2 (hashed subvol is data-client-11) >>>>>>>>> [2020-07-02 12:30:21.661381] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_4_DATA.dat >>>>>>>>> [2020-07-02 12:30:21.748937] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat >>>>>>>>> on data-client-15 (hashed subvol is data-client-7) >>>>>>>>> [2020-07-02 12:30:21.749481] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_4_METADATA.dat >>>>>>>>> [2020-07-02 12:30:21.898593] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on >>>>>>>>> data-client-14 (hashed subvol is data-client-7) >>>>>>>>> [2020-07-02 12:30:21.899442] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_6_DATA.dat >>>>>>>>> [2020-07-02 12:30:22.039337] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat >>>>>>>>> on data-client-10 (hashed subvol is data-client-2) >>>>>>>>> [2020-07-02 12:30:22.040086] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_6_METADATA.dat >>>>>>>>> [2020-07-02 12:30:22.501877] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat >>>>>>>>> on data-client-15 (hashed subvol is data-client-8) >>>>>>>>> [2020-07-02 12:30:22.502712] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECDS1_DATA.dat >>>>>>>>> [2020-07-02 12:30:22.782577] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11 >>>>>>>>> (hashed subvol is data-client-6) >>>>>>>>> [2020-07-02 12:30:22.783777] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat >>>>>>>>> [2020-07-02 12:30:23.146847] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on >>>>>>>>> data-client-17 (hashed subvol is data-client-9) >>>>>>>>> [2020-07-02 12:30:23.148009] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM1_DATA.dat >>>>>>>>> [2020-07-02 12:30:23.229290] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed >>>>>>>>> subvol is data-client-6) >>>>>>>>> [2020-07-02 12:30:23.230151] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat >>>>>>>>> [2020-07-02 12:30:23.889520] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on >>>>>>>>> data-client-2 (hashed subvol is data-client-11) >>>>>>>>> [2020-07-02 12:30:23.896618] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM2_DATA.dat >>>>>>>>> [2020-07-02 12:30:24.093017] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed >>>>>>>>> subvol is data-client-15) >>>>>>>>> [2020-07-02 12:30:24.094117] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat >>>>>>>>> [2020-07-02 12:30:24.345257] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on >>>>>>>>> data-client-17 (hashed subvol is data-client-10) >>>>>>>>> [2020-07-02 12:30:24.346234] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM3_DATA.dat >>>>>>>>> [2020-07-02 12:30:24.425835] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed >>>>>>>>> subvol is data-client-15) >>>>>>>>> [2020-07-02 12:30:24.426880] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat >>>>>>>>> [2020-07-02 12:30:25.158718] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat >>>>>>>>> on data-client-9 (hashed subvol is data-client-19) >>>>>>>>> [2020-07-02 12:30:25.159619] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat >>>>>>>>> [2020-07-02 12:30:25.531479] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed >>>>>>>>> subvol is data-client-10) >>>>>>>>> [2020-07-02 12:30:25.540569] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat >>>>>>>>> [2020-07-02 12:30:25.771692] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat >>>>>>>>> on data-client-11 (hashed subvol is data-client-3) >>>>>>>>> [2020-07-02 12:30:25.772610] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat >>>>>>>>> [2020-07-02 12:30:25.866118] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15 >>>>>>>>> (hashed subvol is data-client-8) >>>>>>>>> [2020-07-02 12:30:25.866917] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat >>>>>>>>> [2020-07-02 12:30:26.424386] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat >>>>>>>>> on data-client-9 (hashed subvol is data-client-18) >>>>>>>>> [2020-07-02 12:30:26.425309] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat >>>>>>>>> [2020-07-02 12:30:26.818852] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10 >>>>>>>>> (hashed subvol is data-client-2) >>>>>>>>> [2020-07-02 12:30:26.819890] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat >>>>>>>>> [2020-07-02 12:30:27.352405] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat >>>>>>>>> on data-client-10 (hashed subvol is data-client-2) >>>>>>>>> [2020-07-02 12:30:27.352914] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat >>>>>>>>> [2020-07-02 12:30:27.521286] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed >>>>>>>>> subvol is data-client-18) >>>>>>>>> [2020-07-02 12:30:27.522325] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat >>>>>>>>> [2020-07-02 12:30:28.566634] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat >>>>>>>>> on data-client-2 (hashed subvol is data-client-11) >>>>>>>>> [2020-07-02 12:30:28.579295] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat >>>>>>>>> [2020-07-02 12:30:28.958028] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat >>>>>>>>> on data-client-7 (hashed subvol is data-client-16) >>>>>>>>> [2020-07-02 12:30:28.959102] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat >>>>>>>>> [2020-07-02 12:30:29.012429] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed >>>>>>>>> subvol is data-client-15) >>>>>>>>> [2020-07-02 12:30:29.013416] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat >>>>>>>>> [2020-07-02 12:30:29.396716] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on >>>>>>>>> data-client-17 (hashed subvol is data-client-10) >>>>>>>>> [2020-07-02 12:30:29.397740] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat >>>>>>>>> [2020-07-02 12:30:29.556312] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed >>>>>>>>> subvol is data-client-18) >>>>>>>>> [2020-07-02 12:30:29.557197] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat >>>>>>>>> [2020-07-02 12:30:30.605354] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9 >>>>>>>>> (hashed subvol is data-client-19) >>>>>>>>> [2020-07-02 12:30:30.606117] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat >>>>>>>>> [2020-07-02 12:30:31.559206] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout - >>>>>>>>> 613576736 - 920330171 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.559255] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99 >>>>>>>>> [2020-07-02 12:30:31.569025] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout - >>>>>>>>> 920330172 - 1227083607 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.569067] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99 >>>>>>>>> [2020-07-02 12:30:31.701849] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout - >>>>>>>>> 3374637116 - 3681390551 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.701895] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid >>>>>>>>> fff324f2-f855-4881-b77c-81e856522373 >>>>>>>>> [2020-07-02 12:30:31.738464] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout - >>>>>>>>> 3681390552 - 3988143987 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.738507] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid >>>>>>>>> fff324f2-f855-4881-b77c-81e856522373 >>>>>>>>> [2020-07-02 12:30:31.857102] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk >>>>>>>>> layout - 3067883680 - 3374637115 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.857147] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>> [2020-07-02 12:30:31.857180] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk >>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.857197] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>> [2020-07-02 12:30:31.917705] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0 >>>>>>>>> - 306753435 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.917781] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>> [2020-07-02 12:30:31.917855] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk >>>>>>>>> layout - 3988213852 - 4294967295 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.917874] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>> [2020-07-02 12:30:32.390945] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout - >>>>>>>>> 3681460416 - 3988213851 - 4159036738 >>>>>>>>> [2020-07-02 12:30:32.390998] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid >>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311 >>>>>>>>> [2020-07-02 12:30:32.391056] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout - >>>>>>>>> 3988213852 - 4294967295 - 4159036738 >>>>>>>>> [2020-07-02 12:30:32.391075] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid >>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311 >>>>>>>>> [2020-07-02 12:33:50.915279] I [MSGID: 109066] >>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming >>>>>>>>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2 >>>>>>>>> (2cb54500-814d-4e85-83e7-e33d9440b18d) >>>>>>>>> (hash=data-client-6/cache=data-client-18) => >>>>>>>>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null)) >>>>>>>>> (hash=data-client-6/cache=<nul>) >>>>>>>>> [2020-07-02 12:34:09.799586] I [MSGID: 109066] >>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming >>>>>>>>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k >>>>>>>>> (99938ee6-6986-4123-9d72-ec09e2310b4f) >>>>>>>>> (hash=data-client-17/cache=data-client-18) => >>>>>>>>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null)) >>>>>>>>> (hash=data-client-17/cache=<nul>) >>>>>>>>> .... >>>>>>>>> >>>>>>>>> >>>>>>>>> Please look into this at top-priority if possible. >>>>>>>>> Let me know if anything else is required. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> Shreyansh Shah >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Shreyansh Shah >>>>>>>> ________ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Community Meeting Calendar: >>>>>>>> >>>>>>>> Schedule - >>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>>>>>>> Bridge: https://bluejeans.com/441850968 >>>>>>>> >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Barak Sason Rofman* >>>>>>> >>>>>>> Gluster Storage Development >>>>>>> >>>>>>> Red Hat Israel <https://www.redhat.com/> >>>>>>> >>>>>>> 34 Jerusalem rd. Ra'anana, 43501 >>>>>>> >>>>>>> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >>>>>>> M: *+972-52-4326355* >>>>>>> @RedHat <https://twitter.com/redhat> Red Hat >>>>>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>>>>> <https://www.facebook.com/redhat.il/> >>>>>>> <https://red.ht/sig> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Shreyansh Shah >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Barak Sason Rofman* >>>>> >>>>> Gluster Storage Development >>>>> >>>>> Red Hat Israel <https://www.redhat.com/> >>>>> >>>>> 34 Jerusalem rd. Ra'anana, 43501 >>>>> >>>>> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >>>>> M: *+972-52-4326355* >>>>> @RedHat <https://twitter.com/redhat> Red Hat >>>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>>> <https://www.facebook.com/redhat.il/> >>>>> <https://red.ht/sig> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Shreyansh Shah >>>> >>> >>> >>> -- >>> *Barak Sason Rofman* >>> >>> Gluster Storage Development >>> >>> Red Hat Israel <https://www.redhat.com/> >>> >>> 34 Jerusalem rd. Ra'anana, 43501 >>> >>> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >>> M: *+972-52-4326355* >>> @RedHat <https://twitter.com/redhat> Red Hat >>> <https://www.linkedin.com/company/red-hat> Red Hat >>> <https://www.facebook.com/redhat.il/> >>> <https://red.ht/sig> >>> >> >> >> -- >> Regards, >> Shreyansh Shah >> > > > -- > *Barak Sason Rofman* > > Gluster Storage Development > > Red Hat Israel <https://www.redhat.com/> > > 34 Jerusalem rd. Ra'anana, 43501 > > bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* > M: *+972-52-4326355* > @RedHat <https://twitter.com/redhat> Red Hat > <https://www.linkedin.com/company/red-hat> Red Hat > <https://www.facebook.com/redhat.il/> > <https://red.ht/sig> >-- Regards, Shreyansh Shah -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200707/6489fa89/attachment.html>
Artem Russakovskii
2020-Jul-08 06:02 UTC
[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance
I think it'd be extremely helpful if gluster had a feature to grab all the necessary logs/debug info (maybe a few variations depending on the bug) so that all the user would have to do is enter a simple command and have gluster generate the whole bug report, ready to be sent to to the gluster team. Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | @ArtemR <http://twitter.com/ArtemR> On Tue, Jul 7, 2020 at 1:47 AM Shreyansh Shah <shreyansh.shah at alpha-grep.com> wrote:> Sounds good, thank you. > > On Tue, Jul 7, 2020 at 2:12 PM Barak Sason Rofman <bsasonro at redhat.com> > wrote: > >> Thanks Shreyansh, >> >> I'll look into it, however I'll likely need some help from more senior >> team members to perform RCA. >> I'll update once I have new insights. >> >> My regards, >> >> On Tue, Jul 7, 2020 at 11:40 AM Shreyansh Shah < >> shreyansh.shah at alpha-grep.com> wrote: >> >>> Hi Barak, >>> Thanks for looking into this and helping me out, >>> The fix-layout was successful, and I ran a rebalance after completion of >>> fix-layout. >>> The rebalance status though did show failure for 3 nodes. >>> >>> On Tue, Jul 7, 2020 at 2:07 PM Barak Sason Rofman <bsasonro at redhat.com> >>> wrote: >>> >>>> Greetings again Shreyansh, >>>> >>>> I'm indeed seeing a lot of errors in the log file - still unsure about >>>> the RC. >>>> You mentioned that prior to running rebalance you ran fix-layout, was >>>> the fix-layout successful? >>>> Another question - did you wait until fix-layout was completed before >>>> running rebalance? >>>> >>>> My thanks, >>>> >>>> On Mon, Jul 6, 2020 at 9:33 PM Shreyansh Shah < >>>> shreyansh.shah at alpha-grep.com> wrote: >>>> >>>>> Hi, >>>>> Attaching rebalance logs >>>>> FYI, we ran "gluster rebalance fix-layout" followed by "gluster >>>>> rebalance" on 20200701 and today we again ran "gluster rebalance fix-layout" >>>>> >>>>> >>>>> PFA >>>>> >>>>> On Mon, Jul 6, 2020 at 11:08 PM Barak Sason Rofman < >>>>> bsasonro at redhat.com> wrote: >>>>> >>>>>> I think it would be best. >>>>>> As I can't say at this point where the problem is originating from, >>>>>> brick logs might also be necessary (I assume I would have a better picture >>>>>> once I have the rebalance logs). >>>>>> >>>>>> Cheers, >>>>>> >>>>>> On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah < >>>>>> shreyansh.shah at alpha-grep.com> wrote: >>>>>> >>>>>>> Hi Barak, >>>>>>> Can provide the rebalance logs. Do you require all the brick logs >>>>>>> (14 in total)? >>>>>>> >>>>>>> On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman < >>>>>>> bsasonro at redhat.com> wrote: >>>>>>> >>>>>>>> Greetings Shreyansh, >>>>>>>> >>>>>>>> Off-hand I can't come up with a reason for these failures. >>>>>>>> In order to start looking into this, access to the full rebalance >>>>>>>> logs is required (possibly brick logs as well). >>>>>>>> Can you provide those? >>>>>>>> >>>>>>>> My regards, >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah < >>>>>>>> shreyansh.shah at alpha-grep.com> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> Did anyone get a chance to look into this? >>>>>>>>> >>>>>>>>> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah < >>>>>>>>> shreyansh.shah at alpha-grep.com> wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> *We are facing "Mismatching layouts for <directory>,gfid = <id>" >>>>>>>>>> errors.* >>>>>>>>>> >>>>>>>>>> We have a distributed glusterfs 5.10, no replication, 2 bricks >>>>>>>>>> (4TB each) on each node, 7 nodes in total. We added new bricks yesterday to >>>>>>>>>> the existing setup. >>>>>>>>>> Post that we did a rebalance fix-layout and then a rebalance >>>>>>>>>> (which is currently still in progress). The status shows "failed" on >>>>>>>>>> certain bricks but "in progress" for others. Adding output for gluster >>>>>>>>>> rebalance status below. >>>>>>>>>> >>>>>>>>>> The glusterfs client logs are flooded with "Mismatching layouts >>>>>>>>>> for <directory>,gfid = <id>" >>>>>>>>>> The performance too seems to have degraded due to this, even >>>>>>>>>> basic commands like `cd` and `ls` are taking more than a minute compared to >>>>>>>>>> sub-second number before brick addition. >>>>>>>>>> Apart from that we also experienced many binaries and files >>>>>>>>>> giving error stale file handle error even though the files were present. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *gluster rebalance status :* >>>>>>>>>> >>>>>>>>>> Node Rebalanced-files size scanned failures >>>>>>>>>> skipped status run time in h:m:s >>>>>>>>>> --------- ----------- ----------- ----------- >>>>>>>>>> ----------- ----------- ------------ -------------- >>>>>>>>>> localhost 176 3.5GB 12790 >>>>>>>>>> 0 8552 in progress 21:36:01 >>>>>>>>>> 10.132.0.72 8232 394.8GB 19995 >>>>>>>>>> 21 26 failed 14:50:30 >>>>>>>>>> 10.132.0.44 12625 1.0TB 50023 >>>>>>>>>> 1 10202 in progress 21:36:00 >>>>>>>>>> 10.132.0.3 21982 956.8GB 79145 >>>>>>>>>> 1 34571 in progress 21:36:00 >>>>>>>>>> 10.132.0.9 7975 355.8GB 20157 >>>>>>>>>> 6 1522 failed 14:51:45 >>>>>>>>>> 10.132.0.73 6293 394.5GB 26414 >>>>>>>>>> 151 8085 failed 14:51:45 >>>>>>>>>> 10.132.0.70 6480 477.1GB 21058 >>>>>>>>>> 27 1787 failed 14:50:32 >>>>>>>>>> Estimated time left for rebalance to complete : 130:56:28 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Logs from one of the clients below:* >>>>>>>>>> >>>>>>>>>> [2020-07-02 12:30:14.971916] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk >>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:14.971935] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>>>> [2020-07-02 12:30:15.032013] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk >>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:15.032059] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>>>> [2020-07-02 12:30:15.032107] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk >>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:15.032153] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>>>> [2020-07-02 12:30:15.093329] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk >>>>>>>>>> layout - 2454306944 - 2761060379 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:15.093373] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>>> [2020-07-02 12:30:15.093460] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk >>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:15.093515] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>>> [2020-07-02 12:30:15.151063] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk >>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:15.151108] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>>> [2020-07-02 12:30:15.151149] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk >>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:15.151162] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>>> [2020-07-02 12:30:15.424321] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk >>>>>>>>>> layout - 920400036 - 1227153471 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:15.424380] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>>> [2020-07-02 12:30:15.424456] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk >>>>>>>>>> layout - 1840730208 - 2147483643 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:15.424484] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>>> [2020-07-02 12:30:15.424525] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk >>>>>>>>>> layout - 1533976772 - 1840730207 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:15.424542] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>>> [2020-07-02 12:30:15.424596] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk >>>>>>>>>> layout - 613646600 - 920400035 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:15.424607] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>>> [2020-07-02 12:30:16.004482] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on >>>>>>>>>> data-client-7 (hashed subvol is data-client-17) >>>>>>>>>> [2020-07-02 12:30:16.005523] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_CDS_1_DATA.dat >>>>>>>>>> [2020-07-02 12:30:16.531047] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat >>>>>>>>>> on data-client-9 (hashed subvol is data-client-19) >>>>>>>>>> [2020-07-02 12:30:16.532086] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_CDS_1_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:18.733229] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on >>>>>>>>>> data-client-17 (hashed subvol is data-client-9) >>>>>>>>>> [2020-07-02 12:30:18.734421] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_CDS_2_DATA.dat >>>>>>>>>> [2020-07-02 12:30:19.171930] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat >>>>>>>>>> on data-client-9 (hashed subvol is data-client-18) >>>>>>>>>> [2020-07-02 12:30:19.172901] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_CDS_2_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:21.028495] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on >>>>>>>>>> data-client-6 (hashed subvol is data-client-15) >>>>>>>>>> [2020-07-02 12:30:21.029836] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_EQ_2_DATA.dat >>>>>>>>>> [2020-07-02 12:30:21.127648] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat >>>>>>>>>> on data-client-11 (hashed subvol is data-client-3) >>>>>>>>>> [2020-07-02 12:30:21.128713] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_EQ_2_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:21.201126] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on >>>>>>>>>> data-client-15 (hashed subvol is data-client-7) >>>>>>>>>> [2020-07-02 12:30:21.201928] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_EQ_3_DATA.dat >>>>>>>>>> [2020-07-02 12:30:21.566158] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat >>>>>>>>>> on data-client-7 (hashed subvol is data-client-16) >>>>>>>>>> [2020-07-02 12:30:21.567123] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_EQ_3_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:21.649357] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on >>>>>>>>>> data-client-2 (hashed subvol is data-client-11) >>>>>>>>>> [2020-07-02 12:30:21.661381] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_EQ_4_DATA.dat >>>>>>>>>> [2020-07-02 12:30:21.748937] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat >>>>>>>>>> on data-client-15 (hashed subvol is data-client-7) >>>>>>>>>> [2020-07-02 12:30:21.749481] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_EQ_4_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:21.898593] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on >>>>>>>>>> data-client-14 (hashed subvol is data-client-7) >>>>>>>>>> [2020-07-02 12:30:21.899442] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_EQ_6_DATA.dat >>>>>>>>>> [2020-07-02 12:30:22.039337] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat >>>>>>>>>> on data-client-10 (hashed subvol is data-client-2) >>>>>>>>>> [2020-07-02 12:30:22.040086] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/BSE_EQ_6_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:22.501877] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat >>>>>>>>>> on data-client-15 (hashed subvol is data-client-8) >>>>>>>>>> [2020-07-02 12:30:22.502712] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_DATA.dat >>>>>>>>>> [2020-07-02 12:30:22.782577] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile >>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11 >>>>>>>>>> (hashed subvol is data-client-6) >>>>>>>>>> [2020-07-02 12:30:22.783777] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:23.146847] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on >>>>>>>>>> data-client-17 (hashed subvol is data-client-9) >>>>>>>>>> [2020-07-02 12:30:23.148009] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSECM1_DATA.dat >>>>>>>>>> [2020-07-02 12:30:23.229290] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile >>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed >>>>>>>>>> subvol is data-client-6) >>>>>>>>>> [2020-07-02 12:30:23.230151] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:23.889520] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on >>>>>>>>>> data-client-2 (hashed subvol is data-client-11) >>>>>>>>>> [2020-07-02 12:30:23.896618] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSECM2_DATA.dat >>>>>>>>>> [2020-07-02 12:30:24.093017] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile >>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed >>>>>>>>>> subvol is data-client-15) >>>>>>>>>> [2020-07-02 12:30:24.094117] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:24.345257] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on >>>>>>>>>> data-client-17 (hashed subvol is data-client-10) >>>>>>>>>> [2020-07-02 12:30:24.346234] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSECM3_DATA.dat >>>>>>>>>> [2020-07-02 12:30:24.425835] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile >>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed >>>>>>>>>> subvol is data-client-15) >>>>>>>>>> [2020-07-02 12:30:24.426880] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:25.158718] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat >>>>>>>>>> on data-client-9 (hashed subvol is data-client-19) >>>>>>>>>> [2020-07-02 12:30:25.159619] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat >>>>>>>>>> [2020-07-02 12:30:25.531479] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed >>>>>>>>>> subvol is data-client-10) >>>>>>>>>> [2020-07-02 12:30:25.540569] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:25.771692] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat >>>>>>>>>> on data-client-11 (hashed subvol is data-client-3) >>>>>>>>>> [2020-07-02 12:30:25.772610] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat >>>>>>>>>> [2020-07-02 12:30:25.866118] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15 >>>>>>>>>> (hashed subvol is data-client-8) >>>>>>>>>> [2020-07-02 12:30:25.866917] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:26.424386] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat >>>>>>>>>> on data-client-9 (hashed subvol is data-client-18) >>>>>>>>>> [2020-07-02 12:30:26.425309] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat >>>>>>>>>> [2020-07-02 12:30:26.818852] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10 >>>>>>>>>> (hashed subvol is data-client-2) >>>>>>>>>> [2020-07-02 12:30:26.819890] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:27.352405] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat >>>>>>>>>> on data-client-10 (hashed subvol is data-client-2) >>>>>>>>>> [2020-07-02 12:30:27.352914] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat >>>>>>>>>> [2020-07-02 12:30:27.521286] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed >>>>>>>>>> subvol is data-client-18) >>>>>>>>>> [2020-07-02 12:30:27.522325] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:28.566634] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat >>>>>>>>>> on data-client-2 (hashed subvol is data-client-11) >>>>>>>>>> [2020-07-02 12:30:28.579295] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat >>>>>>>>>> [2020-07-02 12:30:28.958028] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat >>>>>>>>>> on data-client-7 (hashed subvol is data-client-16) >>>>>>>>>> [2020-07-02 12:30:28.959102] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat >>>>>>>>>> [2020-07-02 12:30:29.012429] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed >>>>>>>>>> subvol is data-client-15) >>>>>>>>>> [2020-07-02 12:30:29.013416] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:29.396716] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on >>>>>>>>>> data-client-17 (hashed subvol is data-client-10) >>>>>>>>>> [2020-07-02 12:30:29.397740] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat >>>>>>>>>> [2020-07-02 12:30:29.556312] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile >>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed >>>>>>>>>> subvol is data-client-18) >>>>>>>>>> [2020-07-02 12:30:29.557197] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat >>>>>>>>>> [2020-07-02 12:30:30.605354] I [MSGID: 109045] >>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>>> deletion of stale linkfile >>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9 >>>>>>>>>> (hashed subvol is data-client-19) >>>>>>>>>> [2020-07-02 12:30:30.606117] I [MSGID: 109069] >>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat >>>>>>>>>> [2020-07-02 12:30:31.559206] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout - >>>>>>>>>> 613576736 - 920330171 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:31.559255] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99 >>>>>>>>>> [2020-07-02 12:30:31.569025] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout - >>>>>>>>>> 920330172 - 1227083607 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:31.569067] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99 >>>>>>>>>> [2020-07-02 12:30:31.701849] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout - >>>>>>>>>> 3374637116 - 3681390551 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:31.701895] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid >>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373 >>>>>>>>>> [2020-07-02 12:30:31.738464] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout - >>>>>>>>>> 3681390552 - 3988143987 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:31.738507] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid >>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373 >>>>>>>>>> [2020-07-02 12:30:31.857102] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk >>>>>>>>>> layout - 3067883680 - 3374637115 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:31.857147] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>>> [2020-07-02 12:30:31.857180] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk >>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:31.857197] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>>> [2020-07-02 12:30:31.917705] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0 >>>>>>>>>> - 306753435 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:31.917781] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>>> [2020-07-02 12:30:31.917855] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk >>>>>>>>>> layout - 3988213852 - 4294967295 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:31.917874] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>>> [2020-07-02 12:30:32.390945] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout - >>>>>>>>>> 3681460416 - 3988213851 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:32.390998] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid >>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311 >>>>>>>>>> [2020-07-02 12:30:32.391056] I [MSGID: 109064] >>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>>> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout - >>>>>>>>>> 3988213852 - 4294967295 - 4159036738 >>>>>>>>>> [2020-07-02 12:30:32.391075] I [MSGID: 109018] >>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid >>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311 >>>>>>>>>> [2020-07-02 12:33:50.915279] I [MSGID: 109066] >>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming >>>>>>>>>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2 >>>>>>>>>> (2cb54500-814d-4e85-83e7-e33d9440b18d) >>>>>>>>>> (hash=data-client-6/cache=data-client-18) => >>>>>>>>>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null)) >>>>>>>>>> (hash=data-client-6/cache=<nul>) >>>>>>>>>> [2020-07-02 12:34:09.799586] I [MSGID: 109066] >>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming >>>>>>>>>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k >>>>>>>>>> (99938ee6-6986-4123-9d72-ec09e2310b4f) >>>>>>>>>> (hash=data-client-17/cache=data-client-18) => >>>>>>>>>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null)) >>>>>>>>>> (hash=data-client-17/cache=<nul>) >>>>>>>>>> .... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Please look into this at top-priority if possible. >>>>>>>>>> Let me know if anything else is required. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Regards, >>>>>>>>>> Shreyansh Shah >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> Shreyansh Shah >>>>>>>>> ________ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Community Meeting Calendar: >>>>>>>>> >>>>>>>>> Schedule - >>>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>>>>>>>> Bridge: https://bluejeans.com/441850968 >>>>>>>>> >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Barak Sason Rofman* >>>>>>>> >>>>>>>> Gluster Storage Development >>>>>>>> >>>>>>>> Red Hat Israel <https://www.redhat.com/> >>>>>>>> >>>>>>>> 34 Jerusalem rd. Ra'anana, 43501 >>>>>>>> >>>>>>>> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >>>>>>>> M: *+972-52-4326355* >>>>>>>> @RedHat <https://twitter.com/redhat> Red Hat >>>>>>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>>>>>> <https://www.facebook.com/redhat.il/> >>>>>>>> <https://red.ht/sig> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Shreyansh Shah >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Barak Sason Rofman* >>>>>> >>>>>> Gluster Storage Development >>>>>> >>>>>> Red Hat Israel <https://www.redhat.com/> >>>>>> >>>>>> 34 Jerusalem rd. Ra'anana, 43501 >>>>>> >>>>>> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >>>>>> M: *+972-52-4326355* >>>>>> @RedHat <https://twitter.com/redhat> Red Hat >>>>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>>>> <https://www.facebook.com/redhat.il/> >>>>>> <https://red.ht/sig> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Shreyansh Shah >>>>> >>>> >>>> >>>> -- >>>> *Barak Sason Rofman* >>>> >>>> Gluster Storage Development >>>> >>>> Red Hat Israel <https://www.redhat.com/> >>>> >>>> 34 Jerusalem rd. Ra'anana, 43501 >>>> >>>> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >>>> M: *+972-52-4326355* >>>> @RedHat <https://twitter.com/redhat> Red Hat >>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>> <https://www.facebook.com/redhat.il/> >>>> <https://red.ht/sig> >>>> >>> >>> >>> -- >>> Regards, >>> Shreyansh Shah >>> >> >> >> -- >> *Barak Sason Rofman* >> >> Gluster Storage Development >> >> Red Hat Israel <https://www.redhat.com/> >> >> 34 Jerusalem rd. Ra'anana, 43501 >> >> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >> M: *+972-52-4326355* >> @RedHat <https://twitter.com/redhat> Red Hat >> <https://www.linkedin.com/company/red-hat> Red Hat >> <https://www.facebook.com/redhat.il/> >> <https://red.ht/sig> >> > > > -- > Regards, > Shreyansh Shah > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200707/47b1369d/attachment.html>