Barak Sason Rofman
2020-Jul-07 08:42 UTC
[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance
Thanks Shreyansh, I'll look into it, however I'll likely need some help from more senior team members to perform RCA. I'll update once I have new insights. My regards, On Tue, Jul 7, 2020 at 11:40 AM Shreyansh Shah < shreyansh.shah at alpha-grep.com> wrote:> Hi Barak, > Thanks for looking into this and helping me out, > The fix-layout was successful, and I ran a rebalance after completion of > fix-layout. > The rebalance status though did show failure for 3 nodes. > > On Tue, Jul 7, 2020 at 2:07 PM Barak Sason Rofman <bsasonro at redhat.com> > wrote: > >> Greetings again Shreyansh, >> >> I'm indeed seeing a lot of errors in the log file - still unsure about >> the RC. >> You mentioned that prior to running rebalance you ran fix-layout, was the >> fix-layout successful? >> Another question - did you wait until fix-layout was completed before >> running rebalance? >> >> My thanks, >> >> On Mon, Jul 6, 2020 at 9:33 PM Shreyansh Shah < >> shreyansh.shah at alpha-grep.com> wrote: >> >>> Hi, >>> Attaching rebalance logs >>> FYI, we ran "gluster rebalance fix-layout" followed by "gluster >>> rebalance" on 20200701 and today we again ran "gluster rebalance fix-layout" >>> >>> >>> PFA >>> >>> On Mon, Jul 6, 2020 at 11:08 PM Barak Sason Rofman <bsasonro at redhat.com> >>> wrote: >>> >>>> I think it would be best. >>>> As I can't say at this point where the problem is originating from, >>>> brick logs might also be necessary (I assume I would have a better picture >>>> once I have the rebalance logs). >>>> >>>> Cheers, >>>> >>>> On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah < >>>> shreyansh.shah at alpha-grep.com> wrote: >>>> >>>>> Hi Barak, >>>>> Can provide the rebalance logs. Do you require all the brick logs (14 >>>>> in total)? >>>>> >>>>> On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman < >>>>> bsasonro at redhat.com> wrote: >>>>> >>>>>> Greetings Shreyansh, >>>>>> >>>>>> Off-hand I can't come up with a reason for these failures. >>>>>> In order to start looking into this, access to the full rebalance >>>>>> logs is required (possibly brick logs as well). >>>>>> Can you provide those? >>>>>> >>>>>> My regards, >>>>>> >>>>>> >>>>>> On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah < >>>>>> shreyansh.shah at alpha-grep.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> Did anyone get a chance to look into this? >>>>>>> >>>>>>> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah < >>>>>>> shreyansh.shah at alpha-grep.com> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> *We are facing "Mismatching layouts for <directory>,gfid = <id>" >>>>>>>> errors.* >>>>>>>> >>>>>>>> We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB >>>>>>>> each) on each node, 7 nodes in total. We added new bricks yesterday to the >>>>>>>> existing setup. >>>>>>>> Post that we did a rebalance fix-layout and then a rebalance (which >>>>>>>> is currently still in progress). The status shows "failed" on certain >>>>>>>> bricks but "in progress" for others. Adding output for gluster rebalance >>>>>>>> status below. >>>>>>>> >>>>>>>> The glusterfs client logs are flooded with "Mismatching layouts for >>>>>>>> <directory>,gfid = <id>" >>>>>>>> The performance too seems to have degraded due to this, even basic >>>>>>>> commands like `cd` and `ls` are taking more than a minute compared to >>>>>>>> sub-second number before brick addition. >>>>>>>> Apart from that we also experienced many binaries and files giving >>>>>>>> error stale file handle error even though the files were present. >>>>>>>> >>>>>>>> >>>>>>>> *gluster rebalance status :* >>>>>>>> >>>>>>>> Node Rebalanced-files size scanned failures >>>>>>>> skipped status run time in h:m:s >>>>>>>> --------- ----------- ----------- ----------- >>>>>>>> ----------- ----------- ------------ -------------- >>>>>>>> localhost 176 3.5GB 12790 >>>>>>>> 0 8552 in progress 21:36:01 >>>>>>>> 10.132.0.72 8232 394.8GB 19995 >>>>>>>> 21 26 failed 14:50:30 >>>>>>>> 10.132.0.44 12625 1.0TB 50023 >>>>>>>> 1 10202 in progress 21:36:00 >>>>>>>> 10.132.0.3 21982 956.8GB 79145 >>>>>>>> 1 34571 in progress 21:36:00 >>>>>>>> 10.132.0.9 7975 355.8GB 20157 >>>>>>>> 6 1522 failed 14:51:45 >>>>>>>> 10.132.0.73 6293 394.5GB 26414 >>>>>>>> 151 8085 failed 14:51:45 >>>>>>>> 10.132.0.70 6480 477.1GB 21058 >>>>>>>> 27 1787 failed 14:50:32 >>>>>>>> Estimated time left for rebalance to complete : 130:56:28 >>>>>>>> >>>>>>>> >>>>>>>> *Logs from one of the clients below:* >>>>>>>> >>>>>>>> [2020-07-02 12:30:14.971916] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk >>>>>>>> layout - 2761060380 - 3067813815 - 4159036738 >>>>>>>> [2020-07-02 12:30:14.971935] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>> [2020-07-02 12:30:15.032013] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk >>>>>>>> layout - 3681390552 - 3988143987 - 4159036738 >>>>>>>> [2020-07-02 12:30:15.032059] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>> [2020-07-02 12:30:15.032107] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk >>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>> [2020-07-02 12:30:15.032153] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>> [2020-07-02 12:30:15.093329] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk >>>>>>>> layout - 2454306944 - 2761060379 - 4159036738 >>>>>>>> [2020-07-02 12:30:15.093373] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>> [2020-07-02 12:30:15.093460] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk >>>>>>>> layout - 2761060380 - 3067813815 - 4159036738 >>>>>>>> [2020-07-02 12:30:15.093515] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>> [2020-07-02 12:30:15.151063] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk >>>>>>>> layout - 3681390552 - 3988143987 - 4159036738 >>>>>>>> [2020-07-02 12:30:15.151108] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>> [2020-07-02 12:30:15.151149] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk >>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>> [2020-07-02 12:30:15.151162] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>> [2020-07-02 12:30:15.424321] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk >>>>>>>> layout - 920400036 - 1227153471 - 4159036738 >>>>>>>> [2020-07-02 12:30:15.424380] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>> [2020-07-02 12:30:15.424456] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk >>>>>>>> layout - 1840730208 - 2147483643 - 4159036738 >>>>>>>> [2020-07-02 12:30:15.424484] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>> [2020-07-02 12:30:15.424525] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk >>>>>>>> layout - 1533976772 - 1840730207 - 4159036738 >>>>>>>> [2020-07-02 12:30:15.424542] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>> [2020-07-02 12:30:15.424596] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk >>>>>>>> layout - 613646600 - 920400035 - 4159036738 >>>>>>>> [2020-07-02 12:30:15.424607] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>> [2020-07-02 12:30:16.004482] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on >>>>>>>> data-client-7 (hashed subvol is data-client-17) >>>>>>>> [2020-07-02 12:30:16.005523] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_CDS_1_DATA.dat >>>>>>>> [2020-07-02 12:30:16.531047] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat >>>>>>>> on data-client-9 (hashed subvol is data-client-19) >>>>>>>> [2020-07-02 12:30:16.532086] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_CDS_1_METADATA.dat >>>>>>>> [2020-07-02 12:30:18.733229] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on >>>>>>>> data-client-17 (hashed subvol is data-client-9) >>>>>>>> [2020-07-02 12:30:18.734421] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_CDS_2_DATA.dat >>>>>>>> [2020-07-02 12:30:19.171930] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat >>>>>>>> on data-client-9 (hashed subvol is data-client-18) >>>>>>>> [2020-07-02 12:30:19.172901] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_CDS_2_METADATA.dat >>>>>>>> [2020-07-02 12:30:21.028495] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on >>>>>>>> data-client-6 (hashed subvol is data-client-15) >>>>>>>> [2020-07-02 12:30:21.029836] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_EQ_2_DATA.dat >>>>>>>> [2020-07-02 12:30:21.127648] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat >>>>>>>> on data-client-11 (hashed subvol is data-client-3) >>>>>>>> [2020-07-02 12:30:21.128713] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_EQ_2_METADATA.dat >>>>>>>> [2020-07-02 12:30:21.201126] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on >>>>>>>> data-client-15 (hashed subvol is data-client-7) >>>>>>>> [2020-07-02 12:30:21.201928] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_EQ_3_DATA.dat >>>>>>>> [2020-07-02 12:30:21.566158] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat >>>>>>>> on data-client-7 (hashed subvol is data-client-16) >>>>>>>> [2020-07-02 12:30:21.567123] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_EQ_3_METADATA.dat >>>>>>>> [2020-07-02 12:30:21.649357] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on >>>>>>>> data-client-2 (hashed subvol is data-client-11) >>>>>>>> [2020-07-02 12:30:21.661381] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_EQ_4_DATA.dat >>>>>>>> [2020-07-02 12:30:21.748937] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat >>>>>>>> on data-client-15 (hashed subvol is data-client-7) >>>>>>>> [2020-07-02 12:30:21.749481] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_EQ_4_METADATA.dat >>>>>>>> [2020-07-02 12:30:21.898593] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on >>>>>>>> data-client-14 (hashed subvol is data-client-7) >>>>>>>> [2020-07-02 12:30:21.899442] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_EQ_6_DATA.dat >>>>>>>> [2020-07-02 12:30:22.039337] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat >>>>>>>> on data-client-10 (hashed subvol is data-client-2) >>>>>>>> [2020-07-02 12:30:22.040086] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/BSE_EQ_6_METADATA.dat >>>>>>>> [2020-07-02 12:30:22.501877] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat >>>>>>>> on data-client-15 (hashed subvol is data-client-8) >>>>>>>> [2020-07-02 12:30:22.502712] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSECDS1_DATA.dat >>>>>>>> [2020-07-02 12:30:22.782577] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile >>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11 >>>>>>>> (hashed subvol is data-client-6) >>>>>>>> [2020-07-02 12:30:22.783777] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat >>>>>>>> [2020-07-02 12:30:23.146847] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on >>>>>>>> data-client-17 (hashed subvol is data-client-9) >>>>>>>> [2020-07-02 12:30:23.148009] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSECM1_DATA.dat >>>>>>>> [2020-07-02 12:30:23.229290] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile >>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed >>>>>>>> subvol is data-client-6) >>>>>>>> [2020-07-02 12:30:23.230151] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat >>>>>>>> [2020-07-02 12:30:23.889520] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on >>>>>>>> data-client-2 (hashed subvol is data-client-11) >>>>>>>> [2020-07-02 12:30:23.896618] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSECM2_DATA.dat >>>>>>>> [2020-07-02 12:30:24.093017] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile >>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed >>>>>>>> subvol is data-client-15) >>>>>>>> [2020-07-02 12:30:24.094117] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat >>>>>>>> [2020-07-02 12:30:24.345257] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on >>>>>>>> data-client-17 (hashed subvol is data-client-10) >>>>>>>> [2020-07-02 12:30:24.346234] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSECM3_DATA.dat >>>>>>>> [2020-07-02 12:30:24.425835] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile >>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed >>>>>>>> subvol is data-client-15) >>>>>>>> [2020-07-02 12:30:24.426880] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat >>>>>>>> [2020-07-02 12:30:25.158718] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat >>>>>>>> on data-client-9 (hashed subvol is data-client-19) >>>>>>>> [2020-07-02 12:30:25.159619] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat >>>>>>>> [2020-07-02 12:30:25.531479] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile >>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed >>>>>>>> subvol is data-client-10) >>>>>>>> [2020-07-02 12:30:25.540569] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat >>>>>>>> [2020-07-02 12:30:25.771692] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat >>>>>>>> on data-client-11 (hashed subvol is data-client-3) >>>>>>>> [2020-07-02 12:30:25.772610] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat >>>>>>>> [2020-07-02 12:30:25.866118] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile >>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15 >>>>>>>> (hashed subvol is data-client-8) >>>>>>>> [2020-07-02 12:30:25.866917] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat >>>>>>>> [2020-07-02 12:30:26.424386] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat >>>>>>>> on data-client-9 (hashed subvol is data-client-18) >>>>>>>> [2020-07-02 12:30:26.425309] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat >>>>>>>> [2020-07-02 12:30:26.818852] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile >>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10 >>>>>>>> (hashed subvol is data-client-2) >>>>>>>> [2020-07-02 12:30:26.819890] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat >>>>>>>> [2020-07-02 12:30:27.352405] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat >>>>>>>> on data-client-10 (hashed subvol is data-client-2) >>>>>>>> [2020-07-02 12:30:27.352914] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat >>>>>>>> [2020-07-02 12:30:27.521286] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile >>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed >>>>>>>> subvol is data-client-18) >>>>>>>> [2020-07-02 12:30:27.522325] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat >>>>>>>> [2020-07-02 12:30:28.566634] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat >>>>>>>> on data-client-2 (hashed subvol is data-client-11) >>>>>>>> [2020-07-02 12:30:28.579295] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat >>>>>>>> [2020-07-02 12:30:28.958028] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat >>>>>>>> on data-client-7 (hashed subvol is data-client-16) >>>>>>>> [2020-07-02 12:30:28.959102] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat >>>>>>>> [2020-07-02 12:30:29.012429] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile >>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed >>>>>>>> subvol is data-client-15) >>>>>>>> [2020-07-02 12:30:29.013416] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat >>>>>>>> [2020-07-02 12:30:29.396716] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on >>>>>>>> data-client-17 (hashed subvol is data-client-10) >>>>>>>> [2020-07-02 12:30:29.397740] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat >>>>>>>> [2020-07-02 12:30:29.556312] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile >>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed >>>>>>>> subvol is data-client-18) >>>>>>>> [2020-07-02 12:30:29.557197] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat >>>>>>>> [2020-07-02 12:30:30.605354] I [MSGID: 109045] >>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>> deletion of stale linkfile >>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9 >>>>>>>> (hashed subvol is data-client-19) >>>>>>>> [2020-07-02 12:30:30.606117] I [MSGID: 109069] >>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat >>>>>>>> [2020-07-02 12:30:31.559206] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout - >>>>>>>> 613576736 - 920330171 - 4159036738 >>>>>>>> [2020-07-02 12:30:31.559255] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99 >>>>>>>> [2020-07-02 12:30:31.569025] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout - >>>>>>>> 920330172 - 1227083607 - 4159036738 >>>>>>>> [2020-07-02 12:30:31.569067] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99 >>>>>>>> [2020-07-02 12:30:31.701849] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout - >>>>>>>> 3374637116 - 3681390551 - 4159036738 >>>>>>>> [2020-07-02 12:30:31.701895] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /processed_data/Indexes/NSEINDEX, gfid >>>>>>>> fff324f2-f855-4881-b77c-81e856522373 >>>>>>>> [2020-07-02 12:30:31.738464] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout - >>>>>>>> 3681390552 - 3988143987 - 4159036738 >>>>>>>> [2020-07-02 12:30:31.738507] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /processed_data/Indexes/NSEINDEX, gfid >>>>>>>> fff324f2-f855-4881-b77c-81e856522373 >>>>>>>> [2020-07-02 12:30:31.857102] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk >>>>>>>> layout - 3067883680 - 3374637115 - 4159036738 >>>>>>>> [2020-07-02 12:30:31.857147] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>> [2020-07-02 12:30:31.857180] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk >>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>> [2020-07-02 12:30:31.857197] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>> [2020-07-02 12:30:31.917705] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0 >>>>>>>> - 306753435 - 4159036738 >>>>>>>> [2020-07-02 12:30:31.917781] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>> [2020-07-02 12:30:31.917855] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk >>>>>>>> layout - 3988213852 - 4294967295 - 4159036738 >>>>>>>> [2020-07-02 12:30:31.917874] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>> [2020-07-02 12:30:32.390945] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout - >>>>>>>> 3681460416 - 3988213851 - 4159036738 >>>>>>>> [2020-07-02 12:30:32.390998] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid >>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311 >>>>>>>> [2020-07-02 12:30:32.391056] I [MSGID: 109064] >>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout - >>>>>>>> 3988213852 - 4294967295 - 4159036738 >>>>>>>> [2020-07-02 12:30:32.391075] I [MSGID: 109018] >>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid >>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311 >>>>>>>> [2020-07-02 12:33:50.915279] I [MSGID: 109066] >>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming >>>>>>>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2 >>>>>>>> (2cb54500-814d-4e85-83e7-e33d9440b18d) >>>>>>>> (hash=data-client-6/cache=data-client-18) => >>>>>>>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null)) >>>>>>>> (hash=data-client-6/cache=<nul>) >>>>>>>> [2020-07-02 12:34:09.799586] I [MSGID: 109066] >>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming >>>>>>>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k >>>>>>>> (99938ee6-6986-4123-9d72-ec09e2310b4f) >>>>>>>> (hash=data-client-17/cache=data-client-18) => >>>>>>>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null)) >>>>>>>> (hash=data-client-17/cache=<nul>) >>>>>>>> .... >>>>>>>> >>>>>>>> >>>>>>>> Please look into this at top-priority if possible. >>>>>>>> Let me know if anything else is required. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Shreyansh Shah >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Shreyansh Shah >>>>>>> ________ >>>>>>> >>>>>>> >>>>>>> >>>>>>> Community Meeting Calendar: >>>>>>> >>>>>>> Schedule - >>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>>>>>> Bridge: https://bluejeans.com/441850968 >>>>>>> >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Barak Sason Rofman* >>>>>> >>>>>> Gluster Storage Development >>>>>> >>>>>> Red Hat Israel <https://www.redhat.com/> >>>>>> >>>>>> 34 Jerusalem rd. Ra'anana, 43501 >>>>>> >>>>>> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >>>>>> M: *+972-52-4326355* >>>>>> @RedHat <https://twitter.com/redhat> Red Hat >>>>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>>>> <https://www.facebook.com/redhat.il/> >>>>>> <https://red.ht/sig> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Shreyansh Shah >>>>> >>>> >>>> >>>> -- >>>> *Barak Sason Rofman* >>>> >>>> Gluster Storage Development >>>> >>>> Red Hat Israel <https://www.redhat.com/> >>>> >>>> 34 Jerusalem rd. Ra'anana, 43501 >>>> >>>> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >>>> M: *+972-52-4326355* >>>> @RedHat <https://twitter.com/redhat> Red Hat >>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>> <https://www.facebook.com/redhat.il/> >>>> <https://red.ht/sig> >>>> >>> >>> >>> -- >>> Regards, >>> Shreyansh Shah >>> >> >> >> -- >> *Barak Sason Rofman* >> >> Gluster Storage Development >> >> Red Hat Israel <https://www.redhat.com/> >> >> 34 Jerusalem rd. Ra'anana, 43501 >> >> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >> M: *+972-52-4326355* >> @RedHat <https://twitter.com/redhat> Red Hat >> <https://www.linkedin.com/company/red-hat> Red Hat >> <https://www.facebook.com/redhat.il/> >> <https://red.ht/sig> >> > > > -- > Regards, > Shreyansh Shah >-- *Barak Sason Rofman* Gluster Storage Development Red Hat Israel <https://www.redhat.com/> 34 Jerusalem rd. Ra'anana, 43501 bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* M: *+972-52-4326355* @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/redhat.il/> <https://red.ht/sig> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200707/4253c597/attachment.html>
Shreyansh Shah
2020-Jul-07 08:46 UTC
[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance
Sounds good, thank you. On Tue, Jul 7, 2020 at 2:12 PM Barak Sason Rofman <bsasonro at redhat.com> wrote:> Thanks Shreyansh, > > I'll look into it, however I'll likely need some help from more senior > team members to perform RCA. > I'll update once I have new insights. > > My regards, > > On Tue, Jul 7, 2020 at 11:40 AM Shreyansh Shah < > shreyansh.shah at alpha-grep.com> wrote: > >> Hi Barak, >> Thanks for looking into this and helping me out, >> The fix-layout was successful, and I ran a rebalance after completion of >> fix-layout. >> The rebalance status though did show failure for 3 nodes. >> >> On Tue, Jul 7, 2020 at 2:07 PM Barak Sason Rofman <bsasonro at redhat.com> >> wrote: >> >>> Greetings again Shreyansh, >>> >>> I'm indeed seeing a lot of errors in the log file - still unsure about >>> the RC. >>> You mentioned that prior to running rebalance you ran fix-layout, was >>> the fix-layout successful? >>> Another question - did you wait until fix-layout was completed before >>> running rebalance? >>> >>> My thanks, >>> >>> On Mon, Jul 6, 2020 at 9:33 PM Shreyansh Shah < >>> shreyansh.shah at alpha-grep.com> wrote: >>> >>>> Hi, >>>> Attaching rebalance logs >>>> FYI, we ran "gluster rebalance fix-layout" followed by "gluster >>>> rebalance" on 20200701 and today we again ran "gluster rebalance fix-layout" >>>> >>>> >>>> PFA >>>> >>>> On Mon, Jul 6, 2020 at 11:08 PM Barak Sason Rofman <bsasonro at redhat.com> >>>> wrote: >>>> >>>>> I think it would be best. >>>>> As I can't say at this point where the problem is originating from, >>>>> brick logs might also be necessary (I assume I would have a better picture >>>>> once I have the rebalance logs). >>>>> >>>>> Cheers, >>>>> >>>>> On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah < >>>>> shreyansh.shah at alpha-grep.com> wrote: >>>>> >>>>>> Hi Barak, >>>>>> Can provide the rebalance logs. Do you require all the brick logs (14 >>>>>> in total)? >>>>>> >>>>>> On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman < >>>>>> bsasonro at redhat.com> wrote: >>>>>> >>>>>>> Greetings Shreyansh, >>>>>>> >>>>>>> Off-hand I can't come up with a reason for these failures. >>>>>>> In order to start looking into this, access to the full rebalance >>>>>>> logs is required (possibly brick logs as well). >>>>>>> Can you provide those? >>>>>>> >>>>>>> My regards, >>>>>>> >>>>>>> >>>>>>> On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah < >>>>>>> shreyansh.shah at alpha-grep.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> Did anyone get a chance to look into this? >>>>>>>> >>>>>>>> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah < >>>>>>>> shreyansh.shah at alpha-grep.com> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> *We are facing "Mismatching layouts for <directory>,gfid = <id>" >>>>>>>>> errors.* >>>>>>>>> >>>>>>>>> We have a distributed glusterfs 5.10, no replication, 2 bricks >>>>>>>>> (4TB each) on each node, 7 nodes in total. We added new bricks yesterday to >>>>>>>>> the existing setup. >>>>>>>>> Post that we did a rebalance fix-layout and then a rebalance >>>>>>>>> (which is currently still in progress). The status shows "failed" on >>>>>>>>> certain bricks but "in progress" for others. Adding output for gluster >>>>>>>>> rebalance status below. >>>>>>>>> >>>>>>>>> The glusterfs client logs are flooded with "Mismatching layouts >>>>>>>>> for <directory>,gfid = <id>" >>>>>>>>> The performance too seems to have degraded due to this, even basic >>>>>>>>> commands like `cd` and `ls` are taking more than a minute compared to >>>>>>>>> sub-second number before brick addition. >>>>>>>>> Apart from that we also experienced many binaries and files giving >>>>>>>>> error stale file handle error even though the files were present. >>>>>>>>> >>>>>>>>> >>>>>>>>> *gluster rebalance status :* >>>>>>>>> >>>>>>>>> Node Rebalanced-files size scanned failures >>>>>>>>> skipped status run time in h:m:s >>>>>>>>> --------- ----------- ----------- ----------- >>>>>>>>> ----------- ----------- ------------ -------------- >>>>>>>>> localhost 176 3.5GB 12790 >>>>>>>>> 0 8552 in progress 21:36:01 >>>>>>>>> 10.132.0.72 8232 394.8GB 19995 >>>>>>>>> 21 26 failed 14:50:30 >>>>>>>>> 10.132.0.44 12625 1.0TB 50023 >>>>>>>>> 1 10202 in progress 21:36:00 >>>>>>>>> 10.132.0.3 21982 956.8GB 79145 >>>>>>>>> 1 34571 in progress 21:36:00 >>>>>>>>> 10.132.0.9 7975 355.8GB 20157 >>>>>>>>> 6 1522 failed 14:51:45 >>>>>>>>> 10.132.0.73 6293 394.5GB 26414 >>>>>>>>> 151 8085 failed 14:51:45 >>>>>>>>> 10.132.0.70 6480 477.1GB 21058 >>>>>>>>> 27 1787 failed 14:50:32 >>>>>>>>> Estimated time left for rebalance to complete : 130:56:28 >>>>>>>>> >>>>>>>>> >>>>>>>>> *Logs from one of the clients below:* >>>>>>>>> >>>>>>>>> [2020-07-02 12:30:14.971916] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk >>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738 >>>>>>>>> [2020-07-02 12:30:14.971935] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>>> [2020-07-02 12:30:15.032013] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk >>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.032059] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>>> [2020-07-02 12:30:15.032107] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk >>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.032153] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc >>>>>>>>> [2020-07-02 12:30:15.093329] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk >>>>>>>>> layout - 2454306944 - 2761060379 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.093373] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>> [2020-07-02 12:30:15.093460] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk >>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.093515] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>> [2020-07-02 12:30:15.151063] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk >>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.151108] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>> [2020-07-02 12:30:15.151149] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk >>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.151162] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222 >>>>>>>>> [2020-07-02 12:30:15.424321] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk >>>>>>>>> layout - 920400036 - 1227153471 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.424380] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>> [2020-07-02 12:30:15.424456] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk >>>>>>>>> layout - 1840730208 - 2147483643 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.424484] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>> [2020-07-02 12:30:15.424525] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk >>>>>>>>> layout - 1533976772 - 1840730207 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.424542] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>> [2020-07-02 12:30:15.424596] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk >>>>>>>>> layout - 613646600 - 920400035 - 4159036738 >>>>>>>>> [2020-07-02 12:30:15.424607] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1 >>>>>>>>> [2020-07-02 12:30:16.004482] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on >>>>>>>>> data-client-7 (hashed subvol is data-client-17) >>>>>>>>> [2020-07-02 12:30:16.005523] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_CDS_1_DATA.dat >>>>>>>>> [2020-07-02 12:30:16.531047] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat >>>>>>>>> on data-client-9 (hashed subvol is data-client-19) >>>>>>>>> [2020-07-02 12:30:16.532086] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_CDS_1_METADATA.dat >>>>>>>>> [2020-07-02 12:30:18.733229] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on >>>>>>>>> data-client-17 (hashed subvol is data-client-9) >>>>>>>>> [2020-07-02 12:30:18.734421] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_CDS_2_DATA.dat >>>>>>>>> [2020-07-02 12:30:19.171930] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat >>>>>>>>> on data-client-9 (hashed subvol is data-client-18) >>>>>>>>> [2020-07-02 12:30:19.172901] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_CDS_2_METADATA.dat >>>>>>>>> [2020-07-02 12:30:21.028495] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on >>>>>>>>> data-client-6 (hashed subvol is data-client-15) >>>>>>>>> [2020-07-02 12:30:21.029836] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_2_DATA.dat >>>>>>>>> [2020-07-02 12:30:21.127648] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat >>>>>>>>> on data-client-11 (hashed subvol is data-client-3) >>>>>>>>> [2020-07-02 12:30:21.128713] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_2_METADATA.dat >>>>>>>>> [2020-07-02 12:30:21.201126] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on >>>>>>>>> data-client-15 (hashed subvol is data-client-7) >>>>>>>>> [2020-07-02 12:30:21.201928] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_3_DATA.dat >>>>>>>>> [2020-07-02 12:30:21.566158] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat >>>>>>>>> on data-client-7 (hashed subvol is data-client-16) >>>>>>>>> [2020-07-02 12:30:21.567123] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_3_METADATA.dat >>>>>>>>> [2020-07-02 12:30:21.649357] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on >>>>>>>>> data-client-2 (hashed subvol is data-client-11) >>>>>>>>> [2020-07-02 12:30:21.661381] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_4_DATA.dat >>>>>>>>> [2020-07-02 12:30:21.748937] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat >>>>>>>>> on data-client-15 (hashed subvol is data-client-7) >>>>>>>>> [2020-07-02 12:30:21.749481] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_4_METADATA.dat >>>>>>>>> [2020-07-02 12:30:21.898593] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on >>>>>>>>> data-client-14 (hashed subvol is data-client-7) >>>>>>>>> [2020-07-02 12:30:21.899442] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_6_DATA.dat >>>>>>>>> [2020-07-02 12:30:22.039337] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat >>>>>>>>> on data-client-10 (hashed subvol is data-client-2) >>>>>>>>> [2020-07-02 12:30:22.040086] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/BSE_EQ_6_METADATA.dat >>>>>>>>> [2020-07-02 12:30:22.501877] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat >>>>>>>>> on data-client-15 (hashed subvol is data-client-8) >>>>>>>>> [2020-07-02 12:30:22.502712] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECDS1_DATA.dat >>>>>>>>> [2020-07-02 12:30:22.782577] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11 >>>>>>>>> (hashed subvol is data-client-6) >>>>>>>>> [2020-07-02 12:30:22.783777] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat >>>>>>>>> [2020-07-02 12:30:23.146847] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on >>>>>>>>> data-client-17 (hashed subvol is data-client-9) >>>>>>>>> [2020-07-02 12:30:23.148009] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM1_DATA.dat >>>>>>>>> [2020-07-02 12:30:23.229290] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed >>>>>>>>> subvol is data-client-6) >>>>>>>>> [2020-07-02 12:30:23.230151] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat >>>>>>>>> [2020-07-02 12:30:23.889520] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on >>>>>>>>> data-client-2 (hashed subvol is data-client-11) >>>>>>>>> [2020-07-02 12:30:23.896618] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM2_DATA.dat >>>>>>>>> [2020-07-02 12:30:24.093017] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed >>>>>>>>> subvol is data-client-15) >>>>>>>>> [2020-07-02 12:30:24.094117] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat >>>>>>>>> [2020-07-02 12:30:24.345257] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on >>>>>>>>> data-client-17 (hashed subvol is data-client-10) >>>>>>>>> [2020-07-02 12:30:24.346234] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM3_DATA.dat >>>>>>>>> [2020-07-02 12:30:24.425835] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed >>>>>>>>> subvol is data-client-15) >>>>>>>>> [2020-07-02 12:30:24.426880] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat >>>>>>>>> [2020-07-02 12:30:25.158718] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat >>>>>>>>> on data-client-9 (hashed subvol is data-client-19) >>>>>>>>> [2020-07-02 12:30:25.159619] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat >>>>>>>>> [2020-07-02 12:30:25.531479] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed >>>>>>>>> subvol is data-client-10) >>>>>>>>> [2020-07-02 12:30:25.540569] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat >>>>>>>>> [2020-07-02 12:30:25.771692] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat >>>>>>>>> on data-client-11 (hashed subvol is data-client-3) >>>>>>>>> [2020-07-02 12:30:25.772610] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat >>>>>>>>> [2020-07-02 12:30:25.866118] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15 >>>>>>>>> (hashed subvol is data-client-8) >>>>>>>>> [2020-07-02 12:30:25.866917] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat >>>>>>>>> [2020-07-02 12:30:26.424386] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat >>>>>>>>> on data-client-9 (hashed subvol is data-client-18) >>>>>>>>> [2020-07-02 12:30:26.425309] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat >>>>>>>>> [2020-07-02 12:30:26.818852] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10 >>>>>>>>> (hashed subvol is data-client-2) >>>>>>>>> [2020-07-02 12:30:26.819890] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat >>>>>>>>> [2020-07-02 12:30:27.352405] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat >>>>>>>>> on data-client-10 (hashed subvol is data-client-2) >>>>>>>>> [2020-07-02 12:30:27.352914] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat >>>>>>>>> [2020-07-02 12:30:27.521286] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed >>>>>>>>> subvol is data-client-18) >>>>>>>>> [2020-07-02 12:30:27.522325] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat >>>>>>>>> [2020-07-02 12:30:28.566634] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat >>>>>>>>> on data-client-2 (hashed subvol is data-client-11) >>>>>>>>> [2020-07-02 12:30:28.579295] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat >>>>>>>>> [2020-07-02 12:30:28.958028] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat >>>>>>>>> on data-client-7 (hashed subvol is data-client-16) >>>>>>>>> [2020-07-02 12:30:28.959102] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat >>>>>>>>> [2020-07-02 12:30:29.012429] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed >>>>>>>>> subvol is data-client-15) >>>>>>>>> [2020-07-02 12:30:29.013416] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat >>>>>>>>> [2020-07-02 12:30:29.396716] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on >>>>>>>>> data-client-17 (hashed subvol is data-client-10) >>>>>>>>> [2020-07-02 12:30:29.397740] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat >>>>>>>>> [2020-07-02 12:30:29.556312] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed >>>>>>>>> subvol is data-client-18) >>>>>>>>> [2020-07-02 12:30:29.557197] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat >>>>>>>>> [2020-07-02 12:30:30.605354] I [MSGID: 109045] >>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting >>>>>>>>> deletion of stale linkfile >>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9 >>>>>>>>> (hashed subvol is data-client-19) >>>>>>>>> [2020-07-02 12:30:30.606117] I [MSGID: 109069] >>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink >>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for >>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat >>>>>>>>> [2020-07-02 12:30:31.559206] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout - >>>>>>>>> 613576736 - 920330171 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.559255] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99 >>>>>>>>> [2020-07-02 12:30:31.569025] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout - >>>>>>>>> 920330172 - 1227083607 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.569067] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99 >>>>>>>>> [2020-07-02 12:30:31.701849] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout - >>>>>>>>> 3374637116 - 3681390551 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.701895] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid >>>>>>>>> fff324f2-f855-4881-b77c-81e856522373 >>>>>>>>> [2020-07-02 12:30:31.738464] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout - >>>>>>>>> 3681390552 - 3988143987 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.738507] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid >>>>>>>>> fff324f2-f855-4881-b77c-81e856522373 >>>>>>>>> [2020-07-02 12:30:31.857102] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk >>>>>>>>> layout - 3067883680 - 3374637115 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.857147] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>> [2020-07-02 12:30:31.857180] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk >>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.857197] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>> [2020-07-02 12:30:31.917705] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0 >>>>>>>>> - 306753435 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.917781] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>> [2020-07-02 12:30:31.917855] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk >>>>>>>>> layout - 3988213852 - 4294967295 - 4159036738 >>>>>>>>> [2020-07-02 12:30:31.917874] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid >>>>>>>>> f8447150-4801-4188-add9-ea295bb88729 >>>>>>>>> [2020-07-02 12:30:32.390945] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout - >>>>>>>>> 3681460416 - 3988213851 - 4159036738 >>>>>>>>> [2020-07-02 12:30:32.390998] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid >>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311 >>>>>>>>> [2020-07-02 12:30:32.391056] I [MSGID: 109064] >>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: >>>>>>>>> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout - >>>>>>>>> 3988213852 - 4294967295 - 4159036738 >>>>>>>>> [2020-07-02 12:30:32.391075] I [MSGID: 109018] >>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for >>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid >>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311 >>>>>>>>> [2020-07-02 12:33:50.915279] I [MSGID: 109066] >>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming >>>>>>>>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2 >>>>>>>>> (2cb54500-814d-4e85-83e7-e33d9440b18d) >>>>>>>>> (hash=data-client-6/cache=data-client-18) => >>>>>>>>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null)) >>>>>>>>> (hash=data-client-6/cache=<nul>) >>>>>>>>> [2020-07-02 12:34:09.799586] I [MSGID: 109066] >>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming >>>>>>>>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k >>>>>>>>> (99938ee6-6986-4123-9d72-ec09e2310b4f) >>>>>>>>> (hash=data-client-17/cache=data-client-18) => >>>>>>>>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null)) >>>>>>>>> (hash=data-client-17/cache=<nul>) >>>>>>>>> .... >>>>>>>>> >>>>>>>>> >>>>>>>>> Please look into this at top-priority if possible. >>>>>>>>> Let me know if anything else is required. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> Shreyansh Shah >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Shreyansh Shah >>>>>>>> ________ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Community Meeting Calendar: >>>>>>>> >>>>>>>> Schedule - >>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>>>>>>> Bridge: https://bluejeans.com/441850968 >>>>>>>> >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Barak Sason Rofman* >>>>>>> >>>>>>> Gluster Storage Development >>>>>>> >>>>>>> Red Hat Israel <https://www.redhat.com/> >>>>>>> >>>>>>> 34 Jerusalem rd. Ra'anana, 43501 >>>>>>> >>>>>>> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >>>>>>> M: *+972-52-4326355* >>>>>>> @RedHat <https://twitter.com/redhat> Red Hat >>>>>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>>>>> <https://www.facebook.com/redhat.il/> >>>>>>> <https://red.ht/sig> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Shreyansh Shah >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Barak Sason Rofman* >>>>> >>>>> Gluster Storage Development >>>>> >>>>> Red Hat Israel <https://www.redhat.com/> >>>>> >>>>> 34 Jerusalem rd. Ra'anana, 43501 >>>>> >>>>> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >>>>> M: *+972-52-4326355* >>>>> @RedHat <https://twitter.com/redhat> Red Hat >>>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>>> <https://www.facebook.com/redhat.il/> >>>>> <https://red.ht/sig> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Shreyansh Shah >>>> >>> >>> >>> -- >>> *Barak Sason Rofman* >>> >>> Gluster Storage Development >>> >>> Red Hat Israel <https://www.redhat.com/> >>> >>> 34 Jerusalem rd. Ra'anana, 43501 >>> >>> bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* >>> M: *+972-52-4326355* >>> @RedHat <https://twitter.com/redhat> Red Hat >>> <https://www.linkedin.com/company/red-hat> Red Hat >>> <https://www.facebook.com/redhat.il/> >>> <https://red.ht/sig> >>> >> >> >> -- >> Regards, >> Shreyansh Shah >> > > > -- > *Barak Sason Rofman* > > Gluster Storage Development > > Red Hat Israel <https://www.redhat.com/> > > 34 Jerusalem rd. Ra'anana, 43501 > > bsasonro at redhat.com <adi at redhat.com> T: *+972-9-7692304* > M: *+972-52-4326355* > @RedHat <https://twitter.com/redhat> Red Hat > <https://www.linkedin.com/company/red-hat> Red Hat > <https://www.facebook.com/redhat.il/> > <https://red.ht/sig> >-- Regards, Shreyansh Shah -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200707/6489fa89/attachment.html>