Googling around I see this has been happening for years, with
apparently no one ever understanding why. The files are now on 2
bricks of each of the 3 replica nodes in my cluster, and the gfids all
appear identical, per below. I guess I'll plan on deleting a copy
directly under one of the bricks after the heal finishes. I'll be
happy to provide more info or logs if someone is interested in looking
further.
pdsh -w glust0[5-7] getfattr -m . -d -e hex /bricks/b*/br/<filepath> |
dshbak | grep gfid
trusted.gfid=0x9c751f5a7f16490dacda32cb7403ccc1
trusted.gfid2path.f2a6b5d7c793e21e=0x39336663336262342d613161332d343039662d613061312d3661646361316232326231662f323139323332335f4252434156322d312d4845524544494356365f53696d62615f3835353233615f4c616e65305f4944543331342e746172
trusted.gfid=0x9c751f5a7f16490dacda32cb7403ccc1
trusted.gfid2path.f2a6b5d7c793e21e=0x39336663336262342d613161332d343039662d613061312d3661646361316232326231662f323139323332335f4252434156322d312d4845524544494356365f53696d62615f3835353233615f4c616e65305f4944543331342e746172
trusted.gfid=0x9c751f5a7f16490dacda32cb7403ccc1
trusted.gfid2path.f2a6b5d7c793e21e=0x39336663336262342d613161332d343039662d613061312d3661646361316232326231662f323139323332335f4252434156322d312d4845524544494356365f53696d62615f3835353233615f4c616e65305f4944543331342e746172
trusted.gfid=0x9c751f5a7f16490dacda32cb7403ccc1
trusted.gfid2path.f2a6b5d7c793e21e=0x39336663336262342d613161332d343039662d613061312d3661646361316232326231662f323139323332335f4252434156322d312d4845524544494356365f53696d62615f3835353233615f4c616e65305f4944543331342e746172
trusted.gfid=0x9c751f5a7f16490dacda32cb7403ccc1
trusted.gfid2path.f2a6b5d7c793e21e=0x39336663336262342d613161332d343039662d613061312d3661646361316232326231662f323139323332335f4252434156322d312d4845524544494356365f53696d62615f3835353233615f4c616e65305f4944543331342e746172
trusted.gfid=0x9c751f5a7f16490dacda32cb7403ccc1
trusted.gfid2path.f2a6b5d7c793e21e=0x39336663336262342d613161332d343039662d613061312d3661646361316232326231662f323139323332335f4252434156322d312d4845524544494356365f53696d62615f3835353233615f4c616e65305f4944543331342e746172
On Fri, Jan 8, 2021 at 1:29 PM Eli V <eliventer at gmail.com>
wrote:>
> I just upgraded a gluster replica 3 gluster from 8.2 -> 8.3 rebooting
> the nodes one by one. The cluster was completely idle during this
> time. After each reboot things seemed fine and a gluster volume heal
> vol & info reported no issues. However, about an hour later I started
> seeing duplicate files in ls listing of some of the directories, and
> doing a heal info now shows files undergoing healing. The list of
> healing files does not contain all the duplicates however. Any idea's
> what's going on, or pointers to debug further?