Gluster users - Nov 2018 - Self-healing not healing 27k files on GlusterFS 4.1.5 3 nodes replica

??????? Original Message ???????
On Friday, November 16, 2018 5:14 AM, Ravishankar N <ravishankar at
redhat.com> wrote:
> Okay, as asked in the previous mail, please share the getfattr output
> from all bricks for these 2 files. I think once we have this, we can try
> either 'adjusting' the the gfid and symlinks on node 2 for dir11
and
> oc_dir or see if we can set afr xattrs on dir10 for self-heal to purge
> everything under it on node 2 and recreate it using the other 2 nodes.
And finally here is the output of a getfattr from both files from the 3 nodes:

FILE 1:
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/fileKey

NODE 1:
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579

NODE 2:
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x48ccb52b788f4361b33fad43157b8ea8
trusted.gfid2path.32a8dc56983f7b8f=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f66696c654b6579

NODE 3:
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579


FILE 2:
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/username.shareKey

NODE 1:
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579

NODE 2:
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0xae880a4f19824bc6a3baabe2e3c62ace
trusted.gfid2path.0c0f97b97351b4af=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f6a6d406d616765726c2e63682e73686172654b6579

NODE 3:
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579

Thanks again in advance for your answer.

Okay so for all files and dirs, node 2 seems to be the bad copy. Try the 
following:

1. On both node 1 and node3, set theafr xattr for dir10:
setfattr -n trusted.afr.myvol-pro-client-1 -v 0x000000000000000100000001 
/data/myvol-private/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10

2. Fuse mount the volume temporarily in some location and from that 
mount point, do a `find .|xargs stat >/dev/null`

3. Run `gluster volume heal $volname`

HTH,
Ravi

On 11/16/2018 09:07 PM, mabi wrote:
> And finally here is the output of a getfattr from both files from the 3
nodes:
>
> FILE 1:
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/fileKey
>
> NODE 1:
> trusted.afr.dirty=0x000000000000000000000000
> trusted.afr.myvol-pro-client-1=0x000000020000000100000000
> trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
>
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579
>
> NODE 2:
> trusted.afr.dirty=0x000000000000000000000000
> trusted.gfid=0x48ccb52b788f4361b33fad43157b8ea8
>
trusted.gfid2path.32a8dc56983f7b8f=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f66696c654b6579
>
> NODE 3:
> trusted.afr.dirty=0x000000000000000000000000
> trusted.afr.myvol-pro-client-1=0x000000020000000100000000
> trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
>
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579
>
>
> FILE 2:
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/username.shareKey
>
> NODE 1:
> trusted.afr.dirty=0x000000000000000000000000
> trusted.afr.myvol-pro-client-1=0x000000020000000100000000
> trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
>
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579
>
> NODE 2:
> trusted.afr.dirty=0x000000000000000000000000
> trusted.gfid=0xae880a4f19824bc6a3baabe2e3c62ace
>
trusted.gfid2path.0c0f97b97351b4af=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f6a6d406d616765726c2e63682e73686172654b6579
>
> NODE 3:
> trusted.afr.dirty=0x000000000000000000000000
> trusted.afr.myvol-pro-client-1=0x000000020000000100000000
> trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
>
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579