Gluster users - Nov 2018 - Self-healing not healing 27k files on GlusterFS 4.1.5 3 nodes replica

??????? Original Message ???????
On Friday, November 9, 2018 2:11 AM, Ravishankar N <ravishankar at
redhat.com> wrote:
> Please re-create the symlink on node 2 to match how it is in the other
> nodes and launch heal again. Check if this is the case for other entries
> too.
> -Ravi
Please ignore my previous mail, I was looking for a symlink with the GFID of
node1 or node 3 on my node2 whereas I should have been looking with the GFID of
node2 of course. I have now found the symlink on node2 pointing to that
problematic directory and it looks like this:

node2# cd /data/myvol-pro/brick/.glusterfs/d9/ac
node2# ls -la | grep d9ac19
lrwxrwxrwx 1 root     root          66 Nov  5 14:12
d9ac192c-e85e-4402-af10-5551f587ed9a ->
../../10/ec/10ec1eb1-c854-4ff2-a36c-325681713093/oc_dir

When you say "re-create the symlink", do you mean I should delete the
current symlink on node2 (d9ac192c-e85e-4402-af10-5551f587ed9a) and re-create it
with the GFID which is used on my node 1 and node 3 like this?

node2# cd /data/myvol-pro/brick/.glusterfs/d9/ac
node2# rm d9ac192c-e85e-4402-af10-5551f587ed9a
node2# cd /data/myvol-pro/brick/.glusterfs/25/e2
node2# ln -s ../../10/ec/10ec1eb1-c854-4ff2-a36c-325681713093/oc_dir
25e2616b-4fb6-4b2a-8945-1afc956fff19

Just want to make sure I understood you correctly before doing that. Could you
please let me know if this is correct?

Thanks

On 11/13/2018 01:09 PM, mabi wrote:
> ??????? Original Message ???????
> On Friday, November 9, 2018 2:11 AM, Ravishankar N <ravishankar at
redhat.com> wrote:
>
>> Please re-create the symlink on node 2 to match how it is in the other
>> nodes and launch heal again. Check if this is the case for other
entries
>> too.
>> -Ravi
> Please ignore my previous mail, I was looking for a symlink with the GFID
of node1 or node 3 on my node2 whereas I should have been looking with the GFID
of node2 of course. I have now found the symlink on node2 pointing to that
problematic directory and it looks like this:
>
> node2# cd /data/myvol-pro/brick/.glusterfs/d9/ac
> node2# ls -la | grep d9ac19
> lrwxrwxrwx 1 root     root          66 Nov  5 14:12
d9ac192c-e85e-4402-af10-5551f587ed9a ->
../../10/ec/10ec1eb1-c854-4ff2-a36c-325681713093/oc_dir
>
> When you say "re-create the symlink", do you mean I should delete
the current symlink on node2 (d9ac192c-e85e-4402-af10-5551f587ed9a) and
re-create it with the GFID which is used on my node 1 and node 3 like this?
I thought it was missing which is why I asked you to create it.? The 
trusted.gfid xattr for any given file or directory must be same in all 3 
bricks.? But it looks like that isn't the case. Are the gfids and the 
symlinks for all the dirs leading to the parent dir of oc_dir same on 
all nodes? (i.e evey directory in 
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/)?
> node2# cd /data/myvol-pro/brick/.glusterfs/d9/ac
> node2# rm d9ac192c-e85e-4402-af10-5551f587ed9a
> node2# cd /data/myvol-pro/brick/.glusterfs/25/e2
> node2# ln -s ../../10/ec/10ec1eb1-c854-4ff2-a36c-325681713093/oc_dir
25e2616b-4fb6-4b2a-8945-1afc956fff19
>
> Just want to make sure I understood you correctly before doing that. Could
you please let me know if this is correct?
Let us see if the parents' gfids are the same before deleting anything. 
Is the heal info still showing 4 entries? Please also share the getfattr 
output of the the parent directory (i.e. dir11) .
Thanks,
Ravi
>
> Thanks