Gluster users - Mar 2021 - Files being healed won't go away

Hi,

I?m running gluster version 3.10.5 across 8 machines. All of the volumes are
Distributed-Replicate. One of the volumes shows a never-going-away list of 13
files when I execute:

gluster volume heal <VOLNAME> info

I?ve tried to run:

gluster volume heal <VOLNAME> 

? but nothing happens.

Most of the files show grid?s but there?s one that is a path. The files belong
to one brick. There are about 4 clients that are version 3.12.15 and I?ve come
across some info that higher version clients can create this problem, but I
doubt it?s a problem.

What can I do to resolve this manually since gluster can?t seem to auto-heal
them? I?ve searched high and low and can?t find any recipes.

Thanks much for any info,
   Sergei
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20210324/e284dd32/attachment.html>

I took one of the gfids and found the actual path to the file on the brick. I
then did:

getfattr -d -m . -e hex <ACTUAL_PATH>

security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.<VOLNAME>-client-6=0x000000020000000100000000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x02000000000000005a7e0cdb000196c6
trusted.gfid=0xa65bac27fd504c899ba4457675ed5518
trusted.pgfid.eb6d6fa2-339e-4dd7-a642-ca755d5f47cf=0x00000001

Which must mean that client-6 had a problem with both the data and metadata.

I then went to client-6 and issued the same getfattr command:

getfattr -d -m . -e hex <ACTUAL_PATH>
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x02000000000000005a7e0cdb00018323
trusted.gfid=0x06df01f614fa40fda3b628265ba36318
trusted.pgfid.eb6d6fa2-339e-4dd7-a642-ca755d5f47cf=0x00000001


As you can see, the gfid is different between the clients for the same file
path. Doing a file operation on this file via the mount results in Input/Output
error. Doing a stat on the parent directory doesn?t do anything either.

What?s a good way to mover forward from here?

Unrelated: what?s dht.linkto attribute used for?

Thanks!
   Sergei

> On Mar 24, 2021, at 4:49 PM, Sergei Gerasenko <gerases at gmail.com>
wrote:
> 
> Hi,
> 
> I?m running gluster version 3.10.5 across 8 machines. All of the volumes
are Distributed-Replicate. One of the volumes shows a never-going-away list of
13 files when I execute:
> 
> gluster volume heal <VOLNAME> info
> 
> I?ve tried to run:
> 
> gluster volume heal <VOLNAME> 
> 
> ? but nothing happens.
> 
> Most of the files show grid?s but there?s one that is a path. The files
belong to one brick. There are about 4 clients that are version 3.12.15 and I?ve
come across some info that higher version clients can create this problem, but I
doubt it?s a problem.
> 
> What can I do to resolve this manually since gluster can?t seem to
auto-heal them? I?ve searched high and low and can?t find any recipes.
> 
> Thanks much for any info,
>    Sergei
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20210324/d809e4d9/attachment.html>