Gluster users - Jul 2017 - Possible stale .glusterfs/indices/xattrop file?

I did a find on this inode number and I could find the file but only on node1
(nothing on node2 and the new arbiternode). Here is an ls -lai of the file
itself on node1:
-rw-r--r-- 1 www-data www-data 32 Jun 19 17:42 fileKey
As you can see it is a 32 bytes file and as you suggested I ran a
"stat" on this very same file through a glusterfs mount (using fuse)
but unfortunately nothing happened. The GFID is still being displayed to be
healed. Just in case here is the output of the stat:
File: ?fileKey?
Size: 32 Blocks: 1 IO Block: 131072 regular file
Device: 1eh/30d Inode: 12086351742306673840 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
Access: 2017-06-19 17:42:35.339773495 +0200
Modify: 2017-06-19 17:42:35.343773437 +0200
Change: 2017-06-19 17:42:35.343773437 +0200
Birth: -
What else can I do or try in order to fix this situation?
> -------- Original Message --------
> Subject: Re: [Gluster-users] Possible stale .glusterfs/indices/xattrop
file?
> Local Time: July 31, 2017 3:27 AM
> UTC Time: July 31, 2017 1:27 AM
> From: ravishankar at redhat.com
> To: mabi <mabi at protonmail.ch>
> Gluster Users <gluster-users at gluster.org>
>
> On 07/30/2017 02:24 PM, mabi wrote:
>
>> Hi Ravi,
>> Thanks for your hints. Below you will find the answer to your
questions.
>> First I tried to start the healing process by running:
>> gluster volume heal myvolume
>> and then as you suggested watch the output of the glustershd.log file
but nothing appeared in that log file after running the above command. I checked
the files which need to be healing using the "heal <volume>
info" command and it still shows that very same GFID on node2 to be healed.
So nothing changed here.
>> The file
/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397
is only on node2 and not on my nod1 nor on my arbiternode. This file seems to be
a regular file and not a symlink. Here is the output of the stat command on it
from my node2:
>> File:
?/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397?
>> Size: 0 Blocks: 1 IO Block: 512 regular empty file
>> Device: 25h/37d Inode: 2798404 Links: 2
>
> Okay, link count of 2 means there is a hardlink somewhere on the brick. Try
the find command again. I see that the inode number is 2798404, not the one you
shared in your first mail. Once you find the path to the file, do a stat of the
file from mount. This should create the entry in the other 2 bricks and do the
heal. But FWIW, this seems to be a zero byte file.
> Regards,
> Ravi
>
>> Access: (0000/----------) Uid: ( 0/ root) Gid: ( 0/ root)
>> Access: 2017-04-28 22:51:15.215775269 +0200
>> Modify: 2017-04-28 22:51:15.215775269 +0200
>> Change: 2017-07-30 08:39:03.700872312 +0200
>> Birth: -
>> I hope this is enough info for a starter, else let me know if you need
any more info. I would be glad to resolve this weird file which needs to be
healed but can not.
>> Best regards,
>> Mabi
>>
>>> -------- Original Message --------
>>> Subject: Re: [Gluster-users] Possible stale
.glusterfs/indices/xattrop file?
>>> Local Time: July 30, 2017 3:31 AM
>>> UTC Time: July 30, 2017 1:31 AM
>>> From: ravishankar at redhat.com
>>> To: mabi [<mabi at protonmail.ch>](mailto:mabi at
protonmail.ch), Gluster Users [<gluster-users at
gluster.org>](mailto:gluster-users at gluster.org)
>>>
>>> On 07/29/2017 04:36 PM, mabi wrote:
>>>
>>>> Hi,
>>>> Sorry for mailing again but as mentioned in my previous mail, I
have added an arbiter node to my replica 2 volume and it seem to have gone fine
except for the fact that there is one single file which needs healing and does
not get healed as you can see here from the output of a "heal info":
>>>> Brick node1.domain.tld:/data/myvolume/brick
>>>> Status: Connected
>>>> Number of entries: 0
>>>> Brick node2.domain.tld:/data/myvolume/brick
>>>> <gfid:29e0d13e-1217-41cc-9bda-1fbbf781c397>
>>>> Status: Connected
>>>> Number of entries: 1
>>>> Brick arbiternode.domain.tld:/srv/glusterfs/myvolume/brick
>>>> Status: Connected
>>>> Number of entries: 0
>>>> On my node2 the respective .glusterfs/indices/xattrop directory
contains two files as you can see below:
>>>> ls -lai /data/myvolume/brick/.glusterfs/indices/xattrop
>>>> total 76180
>>>> 10 drw------- 2 root root 4 Jul 29 12:15 .
>>>> 9 drw------- 5 root root 5 Apr 28 22:15 ..
>>>> 2798404 ---------- 2 root root 0 Apr 28 22:51
29e0d13e-1217-41cc-9bda-1fbbf781c397
>>>> 2798404 ---------- 2 root root 0 Apr 28 22:51
xattrop-6fa49ad5-71dd-4ec2-9246-7b302ab92d38
>>>> I tried to find the real file on my brick where this xattrop
file points to using its inode number (command: find /data/myvolume/brick/data
-inum 8394642) but it does not find any associated file.
>>>> So my question here is, is it possible that this is a stale
file which just forgot to get deleted from the indices/xattrop file by gluster
for some unknown reason? If yes is it safe for me to delete these two files? or
what would be the correct process in that case?
>>>
>>> The 'xattrop-6fa...' is the base entry. gfids of files that
need heal are hard linked to this entry, so nothing needs to be done for it. But
you need to find out why '29e0d13...' is not healing. Launch the heal
and observe the glustershd logs for errors. I suppose the inode number for
.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397 is what is 8394642. Is
.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397 a regular file or symlink?
Does it exist in the other 2 bricks? What is the link count (as seen from stat
<file>)?
>>> -Ravi
>>>
>>>> Thank you for your input.
>>>> Mabi
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>>
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170731/abfa4bed/attachment.html>

On 07/31/2017 12:20 PM, mabi wrote:
> I did a find on this inode number and I could find the file but only 
> on node1 (nothing on node2 and the new arbiternode). Here is an ls 
> -lai of the file itself on node1:
Sorry I don't understand, isn't that (XFS) inode number specific to 
node2's brick? If you want to use the same command, maybe you should try 
`find /data/myvolume/brick -samefile 
/data/myvolume/brick/.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397` 
on all 3 bricks.
>
> -rw-r--r-- 1 www-data www-data   32 Jun 19 17:42 fileKey
>
> As you can see it is a 32 bytes file and as you suggested I ran a 
> "stat" on this very same file through a glusterfs mount (using
fuse)
> but unfortunately nothing happened. The GFID is still being displayed 
> to be healed.  Just in case here is the output of the stat:
>
>   File: ?fileKey?
>   Size: 32        Blocks: 1          IO Block: 131072 regular file
> Device: 1eh/30d Inode: 12086351742306673840  Links: 1
> Access: (0644/-rw-r--r--)  Uid: (   33/www-data)   Gid: ( 33/www-data)
> Access: 2017-06-19 17:42:35.339773495 +0200
> Modify: 2017-06-19 17:42:35.343773437 +0200
> Change: 2017-06-19 17:42:35.343773437 +0200
> Birth: -
>
Is this 'fileKey' on node1 having the same gfid (see getfattr output)? 
Looks like it is missing the hardlink inside .glusterfs folder since the 
link count is only 1.
Thanks,
Ravi
> What else can I do or try in order to fix this situation?
>
>
>
>
>> -------- Original Message --------
>> Subject: Re: [Gluster-users] Possible stale 
>> .glusterfs/indices/xattrop file?
>> Local Time: July 31, 2017 3:27 AM
>> UTC Time: July 31, 2017 1:27 AM
>> From: ravishankar at redhat.com
>> To: mabi <mabi at protonmail.ch>
>> Gluster Users <gluster-users at gluster.org>
>>
>>
>>
>>
>> On 07/30/2017 02:24 PM, mabi wrote:
>>> Hi Ravi,
>>>
>>> Thanks for your hints. Below you will find the answer to your
questions.
>>>
>>> First I tried to start the healing process by running:
>>>
>>> gluster volume heal myvolume
>>>
>>> and then as you suggested watch the output of the glustershd.log 
>>> file but nothing appeared in that log file after running the above 
>>> command. I checked the files which need to be healing using the 
>>> "heal <volume> info" command and it still shows
that very same GFID
>>> on node2 to be healed. So nothing changed here.
>>>
>>> The file 
>>>
/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397
>>> is only on node2 and not on my nod1 nor on my arbiternode. This
file
>>> seems to be a regular file and not a symlink. Here is the output of
>>> the stat command on it from my node2:
>>>
>>>   File: 
>>>
?/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397?
>>>   Size: 0         Blocks: 1          IO Block: 512 regular empty
file
>>> Device: 25h/37d Inode: 2798404     Links: 2
>>
>> Okay, link count of 2 means there is a hardlink somewhere on the 
>> brick. Try the find command again. I see that the inode number is 
>> 2798404, not the one you shared in your first mail. Once you find the 
>> path to the file, do a stat of the file from mount. This should 
>> create the entry in the other 2 bricks and do the heal. But FWIW, 
>> this seems to be a zero byte file.
>>
>> Regards,
>> Ravi
>>
>>> Access: (0000/----------)  Uid: (    0/    root)   Gid: (    0/   
root)
>>> Access: 2017-04-28 22:51:15.215775269 +0200
>>> Modify: 2017-04-28 22:51:15.215775269 +0200
>>> Change: 2017-07-30 08:39:03.700872312 +0200
>>> Birth: -
>>>
>>> I hope this is enough info for a starter, else let me know if you 
>>> need any more info. I would be glad to resolve this weird file
which
>>> needs to be healed but can not.
>>>
>>> Best regards,
>>> Mabi
>>>
>>>
>>>
>>>> -------- Original Message --------
>>>> Subject: Re: [Gluster-users] Possible stale 
>>>> .glusterfs/indices/xattrop file?
>>>> Local Time: July 30, 2017 3:31 AM
>>>> UTC Time: July 30, 2017 1:31 AM
>>>> From: ravishankar at redhat.com
>>>> To: mabi <mabi at protonmail.ch>, Gluster Users 
>>>> <gluster-users at gluster.org>
>>>>
>>>>
>>>>
>>>>
>>>> On 07/29/2017 04:36 PM, mabi wrote:
>>>>> Hi,
>>>>>
>>>>> Sorry for mailing again but as mentioned in my previous
mail, I
>>>>> have added an arbiter node to my replica 2 volume and it
seem to
>>>>> have gone fine except for the fact that there is one single
file
>>>>> which needs healing and does not get healed as you can see
here
>>>>> from the output of a "heal info":
>>>>>
>>>>> Brick node1.domain.tld:/data/myvolume/brick
>>>>> Status: Connected
>>>>> Number of entries: 0
>>>>>
>>>>> Brick node2.domain.tld:/data/myvolume/brick
>>>>> <gfid:29e0d13e-1217-41cc-9bda-1fbbf781c397>
>>>>> Status: Connected
>>>>> Number of entries: 1
>>>>>
>>>>> Brick arbiternode.domain.tld:/srv/glusterfs/myvolume/brick
>>>>> Status: Connected
>>>>> Number of entries: 0
>>>>>
>>>>> On my node2 the respective .glusterfs/indices/xattrop
directory
>>>>> contains two files as you can see below:
>>>>>
>>>>> ls -lai /data/myvolume/brick/.glusterfs/indices/xattrop
>>>>> total 76180
>>>>>      10 drw------- 2 root root 4 Jul 29 12:15 .
>>>>>       9 drw------- 5 root root 5 Apr 28 22:15 ..
>>>>> 2798404 ---------- 2 root root 0 Apr 28 22:51 
>>>>> 29e0d13e-1217-41cc-9bda-1fbbf781c397
>>>>> 2798404 ---------- 2 root root 0 Apr 28 22:51 
>>>>> xattrop-6fa49ad5-71dd-4ec2-9246-7b302ab92d38
>>>>>
>>>>>
>>>>>
>>>>> I tried to find the real file on my brick where this
xattrop file
>>>>> points to using its inode number (command: find 
>>>>> /data/myvolume/brick/data -inum 8394642) but it does not
find any
>>>>> associated file.
>>>>>
>>>>> So my question here is, is it possible that this is a stale
file
>>>>> which just forgot to get deleted from the indices/xattrop
file by
>>>>> gluster for some unknown reason? If yes is it safe for me
to
>>>>> delete these two files? or what would be the correct
process in
>>>>> that case?
>>>> The 'xattrop-6fa...' is the base entry. gfids of files
that need
>>>> heal are hard linked to this entry, so nothing needs to be done
for
>>>> it. But you need to find out why '29e0d13...' is not
healing.
>>>> Launch the heal and observe the glustershd logs for errors. I 
>>>> suppose the inode number for 
>>>> .glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397 is what
is
>>>> 8394642.  Is
.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397
>>>> a regular file or  symlink? Does it exist in the other 2
bricks?
>>>> What is the link count (as seen from stat <file>)?
>>>> -Ravi
>>>>
>>>>>
>>>>> Thank you for your input.
>>>>> Mabi
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170731/bb3ecd4d/attachment.html>

To quickly resume my current situation:
on node2 I have found the following file xattrop/indices file which matches the
GFID of the "heal info" command (below is there output of "ls
-lai":
2798404 ---------- 2 root root 0 Apr 28 22:51
/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397
As you can see this file has inode number 2798404, so I ran the following
command on all my nodes (node1, node2 and arbiternode):
sudo find /data/myvolume/brick -inum 2798404 -ls
Here below are the results for all 3 nodes:
node1:
2798404 19 -rw-r--r-- 2 www-data www-data 32 Jun 19 17:42
/data/myvolume/brick/.glusterfs/e6/5b/e65b77e2-a4c4-4824-a7bb-58df969ce4b0
2798404 19 -rw-r--r-- 2 www-data www-data 32 Jun 19 17:42
/data/myvolume/brick/<REMOVED_DIRECTORIES_IN_BETWEEN>/fileKey
node2:
2798404 1 ---------- 2 root root 0 Apr 28 22:51
/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397
2798404 1 ---------- 2 root root 0 Apr 28 22:51
/data/myvolume/brick/.glusterfs/indices/xattrop/xattrop-6fa49ad5-71dd-4ec2-9246-7b302ab92d38
arbirternode:
NOTHING
As you requested I have tried to run on node1 a getfattr on the fileKey file by
using the following command:
getfattr -m . -d -e hex fileKey
but there is no output. I am not familiar with the getfattr command so maybe I
am using the wrong parameters, could you help me with that?
> -------- Original Message --------
> Subject: Re: [Gluster-users] Possible stale .glusterfs/indices/xattrop
file?
> Local Time: July 31, 2017 9:25 AM
> UTC Time: July 31, 2017 7:25 AM
> From: ravishankar at redhat.com
> To: mabi <mabi at protonmail.ch>
> Gluster Users <gluster-users at gluster.org>
> On 07/31/2017 12:20 PM, mabi wrote:
>
>> I did a find on this inode number and I could find the file but only on
node1 (nothing on node2 and the new arbiternode). Here is an ls -lai of the file
itself on node1:
>
> Sorry I don't understand, isn't that (XFS) inode number specific to
node2's brick? If you want to use the same command, maybe you should try
`find /data/myvolume/brick -samefile
/data/myvolume/brick/.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397` on
all 3 bricks.
>
>> -rw-r--r-- 1 www-data www-data 32 Jun 19 17:42 fileKey
>> As you can see it is a 32 bytes file and as you suggested I ran a
"stat" on this very same file through a glusterfs mount (using fuse)
but unfortunately nothing happened. The GFID is still being displayed to be
healed. Just in case here is the output of the stat:
>> File: ?fileKey?
>> Size: 32 Blocks: 1 IO Block: 131072 regular file
>> Device: 1eh/30d Inode: 12086351742306673840 Links: 1
>> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>> Access: 2017-06-19 17:42:35.339773495 +0200
>> Modify: 2017-06-19 17:42:35.343773437 +0200
>> Change: 2017-06-19 17:42:35.343773437 +0200
>> Birth: -
>
> Is this 'fileKey' on node1 having the same gfid (see getfattr
output)? Looks like it is missing the hardlink inside .glusterfs folder since
the link count is only 1.
> Thanks,
> Ravi
>
>> What else can I do or try in order to fix this situation?
>>
>>> -------- Original Message --------
>>> Subject: Re: [Gluster-users] Possible stale
.glusterfs/indices/xattrop file?
>>> Local Time: July 31, 2017 3:27 AM
>>> UTC Time: July 31, 2017 1:27 AM
>>> From: ravishankar at redhat.com
>>> To: mabi [<mabi at protonmail.ch>](mailto:mabi at
protonmail.ch)
>>> Gluster Users [<gluster-users at
gluster.org>](mailto:gluster-users at gluster.org)
>>>
>>> On 07/30/2017 02:24 PM, mabi wrote:
>>>
>>>> Hi Ravi,
>>>> Thanks for your hints. Below you will find the answer to your
questions.
>>>> First I tried to start the healing process by running:
>>>> gluster volume heal myvolume
>>>> and then as you suggested watch the output of the
glustershd.log file but nothing appeared in that log file after running the
above command. I checked the files which need to be healing using the "heal
<volume> info" command and it still shows that very same GFID on
node2 to be healed. So nothing changed here.
>>>> The file
/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397
is only on node2 and not on my nod1 nor on my arbiternode. This file seems to be
a regular file and not a symlink. Here is the output of the stat command on it
from my node2:
>>>> File:
?/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397?
>>>> Size: 0 Blocks: 1 IO Block: 512 regular empty file
>>>> Device: 25h/37d Inode: 2798404 Links: 2
>>>
>>> Okay, link count of 2 means there is a hardlink somewhere on the
brick. Try the find command again. I see that the inode number is 2798404, not
the one you shared in your first mail. Once you find the path to the file, do a
stat of the file from mount. This should create the entry in the other 2 bricks
and do the heal. But FWIW, this seems to be a zero byte file.
>>> Regards,
>>> Ravi
>>>
>>>> Access: (0000/----------) Uid: ( 0/ root) Gid: ( 0/ root)
>>>> Access: 2017-04-28 22:51:15.215775269 +0200
>>>> Modify: 2017-04-28 22:51:15.215775269 +0200
>>>> Change: 2017-07-30 08:39:03.700872312 +0200
>>>> Birth: -
>>>> I hope this is enough info for a starter, else let me know if
you need any more info. I would be glad to resolve this weird file which needs
to be healed but can not.
>>>> Best regards,
>>>> Mabi
>>>>
>>>>> -------- Original Message --------
>>>>> Subject: Re: [Gluster-users] Possible stale
.glusterfs/indices/xattrop file?
>>>>> Local Time: July 30, 2017 3:31 AM
>>>>> UTC Time: July 30, 2017 1:31 AM
>>>>> From: ravishankar at redhat.com
>>>>> To: mabi [<mabi at protonmail.ch>](mailto:mabi at
protonmail.ch), Gluster Users [<gluster-users at
gluster.org>](mailto:gluster-users at gluster.org)
>>>>>
>>>>> On 07/29/2017 04:36 PM, mabi wrote:
>>>>>
>>>>>> Hi,
>>>>>> Sorry for mailing again but as mentioned in my previous
mail, I have added an arbiter node to my replica 2 volume and it seem to have
gone fine except for the fact that there is one single file which needs healing
and does not get healed as you can see here from the output of a "heal
info":
>>>>>> Brick node1.domain.tld:/data/myvolume/brick
>>>>>> Status: Connected
>>>>>> Number of entries: 0
>>>>>> Brick node2.domain.tld:/data/myvolume/brick
>>>>>> <gfid:29e0d13e-1217-41cc-9bda-1fbbf781c397>
>>>>>> Status: Connected
>>>>>> Number of entries: 1
>>>>>> Brick
arbiternode.domain.tld:/srv/glusterfs/myvolume/brick
>>>>>> Status: Connected
>>>>>> Number of entries: 0
>>>>>> On my node2 the respective .glusterfs/indices/xattrop
directory contains two files as you can see below:
>>>>>> ls -lai /data/myvolume/brick/.glusterfs/indices/xattrop
>>>>>> total 76180
>>>>>> 10 drw------- 2 root root 4 Jul 29 12:15 .
>>>>>> 9 drw------- 5 root root 5 Apr 28 22:15 ..
>>>>>> 2798404 ---------- 2 root root 0 Apr 28 22:51
29e0d13e-1217-41cc-9bda-1fbbf781c397
>>>>>> 2798404 ---------- 2 root root 0 Apr 28 22:51
xattrop-6fa49ad5-71dd-4ec2-9246-7b302ab92d38
>>>>>> I tried to find the real file on my brick where this
xattrop file points to using its inode number (command: find
/data/myvolume/brick/data -inum 8394642) but it does not find any associated
file.
>>>>>> So my question here is, is it possible that this is a
stale file which just forgot to get deleted from the indices/xattrop file by
gluster for some unknown reason? If yes is it safe for me to delete these two
files? or what would be the correct process in that case?
>>>>>
>>>>> The 'xattrop-6fa...' is the base entry. gfids of
files that need heal are hard linked to this entry, so nothing needs to be done
for it. But you need to find out why '29e0d13...' is not healing. Launch
the heal and observe the glustershd logs for errors. I suppose the inode number
for .glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397 is what is 8394642. Is
.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397 a regular file or symlink?
Does it exist in the other 2 bricks? What is the link count (as seen from stat
<file>)?
>>>>> -Ravi
>>>>>
>>>>>> Thank you for your input.
>>>>>> Mabi
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>>
>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170731/7bd441f5/attachment.html>