[Gluster-users] self-heal not working

Ravishankar N
Mon, 28 Aug 2017 14:29:43 -0000

Great, can you raise a bug for the issue so that it is easier to keep
track (plus you'll be notified if the patch is posted) of it? The
general guidelines are @
https://gluster.readthedocs.io/en/latest/Contributors-Guide/Bug-Reporting-Guidelines
but you just need to provide whatever you described in this email thread
in the bug:

i.e. volume info, heal info, getfattr and stat output of the file in
question.

Thanks!
Ravi

On 08/28/2017 07:49 PM, mabi wrote:
> Thank you for the command. I ran it on all my nodes and now finally
> the the self-heal daemon does not report any files to be healed.
> Hopefully this scenario can get handled properly in newer versions of
> GlusterFS.
>
>
>
>> -------- Original Message --------
>> Subject: Re: [Gluster-users] self-heal not working
>> Local Time: August 28, 2017 10:41 AM
>> UTC Time: August 28, 2017 8:41 AM
>> From: ravishankar at redhat.com
>> To: mabi <mabi at protonmail.ch>
>> Ben Turner <bturner at redhat.com>, Gluster Users
>> <gluster-users at gluster.org>
>>
>>
>>
>>
>> On 08/28/2017 01:29 PM, mabi wrote:
>>> Excuse me for my naive questions but how do I reset the afr.dirty
>>> xattr on the file to be healed? and do I need to do that through a
>>> FUSE mount? or simply on every bricks directly?
>>>
>>>
>> Directly on the bricks: `setfattr -n trusted.afr.dirty -v
>> 0x000000000000000000000000
>>
/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png`
>> -Ravi
>>
>>>
>>>> -------- Original Message --------
>>>> Subject: Re: [Gluster-users] self-heal not working
>>>> Local Time: August 28, 2017 5:58 AM
>>>> UTC Time: August 28, 2017 3:58 AM
>>>> From: ravishankar at redhat.com
>>>> To: Ben Turner <bturner at redhat.com>, mabi <mabi at
protonmail.ch>
>>>> Gluster Users <gluster-users at gluster.org>
>>>>
>>>>
>>>>
>>>> On 08/28/2017 01:57 AM, Ben Turner wrote:
>>>> > ----- Original Message -----
>>>> >> From: "mabi" <mabi at protonmail.ch>
>>>> >> To: "Ravishankar N" <ravishankar at
redhat.com>
>>>> >> Cc: "Ben Turner" <bturner at
redhat.com>, "Gluster Users"
>>>> <gluster-users at gluster.org>
>>>> >> Sent: Sunday, August 27, 2017 3:15:33 PM
>>>> >> Subject: Re: [Gluster-users] self-heal not working
>>>> >>
>>>> >> Thanks Ravi for your analysis. So as far as I
understand nothing
>>>> to worry
>>>> >> about but my question now would be: how do I get rid
of this
>>>> file from the
>>>> >> heal info?
>>>> > Correct me if I am wrong but clearing this is just a
matter of
>>>> resetting the afr.dirty xattr? @Ravi - Is this correct?
>>>>
>>>> Yes resetting the xattr and launching index heal or running
heal-info
>>>> command should serve as a workaround.
>>>> -Ravi
>>>>
>>>> >
>>>> > -b
>>>> >
>>>> >>> -------- Original Message --------
>>>> >>> Subject: Re: [Gluster-users] self-heal not working
>>>> >>> Local Time: August 27, 2017 3:45 PM
>>>> >>> UTC Time: August 27, 2017 1:45 PM
>>>> >>> From: ravishankar at redhat.com
>>>> >>> To: mabi <mabi at protonmail.ch>
>>>> >>> Ben Turner <bturner at redhat.com>, Gluster
Users
>>>> <gluster-users at gluster.org>
>>>> >>>
>>>> >>> Yes, the shds did pick up the file for healing (I
saw messages
>>>> like " got
>>>> >>> entry: 1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea")
but no error
>>>> afterwards.
>>>> >>>
>>>> >>> Anyway I reproduced it by manually setting the
afr.dirty bit
>>>> for a zero
>>>> >>> byte file on all 3 bricks. Since there are no afr
pending xattrs
>>>> >>> indicating good/bad copies and all files are zero
bytes, the data
>>>> >>> self-heal algorithm just picks the file with the
latest ctime
>>>> as source.
>>>> >>> In your case that was the arbiter brick. In the
code, there is
>>>> a check to
>>>> >>> prevent data heals if arbiter is the source. So
heal was not
>>>> happening and
>>>> >>> the entries were not removed from heal-info
output.
>>>> >>>
>>>> >>> Perhaps we should add a check in the code to just
remove the
>>>> entries from
>>>> >>> heal-info if size is zero bytes in all bricks.
>>>> >>>
>>>> >>> -Ravi
>>>> >>>
>>>> >>> On 08/25/2017 06:33 PM, mabi wrote:
>>>> >>>
>>>> >>>> Hi Ravi,
>>>> >>>>
>>>> >>>> Did you get a chance to have a look at the log
files I have
>>>> attached in my
>>>> >>>> last mail?
>>>> >>>>
>>>> >>>> Best,
>>>> >>>> Mabi
>>>> >>>>
>>>> >>>>> -------- Original Message --------
>>>> >>>>> Subject: Re: [Gluster-users] self-heal not
working
>>>> >>>>> Local Time: August 24, 2017 12:08 PM
>>>> >>>>> UTC Time: August 24, 2017 10:08 AM
>>>> >>>>> From: mabi at protonmail.ch
>>>> >>>>> To: Ravishankar N
>>>> >>>>> [<ravishankar at
redhat.com>](mailto:ravishankar at redhat.com)
>>>> >>>>> Ben Turner [<bturner at
redhat.com>](mailto:bturner at redhat.com),
>>>> Gluster
>>>> >>>>> Users
>>>> [<gluster-users at gluster.org>](mailto:gluster-users at
gluster.org)
>>>> >>>>>
>>>> >>>>> Thanks for confirming the command. I have
now enabled DEBUG
>>>> >>>>> client-log-level, run a heal and then
attached the glustershd
>>>> log files
>>>> >>>>> of all 3 nodes in this mail.
>>>> >>>>>
>>>> >>>>> The volume concerned is called myvol-pro,
the other 3 volumes
>>>> have no
>>>> >>>>> problem so far.
>>>> >>>>>
>>>> >>>>> Also note that in the mean time it looks
like the file has
>>>> been deleted
>>>> >>>>> by the user and as such the heal info
command does not show
>>>> the file
>>>> >>>>> name anymore but just is GFID which is:
>>>> >>>>>
>>>> >>>>> gfid:1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea
>>>> >>>>>
>>>> >>>>> Hope that helps for debugging this issue.
>>>> >>>>>
>>>> >>>>>> -------- Original Message --------
>>>> >>>>>> Subject: Re: [Gluster-users] self-heal
not working
>>>> >>>>>> Local Time: August 24, 2017 5:58 AM
>>>> >>>>>> UTC Time: August 24, 2017 3:58 AM
>>>> >>>>>> From: ravishankar at redhat.com
>>>> >>>>>> To: mabi [<mabi at
protonmail.ch>](mailto:mabi at protonmail.ch)
>>>> >>>>>> Ben Turner
>>>> [<bturner at redhat.com>](mailto:bturner at redhat.com),
Gluster
>>>> >>>>>> Users
>>>> [<gluster-users at gluster.org>](mailto:gluster-users at
gluster.org)
>>>> >>>>>>
>>>> >>>>>> Unlikely. In your case only the
afr.dirty is set, not the
>>>> >>>>>> afr.volname-client-xx xattr.
>>>> >>>>>>
>>>> >>>>>> `gluster volume set myvolume
diagnostics.client-log-level
>>>> DEBUG` is
>>>> >>>>>> right.
>>>> >>>>>>
>>>> >>>>>> On 08/23/2017 10:31 PM, mabi wrote:
>>>> >>>>>>
>>>> >>>>>>> I just saw the following bug which
was fixed in 3.8.15:
>>>> >>>>>>>
>>>> >>>>>>>
https://bugzilla.redhat.com/show_bug.cgi?id=1471613
>>>> >>>>>>>
>>>> >>>>>>> Is it possible that the problem I
described in this post is
>>>> related to
>>>> >>>>>>> that bug?
>>>> >>>>>>>
>>>> >>>>>>>> -------- Original Message
--------
>>>> >>>>>>>> Subject: Re: [Gluster-users]
self-heal not working
>>>> >>>>>>>> Local Time: August 22, 2017
11:51 AM
>>>> >>>>>>>> UTC Time: August 22, 2017 9:51
AM
>>>> >>>>>>>> From: ravishankar at
redhat.com
>>>> >>>>>>>> To: mabi [<mabi at
protonmail.ch>](mailto:mabi at protonmail.ch)
>>>> >>>>>>>> Ben Turner
>>>> [<bturner at redhat.com>](mailto:bturner at redhat.com),
Gluster
>>>> >>>>>>>> Users
>>>> [<gluster-users at gluster.org>](mailto:gluster-users at
gluster.org)
>>>> >>>>>>>>
>>>> >>>>>>>> On 08/22/2017 02:30 PM, mabi
wrote:
>>>> >>>>>>>>
>>>> >>>>>>>>> Thanks for the additional
hints, I have the following 2
>>>> questions
>>>> >>>>>>>>> first:
>>>> >>>>>>>>>
>>>> >>>>>>>>> - In order to launch the
index heal is the following
>>>> command correct:
>>>> >>>>>>>>> gluster volume heal
myvolume
>>>> >>>>>>>> Yes
>>>> >>>>>>>>
>>>> >>>>>>>>> - If I run a "volume
start force" will it have any short
>>>> disruptions
>>>> >>>>>>>>> on my clients which mount
the volume through FUSE? If
>>>> yes, how long?
>>>> >>>>>>>>> This is a production
system that"s why I am asking.
>>>> >>>>>>>> No. You can actually create a
test volume on your personal
>>>> linux box
>>>> >>>>>>>> to try these kinds of things
without needing multiple
>>>> machines. This
>>>> >>>>>>>> is how we develop and test our
patches :)
>>>> >>>>>>>> "gluster volume create
testvol replica 3
>>>> /home/mabi/bricks/brick{1..3}
>>>> >>>>>>>> force` and so on.
>>>> >>>>>>>>
>>>> >>>>>>>> HTH,
>>>> >>>>>>>> Ravi
>>>> >>>>>>>>
>>>> >>>>>>>>>> -------- Original
Message --------
>>>> >>>>>>>>>> Subject: Re:
[Gluster-users] self-heal not working
>>>> >>>>>>>>>> Local Time: August 22,
2017 6:26 AM
>>>> >>>>>>>>>> UTC Time: August 22,
2017 4:26 AM
>>>> >>>>>>>>>> From: ravishankar at
redhat.com
>>>> >>>>>>>>>> To: mabi
>>>> [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch),
Ben
>>>> >>>>>>>>>> Turner [<bturner at
redhat.com>](mailto:bturner at redhat.com)
>>>> >>>>>>>>>> Gluster Users
>>>> >>>>>>>>>>
>>>> [<gluster-users at gluster.org>](mailto:gluster-users at
gluster.org)
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Explore the following:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> - Launch index heal
and look at the glustershd logs of
>>>> all bricks
>>>> >>>>>>>>>> for possible errors
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> - See if the
glustershd in each node is connected to all
>>>> bricks.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> - If not try to
restart shd by `volume start force`
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> - Launch index heal
again and try.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> - Try debugging the
shd log by setting client-log-level
>>>> to DEBUG
>>>> >>>>>>>>>> temporarily.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> On 08/22/2017 03:19
AM, mabi wrote:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>> Sure, it
doesn"t look like a split brain based on the
>>>> output:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Brick
node1.domain.tld:/data/myvolume/brick
>>>> >>>>>>>>>>> Status: Connected
>>>> >>>>>>>>>>> Number of entries
in split-brain: 0
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Brick
node2.domain.tld:/data/myvolume/brick
>>>> >>>>>>>>>>> Status: Connected
>>>> >>>>>>>>>>> Number of entries
in split-brain: 0
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Brick
node3.domain.tld:/srv/glusterfs/myvolume/brick
>>>> >>>>>>>>>>> Status: Connected
>>>> >>>>>>>>>>> Number of entries
in split-brain: 0
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>> --------
Original Message --------
>>>> >>>>>>>>>>>> Subject: Re:
[Gluster-users] self-heal not working
>>>> >>>>>>>>>>>> Local Time:
August 21, 2017 11:35 PM
>>>> >>>>>>>>>>>> UTC Time:
August 21, 2017 9:35 PM
>>>> >>>>>>>>>>>> From: bturner
at redhat.com
>>>> >>>>>>>>>>>> To: mabi
[<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)
>>>> >>>>>>>>>>>> Gluster Users
>>>> >>>>>>>>>>>>
>>>> [<gluster-users at gluster.org>](mailto:gluster-users at
gluster.org)
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Can you also
provide:
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> gluster v heal
<my vol> info split-brain
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> If it is split
brain just delete the incorrect file
>>>> from the brick
>>>> >>>>>>>>>>>> and run heal
again. I haven"t tried this with arbiter
>>>> but I
>>>> >>>>>>>>>>>> assume the
process is the same.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> -b
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> ----- Original
Message -----
>>>> >>>>>>>>>>>>> From:
"mabi"
>>>> [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)
>>>> >>>>>>>>>>>>> To:
"Ben Turner"
>>>> >>>>>>>>>>>>>
[<bturner at redhat.com>](mailto:bturner at redhat.com)
>>>> >>>>>>>>>>>>> Cc:
"Gluster Users"
>>>> >>>>>>>>>>>>>
>>>> [<gluster-users at gluster.org>](mailto:gluster-users at
gluster.org)
>>>> >>>>>>>>>>>>> Sent:
Monday, August 21, 2017 4:55:59 PM
>>>> >>>>>>>>>>>>> Subject:
Re: [Gluster-users] self-heal not working
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> Hi Ben,
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> So it is
really a 0 kBytes file everywhere (all nodes
>>>> including
>>>> >>>>>>>>>>>>> the
arbiter
>>>> >>>>>>>>>>>>> and from
the client).
>>>> >>>>>>>>>>>>> Here below
you will find the output you requested.
>>>> Hopefully that
>>>> >>>>>>>>>>>>> will help
>>>> >>>>>>>>>>>>> to find
out why this specific file is not healing...
>>>> Let me know
>>>> >>>>>>>>>>>>> if you
need
>>>> >>>>>>>>>>>>> any more
information. Btw node3 is my arbiter node.
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> NODE1:
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> STAT:
>>>> >>>>>>>>>>>>> File:
>>>> >>>>>>>>>>>>>
>>>>
?/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png?
>>>> >>>>>>>>>>>>> Size: 0
Blocks: 38 IO Block: 131072 regular empty file
>>>> >>>>>>>>>>>>> Device:
24h/36d Inode: 10033884 Links: 2
>>>> >>>>>>>>>>>>> Access:
(0644/-rw-r--r--) Uid: ( 33/www-data) Gid: (
>>>> 33/www-data)
>>>> >>>>>>>>>>>>> Access:
2017-08-14 17:04:55.530681000 +0200
>>>> >>>>>>>>>>>>> Modify:
2017-08-14 17:11:46.407404779 +0200
>>>> >>>>>>>>>>>>> Change:
2017-08-14 17:11:46.407404779 +0200
>>>> >>>>>>>>>>>>> Birth: -
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> GETFATTR:
>>>> >>>>>>>>>>>>>
trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>> >>>>>>>>>>>>>
trusted.bit-rot.version=0sAgAAAAAAAABZhuknAAlJAg=>>>>
>>>>>>>>>>>>>
trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g=>>>>
>>>>>>>>>>>>>
>>>>
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOyo>>>>
>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> NODE2:
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> STAT:
>>>> >>>>>>>>>>>>> File:
>>>> >>>>>>>>>>>>>
>>>>
?/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png?
>>>> >>>>>>>>>>>>> Size: 0
Blocks: 38 IO Block: 131072 regular empty file
>>>> >>>>>>>>>>>>> Device:
26h/38d Inode: 10031330 Links: 2
>>>> >>>>>>>>>>>>> Access:
(0644/-rw-r--r--) Uid: ( 33/www-data) Gid: (
>>>> 33/www-data)
>>>> >>>>>>>>>>>>> Access:
2017-08-14 17:04:55.530681000 +0200
>>>> >>>>>>>>>>>>> Modify:
2017-08-14 17:11:46.403704181 +0200
>>>> >>>>>>>>>>>>> Change:
2017-08-14 17:11:46.403704181 +0200
>>>> >>>>>>>>>>>>> Birth: -
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> GETFATTR:
>>>> >>>>>>>>>>>>>
trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>> >>>>>>>>>>>>>
trusted.bit-rot.version=0sAgAAAAAAAABZhu6wAA8Hpw=>>>>
>>>>>>>>>>>>>
trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g=>>>>
>>>>>>>>>>>>>
>>>>
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOVE>>>>
>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> NODE3:
>>>> >>>>>>>>>>>>> STAT:
>>>> >>>>>>>>>>>>> File:
>>>> >>>>>>>>>>>>>
>>>>
/srv/glusterfs/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>> >>>>>>>>>>>>> Size: 0
Blocks: 0 IO Block: 4096 regular empty file
>>>> >>>>>>>>>>>>> Device:
ca11h/51729d Inode: 405208959 Links: 2
>>>> >>>>>>>>>>>>> Access:
(0644/-rw-r--r--) Uid: ( 33/www-data) Gid: (
>>>> 33/www-data)
>>>> >>>>>>>>>>>>> Access:
2017-08-14 17:04:55.530681000 +0200
>>>> >>>>>>>>>>>>> Modify:
2017-08-14 17:04:55.530681000 +0200
>>>> >>>>>>>>>>>>> Change:
2017-08-14 17:11:46.604380051 +0200
>>>> >>>>>>>>>>>>> Birth: -
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> GETFATTR:
>>>> >>>>>>>>>>>>>
trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>> >>>>>>>>>>>>>
trusted.bit-rot.version=0sAgAAAAAAAABZe6ejAAKPAg=>>>>
>>>>>>>>>>>>>
trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g=>>>>
>>>>>>>>>>>>>
>>>>
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOc4>>>>
>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> CLIENT
GLUSTER MOUNT:
>>>> >>>>>>>>>>>>> STAT:
>>>> >>>>>>>>>>>>> File:
>>>> >>>>>>>>>>>>>
>>>>
"/mnt/myvolume/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png"
>>>> >>>>>>>>>>>>> Size: 0
Blocks: 0 IO Block: 131072 regular empty file
>>>> >>>>>>>>>>>>> Device:
1eh/30d Inode: 11897049013408443114 Links: 1
>>>> >>>>>>>>>>>>> Access:
(0644/-rw-r--r--) Uid: ( 33/www-data) Gid: (
>>>> 33/www-data)
>>>> >>>>>>>>>>>>> Access:
2017-08-14 17:04:55.530681000 +0200
>>>> >>>>>>>>>>>>> Modify:
2017-08-14 17:11:46.407404779 +0200
>>>> >>>>>>>>>>>>> Change:
2017-08-14 17:11:46.407404779 +0200
>>>> >>>>>>>>>>>>> Birth: -
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>
-------- Original Message --------
>>>> >>>>>>>>>>>>>>
Subject: Re: [Gluster-users] self-heal not working
>>>> >>>>>>>>>>>>>> Local
Time: August 21, 2017 9:34 PM
>>>> >>>>>>>>>>>>>> UTC
Time: August 21, 2017 7:34 PM
>>>> >>>>>>>>>>>>>> From:
bturner at redhat.com
>>>> >>>>>>>>>>>>>> To:
mabi
>>>> [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)
>>>> >>>>>>>>>>>>>>
Gluster Users
>>>> >>>>>>>>>>>>>>
>>>> [<gluster-users at gluster.org>](mailto:gluster-users at
gluster.org)
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> -----
Original Message -----
>>>> >>>>>>>>>>>>>>>
From: "mabi"
>>>> [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)
>>>> >>>>>>>>>>>>>>>
To: "Gluster Users"
>>>> >>>>>>>>>>>>>>>
>>>> [<gluster-users at gluster.org>](mailto:gluster-users at
gluster.org)
>>>> >>>>>>>>>>>>>>>
Sent: Monday, August 21, 2017 9:28:24 AM
>>>> >>>>>>>>>>>>>>>
Subject: [Gluster-users] self-heal not working
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
Hi,
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> I
have a replicat 2 with arbiter GlusterFS 3.8.11
>>>> cluster and
>>>> >>>>>>>>>>>>>>>
there is
>>>> >>>>>>>>>>>>>>>
currently one file listed to be healed as you can
>>>> see below
>>>> >>>>>>>>>>>>>>>
but never gets
>>>> >>>>>>>>>>>>>>>
healed by the self-heal daemon:
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
Brick node1.domain.tld:/data/myvolume/brick
>>>> >>>>>>>>>>>>>>>
>>>> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>> >>>>>>>>>>>>>>>
Status: Connected
>>>> >>>>>>>>>>>>>>>
Number of entries: 1
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
Brick node2.domain.tld:/data/myvolume/brick
>>>> >>>>>>>>>>>>>>>
>>>> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>> >>>>>>>>>>>>>>>
Status: Connected
>>>> >>>>>>>>>>>>>>>
Number of entries: 1
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
Brick node3.domain.tld:/srv/glusterfs/myvolume/brick
>>>> >>>>>>>>>>>>>>>
>>>> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>> >>>>>>>>>>>>>>>
Status: Connected
>>>> >>>>>>>>>>>>>>>
Number of entries: 1
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> As
once recommended on this mailing list I have
>>>> mounted that
>>>> >>>>>>>>>>>>>>>
glusterfs
>>>> >>>>>>>>>>>>>>>
volume
>>>> >>>>>>>>>>>>>>>
temporarily through fuse/glusterfs and ran a "stat"
>>>> on that
>>>> >>>>>>>>>>>>>>>
file which is
>>>> >>>>>>>>>>>>>>>
listed above but nothing happened.
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
The file itself is available on all 3 nodes/bricks
>>>> but on the
>>>> >>>>>>>>>>>>>>>
last node it
>>>> >>>>>>>>>>>>>>>
has a different date. By the way this file is 0
>>>> kBytes big. Is
>>>> >>>>>>>>>>>>>>>
that maybe
>>>> >>>>>>>>>>>>>>>
the reason why the self-heal does not work?
>>>> >>>>>>>>>>>>>> Is the
file actually 0 bytes or is it just 0 bytes
>>>> on the
>>>> >>>>>>>>>>>>>>
arbiter(0 bytes
>>>> >>>>>>>>>>>>>> are
expected on the arbiter, it just stores
>>>> metadata)? Can you
>>>> >>>>>>>>>>>>>> send
us the
>>>> >>>>>>>>>>>>>> output
from stat on all 3 nodes:
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> $ stat
<file on back end brick>
>>>> >>>>>>>>>>>>>> $
getfattr -d -m - <file on back end brick>
>>>> >>>>>>>>>>>>>> $ stat
<file from gluster mount>
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> Lets
see what things look like on the back end, it
>>>> should tell
>>>> >>>>>>>>>>>>>> us why
>>>> >>>>>>>>>>>>>>
healing is failing.
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> -b
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
And how can I now make this file to heal?
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
Thanks,
>>>> >>>>>>>>>>>>>>>
Mabi
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
_______________________________________________
>>>> >>>>>>>>>>>>>>>
Gluster-users mailing list
>>>> >>>>>>>>>>>>>>>
Gluster-users at gluster.org
>>>> >>>>>>>>>>>>>>>
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>> >>>>>>>>>>>
_______________________________________________
>>>> >>>>>>>>>>> Gluster-users
mailing list
>>>> >>>>>>>>>>> Gluster-users at
gluster.org
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170828/046e2836/attachment.html>