[Gluster-users] self-heal not working

mabi
Mon, 28 Aug 2017 14:19:55 -0000

Thank you for the command. I ran it on all my nodes and now finally the the
self-heal daemon does not report any files to be healed. Hopefully this scenario
can get handled properly in newer versions of GlusterFS.

> -------- Original Message --------
> Subject: Re: [Gluster-users] self-heal not working
> Local Time: August 28, 2017 10:41 AM
> UTC Time: August 28, 2017 8:41 AM
> From: ravishankar at redhat.com
> To: mabi <mabi at protonmail.ch>
> Ben Turner <bturner at redhat.com>, Gluster Users <gluster-users
at gluster.org>
>
> On 08/28/2017 01:29 PM, mabi wrote:
>
>> Excuse me for my naive questions but how do I reset the afr.dirty xattr
on the file to be healed? and do I need to do that through a FUSE mount? or
simply on every bricks directly?
>
> Directly on the bricks: `setfattr -n trusted.afr.dirty -v
0x000000000000000000000000
/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png`
> -Ravi
>
>>> -------- Original Message --------
>>> Subject: Re: [Gluster-users] self-heal not working
>>> Local Time: August 28, 2017 5:58 AM
>>> UTC Time: August 28, 2017 3:58 AM
>>> From: ravishankar at redhat.com
>>> To: Ben Turner [<bturner at redhat.com>](mailto:bturner at
redhat.com), mabi [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)
>>> Gluster Users [<gluster-users at
gluster.org>](mailto:gluster-users at gluster.org)
>>>
>>> On 08/28/2017 01:57 AM, Ben Turner wrote:
>>>> ----- Original Message -----
>>>>> From: "mabi" [<mabi at
protonmail.ch>](mailto:mabi at protonmail.ch)
>>>>> To: "Ravishankar N" [<ravishankar at
redhat.com>](mailto:ravishankar at redhat.com)
>>>>> Cc: "Ben Turner" [<bturner at
redhat.com>](mailto:bturner at redhat.com), "Gluster Users"
[<gluster-users at gluster.org>](mailto:gluster-users at gluster.org)
>>>>> Sent: Sunday, August 27, 2017 3:15:33 PM
>>>>> Subject: Re: [Gluster-users] self-heal not working
>>>>>
>>>>> Thanks Ravi for your analysis. So as far as I understand
nothing to worry
>>>>> about but my question now would be: how do I get rid of
this file from the
>>>>> heal info?
>>>> Correct me if I am wrong but clearing this is just a matter of
resetting the afr.dirty xattr? @Ravi - Is this correct?
>>>
>>> Yes resetting the xattr and launching index heal or running
heal-info
>>> command should serve as a workaround.
>>> -Ravi
>>>
>>>>
>>>> -b
>>>>
>>>>>> -------- Original Message --------
>>>>>> Subject: Re: [Gluster-users] self-heal not working
>>>>>> Local Time: August 27, 2017 3:45 PM
>>>>>> UTC Time: August 27, 2017 1:45 PM
>>>>>> From: ravishankar at redhat.com
>>>>>> To: mabi [<mabi at protonmail.ch>](mailto:mabi at
protonmail.ch)
>>>>>> Ben Turner [<bturner at
redhat.com>](mailto:bturner at redhat.com), Gluster Users [<gluster-users
at gluster.org>](mailto:gluster-users at gluster.org)
>>>>>>
>>>>>> Yes, the shds did pick up the file for healing (I saw
messages like " got
>>>>>> entry: 1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea") but
no error afterwards.
>>>>>>
>>>>>> Anyway I reproduced it by manually setting the
afr.dirty bit for a zero
>>>>>> byte file on all 3 bricks. Since there are no afr
pending xattrs
>>>>>> indicating good/bad copies and all files are zero
bytes, the data
>>>>>> self-heal algorithm just picks the file with the latest
ctime as source.
>>>>>> In your case that was the arbiter brick. In the code,
there is a check to
>>>>>> prevent data heals if arbiter is the source. So heal
was not happening and
>>>>>> the entries were not removed from heal-info output.
>>>>>>
>>>>>> Perhaps we should add a check in the code to just
remove the entries from
>>>>>> heal-info if size is zero bytes in all bricks.
>>>>>>
>>>>>> -Ravi
>>>>>>
>>>>>> On 08/25/2017 06:33 PM, mabi wrote:
>>>>>>
>>>>>>> Hi Ravi,
>>>>>>>
>>>>>>> Did you get a chance to have a look at the log
files I have attached in my
>>>>>>> last mail?
>>>>>>>
>>>>>>> Best,
>>>>>>> Mabi
>>>>>>>
>>>>>>>> -------- Original Message --------
>>>>>>>> Subject: Re: [Gluster-users] self-heal not
working
>>>>>>>> Local Time: August 24, 2017 12:08 PM
>>>>>>>> UTC Time: August 24, 2017 10:08 AM
>>>>>>>> From: mabi at protonmail.ch
>>>>>>>> To: Ravishankar N
>>>>>>>> [[<ravishankar at
redhat.com>](mailto:ravishankar at redhat.com)](mailto:ravishankar at
redhat.com)
>>>>>>>> Ben Turner [[<bturner at
redhat.com>](mailto:bturner at redhat.com)](mailto:bturner at redhat.com),
Gluster
>>>>>>>> Users [[<gluster-users at
gluster.org>](mailto:gluster-users at gluster.org)](mailto:gluster-users at
gluster.org)
>>>>>>>>
>>>>>>>> Thanks for confirming the command. I have now
enabled DEBUG
>>>>>>>> client-log-level, run a heal and then attached
the glustershd log files
>>>>>>>> of all 3 nodes in this mail.
>>>>>>>>
>>>>>>>> The volume concerned is called myvol-pro, the
other 3 volumes have no
>>>>>>>> problem so far.
>>>>>>>>
>>>>>>>> Also note that in the mean time it looks like
the file has been deleted
>>>>>>>> by the user and as such the heal info command
does not show the file
>>>>>>>> name anymore but just is GFID which is:
>>>>>>>>
>>>>>>>> gfid:1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea
>>>>>>>>
>>>>>>>> Hope that helps for debugging this issue.
>>>>>>>>
>>>>>>>>> -------- Original Message --------
>>>>>>>>> Subject: Re: [Gluster-users] self-heal not
working
>>>>>>>>> Local Time: August 24, 2017 5:58 AM
>>>>>>>>> UTC Time: August 24, 2017 3:58 AM
>>>>>>>>> From: ravishankar at redhat.com
>>>>>>>>> To: mabi [[<mabi at
protonmail.ch>](mailto:mabi at protonmail.ch)](mailto:mabi at protonmail.ch)
>>>>>>>>> Ben Turner [[<bturner at
redhat.com>](mailto:bturner at redhat.com)](mailto:bturner at redhat.com),
Gluster
>>>>>>>>> Users [[<gluster-users at
gluster.org>](mailto:gluster-users at gluster.org)](mailto:gluster-users at
gluster.org)
>>>>>>>>>
>>>>>>>>> Unlikely. In your case only the afr.dirty
is set, not the
>>>>>>>>> afr.volname-client-xx xattr.
>>>>>>>>>
>>>>>>>>> `gluster volume set myvolume
diagnostics.client-log-level DEBUG` is
>>>>>>>>> right.
>>>>>>>>>
>>>>>>>>> On 08/23/2017 10:31 PM, mabi wrote:
>>>>>>>>>
>>>>>>>>>> I just saw the following bug which was
fixed in 3.8.15:
>>>>>>>>>>
>>>>>>>>>>
https://bugzilla.redhat.com/show_bug.cgi?id=1471613
>>>>>>>>>>
>>>>>>>>>> Is it possible that the problem I
described in this post is related to
>>>>>>>>>> that bug?
>>>>>>>>>>
>>>>>>>>>>> -------- Original Message --------
>>>>>>>>>>> Subject: Re: [Gluster-users]
self-heal not working
>>>>>>>>>>> Local Time: August 22, 2017 11:51
AM
>>>>>>>>>>> UTC Time: August 22, 2017 9:51 AM
>>>>>>>>>>> From: ravishankar at redhat.com
>>>>>>>>>>> To: mabi [[<mabi at
protonmail.ch>](mailto:mabi at protonmail.ch)](mailto:mabi at protonmail.ch)
>>>>>>>>>>> Ben Turner [[<bturner at
redhat.com>](mailto:bturner at redhat.com)](mailto:bturner at redhat.com),
Gluster
>>>>>>>>>>> Users [[<gluster-users at
gluster.org>](mailto:gluster-users at gluster.org)](mailto:gluster-users at
gluster.org)
>>>>>>>>>>>
>>>>>>>>>>> On 08/22/2017 02:30 PM, mabi wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the additional
hints, I have the following 2 questions
>>>>>>>>>>>> first:
>>>>>>>>>>>>
>>>>>>>>>>>> - In order to launch the index
heal is the following command correct:
>>>>>>>>>>>> gluster volume heal myvolume
>>>>>>>>>>> Yes
>>>>>>>>>>>
>>>>>>>>>>>> - If I run a "volume start
force" will it have any short disruptions
>>>>>>>>>>>> on my clients which mount the
volume through FUSE? If yes, how long?
>>>>>>>>>>>> This is a production system
that"s why I am asking.
>>>>>>>>>>> No. You can actually create a test
volume on your personal linux box
>>>>>>>>>>> to try these kinds of things
without needing multiple machines. This
>>>>>>>>>>> is how we develop and test our
patches :)
>>>>>>>>>>> "gluster volume create testvol
replica 3 /home/mabi/bricks/brick{1..3}
>>>>>>>>>>> force` and so on.
>>>>>>>>>>>
>>>>>>>>>>> HTH,
>>>>>>>>>>> Ravi
>>>>>>>>>>>
>>>>>>>>>>>>> -------- Original Message
--------
>>>>>>>>>>>>> Subject: Re:
[Gluster-users] self-heal not working
>>>>>>>>>>>>> Local Time: August 22, 2017
6:26 AM
>>>>>>>>>>>>> UTC Time: August 22, 2017
4:26 AM
>>>>>>>>>>>>> From: ravishankar at
redhat.com
>>>>>>>>>>>>> To: mabi [[<mabi at
protonmail.ch>](mailto:mabi at protonmail.ch)](mailto:mabi at protonmail.ch),
Ben
>>>>>>>>>>>>> Turner [[<bturner at
redhat.com>](mailto:bturner at redhat.com)](mailto:bturner at redhat.com)
>>>>>>>>>>>>> Gluster Users
>>>>>>>>>>>>> [[<gluster-users at
gluster.org>](mailto:gluster-users at gluster.org)](mailto:gluster-users at
gluster.org)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Explore the following:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Launch index heal and
look at the glustershd logs of all bricks
>>>>>>>>>>>>> for possible errors
>>>>>>>>>>>>>
>>>>>>>>>>>>> - See if the glustershd in
each node is connected to all bricks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - If not try to restart shd
by `volume start force`
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Launch index heal again
and try.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Try debugging the shd log
by setting client-log-level to DEBUG
>>>>>>>>>>>>> temporarily.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 08/22/2017 03:19 AM,
mabi wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sure, it doesn"t
look like a split brain based on the output:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Brick
node1.domain.tld:/data/myvolume/brick
>>>>>>>>>>>>>> Status: Connected
>>>>>>>>>>>>>> Number of entries in
split-brain: 0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Brick
node2.domain.tld:/data/myvolume/brick
>>>>>>>>>>>>>> Status: Connected
>>>>>>>>>>>>>> Number of entries in
split-brain: 0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Brick
node3.domain.tld:/srv/glusterfs/myvolume/brick
>>>>>>>>>>>>>> Status: Connected
>>>>>>>>>>>>>> Number of entries in
split-brain: 0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -------- Original
Message --------
>>>>>>>>>>>>>>> Subject: Re:
[Gluster-users] self-heal not working
>>>>>>>>>>>>>>> Local Time: August
21, 2017 11:35 PM
>>>>>>>>>>>>>>> UTC Time: August
21, 2017 9:35 PM
>>>>>>>>>>>>>>> From: bturner at
redhat.com
>>>>>>>>>>>>>>> To: mabi [[<mabi
at protonmail.ch>](mailto:mabi at protonmail.ch)](mailto:mabi at
protonmail.ch)
>>>>>>>>>>>>>>> Gluster Users
>>>>>>>>>>>>>>> [[<gluster-users
at gluster.org>](mailto:gluster-users at gluster.org)](mailto:gluster-users
at gluster.org)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you also
provide:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> gluster v heal
<my vol> info split-brain
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If it is split
brain just delete the incorrect file from the brick
>>>>>>>>>>>>>>> and run heal again.
I haven"t tried this with arbiter but I
>>>>>>>>>>>>>>> assume the process
is the same.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -b
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----- Original
Message -----
>>>>>>>>>>>>>>>> From:
"mabi" [[<mabi at protonmail.ch>](mailto:mabi at
protonmail.ch)](mailto:mabi at protonmail.ch)
>>>>>>>>>>>>>>>> To: "Ben
Turner"
>>>>>>>>>>>>>>>> [[<bturner
at redhat.com>](mailto:bturner at redhat.com)](mailto:bturner at redhat.com)
>>>>>>>>>>>>>>>> Cc:
"Gluster Users"
>>>>>>>>>>>>>>>>
[[<gluster-users at gluster.org>](mailto:gluster-users at
gluster.org)](mailto:gluster-users at gluster.org)
>>>>>>>>>>>>>>>> Sent: Monday,
August 21, 2017 4:55:59 PM
>>>>>>>>>>>>>>>> Subject: Re:
[Gluster-users] self-heal not working
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Ben,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So it is really
a 0 kBytes file everywhere (all nodes including
>>>>>>>>>>>>>>>> the arbiter
>>>>>>>>>>>>>>>> and from the
client).
>>>>>>>>>>>>>>>> Here below you
will find the output you requested. Hopefully that
>>>>>>>>>>>>>>>> will help
>>>>>>>>>>>>>>>> to find out why
this specific file is not healing... Let me know
>>>>>>>>>>>>>>>> if you need
>>>>>>>>>>>>>>>> any more
information. Btw node3 is my arbiter node.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> NODE1:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> STAT:
>>>>>>>>>>>>>>>> File:
>>>>>>>>>>>>>>>>
?/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png?
>>>>>>>>>>>>>>>> Size: 0 Blocks:
38 IO Block: 131072 regular empty file
>>>>>>>>>>>>>>>> Device: 24h/36d
Inode: 10033884 Links: 2
>>>>>>>>>>>>>>>> Access:
(0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>>>>>>>>>>>> Access:
2017-08-14 17:04:55.530681000 +0200
>>>>>>>>>>>>>>>> Modify:
2017-08-14 17:11:46.407404779 +0200
>>>>>>>>>>>>>>>> Change:
2017-08-14 17:11:46.407404779 +0200
>>>>>>>>>>>>>>>> Birth: -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> GETFATTR:
>>>>>>>>>>>>>>>>
trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>>>>>>>>>>>>>>
trusted.bit-rot.version=0sAgAAAAAAAABZhuknAAlJAg=>>>>>>>>>>>>>>>>
trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g=>>>>>>>>>>>>>>>>
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOyo>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> NODE2:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> STAT:
>>>>>>>>>>>>>>>> File:
>>>>>>>>>>>>>>>>
?/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png?
>>>>>>>>>>>>>>>> Size: 0 Blocks:
38 IO Block: 131072 regular empty file
>>>>>>>>>>>>>>>> Device: 26h/38d
Inode: 10031330 Links: 2
>>>>>>>>>>>>>>>> Access:
(0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>>>>>>>>>>>> Access:
2017-08-14 17:04:55.530681000 +0200
>>>>>>>>>>>>>>>> Modify:
2017-08-14 17:11:46.403704181 +0200
>>>>>>>>>>>>>>>> Change:
2017-08-14 17:11:46.403704181 +0200
>>>>>>>>>>>>>>>> Birth: -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> GETFATTR:
>>>>>>>>>>>>>>>>
trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>>>>>>>>>>>>>>
trusted.bit-rot.version=0sAgAAAAAAAABZhu6wAA8Hpw=>>>>>>>>>>>>>>>>
trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g=>>>>>>>>>>>>>>>>
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOVE>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> NODE3:
>>>>>>>>>>>>>>>> STAT:
>>>>>>>>>>>>>>>> File:
>>>>>>>>>>>>>>>>
/srv/glusterfs/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>>>>>>>>>>>> Size: 0 Blocks:
0 IO Block: 4096 regular empty file
>>>>>>>>>>>>>>>> Device:
ca11h/51729d Inode: 405208959 Links: 2
>>>>>>>>>>>>>>>> Access:
(0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>>>>>>>>>>>> Access:
2017-08-14 17:04:55.530681000 +0200
>>>>>>>>>>>>>>>> Modify:
2017-08-14 17:04:55.530681000 +0200
>>>>>>>>>>>>>>>> Change:
2017-08-14 17:11:46.604380051 +0200
>>>>>>>>>>>>>>>> Birth: -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> GETFATTR:
>>>>>>>>>>>>>>>>
trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>>>>>>>>>>>>>>
trusted.bit-rot.version=0sAgAAAAAAAABZe6ejAAKPAg=>>>>>>>>>>>>>>>>
trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g=>>>>>>>>>>>>>>>>
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOc4>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CLIENT GLUSTER
MOUNT:
>>>>>>>>>>>>>>>> STAT:
>>>>>>>>>>>>>>>> File:
>>>>>>>>>>>>>>>>
"/mnt/myvolume/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png"
>>>>>>>>>>>>>>>> Size: 0 Blocks:
0 IO Block: 131072 regular empty file
>>>>>>>>>>>>>>>> Device: 1eh/30d
Inode: 11897049013408443114 Links: 1
>>>>>>>>>>>>>>>> Access:
(0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>>>>>>>>>>>> Access:
2017-08-14 17:04:55.530681000 +0200
>>>>>>>>>>>>>>>> Modify:
2017-08-14 17:11:46.407404779 +0200
>>>>>>>>>>>>>>>> Change:
2017-08-14 17:11:46.407404779 +0200
>>>>>>>>>>>>>>>> Birth: -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --------
Original Message --------
>>>>>>>>>>>>>>>>> Subject:
Re: [Gluster-users] self-heal not working
>>>>>>>>>>>>>>>>> Local Time:
August 21, 2017 9:34 PM
>>>>>>>>>>>>>>>>> UTC Time:
August 21, 2017 7:34 PM
>>>>>>>>>>>>>>>>> From:
bturner at redhat.com
>>>>>>>>>>>>>>>>> To: mabi
[[<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)](mailto:mabi at
protonmail.ch)
>>>>>>>>>>>>>>>>> Gluster
Users
>>>>>>>>>>>>>>>>>
[[<gluster-users at gluster.org>](mailto:gluster-users at
gluster.org)](mailto:gluster-users at gluster.org)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -----
Original Message -----
>>>>>>>>>>>>>>>>>> From:
"mabi" [[<mabi at protonmail.ch>](mailto:mabi at
protonmail.ch)](mailto:mabi at protonmail.ch)
>>>>>>>>>>>>>>>>>> To:
"Gluster Users"
>>>>>>>>>>>>>>>>>>
[[<gluster-users at gluster.org>](mailto:gluster-users at
gluster.org)](mailto:gluster-users at gluster.org)
>>>>>>>>>>>>>>>>>> Sent:
Monday, August 21, 2017 9:28:24 AM
>>>>>>>>>>>>>>>>>>
Subject: [Gluster-users] self-heal not working
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have
a replicat 2 with arbiter GlusterFS 3.8.11 cluster and
>>>>>>>>>>>>>>>>>> there
is
>>>>>>>>>>>>>>>>>>
currently one file listed to be healed as you can see below
>>>>>>>>>>>>>>>>>> but
never gets
>>>>>>>>>>>>>>>>>> healed
by the self-heal daemon:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Brick
node1.domain.tld:/data/myvolume/brick
>>>>>>>>>>>>>>>>>>
/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>>>>>>>>>>>>>> Status:
Connected
>>>>>>>>>>>>>>>>>> Number
of entries: 1
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Brick
node2.domain.tld:/data/myvolume/brick
>>>>>>>>>>>>>>>>>>
/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>>>>>>>>>>>>>> Status:
Connected
>>>>>>>>>>>>>>>>>> Number
of entries: 1
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Brick
node3.domain.tld:/srv/glusterfs/myvolume/brick
>>>>>>>>>>>>>>>>>>
/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>>>>>>>>>>>>>> Status:
Connected
>>>>>>>>>>>>>>>>>> Number
of entries: 1
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As once
recommended on this mailing list I have mounted that
>>>>>>>>>>>>>>>>>>
glusterfs
>>>>>>>>>>>>>>>>>> volume
>>>>>>>>>>>>>>>>>>
temporarily through fuse/glusterfs and ran a "stat" on that
>>>>>>>>>>>>>>>>>> file
which is
>>>>>>>>>>>>>>>>>> listed
above but nothing happened.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The
file itself is available on all 3 nodes/bricks but on the
>>>>>>>>>>>>>>>>>> last
node it
>>>>>>>>>>>>>>>>>> has a
different date. By the way this file is 0 kBytes big. Is
>>>>>>>>>>>>>>>>>> that
maybe
>>>>>>>>>>>>>>>>>> the
reason why the self-heal does not work?
>>>>>>>>>>>>>>>>> Is the file
actually 0 bytes or is it just 0 bytes on the
>>>>>>>>>>>>>>>>> arbiter(0
bytes
>>>>>>>>>>>>>>>>> are
expected on the arbiter, it just stores metadata)? Can you
>>>>>>>>>>>>>>>>> send us the
>>>>>>>>>>>>>>>>> output from
stat on all 3 nodes:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> $ stat
<file on back end brick>
>>>>>>>>>>>>>>>>> $ getfattr
-d -m - <file on back end brick>
>>>>>>>>>>>>>>>>> $ stat
<file from gluster mount>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Lets see
what things look like on the back end, it should tell
>>>>>>>>>>>>>>>>> us why
>>>>>>>>>>>>>>>>> healing is
failing.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -b
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> And how
can I now make this file to heal?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Mabi
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>>>>>>>>
Gluster-users mailing list
>>>>>>>>>>>>>>>>>>
Gluster-users at gluster.org
>>>>>>>>>>>>>>>>>>
http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>>>> Gluster-users mailing
list
>>>>>>>>>>>>>> Gluster-users at
gluster.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170828/326a8b91/attachment.html>