Same issue brought up glusterd on problem node heal count still stuck at
6330.
Ran gluster v heal GUSTER1 full
glustershd on problem node shows a sweep starting and finishing in
seconds. Other 2 nodes show no activity in log. They should start a sweep
too shouldn't they?
Tried starting from scratch
kill -15 brickpid
rm -Rf /brick
mkdir -p /brick
mkdir mkdir /gsmount/fake2
setfattr -n "user.some-name" -v "some-value" /gsmount/fake2
Heals visible dirs instantly then stops.
gluster v heal GLUSTER1 full
see sweep star on problem node and end almost instantly. no files added t
heal list no files healed no more logging
[2016-08-30 23:11:31.544331] I [MSGID: 108026]
[afr-self-heald.c:646:afr_shd_full_healer] 0-GLUSTER1-replicate-0: starting
full sweep on subvol GLUSTER1-client-1
[2016-08-30 23:11:33.776235] I [MSGID: 108026]
[afr-self-heald.c:656:afr_shd_full_healer] 0-GLUSTER1-replicate-0: finished
full sweep on subvol GLUSTER1-client-1
same results no matter which node you run command on. Still stuck with
6330 files showing needing healed out of 19k. still showing in logs no
heals are occuring.
Is their a way to forcibly reset any prior heal data? Could it be stuck on
some past failed heal start?
*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
On Tue, Aug 30, 2016 at 10:03 AM, David Gossage <dgossage at
carouselchecks.com> wrote:
> On Tue, Aug 30, 2016 at 10:02 AM, David Gossage <
> dgossage at carouselchecks.com> wrote:
>
>> updated test server to 3.8.3
>>
>> Brick1: 192.168.71.10:/gluster2/brick1/1
>> Brick2: 192.168.71.11:/gluster2/brick2/1
>> Brick3: 192.168.71.12:/gluster2/brick3/1
>> Options Reconfigured:
>> cluster.granular-entry-heal: on
>> performance.readdir-ahead: on
>> performance.read-ahead: off
>> nfs.disable: on
>> nfs.addr-namelookup: off
>> nfs.enable-ino32: off
>> cluster.background-self-heal-count: 16
>> cluster.self-heal-window-size: 1024
>> performance.quick-read: off
>> performance.io-cache: off
>> performance.stat-prefetch: off
>> cluster.eager-lock: enable
>> network.remote-dio: on
>> cluster.quorum-type: auto
>> cluster.server-quorum-type: server
>> storage.owner-gid: 36
>> storage.owner-uid: 36
>> server.allow-insecure: on
>> features.shard: on
>> features.shard-block-size: 64MB
>> performance.strict-o-direct: off
>> cluster.locking-scheme: granular
>>
>> kill -15 brickpid
>> rm -Rf /gluster2/brick3
>> mkdir -p /gluster2/brick3/1
>> mkdir mkdir
/rhev/data-center/mnt/glusterSD/192.168.71.10\:_glustershard
>> /fake2
>> setfattr -n "user.some-name" -v "some-value"
>> /rhev/data-center/mnt/glusterSD/192.168.71.10\:_glustershard/fake2
>> gluster v start glustershard force
>>
>> at this point brick process starts and all visible files including new
>> dir are made on brick
>> handful of shards are in heal statistics still but no .shard directory
>> created and no increase in shard count
>>
>> gluster v heal glustershard
>>
>> At this point still no increase in count or dir made no additional
>> activity in logs for healing generated. waited few minutes tailing
logs to
>> check if anything kicked in.
>>
>> gluster v heal glustershard full
>>
>> gluster shards added to list and heal commences. logs show full sweep
>> starting on all 3 nodes. though this time it only shows as finishing
on
>> one which looks to be the one that had brick deleted.
>>
>> [2016-08-30 14:45:33.098589] I [MSGID: 108026]
>> [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
>> starting full sweep on subvol glustershard-client-0
>> [2016-08-30 14:45:33.099492] I [MSGID: 108026]
>> [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
>> starting full sweep on subvol glustershard-client-1
>> [2016-08-30 14:45:33.100093] I [MSGID: 108026]
>> [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
>> starting full sweep on subvol glustershard-client-2
>> [2016-08-30 14:52:29.760213] I [MSGID: 108026]
>> [afr-self-heald.c:656:afr_shd_full_healer] 0-glustershard-replicate-0:
>> finished full sweep on subvol glustershard-client-2
>>
>
> Just realized its still healing so that may be why sweep on 2 other bricks
> haven't replied as finished.
>
>>
>>
>> my hope is that later tonight a full heal will work on production. Is
it
>> possible self-heal daemon can get stale or stop listening but still
show as
>> active? Would stopping and starting self-heal daemon from gluster cli
>> before doing these heals be helpful?
>>
>>
>> On Tue, Aug 30, 2016 at 9:29 AM, David Gossage <
>> dgossage at carouselchecks.com> wrote:
>>
>>> On Tue, Aug 30, 2016 at 8:52 AM, David Gossage <
>>> dgossage at carouselchecks.com> wrote:
>>>
>>>> On Tue, Aug 30, 2016 at 8:01 AM, Krutika Dhananjay <kdhananj
at redhat.com
>>>> > wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2016 at 6:20 PM, Krutika Dhananjay <
>>>>> kdhananj at redhat.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 30, 2016 at 6:07 PM, David Gossage <
>>>>>> dgossage at carouselchecks.com> wrote:
>>>>>>
>>>>>>> On Tue, Aug 30, 2016 at 7:18 AM, Krutika Dhananjay
<
>>>>>>> kdhananj at redhat.com> wrote:
>>>>>>>
>>>>>>>> Could you also share the glustershd logs?
>>>>>>>>
>>>>>>>
>>>>>>> I'll get them when I get to work sure
>>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I tried the same steps that you mentioned
multiple times, but heal
>>>>>>>> is running to completion without any issues.
>>>>>>>>
>>>>>>>> It must be said that 'heal full'
traverses the files and
>>>>>>>> directories in a depth-first order and does
heals also in the same order.
>>>>>>>> But if it gets interrupted in the middle (say
because self-heal-daemon was
>>>>>>>> either intentionally or unintentionally brought
offline and then brought
>>>>>>>> back up), self-heal will only pick up the
entries that are so far marked as
>>>>>>>> new-entries that need heal which it will find
in indices/xattrop directory.
>>>>>>>> What this means is that those files and
directories that were not visited
>>>>>>>> during the crawl, will remain untouched and
unhealed in this second
>>>>>>>> iteration of heal, unless you execute a
'heal-full' again.
>>>>>>>>
>>>>>>>
>>>>>>> So should it start healing shards as it crawls or
not until after it
>>>>>>> crawls the entire .shard directory? At the pace it
was going that could be
>>>>>>> a week with one node appearing in the cluster but
with no shard files if
>>>>>>> anything tries to access a file on that node. From
my experience other day
>>>>>>> telling it to heal full again did nothing
regardless of node used.
>>>>>>>
>>>>>>
>>>>> Crawl is started from '/' of the volume. Whenever
self-heal detects
>>>>> during the crawl that a file or directory is present in
some brick(s) and
>>>>> absent in others, it creates the file on the bricks where
it is absent and
>>>>> marks the fact that the file or directory might need
data/entry and
>>>>> metadata heal too (this also means that an index is created
under
>>>>> .glusterfs/indices/xattrop of the src bricks). And the
data/entry and
>>>>> metadata heal are picked up and done in
>>>>>
>>>> the background with the help of these indices.
>>>>>
>>>>
>>>> Looking at my 3rd node as example i find nearly an exact same
number of
>>>> files in xattrop dir as reported by heal count at time I
brought down node2
>>>> to try and alleviate read io errors that seemed to occur from
what I was
>>>> guessing as attempts to use the node with no shards for reads.
>>>>
>>>> Also attached are the glustershd logs from the 3 nodes, along
with the
>>>> test node i tried yesterday with same results.
>>>>
>>>
>>> Looking at my own logs I notice that a full sweep was only ever
recorded
>>> in glustershd.log on 2nd node with missing directory. I believe I
should
>>> have found a sweep begun on every node correct?
>>>
>>> On my test dev when it did work I do see that
>>>
>>> [2016-08-30 13:56:25.223333] I [MSGID: 108026]
>>> [afr-self-heald.c:646:afr_shd_full_healer]
0-glustershard-replicate-0:
>>> starting full sweep on subvol glustershard-client-0
>>> [2016-08-30 13:56:25.223522] I [MSGID: 108026]
>>> [afr-self-heald.c:646:afr_shd_full_healer]
0-glustershard-replicate-0:
>>> starting full sweep on subvol glustershard-client-1
>>> [2016-08-30 13:56:25.224616] I [MSGID: 108026]
>>> [afr-self-heald.c:646:afr_shd_full_healer]
0-glustershard-replicate-0:
>>> starting full sweep on subvol glustershard-client-2
>>> [2016-08-30 14:18:48.333740] I [MSGID: 108026]
>>> [afr-self-heald.c:656:afr_shd_full_healer]
0-glustershard-replicate-0:
>>> finished full sweep on subvol glustershard-client-2
>>> [2016-08-30 14:18:48.356008] I [MSGID: 108026]
>>> [afr-self-heald.c:656:afr_shd_full_healer]
0-glustershard-replicate-0:
>>> finished full sweep on subvol glustershard-client-1
>>> [2016-08-30 14:18:49.637811] I [MSGID: 108026]
>>> [afr-self-heald.c:656:afr_shd_full_healer]
0-glustershard-replicate-0:
>>> finished full sweep on subvol glustershard-client-0
>>>
>>> While when looking at past few days of the 3 prod nodes i only
found
>>> that on my 2nd node
>>> [2016-08-27 01:26:42.638772] I [MSGID: 108026]
>>> [afr-self-heald.c:646:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
>>> starting full sweep on subvol GLUSTER1-client-1
>>> [2016-08-27 11:37:01.732366] I [MSGID: 108026]
>>> [afr-self-heald.c:656:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
>>> finished full sweep on subvol GLUSTER1-client-1
>>> [2016-08-27 12:58:34.597228] I [MSGID: 108026]
>>> [afr-self-heald.c:646:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
>>> starting full sweep on subvol GLUSTER1-client-1
>>> [2016-08-27 12:59:28.041173] I [MSGID: 108026]
>>> [afr-self-heald.c:656:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
>>> finished full sweep on subvol GLUSTER1-client-1
>>> [2016-08-27 20:03:42.560188] I [MSGID: 108026]
>>> [afr-self-heald.c:646:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
>>> starting full sweep on subvol GLUSTER1-client-1
>>> [2016-08-27 20:03:44.278274] I [MSGID: 108026]
>>> [afr-self-heald.c:656:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
>>> finished full sweep on subvol GLUSTER1-client-1
>>> [2016-08-27 21:00:42.603315] I [MSGID: 108026]
>>> [afr-self-heald.c:646:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
>>> starting full sweep on subvol GLUSTER1-client-1
>>> [2016-08-27 21:00:46.148674] I [MSGID: 108026]
>>> [afr-self-heald.c:656:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
>>> finished full sweep on subvol GLUSTER1-client-1
>>>
>>>
>>>
>>>
>>>
>>>>
>>>>>
>>>>>>>
>>>>>>>> My suspicion is that this is what happened on
your setup. Could you
>>>>>>>> confirm if that was the case?
>>>>>>>>
>>>>>>>
>>>>>>> Brick was brought online with force start then a
full heal
>>>>>>> launched. Hours later after it became evident that
it was not adding new
>>>>>>> files to heal I did try restarting self-heal daemon
and relaunching full
>>>>>>> heal again. But this was after the heal had
basically already failed to
>>>>>>> work as intended.
>>>>>>>
>>>>>>
>>>>>> OK. How did you figure it was not adding any new files?
I need to
>>>>>> know what places you were monitoring to come to this
conclusion.
>>>>>>
>>>>>> -Krutika
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> As for those logs, I did manager to do
something that caused these
>>>>>>>> warning messages you shared earlier to appear
in my client and server logs.
>>>>>>>> Although these logs are annoying and a bit
scary too, they didn't
>>>>>>>> do any harm to the data in my volume. Why they
appear just after a brick is
>>>>>>>> replaced and under no other circumstances is
something I'm still
>>>>>>>> investigating.
>>>>>>>>
>>>>>>>> But for future, it would be good to follow the
steps Anuradha gave
>>>>>>>> as that would allow self-heal to at least
detect that it has some repairing
>>>>>>>> to do whenever it is restarted whether
intentionally or otherwise.
>>>>>>>>
>>>>>>>
>>>>>>> I followed those steps as described on my test box
and ended up with
>>>>>>> exact same outcome of adding shards at an agonizing
slow pace and no
>>>>>>> creation of .shard directory or heals on shard
directory. Directories
>>>>>>> visible from mount healed quickly. This was with
one VM so it has only 800
>>>>>>> shards as well. After hours at work it had added a
total of 33 shards to
>>>>>>> be healed. I sent those logs yesterday as well
though not the glustershd.
>>>>>>>
>>>>>>> Does replace-brick command copy files in same
manner? For these
>>>>>>> purposes I am contemplating just skipping the heal
route.
>>>>>>>
>>>>>>>
>>>>>>>> -Krutika
>>>>>>>>
>>>>>>>> On Tue, Aug 30, 2016 at 2:22 AM, David Gossage
<
>>>>>>>> dgossage at carouselchecks.com> wrote:
>>>>>>>>
>>>>>>>>> attached brick and client logs from test
machine where same
>>>>>>>>> behavior occurred not sure if anything new
is there. its still on 3.8.2
>>>>>>>>>
>>>>>>>>> Number of Bricks: 1 x 3 = 3
>>>>>>>>> Transport-type: tcp
>>>>>>>>> Bricks:
>>>>>>>>> Brick1: 192.168.71.10:/gluster2/brick1/1
>>>>>>>>> Brick2: 192.168.71.11:/gluster2/brick2/1
>>>>>>>>> Brick3: 192.168.71.12:/gluster2/brick3/1
>>>>>>>>> Options Reconfigured:
>>>>>>>>> cluster.locking-scheme: granular
>>>>>>>>> performance.strict-o-direct: off
>>>>>>>>> features.shard-block-size: 64MB
>>>>>>>>> features.shard: on
>>>>>>>>> server.allow-insecure: on
>>>>>>>>> storage.owner-uid: 36
>>>>>>>>> storage.owner-gid: 36
>>>>>>>>> cluster.server-quorum-type: server
>>>>>>>>> cluster.quorum-type: auto
>>>>>>>>> network.remote-dio: on
>>>>>>>>> cluster.eager-lock: enable
>>>>>>>>> performance.stat-prefetch: off
>>>>>>>>> performance.io-cache: off
>>>>>>>>> performance.quick-read: off
>>>>>>>>> cluster.self-heal-window-size: 1024
>>>>>>>>> cluster.background-self-heal-count: 16
>>>>>>>>> nfs.enable-ino32: off
>>>>>>>>> nfs.addr-namelookup: off
>>>>>>>>> nfs.disable: on
>>>>>>>>> performance.read-ahead: off
>>>>>>>>> performance.readdir-ahead: on
>>>>>>>>> cluster.granular-entry-heal: on
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 29, 2016 at 2:20 PM, David
Gossage <
>>>>>>>>> dgossage at carouselchecks.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Mon, Aug 29, 2016 at 7:01 AM,
Anuradha Talur <
>>>>>>>>>> atalur at redhat.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> > From: "David
Gossage" <dgossage at carouselchecks.com>
>>>>>>>>>>> > To: "Anuradha Talur"
<atalur at redhat.com>
>>>>>>>>>>> > Cc: "gluster-users at
gluster.org List" <
>>>>>>>>>>> Gluster-users at gluster.org>,
"Krutika Dhananjay" <
>>>>>>>>>>> kdhananj at redhat.com>
>>>>>>>>>>> > Sent: Monday, August 29, 2016
5:12:42 PM
>>>>>>>>>>> > Subject: Re: [Gluster-users]
3.8.3 Shards Healing Glacier Slow
>>>>>>>>>>> >
>>>>>>>>>>> > On Mon, Aug 29, 2016 at 5:39
AM, Anuradha Talur <
>>>>>>>>>>> atalur at redhat.com> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > > Response inline.
>>>>>>>>>>> > >
>>>>>>>>>>> > > ----- Original Message
-----
>>>>>>>>>>> > > > From: "Krutika
Dhananjay" <kdhananj at redhat.com>
>>>>>>>>>>> > > > To: "David
Gossage" <dgossage at carouselchecks.com>
>>>>>>>>>>> > > > Cc:
"gluster-users at gluster.org List" <
>>>>>>>>>>> Gluster-users at gluster.org>
>>>>>>>>>>> > > > Sent: Monday, August
29, 2016 3:55:04 PM
>>>>>>>>>>> > > > Subject: Re:
[Gluster-users] 3.8.3 Shards Healing Glacier
>>>>>>>>>>> Slow
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > Could you attach
both client and brick logs? Meanwhile I
>>>>>>>>>>> will try these
>>>>>>>>>>> > > steps
>>>>>>>>>>> > > > out on my machines
and see if it is easily recreatable.
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > -Krutika
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > On Mon, Aug 29, 2016
at 2:31 PM, David Gossage <
>>>>>>>>>>> > > dgossage at
carouselchecks.com
>>>>>>>>>>> > > > > wrote:
>>>>>>>>>>> > > >
>>>>>>>>>>> > > >
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > Centos 7 Gluster
3.8.3
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > Brick1:
ccgl1.gl.local:/gluster1/BRICK1/1
>>>>>>>>>>> > > > Brick2:
ccgl2.gl.local:/gluster1/BRICK1/1
>>>>>>>>>>> > > > Brick3:
ccgl4.gl.local:/gluster1/BRICK1/1
>>>>>>>>>>> > > > Options
Reconfigured:
>>>>>>>>>>> > > >
cluster.data-self-heal-algorithm: full
>>>>>>>>>>> > > >
cluster.self-heal-daemon: on
>>>>>>>>>>> > > >
cluster.locking-scheme: granular
>>>>>>>>>>> > > >
features.shard-block-size: 64MB
>>>>>>>>>>> > > > features.shard: on
>>>>>>>>>>> > > >
performance.readdir-ahead: on
>>>>>>>>>>> > > > storage.owner-uid:
36
>>>>>>>>>>> > > > storage.owner-gid:
36
>>>>>>>>>>> > > >
performance.quick-read: off
>>>>>>>>>>> > > >
performance.read-ahead: off
>>>>>>>>>>> > > >
performance.io-cache: off
>>>>>>>>>>> > > >
performance.stat-prefetch: on
>>>>>>>>>>> > > > cluster.eager-lock:
enable
>>>>>>>>>>> > > > network.remote-dio:
enable
>>>>>>>>>>> > > > cluster.quorum-type:
auto
>>>>>>>>>>> > > >
cluster.server-quorum-type: server
>>>>>>>>>>> > > >
server.allow-insecure: on
>>>>>>>>>>> > > >
cluster.self-heal-window-size: 1024
>>>>>>>>>>> > > >
cluster.background-self-heal-count: 16
>>>>>>>>>>> > > >
performance.strict-write-ordering: off
>>>>>>>>>>> > > > nfs.disable: on
>>>>>>>>>>> > > > nfs.addr-namelookup:
off
>>>>>>>>>>> > > > nfs.enable-ino32:
off
>>>>>>>>>>> > > >
cluster.granular-entry-heal: on
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > Friday did rolling
upgrade from 3.8.3->3.8.3 no issues.
>>>>>>>>>>> > > > Following steps
detailed in previous recommendations began
>>>>>>>>>>> proces of
>>>>>>>>>>> > > > replacing and
healngbricks one node at a time.
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > 1) kill pid of brick
>>>>>>>>>>> > > > 2) reconfigure brick
from raid6 to raid10
>>>>>>>>>>> > > > 3) recreate
directory of brick
>>>>>>>>>>> > > > 4) gluster volume
start <> force
>>>>>>>>>>> > > > 5) gluster volume
heal <> full
>>>>>>>>>>> > > Hi,
>>>>>>>>>>> > >
>>>>>>>>>>> > > I'd suggest that full
heal is not used. There are a few bugs
>>>>>>>>>>> in full heal.
>>>>>>>>>>> > > Better safe than sorry ;)
>>>>>>>>>>> > > Instead I'd suggest
the following steps:
>>>>>>>>>>> > >
>>>>>>>>>>> > > Currently I brought the
node down by systemctl stop glusterd
>>>>>>>>>>> as I was
>>>>>>>>>>> > getting sporadic io issues and
a few VM's paused so hoping
>>>>>>>>>>> that will help.
>>>>>>>>>>> > I may wait to do this till
around 4PM when most work is done
>>>>>>>>>>> in case it
>>>>>>>>>>> > shoots load up.
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > > 1) kill pid of brick
>>>>>>>>>>> > > 2) to configuring of
brick that you need
>>>>>>>>>>> > > 3) recreate brick dir
>>>>>>>>>>> > > 4) while the brick is
still down, from the mount point:
>>>>>>>>>>> > > a) create a dummy non
existent dir under / of mount.
>>>>>>>>>>> > >
>>>>>>>>>>> >
>>>>>>>>>>> > so if noee 2 is down brick,
pick node for example 3 and make a
>>>>>>>>>>> test dir
>>>>>>>>>>> > under its brick directory that
doesnt exist on 2 or should I
>>>>>>>>>>> be dong this
>>>>>>>>>>> > over a gluster mount?
>>>>>>>>>>> You should be doing this over
gluster mount.
>>>>>>>>>>> >
>>>>>>>>>>> > > b) set a non existent
extended attribute on / of mount.
>>>>>>>>>>> > >
>>>>>>>>>>> >
>>>>>>>>>>> > Could you give me an example
of an attribute to set? I've
>>>>>>>>>>> read a tad on
>>>>>>>>>>> > this, and looked up attributes
but haven't set any yet myself.
>>>>>>>>>>> >
>>>>>>>>>>> Sure. setfattr -n
"user.some-name" -v "some-value"
>>>>>>>>>>> <path-to-mount>
>>>>>>>>>>> > Doing these steps will ensure
that heal happens only from
>>>>>>>>>>> updated brick to
>>>>>>>>>>> > > down brick.
>>>>>>>>>>> > > 5) gluster v start
<> force
>>>>>>>>>>> > > 6) gluster v heal
<>
>>>>>>>>>>> > >
>>>>>>>>>>> >
>>>>>>>>>>> > Will it matter if somewhere in
gluster the full heal command
>>>>>>>>>>> was run other
>>>>>>>>>>> > day? Not sure if it
eventually stops or times out.
>>>>>>>>>>> >
>>>>>>>>>>> full heal will stop once the crawl
is done. So if you want to
>>>>>>>>>>> trigger heal again,
>>>>>>>>>>> run gluster v heal <>.
Actually even brick up or volume start
>>>>>>>>>>> force should
>>>>>>>>>>> trigger the heal.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Did this on test bed today. its one
server with 3 bricks on same
>>>>>>>>>> machine so take that for what its
worth. also it still runs 3.8.2. Maybe
>>>>>>>>>> ill update and re-run test.
>>>>>>>>>>
>>>>>>>>>> killed brick
>>>>>>>>>> deleted brick dir
>>>>>>>>>> recreated brick dir
>>>>>>>>>> created fake dir on gluster mount
>>>>>>>>>> set suggested fake attribute on it
>>>>>>>>>> ran volume start <> force
>>>>>>>>>>
>>>>>>>>>> looked at files it said needed healing
and it was just 8 shards
>>>>>>>>>> that were modified for few minutes I
ran through steps
>>>>>>>>>>
>>>>>>>>>> gave it few minutes and it stayed same
>>>>>>>>>> ran gluster volume <> heal
>>>>>>>>>>
>>>>>>>>>> it healed all the directories and files
you can see over mount
>>>>>>>>>> including fakedir.
>>>>>>>>>>
>>>>>>>>>> same issue for shards though. it adds
more shards to heal at
>>>>>>>>>> glacier pace. slight jump in speed if
I stat every file and dir in VM
>>>>>>>>>> running but not all shards.
>>>>>>>>>>
>>>>>>>>>> It started with 8 shards to heal and is
now only at 33 out of 800
>>>>>>>>>> and probably wont finish adding for few
days at rate it goes.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> > >
>>>>>>>>>>> > > > 1st node worked as
expected took 12 hours to heal 1TB
>>>>>>>>>>> data. Load was
>>>>>>>>>>> > > little
>>>>>>>>>>> > > > heavy but nothing
shocking.
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > About an hour after
node 1 finished I began same process
>>>>>>>>>>> on node2. Heal
>>>>>>>>>>> > > > proces kicked in as
before and the files in directories
>>>>>>>>>>> visible from
>>>>>>>>>>> > > mount
>>>>>>>>>>> > > > and .glusterfs
healed in short time. Then it began crawl
>>>>>>>>>>> of .shard adding
>>>>>>>>>>> > > > those files to heal
count at which point the entire proces
>>>>>>>>>>> ground to a
>>>>>>>>>>> > > halt
>>>>>>>>>>> > > > basically. After 48
hours out of 19k shards it has added
>>>>>>>>>>> 5900 to heal
>>>>>>>>>>> > > list.
>>>>>>>>>>> > > > Load on all 3
machnes is negligible. It was suggested to
>>>>>>>>>>> change this
>>>>>>>>>>> > > value
>>>>>>>>>>> > > > to full
cluster.data-self-heal-algorithm and restart
>>>>>>>>>>> volume which I
>>>>>>>>>>> > > did. No
>>>>>>>>>>> > > > efffect. Tried
relaunching heal no effect, despite any
>>>>>>>>>>> node picked. I
>>>>>>>>>>> > > > started each VM and
performed a stat of all files from
>>>>>>>>>>> within it, or a
>>>>>>>>>>> > > full
>>>>>>>>>>> > > > virus scan and that
seemed to cause short small spikes in
>>>>>>>>>>> shards added,
>>>>>>>>>>> > > but
>>>>>>>>>>> > > > not by much. Logs
are showing no real messages indicating
>>>>>>>>>>> anything is
>>>>>>>>>>> > > going
>>>>>>>>>>> > > > on. I get hits to
brick log on occasion of null lookups
>>>>>>>>>>> making me think
>>>>>>>>>>> > > its
>>>>>>>>>>> > > > not really crawling
shards directory but waiting for a
>>>>>>>>>>> shard lookup to
>>>>>>>>>>> > > add
>>>>>>>>>>> > > > it. I'll get
following in brick log but not constant and
>>>>>>>>>>> sometime
>>>>>>>>>>> > > multiple
>>>>>>>>>>> > > > for same shard.
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > [2016-08-29
08:31:57.478125] W [MSGID: 115009]
>>>>>>>>>>> > > >
[server-resolve.c:569:server_resolve] 0-GLUSTER1-server:
>>>>>>>>>>> no resolution
>>>>>>>>>>> > > type
>>>>>>>>>>> > > > for (null) (LOOKUP)
>>>>>>>>>>> > > > [2016-08-29
08:31:57.478170] E [MSGID: 115050]
>>>>>>>>>>> > > >
[server-rpc-fops.c:156:server_lookup_cbk]
>>>>>>>>>>> 0-GLUSTER1-server: 12591783:
>>>>>>>>>>> > > > LOOKUP (null)
(00000000-0000-0000-00
>>>>>>>>>>> > > >
00-000000000000/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.221)
>>>>>>>>>>> ==> (Invalid
>>>>>>>>>>> > > > argument) [Invalid
argument]
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > This one repeated
about 30 times in row then nothing for
>>>>>>>>>>> 10 minutes then
>>>>>>>>>>> > > one
>>>>>>>>>>> > > > hit for one
different shard by itself.
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > How can I determine
if Heal is actually running? How can I
>>>>>>>>>>> kill it or
>>>>>>>>>>> > > force
>>>>>>>>>>> > > > restart? Does node I
start it from determine which
>>>>>>>>>>> directory gets
>>>>>>>>>>> > > crawled to
>>>>>>>>>>> > > > determine heals?
>>>>>>>>>>> > > >
>>>>>>>>>>> > > > David Gossage
>>>>>>>>>>> > > > Carousel Checks Inc.
| System Administrator
>>>>>>>>>>> > > > Office 708.613.2284
>>>>>>>>>>> > > >
>>>>>>>>>>> > > >
_______________________________________________
>>>>>>>>>>> > > > Gluster-users
mailing list
>>>>>>>>>>> > > > Gluster-users at
gluster.org
>>>>>>>>>>> > > >
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>> > > >
>>>>>>>>>>> > > >
>>>>>>>>>>> > > >
_______________________________________________
>>>>>>>>>>> > > > Gluster-users
mailing list
>>>>>>>>>>> > > > Gluster-users at
gluster.org
>>>>>>>>>>> > > >
http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>> > >
>>>>>>>>>>> > > --
>>>>>>>>>>> > > Thanks,
>>>>>>>>>>> > > Anuradha.
>>>>>>>>>>> > >
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Anuradha.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160830/663dac03/attachment.html>