similar to: Peer isolation while healing

Hi, There is no way to isolate the healing peer. Healing happens from the good brick to the bad brick. I guess your replica bricks are on a different peers. If you try to isolate the healing peer, it will stop the healing process itself. What is the error you are getting while writing? It would be helpful to debug the issue, if you can provide us the output of the following commands: gluster

That make sense ^_^ Unfortunately I haven't kept the interresting data you need. Basically I had some write errors on my gluster clients when my monitoring tool tested mkdir & create files. The server's load was huge during the healing (cpu at 100%), and the disk latency increased a lot. That may be the source of my write errors, we'll know for sure next time... I'll keep

On Mon, Oct 09, 2017 at 03:29:41PM +0200, ML wrote: > The server's load was huge during the healing (cpu at 100%), and the > disk latency increased a lot. Depending on the file sizes, you might want to consider changing the heal algortithm. Might be better to just re-download the whole file / shard than to try and heal it, assuming you don't have big files. That would free up the

I'm trying to manually garbage data on bricks (when the volume is stopped) and then check whether healing is possible. For example: Start: # glusterd --debug Bricks (on EXT4 mounted with 'rw,realtime'): # mkdir /root/data0 # mkdir /root/data1 # mkdir /root/data2 Volume: # gluster volume create gv0 replica 3 [local-ip]:/root/data0 [local-ip]:/root/data1 [local-ip]:/root/data2

Hi Richard, Thanks for the informations. As you said there is gfid mismatch for the file. On brick-1 & brick-2 the gfids are same & on brick-3 the gfid is different. This is not considered as split-brain because we have two good copies here. Gluster 3.10 does not have a method to resolve this situation other than the manual intervention [1]. Basically what you need to do is remove the

Hi folks, I'm troubled moving an arbiter brick to another server because of I/O load issues. My setup is as follows: # gluster volume info Volume Name: myvol Type: Distributed-Replicate Volume ID: 43ba517a-ac09-461e-99da-a197759a7dc8 Status: Started Snapshot Count: 0 Number of Bricks: 3 x (2 + 1) = 9 Transport-type: tcp Bricks: Brick1: gv0:/data/glusterfs Brick2: gv1:/data/glusterfs Brick3:

Hi Karthik, Thank you for your reply. The heal is still undergoing, as the /var/log/glusterfs/glustershd.log keeps growing, and there's a lot of pending entries in the heal info. The gluster version is 3.10.9 and 3.10.10 (the version update in progress). It doesn't have info summary [yet?], and the heal info is way too long to attach here. (It takes more than 20 minutes just to collect

Hi Karthik, Thank you very much, you made me much more relaxed. Below is getfattr output for a file from all the bricks: root at gv2 ~ # getfattr -d -e hex -m . /data/glusterfs/testset/306/30677af808ad578916f54783904e6342.pack getfattr: Removing leading '/' from absolute path names # file: data/glusterfs/testset/306/30677af808ad578916f54783904e6342.pack

Hi, I am wondering why the other brick is not showing any entry in split brain in the heal info split-brain output. Can you give the output of stat & getfattr -d -m . -e hex <file-path-on-brick> from both the bricks. Regards, Karthik On Mon, Feb 5, 2018 at 5:03 PM, Alex K <rightkicktech at gmail.com> wrote: > After stoping/starting the volume I have: > > gluster volume

Hey Matt, >From the xattr output, it looks like the files are not present on the arbiter brick & needs healing. But on the parent it does not have the pending markers set for those entries. The workaround for this is you need to do a lookup on the file which needs heal from the mount, so it will create the entry on the arbiter brick and then run the volume heal to do the healing. Follow

Hey, Did the heal completed and you still have some entries pending heal? If yes then can you provide the following informations to debug the issue. 1. Which version of gluster you are running 2. gluster volume heal <volname> info summary or gluster volume heal <volname> info 3. getfattr -d -e hex -m . <filepath-on-brick> output of any one of the which is pending heal from all

Hi Karthik, I tried to delete one file at one node and that is probably the reason. After several deletes seems that I deleted some files that shouldn't and the ovirt engine hosted on this volume was not able to start. Now I am setting up the engine from scratch... In case I see this kind of split brain again I will get back before I start deleting :) Alex On Mon, Feb 5, 2018 at 2:34 PM,

Hi Matt, Run these commands on all the bricks of the replica pair to get the attrs set on the backend. On the bricks of first replica set: getfattr -d -e hex -m . <brick path>/.glusterfs/10/86/ 108694db-c039-4b7c-bd3d-ad6a15d811a2 On the fourth replica set: getfattr -d -e hex -m . <brick path>/.glusterfs/ e0/c5/e0c56bf7-8bfe-46ca-bde1-e46b92d33df3 Also run the "gluster volume

Attached is the heal log for the volume as well as the shd log. >> Run these commands on all the bricks of the replica pair to get the attrs set on the backend. [root at tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m . /exp/b1/gv0/.glusterfs/10/86/108694db-c039-4b7c-bd3d-ad6a15d811a2 getfattr: Removing leading '/' from absolute path names # file:

On Wed, Mar 14, 2018 at 3:36 PM, Anatoliy Dmytriyev <tolid at tolid.eu.org> wrote: > Hi Karthik, > > > Thanks a lot for the explanation. > > Does it mean a distributed volume health can be checked only by "gluster > volume status " command? > Yes. I am not aware of any other command which can give the status of plain distribute volume which is similar to

Hi Anatoliy, The heal command is basically used to heal any mismatching contents between replica copies of the files. For the command "gluster volume heal <volname>" to succeed, you should have the self-heal-daemon running, which is true only if your volume is of type replicate/disperse. In your case you have a plain distribute volume where you do not store the replica of any

On Wed, Mar 14, 2018 at 5:42 PM, Karthik Subrahmanya <ksubrahm at redhat.com> wrote: > > > On Wed, Mar 14, 2018 at 3:36 PM, Anatoliy Dmytriyev <tolid at tolid.eu.org> > wrote: > >> Hi Karthik, >> >> >> Thanks a lot for the explanation. >> >> Does it mean a distributed volume health can be checked only by "gluster >> volume

Hi Rolf, answers follow inline... On Thu, Nov 9, 2017 at 3:20 PM, Rolf Larsen <rolf at jotta.no> wrote: > Hi, > > We ran a test on GlusterFS 3.12.1 with erasurecoded volumes 8+2 with 10 > bricks (default config,tested with 100gb, 200gb, 400gb bricksizes,10gbit > nics) > > 1. > Tests show that healing takes about double the time on healing 200gb vs > 100, and

Someone on the #gluster-users irc channel said the following : "Decreasing features.locks-revocation-max-blocked to an absurdly low number is letting our distributed-disperse set heal again." Is this something to concider? Does anyone else have experience with tweaking this to speed up healing? Sent from my iPhone > On 9 Nov 2017, at 18:00, Serkan ?oban <cobanserkan at

Hi Karthik, Thanks a lot for the explanation. Does it mean a distributed volume health can be checked only by "gluster volume status " command? And one more question: cluster.min-free-disk is 10% by default. What kind of "side effects" can we face if this option will be reduced to, for example, 5%? Could you point to any best practice document(s)? Regards, Anatoliy