Gluster users - Oct 2015 - Replica 3 cluster, file being healed on all 3 nodes

Have been testing failure mode by killing the gluster processes on a node
(killall glusterd glusterfsd).

Pleasantly surprised how much smoother gluster has got at this since 3.5,
heal takes a while but it no longer kills cluster performance. Probably
helps that I'm using replica 3 now rather than replica 2.

However I managed to create a state where a file was being healed on all
three nodes (probably y live migrating a VM while it was being healed). I
didn't think that was possible without creating a split brain problem, but
it eventually got all the way to being healed.


Is this the diff algorithm in play? is it comparing blocks on all three
nodes using the ones that are identical across two or three nodes? If it
encountered a block that was different on all three ndes would that result
in split brain?



-- 
Lindsay
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20151012/b6958831/attachment.html>

----- Original Message -----
> From: "Lindsay Mathieson" <lindsay.mathieson at gmail.com>
> To: "gluster-users" <gluster-users at gluster.org>
> Sent: Monday, October 12, 2015 6:04:01 PM
> Subject: [Gluster-users] Replica 3 cluster, file being healed on all 3
nodes
> Have been testing failure mode by killing the gluster processes on a node
> (killall glusterd glusterfsd).
> Pleasantly surprised how much smoother gluster has got at this since 3.5,
> heal takes a while but it no longer kills cluster performance. Probably
> helps that I'm using replica 3 now rather than replica 2.
> However I managed to create a state where a file was being healed on all
> three nodes (probably y live migrating a VM while it was being healed). I
> didn't think that was possible without creating a split brain problem,
but
> it eventually got all the way to being healed.
I don't think it is possible for heal of this image to be happening on all
three nodes.
> Is this the diff algorithm in play? is it comparing blocks on all three
nodes
> using the ones that are identical across two or three nodes? If it
> encountered a block that was different on all three ndes would that result
> in split brain?
Yes, diff heal performs rolling checksum of every block at same offset on all
three copies of the file to see if that block needs heal.

As far as split-brain is concerned, self-heal daemon would not even attempt to
heal the file if it detects that the file is in split-brain (due to the absence
of at least one good copy).
It is possible for the three copies of the file to have different checksums
(when client quorum is disabled) but one of them MUST be a good copy for heal to
happen.

-Krutika 
> --
> Lindsay
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20151013/bc4844ee/attachment.html>