On 18/05/2016 11:41 PM, Krutika Dhananjay wrote:> I will try to recreate this issue tomorrow on my machines with the > steps that Lindsay provided in this thread. I will let you know the > result soon after that.Thanks Krutika, I've been trying to get the shard stats you wanted, but by the time the heal info completed, the shard in question have been healed ... The node in question is the last node on the list :) I'll swap them around and try tomorrow. One thought - since the VM's are active while the brick is removed/re-added, could it be the shards that are written while the brick is added that are the reverse healing shards? -- Lindsay Mathieson -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160519/1e798e94/attachment.html>
On 19/05/2016 12:17 AM, Lindsay Mathieson wrote:> One thought - since the VM's are active while the brick is > removed/re-added, could it be the shards that are written while the > brick is added that are the reverse healing shards?I tested by: - removing brick 3 - erasing brick 3 - closing down all VM's - adding new brick 3 - waiting until heal number reached its max and started decreasing There were no reverse heals - Started the VM's backup. No real issues there though one showed IO errors, presumably due to shards being locked as they were healed. - VM's started ok, no reverse heals were noted and eventually Brick 3 was fully healed. The VM's do not appear to be corrupted. So it would appear the problem is adding a brick while the volume is being written to. Cheers, -- Lindsay Mathieson -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160519/0bacaaf1/attachment.html>