Pranith Kumar Karampuri
2016-Mar-26 13:58 UTC
[Gluster-users] Very poor heal behaviour in 3.7.9
On 03/26/2016 07:20 PM, Lindsay Mathieson wrote:> On 26/03/2016 11:32 PM, Pranith Kumar Karampuri wrote: >> Yes, this is a bug we are addressing for 3.7.10. The patch is already >> merged. http://review.gluster.org/13564 > > > Excellent, thanks Pranith. > > Is that the same issue I posted earlier re "gluster volume heal info" > appearing to block I/O? >I don't think it is heal info that is blocking I/O. I think it is client triggering heal and block the fop until heal completes that results in this pattern. This data-heal disabling should get you out of this problem. Pranith
Kayra Otaner | BilgiO
2016-Mar-26 14:06 UTC
[Gluster-users] Very poor heal behaviour in 3.7.9
I will test this patch for the issue we've posted couple days ago. Our problem showed itself during self heal and autoheal operations especially when replicating huge number of files sitting in one directory (300K+ files) On Sat, Mar 26, 2016, 15:58 Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:> > > On 03/26/2016 07:20 PM, Lindsay Mathieson wrote: > > On 26/03/2016 11:32 PM, Pranith Kumar Karampuri wrote: > >> Yes, this is a bug we are addressing for 3.7.10. The patch is already > >> merged. http://review.gluster.org/13564 > > > > > > Excellent, thanks Pranith. > > > > Is that the same issue I posted earlier re "gluster volume heal info" > > appearing to block I/O? > > > I don't think it is heal info that is blocking I/O. I think it is client > triggering heal and block the fop until heal completes that results in > this pattern. This data-heal disabling should get you out of this problem. > > Pranith > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160326/974ab770/attachment.html>
On 26/03/2016 11:58 PM, Pranith Kumar Karampuri wrote:>> Is that the same issue I posted earlier re "gluster volume heal info" >> appearing to block I/O? >> > I don't think it is heal info that is blocking I/O. I think it is > client triggering heal and block the fop until heal completes that > results in this pattern. This data-heal disabling should get you out > of this problem.I tried it earlier and it didn't seem to help. Does anything need to be restarted after cluster.data-self-heal is set off? -- Lindsay Mathieson
On 26/03/2016 11:58 PM, Pranith Kumar Karampuri wrote:> I don't think it is heal info that is blocking I/O. I think it is > client triggering heal and block the fop until heal completes that > results in this pattern. This data-heal disabling should get you out > of this problem.I'm not sure this is the case. Disabling data-heal didn't help. And I wasn't observing slow i/o, it was blocked altogether for over an hour. While this was happening cpu and iowait were under 5%. The moment I cancelled the "gluster volume heal <ds> info" call i/o resumed. Unfortunately I can't resume testing right now as the cluster is being backed up, takes around 18 hours. -- Lindsay Mathieson