>Good news is we already have a WIP patch review.glusterd.org/10851 tointroduce multi threaded shd. Credits to Richard/Shreyas from facebook for this. IIRC, we also have a BZ for the same>Isnt it the same bugzilla ( https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the commit log? --Humble On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee <atin.mukherjee83 at gmail.com> wrote:> -Atin > Sent from one plus one > On Oct 13, 2015 3:16 AM, "Ben Turner" <bturner at redhat.com> wrote: > > > > ----- Original Message ----- > > > From: "Lindsay Mathieson" <lindsay.mathieson at gmail.com> > > > To: "gluster-users" <gluster-users at gluster.org> > > > Sent: Friday, October 9, 2015 9:18:11 AM > > > Subject: [Gluster-users] Speed up heal performance > > > > > > Is there any way to max out heal performance? My cluster is unused > overnight, > > > and lightly used at lunchtimes, it would be handy to speed up a heal. > > > > > > The only tuneable I found was cluster.self-heal-window-size, which > doesn't > > > seem to make much difference. > > > > I don't know of any way to speed this up, maybe someone else could chime > in here that knows the heal daemon better than me. Maybe you could open an > RFE on this? In my testing I only see 2 files getting healed at a time per > replica pair. I would like to see this be multi threaded(if its not > already) with the ability to tune it to control resource usage(similar to > what we did in the rebalance refactoring done recently). If you let me > know the BZ # I'll add my data + suggestions, I have been testing this > pretty extensively in recent weeks and good data + some ideas on how to > speed things up. > Good news is we already have a WIP patch review.glusterd.org/10851 to > introduce multi threaded shd. Credits to Richard/Shreyas from facebook for > this. IIRC, we also have a BZ for the same but the patch is in rfc as of > now. AFAIK, this is a candidate to land in 3.8 as well, Vijay can correct > me otherwise. > > > > -b > > > > > thanks, > > > -- > > > Lindsay > > > > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org > > > http://www.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151013/964fd88b/attachment.html>
On 10/13/2015 03:44 PM, Humble Devassy Chirammal wrote:>>Good news is we already have a WIP patch review.glusterd.org/10851 > <http://review.glusterd.org/10851> to introduce multi threaded shd. > Credits to Richard/Shreyas from facebook for this. IIRC, we also have a > BZ for the same >> > Isnt it the same bugzilla > (https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the > commit log?It is, just that I was too lazy to scroll down for the bug as topic would have been also ideally be the same (bug-XXXXXXX)> > --Humble > > > On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee > <atin.mukherjee83 at gmail.com <mailto:atin.mukherjee83 at gmail.com>> wrote: > > -Atin > Sent from one plus one > On Oct 13, 2015 3:16 AM, "Ben Turner" <bturner at redhat.com > <mailto:bturner at redhat.com>> wrote: > > > > ----- Original Message ----- > > > From: "Lindsay Mathieson" <lindsay.mathieson at gmail.com <mailto:lindsay.mathieson at gmail.com>> > > > To: "gluster-users" <gluster-users at gluster.org <mailto:gluster-users at gluster.org>> > > > Sent: Friday, October 9, 2015 9:18:11 AM > > > Subject: [Gluster-users] Speed up heal performance > > > > > > Is there any way to max out heal performance? My cluster is unused overnight, > > > and lightly used at lunchtimes, it would be handy to speed up a heal. > > > > > > The only tuneable I found was cluster.self-heal-window-size, which doesn't > > > seem to make much difference. > > > > I don't know of any way to speed this up, maybe someone else could chime in here that knows the heal daemon better than me. Maybe you could open an RFE on this? In my testing I only see 2 files getting healed at a time per replica pair. I would like to see this be multi threaded(if its not already) with the ability to tune it to control resource usage(similar to what we did in the rebalance refactoring done recently). If you let me know the BZ # I'll add my data + suggestions, I have been testing this pretty extensively in recent weeks and good data + some ideas on how to speed things up. > Good news is we already have a WIP patch review.glusterd.org/10851 > <http://review.glusterd.org/10851> to introduce multi threaded shd. > Credits to Richard/Shreyas from facebook for this. IIRC, we also > have a BZ for the same but the patch is in rfc as of now. AFAIK, > this is a candidate to land in 3.8 as well, Vijay can correct me > otherwise. > > > > -b > > > > > thanks, > > > -- > > > Lindsay > > > > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > > > http://www.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > > http://www.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >
On 13 October 2015 at 20:14, Humble Devassy Chirammal < humble.devassy at gmail.com> wrote:> >Good news is we already have a WIP patch review.glusterd.org/10851 to > introduce multi threaded shd. Credits to Richard/Shreyas from facebook for > this. IIRC, we also have a BZ for the same > > > Isnt it the same bugzilla ( > https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the > commit log?I'm confused, the bug report states that its already in 3.6? what settings would control the self heal threads? -- Lindsay -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151013/c3e1d294/attachment.html>
----- Original Message -----> From: "Humble Devassy Chirammal" <humble.devassy at gmail.com> > To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com> > Cc: "Ben Turner" <bturner at redhat.com>, "gluster-users" <gluster-users at gluster.org> > Sent: Tuesday, October 13, 2015 6:14:46 AM > Subject: Re: [Gluster-users] Speed up heal performance > > >Good news is we already have a WIP patch review.glusterd.org/10851 to > introduce multi threaded shd. Credits to Richard/Shreyas from facebook for > this. IIRC, we also have a BZ for the same > > > Isnt it the same bugzilla ( > https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the > commit log?@Lindsay - No need for a BZ, the above BZ should suffice. @Anyone - In the commit I see: { .key = "cluster.shd-max-threads", .voltype = "cluster/replicate", .option = "shd-max-threads", .op_version = 1, .flags = OPT_FLAG_CLIENT_OPT }, { .key = "cluster.shd-thread-batch-size", .voltype = "cluster/replicate", .option = "shd-thread-batch-size", .op_version = 1, .flags = OPT_FLAG_CLIENT_OPT }, So we can tune max threads and thread batch size? I understand max threads, but what is batch size? In my testing on 10G NICs with a backend that will service 10G throughput I see about 1.5 GB per minute of SH throughput. To Lindsay's other point, will this patch improve SH throughput? My systems can write at 1.5 GB / Sec and NICs can to 1.2 GB / sec but I only see ~1.5 GB per _minute_ of SH throughput. If we can not only make SH multi threaded, but improve the performance of a single thread that would be awesome. Super bonus points if we can have some sort of tunible that can limit the bandwidth each thread can consume. It would be great to be able to crank things up when the systems aren't busy and slow things down when load increases. -b> > --Humble > > > On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee <atin.mukherjee83 at gmail.com> > wrote: > > > -Atin > > Sent from one plus one > > On Oct 13, 2015 3:16 AM, "Ben Turner" <bturner at redhat.com> wrote: > > > > > > ----- Original Message ----- > > > > From: "Lindsay Mathieson" <lindsay.mathieson at gmail.com> > > > > To: "gluster-users" <gluster-users at gluster.org> > > > > Sent: Friday, October 9, 2015 9:18:11 AM > > > > Subject: [Gluster-users] Speed up heal performance > > > > > > > > Is there any way to max out heal performance? My cluster is unused > > overnight, > > > > and lightly used at lunchtimes, it would be handy to speed up a heal. > > > > > > > > The only tuneable I found was cluster.self-heal-window-size, which > > doesn't > > > > seem to make much difference. > > > > > > I don't know of any way to speed this up, maybe someone else could chime > > in here that knows the heal daemon better than me. Maybe you could open an > > RFE on this? In my testing I only see 2 files getting healed at a time per > > replica pair. I would like to see this be multi threaded(if its not > > already) with the ability to tune it to control resource usage(similar to > > what we did in the rebalance refactoring done recently). If you let me > > know the BZ # I'll add my data + suggestions, I have been testing this > > pretty extensively in recent weeks and good data + some ideas on how to > > speed things up. > > Good news is we already have a WIP patch review.glusterd.org/10851 to > > introduce multi threaded shd. Credits to Richard/Shreyas from facebook for > > this. IIRC, we also have a BZ for the same but the patch is in rfc as of > > now. AFAIK, this is a candidate to land in 3.8 as well, Vijay can correct > > me otherwise. > > > > > > -b > > > > > > > thanks, > > > > -- > > > > Lindsay > > > > > > > > _______________________________________________ > > > > Gluster-users mailing list > > > > Gluster-users at gluster.org > > > > http://www.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org > > > http://www.gluster.org/mailman/listinfo/gluster-users > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-users > > >