[Gluster-users] Speed up heal performance

Ben Turner
2015-10-14T15:38:43Z

----- Original Message -----
> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> To: "Ben Turner" <bturner at redhat.com>, "Humble
Devassy Chirammal" <humble.devassy at gmail.com>, "Atin
Mukherjee"
> <atin.mukherjee83 at gmail.com>
> Cc: "gluster-users" <gluster-users at gluster.org>
> Sent: Wednesday, October 14, 2015 1:39:14 AM
> Subject: Re: [Gluster-users] Speed up heal performance
>
>
>
> On 10/13/2015 07:11 PM, Ben Turner wrote:
> > ----- Original Message -----
> >> From: "Humble Devassy Chirammal" <humble.devassy at
gmail.com>
> >> To: "Atin Mukherjee" <atin.mukherjee83 at
gmail.com>
> >> Cc: "Ben Turner" <bturner at redhat.com>,
"gluster-users"
> >> <gluster-users at gluster.org>
> >> Sent: Tuesday, October 13, 2015 6:14:46 AM
> >> Subject: Re: [Gluster-users] Speed up heal performance
> >>
> >>> Good news is we already have a WIP patch
review.glusterd.org/10851 to
> >> introduce multi threaded shd. Credits to Richard/Shreyas from
facebook for
> >> this. IIRC, we also have a BZ for the same
> >> Isnt it the same bugzilla (
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in
the
> >> commit log?
> > @Lindsay - No need for a BZ, the above BZ should suffice.
> >
> > @Anyone - In the commit I see:
> >
> >          { .key        = "cluster.shd-max-threads",
> >            .voltype    = "cluster/replicate",
> >            .option     = "shd-max-threads",
> >            .op_version = 1,
> >            .flags      = OPT_FLAG_CLIENT_OPT
> >          },
> >          { .key        = "cluster.shd-thread-batch-size",
> >            .voltype    = "cluster/replicate",
> >            .option     = "shd-thread-batch-size",
> >            .op_version = 1,
> >            .flags      = OPT_FLAG_CLIENT_OPT
> >          },
> >
> > So we can tune max threads and thread batch size?  I understand max
> > threads, but what is batch size?  In my testing on 10G NICs with a
backend
> > that will service 10G throughput I see about 1.5 GB per minute of SH
> > throughput.  To Lindsay's other point, will this patch improve SH
> > throughput?  My systems can write at 1.5 GB / Sec and NICs can to 1.2
GB /
> > sec but I only see ~1.5 GB per _minute_ of SH throughput.  If we can
not
> > only make SH multi threaded, but improve the performance of a single
> > thread that would be awesome.  Super bonus points if we can have some
sort
> > of tunible that can limit the bandwidth each thread can consume.  It
would
> > be great to be able to crank things up when the systems aren't
busy and
> > slow things down when load increases.
> This patch is not merged because I thought we needed throttling feature
> to go in before we can merge this for better control of the self-heal
> speed. We are doing that for 3.8. So expect to see both of these for 3.8.

Great news!  You da man Pranith, next time I am on your side of the world beers
are on me :)

-b

>
> Pranith
> >
> > -b
> >
> >
> >> --Humble
> >>
> >>
> >> On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee
> >> <atin.mukherjee83 at gmail.com>
> >> wrote:
> >>
> >>> -Atin
> >>> Sent from one plus one
> >>> On Oct 13, 2015 3:16 AM, "Ben Turner" <bturner at
redhat.com> wrote:
> >>>> ----- Original Message -----
> >>>>> From: "Lindsay Mathieson"
<lindsay.mathieson at gmail.com>
> >>>>> To: "gluster-users" <gluster-users at
gluster.org>
> >>>>> Sent: Friday, October 9, 2015 9:18:11 AM
> >>>>> Subject: [Gluster-users] Speed up heal performance
> >>>>>
> >>>>> Is there any way to max out heal performance? My
cluster is unused
> >>> overnight,
> >>>>> and lightly used at lunchtimes, it would be handy to
speed up a heal.
> >>>>>
> >>>>> The only tuneable I found was
cluster.self-heal-window-size, which
> >>> doesn't
> >>>>> seem to make much difference.
> >>>> I don't know of any way to speed this up, maybe
someone else could chime
> >>> in here that knows the heal daemon better than me.  Maybe you
could open
> >>> an
> >>> RFE on this?  In my testing I only see 2 files getting healed
at a time
> >>> per
> >>> replica pair.  I would like to see this be multi threaded(if
its not
> >>> already) with the ability to tune it to control resource
usage(similar to
> >>> what we did in the rebalance refactoring done recently).  If
you let me
> >>> know the BZ # I'll add my data + suggestions, I have been
testing this
> >>> pretty extensively in recent weeks and good data + some ideas
on how to
> >>> speed things up.
> >>> Good news is we already have a WIP patch
review.glusterd.org/10851 to
> >>> introduce multi threaded shd. Credits to Richard/Shreyas from
facebook
> >>> for
> >>> this. IIRC, we also have a BZ for the same but the patch is in
rfc as of
> >>> now. AFAIK, this is a candidate to land in 3.8 as well, Vijay
can correct
> >>> me otherwise.
> >>>> -b
> >>>>
> >>>>> thanks,
> >>>>> --
> >>>>> Lindsay
> >>>>>
> >>>>> _______________________________________________
> >>>>> Gluster-users mailing list
> >>>>> Gluster-users at gluster.org
> >>>>> http://www.gluster.org/mailman/listinfo/gluster-users
> >>>> _______________________________________________
> >>>> Gluster-users mailing list
> >>>> Gluster-users at gluster.org
> >>>> http://www.gluster.org/mailman/listinfo/gluster-users
> >>> _______________________________________________
> >>> Gluster-users mailing list
> >>> Gluster-users at gluster.org
> >>> http://www.gluster.org/mailman/listinfo/gluster-users
> >>>
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
>
>