Gluster users - Aug 2016 - canceling full heal 3.8

On Sat, Aug 27, 2016 at 5:35 PM, David Gossage <dgossage at
carouselchecks.com>
wrote:
> On Aug 27, 2016 4:37 PM, "Lindsay Mathieson"
<lindsay.mathieson at gmail.com>
> wrote:
> >
> > On 28/08/2016 6:07 AM, David Gossage wrote:
> >>
> >> 7 hours after starting full heal shards still haven't started
healing,
> and count from heal statistics heal-count has only reached 1800 out of
> 19000 shards.  shards dir hasn't even been recreated yet.  Creation of
the
> non sharded stubs (do they have a more official term?) in the visible mount
> point was as speedy as expected.  shards are painfully slow.
> >
> >
> >
> > Is your CPU usage through the roof?
>
> Currently it has almost no activity.  First node yesterday got a bit
> high.  But 2nd node today that has issues is pretty low.
> >
> > If you haven't already, I'd suggest
> >
> > - changing "cluster.data-self-heal-algorithm" to
"full"
> >
> > - And restarting the gluster volume if possible
> >
>
> I'll shut down vm's later tonight and see if that helps at all.
>
applied  "cluster.data-self-heal-algorithm" to "full"

stopped volume - started volume

cpu activity barely noticeable. heal count crawling at 1 new addition to
list every minute or 2 and stil just building a list it hasnt started
making a .shard directory
> >
> > I have a suspicion something changed recently with heal, I've
noticed
> that it takes a long time (hours) to kick in when the diff algorithm is
> used. I don't recall it doing this with 3.7.11
> >
> >
> > --
> > Lindsay Mathieson
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160827/c9102c89/attachment.html>

On Sat, Aug 27, 2016 at 9:55 PM, David Gossage <dgossage at
carouselchecks.com>
wrote:
> On Sat, Aug 27, 2016 at 5:35 PM, David Gossage <
> dgossage at carouselchecks.com> wrote:
>
>> On Aug 27, 2016 4:37 PM, "Lindsay Mathieson"
<lindsay.mathieson at gmail.com>
>> wrote:
>> >
>> > On 28/08/2016 6:07 AM, David Gossage wrote:
>> >>
>> >> 7 hours after starting full heal shards still haven't
started healing,
>> and count from heal statistics heal-count has only reached 1800 out of
>> 19000 shards.  shards dir hasn't even been recreated yet.  Creation
of the
>> non sharded stubs (do they have a more official term?) in the visible
mount
>> point was as speedy as expected.  shards are painfully slow.
>> >
>> >
>> >
>> > Is your CPU usage through the roof?
>>
>> Currently it has almost no activity.  First node yesterday got a bit
>> high.  But 2nd node today that has issues is pretty low.
>> >
>> > If you haven't already, I'd suggest
>> >
>> > - changing "cluster.data-self-heal-algorithm" to
"full"
>> >
>> > - And restarting the gluster volume if possible
>> >
>>
>> I'll shut down vm's later tonight and see if that helps at all.
>>
>
> applied  "cluster.data-self-heal-algorithm" to "full"
>
> stopped volume - started volume
>
> cpu activity barely noticeable. heal count crawling at 1 new addition to
> list every minute or 2 and stil just building a list it hasnt started
> making a .shard directory
>
logging into each linux vm and running from / 'find . | xargs stat '
seeme
dto make count ump a bit faster. wasnt quite sure best way to repeatthat
for windows vm's so just ran full system virus scans.

Still after 15+ plus hours now its listed 3600/19000 shards in to be healed
list and started healing none.
> >
>> > I have a suspicion something changed recently with heal, I've
noticed
>> that it takes a long time (hours) to kick in when the diff algorithm is
>> used. I don't recall it doing this with 3.7.11
>> >
>> >
>> > --
>> > Lindsay Mathieson
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160827/331bf935/attachment.html>