Gluster users - Sep 2017 - high (sys) cpu util and high load

Hi

I'm seeing quite high cpu sys utilisation and an increased system load the
past few days on my servers. It appears it doesn't start at exactly the
same time for the different servers, but I've not (yet) been able to pin
the cpu usage to a specific task or entries in the logs.

The cluster is running distribute with 8nodes and 4 bricks each version 3.10
The nodes have 32 (HT) cores and 64 gigs of ram.

Does anyone know what I can attribute this behaviour to? Or how I can
figure out what is causing it?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170904/e65f370a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2017-09-04 at 15.53.31.png
Type: image/png
Size: 39302 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170904/e65f370a/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2017-09-04 at 15.51.24.png
Type: image/png
Size: 57241 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170904/e65f370a/attachment-0001.png>

On 09/04/2017 10:00 AM, Ingard Mev?g wrote:
> Hi
> 
> I'm seeing quite high cpu sys utilisation and an increased system load 
> the past few days on my servers. It appears it doesn't start at exactly
> the same time for the different servers, but I've not (yet) been able
to
> pin the cpu usage to a specific task or entries in the logs.
> 
> The cluster is running distribute with 8nodes and 4 bricks each version
3.10
> The nodes have 32 (HT) cores and 64 gigs of ram.
> 
> Does anyone know what I can attribute this behaviour to? Or how I can 
> figure out what is causing it?
> 
Looking for logs of servers and clients around the spike interval is a 
good place to start. Operations like self-heal in gluster can cause 
higher resource utilization than normal.

HTH,
Vijay

2017-09-05 19:58 GMT+02:00 Vijay Bellur <vbellur at redhat.com>:
> On 09/04/2017 10:00 AM, Ingard Mev?g wrote:
>
>> Hi
>>
>> I'm seeing quite high cpu sys utilisation and an increased system
load
>> the past few days on my servers. It appears it doesn't start at
exactly the
>> same time for the different servers, but I've not (yet) been able
to pin
>> the cpu usage to a specific task or entries in the logs.
>>
>> The cluster is running distribute with 8nodes and 4 bricks each version
>> 3.10
>> The nodes have 32 (HT) cores and 64 gigs of ram.
>>
>> Does anyone know what I can attribute this behaviour to? Or how I can
>> figure out what is causing it?
>>
>>
> Looking for logs of servers and clients around the spike interval is a
> good place to start. Operations like self-heal in gluster can cause higher
> resource utilization than normal.
>
I did think that it might be related to self-heal or rebalance or something
similar as it seemingly started at different times for the different
members of the cluster. But I couldn't find anything in the logs that
suggested that some background operation was running. I did notice the call
log saying that a lot of the bricks had Pending tasks. Could it be
connected to that? Or some other resource being starved?
Would gluster volume status show self heal or distribute running?

>
> HTH,
> Vijay
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170905/14a16e6b/attachment.html>