Hi I'm seeing quite high cpu sys utilisation and an increased system load the past few days on my servers. It appears it doesn't start at exactly the same time for the different servers, but I've not (yet) been able to pin the cpu usage to a specific task or entries in the logs. The cluster is running distribute with 8nodes and 4 bricks each version 3.10 The nodes have 32 (HT) cores and 64 gigs of ram. Does anyone know what I can attribute this behaviour to? Or how I can figure out what is causing it? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170904/e65f370a/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2017-09-04 at 15.53.31.png Type: image/png Size: 39302 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170904/e65f370a/attachment.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2017-09-04 at 15.51.24.png Type: image/png Size: 57241 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170904/e65f370a/attachment-0001.png>
On 09/04/2017 10:00 AM, Ingard Mev?g wrote:> Hi > > I'm seeing quite high cpu sys utilisation and an increased system load > the past few days on my servers. It appears it doesn't start at exactly > the same time for the different servers, but I've not (yet) been able to > pin the cpu usage to a specific task or entries in the logs. > > The cluster is running distribute with 8nodes and 4 bricks each version 3.10 > The nodes have 32 (HT) cores and 64 gigs of ram. > > Does anyone know what I can attribute this behaviour to? Or how I can > figure out what is causing it? >Looking for logs of servers and clients around the spike interval is a good place to start. Operations like self-heal in gluster can cause higher resource utilization than normal. HTH, Vijay
2017-09-05 19:58 GMT+02:00 Vijay Bellur <vbellur at redhat.com>:> On 09/04/2017 10:00 AM, Ingard Mev?g wrote: > >> Hi >> >> I'm seeing quite high cpu sys utilisation and an increased system load >> the past few days on my servers. It appears it doesn't start at exactly the >> same time for the different servers, but I've not (yet) been able to pin >> the cpu usage to a specific task or entries in the logs. >> >> The cluster is running distribute with 8nodes and 4 bricks each version >> 3.10 >> The nodes have 32 (HT) cores and 64 gigs of ram. >> >> Does anyone know what I can attribute this behaviour to? Or how I can >> figure out what is causing it? >> >> > Looking for logs of servers and clients around the spike interval is a > good place to start. Operations like self-heal in gluster can cause higher > resource utilization than normal. >I did think that it might be related to self-heal or rebalance or something similar as it seemingly started at different times for the different members of the cluster. But I couldn't find anything in the logs that suggested that some background operation was running. I did notice the call log saying that a lot of the bricks had Pending tasks. Could it be connected to that? Or some other resource being starved? Would gluster volume status show self heal or distribute running?> > HTH, > Vijay > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170905/14a16e6b/attachment.html>