Pranith Kumar Karampuri
2018-Aug-17 06:26 UTC
[Gluster-users] Gluter 3.12.12: performance during heal and in general
As per the output, all io-threads are using a lot of CPU. It is better to check what the volume profile is to see what is leading to so much work for io-threads. Please follow the documentation at https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/ section: "Running GlusterFS Volume Profile Command"and attach output of "gluster volume profile info", On Fri, Aug 17, 2018 at 11:24 AM Hu Bert <revirii at googlemail.com> wrote:> Good morning, > > i ran the command during 100% CPU usage and attached the file. > Hopefully it helps. > > 2018-08-17 7:33 GMT+02:00 Pranith Kumar Karampuri <pkarampu at redhat.com>: > > Could you do the following on one of the nodes where you are observing > high > > CPU usage and attach that file to this thread? We can find what > > threads/processes are leading to high usage. Do this for say 10 minutes > when > > you see the ~100% CPU. > > > > top -bHd 5 > /tmp/top.${HOSTNAME}.txt > > > > On Wed, Aug 15, 2018 at 2:37 PM Hu Bert <revirii at googlemail.com> wrote: > >> > >> Hello again :-) > >> > >> The self heal must have finished as there are no log entries in > >> glustershd.log files anymore. According to munin disk latency (average > >> io wait) has gone down to 100 ms, and disk utilization has gone down > >> to ~60% - both on all servers and hard disks. > >> > >> But now system load on 2 servers (which were in the good state) > >> fluctuates between 60 and 100; the server with the formerly failed > >> disk has a load of 20-30.I've uploaded some munin graphics of the cpu > >> usage: > >> > >> https://abload.de/img/gluster11_cpu31d3a.png > >> https://abload.de/img/gluster12_cpu8sem7.png > >> https://abload.de/img/gluster13_cpud7eni.png > >> > >> This can't be normal. 2 of the servers under heavy load and one not > >> that much. Does anyone have an explanation of this strange behaviour? > >> > >> > >> Thx :-) > >> > >> 2018-08-14 9:37 GMT+02:00 Hu Bert <revirii at googlemail.com>: > >> > Hi there, > >> > > >> > well, it seems the heal has finally finished. Couldn't see/find any > >> > related log message; is there such a message in a specific log file? > >> > > >> > But i see the same behaviour when the last heal finished: all CPU > >> > cores are consumed by brick processes; not only by the formerly failed > >> > bricksdd1, but by all 4 brick processes (and their threads). Load goes > >> > up to > 100 on the 2 servers with the not-failed brick, and > >> > glustershd.log gets filled with a lot of entries. Load on the server > >> > with the then failed brick not that high, but still ~60. > >> > > >> > Is this behaviour normal? Is there some post-heal after a heal has > >> > finished? > >> > > >> > thx in advance :-) > > > > > > > > -- > > Pranith >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180817/9df59f16/attachment.html>
Pranith Kumar Karampuri
2018-Aug-17 06:31 UTC
[Gluster-users] Gluter 3.12.12: performance during heal and in general
Please do volume profile also for around 10 minutes when CPU% is high. On Fri, Aug 17, 2018 at 11:56 AM Pranith Kumar Karampuri < pkarampu at redhat.com> wrote:> As per the output, all io-threads are using a lot of CPU. It is better to > check what the volume profile is to see what is leading to so much work for > io-threads. Please follow the documentation at > https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/ > section: "Running GlusterFS Volume Profile Command"and attach output of "gluster > volume profile info", > > On Fri, Aug 17, 2018 at 11:24 AM Hu Bert <revirii at googlemail.com> wrote: > >> Good morning, >> >> i ran the command during 100% CPU usage and attached the file. >> Hopefully it helps. >> >> 2018-08-17 7:33 GMT+02:00 Pranith Kumar Karampuri <pkarampu at redhat.com>: >> > Could you do the following on one of the nodes where you are observing >> high >> > CPU usage and attach that file to this thread? We can find what >> > threads/processes are leading to high usage. Do this for say 10 minutes >> when >> > you see the ~100% CPU. >> > >> > top -bHd 5 > /tmp/top.${HOSTNAME}.txt >> > >> > On Wed, Aug 15, 2018 at 2:37 PM Hu Bert <revirii at googlemail.com> wrote: >> >> >> >> Hello again :-) >> >> >> >> The self heal must have finished as there are no log entries in >> >> glustershd.log files anymore. According to munin disk latency (average >> >> io wait) has gone down to 100 ms, and disk utilization has gone down >> >> to ~60% - both on all servers and hard disks. >> >> >> >> But now system load on 2 servers (which were in the good state) >> >> fluctuates between 60 and 100; the server with the formerly failed >> >> disk has a load of 20-30.I've uploaded some munin graphics of the cpu >> >> usage: >> >> >> >> https://abload.de/img/gluster11_cpu31d3a.png >> >> https://abload.de/img/gluster12_cpu8sem7.png >> >> https://abload.de/img/gluster13_cpud7eni.png >> >> >> >> This can't be normal. 2 of the servers under heavy load and one not >> >> that much. Does anyone have an explanation of this strange behaviour? >> >> >> >> >> >> Thx :-) >> >> >> >> 2018-08-14 9:37 GMT+02:00 Hu Bert <revirii at googlemail.com>: >> >> > Hi there, >> >> > >> >> > well, it seems the heal has finally finished. Couldn't see/find any >> >> > related log message; is there such a message in a specific log file? >> >> > >> >> > But i see the same behaviour when the last heal finished: all CPU >> >> > cores are consumed by brick processes; not only by the formerly >> failed >> >> > bricksdd1, but by all 4 brick processes (and their threads). Load >> goes >> >> > up to > 100 on the 2 servers with the not-failed brick, and >> >> > glustershd.log gets filled with a lot of entries. Load on the server >> >> > with the then failed brick not that high, but still ~60. >> >> > >> >> > Is this behaviour normal? Is there some post-heal after a heal has >> >> > finished? >> >> > >> >> > thx in advance :-) >> > >> > >> > >> > -- >> > Pranith >> > > > -- > Pranith >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180817/81f95a2b/attachment.html>