Subscriber
2017-Jan-18 13:34 UTC
[CentOS] Increase CPU usage on HV after upgrade (7.2 -> 7.3)
Hello , After upgrading the system from CentOS 7.2.1511 to CenOS 7.3.1611 I see that the average processing time has increased from 5-7% to 12-15% (doubled). Not critical but it is not pleasant. Server as KVM with 5 virtual machines. Someone noticed something similar? If so, how to fix that? Thx.
Gianluca Cecchi
2017-Jan-18 13:54 UTC
[CentOS] Increase CPU usage on HV after upgrade (7.2 -> 7.3)
On Wed, Jan 18, 2017 at 2:34 PM, Subscriber <ml-lists at agoris.net.ua> wrote:> Hello , > > After upgrading the system from CentOS 7.2.1511 to CenOS 7.3.1611 I > see that the average processing time has increased from 5-7% to 12-15% > (doubled). Not critical but it is not pleasant. Server as KVM with 5 > virtual machines. Someone noticed something similar? If so, how to fix > that? > > Thx. > > >In the mean time, if you have not disabled it, you should find some collected statistics from sysstat/sar. Look at the sarXX files under /var/log/sa. They should be kept for 30 day by default in CentOS 7. So you can compare cpu, mem, I/O profiles before and after the upgrade. If you have access to Red Hat documents you can look also here: https://access.redhat.com/articles/325783 or in general some articles like this: https://www.blackmoreops.com/2014/06/18/sysstat-sar-examples-usage/ and to create pdf graphics with kSar https://www.thomas-krenn.com/en/wiki/Linux_Performance_Analysis_using_kSar HIH, Gianluca
Gordon Messmer
2017-Jan-18 21:52 UTC
[CentOS] Increase CPU usage on HV after upgrade (7.2 -> 7.3)
On 01/18/2017 05:34 AM, Subscriber wrote:> Someone noticed something similar?How is your storage arranged, and what kind of IO patterns do those VMs have? During recent testing, I found that the read performance of software RAID volumes was worse under 7.3 than it was under 7.2. Most other IO had improved significantly: https://plus.google.com/+GordonMessmer/posts/eSe6iNmk1Fs?sfc=false
Sandro Bonazzola
2017-Jan-19 10:12 UTC
[CentOS-virt] Increase CPU usage on HV after upgrade (7.2 -> 7.3)
On Wed, Jan 18, 2017 at 2:34 PM, Subscriber <ml-lists at agoris.net.ua> wrote:> Hello , > > After upgrading the system from CentOS 7.2.1511 to CenOS 7.3.1611 I > see that the average processing time has increased from 5-7% to 12-15% > (doubled). Not critical but it is not pleasant. Server as KVM with 5 > virtual machines. Someone noticed something similar? If so, how to fix > that? > >Discussed with KVM guys, here's the answer: 1% per VM seems to much on idle VM; if the VM is not idle you are getting better performance in exchange. For idle VMs the difference should be roughly 0.1% per VM.> Thx. > > > _______________________________________________ > CentOS-virt mailing list > CentOS-virt at centos.org > https://lists.centos.org/mailman/listinfo/centos-virt >-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20170119/c034821c/attachment-0002.html>
Subscriber
2017-Jan-19 14:29 UTC
[CentOS] Increase CPU usage on HV after upgrade (7.2 -> 7.3)
Hello Gordon, Wednesday, January 18, 2017, 11:52:35 PM, you wrote:> On 01/18/2017 05:34 AM, Subscriber wrote: >> Someone noticed something similar?> How is your storage arranged,It is software RAID1 + LVM> and what kind of IO patterns do those VMs > have?Do not quite understand. What do you mean?> During recent testing, I found that the read performance of software > RAID volumes was worse under 7.3 than it was under 7.2. Most other IO > had improved significantly:> https://plus.google.com/+GordonMessmer/posts/eSe6iNmk1Fs?sfc=false
Subscriber
2017-Jan-19 14:44 UTC
[CentOS] Increase CPU usage on HV after upgrade (7.2 -> 7.3)
Hello Gianluca, Wednesday, January 18, 2017, 3:54:15 PM, you wrote:> In the mean time, if you have not disabled it, you should find some > collected statistics from sysstat/sar. > Look at the sarXX files under /var/log/sa. They should be kept for 30 day > by default in CentOS 7.Unfortunately, on that host such statistics is disabled.> So you can compare cpu, mem, I/O profiles before and after the upgrade. > If you have access to Red Hat documents you can look also here: > https://access.redhat.com/articles/325783> or in general some articles like this: > https://www.blackmoreops.com/2014/06/18/sysstat-sar-examples-usage/ > and to create pdf graphics with kSar > https://www.thomas-krenn.com/en/wiki/Linux_Performance_Analysis_using_kSar> HIH, > Gianluca > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos-- Best regards, Subscriber mailto:ml-lists at agoris.net.ua
Subscriber
2017-Jan-22 15:19 UTC
[CentOS] [CentOS-virt] Increase CPU usage on HV after upgrade (7.2 -> 7.3)
Hello Sandro, Friday, January 20, 2017, 6:54:02 PM, you wrote:> On Thu, Jan 19, 2017 at 3:40 PM, Subscriber <ml-lists at agoris.net.ua> wrote: > Hello Sandro,> Thursday, January 19, 2017, 12:12:25 PM, you wrote:>> Discussed with KVM guys, here's the answer: >> 1% per VM seems to much on idle VM; if the VM is not idle you are >> getting better performance in exchange. For idle VMs the difference >> should be roughly 0.1% per VM.> I compared the graphics load on the CPU before and after the upgrade. > After the upgrade is clearly increased CPU System time load. In all > of this mode and load VM themselves became approximately the same.> Can you share exact version of kvm, libvirt and qemu-kvm / qemu-kvm-ev you're using?kvm_intel In CentOS 7.2.1511: kernel-3.10.0-327.36.3.el7.x86_64 qemu-kvm-1.5.3-105.el7_2.7.x86_64 libvirt-daemon-kvm-1.2.17-13.el7_2.5.x86_64 After upgrade from 7.2.1511 to 7.3.1611: It was (after upgrading to this version, I noticed described in the first report the situation): kernel-3.10.0-514.2.2.el7.x86_64 qemu-kvm-1.5.3-126.el7.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.2.x86_64 Now (as of 2017/01/22): kernel-3.10.0-514.6.1.el7.x86_64 qemu-kvm-1.5.3-126.el7_3.3.x86_64 libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64 It takes 1-2 days for monitoring, to determine whether the situation has changed since the last update.
-=X.L.O.R.D=-
2017-Jan-23 13:05 UTC
[CentOS-virt] Increase CPU usage on HV after upgrade (7.2 -> 7.3)
It is hard to define that extra % are leak due to upgrade, maybe should try on little heavy load VM and see if it is still constant. Xlord From: CentOS-virt [mailto:centos-virt-bounces at centos.org] On Behalf Of Sandro Bonazzola Sent: Thursday, January 19, 2017 6:12 PM To: Subscriber <ml-lists at agoris.net.ua>; Discussion about the virtualization on CentOS <centos-virt at centos.org> Cc: centos at centos.org Subject: Re: [CentOS-virt] Increase CPU usage on HV after upgrade (7.2 -> 7.3) On Wed, Jan 18, 2017 at 2:34 PM, Subscriber <ml-lists at agoris.net.ua <mailto:ml-lists at agoris.net.ua> > wrote: Hello , After upgrading the system from CentOS 7.2.1511 to CenOS 7.3.1611 I see that the average processing time has increased from 5-7% to 12-15% (doubled). Not critical but it is not pleasant. Server as KVM with 5 virtual machines. Someone noticed something similar? If so, how to fix that? Discussed with KVM guys, here's the answer: 1% per VM seems to much on idle VM; if the VM is not idle you are getting better performance in exchange. For idle VMs the difference should be roughly 0.1% per VM. Thx. _______________________________________________ CentOS-virt mailing list CentOS-virt at centos.org <mailto:CentOS-virt at centos.org> https://lists.centos.org/mailman/listinfo/centos-virt -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com <http://redhat.com> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20170123/ede50975/attachment-0002.html>