CentOS - Jan 2017 - Increase CPU usage on HV after upgrade (7.2 -> 7.3)

Hello ,

After upgrading the system from CentOS 7.2.1511 to CenOS 7.3.1611 I
see that the average processing time has increased from 5-7% to 12-15%
(doubled). Not critical but it is not pleasant. Server as KVM with 5
virtual machines. Someone noticed something similar? If so, how to fix
that? 

Thx.

On Wed, Jan 18, 2017 at 2:34 PM, Subscriber <ml-lists at agoris.net.ua>
wrote:
> Hello ,
>
> After upgrading the system from CentOS 7.2.1511 to CenOS 7.3.1611 I
> see that the average processing time has increased from 5-7% to 12-15%
> (doubled). Not critical but it is not pleasant. Server as KVM with 5
> virtual machines. Someone noticed something similar? If so, how to fix
> that?
>
> Thx.
>
>
>
In the mean time, if you have not disabled it, you should find some
collected statistics from sysstat/sar.
Look at the sarXX files under /var/log/sa. They should be kept for 30 day
by default in CentOS 7.
So you can compare cpu, mem, I/O profiles before and after the upgrade.
If you have access to Red Hat documents you can look also here:
https://access.redhat.com/articles/325783

or in general some articles like this:
https://www.blackmoreops.com/2014/06/18/sysstat-sar-examples-usage/
and to create pdf graphics with kSar
https://www.thomas-krenn.com/en/wiki/Linux_Performance_Analysis_using_kSar

HIH,
Gianluca

On 01/18/2017 05:34 AM, Subscriber wrote:
> Someone noticed something similar?

How is your storage arranged, and what kind of IO patterns do those VMs 
have?

During recent testing, I found that the read performance of software 
RAID volumes was worse under 7.3 than it was under 7.2. Most other IO 
had improved significantly:

https://plus.google.com/+GordonMessmer/posts/eSe6iNmk1Fs?sfc=false

On Wed, Jan 18, 2017 at 2:34 PM, Subscriber <ml-lists at agoris.net.ua>
wrote:
> Hello ,
>
> After upgrading the system from CentOS 7.2.1511 to CenOS 7.3.1611 I
> see that the average processing time has increased from 5-7% to 12-15%
> (doubled). Not critical but it is not pleasant. Server as KVM with 5
> virtual machines. Someone noticed something similar? If so, how to fix
> that?
>
>
Discussed with KVM guys, here's the answer:
1% per VM seems to much on idle VM; if the VM is not idle you are
getting better performance in exchange.  For idle VMs the difference
should be roughly 0.1% per VM.




> Thx.
>
>
> _______________________________________________
> CentOS-virt mailing list
> CentOS-virt at centos.org
> https://lists.centos.org/mailman/listinfo/centos-virt
>


-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20170119/c034821c/attachment-0002.html>

Hello Gordon,

Wednesday, January 18, 2017, 11:52:35 PM, you wrote:
> On 01/18/2017 05:34 AM, Subscriber wrote:
>> Someone noticed something similar?
> How is your storage arranged,
It is software RAID1 + LVM
> and what kind of IO patterns do those VMs
> have?
Do not quite understand. What do you mean?
> During recent testing, I found that the read performance of software 
> RAID volumes was worse under 7.3 than it was under 7.2. Most other IO 
> had improved significantly:
> https://plus.google.com/+GordonMessmer/posts/eSe6iNmk1Fs?sfc=false

Hello Gianluca,

Wednesday, January 18, 2017, 3:54:15 PM, you wrote:
> In the mean time, if you have not disabled it, you should find some
> collected statistics from sysstat/sar.
> Look at the sarXX files under /var/log/sa. They should be kept for 30 day
> by default in CentOS 7.
Unfortunately, on that host such statistics is disabled.
> So you can compare cpu, mem, I/O profiles before and after the upgrade.
> If you have access to Red Hat documents you can look also here:
> https://access.redhat.com/articles/325783
> or in general some articles like this:
> https://www.blackmoreops.com/2014/06/18/sysstat-sar-examples-usage/
> and to create pdf graphics with kSar
> https://www.thomas-krenn.com/en/wiki/Linux_Performance_Analysis_using_kSar
> HIH,
> Gianluca
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos


-- 
Best regards,
 Subscriber                            mailto:ml-lists at agoris.net.ua

Hello Sandro,

Friday, January 20, 2017, 6:54:02 PM, you wrote:


> On Thu, Jan 19, 2017 at 3:40 PM, Subscriber <ml-lists at
agoris.net.ua> wrote:
> Hello Sandro,
> Thursday, January 19, 2017, 12:12:25 PM, you wrote:
>> Discussed with KVM guys, here's the answer:
>> 1% per VM seems to much on idle VM; if the VM is not idle you are
>> getting better performance in exchange.  For idle VMs the difference
>> should be roughly 0.1% per VM.
> I compared the graphics load on the CPU before and after the upgrade.
> After  the  upgrade is clearly increased CPU System time load.  In all
> of this mode and load VM themselves became approximately the same.

> Can you share exact version of kvm, libvirt and qemu-kvm / qemu-kvm-ev
you're using?
kvm_intel

In CentOS 7.2.1511:

kernel-3.10.0-327.36.3.el7.x86_64
qemu-kvm-1.5.3-105.el7_2.7.x86_64
libvirt-daemon-kvm-1.2.17-13.el7_2.5.x86_64

After upgrade from 7.2.1511 to 7.3.1611:
It was (after upgrading to this version, I noticed described in the first
report the situation):

kernel-3.10.0-514.2.2.el7.x86_64
qemu-kvm-1.5.3-126.el7.x86_64
libvirt-daemon-kvm-2.0.0-10.el7_3.2.x86_64

Now (as of 2017/01/22):

kernel-3.10.0-514.6.1.el7.x86_64
qemu-kvm-1.5.3-126.el7_3.3.x86_64
libvirt-daemon-kvm-2.0.0-10.el7_3.4.x86_64

It takes 1-2 days for monitoring, to determine whether the situation
has changed since the last update.

It is hard to define that extra % are leak due to upgrade, maybe should try on
little heavy load VM and see if it is still constant.

 

Xlord

 

From: CentOS-virt [mailto:centos-virt-bounces at centos.org] On Behalf Of Sandro
Bonazzola
Sent: Thursday, January 19, 2017 6:12 PM
To: Subscriber <ml-lists at agoris.net.ua>; Discussion about the
virtualization on CentOS <centos-virt at centos.org>
Cc: centos at centos.org
Subject: Re: [CentOS-virt] Increase CPU usage on HV after upgrade (7.2 ->
7.3)

 

 

 

On Wed, Jan 18, 2017 at 2:34 PM, Subscriber <ml-lists at agoris.net.ua
<mailto:ml-lists at agoris.net.ua> > wrote:

Hello ,

After upgrading the system from CentOS 7.2.1511 to CenOS 7.3.1611 I
see that the average processing time has increased from 5-7% to 12-15%
(doubled). Not critical but it is not pleasant. Server as KVM with 5
virtual machines. Someone noticed something similar? If so, how to fix
that?

 

Discussed with KVM guys, here's the answer:

1% per VM seems to much on idle VM; if the VM is not idle you are
getting better performance in exchange.  For idle VMs the difference
should be roughly 0.1% per VM.

 

 

 

Thx.


_______________________________________________
CentOS-virt mailing list
CentOS-virt at centos.org <mailto:CentOS-virt at centos.org> 
https://lists.centos.org/mailman/listinfo/centos-virt





 

-- 

Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com <http://redhat.com> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20170123/ede50975/attachment-0002.html>