similar to: Nested KVM issue

________________________________ From: centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica <laurentiu at soica.ro> Sent: Sunday, August 14, 2016 10:17 AM To: Discussion about the virtualization on CentOS Subject: Re: [CentOS-virt] Nested KVM issue More details on the subject: I suppose it is a nested KVM issue because it raised after I

Sorry, How you trigger the problem ? B. ________________________________ From: centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica <laurentiu at soica.ro> Sent: Tuesday, August 16, 2016 3:28 AM To: Discussion about the virtualization on CentOS Subject: Re: [CentOS-virt] Nested KVM issue Hello, The issue reproduced again and it

Is KSM enabled on your Compute Nodes ( presuming CentOS 7.2 on bare metal ) ? ________________________________ From: centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica <laurentiu at soica.ro> Sent: Tuesday, August 16, 2016 5:25 AM To: Discussion about the virtualization on CentOS Subject: Re: [CentOS-virt] Nested KVM issue Running the

I would enable ksmtuned logging ,if it has been done verify logs ________________________________ From: centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica <laurentiu at soica.ro> Sent: Tuesday, August 16, 2016 7:16 AM To: Discussion about the virtualization on CentOS Subject: Re: [CentOS-virt] Nested KVM issue Yes. It is on both

It sounds weird, but attempt to disable KSM and see would it help or no ? ________________________________ From: centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica <laurentiu at soica.ro> Sent: Wednesday, August 17, 2016 4:56 AM To: Discussion about the virtualization on CentOS Subject: Re: [CentOS-virt] Nested KVM issue Enabled the

Reports posted look good for me. Config should provide the best available performance for cloud VM (L2) on Compute Node. 1. Please, remind me what goes wrong from your standpoint ? 2. Which CPU is installed on Compute Node && how much RAM ? Actually , my concern is :- Number_of_ Cloud_VMs versus Number_CPU_Cores ( not threads) Please, check `top` report in

For myself KSM is unpredictable feature. The problem is Compute, just this node does "copy on write" , so only Compute. My concern exactly is where would it lead to worse or better Guest behavior ? I am not expecting complete fix. I would track via top/htop and dmesg via Cron on 1-2 hr period. ________________________________ From: centos-virt-bounces at centos.org

I would attempt to decrease number of VCPUS allocated to cloud VMs. Say try 4 => 2 . My guess there is not enough VCPUs to run OS itself. I also guess CPU model << Haswell. Please , confirm ( or not) if possible. Since Haswell was launched via my experience Intel Xeons based on this kernel (or latter kernels ) behaves much better then SandyBridge or IvyBridge based. Boris.

No luck with qemu-kvm-ev, the behavior is the same. Running perf record -a -g on the baremetal shows that most of the CPU time is in _raw_spin_lock Children Self Command Shared Object Symbol - 93.62% 93.62% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock - _raw_spin_lock + 45.30%

So basically you're saying that this THP feature is broken for CentOS 7 ? The feature is enabled by default. Is there a thread opened on this issue ? ?n mie., 24 aug. 2016 la 18:13, <m.roth at 5-cent.us> a scris: > Laurentiu Soica wrote: > > Hello, > > > > I have a CentOS 7 installation on baremetal with 2 CPUs, 10 cores each > and > > HT enabled, 128 GB

Hello, I have a CentOS 7 installation on baremetal with 2 CPUs, 10 cores each and HT enabled, 128 GB RAM. The system has transparent hugetables enabled. cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never The system reports anonymous hugepages pages usage and a size of hugepage of 2048Kb cat /proc/meminfo |grep -i hugepages|grep AnonHugePages AnonHugePages: 35491840 kB

More details on the subject: I suppose it is a nested KVM issue because it raised after I enabled the nested KVM feature. Without it, anyway, the second level VMs are unusable in terms of performance. I am using CentOS 7 with: kernel: 3.10.0-327.22.2.el7.x86_64 qemu-kvm:1.5.3-105.el7_2.4 libvirt:1.2.17-13.el7_2.5 on both the baremetal and the compute VM. The only workaround now is to shutdown

Both baremetal and compute ? Are there any other metrics do you consider useful to collect for troubleshooting purposes ? ?n mie., 17 aug. 2016 la 13:04, Boris Derzhavets <bderzhavets at hotmail.com> a scris: > It sounds weird, but attempt to disable KSM and see would it help or no ? > > > ------------------------------ > *From:* centos-virt-bounces at centos.org

Enabled the logging on both compute and baremetal. Nothing strange in logs: on baremetal : Wed Aug 17 11:51:01 EEST 2016: committed 62310764 free 58501808 Wed Aug 17 11:51:01 EEST 2016: 87025667 < 123574516 and free > 24714903, stop ksm on compute: Wed Aug 17 08:52:52 UTC 2016: committed 24547132 free 76730936 Wed Aug 17 08:52:52 UTC 2016: 45139624 < 102962460 and free > 20592492,

Yes. It is on both baremetal and compute node. ?n mar., 16 aug. 2016 la 13:37, Boris Derzhavets <bderzhavets at hotmail.com> a scris: > Is KSM enabled on your Compute Nodes ( presuming CentOS 7.2 on bare metal > ) ? > > > ------------------------------ > *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> > on behalf of Laurentiu Soica

I've tried with KSM disabled and nothing changed. I've upgraded KVM to qemu-kvm-ev. I'm waiting to see if there are any improvements and report back. ?n mie., 17 aug. 2016 la 15:10, Boris Derzhavets <bderzhavets at hotmail.com> a scris: > For myself KSM is unpredictable feature. The problem is Compute, just this > node > > does "copy on write" , so only

Running the compute node for several days simply triggers it. ?n mar., 16 aug. 2016 la 12:12, Boris Derzhavets <bderzhavets at hotmail.com> a scris: > Sorry, > > How you trigger the problem ? > > B. > > > ------------------------------ > *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> > on behalf of Laurentiu Soica

Hello, 1. <domain type='kvm' id='6'> <name>baremetalbrbm_1</name> <uuid>534e9b54-5e4c-4acb-adcf-793f841551a7</uuid> <memory unit='KiB'>104857600</memory> <currentMemory unit='KiB'>104857600</currentMemory> <vcpu placement='static'>36</vcpu> <resource>

Hello, The issue reproduced again and it doesn't look like a swap problem. Some details: on the baremetal, from top: top - 08:08:52 up 5 days, 16:43, 3 users, load average: 36.19, 36.05, 36.05 Tasks: 493 total, 1 running, 492 sleeping, 0 stopped, 0 zombie %Cpu(s): 3.5 us, 87.9 sy, 0.0 ni, 8.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 12357451+total, 14296000 free,

The CPUs are IvyBridge microarchitecture, Xeon E5-2670 v2. I could try lowering the vCPUs but I doubt it would help. Please note that the same VMs are running just fine (with a load of 2 out of an acceptable 36 on the compute node) for about 3 days after a restart. ?n lun., 15 aug. 2016 la 08:42, Boris Derzhavets <bderzhavets at hotmail.com> a scris: > I would attempt to decrease