No luck with qemu-kvm-ev, the behavior is the same. Running perf record -a -g on the baremetal shows that most of the CPU time is in _raw_spin_lock Children Self Command Shared Object Symbol - 93.62% 93.62% qemu-kvm [kernel.kallsyms] [k] _raw_spin_lock - _raw_spin_lock + 45.30% kvm_mmu_sync_roots + 28.49% kvm_mmu_load + 25.00% mmu_free_roots + 1.12% tdp_page_fault ?n joi, 18 aug. 2016 la 11:59, Laurentiu Soica <laurentiu at soica.ro> a scris:> I've tried with KSM disabled and nothing changed. > > I've upgraded KVM to qemu-kvm-ev. I'm waiting to see if there are any > improvements and report back. > > ?n mie., 17 aug. 2016 la 15:10, Boris Derzhavets <bderzhavets at hotmail.com> > a scris: > >> For myself KSM is unpredictable feature. The problem is Compute, just >> this node >> >> does "copy on write" , so only Compute. >> >> My concern exactly is where would it lead to worse or better Guest >> behavior ? >> >> I am not expecting complete fix. I would track via top/htop and dmesg >> via Cron on 1-2 hr >> >> period. >> >> >> ------------------------------ >> *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> >> on behalf of Laurentiu Soica <laurentiu at soica.ro> >> *Sent:* Wednesday, August 17, 2016 6:38 AM >> >> *To:* Discussion about the virtualization on CentOS >> *Subject:* Re: [CentOS-virt] Nested KVM issue >> Both baremetal and compute ? Are there any other metrics do you consider >> useful to collect for troubleshooting purposes ? >> ?n mie., 17 aug. 2016 la 13:04, Boris Derzhavets <bderzhavets at hotmail.com> >> a scris: >> >>> It sounds weird, but attempt to disable KSM and see would it help or no ? >>> >>> >>> ------------------------------ >>> *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> >>> on behalf of Laurentiu Soica <laurentiu at soica.ro> >>> *Sent:* Wednesday, August 17, 2016 4:56 AM >>> >>> *To:* Discussion about the virtualization on CentOS >>> *Subject:* Re: [CentOS-virt] Nested KVM issue >>> Enabled the logging on both compute and baremetal. Nothing strange in >>> logs: >>> >>> on baremetal : >>> Wed Aug 17 11:51:01 EEST 2016: committed 62310764 free 58501808 >>> Wed Aug 17 11:51:01 EEST 2016: 87025667 < 123574516 and free > 24714903, >>> stop ksm >>> >>> on compute: >>> Wed Aug 17 08:52:52 UTC 2016: committed 24547132 free 76730936 >>> Wed Aug 17 08:52:52 UTC 2016: 45139624 < 102962460 and free > 20592492, >>> stop ksm >>> >>> and the compute node is again at 100% CPU utilization. >>> >>> >>> >>> ?n mar., 16 aug. 2016 la 15:26, Boris Derzhavets < >>> bderzhavets at hotmail.com> a scris: >>> >>>> I would enable ksmtuned logging ,if it has been done verify logs >>>> >>>> >>>> ------------------------------ >>>> *From:* centos-virt-bounces at centos.org <centos-virt-bounces at centos.org> >>>> on behalf of Laurentiu Soica <laurentiu at soica.ro> >>>> *Sent:* Tuesday, August 16, 2016 7:16 AM >>>> >>>> *To:* Discussion about the virtualization on CentOS >>>> *Subject:* Re: [CentOS-virt] Nested KVM issue >>>> Yes. It is on both baremetal and compute node. >>>> >>>> >>>> _______________________________________________ >>> CentOS-virt mailing list >>> CentOS-virt at centos.org >>> https://lists.centos.org/mailman/listinfo/centos-virt >>> >> _______________________________________________ >> CentOS-virt mailing list >> CentOS-virt at centos.org >> https://lists.centos.org/mailman/listinfo/centos-virt >> > --Laurentiu Soica -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20160822/392b8328/attachment-0002.html>
Laurentiu, Just to chip in, without thoroughly checking the thread (which I'd like to apologize in advance for): have you tried to test other kernel instead of the stock one? You can have a look at http://elrepo.org/tiki/kernel-ml and http://elrepo.org/tiki/kernel-lt and use them for some comparison testing. Cheers, --- Adi Pircalabu On 22-08-2016 18:31, Laurentiu Soica wrote:> No luck with qemu-kvm-ev, the behavior is the same. Running perf > record -a -g on the baremetal shows that most of the CPU time is in > _raw_spin_lock > > Children Self Command Shared Object > Symbol > - 93.62% > 93.62% qemu-kvm [kernel.kallsyms] [k] > _raw_spin_lock > > > - _raw_spin_lock > > > + 45.30% kvm_mmu_sync_roots > > > + 28.49% kvm_mmu_load > > > + 25.00% mmu_free_roots > > > + 1.12% tdp_page_fault > > ?n joi, 18 aug. 2016 la 11:59, Laurentiu Soica a > scris: > >> I've tried with KSM disabled and nothing changed. >> >> I've upgraded KVM to qemu-kvm-ev. I'm waiting to see if there are >> any improvements and report back. >> >> ?n mie., 17 aug. 2016 la 15:10, Boris Derzhavets >> <bderzhavets at hotmail.com> a scris: >> >> For myself KSM is unpredictable feature. The problem is Compute, >> just this node >> >> does "copy on write" , so only Compute. >> >> My concern exactly is where would it lead to worse or better Guest >> behavior ? >> >> I am not expecting complete fix. I would track via top/htop and >> dmesg via Cron on 1-2 hr >> >> period. >> >> ------------------------- >> >> FROM: centos-virt-bounces at centos.org >> <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica >> <laurentiu at soica.ro> >> >> SENT: Wednesday, August 17, 2016 6:38 AM >> >> TO: Discussion about the virtualization on CentOS >> SUBJECT: Re: [CentOS-virt] Nested KVM issue >> >> Both baremetal and compute ? Are there any other metrics do you >> consider useful to collect for troubleshooting purposes ? >> >> ?n mie., 17 aug. 2016 la 13:04, Boris Derzhavets >> a scris: >> >> It sounds weird, but attempt to disable KSM and see would it help or >> no ? >> >> ------------------------- >> >> FROM: centos-virt-bounces at centos.org >> <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica >> >> SENT: Wednesday, August 17, 2016 4:56 AM >> >> TO: Discussion about the virtualization on CentOS >> SUBJECT: Re: [CentOS-virt] Nested KVM issue >> >> Enabled the logging on both compute and baremetal. Nothing strange >> in logs: >> >> on baremetal : >> >> Wed Aug 17 11:51:01 EEST 2016: committed 62310764 free 58501808 >> Wed Aug 17 11:51:01 EEST 2016: 87025667 < 123574516 and free > >> 24714903, stop ksm >> >> on compute: >> >> Wed Aug 17 08:52:52 UTC 2016: committed 24547132 free 76730936 >> Wed Aug 17 08:52:52 UTC 2016: 45139624 < 102962460 and free > >> 20592492, stop ksm >> >> and the compute node is again at 100% CPU utilization. >> >> ?n mar., 16 aug. 2016 la 15:26, Boris Derzhavets >> a scris: >> >> I would enable ksmtuned logging ,if it has been done verify logs >> >> ------------------------- >> >> FROM: centos-virt-bounces at centos.org >> <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica >> >> SENT: Tuesday, August 16, 2016 7:16 AM >> >> TO: Discussion about the virtualization on CentOS >> SUBJECT: Re: [CentOS-virt] Nested KVM issue >> >> Yes. It is on both baremetal and compute node. >> >> _______________________________________________ >> CentOS-virt mailing list >> CentOS-virt at centos.org >> https://lists.centos.org/mailman/listinfo/centos-virt > _______________________________________________ > CentOS-virt mailing list > CentOS-virt at centos.org > https://lists.centos.org/mailman/listinfo/centos-virt > > -- > > Laurentiu Soica > _______________________________________________ > CentOS-virt mailing list > CentOS-virt at centos.org > https://lists.centos.org/mailman/listinfo/centos-virt
I've tried the kernel-lt with same results. For kernel-ml, the baremetal wasn't able to boot at that time. ?n mar., 23 aug. 2016 la 02:22, Adi Pircalabu <adi at ddns.com.au> a scris:> Laurentiu, > > Just to chip in, without thoroughly checking the thread (which I'd like > to apologize in advance for): have you tried to test other kernel > instead of the stock one? You can have a look at > http://elrepo.org/tiki/kernel-ml and http://elrepo.org/tiki/kernel-lt > and use them for some comparison testing. > Cheers, > > --- > Adi Pircalabu > > On 22-08-2016 18:31, Laurentiu Soica wrote: > > No luck with qemu-kvm-ev, the behavior is the same. Running perf > > record -a -g on the baremetal shows that most of the CPU time is in > > _raw_spin_lock > > > > Children Self Command Shared Object > > Symbol > > - 93.62% > > 93.62% qemu-kvm [kernel.kallsyms] [k] > > _raw_spin_lock > > > > > > - _raw_spin_lock > > > > > > + 45.30% kvm_mmu_sync_roots > > > > > > + 28.49% kvm_mmu_load > > > > > > + 25.00% mmu_free_roots > > > > > > + 1.12% tdp_page_fault > > > > ?n joi, 18 aug. 2016 la 11:59, Laurentiu Soica a > > scris: > > > >> I've tried with KSM disabled and nothing changed. > >> > >> I've upgraded KVM to qemu-kvm-ev. I'm waiting to see if there are > >> any improvements and report back. > >> > >> ?n mie., 17 aug. 2016 la 15:10, Boris Derzhavets > >> <bderzhavets at hotmail.com> a scris: > >> > >> For myself KSM is unpredictable feature. The problem is Compute, > >> just this node > >> > >> does "copy on write" , so only Compute. > >> > >> My concern exactly is where would it lead to worse or better Guest > >> behavior ? > >> > >> I am not expecting complete fix. I would track via top/htop and > >> dmesg via Cron on 1-2 hr > >> > >> period. > >> > >> ------------------------- > >> > >> FROM: centos-virt-bounces at centos.org > >> <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica > >> <laurentiu at soica.ro> > >> > >> SENT: Wednesday, August 17, 2016 6:38 AM > >> > >> TO: Discussion about the virtualization on CentOS > >> SUBJECT: Re: [CentOS-virt] Nested KVM issue > >> > >> Both baremetal and compute ? Are there any other metrics do you > >> consider useful to collect for troubleshooting purposes ? > >> > >> ?n mie., 17 aug. 2016 la 13:04, Boris Derzhavets > >> a scris: > >> > >> It sounds weird, but attempt to disable KSM and see would it help or > >> no ? > >> > >> ------------------------- > >> > >> FROM: centos-virt-bounces at centos.org > >> <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica > >> > >> SENT: Wednesday, August 17, 2016 4:56 AM > >> > >> TO: Discussion about the virtualization on CentOS > >> SUBJECT: Re: [CentOS-virt] Nested KVM issue > >> > >> Enabled the logging on both compute and baremetal. Nothing strange > >> in logs: > >> > >> on baremetal : > >> > >> Wed Aug 17 11:51:01 EEST 2016: committed 62310764 free 58501808 > >> Wed Aug 17 11:51:01 EEST 2016: 87025667 < 123574516 and free > > >> 24714903, stop ksm > >> > >> on compute: > >> > >> Wed Aug 17 08:52:52 UTC 2016: committed 24547132 free 76730936 > >> Wed Aug 17 08:52:52 UTC 2016: 45139624 < 102962460 and free > > >> 20592492, stop ksm > >> > >> and the compute node is again at 100% CPU utilization. > >> > >> ?n mar., 16 aug. 2016 la 15:26, Boris Derzhavets > >> a scris: > >> > >> I would enable ksmtuned logging ,if it has been done verify logs > >> > >> ------------------------- > >> > >> FROM: centos-virt-bounces at centos.org > >> <centos-virt-bounces at centos.org> on behalf of Laurentiu Soica > >> > >> SENT: Tuesday, August 16, 2016 7:16 AM > >> > >> TO: Discussion about the virtualization on CentOS > >> SUBJECT: Re: [CentOS-virt] Nested KVM issue > >> > >> Yes. It is on both baremetal and compute node. > >> > >> _______________________________________________ > >> CentOS-virt mailing list > >> CentOS-virt at centos.org > >> https://lists.centos.org/mailman/listinfo/centos-virt > > _______________________________________________ > > CentOS-virt mailing list > > CentOS-virt at centos.org > > https://lists.centos.org/mailman/listinfo/centos-virt > > > > -- > > > > Laurentiu Soica > > _______________________________________________ > > CentOS-virt mailing list > > CentOS-virt at centos.org > > https://lists.centos.org/mailman/listinfo/centos-virt >-- Laurentiu Soica -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20160823/21b5ae6d/attachment-0002.html>