Chris Wik
2017-Feb-28 22:36 UTC
[CentOS-virt] NUMA placement failed, performance might be affected
I just did a yum update on a CentOS 7 / Xen 4.6 server which took me from kernel-3.18.34-20.el7.x86_64 -> kernel-3.18.44-20.el7.x86_64 After rebooting, the following notice is printed immediately upon xl create'ing a domain: libxl: notice: libxl_numa.c:499:libxl__get_numa_candidate: NUMA placement failed, performance might be affected Indeed performance is significantly degraded. This issue didn't happen on the older kernel. Any ideas where it came from or how it can be fixed? # cat /proc/cpuinfo? processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz stepping : 2 microcode : 0x36 cpu MHz : 3500.090 cache size : 15360 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon rep_good nopl nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm ida arat epb pln pts dtherm fsgsbase bmi1 avx2 bmi2 erms xsaveopt bugs : bogomips : 7000.18 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: # xl info host ? ? ? ? ? ? ? ? ? : ****.localdomain release ? ? ? ? ? ? ? ?: 3.18.44-20.el7.x86_64 version ? ? ? ? ? ? ? ?: #1 SMP Tue Oct 25 11:42:52 UTC 2016 machine ? ? ? ? ? ? ? ?: x86_64 nr_cpus ? ? ? ? ? ? ? ?: 1 max_cpu_id ? ? ? ? ? ? : 15 nr_nodes ? ? ? ? ? ? ? : 2 cores_per_socket ? ? ? : 1 threads_per_core ? ? ? : 1 cpu_mhz ? ? ? ? ? ? ? ?: 3500 hw_caps ? ? ? ? ? ? ? ?: bfebfbff:2c100800:00000000:00007f00:77fefbff:00000000:00000021:000037ab virt_caps ? ? ? ? ? ? ?: hvm hvm_directio total_memory ? ? ? ? ? : 130958 free_memory ? ? ? ? ? ?: 28003 sharing_freed_memory ? : 0 sharing_used_memory ? ?: 0 outstanding_claims ? ? : 0 free_cpus ? ? ? ? ? ? ?: 0 xen_major ? ? ? ? ? ? ?: 4 xen_minor ? ? ? ? ? ? ?: 6 xen_extra ? ? ? ? ? ? ?: .3-7.el7 xen_version ? ? ? ? ? ?: 4.6.3-7.el7 xen_caps ? ? ? ? ? ? ? : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64? xen_scheduler ? ? ? ? ?: credit xen_pagesize ? ? ? ? ? : 4096 platform_params ? ? ? ?: virt_start=0xffff800000000000 xen_changeset ? ? ? ? ?: Wed Feb 15 02:12:22 2017 -0600 git:c359c64-dirty xen_commandline ? ? ? ?: placeholder dom0_mem=1024M,max:1024M cpuinfo com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all cc_compiler ? ? ? ? ? ?: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) cc_compile_by ? ? ? ? ?: mockbuild cc_compile_domain ? ? ?: centos.org cc_compile_date ? ? ? ?: Wed Feb 15 12:18:43 UTC 2017 xend_config_format ? ? : 4 Thanks in advance, Chris -- Chris?Wik Anu?Internet?Services www.anu.net?|?www.cwik.ch -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20170228/49532d2f/attachment-0002.html>
Chris Wik
2017-Feb-28 23:06 UTC
[CentOS-virt] NUMA placement failed, performance might be affected
Solved. For the archives: Noticed in xl info output that only 1 core was recognised, even though nr_cpus should show 16 on this box. Tried rebooting and selecting the old .34 kernel, console printed "smpboot: do_boot_cpu failed(-1) to wakeup CPU#1" and CentOS failed to boot. Powered down the box fully and started back up with the latest .44 kernel and everything is working fine now. Not enough of an expert to know if this is a bug in the kernel, a problem with the BIOS, bad hardware, a completely random stroke of bad luck or a cosmic sign. Happy it works now. If anyone has experienced this and has any suggestions they would be received with gratitude. Chris -- Chris?Wik Anu?Internet?Services www.anu.net?|?www.cwik.ch From: Chris Wik <chris at anu.net> To: <centos-virt at centos.org> Sent: 28/02/2017 11:36 PM Subject: [CentOS-virt] NUMA placement failed, performance might be affected I just did a yum update on a CentOS 7 / Xen 4.6 server which took me from kernel-3.18.34-20.el7.x86_64 -> kernel-3.18.44-20.el7.x86_64 After rebooting, the following notice is printed immediately upon xl create'ing a domain: libxl: notice: libxl_numa.c:499:libxl__get_numa_candidate: NUMA placement failed, performance might be affected Indeed performance is significantly degraded. This issue didn't happen on the older kernel. Any ideas where it came from or how it can be fixed? # cat /proc/cpuinfo? processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz stepping : 2 microcode : 0x36 cpu MHz : 3500.090 cache size : 15360 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon rep_good nopl nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm ida arat epb pln pts dtherm fsgsbase bmi1 avx2 bmi2 erms xsaveopt bugs : bogomips : 7000.18 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: # xl info host ? ? ? ? ? ? ? ? ? : ****.localdomain release ? ? ? ? ? ? ? ?: 3.18.44-20.el7.x86_64 version ? ? ? ? ? ? ? ?: #1 SMP Tue Oct 25 11:42:52 UTC 2016 machine ? ? ? ? ? ? ? ?: x86_64 nr_cpus ? ? ? ? ? ? ? ?: 1 max_cpu_id ? ? ? ? ? ? : 15 nr_nodes ? ? ? ? ? ? ? : 2 cores_per_socket ? ? ? : 1 threads_per_core ? ? ? : 1 cpu_mhz ? ? ? ? ? ? ? ?: 3500 hw_caps ? ? ? ? ? ? ? ?: bfebfbff:2c100800:00000000:00007f00:77fefbff:00000000:00000021:000037ab virt_caps ? ? ? ? ? ? ?: hvm hvm_directio total_memory ? ? ? ? ? : 130958 free_memory ? ? ? ? ? ?: 28003 sharing_freed_memory ? : 0 sharing_used_memory ? ?: 0 outstanding_claims ? ? : 0 free_cpus ? ? ? ? ? ? ?: 0 xen_major ? ? ? ? ? ? ?: 4 xen_minor ? ? ? ? ? ? ?: 6 xen_extra ? ? ? ? ? ? ?: .3-7.el7 xen_version ? ? ? ? ? ?: 4.6.3-7.el7 xen_caps ? ? ? ? ? ? ? : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64? xen_scheduler ? ? ? ? ?: credit xen_pagesize ? ? ? ? ? : 4096 platform_params ? ? ? ?: virt_start=0xffff800000000000 xen_changeset ? ? ? ? ?: Wed Feb 15 02:12:22 2017 -0600 git:c359c64-dirty xen_commandline ? ? ? ?: placeholder dom0_mem=1024M,max:1024M cpuinfo com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all cc_compiler ? ? ? ? ? ?: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) cc_compile_by ? ? ? ? ?: mockbuild cc_compile_domain ? ? ?: centos.org cc_compile_date ? ? ? ?: Wed Feb 15 12:18:43 UTC 2017 xend_config_format ? ? : 4 Thanks in advance, Chris -- Chris?Wik Anu?Internet?Services www.anu.net?|?www.cwik.ch _______________________________________________ CentOS-virt mailing list CentOS-virt at centos.org https://lists.centos.org/mailman/listinfo/centos-virt -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20170301/5bb1e8c0/attachment-0002.html>
-=X.L.O.R.D=-
2017-Mar-01 16:18 UTC
[CentOS-virt] NUMA placement failed, performance might be affected
Chris, Check your cpuinfo from flag which did not indicate any ?VMX? support for your CPU virtualization support on the board. Xlord From: CentOS-virt [mailto:centos-virt-bounces at centos.org] On Behalf Of Chris Wik Sent: Wednesday, March 1, 2017 7:07 AM To: Discussion about the virtualization on CentOS <centos-virt at centos.org> Subject: Re: [CentOS-virt] NUMA placement failed, performance might be affected Solved. For the archives: Noticed in xl info output that only 1 core was recognised, even though nr_cpus should show 16 on this box. Tried rebooting and selecting the old .34 kernel, console printed "smpboot: do_boot_cpu failed(-1) to wakeup CPU#1" and CentOS failed to boot. Powered down the box fully and started back up with the latest .44 kernel and everything is working fine now. Not enough of an expert to know if this is a bug in the kernel, a problem with the BIOS, bad hardware, a completely random stroke of bad luck or a cosmic sign. Happy it works now. If anyone has experienced this and has any suggestions they would be received with gratitude. Chris -- Chris Wik Anu Internet Services www.anu.net <http://www.anu.net> | www.cwik.ch <http://www.cwik.ch> From: Chris Wik <chris at anu.net <mailto:chris at anu.net> > To: <centos-virt at centos.org <mailto:centos-virt at centos.org> > Sent: 28/02/2017 11:36 PM Subject: [CentOS-virt] NUMA placement failed, performance might be affected I just did a yum update on a CentOS 7 / Xen 4.6 server which took me from kernel-3.18.34-20.el7.x86_64 -> kernel-3.18.44-20.el7.x86_64 After rebooting, the following notice is printed immediately upon xl create'ing a domain: libxl: notice: libxl_numa.c:499:libxl__get_numa_candidate: NUMA placement failed, performance might be affected Indeed performance is significantly degraded. This issue didn't happen on the older kernel. Any ideas where it came from or how it can be fixed? # cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz stepping : 2 microcode : 0x36 cpu MHz : 3500.090 cache size : 15360 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 15 wp : yes flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon rep_good nopl nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm ida arat epb pln pts dtherm fsgsbase bmi1 avx2 bmi2 erms xsaveopt bugs : bogomips : 7000.18 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: # xl info host : ****.localdomain release : 3.18.44-20.el7.x86_64 version : #1 SMP Tue Oct 25 11:42:52 UTC 2016 machine : x86_64 nr_cpus : 1 max_cpu_id : 15 nr_nodes : 2 cores_per_socket : 1 threads_per_core : 1 cpu_mhz : 3500 hw_caps : bfebfbff:2c100800:00000000:00007f00:77fefbff:00000000:00000021:000037ab virt_caps : hvm hvm_directio total_memory : 130958 free_memory : 28003 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 6 xen_extra : .3-7.el7 xen_version : 4.6.3-7.el7 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : Wed Feb 15 02:12:22 2017 -0600 git:c359c64-dirty xen_commandline : placeholder dom0_mem=1024M,max:1024M cpuinfo com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all cc_compiler : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) cc_compile_by : mockbuild cc_compile_domain : centos.org cc_compile_date : Wed Feb 15 12:18:43 UTC 2017 xend_config_format : 4 Thanks in advance, Chris -- Chris Wik Anu Internet Services www.anu.net <http://www.anu.net> | www.cwik.ch <http://www.cwik.ch> _______________________________________________ CentOS-virt mailing list CentOS-virt at centos.org <mailto:CentOS-virt at centos.org> https://lists.centos.org/mailman/listinfo/centos-virt -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20170302/d9de7f55/attachment-0002.html>