thr3ads.net - CentOS virt - [CentOS-virt] NUMA placement failed, performance might be affected [Feb 2017]

If this information is useful, please help other people find it:
Share via:

Chris Wik

2017-Feb-28 22:36 UTC

[CentOS-virt] NUMA placement failed, performance might be affected

I just did a yum update on a CentOS 7 / Xen 4.6 server which took me from
kernel-3.18.34-20.el7.x86_64 -> kernel-3.18.44-20.el7.x86_64

After rebooting, the following notice is printed immediately upon xl
create'ing a domain: libxl: notice:
libxl_numa.c:499:libxl__get_numa_candidate: NUMA placement failed, performance
might be affected


Indeed performance is significantly degraded. This issue didn't happen on
the older kernel. Any ideas where it came from or how it can be fixed?




# cat /proc/cpuinfo?
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz
stepping : 2
microcode : 0x36
cpu MHz : 3500.090
cache size : 15360 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr
sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon rep_good nopl nonstop_tsc
eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt
tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm ida arat epb
pln pts dtherm fsgsbase bmi1 avx2 bmi2 erms xsaveopt
bugs :
bogomips : 7000.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:



# xl info
host ? ? ? ? ? ? ? ? ? : ****.localdomain
release ? ? ? ? ? ? ? ?: 3.18.44-20.el7.x86_64
version ? ? ? ? ? ? ? ?: #1 SMP Tue Oct 25 11:42:52 UTC 2016
machine ? ? ? ? ? ? ? ?: x86_64
nr_cpus ? ? ? ? ? ? ? ?: 1
max_cpu_id ? ? ? ? ? ? : 15
nr_nodes ? ? ? ? ? ? ? : 2
cores_per_socket ? ? ? : 1
threads_per_core ? ? ? : 1
cpu_mhz ? ? ? ? ? ? ? ?: 3500
hw_caps ? ? ? ? ? ? ? ?:
bfebfbff:2c100800:00000000:00007f00:77fefbff:00000000:00000021:000037ab
virt_caps ? ? ? ? ? ? ?: hvm hvm_directio
total_memory ? ? ? ? ? : 130958
free_memory ? ? ? ? ? ?: 28003
sharing_freed_memory ? : 0
sharing_used_memory ? ?: 0
outstanding_claims ? ? : 0
free_cpus ? ? ? ? ? ? ?: 0
xen_major ? ? ? ? ? ? ?: 4
xen_minor ? ? ? ? ? ? ?: 6
xen_extra ? ? ? ? ? ? ?: .3-7.el7
xen_version ? ? ? ? ? ?: 4.6.3-7.el7
xen_caps ? ? ? ? ? ? ? : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64?
xen_scheduler ? ? ? ? ?: credit
xen_pagesize ? ? ? ? ? : 4096
platform_params ? ? ? ?: virt_start=0xffff800000000000
xen_changeset ? ? ? ? ?: Wed Feb 15 02:12:22 2017 -0600 git:c359c64-dirty
xen_commandline ? ? ? ?: placeholder dom0_mem=1024M,max:1024M cpuinfo
com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all
cc_compiler ? ? ? ? ? ?: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
cc_compile_by ? ? ? ? ?: mockbuild
cc_compile_domain ? ? ?: centos.org
cc_compile_date ? ? ? ?: Wed Feb 15 12:18:43 UTC 2017
xend_config_format ? ? : 4




Thanks in advance,


Chris
--
 Chris?Wik
 Anu?Internet?Services
 www.anu.net?|?www.cwik.ch
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20170228/49532d2f/attachment-0002.html>

Chris Wik

2017-Feb-28 23:06 UTC

head link

[CentOS-virt] NUMA placement failed, performance might be affected

Solved.


For the archives:


Noticed in xl info output that only 1 core was recognised, even though nr_cpus
should show 16 on this box.



Tried rebooting and selecting the old .34 kernel, console printed "smpboot:
do_boot_cpu failed(-1) to wakeup CPU#1" and CentOS failed to boot. Powered
down the box fully and started back up with the latest .44 kernel and everything
is working fine now.


Not enough of an expert to know if this is a bug in the kernel, a problem with
the BIOS, bad hardware, a completely random stroke of bad luck or a cosmic sign.
Happy it works now. If anyone has experienced this and has any suggestions they
would be received with gratitude.


Chris

--
 Chris?Wik
 Anu?Internet?Services
 www.anu.net?|?www.cwik.ch



 From:   Chris Wik <chris at anu.net> 
 To:   <centos-virt at centos.org> 
 Sent:   28/02/2017 11:36 PM 
 Subject:   [CentOS-virt] NUMA placement failed, performance might be affected 



I just did a yum update on a CentOS 7 / Xen 4.6 server which took me from
kernel-3.18.34-20.el7.x86_64 -> kernel-3.18.44-20.el7.x86_64

After rebooting, the following notice is printed immediately upon xl
create'ing a domain: libxl: notice:
libxl_numa.c:499:libxl__get_numa_candidate: NUMA placement failed, performance
might be affected


Indeed performance is significantly degraded. This issue didn't happen on
the older kernel. Any ideas where it came from or how it can be fixed?




# cat /proc/cpuinfo?
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz
stepping : 2
microcode : 0x36
cpu MHz : 3500.090
cache size : 15360 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr
sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon rep_good nopl nonstop_tsc
eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt
tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm ida arat epb
pln pts dtherm fsgsbase bmi1 avx2 bmi2 erms xsaveopt
bugs :
bogomips : 7000.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:



# xl info
host ? ? ? ? ? ? ? ? ? : ****.localdomain
release ? ? ? ? ? ? ? ?: 3.18.44-20.el7.x86_64
version ? ? ? ? ? ? ? ?: #1 SMP Tue Oct 25 11:42:52 UTC 2016
machine ? ? ? ? ? ? ? ?: x86_64
nr_cpus ? ? ? ? ? ? ? ?: 1
max_cpu_id ? ? ? ? ? ? : 15
nr_nodes ? ? ? ? ? ? ? : 2
cores_per_socket ? ? ? : 1
threads_per_core ? ? ? : 1
cpu_mhz ? ? ? ? ? ? ? ?: 3500
hw_caps ? ? ? ? ? ? ? ?:
bfebfbff:2c100800:00000000:00007f00:77fefbff:00000000:00000021:000037ab
virt_caps ? ? ? ? ? ? ?: hvm hvm_directio
total_memory ? ? ? ? ? : 130958
free_memory ? ? ? ? ? ?: 28003
sharing_freed_memory ? : 0
sharing_used_memory ? ?: 0
outstanding_claims ? ? : 0
free_cpus ? ? ? ? ? ? ?: 0
xen_major ? ? ? ? ? ? ?: 4
xen_minor ? ? ? ? ? ? ?: 6
xen_extra ? ? ? ? ? ? ?: .3-7.el7
xen_version ? ? ? ? ? ?: 4.6.3-7.el7
xen_caps ? ? ? ? ? ? ? : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64?
xen_scheduler ? ? ? ? ?: credit
xen_pagesize ? ? ? ? ? : 4096
platform_params ? ? ? ?: virt_start=0xffff800000000000
xen_changeset ? ? ? ? ?: Wed Feb 15 02:12:22 2017 -0600 git:c359c64-dirty
xen_commandline ? ? ? ?: placeholder dom0_mem=1024M,max:1024M cpuinfo
com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all
cc_compiler ? ? ? ? ? ?: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
cc_compile_by ? ? ? ? ?: mockbuild
cc_compile_domain ? ? ?: centos.org
cc_compile_date ? ? ? ?: Wed Feb 15 12:18:43 UTC 2017
xend_config_format ? ? : 4




Thanks in advance,


Chris
--
 Chris?Wik
 Anu?Internet?Services
 www.anu.net?|?www.cwik.ch

_______________________________________________ 
CentOS-virt mailing list 
CentOS-virt at centos.org 
https://lists.centos.org/mailman/listinfo/centos-virt 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20170301/5bb1e8c0/attachment-0002.html>

-=X.L.O.R.D=-

2017-Mar-01 16:18 UTC

head link

[CentOS-virt] NUMA placement failed, performance might be affected

Chris,

Check your cpuinfo from flag which did not indicate any ?VMX? support for your
CPU virtualization support on the board.

 

Xlord

 

From: CentOS-virt [mailto:centos-virt-bounces at centos.org] On Behalf Of Chris
Wik
Sent: Wednesday, March 1, 2017 7:07 AM
To: Discussion about the virtualization on CentOS <centos-virt at
centos.org>
Subject: Re: [CentOS-virt] NUMA placement failed, performance might be affected

 

Solved.

 

For the archives:

 

Noticed in xl info output that only 1 core was recognised, even though nr_cpus
should show 16 on this box.

 

Tried rebooting and selecting the old .34 kernel, console printed "smpboot:
do_boot_cpu failed(-1) to wakeup CPU#1" and CentOS failed to boot. Powered
down the box fully and started back up with the latest .44 kernel and everything
is working fine now.

 

Not enough of an expert to know if this is a bug in the kernel, a problem with
the BIOS, bad hardware, a completely random stroke of bad luck or a cosmic sign.
Happy it works now. If anyone has experienced this and has any suggestions they
would be received with gratitude.

 

Chris


--
Chris Wik
Anu Internet Services
www.anu.net <http://www.anu.net>  | www.cwik.ch <http://www.cwik.ch>



From: Chris Wik <chris at anu.net <mailto:chris at anu.net> > 
To: <centos-virt at centos.org <mailto:centos-virt at centos.org> > 
Sent: 28/02/2017 11:36 PM 
Subject: [CentOS-virt] NUMA placement failed, performance might be affected 

I just did a yum update on a CentOS 7 / Xen 4.6 server which took me from
kernel-3.18.34-20.el7.x86_64 -> kernel-3.18.44-20.el7.x86_64

 

After rebooting, the following notice is printed immediately upon xl
create'ing a domain: libxl: notice:
libxl_numa.c:499:libxl__get_numa_candidate: NUMA placement failed, performance
might be affected

 

Indeed performance is significantly degraded. This issue didn't happen on
the older kernel. Any ideas where it came from or how it can be fixed?

 

# cat /proc/cpuinfo 

processor     : 0

vendor_id    : GenuineIntel

cpu family   : 6

model                      : 63

model name : Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz

stepping       : 2

microcode    : 0x36

cpu MHz                 : 3500.090

cache size    : 15360 KB

physical id   : 0

siblings        : 1

core id                     : 0

cpu cores     : 1

apicid                       : 0

initial apicid : 0

fpu               : yes

fpu_exception         : yes

cpuid level   : 15

wp               : yes

flags             : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush
acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon rep_good
nopl nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2
movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm
ida arat epb pln pts dtherm fsgsbase bmi1 avx2 bmi2 erms xsaveopt

bugs             :

bogomips     : 7000.18

clflush size   : 64

cache_alignment      : 64

address sizes            : 46 bits physical, 48 bits virtual

power management:

 

# xl info

host                   : ****.localdomain

release                : 3.18.44-20.el7.x86_64

version                : #1 SMP Tue Oct 25 11:42:52 UTC 2016

machine                : x86_64

nr_cpus                : 1

max_cpu_id             : 15

nr_nodes               : 2

cores_per_socket       : 1

threads_per_core       : 1

cpu_mhz                : 3500

hw_caps                :
bfebfbff:2c100800:00000000:00007f00:77fefbff:00000000:00000021:000037ab

virt_caps              : hvm hvm_directio

total_memory           : 130958

free_memory            : 28003

sharing_freed_memory   : 0

sharing_used_memory    : 0

outstanding_claims     : 0

free_cpus              : 0

xen_major              : 4

xen_minor              : 6

xen_extra              : .3-7.el7

xen_version            : 4.6.3-7.el7

xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64

xen_scheduler          : credit

xen_pagesize           : 4096

platform_params        : virt_start=0xffff800000000000

xen_changeset          : Wed Feb 15 02:12:22 2017 -0600 git:c359c64-dirty

xen_commandline        : placeholder dom0_mem=1024M,max:1024M cpuinfo
com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all

cc_compiler            : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)

cc_compile_by          : mockbuild

cc_compile_domain      : centos.org

cc_compile_date        : Wed Feb 15 12:18:43 UTC 2017

xend_config_format     : 4

 

 

Thanks in advance,

 

Chris


--
Chris Wik
Anu Internet Services
www.anu.net <http://www.anu.net>  | www.cwik.ch <http://www.cwik.ch>



_______________________________________________ 
CentOS-virt mailing list 
CentOS-virt at centos.org <mailto:CentOS-virt at centos.org>  
https://lists.centos.org/mailman/listinfo/centos-virt 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20170302/d9de7f55/attachment-0002.html>

Possibly Parallel Threads

Search for more possibly parallel threads

CentOS virt - Feb 2017 - NUMA placement failed, performance might be affected

[CentOS-virt] NUMA placement failed, performance might be affected

[CentOS-virt] NUMA placement failed, performance might be affected

[CentOS-virt] NUMA placement failed, performance might be affected

Possibly Parallel Threads