Grzegorz Bakalarski
2014-Jan-07 16:21 UTC
xen 4.1.4 problem with vCPUs assignment to physical CPUs
Hi We have two identical hardware machines (Dell R815 AMD Opteron 4x 6172 + 192GB RAM each). One is running older Debian squeezy with Xen 4.1.2 (from backports) with home made kernel based on 3.2.0-4 linux kernel. This behaves normally Other one is running new Debian wheezy with Xen 4.1.4 with the same home made kernel as first one. This behaves abnormally. Both machines run few PV guests. It really few - older/normal runs 2 VMs (16 & 2 vCPUs == total 18, and newer/abnormal runs 5 VMs ( 16 & 8 & 4 & 6 & 8 == total 42) What is abnormal???? We test bigest (i.e. 16 CPU) guests. When running full load on normal VM, i.e. 16 number crunching proceses xentop shows utilization of 1600% CPU for the normal VM. Also when fully loaded xm vcpu-list shows that vCPUs are placed on different physical CPUs and Steal Time shown by top is under 1%. The same run on newer xen results in only 1200% CPU utilization (regardless of 16 or 32 or 128 running heavy processes), and steal time varies from 20 to 50%. xm vcpu-list shows that even for very overloaded guest 8 vCPUs are correctly assigned to 8 physical CPUs but other 8 vCPUs are by pair assigned to 4 physical CPUs (i.e. 2 vCPUs per one physical CPUs). BTW other guess machines are very very light loaded (less than 100% i.e. less then one vCPU) Both domU see 16 cpu. Home made kernel is almost debian kernel with NUMA switched off (in kernel). We restarted suspicious VM but no change. We cannot restart dom0 at the moment ... Any hints what going wrong? Will cpu pinnig help? Would creating cpu pool help ???? TIA GB PS Here xm info -n for both machines: ^^^^^^^ normal ^^^^^^^^^^^^^^^^^^^^ host : build1-dom0 release : 3.2.0-1-amd64 version : #1 SMP Wed Feb 29 15:14:57 UTC 2012 machine : x86_64 nr_cpus : 48 nr_nodes : 8 cores_per_socket : 12 threads_per_core : 1 cpu_mhz : 2200 hw_caps : 178bf3ff:efd3fbff:00000000:00001710:00802001:00000000:000837ff:00000000 virt_caps : hvm total_memory : 196598 free_memory : 119218 free_cpus : 0 cpu_topology : cpu: core socket node 0: 0 0 0 1: 1 0 0 2: 2 0 0 3: 3 0 0 4: 4 0 0 5: 5 0 0 6: 6 0 1 7: 7 0 1 8: 8 0 1 9: 9 0 1 10: 10 0 1 11: 11 0 1 12: 0 1 2 13: 1 1 2 14: 2 1 2 15: 3 1 2 16: 4 1 2 17: 5 1 2 18: 6 1 3 19: 7 1 3 20: 8 1 3 21: 9 1 3 22: 10 1 3 23: 11 1 3 24: 0 2 4 25: 1 2 4 26: 2 2 4 27: 3 2 4 28: 4 2 4 29: 5 2 4 30: 6 2 5 31: 7 2 5 32: 8 2 5 33: 9 2 5 34: 10 2 5 35: 11 2 5 36: 0 3 6 37: 1 3 6 38: 2 3 6 39: 3 3 6 40: 4 3 6 41: 5 3 6 42: 6 3 7 43: 7 3 7 44: 8 3 7 45: 9 3 7 46: 10 3 7 47: 11 3 7 numa_info : none xen_major : 4 xen_minor : 1 xen_extra : .2 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable xen_commandline : placeholder com2=115200,8n1 vga=mode-0x0319 console=com2,vga guest_loglvl=error dom0_mem=6144M cc_compiler : gcc version 4.6.2 (Debian 4.6.2-6) cc_compile_by : waldi cc_compile_domain : debian.org cc_compile_date : Sat Dec 10 19:58:21 UTC 2011 xend_config_format : 4 ============ABnormal=============host : prod3-dom0 release : 3.2.0-1-amd64 version : #1 SMP Wed Feb 29 15:14:57 UTC 2012 machine : x86_64 nr_cpus : 48 nr_nodes : 8 cores_per_socket : 12 threads_per_core : 1 cpu_mhz : 2200 hw_caps : 178bf3ff:efd3fbff:00000000:00001310:00802001:00000000:000837ff:00000000 virt_caps : hvm total_memory : 196598 free_memory : 57776 free_cpus : 0 cpu_topology : cpu: core socket node 0: 0 0 0 1: 1 0 0 2: 2 0 0 3: 3 0 0 4: 4 0 0 5: 5 0 0 6: 6 0 1 7: 7 0 1 8: 8 0 1 9: 9 0 1 10: 10 0 1 11: 11 0 1 12: 0 1 2 13: 1 1 2 14: 2 1 2 15: 3 1 2 16: 4 1 2 17: 5 1 2 18: 6 1 3 19: 7 1 3 20: 8 1 3 21: 9 1 3 22: 10 1 3 23: 11 1 3 24: 0 2 4 25: 1 2 4 26: 2 2 4 27: 3 2 4 28: 4 2 4 29: 5 2 4 30: 6 2 5 31: 7 2 5 32: 8 2 5 33: 9 2 5 34: 10 2 5 35: 11 2 5 36: 0 3 6 37: 1 3 6 38: 2 3 6 39: 3 3 6 40: 4 3 6 41: 5 3 6 42: 6 3 7 43: 7 3 7 44: 8 3 7 45: 9 3 7 46: 10 3 7 47: 11 3 7 numa_info : none xen_major : 4 xen_minor : 1 xen_extra : .4 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable xen_commandline : placeholder com2=115200,8n1 vga=mode-0x0319 console=com2,vga guest_loglvl=error dom0_mem=6144M cc_compiler : gcc version 4.7.2 (Debian 4.7.2-5) cc_compile_by : carnil cc_compile_domain : debian.org cc_compile_date : Sun May 5 14:44:49 UTC 2013 xend_config_format : 4