Doug Goldstein
2013-Jan-23 21:45 UTC
[libvirt-users] VMs fail to start with NUMA configuration
I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + qemu
1.2.2 applied on top plus a number of stability patches). Having issue
where my VMs fail to start with the following message:
kvm_init_vcpu failed: Cannot allocate memory
Following the instructions at
http://libvirt.org/formatdomain.html#elementsNUMATuning I've added the
following to my VCPU configuration:
<vcpu placement='auto'>2</vcpu>
Which libvirt expands out as I'd expect per the documentation to:
<vcpu placement='auto'>2</vcpu>
<numatune>
<memory tune='strict' placement='auto'/>
</numatune>
However, the VMs won't start and the system is no low on memory.
# numactl --hardware
available: 8 nodes (0-7)
node 0 cpus: 0 4 8 12 16 20 24 28
node 0 size: 16374 MB
node 0 free: 11899 MB
node 1 cpus: 32 36 40 44 48 52 56 60
node 1 size: 16384 MB
node 1 free: 15318 MB
node 2 cpus: 2 6 10 14 18 22 26 30
node 2 size: 16384 MB
node 2 free: 15766 MB
node 3 cpus: 34 38 42 46 50 54 58 62
node 3 size: 16384 MB
node 3 free: 15347 MB
node 4 cpus: 3 7 11 15 19 23 27 31
node 4 size: 16384 MB
node 4 free: 15041 MB
node 5 cpus: 35 39 43 47 51 55 59 63
node 5 size: 16384 MB
node 5 free: 15202 MB
node 6 cpus: 1 5 9 13 17 21 25 29
node 6 size: 16384 MB
node 6 free: 15197 MB
node 7 cpus: 33 37 41 45 49 53 57 61
node 7 size: 16368 MB
node 7 free: 15669 MB
The system has 4 Opteron 6272 which add up to a total of 64 cores, 16
cores per socket. These are the CPUs that Dan B noticed issues with in
the past regarding topology but I thought this was resolved. But the
capabilities are posted below, you'll notice the topology is
incorrect.
# virsh capabilities
<capabilities>
<host>
<uuid>44454c4c-5300-1038-8031-c4c04f545331</uuid>
<cpu>
<arch>x86_64</arch>
<model>Opteron_G4</model>
<vendor>AMD</vendor>
<topology sockets='1' cores='8'
threads='2'/>
<feature name='nodeid_msr'/>
<feature name='wdt'/>
<feature name='skinit'/>
<feature name='ibs'/>
<feature name='osvw'/>
<feature name='cr8legacy'/>
<feature name='extapic'/>
<feature name='cmp_legacy'/>
<feature name='fxsr_opt'/>
<feature name='mmxext'/>
<feature name='osxsave'/>
<feature name='monitor'/>
<feature name='ht'/>
<feature name='vme'/>
</cpu>
<power_management>
<suspend_disk/>
</power_management>
<migration_features>
<live/>
<uri_transports>
<uri_transport>tcp</uri_transport>
</uri_transports>
</migration_features>
<topology>
<cells num='8'>
<cell id='0'>
<cpus num='8'>
<cpu id='0'/>
<cpu id='4'/>
<cpu id='8'/>
<cpu id='12'/>
<cpu id='16'/>
<cpu id='20'/>
<cpu id='24'/>
<cpu id='28'/>
</cpus>
</cell>
<cell id='1'>
<cpus num='8'>
<cpu id='32'/>
<cpu id='36'/>
<cpu id='40'/>
<cpu id='44'/>
<cpu id='48'/>
<cpu id='52'/>
<cpu id='56'/>
<cpu id='60'/>
</cpus>
</cell>
<cell id='2'>
<cpus num='8'>
<cpu id='2'/>
<cpu id='6'/>
<cpu id='10'/>
<cpu id='14'/>
<cpu id='18'/>
<cpu id='22'/>
<cpu id='26'/>
<cpu id='30'/>
</cpus>
</cell>
<cell id='3'>
<cpus num='8'>
<cpu id='34'/>
<cpu id='38'/>
<cpu id='42'/>
<cpu id='46'/>
<cpu id='50'/>
<cpu id='54'/>
<cpu id='58'/>
<cpu id='62'/>
</cpus>
</cell>
<cell id='4'>
<cpus num='8'>
<cpu id='3'/>
<cpu id='7'/>
<cpu id='11'/>
<cpu id='15'/>
<cpu id='19'/>
<cpu id='23'/>
<cpu id='27'/>
<cpu id='31'/>
</cpus>
</cell>
<cell id='5'>
<cpus num='8'>
<cpu id='35'/>
<cpu id='39'/>
<cpu id='43'/>
<cpu id='47'/>
<cpu id='51'/>
<cpu id='55'/>
<cpu id='59'/>
<cpu id='63'/>
</cpus>
</cell>
<cell id='6'>
<cpus num='8'>
<cpu id='1'/>
<cpu id='5'/>
<cpu id='9'/>
<cpu id='13'/>
<cpu id='17'/>
<cpu id='21'/>
<cpu id='25'/>
<cpu id='29'/>
</cpus>
</cell>
<cell id='7'>
<cpus num='8'>
<cpu id='33'/>
<cpu id='37'/>
<cpu id='41'/>
<cpu id='45'/>
<cpu id='49'/>
<cpu id='53'/>
<cpu id='57'/>
<cpu id='61'/>
</cpus>
</cell>
</cells>
</topology>
<secmodel>
<model>none</model>
<doi>0</doi>
</secmodel>
<secmodel>
<model>dac</model>
<doi>0</doi>
</secmodel>
</host>
<guest>
<os_type>hvm</os_type>
<arch name='i686'>
<wordsize>32</wordsize>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<machine>pc-1.2</machine>
<machine canonical='pc-1.2'>pc</machine>
<machine>pc-1.1</machine>
<machine>pc-1.0</machine>
<machine>pc-0.15</machine>
<machine>pc-0.14</machine>
<machine>pc-0.13</machine>
<machine>pc-0.12</machine>
<machine>pc-0.11</machine>
<machine>pc-0.10</machine>
<machine>isapc</machine>
<machine>none</machine>
<domain type='qemu'>
</domain>
<domain type='kvm'>
<emulator>/usr/bin/qemu-kvm</emulator>
<machine>pc-1.2</machine>
<machine canonical='pc-1.2'>pc</machine>
<machine>pc-1.1</machine>
<machine>pc-1.0</machine>
<machine>pc-0.15</machine>
<machine>pc-0.14</machine>
<machine>pc-0.13</machine>
<machine>pc-0.12</machine>
<machine>pc-0.11</machine>
<machine>pc-0.10</machine>
<machine>isapc</machine>
<machine>none</machine>
</domain>
</arch>
<features>
<cpuselection/>
<deviceboot/>
<pae/>
<nonpae/>
<acpi default='on' toggle='yes'/>
<apic default='on' toggle='no'/>
</features>
</guest>
<guest>
<os_type>hvm</os_type>
<arch name='x86_64'>
<wordsize>64</wordsize>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<machine>pc-1.2</machine>
<machine canonical='pc-1.2'>pc</machine>
<machine>pc-1.1</machine>
<machine>pc-1.0</machine>
<machine>pc-0.15</machine>
<machine>pc-0.14</machine>
<machine>pc-0.13</machine>
<machine>pc-0.12</machine>
<machine>pc-0.11</machine>
<machine>pc-0.10</machine>
<machine>isapc</machine>
<machine>none</machine>
<domain type='qemu'>
</domain>
<domain type='kvm'>
<emulator>/usr/bin/qemu-kvm</emulator>
<machine>pc-1.2</machine>
<machine canonical='pc-1.2'>pc</machine>
<machine>pc-1.1</machine>
<machine>pc-1.0</machine>
<machine>pc-0.15</machine>
<machine>pc-0.14</machine>
<machine>pc-0.13</machine>
<machine>pc-0.12</machine>
<machine>pc-0.11</machine>
<machine>pc-0.10</machine>
<machine>isapc</machine>
<machine>none</machine>
</domain>
</arch>
<features>
<cpuselection/>
<deviceboot/>
<acpi default='on' toggle='yes'/>
<apic default='on' toggle='no'/>
</features>
</guest>
</capabilities>
Any suggestions are appreciated.
--
Doug Goldstein
Doug Goldstein
2013-Jan-24 04:11 UTC
[libvirt-users] VMs fail to start with NUMA configuration
On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein <cardoe at gentoo.org> wrote:> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + qemu > 1.2.2 applied on top plus a number of stability patches). Having issue > where my VMs fail to start with the following message: > > kvm_init_vcpu failed: Cannot allocate memoryI've tried libvirt-1.0.1 on this machine and had the same results. The capabilities have been fixed on that machine however and now display: <topology sockets='1' cores='64' threads='1'/> I also forgot to mention that LIBVIRT_DEBUG=1 and running the daemon with --verbose didn't have more details than what I originally posted. -- Doug Goldstein