Re: [libvirt-users] NUMA VM and assigning interfaces

Amir Shehata
2016-11-21T21:48:29Z

Laine,

Thank you for your response.

I did use the numa element instead of using qemu commandline and that
worked:

 16   <cpu mode='custom' match='exact'>
...
 34     <numa>
 35       <cell id='0' cpus='0-1' memory='1048576'
unit='KiB'/>
 36       <cell id='1' cpus='2-3' memory='1048576'
unit='KiB'/>
 37     </numa>
 38   </cpu>

However, whenever I try to use the pci-expander-bus I get:
error: XML document failed to validate against schema: Unable to validate
doc against /usr/share/libvirt/schemas/domain.rng
Extra element devices in interleave
Element domain failed to validate content

I tried the following

 93     <controller type='pci' index='1'
model='pci-expander-bus'>
 94       <address type='pci' domain='0x0000'
bus='0x00' slot='0x0b'
function='0x0'/>
 95     </controller>

and

 93     <controller type='pci' index='1'
model='pci-expander-bus'>
 94       <target busNr='254'>
 95         <node>1</node>
 96       </target>
 97       <address type='pci' domain='0x0000'
bus='0x00' slot='0x0b'
function='0x0'/>
 98     </controller>


I got the syntax from:
https://www.redhat.com/archives/libvir-list/2016-March/msg01224.html

Just to give you a bit more info. This is running on:
Linux ashehata-DellPE 3.19.0-51-generic #58-Ubuntu SMP Fri Feb 26 21:22:26
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

below is a dump of the working XML file I'm using

thanks
amir


<domain type='kvm'
xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>MR_test01</name>
  <uuid>59eafc3c-9d9e-4dab-bac9-48003048c6dc</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64'
machine='pc-i440fx-utopic'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>core2duo</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='cx16'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='acpi'/>
    <numa>
      <cell id='0' cpus='0-1' memory='1048576'
unit='KiB'/>
      <cell id='1' cpus='2-3' memory='1048576'
unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source
file='/media/ashehata/Lustre03/kvm-linux2/disk1.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x09'
function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source
file='/media/ashehata/Lustre03/kvm-linux2/disk4.qcow2'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x0b'
function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0'
target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'
model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x08'
function='0x7'/>
    </controller>
    <controller type='usb' index='0'
model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x08'
function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0'
model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x08'
function='0x1'/>
    </controller>
    <controller type='usb' index='0'
model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x08'
function='0x2'/>
    </controller>
    <controller type='pci' index='0'
model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x07'
function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:da:1b:af'/>

      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:84:0a:66'/>

      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:17:9d:cd'/>

      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0'
bus='0' port='1'/>
    </channel>



    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x06'
function='0x0'/>
    </sound>

      <model type='qxl' ram='65536' vram='65536'
vgamem='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>

    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x0a'
function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-gdb'/>
    <qemu:arg value='tcp::1201'/>
  </qemu:commandline>
</domain>


On 21 November 2016 at 11:40, Laine Stump <laine@laine.org> wrote:

> On 11/21/2016 12:34 PM, Amir Shehata wrote:
>
>> Hello,
>>
>> Hope all is well.
>>
>> I've been looking at how I can create a virtual machine which is
NUMA
>> capable. I was able to do that by:
>>
>> 140   <qemu:commandline>
>> 143     <qemu:arg value='-numa'/>
>> 144     <qemu:arg value='node'/>
>> 145     <qemu:arg value='-numa'/>
>> 146     <qemu:arg value='node'/>
>> 147   </qemu:commandline>
>>
>
> qemu commandline passthrough should only be used for qemu features that
> libvirt doesn't support directly (and even then it's only intended
as a
> temporary stopgap until the feature gains official libvirt support). NUMA
> has been supported by libvirt for quite a long time.
>
> If you open http://www.libvirt.org/formatdomain.html and search for
> "numa" you'll find descriptions of the configuration for
defining NUMA
> memory regions, and assigning particular CPUs to particular nodes. The is
> also a small bit about defining a pci-expander-bus (a type of pci
> controller) and tying it to a particular NUMA node - this is how you can
> place  a guest device on a particular NUMA node; just assign the device a
> PCI address that has the given pci-expander-bus' "index" as
its "bus"
> attribute.
>
>
>> This creates two NUMA nodes.
>>
>> Now I would like to assign my virtual network interfaces, of which I
have
>> three, to the different NUMA nodes on the VM.
>>
>> Anyone know how I can do that? Any help/pointers would be much
>> appreciated.
>>
>> I'm trying to create a VM that will enable me to test a NUMA based
>> feature.
>>
>
> The one thing I'm unclear about is whether the guest can be setup with
> NUMA nodes when the host itself doesn't have multiple NUMA nodes...
>
>