libvirt users - Feb 2014 - Help? Running into problems with migrateToURI2() and virDomainDefCheckABIStability()

Hi,

We've run into a problem with libvirt 1.1.2 and are looking for some 
comments on whether this is a bug or design intent.

We're trying to use migrateToURI() but we're using a few things 
(numatune, vcpu mask, etc.) that may need adjustment during the 
migration.  We found that migrateToURI2() mostly works if we use XML 
created by copying the domain XML from the running instance and 
modifying the appropriate sections.

The problem that we're seeing is that the serial console checking in 
libvirt/src/conf/domain_conf.c::virDomainDefCheckABIStability() is 
failing even though we haven't touched the serial console XML:

2014-02-10 19:41:07.196 1264 ERROR nova.virt.libvirt.driver [-] 
[instance: 42a841a8-034f-4565-937e-adb396a2ec48] Live Migration failure: 
unsupported configuration: Target domain console count 1 does not match 
source 0


On occasion it passes if the devices/serial/source element happens to be 
the same after migration, but this seems to be a fluke.

If we remove the devices/serial/source element in the XML it seems to 
make the serial console check work on the first migration, but upon 
doing subsequent migration it fails with a console count error:

2014-02-07 16:11:32.459 28759 ERROR nova.virt.libvirt.driver [-] 
[instance: 1d7c470d-4337-450a-8a40-d0fb375ca2c1] Live Migration failure: 
unsupported configuration: Target domain console count 0 does not match 
source 1

Are we doing something wrong?  Is there an issue with the console check 
code?  Is there a better way to do what we're trying to do?

Any help would be appreciated...

Thanks,
Chris

On 02/10/2014 06:46 PM, Chris Friesen wrote:
> Hi,
> 
> We've run into a problem with libvirt 1.1.2 and are looking for some
comments
> on whether this is a bug or design intent.
> 
> We're trying to use migrateToURI() but we're using a few things
(numatune,
> vcpu mask, etc.) that may need adjustment during the migration.  We found
that
> migrateToURI2() mostly works if we use XML created by copying the domain
XML
> from the running instance and modifying the appropriate sections.
> 
> The problem that we're seeing is that the serial console checking in
> libvirt/src/conf/domain_conf.c::virDomainDefCheckABIStability() is failing
> even though we haven't touched the serial console XML:
> 
> 2014-02-10 19:41:07.196 1264 ERROR nova.virt.libvirt.driver [-] [instance:
> 42a841a8-034f-4565-937e-adb396a2ec48] Live Migration failure: unsupported
> configuration: Target domain console count 1 does not match source 0
> 
> 
> On occasion it passes if the devices/serial/source element happens to be
the
> same after migration, but this seems to be a fluke.
> 
> If we remove the devices/serial/source element in the XML it seems to make
the
> serial console check work on the first migration, but upon doing subsequent
> migration it fails with a console count error:
> 
> 2014-02-07 16:11:32.459 28759 ERROR nova.virt.libvirt.driver [-] [instance:
> 1d7c470d-4337-450a-8a40-d0fb375ca2c1] Live Migration failure: unsupported
> configuration: Target domain console count 0 does not match source 1
> 
> Are we doing something wrong?  Is there an issue with the console check
code?
> Is there a better way to do what we're trying to do?
> 
> Any help would be appreciated...
> 
I think we will have to see the XML your passing before and after edit to
figure out what's going wrong. Also, showing all the arguments you are
invoking the migration APIs with.

This could have to do with some back compatible stuff we do with <console>
handling: basically, <console type='pty'/> implies <serial
type='pty'/> and
vice versa. So if one is available in the XML, libvirt will force add another.
If you clear out the <serial> bit, then pass that to libvirt, it will just
fill it in with the matching console details. So you might need to remove
<serial> and the matching <console> piece.

- Cole

On 02/11/2014 04:45 PM, Cole Robinson wrote:
> On 02/10/2014 06:46 PM, Chris Friesen wrote:
>> Hi,
>>
>> We've run into a problem with libvirt 1.1.2 and are looking for
some comments
>> on whether this is a bug or design intent.
>>
>> We're trying to use migrateToURI() but we're using a few things
(numatune,
>> vcpu mask, etc.) that may need adjustment during the migration.  We
found that
>> migrateToURI2() mostly works if we use XML created by copying the
domain XML
>> from the running instance and modifying the appropriate sections.
>>
>> The problem that we're seeing is that the serial console checking
in
>> libvirt/src/conf/domain_conf.c::virDomainDefCheckABIStability() is
failing
>> even though we haven't touched the serial console XML:
> I think we will have to see the XML your passing before and after edit to
> figure out what's going wrong. Also, showing all the arguments you are
> invoking the migration APIs with.

Okay, this is going to be a long message.  :)

We're calling it as:

dom.migrateToURI2(duri, muri, xml_str, logical_sum, None,
CONF.live_migration_bandwidth)

Here are the supplied arguments:
duri = u'qemu+tcp://compute-0/system'
muri = u'tcp://compute-0'
flaglist = ['VIR_MIGRATE_UNDEFINE_SOURCE',
'VIR_MIGRATE_PEER2PEER', 'VIR_MIGRATE_LIVE']
logical_sum = 19,  (logical OR of flaglist)
CONF.live_migration_bandwidth = 0

Grabbing original XML prior to migration:

dom_xml = etree.fromstring(dom.XMLDesc(0))
xml_str = etree.tostring(dom_xml)

xml_str (BEFORE) <domain type="qemu" id="2">
  <name>instance-00000006</name>
  <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid>
  <dpdk>
    <process type="secondary"/>
    <file prefix="vs"/>
    <cpu list="0"/>
    <memory channels="4"/>
  </dpdk>
  <memory unit="KiB">262144</memory>
  <currentMemory unit="KiB">262144</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static" cpuset="1-2">2</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="1"/>
    <vcpupin vcpu="1" cpuset="2"/>
    <emulatorpin cpuset="1-2"/>
  </cputune>
  <numatune>
    <memory mode="preferred" nodeset="0"/>
  </numatune>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">OpenStack
Foundation</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2013.2.2</entry>
      <entry
name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry>
      <entry
name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry>
    </system>
  </sysinfo>
  <os>
    <type arch="x86_64"
machine="pc-i440fx-1.4">hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset="utc"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"
cache="none"/>
      <source
file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/>
      <target dev="vda" bus="virtio"/>
      <alias name="virtio-disk0"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x05" function="0x0"/>
    </disk>
    <controller type="usb" index="0">
      <alias name="usb0"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="0"
model="pci-root">
      <alias name="pci.0"/>
    </controller>
    <interface type="vswitch">
      <mac address="fa:16:3e:fb:34:6b"/>
      <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/>
      <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/>
      <model type="e1000"/>
      <alias name="net0"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x03" function="0x0"/>
    </interface>
    <interface type="vswitch">
      <mac address="fa:16:3e:d5:b9:50"/>
      <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/>
      <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/>
      <model type="e1000"/>
      <alias name="net1"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x04" function="0x0"/>
    </interface>
    <serial type="file">
      <source
path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source
path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>
    <input type="tablet" bus="usb">
      <alias name="input0"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <graphics type="vnc" port="5900"
autoport="yes" listen="0.0.0.0" keymap="en-us">
      <listen type="address" address="0.0.0.0"/>
    </graphics>
    <video>
      <model type="cirrus" vram="9216"
heads="1"/>
      <alias name="video0"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x02" function="0x0"/>
    </video>
    <memballoon model="virtio">
      <alias name="balloon0"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x06" function="0x0"/>
    </memballoon>
  </devices>
  <seclabel type="none"/>
</domain>



This is the modified XML, specifically updating vcpu, cputune, numatune.
In this case they actually have the same values as above but there are
whitespace differences:


xml_str (MODIFIED) <domain type="qemu" id="2">
  <name>instance-00000006</name>
  <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid>
  <dpdk>
    <process type="secondary"/>
    <file prefix="vs"/>
    <cpu list="0"/>
    <memory channels="4"/>
  </dpdk>
  <memory unit="KiB">262144</memory>
  <currentMemory unit="KiB">262144</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static" cpuset="1,2">2</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="1"/><vcpupin
vcpu="1" cpuset="2"/><emulatorpin
cpuset="1,2"/></cputune>
  <numatune>
    <memory mode="preferred"
nodeset="0"/></numatune>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">OpenStack
Foundation</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2013.2.2</entry>
      <entry
name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry>
      <entry
name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry>
    </system>
  </sysinfo>
  <os>
    <type arch="x86_64"
machine="pc-i440fx-1.4">hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset="utc"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"
cache="none"/>
      <source
file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/>
      <target dev="vda" bus="virtio"/>
      <alias name="virtio-disk0"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x05" function="0x0"/>
    </disk>
    <controller type="usb" index="0">
      <alias name="usb0"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="0"
model="pci-root">
      <alias name="pci.0"/>
    </controller>
    <interface type="vswitch">
      <mac address="fa:16:3e:fb:34:6b"/>
      <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/>
      <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/>
      <model type="e1000"/>
      <alias name="net0"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x03" function="0x0"/>
    </interface>
    <interface type="vswitch">
      <mac address="fa:16:3e:d5:b9:50"/>
      <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/>
      <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/>
      <model type="e1000"/>
      <alias name="net1"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x04" function="0x0"/>
    </interface>
    <serial type="file">
      <source
path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source
path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>
    <input type="tablet" bus="usb">
      <alias name="input0"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <graphics type="vnc" port="5900"
autoport="yes" listen="0.0.0.0" keymap="en-us">
      <listen type="address" address="0.0.0.0"/>
    </graphics>
    <video>
      <model type="cirrus" vram="9216"
heads="1"/>
      <alias name="video0"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x02" function="0x0"/>
    </video>
    <memballoon model="virtio">
      <alias name="balloon0"/>
      <address type="pci" domain="0x0000"
bus="0x00" slot="0x06" function="0x0"/>
    </memballoon>
  </devices>
  <seclabel type="none"/>
</domain>



Calling migrateToURI2 then gives the following error :
2014-02-12 03:37:17.738 1251 ERROR nova.virt.libvirt.driver [-] [instance:
2e13869b-edc0-40dc-92e3-1f2ffde60cce] Live Migration failure: unsupported
configuration: Target domain console count 1 does not match source 0


If we remove the this section :
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>

Then we get the following error :
2014-02-12 04:19:12.689 1278 ERROR nova.virt.libvirt.driver [-] [instance:
90335c32-a7ae-4c99-8e5a-cd177f6b3c22] Live Migration failure: unsupported
configuration: Target domain serial port count 1 does not match source 2

If we remove just the following line:
      <source path="/dev/pts/1"/>

so the modified portion looks like this,
    <serial type="file">
      <source
path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source
path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>
    <input type="tablet" bus="usb">

Then we get this error :

2014-02-12 04:29:06.402 5175 ERROR nova.virt.libvirt.driver [-] [instance:
00b99f94-436e-406b-8459-83e77758f154] Live Migration failure: unsupported
configuration: Target domain console count 1 does not match source 0

If we modify it with empty <serial type='pty'> element like this:

    <serial type="file">
      <source
path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      </serial>
    <console type="file">
      <source
path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>

Then we get this error:

2014-02-12 04:36:45.686 5358 ERROR nova.virt.libvirt.driver [-] [instance:
b1103349-8c9f-4315-9144-03fc22f09be1] Live Migration failure: unsupported
configuration: Target domain console count 1 does not match source 0

If we modify it with empty <serial type='pty'> element and also
remove
<console type='file'> section completely like this :

    <serial type="file">
      <source
path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
    </serial>

Then this successfully migrates!

However, it then fails on subsequent migration back to the original host.  Prior
to migrating
back the domain XML serial/console section looks like:

    <serial type="file">
      <source
path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
      <source path="/dev/pts/1"/>
      <target port="1"/>
      <alias name="serial1"/>
    </serial>
    <console type="file">
      <source
path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target type="serial" port="0"/>
      <alias name="serial0"/>
    </console>

After modifying empty <serial type='pty'> element and removing the
<console type='file'> section
it looks like this :

    <serial type="file">
      <source
path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/>
      <target port="0"/>
      <alias name="serial0"/>
    </serial>
    <serial type="pty">
    </serial>

But this give the following error when migrating back:

2014-02-12 06:00:19.831 7125 ERROR nova.virt.libvirt.driver [-] [instance:
8cccb4ad-2abc-4513-b8fd-bfb811301677] Live Migration failure: unsupported
configuration: Target domain console count 0 does not match source 1



Chris