Re: support for live migration with PCI passthrough devices

Laine Stump
2020-08-25T16:08:14Z

On 8/25/20 8:56 AM, Michal Privoznik wrote:
> On 8/25/20 1:40 PM, Henry lol wrote:
>> Hi guys,
>>
>> I'm wondering whether libvirt supports live migration for the VM
with
>> PCI passthrough devices.
>> or it must be assumed before live migration that all passthrough
>> devices be unplugged?
>
> Unfortunately, this is still not supported. The problem is that PCI
> devices themselves are not capable of dumping their internal state and
> restoring from it on destination.
>
> There is a long thread started last month that discuss what the
> interface should look like, but at this point I guess we are still far
> away from it:
>
> https://www.redhat.com/archives/libvir-list/2020-July/msg00675.html
>
>>
>> If so, all unplugged devices should be manually hot-plugged to the VM
>> after migration??
>
> This is the usual mode of operation, yes.

There is one exception to this - if the device to be migrated is the VF
of an SRIOV-capable network card, then there is machinery in qemu to
automatically unplug the device before migration starts, and then
automatically plug in a new similar device on the other end after the
guest has started there.

This all works in concert with a virtio-net emulated network device that
is bonded together in the guest (linux only) to provide a single network
device in the guest - when the VF is visible and working, it will be
preferred for all traffic, but if the VF is missing then the virtio-net
emulated device will be used.

The result is that under normal operation the vfio-assigned VF is used
for all network traffic, but during migration when the VF has been
unplugged, the virtio-net device is used, so there is no disruption in
network traffic, just a bit of degraded performance during migration.

Here is a description of how this is configured in a libvirt domain
definition:


https://www.libvirt.org/formatdomain.html#teaming-a-virtio-hostdev-nic-pair


This is all fairly new, and unfortunately doesn't work properly for all
SRIOV cards.