libvirt users - Aug 2020 - support for live migration with PCI passthrough devices

Hi guys,

I'm wondering whether libvirt supports live migration for the VM with PCI
passthrough devices.
or it must be assumed before live migration that all passthrough devices be
unplugged?

If so, all unplugged devices should be manually hot-plugged to the VM after
migration??


Thanks.

On 8/25/20 1:40 PM, Henry lol wrote:
> Hi guys,
> 
> I'm wondering whether libvirt supports live migration for the VM with 
> PCI passthrough devices.
> or it must be assumed before live migration that all passthrough devices 
> be unplugged?
Unfortunately, this is still not supported. The problem is that PCI 
devices themselves are not capable of dumping their internal state and 
restoring from it on destination.

There is a long thread started last month that discuss what the 
interface should look like, but at this point I guess we are still far 
away from it:

https://www.redhat.com/archives/libvir-list/2020-July/msg00675.html
> 
> If so, all unplugged devices should be manually hot-plugged to the VM 
> after migration??
This is the usual mode of operation, yes.

Michal

On 8/25/20 8:56 AM, Michal Privoznik wrote:
> On 8/25/20 1:40 PM, Henry lol wrote:
>> Hi guys,
>>
>> I'm wondering whether libvirt supports live migration for the VM
with
>> PCI passthrough devices.
>> or it must be assumed before live migration that all passthrough 
>> devices be unplugged?
> 
> Unfortunately, this is still not supported. The problem is that PCI 
> devices themselves are not capable of dumping their internal state and 
> restoring from it on destination.
> 
> There is a long thread started last month that discuss what the 
> interface should look like, but at this point I guess we are still far 
> away from it:
> 
> https://www.redhat.com/archives/libvir-list/2020-July/msg00675.html
> 
>>
>> If so, all unplugged devices should be manually hot-plugged to the VM 
>> after migration??
> 
> This is the usual mode of operation, yes.
There is one exception to this - if the device to be migrated is the VF 
of an SRIOV-capable network card, then there is machinery in qemu to 
automatically unplug the device before migration starts, and then 
automatically plug in a new similar device on the other end after the 
guest has started there.

This all works in concert with a virtio-net emulated network device that 
is bonded together in the guest (linux only) to provide a single network 
device in the guest - when the VF is visible and working, it will be 
preferred for all traffic, but if the VF is missing then the virtio-net 
emulated device will be used.

The result is that under normal operation the vfio-assigned VF is used 
for all network traffic, but during migration when the VF has been 
unplugged, the virtio-net device is used, so there is no disruption in 
network traffic, just a bit of degraded performance during migration.

Here is a description of how this is configured in a libvirt domain 
definition:

 
https://www.libvirt.org/formatdomain.html#teaming-a-virtio-hostdev-nic-pair


This is all fairly new, and unfortunately doesn't work properly for all 
SRIOV cards.

On Tue, Aug 25, 2020 at 02:56:49PM +0200, Michal Privoznik
wrote:
> On 8/25/20 1:40 PM, Henry lol wrote:
> > Hi guys,
> >
> > I'm wondering whether libvirt supports live migration for the VM
with
> > PCI passthrough devices.
> > or it must be assumed before live migration that all passthrough
devices
> > be unplugged?
>
> Unfortunately, this is still not supported. The problem is that PCI devices
> themselves are not capable of dumping their internal state and restoring
> from it on destination.
>
> There is a long thread started last month that discuss what the interface
> should look like, but at this point I guess we are still far away from it:
>
> https://www.redhat.com/archives/libvir-list/2020-July/msg00675.html
Just a little note, even with such an interface in place it will still take
time for vendors of conventional physical devices to adopt it, so realistically
the main focus here are mediated devices and SRIOV-capable devices.

Erik