Hi guys, I'm wondering whether libvirt supports live migration for the VM with PCI passthrough devices. or it must be assumed before live migration that all passthrough devices be unplugged? If so, all unplugged devices should be manually hot-plugged to the VM after migration?? Thanks.
Michal Privoznik
2020-Aug-25 12:56 UTC
Re: support for live migration with PCI passthrough devices
On 8/25/20 1:40 PM, Henry lol wrote:> Hi guys, > > I'm wondering whether libvirt supports live migration for the VM with > PCI passthrough devices. > or it must be assumed before live migration that all passthrough devices > be unplugged?Unfortunately, this is still not supported. The problem is that PCI devices themselves are not capable of dumping their internal state and restoring from it on destination. There is a long thread started last month that discuss what the interface should look like, but at this point I guess we are still far away from it: https://www.redhat.com/archives/libvir-list/2020-July/msg00675.html> > If so, all unplugged devices should be manually hot-plugged to the VM > after migration??This is the usual mode of operation, yes. Michal
Laine Stump
2020-Aug-25 16:08 UTC
Re: support for live migration with PCI passthrough devices
On 8/25/20 8:56 AM, Michal Privoznik wrote:> On 8/25/20 1:40 PM, Henry lol wrote: >> Hi guys, >> >> I'm wondering whether libvirt supports live migration for the VM with >> PCI passthrough devices. >> or it must be assumed before live migration that all passthrough >> devices be unplugged? > > Unfortunately, this is still not supported. The problem is that PCI > devices themselves are not capable of dumping their internal state and > restoring from it on destination. > > There is a long thread started last month that discuss what the > interface should look like, but at this point I guess we are still far > away from it: > > https://www.redhat.com/archives/libvir-list/2020-July/msg00675.html > >> >> If so, all unplugged devices should be manually hot-plugged to the VM >> after migration?? > > This is the usual mode of operation, yes.There is one exception to this - if the device to be migrated is the VF of an SRIOV-capable network card, then there is machinery in qemu to automatically unplug the device before migration starts, and then automatically plug in a new similar device on the other end after the guest has started there. This all works in concert with a virtio-net emulated network device that is bonded together in the guest (linux only) to provide a single network device in the guest - when the VF is visible and working, it will be preferred for all traffic, but if the VF is missing then the virtio-net emulated device will be used. The result is that under normal operation the vfio-assigned VF is used for all network traffic, but during migration when the VF has been unplugged, the virtio-net device is used, so there is no disruption in network traffic, just a bit of degraded performance during migration. Here is a description of how this is configured in a libvirt domain definition: https://www.libvirt.org/formatdomain.html#teaming-a-virtio-hostdev-nic-pair This is all fairly new, and unfortunately doesn't work properly for all SRIOV cards.
Erik Skultety
2020-Aug-26 06:27 UTC
Re: support for live migration with PCI passthrough devices
On Tue, Aug 25, 2020 at 02:56:49PM +0200, Michal Privoznik wrote:> On 8/25/20 1:40 PM, Henry lol wrote: > > Hi guys, > > > > I'm wondering whether libvirt supports live migration for the VM with > > PCI passthrough devices. > > or it must be assumed before live migration that all passthrough devices > > be unplugged? > > Unfortunately, this is still not supported. The problem is that PCI devices > themselves are not capable of dumping their internal state and restoring > from it on destination. > > There is a long thread started last month that discuss what the interface > should look like, but at this point I guess we are still far away from it: > > https://www.redhat.com/archives/libvir-list/2020-July/msg00675.htmlJust a little note, even with such an interface in place it will still take time for vendors of conventional physical devices to adopt it, so realistically the main focus here are mediated devices and SRIOV-capable devices. Erik