Linux Virtualization - Dec 2017 - [RFC] virtio-net: help live migrate SR-IOV devices

On Wed, Dec 6, 2017 at 11:28 PM, achiad shochat
<achiad.mellanox at gmail.com> wrote:
> On 5 December 2017 at 21:20, Michael S. Tsirkin <mst at redhat.com>
wrote:
>> On Tue, Dec 05, 2017 at 11:59:17AM +0200, achiad shochat wrote:
>>> Then we'll have a single solution for both netvsc and virtio
(and any
>>> other PV device).
>>> And we could handle the VF DMA dirt issue agnostically.
>>
>> For the record, I won't block patches adding this kist to virtio
>> on the basis that they must be generic. It's not a lot
>> of code, implementation can come first, prettify later.
>
> It's not a lot of code either way.
> So I fail to understand why not to do it right from the beginning.
> For the record...
What isn't a lot of code? If you are talking about the DMA dirtying
then I would have to disagree. The big problem with the DMA is that we
have to mark a page as dirty and non-migratable as soon as it is
mapped for Rx DMA. It isn't until the driver has either unmapped the
page or the device has been disabled that we can then allow the page
to be migrated for being dirty. That ends up being the way we have to
support this if we don't have the bonding solution.

With the bonding solution we could look at doing a lightweight DMA
dirtying which would just require flagging pages as dirty after an
unmap or sync call is performed. However it requires that we shut down
the driver/device before we can complete the migration which means we
have to have the paravirtualized fail-over approach.

As far as indicating that the interfaces are meant to be enslaved I
wonder if we couldn't look at tweaking the PCI layout of the guest and
use that to indicate that a given set of interfaces are meant to be
bonded. For example the VFs are all meant to work as a part of a
multi-function device. What if we were to make virtio-net function 0
of a PCI/PCIe device, and then place any direct assigned VFs that are
meant to be a part of the bond in functions 1-7 of the device? Then it
isn't too far off from the model we have on the host where if the VF
goes away we would expect to see the traffic on the PF that is usually
occupying function 0 of a given device.

On Thu, Dec 07, 2017 at 08:45:33AM -0800, Alexander Duyck
wrote:
> As far as indicating that the interfaces are meant to be enslaved I
> wonder if we couldn't look at tweaking the PCI layout of the guest and
> use that to indicate that a given set of interfaces are meant to be
> bonded. For example the VFs are all meant to work as a part of a
> multi-function device. What if we were to make virtio-net function 0
> of a PCI/PCIe device, and then place any direct assigned VFs that are
> meant to be a part of the bond in functions 1-7 of the device? Then it
> isn't too far off from the model we have on the host where if the VF
> goes away we would expect to see the traffic on the PF that is usually
> occupying function 0 of a given device.
This pretty much precludes removing them with hotplug.

But as long as we are happy to limit this to pci devices,
maybe we should put them behind a pci bridge.

All devices behind a bridge would be assumed to have
the same backend.

QEMU has pci-bridge-seat which we could reuse for this -
just need to build a similar pci express bridge.

-- 
MST