On Thu, Dec 17, 2020 at 05:02:49PM +0800, Jason Wang
wrote:>
> On 2020/12/17 ??3:58, Michael S. Tsirkin wrote:
> > On Thu, Dec 17, 2020 at 11:30:18AM +0800, Jason Wang wrote:
> > > On 2020/12/16 ??5:47, Michael S. Tsirkin wrote:
> > > > On Wed, Dec 16, 2020 at 02:47:57PM +0800, Jason Wang wrote:
> > > > > Hi All:
> > > > >
> > > > > This series tries to add the support for control
virtqueue in vDPA.
> > > > >
> > > > > Control virtqueue is used by networking device for
accepting various
> > > > > commands from the driver. It's a must to support
multiqueue and other
> > > > > configurations.
> > > > >
> > > > > When used by vhost-vDPA bus driver for VM, the control
virtqueue
> > > > > should be shadowed via userspace VMM (Qemu) instead of
being assigned
> > > > > directly to Guest. This is because Qemu needs to know
the device state
> > > > > in order to start and stop device correctly (e.g for
Live Migration).
> > > > >
> > > > > This requies to isolate the memory mapping for control
virtqueue
> > > > > presented by vhost-vDPA to prevent guest from accesing
it directly.
> > > > > To achieve this, vDPA introduce two new abstractions:
> > > > >
> > > > > - address space: identified through address space id
(ASID) and a set
> > > > > of memory mapping in maintained
> > > > > - virtqueue group: the minimal set of virtqueues that
must share an
> > > > > address space
> > > > How will this support the pretty common case where control
vq
> > > > is programmed by the kernel through the PF, and others by
the VFs?
> > >
> > > In this case, the VF parent need to provide a software control vq
and decode
> > > the command then send them to VF.
> >
> > But how does that tie to the address space infrastructure?
>
>
> In this case, address space is not a must.
That's ok, problem is I don't see how address space is going
to work in this case at all.
There's no address space there that userspace/guest can control.
> But the idea is to make control
> vq works for all types of hardware:
>
> 1) control virtqueue is implemented via VF/PF communication
> 2) control virtqueue is implemented by VF but not through DMA
> 3) control virtqueue is implemented by VF DMA, it could be either a
hardware
> control virtqueue or other type of DMA
>
> The address space is a must for 3) to work and can work for both 1) and 2).
>
>
> >
> >
> >
> > > >
> > > > I actually thought the way to support it is by exposing
> > > > something like an "inject buffers" API which sends
data to a given VQ.
> > > > Maybe an ioctl, and maybe down the road uio ring can support
batching
> > > > these ....
> > >
> > > So the virtuqueue allows the request to be processed
asynchronously (e.g
> > > driver may choose to use interrupt for control vq). This means we
need to
> > > support that in uAPI level.
> > I don't think we need to make it async, just a regular ioctl will
do.
> > In fact no guest uses the asynchronous property.
>
>
> It was not forbidden by the spec then we need to support that. E.g we can
> not assume driver doesn't assign interrupt for cvq.
>
>
> >
> >
> > > And if we manage to do that, it's just another
> > > type of virtqueue.
> > >
> > > For virtio-vDPA, this also means the extensions for queue
processing which
> > > is a functional duplication.
> > I don't see why, just send it to the actual control vq :)
>
>
> But in the case you've pointed out, there's no hardware control vq
in fact.
>
>
> >
> > > Using what proposed in this series, we don't
> > > need any changes for kernel virtio drivers.
> > >
> > > What's more important, this series could be used for future
features that
> > > requires DMA isolation between virtqueues:
> > >
> > > - report dirty pages via virtqueue
> > > - sub function level device slicing
> >
> > I agree these are nice to have, but I am not sure basic control vq
must
> > be tied to that.
>
>
> If the control virtqueue is implemented via DMA through VF, it looks like a
> must.
>
> Thanks
>
>
> >
> > > ...
> > >
> > > Thanks
> > >
> > >
> > > >
> > > > > Device needs to advertise the following attributes to
vDPA:
> > > > >
> > > > > - the number of address spaces supported in the device
> > > > > - the number of virtqueue groups supported in the
device
> > > > > - the mappings from a specific virtqueue to its
virtqueue groups
> > > > >
> > > > > The mappings from virtqueue to virtqueue groups is
fixed and defined
> > > > > by vDPA device driver. E.g:
> > > > >
> > > > > - For the device that has hardware ASID support, it can
simply
> > > > > advertise a per virtqueue virtqueue group.
> > > > > - For the device that does not have hardware ASID
support, it can
> > > > > simply advertise a single virtqueue group that
contains all
> > > > > virtqueues. Or if it wants a software emulated
control virtqueue, it
> > > > > can advertise two virtqueue groups, one is for cvq,
another is for
> > > > > the rest virtqueues.
> > > > >
> > > > > vDPA also allow to change the association between
virtqueue group and
> > > > > address space. So in the case of control virtqueue,
userspace
> > > > > VMM(Qemu) may use a dedicated address space for the
control virtqueue
> > > > > group to isolate the memory mapping.
> > > > >
> > > > > The vhost/vhost-vDPA is also extend for the userspace
to:
> > > > >
> > > > > - query the number of virtqueue groups and address
spaces supported by
> > > > > the device
> > > > > - query the virtqueue group for a specific virtqueue
> > > > > - assocaite a virtqueue group with an address space
> > > > > - send ASID based IOTLB commands
> > > > >
> > > > > This will help userspace VMM(Qemu) to detect whether
the control vq
> > > > > could be supported and isolate memory mappings of
control virtqueue
> > > > > from the others.
> > > > >
> > > > > To demonstrate the usage, vDPA simulator is extended to
support
> > > > > setting MAC address via a emulated control virtqueue.
> > > > >
> > > > > Please review.
> > > > >
> > > > > Changes since RFC:
> > > > >
> > > > > - tweak vhost uAPI documentation
> > > > > - switch to use device specific IOTLB really in patch 4
> > > > > - tweak the commit log
> > > > > - fix that ASID in vhost is claimed to be 32 actually
but 16bit
> > > > > actually
> > > > > - fix use after free when using ASID with IOTLB
batching requests
> > > > > - switch to use Stefano's patch for having
separated iov
> > > > > - remove unused "used_as" variable
> > > > > - fix the iotlb/asid checking in vhost_vdpa_unmap()
> > > > >
> > > > > Thanks
> > > > >
> > > > > Jason Wang (20):
> > > > > vhost: move the backend feature bits to
vhost_types.h
> > > > > virtio-vdpa: don't set callback if virtio
doesn't need it
> > > > > vhost-vdpa: passing iotlb to IOMMU mapping helpers
> > > > > vhost-vdpa: switch to use vhost-vdpa specific IOTLB
> > > > > vdpa: add the missing comment for nvqs in struct
vdpa_device
> > > > > vdpa: introduce virtqueue groups
> > > > > vdpa: multiple address spaces support
> > > > > vdpa: introduce config operations for associating
ASID to a virtqueue
> > > > > group
> > > > > vhost_iotlb: split out IOTLB initialization
> > > > > vhost: support ASID in IOTLB API
> > > > > vhost-vdpa: introduce asid based IOTLB
> > > > > vhost-vdpa: introduce uAPI to get the number of
virtqueue groups
> > > > > vhost-vdpa: introduce uAPI to get the number of
address spaces
> > > > > vhost-vdpa: uAPI to get virtqueue group id
> > > > > vhost-vdpa: introduce uAPI to set group ASID
> > > > > vhost-vdpa: support ASID based IOTLB API
> > > > > vdpa_sim: advertise VIRTIO_NET_F_MTU
> > > > > vdpa_sim: factor out buffer completion logic
> > > > > vdpa_sim: filter destination mac address
> > > > > vdpasim: control virtqueue support
> > > > >
> > > > > Stefano Garzarella (1):
> > > > > vdpa_sim: split vdpasim_virtqueue's iov field
in out_iov and in_iov
> > > > >
> > > > > drivers/vdpa/ifcvf/ifcvf_main.c | 9 +-
> > > > > drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 +-
> > > > > drivers/vdpa/vdpa.c | 8 +-
> > > > > drivers/vdpa/vdpa_sim/vdpa_sim.c | 292
++++++++++++++++++++++++------
> > > > > drivers/vhost/iotlb.c | 23 ++-
> > > > > drivers/vhost/vdpa.c | 246
++++++++++++++++++++-----
> > > > > drivers/vhost/vhost.c | 23 ++-
> > > > > drivers/vhost/vhost.h | 4 +-
> > > > > drivers/virtio/virtio_vdpa.c | 2 +-
> > > > > include/linux/vdpa.h | 42 ++++-
> > > > > include/linux/vhost_iotlb.h | 2 +
> > > > > include/uapi/linux/vhost.h | 25 ++-
> > > > > include/uapi/linux/vhost_types.h | 10 +-
> > > > > 13 files changed, 561 insertions(+), 136
deletions(-)
> > > > >
> > > > > --
> > > > > 2.25.1