Linux Virtualization - Apr 2023 - [PATCH vhost v6 08/11] virtio_ring: introduce virtqueue_dma_dev()

On Tue, Apr 11, 2023 at 10:01?AM Xuan Zhuo <xuanzhuo at linux.alibaba.com>
wrote:
>
> On Mon, 10 Apr 2023 08:27:14 -0700, Christoph Hellwig <hch at
infradead.org> wrote:
> > On Mon, Apr 10, 2023 at 01:14:13PM +0800, Jason Wang wrote:
> > > > But rethink, this is unreliable. Some platforms have
returned their own ops,
> > > > including X86.
> > > >
> > > > Because the priority of dev->dma_ops is higher than
get_arch_dma_ops(), we
> > > > should not let dev->dma_ops be null. We should define a
set of dma_ops to
> > > > actively implement direct dMA.
> > >
> > > Then this will duplicate with existing DMA API helpers. Could we
tweak
> > > the sparc or introduce a new flag for the device structure then
the
> > > DMA API knows it needs to use direct mapping?
> > >
> > > Adding Christoph for more comments.
> >
> > Code outside of the core kernel/dma/ code has no business doing
> > anything with the dma_ops.
>
> Do you mean we should not change the dma_ops from the outside of the core
> kernel/dma/?
>
> How about adding one new "dma_direct" to struct devive?
>
> --- a/include/linux/dma-map-ops.h
> +++ b/include/linux/dma-map-ops.h
> @@ -88,6 +88,9 @@ struct dma_map_ops {
>
>  static inline const struct dma_map_ops *get_dma_ops(struct device *dev)
>  {
> +       if (dev->dma_direct)
> +               return NULL;
> +
>         if (dev->dma_ops)
>                 return dev->dma_ops;
>         return get_arch_dma_ops();
>
>
> Thanks.
>
>
>
> > WTF is going on?
We want to support AF_XDP for virtio-net. It means AF_XDP needs to
know the dma device to perform DMA mapping. So we introduce a helper
to expose the dma dev of the virtio device.

This works fine as long as VIRTIO_F_ACCESS_PLATFORM is negotiated. But
when it is not, the virtio driver needs to use a physical address so
we want to expose the virtio device without dma_ops in the hope that
it will go for direct mapping where the physical address is used. But
it may not work on some specific setups (arches that assume an IOMMU
or have arch dma ops).

Thanks
>

On Tue, Apr 11, 2023 at 10:09:40AM +0800, Jason Wang
wrote:
> We want to support AF_XDP for virtio-net. It means AF_XDP needs to
> know the dma device to perform DMA mapping. So we introduce a helper
> to expose the dma dev of the virtio device.
The whole virtio architecture is based around the core code doing
the DMA mapping.  I can't see how you can just export a helper to
expose the dma device.  You'd have to complete rework the layering
of the virtio code if you want to do it in the upper level drivers.
And why would you want to do this?  The low-level code is the only
piece that can actually know if you need to do a dma mapping.  All
the kernel subsystems that don't do it inside the low-level drivers
or helpers closely associtated are a giant and hard to fix map
(see usb for the prime exhibit).

So the first question is:  why do you want this for XF_ADP, and
the next question will be how to do that without making a complete
mess.
> This works fine as long as VIRTIO_F_ACCESS_PLATFORM is negotiated. But
> when it is not, the virtio driver needs to use a physical address so
> we want to expose the virtio device without dma_ops in the hope that
> it will go for direct mapping where the physical address is used. But
> it may not work on some specific setups (arches that assume an IOMMU
> or have arch dma ops).
The DMA device for virtio_pci is the underlying PCI device, always.
!VIRTIO_F_ACCESS_PLATFORM means there is no dma device at all.  Because
of all these things you can't just expose a pointer to the dma_device
as that is just a completely wrong way of thinking about the problem.