Jason Wang
2023-Aug-08 03:49 UTC
[PATCH vhost v11 05/10] virtio_ring: introduce virtqueue_dma_dev()
On Tue, Aug 8, 2023 at 11:12?AM Xuan Zhuo <xuanzhuo at linux.alibaba.com> wrote:> > On Tue, 8 Aug 2023 11:08:09 +0800, Jason Wang <jasowang at redhat.com> wrote: > > On Tue, Aug 8, 2023 at 10:52?AM Xuan Zhuo <xuanzhuo at linux.alibaba.com> wrote: > > > > > > On Tue, 8 Aug 2023 10:26:04 +0800, Jason Wang <jasowang at redhat.com> wrote: > > > > On Mon, Aug 7, 2023 at 2:15?PM Xuan Zhuo <xuanzhuo at linux.alibaba.com> wrote: > > > > > > > > > > On Wed, 2 Aug 2023 09:49:31 +0800, Xuan Zhuo <xuanzhuo at linux.alibaba.com> wrote: > > > > > > On Tue, 1 Aug 2023 12:17:47 -0400, "Michael S. Tsirkin" <mst at redhat.com> wrote: > > > > > > > On Fri, Jul 28, 2023 at 02:02:33PM +0800, Xuan Zhuo wrote: > > > > > > > > On Tue, 25 Jul 2023 19:07:23 +0800, Xuan Zhuo <xuanzhuo at linux.alibaba.com> wrote: > > > > > > > > > On Tue, 25 Jul 2023 03:34:34 -0400, "Michael S. Tsirkin" <mst at redhat.com> wrote: > > > > > > > > > > On Tue, Jul 25, 2023 at 10:13:48AM +0800, Xuan Zhuo wrote: > > > > > > > > > > > On Mon, 24 Jul 2023 09:43:42 -0700, Christoph Hellwig <hch at infradead.org> wrote: > > > > > > > > > > > > On Thu, Jul 20, 2023 at 01:21:07PM -0400, Michael S. Tsirkin wrote: > > > > > > > > > > > > > Well I think we can add wrappers like virtio_dma_sync and so on. > > > > > > > > > > > > > There are NOP for non-dma so passing the dma device is harmless. > > > > > > > > > > > > > > > > > > > > > > > > Yes, please. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am not sure I got this fully. > > > > > > > > > > > > > > > > > > > > > > Are you mean this: > > > > > > > > > > > https://lore.kernel.org/all/20230214072704.126660-8-xuanzhuo at linux.alibaba.com/ > > > > > > > > > > > https://lore.kernel.org/all/20230214072704.126660-9-xuanzhuo at linux.alibaba.com/ > > > > > > > > > > > > > > > > > > > > > > Then the driver must do dma operation(map and sync) by these virtio_dma_* APIs. > > > > > > > > > > > No care the device is non-dma device or dma device. > > > > > > > > > > > > > > > > > > > > yes > > > > > > > > > > > > > > > > > > > > > Then the AF_XDP must use these virtio_dma_* APIs for virtio device. > > > > > > > > > > > > > > > > > > > > We'll worry about AF_XDP when the patch is posted. > > > > > > > > > > > > > > > > > > YES. > > > > > > > > > > > > > > > > > > We discussed it. They voted 'no'. > > > > > > > > > > > > > > > > > > http://lore.kernel.org/all/20230424082856.15c1e593 at kernel.org > > > > > > > > > > > > > > > > > > > > > > > > Hi guys, this topic is stuck again. How should I proceed with this work? > > > > > > > > > > > > > > > > Let me briefly summarize: > > > > > > > > 1. The problem with adding virtio_dma_{map, sync} api is that, for AF_XDP and > > > > > > > > the driver layer, we need to support these APIs. The current conclusion of > > > > > > > > AF_XDP is no. > > > > > > > > > > > > > > > > 2. Set dma_set_mask_and_coherent, then we can use DMA API uniformly inside > > > > > > > > driver. This idea seems to be inconsistent with the framework design of DMA. The > > > > > > > > conclusion is no. > > > > > > > > > > > > > > > > 3. We noticed that if the virtio device supports VIRTIO_F_ACCESS_PLATFORM, it > > > > > > > > uses DMA API. And this type of device is the future direction, so we only > > > > > > > > support DMA premapped for this type of virtio device. The problem with this > > > > > > > > solution is that virtqueue_dma_dev() only returns dev in some cases, because > > > > > > > > VIRTIO_F_ACCESS_PLATFORM is supported in such cases. > > > > > > > > Could you explain the issue a little bit more? > > > > > > > > E.g if we limit AF_XDP to ACESS_PLATFROM only, why does > > > > virtqueue_dma_dev() only return dev in some cases? > > > > > > The behavior of virtqueue_dma_dev() is not related to AF_XDP. > > > > > > The return value of virtqueue_dma_dev() is used for the DMA APIs. So it can > > > return dma dev when the virtio is with ACCESS_PLATFORM. If virtio is without > > > ACCESS_PLATFORM then it MUST return NULL. > > > > > > In the virtio-net driver, if the virtqueue_dma_dev() returns dma dev, > > > we can enable AF_XDP. If not, we return error to AF_XDP. > > > > Yes, as discussed, just having wrappers in the virtio_ring and doing > > the switch there. Then can virtio-net use them without worrying about > > DMA details? > > > Yes. In the virtio drivers, we can use the wrappers. That is ok. > > But we also need to support virtqueue_dma_dev() for AF_XDP, because that the > AF_XDP will not use the wrappers.You mean AF_XDP core or other? Could you give me an example? Thanks> > So that is ok for you? > > Thanks. > > > > > Thanks > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > >Otherwise NULL is returned. > > > > > > > > This option is currently NO. > > > > > > > > > > > > > > > > So I'm wondering what should I do, from a DMA point of view, is there any > > > > > > > > solution in case of using DMA API? > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > > I think it's ok at this point, Christoph just asked you > > > > > > > to add wrappers for map/unmap for use in virtio code. > > > > > > > Seems like a cosmetic change, shouldn't be hard. > > > > > > > > > > > > Yes, that is not hard, I has this code. > > > > > > > > > > > > But, you mean that the wrappers is just used for the virtio driver code? > > > > > > And we also offer the API virtqueue_dma_dev() at the same time? > > > > > > Then the driver will has two chooses to do DMA. > > > > > > > > > > > > Is that so? > > > > > > > > > > Ping. > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > Otherwise I haven't seen significant comments. > > > > > > > > > > > > > > > > > > > > > Christoph do I summarize what you are saying correctly? > > > > > > > -- > > > > > > > MST > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Xuan Zhuo
2023-Aug-08 03:54 UTC
[PATCH vhost v11 05/10] virtio_ring: introduce virtqueue_dma_dev()
On Tue, 8 Aug 2023 11:49:08 +0800, Jason Wang <jasowang at redhat.com> wrote:> On Tue, Aug 8, 2023 at 11:12?AM Xuan Zhuo <xuanzhuo at linux.alibaba.com> wrote: > > > > On Tue, 8 Aug 2023 11:08:09 +0800, Jason Wang <jasowang at redhat.com> wrote: > > > On Tue, Aug 8, 2023 at 10:52?AM Xuan Zhuo <xuanzhuo at linux.alibaba.com> wrote: > > > > > > > > On Tue, 8 Aug 2023 10:26:04 +0800, Jason Wang <jasowang at redhat.com> wrote: > > > > > On Mon, Aug 7, 2023 at 2:15?PM Xuan Zhuo <xuanzhuo at linux.alibaba.com> wrote: > > > > > > > > > > > > On Wed, 2 Aug 2023 09:49:31 +0800, Xuan Zhuo <xuanzhuo at linux.alibaba.com> wrote: > > > > > > > On Tue, 1 Aug 2023 12:17:47 -0400, "Michael S. Tsirkin" <mst at redhat.com> wrote: > > > > > > > > On Fri, Jul 28, 2023 at 02:02:33PM +0800, Xuan Zhuo wrote: > > > > > > > > > On Tue, 25 Jul 2023 19:07:23 +0800, Xuan Zhuo <xuanzhuo at linux.alibaba.com> wrote: > > > > > > > > > > On Tue, 25 Jul 2023 03:34:34 -0400, "Michael S. Tsirkin" <mst at redhat.com> wrote: > > > > > > > > > > > On Tue, Jul 25, 2023 at 10:13:48AM +0800, Xuan Zhuo wrote: > > > > > > > > > > > > On Mon, 24 Jul 2023 09:43:42 -0700, Christoph Hellwig <hch at infradead.org> wrote: > > > > > > > > > > > > > On Thu, Jul 20, 2023 at 01:21:07PM -0400, Michael S. Tsirkin wrote: > > > > > > > > > > > > > > Well I think we can add wrappers like virtio_dma_sync and so on. > > > > > > > > > > > > > > There are NOP for non-dma so passing the dma device is harmless. > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, please. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am not sure I got this fully. > > > > > > > > > > > > > > > > > > > > > > > > Are you mean this: > > > > > > > > > > > > https://lore.kernel.org/all/20230214072704.126660-8-xuanzhuo at linux.alibaba.com/ > > > > > > > > > > > > https://lore.kernel.org/all/20230214072704.126660-9-xuanzhuo at linux.alibaba.com/ > > > > > > > > > > > > > > > > > > > > > > > > Then the driver must do dma operation(map and sync) by these virtio_dma_* APIs. > > > > > > > > > > > > No care the device is non-dma device or dma device. > > > > > > > > > > > > > > > > > > > > > > yes > > > > > > > > > > > > > > > > > > > > > > > Then the AF_XDP must use these virtio_dma_* APIs for virtio device. > > > > > > > > > > > > > > > > > > > > > > We'll worry about AF_XDP when the patch is posted. > > > > > > > > > > > > > > > > > > > > YES. > > > > > > > > > > > > > > > > > > > > We discussed it. They voted 'no'. > > > > > > > > > > > > > > > > > > > > http://lore.kernel.org/all/20230424082856.15c1e593 at kernel.org > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi guys, this topic is stuck again. How should I proceed with this work? > > > > > > > > > > > > > > > > > > Let me briefly summarize: > > > > > > > > > 1. The problem with adding virtio_dma_{map, sync} api is that, for AF_XDP and > > > > > > > > > the driver layer, we need to support these APIs. The current conclusion of > > > > > > > > > AF_XDP is no. > > > > > > > > > > > > > > > > > > 2. Set dma_set_mask_and_coherent, then we can use DMA API uniformly inside > > > > > > > > > driver. This idea seems to be inconsistent with the framework design of DMA. The > > > > > > > > > conclusion is no. > > > > > > > > > > > > > > > > > > 3. We noticed that if the virtio device supports VIRTIO_F_ACCESS_PLATFORM, it > > > > > > > > > uses DMA API. And this type of device is the future direction, so we only > > > > > > > > > support DMA premapped for this type of virtio device. The problem with this > > > > > > > > > solution is that virtqueue_dma_dev() only returns dev in some cases, because > > > > > > > > > VIRTIO_F_ACCESS_PLATFORM is supported in such cases. > > > > > > > > > > Could you explain the issue a little bit more? > > > > > > > > > > E.g if we limit AF_XDP to ACESS_PLATFROM only, why does > > > > > virtqueue_dma_dev() only return dev in some cases? > > > > > > > > The behavior of virtqueue_dma_dev() is not related to AF_XDP. > > > > > > > > The return value of virtqueue_dma_dev() is used for the DMA APIs. So it can > > > > return dma dev when the virtio is with ACCESS_PLATFORM. If virtio is without > > > > ACCESS_PLATFORM then it MUST return NULL. > > > > > > > > In the virtio-net driver, if the virtqueue_dma_dev() returns dma dev, > > > > we can enable AF_XDP. If not, we return error to AF_XDP. > > > > > > Yes, as discussed, just having wrappers in the virtio_ring and doing > > > the switch there. Then can virtio-net use them without worrying about > > > DMA details? > > > > > > Yes. In the virtio drivers, we can use the wrappers. That is ok. > > > > But we also need to support virtqueue_dma_dev() for AF_XDP, because that the > > AF_XDP will not use the wrappers. > > You mean AF_XDP core or other? Could you give me an example?Yes. The AF_XDP core. Now the AF_XDP core will do the dma operation. Because that the memory is allocated by the user from the user space. So before putting the memory to the driver, the AF_XDP will do the dma mapping. int xp_dma_map(struct xsk_buff_pool *pool, struct device *dev, unsigned long attrs, struct page **pages, u32 nr_pages) { struct xsk_dma_map *dma_map; dma_addr_t dma; int err; u32 i; dma_map = xp_find_dma_map(pool); if (dma_map) { err = xp_init_dma_info(pool, dma_map); if (err) return err; refcount_inc(&dma_map->users); return 0; } dma_map = xp_create_dma_map(dev, pool->netdev, nr_pages, pool->umem); if (!dma_map) return -ENOMEM; for (i = 0; i < dma_map->dma_pages_cnt; i++) { dma = dma_map_page_attrs(dev, pages[i], 0, PAGE_SIZE, DMA_BIDIRECTIONAL, attrs); if (dma_mapping_error(dev, dma)) { __xp_dma_unmap(dma_map, attrs); return -ENOMEM; } if (dma_need_sync(dev, dma)) dma_map->dma_need_sync = true; dma_map->dma_pages[i] = dma; } if (pool->unaligned) xp_check_dma_contiguity(dma_map); err = xp_init_dma_info(pool, dma_map); if (err) { __xp_dma_unmap(dma_map, attrs); return err; } return 0; } EXPORT_SYMBOL(xp_dma_map); Thanks.> > Thanks > > > > > So that is ok for you? > > > > Thanks. > > > > > > > > Thanks > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > >Otherwise NULL is returned. > > > > > > > > > This option is currently NO. > > > > > > > > > > > > > > > > > > So I'm wondering what should I do, from a DMA point of view, is there any > > > > > > > > > solution in case of using DMA API? > > > > > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > > > > > > > > > > > I think it's ok at this point, Christoph just asked you > > > > > > > > to add wrappers for map/unmap for use in virtio code. > > > > > > > > Seems like a cosmetic change, shouldn't be hard. > > > > > > > > > > > > > > Yes, that is not hard, I has this code. > > > > > > > > > > > > > > But, you mean that the wrappers is just used for the virtio driver code? > > > > > > > And we also offer the API virtqueue_dma_dev() at the same time? > > > > > > > Then the driver will has two chooses to do DMA. > > > > > > > > > > > > > > Is that so? > > > > > > > > > > > > Ping. > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > Otherwise I haven't seen significant comments. > > > > > > > > > > > > > > > > > > > > > > > > Christoph do I summarize what you are saying correctly? > > > > > > > > -- > > > > > > > > MST > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >