Christian König
2023-Jan-18 15:37 UTC
[Nouveau] [PATCH drm-next 00/14] [RFC] DRM GPUVA Manager & Nouveau VM_BIND UAPI
Am 18.01.23 um 16:34 schrieb Danilo Krummrich:> Hi Christian, > > On 1/18/23 09:53, Christian K?nig wrote: >> Am 18.01.23 um 07:12 schrieb Danilo Krummrich: >>> This patch series provides a new UAPI for the Nouveau driver in >>> order to >>> support Vulkan features, such as sparse bindings and sparse residency. >>> >>> Furthermore, with the DRM GPUVA manager it provides a new DRM core >>> feature to >>> keep track of GPU virtual address (VA) mappings in a more generic way. >>> >>> The DRM GPUVA manager is indented to help drivers implement >>> userspace-manageable >>> GPU VA spaces in reference to the Vulkan API. In order to achieve >>> this goal it >>> serves the following purposes in this context. >>> >>> ???? 1) Provide a dedicated range allocator to track GPU VA >>> allocations and >>> ??????? mappings, making use of the drm_mm range allocator. >> >> This means that the ranges are allocated by the kernel? If yes that's >> a really really bad idea. > > No, it's just for keeping track of the ranges userspace has allocated.Ok, that makes more sense. So basically you have an IOCTL which asks kernel for a free range? Or what exactly is the drm_mm used for here? Regards, Christian.> > - Danilo > >> >> Regards, >> Christian. >> >>> >>> ???? 2) Generically connect GPU VA mappings to their backing >>> buffers, in >>> ??????? particular DRM GEM objects. >>> >>> ???? 3) Provide a common implementation to perform more complex mapping >>> ??????? operations on the GPU VA space. In particular splitting and >>> merging >>> ??????? of GPU VA mappings, e.g. for intersecting mapping requests >>> or partial >>> ??????? unmap requests. >>> >>> The new VM_BIND Nouveau UAPI build on top of the DRM GPUVA manager, >>> itself >>> providing the following new interfaces. >>> >>> ???? 1) Initialize a GPU VA space via the new >>> DRM_IOCTL_NOUVEAU_VM_INIT ioctl >>> ??????? for UMDs to specify the portion of VA space managed by the >>> kernel and >>> ??????? userspace, respectively. >>> >>> ???? 2) Allocate and free a VA space region as well as bind and >>> unbind memory >>> ??????? to the GPUs VA space via the new DRM_IOCTL_NOUVEAU_VM_BIND >>> ioctl. >>> >>> ???? 3) Execute push buffers with the new DRM_IOCTL_NOUVEAU_EXEC ioctl. >>> >>> Both, DRM_IOCTL_NOUVEAU_VM_BIND and DRM_IOCTL_NOUVEAU_EXEC, make use >>> of the DRM >>> scheduler to queue jobs and support asynchronous processing with DRM >>> syncobjs >>> as synchronization mechanism. >>> >>> By default DRM_IOCTL_NOUVEAU_VM_BIND does synchronous processing, >>> DRM_IOCTL_NOUVEAU_EXEC supports asynchronous processing only. >>> >>> The new VM_BIND UAPI for Nouveau makes also use of drm_exec >>> (execution context >>> for GEM buffers) by Christian K?nig. Since the patch implementing >>> drm_exec was >>> not yet merged into drm-next it is part of this series, as well as a >>> small fix >>> for this patch, which was found while testing this series. >>> >>> This patch series is also available at [1]. >>> >>> There is a Mesa NVK merge request by Dave Airlie [2] implementing the >>> corresponding userspace parts for this series. >>> >>> The Vulkan CTS test suite passes the sparse binding and sparse >>> residency test >>> cases for the new UAPI together with Dave's Mesa work. >>> >>> There are also some test cases in the igt-gpu-tools project [3] for >>> the new UAPI >>> and hence the DRM GPU VA manager. However, most of them are testing >>> the DRM GPU >>> VA manager's logic through Nouveau's new UAPI and should be >>> considered just as >>> helper for implementation. >>> >>> However, I absolutely intend to change those test cases to proper >>> kunit test >>> cases for the DRM GPUVA manager, once and if we agree on it's >>> usefulness and >>> design. >>> >>> [1] >>> https://gitlab.freedesktop.org/nouvelles/kernel/-/tree/new-uapi-drm-next >>> / >>> https://gitlab.freedesktop.org/nouvelles/kernel/-/merge_requests/1 >>> [2] https://gitlab.freedesktop.org/nouveau/mesa/-/merge_requests/150/ >>> [3] >>> https://gitlab.freedesktop.org/dakr/igt-gpu-tools/-/tree/wip_nouveau_vm_bind >>> >>> I also want to give credit to Dave Airlie, who contributed a lot of >>> ideas to >>> this patch series. >>> >>> Christian K?nig (1): >>> ?? drm: execution context for GEM buffers >>> >>> Danilo Krummrich (13): >>> ?? drm/exec: fix memory leak in drm_exec_prepare_obj() >>> ?? drm: manager to keep track of GPUs VA mappings >>> ?? drm: debugfs: provide infrastructure to dump a DRM GPU VA space >>> ?? drm/nouveau: new VM_BIND uapi interfaces >>> ?? drm/nouveau: get vmm via nouveau_cli_vmm() >>> ?? drm/nouveau: bo: initialize GEM GPU VA interface >>> ?? drm/nouveau: move usercopy helpers to nouveau_drv.h >>> ?? drm/nouveau: fence: fail to emit when fence context is killed >>> ?? drm/nouveau: chan: provide nouveau_channel_kill() >>> ?? drm/nouveau: nvkm/vmm: implement raw ops to manage uvmm >>> ?? drm/nouveau: implement uvmm for user mode bindings >>> ?? drm/nouveau: implement new VM_BIND UAPI >>> ?? drm/nouveau: debugfs: implement DRM GPU VA debugfs >>> >>> ? Documentation/gpu/driver-uapi.rst???????????? |?? 11 + >>> ? Documentation/gpu/drm-mm.rst????????????????? |?? 43 + >>> ? drivers/gpu/drm/Kconfig?????????????????????? |??? 6 + >>> ? drivers/gpu/drm/Makefile????????????????????? |??? 3 + >>> ? drivers/gpu/drm/amd/amdgpu/Kconfig??????????? |??? 1 + >>> ? drivers/gpu/drm/drm_debugfs.c???????????????? |?? 56 + >>> ? drivers/gpu/drm/drm_exec.c??????????????????? |? 294 ++++ >>> ? drivers/gpu/drm/drm_gem.c???????????????????? |??? 3 + >>> ? drivers/gpu/drm/drm_gpuva_mgr.c?????????????? | 1323 >>> +++++++++++++++++ >>> ? drivers/gpu/drm/nouveau/Kbuild??????????????? |??? 3 + >>> ? drivers/gpu/drm/nouveau/Kconfig?????????????? |??? 2 + >>> ? drivers/gpu/drm/nouveau/include/nvif/if000c.h |?? 23 +- >>> ? drivers/gpu/drm/nouveau/include/nvif/vmm.h??? |?? 17 +- >>> ? .../gpu/drm/nouveau/include/nvkm/subdev/mmu.h |?? 10 + >>> ? drivers/gpu/drm/nouveau/nouveau_abi16.c?????? |?? 23 + >>> ? drivers/gpu/drm/nouveau/nouveau_abi16.h?????? |??? 1 + >>> ? drivers/gpu/drm/nouveau/nouveau_bo.c????????? |? 152 +- >>> ? drivers/gpu/drm/nouveau/nouveau_bo.h????????? |??? 2 +- >>> ? drivers/gpu/drm/nouveau/nouveau_chan.c??????? |?? 16 +- >>> ? drivers/gpu/drm/nouveau/nouveau_chan.h??????? |??? 1 + >>> ? drivers/gpu/drm/nouveau/nouveau_debugfs.c???? |?? 24 + >>> ? drivers/gpu/drm/nouveau/nouveau_drm.c???????? |?? 25 +- >>> ? drivers/gpu/drm/nouveau/nouveau_drv.h???????? |?? 92 +- >>> ? drivers/gpu/drm/nouveau/nouveau_exec.c??????? |? 310 ++++ >>> ? drivers/gpu/drm/nouveau/nouveau_exec.h??????? |?? 55 + >>> ? drivers/gpu/drm/nouveau/nouveau_fence.c?????? |??? 7 + >>> ? drivers/gpu/drm/nouveau/nouveau_fence.h?????? |??? 2 +- >>> ? drivers/gpu/drm/nouveau/nouveau_gem.c???????? |?? 83 +- >>> ? drivers/gpu/drm/nouveau/nouveau_mem.h???????? |??? 5 + >>> ? drivers/gpu/drm/nouveau/nouveau_prime.c?????? |??? 2 +- >>> ? drivers/gpu/drm/nouveau/nouveau_sched.c?????? |? 780 ++++++++++ >>> ? drivers/gpu/drm/nouveau/nouveau_sched.h?????? |?? 98 ++ >>> ? drivers/gpu/drm/nouveau/nouveau_svm.c???????? |??? 2 +- >>> ? drivers/gpu/drm/nouveau/nouveau_uvmm.c??????? |? 575 +++++++ >>> ? drivers/gpu/drm/nouveau/nouveau_uvmm.h??????? |?? 68 + >>> ? drivers/gpu/drm/nouveau/nouveau_vmm.c???????? |??? 4 +- >>> ? drivers/gpu/drm/nouveau/nvif/vmm.c??????????? |?? 73 +- >>> ? .../gpu/drm/nouveau/nvkm/subdev/mmu/uvmm.c??? |? 168 ++- >>> ? .../gpu/drm/nouveau/nvkm/subdev/mmu/uvmm.h??? |??? 1 + >>> ? drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.c |?? 32 +- >>> ? drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.h |??? 3 + >>> ? include/drm/drm_debugfs.h???????????????????? |?? 25 + >>> ? include/drm/drm_drv.h???????????????????????? |??? 6 + >>> ? include/drm/drm_exec.h??????????????????????? |? 144 ++ >>> ? include/drm/drm_gem.h???????????????????????? |?? 75 + >>> ? include/drm/drm_gpuva_mgr.h?????????????????? |? 527 +++++++ >>> ? include/uapi/drm/nouveau_drm.h??????????????? |? 216 +++ >>> ? 47 files changed, 5266 insertions(+), 126 deletions(-) >>> ? create mode 100644 drivers/gpu/drm/drm_exec.c >>> ? create mode 100644 drivers/gpu/drm/drm_gpuva_mgr.c >>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.c >>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.h >>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.c >>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.h >>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.c >>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.h >>> ? create mode 100644 include/drm/drm_exec.h >>> ? create mode 100644 include/drm/drm_gpuva_mgr.h >>> >>> >>> base-commit: 0b45ac1170ea6416bc1d36798414c04870cd356d >> >
Danilo Krummrich
2023-Jan-18 16:19 UTC
[Nouveau] [PATCH drm-next 00/14] [RFC] DRM GPUVA Manager & Nouveau VM_BIND UAPI
On 1/18/23 16:37, Christian K?nig wrote:> Am 18.01.23 um 16:34 schrieb Danilo Krummrich: >> Hi Christian, >> >> On 1/18/23 09:53, Christian K?nig wrote: >>> Am 18.01.23 um 07:12 schrieb Danilo Krummrich: >>>> This patch series provides a new UAPI for the Nouveau driver in >>>> order to >>>> support Vulkan features, such as sparse bindings and sparse residency. >>>> >>>> Furthermore, with the DRM GPUVA manager it provides a new DRM core >>>> feature to >>>> keep track of GPU virtual address (VA) mappings in a more generic way. >>>> >>>> The DRM GPUVA manager is indented to help drivers implement >>>> userspace-manageable >>>> GPU VA spaces in reference to the Vulkan API. In order to achieve >>>> this goal it >>>> serves the following purposes in this context. >>>> >>>> ???? 1) Provide a dedicated range allocator to track GPU VA >>>> allocations and >>>> ??????? mappings, making use of the drm_mm range allocator. >>> >>> This means that the ranges are allocated by the kernel? If yes that's >>> a really really bad idea. >> >> No, it's just for keeping track of the ranges userspace has allocated. > > Ok, that makes more sense. > > So basically you have an IOCTL which asks kernel for a free range? Or > what exactly is the drm_mm used for here?Not even that, userspace provides both the base address and the range, the kernel really just keeps track of things. Though, writing a UAPI on top of the GPUVA manager asking for a free range instead would be possible by just adding the corresponding wrapper functions to get a free hole. Currently, and that's what I think I read out of your question, the main benefit of using drm_mm over simply stuffing the entries into a list or something boils down to easier collision detection and iterating sub-ranges of the whole VA space.> > Regards, > Christian. > >> >> - Danilo >> >>> >>> Regards, >>> Christian. >>> >>>> >>>> ???? 2) Generically connect GPU VA mappings to their backing >>>> buffers, in >>>> ??????? particular DRM GEM objects. >>>> >>>> ???? 3) Provide a common implementation to perform more complex mapping >>>> ??????? operations on the GPU VA space. In particular splitting and >>>> merging >>>> ??????? of GPU VA mappings, e.g. for intersecting mapping requests >>>> or partial >>>> ??????? unmap requests. >>>> >>>> The new VM_BIND Nouveau UAPI build on top of the DRM GPUVA manager, >>>> itself >>>> providing the following new interfaces. >>>> >>>> ???? 1) Initialize a GPU VA space via the new >>>> DRM_IOCTL_NOUVEAU_VM_INIT ioctl >>>> ??????? for UMDs to specify the portion of VA space managed by the >>>> kernel and >>>> ??????? userspace, respectively. >>>> >>>> ???? 2) Allocate and free a VA space region as well as bind and >>>> unbind memory >>>> ??????? to the GPUs VA space via the new DRM_IOCTL_NOUVEAU_VM_BIND >>>> ioctl. >>>> >>>> ???? 3) Execute push buffers with the new DRM_IOCTL_NOUVEAU_EXEC ioctl. >>>> >>>> Both, DRM_IOCTL_NOUVEAU_VM_BIND and DRM_IOCTL_NOUVEAU_EXEC, make use >>>> of the DRM >>>> scheduler to queue jobs and support asynchronous processing with DRM >>>> syncobjs >>>> as synchronization mechanism. >>>> >>>> By default DRM_IOCTL_NOUVEAU_VM_BIND does synchronous processing, >>>> DRM_IOCTL_NOUVEAU_EXEC supports asynchronous processing only. >>>> >>>> The new VM_BIND UAPI for Nouveau makes also use of drm_exec >>>> (execution context >>>> for GEM buffers) by Christian K?nig. Since the patch implementing >>>> drm_exec was >>>> not yet merged into drm-next it is part of this series, as well as a >>>> small fix >>>> for this patch, which was found while testing this series. >>>> >>>> This patch series is also available at [1]. >>>> >>>> There is a Mesa NVK merge request by Dave Airlie [2] implementing the >>>> corresponding userspace parts for this series. >>>> >>>> The Vulkan CTS test suite passes the sparse binding and sparse >>>> residency test >>>> cases for the new UAPI together with Dave's Mesa work. >>>> >>>> There are also some test cases in the igt-gpu-tools project [3] for >>>> the new UAPI >>>> and hence the DRM GPU VA manager. However, most of them are testing >>>> the DRM GPU >>>> VA manager's logic through Nouveau's new UAPI and should be >>>> considered just as >>>> helper for implementation. >>>> >>>> However, I absolutely intend to change those test cases to proper >>>> kunit test >>>> cases for the DRM GPUVA manager, once and if we agree on it's >>>> usefulness and >>>> design. >>>> >>>> [1] >>>> https://gitlab.freedesktop.org/nouvelles/kernel/-/tree/new-uapi-drm-next / >>>> https://gitlab.freedesktop.org/nouvelles/kernel/-/merge_requests/1 >>>> [2] https://gitlab.freedesktop.org/nouveau/mesa/-/merge_requests/150/ >>>> [3] >>>> https://gitlab.freedesktop.org/dakr/igt-gpu-tools/-/tree/wip_nouveau_vm_bind >>>> >>>> I also want to give credit to Dave Airlie, who contributed a lot of >>>> ideas to >>>> this patch series. >>>> >>>> Christian K?nig (1): >>>> ?? drm: execution context for GEM buffers >>>> >>>> Danilo Krummrich (13): >>>> ?? drm/exec: fix memory leak in drm_exec_prepare_obj() >>>> ?? drm: manager to keep track of GPUs VA mappings >>>> ?? drm: debugfs: provide infrastructure to dump a DRM GPU VA space >>>> ?? drm/nouveau: new VM_BIND uapi interfaces >>>> ?? drm/nouveau: get vmm via nouveau_cli_vmm() >>>> ?? drm/nouveau: bo: initialize GEM GPU VA interface >>>> ?? drm/nouveau: move usercopy helpers to nouveau_drv.h >>>> ?? drm/nouveau: fence: fail to emit when fence context is killed >>>> ?? drm/nouveau: chan: provide nouveau_channel_kill() >>>> ?? drm/nouveau: nvkm/vmm: implement raw ops to manage uvmm >>>> ?? drm/nouveau: implement uvmm for user mode bindings >>>> ?? drm/nouveau: implement new VM_BIND UAPI >>>> ?? drm/nouveau: debugfs: implement DRM GPU VA debugfs >>>> >>>> ? Documentation/gpu/driver-uapi.rst???????????? |?? 11 + >>>> ? Documentation/gpu/drm-mm.rst????????????????? |?? 43 + >>>> ? drivers/gpu/drm/Kconfig?????????????????????? |??? 6 + >>>> ? drivers/gpu/drm/Makefile????????????????????? |??? 3 + >>>> ? drivers/gpu/drm/amd/amdgpu/Kconfig??????????? |??? 1 + >>>> ? drivers/gpu/drm/drm_debugfs.c???????????????? |?? 56 + >>>> ? drivers/gpu/drm/drm_exec.c??????????????????? |? 294 ++++ >>>> ? drivers/gpu/drm/drm_gem.c???????????????????? |??? 3 + >>>> ? drivers/gpu/drm/drm_gpuva_mgr.c?????????????? | 1323 >>>> +++++++++++++++++ >>>> ? drivers/gpu/drm/nouveau/Kbuild??????????????? |??? 3 + >>>> ? drivers/gpu/drm/nouveau/Kconfig?????????????? |??? 2 + >>>> ? drivers/gpu/drm/nouveau/include/nvif/if000c.h |?? 23 +- >>>> ? drivers/gpu/drm/nouveau/include/nvif/vmm.h??? |?? 17 +- >>>> ? .../gpu/drm/nouveau/include/nvkm/subdev/mmu.h |?? 10 + >>>> ? drivers/gpu/drm/nouveau/nouveau_abi16.c?????? |?? 23 + >>>> ? drivers/gpu/drm/nouveau/nouveau_abi16.h?????? |??? 1 + >>>> ? drivers/gpu/drm/nouveau/nouveau_bo.c????????? |? 152 +- >>>> ? drivers/gpu/drm/nouveau/nouveau_bo.h????????? |??? 2 +- >>>> ? drivers/gpu/drm/nouveau/nouveau_chan.c??????? |?? 16 +- >>>> ? drivers/gpu/drm/nouveau/nouveau_chan.h??????? |??? 1 + >>>> ? drivers/gpu/drm/nouveau/nouveau_debugfs.c???? |?? 24 + >>>> ? drivers/gpu/drm/nouveau/nouveau_drm.c???????? |?? 25 +- >>>> ? drivers/gpu/drm/nouveau/nouveau_drv.h???????? |?? 92 +- >>>> ? drivers/gpu/drm/nouveau/nouveau_exec.c??????? |? 310 ++++ >>>> ? drivers/gpu/drm/nouveau/nouveau_exec.h??????? |?? 55 + >>>> ? drivers/gpu/drm/nouveau/nouveau_fence.c?????? |??? 7 + >>>> ? drivers/gpu/drm/nouveau/nouveau_fence.h?????? |??? 2 +- >>>> ? drivers/gpu/drm/nouveau/nouveau_gem.c???????? |?? 83 +- >>>> ? drivers/gpu/drm/nouveau/nouveau_mem.h???????? |??? 5 + >>>> ? drivers/gpu/drm/nouveau/nouveau_prime.c?????? |??? 2 +- >>>> ? drivers/gpu/drm/nouveau/nouveau_sched.c?????? |? 780 ++++++++++ >>>> ? drivers/gpu/drm/nouveau/nouveau_sched.h?????? |?? 98 ++ >>>> ? drivers/gpu/drm/nouveau/nouveau_svm.c???????? |??? 2 +- >>>> ? drivers/gpu/drm/nouveau/nouveau_uvmm.c??????? |? 575 +++++++ >>>> ? drivers/gpu/drm/nouveau/nouveau_uvmm.h??????? |?? 68 + >>>> ? drivers/gpu/drm/nouveau/nouveau_vmm.c???????? |??? 4 +- >>>> ? drivers/gpu/drm/nouveau/nvif/vmm.c??????????? |?? 73 +- >>>> ? .../gpu/drm/nouveau/nvkm/subdev/mmu/uvmm.c??? |? 168 ++- >>>> ? .../gpu/drm/nouveau/nvkm/subdev/mmu/uvmm.h??? |??? 1 + >>>> ? drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.c |?? 32 +- >>>> ? drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.h |??? 3 + >>>> ? include/drm/drm_debugfs.h???????????????????? |?? 25 + >>>> ? include/drm/drm_drv.h???????????????????????? |??? 6 + >>>> ? include/drm/drm_exec.h??????????????????????? |? 144 ++ >>>> ? include/drm/drm_gem.h???????????????????????? |?? 75 + >>>> ? include/drm/drm_gpuva_mgr.h?????????????????? |? 527 +++++++ >>>> ? include/uapi/drm/nouveau_drm.h??????????????? |? 216 +++ >>>> ? 47 files changed, 5266 insertions(+), 126 deletions(-) >>>> ? create mode 100644 drivers/gpu/drm/drm_exec.c >>>> ? create mode 100644 drivers/gpu/drm/drm_gpuva_mgr.c >>>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.c >>>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_exec.h >>>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.c >>>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_sched.h >>>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.c >>>> ? create mode 100644 drivers/gpu/drm/nouveau/nouveau_uvmm.h >>>> ? create mode 100644 include/drm/drm_exec.h >>>> ? create mode 100644 include/drm/drm_gpuva_mgr.h >>>> >>>> >>>> base-commit: 0b45ac1170ea6416bc1d36798414c04870cd356d >>> >> >