Nouveau - Sep 2024 - [RFC 00/29] Introduce NVIDIA GPU Virtualization (vGPU) Support

On Thu, Sep 26, 2024 at 06:43:44AM +0000, Tian, Kevin wrote:
> Then there comes an open whether VFIO is a right place to host such
> vendor specific provisioning interface. The existing mdev type based
> provisioning mechanism was considered a bad fit already.
> IIRC the previous discussion came to suggest putting the provisioning
> interface in the PF driver. There may be chance to generalize and
> move to VFIO but no idea what it will be until multiple drivers already
> demonstrate their own implementations as the base for discussion.
I am looking at fwctl do to alot of this in the SRIOV world.

You'd provision the VF prior to opening VFIO using the fwctl interface
and the VFIO would perceive a VF that has exactly the required
properties. At least for SRIOV where the VM is talking directly to
device FW, mdev/paravirtualization would be different.
> But now seems you prefer to vendors putting their own provisioning
> interface in VFIO directly?
Maybe not, just that drm isn't the right place either. If the we do
fwctl stuff then the VF provisioning would be done through a fwctl
driver.

I'm not entirely sure yet what this whole 'mgr' component is
actually
doing though.

Jason

On Thu, Sep 26, 2024 at 09:55:28AM -0300, Jason Gunthorpe wrote:
> I'm not entirely sure yet what this whole 'mgr' component is
actually
> doing though.
Looking more closely I think some of it is certainly appropriate to be
in vfio. Like when something opens the VFIO device it should allocate
the PF device resources from FW, setup kernel structures and so on to
allow the about to be opened VF to work. That is good VFIO topics. IOW
if you don't open any VFIO devices there would be a minimal overhead

But that stuff shouldn't be shunted into some weird "mgr", it
should
just be inside the struct vfio_device subclass inside the variant
driver.

How to get the provisioning into the kernel prior to VFIO open, and
what kind of control object should exist for the hypervisor side of
the VF, I'm not sure. In mlx5 we used devlink and a netdev/rdma
"respresentor" for alot of this complex control stuff.

Jason