Nouveau - Oct 2025 - [PATCH v2 1/2] rust: pci: skip probing VFs if driver doesn't support VFs

On Thu, Oct 02, 2025 at 08:17:10PM +0200, Danilo Krummrich
wrote:
> On 10/2/25 8:05 PM, Jason Gunthorpe wrote:
> > On Thu, Oct 02, 2025 at 10:49:21AM -0700, John Hubbard wrote:
> >>> Forgot to add: But I think Zhi explained that this is not
necessary and can be
> >>> controlled by the VFIO driver, i.e. the PCI driver that binds
to the VF itself.
> >>
> >> Yes, this is the direction that I originally (3 whole days ago,
haha) had in mind,
> >> after talking with Zhi and a few others: nova-core handles PFs,
and the VFIO driver
> >> handles the VFs, and use the "is virtual" logic to sort
them out.
> > 
> > To be clear, no matter what the VFIO driver bound to the VF should not
> > become entangled with any aux devices.
> > 
> > The VFIO VF driver uses pci_iov_get_pf_drvdata() to reach into the PF
> > to request the PF's help. Eg for live migration or things of that
> > nature.
> 
> Ick! The VF driver should never mess with the PF driver's private data.
> Instead the PF driver should provide an API for the VF driver to get things
done
> on behalf.
This exactly how this function is used.

The core PF driver provides an API:

struct mlx5_core_dev *mlx5_vf_get_core_dev(struct pci_dev *pdev)

Which takes in the VF as pdev and internally it invokes:

	mdev = pci_iov_get_pf_drvdata(pdev, &mlx5_core_driver);

Which asserts the PF is bound to the right driver, which asserts the
format of the drvdata is known, and now we have a proper handle to use
in the rest of API surface the VF driver will use.

By forcing the ops into the signature we strongly encourage people to
follow this design pattern and provide APIs, otherwise you have to
export the ops to call pci_iov_get_pf_drvdata()

There really isn't another good way to obtain the 'struct
mlx5_core_dev' from a simple VF pci_dev.
> It also has the implication that we need to guarantee that PF driver unbind
will
> also unbind all VFs, but we need this guarantee anyways. 
This is another reason why pci_iov_get_pf_drvdata() exists.

/**
 * pci_iov_get_pf_drvdata - Return the drvdata of a PF
 * @dev: VF pci_dev
 * @pf_driver: Device driver required to own the PF
 *
 * This must be called from a context that ensures that a VF driver is attached.
 * The value returned is invalid once the VF driver completes its remove()
 * callback.
 *
 * Locking is achieved by the driver core. A VF driver cannot be probed until
 * pci_enable_sriov() is called and pci_disable_sriov() does not return until
 * all VF drivers have completed their remove().
 *
 * The PF driver must call pci_disable_sriov() before it begins to destroy the
 * drvdata.
 */

Meaning nova-core has to guarentee to call pci_disable_sriov() before
remove completes or before a failing probe returns as part of
implementing SRIOV support.

Userspace cannot create VFs until those calls are done.

Jason

On 10/2/25 8:31 PM, Jason Gunthorpe wrote:
> This exactly how this function is used.
> 
> The core PF driver provides an API:
> 
> struct mlx5_core_dev *mlx5_vf_get_core_dev(struct pci_dev *pdev)
> 
> Which takes in the VF as pdev and internally it invokes:
> 
> 	mdev = pci_iov_get_pf_drvdata(pdev, &mlx5_core_driver);
Oh, I see, that makes sense then. Thanks for clarifying. I think I already had
in mind how this would look like in the Rust abstraction, and there we don't
need pci_iov_get_pf_drvdata() to achieve the same thing.
> /**
>  * pci_iov_get_pf_drvdata - Return the drvdata of a PF
>  * @dev: VF pci_dev
>  * @pf_driver: Device driver required to own the PF
>  *
>  * This must be called from a context that ensures that a VF driver is
attached.
>  * The value returned is invalid once the VF driver completes its remove()
>  * callback.
>  *
>  * Locking is achieved by the driver core. A VF driver cannot be probed
until
>  * pci_enable_sriov() is called and pci_disable_sriov() does not return
until
>  * all VF drivers have completed their remove().
>  *
>  * The PF driver must call pci_disable_sriov() before it begins to destroy
the
>  * drvdata.
>  */
> 
> Meaning nova-core has to guarentee to call pci_disable_sriov() before
> remove completes or before a failing probe returns as part of
> implementing SRIOV support.
Yes, I already thought about this. In the context of adding support for SR-IOV
in the Rust abstractions I'm planning on sending an RFC to let the subsystem
provide this guarantee instead (at least under certain conditions).

This will allow us to assert the device to be bound by the type system in the
Rust PCI abstraction, rather than having the driver to provide a guarantee to
call pci_disable_sriov() manually. :)