Nouveau - Oct 2025 - [PATCH v2 1/2] rust: pci: skip probing VFs if driver doesn't support VFs

On Thu Oct 2, 2025 at 7:05 PM CEST, Jason Gunthorpe
wrote:
> On Thu, Oct 02, 2025 at 06:05:28PM +0200, Danilo Krummrich wrote:
>> On Thu Oct 2, 2025 at 5:23 PM CEST, Jason Gunthorpe wrote:
>> > This is not what I've been told, the VF driver has significant
>> > programming model differences in the NVIDIA model, and supports
>> > different commands.
>> 
>> Ok, that means there are some more fundamental differences between the
host PF
>> and the "VM PF" code that we have to deal with.
>
> That was my understanding.
>  
>> But that doesn't necessarily require that the VF parts of the host
have to be in
>> nova-core as well, i.e. with the information we have we can
differentiate
>> between PF, VF and PF in the VM (indicated by a device register).
>
> I'm not entirely sure what you mean by this..
>
> The driver to operate the function in "vGPU" mode as indicated by
the
> register has to be in nova-core, since there is only one device ID.
Yes, the PF driver on the host and the PF (from VM perspective) driver in the VM
have to be that same. But the VF driver on the host can still be a seaparate
one.
>> > If you look at the VFIO driver RFC it basically does no mediation,
it
>> > isn't intercepting MMIO - the guest sees the BARs directly.
Most of
>> > the code is "profiling" from what I can tell. Some
config space
>> > meddling.
>> 
>> Sure, there is no mediation in that sense, but it needs quite some
setup
>> regardless, no?
>>
>> I thought there is a significant amount of semantics that is different
between
>> booting the PF and the VF on the host.
>
> I think it would be good to have Zhi clarify more of this, but from
> what I understand are at least three activites comingled all together:
>
>  1) Boot the PF in "vGPU" mode so it can enable SRIOV
Ok, this might be where the confusion above comes from. When I talk about
nova-core in vGPU mode I mean nova-core running in the VM on the (from VM
perspective) PF.

But you seem to mean nova-core running on the host PF with vGPU on top? That of
course has to be in nova-core.
>  2) Enable SRIOV and profile VFs to allocate HW resources to them
I think that's partially in nova-core and partially in vGPU; nova-core
providing
the abstraction of the corresponding firmware / hardware interfaces and vGPU
controlling the semantics of the resource handling?

This is what I thought vGPU has a secondary part for where it binds to nova-core
through the auxiliary bus, i.e. vGPU consisting out of two drivers actually; the
VFIO parts and a "per VF resource controller".
>  3) VFIO variant driver to convert the VF into a "VM PF" with
whatever
>     mediation and enhancement needed
That should be vGPU only land.

On Thu Oct 2, 2025 at 7:37 PM CEST, Danilo Krummrich
wrote:
> On Thu Oct 2, 2025 at 7:05 PM CEST, Jason Gunthorpe wrote:
>> On Thu, Oct 02, 2025 at 06:05:28PM +0200, Danilo Krummrich wrote:
>>> On Thu Oct 2, 2025 at 5:23 PM CEST, Jason Gunthorpe wrote:
>>> > This is not what I've been told, the VF driver has
significant
>>> > programming model differences in the NVIDIA model, and
supports
>>> > different commands.
>>> 
>>> Ok, that means there are some more fundamental differences between
the host PF
>>> and the "VM PF" code that we have to deal with.
>>
>> That was my understanding.
>>  
>>> But that doesn't necessarily require that the VF parts of the
host have to be in
>>> nova-core as well, i.e. with the information we have we can
differentiate
>>> between PF, VF and PF in the VM (indicated by a device register).
>>
>> I'm not entirely sure what you mean by this..
>>
>> The driver to operate the function in "vGPU" mode as
indicated by the
>> register has to be in nova-core, since there is only one device ID.
>
> Yes, the PF driver on the host and the PF (from VM perspective) driver in
the VM
> have to be that same. But the VF driver on the host can still be a
seaparate
> one.
>
>>> > If you look at the VFIO driver RFC it basically does no
mediation, it
>>> > isn't intercepting MMIO - the guest sees the BARs
directly. Most of
>>> > the code is "profiling" from what I can tell. Some
config space
>>> > meddling.
>>> 
>>> Sure, there is no mediation in that sense, but it needs quite some
setup
>>> regardless, no?
>>>
>>> I thought there is a significant amount of semantics that is
different between
>>> booting the PF and the VF on the host.
>>
>> I think it would be good to have Zhi clarify more of this, but from
>> what I understand are at least three activites comingled all together:
>>
>>  1) Boot the PF in "vGPU" mode so it can enable SRIOV
>
> Ok, this might be where the confusion above comes from. When I talk about
> nova-core in vGPU mode I mean nova-core running in the VM on the (from VM
> perspective) PF.
>
> But you seem to mean nova-core running on the host PF with vGPU on top?
That of
> course has to be in nova-core.
>
>>  2) Enable SRIOV and profile VFs to allocate HW resources to them
>
> I think that's partially in nova-core and partially in vGPU; nova-core
providing
> the abstraction of the corresponding firmware / hardware interfaces and
vGPU
> controlling the semantics of the resource handling?
>
> This is what I thought vGPU has a secondary part for where it binds to
nova-core
> through the auxiliary bus, i.e. vGPU consisting out of two drivers
actually; the
> VFIO parts and a "per VF resource controller".
Forgot to add: But I think Zhi explained that this is not necessary and can be
controlled by the VFIO driver, i.e. the PCI driver that binds to the VF itself.
>>  3) VFIO variant driver to convert the VF into a "VM PF" with
whatever
>>     mediation and enhancement needed
>
> That should be vGPU only land.

On Thu, Oct 02, 2025 at 07:37:45PM +0200, Danilo Krummrich
wrote:
> > The driver to operate the function in "vGPU" mode as
indicated by the
> > register has to be in nova-core, since there is only one device ID.
> 
> Yes, the PF driver on the host and the PF (from VM perspective) driver in
the VM
> have to be that same. But the VF driver on the host can still be a
seaparate
> one.
In most cases it is going to be bound to a vfio driver..

However, if you actually want a DRM subsystem device on the VF without
a VM I don't know why you'd use a different driver than the one used
by the VM on the very same VF, with the very same register programming
model..
> > I think it would be good to have Zhi clarify more of this, but from
> > what I understand are at least three activites comingled all together:
> >
> >  1) Boot the PF in "vGPU" mode so it can enable SRIOV
> 
> Ok, this might be where the confusion above comes from. When I talk about
> nova-core in vGPU mode I mean nova-core running in the VM on the (from VM
> perspective) PF.
I would call this nova-core running on a VF (assigned to a VM)

Not sure "vgpu" is a helpful word here, lets try to talk about what
.ko's and struct device_drivers's the various codes should live in..
> But you seem to mean nova-core running on the host PF with vGPU on top?
That of
> course has to be in nova-core.
Yes, #1 would be implemented as part of nova-core.ko and it's
pci_driver. As I understand it around firmware loading nova-core has
to tell the FW if it wants to enable "vGPU" mode or not.

If it doesn't then the sriov_configure op should be inhibited and #2
disabled. If it does then sriov_configure should work, #2 is enabled,
and DRM on the PF is disabled.
> >  2) Enable SRIOV and profile VFs to allocate HW resources to them
> 
> I think that's partially in nova-core and partially in vGPU; nova-core
providing
> the abstraction of the corresponding firmware / hardware interfaces and
vGPU
> controlling the semantics of the resource handling?
> This is what I thought vGPU has a secondary part for where it binds to
nova-core
> through the auxiliary bus, i.e. vGPU consisting out of two drivers
actually; the
> VFIO parts and a "per VF resource controller".
This is certainly one option, you can put #2 in an aux driver of the
PF in a nova-sriov.ko module that is fully divorced from VFIO. It
might go along with a nova-fwctl.ko module too.

You could also just embed it in nova-core.ko and have it activate when
the PF is booted in "vGPU" mode.

Broadly I would suggest the latter. aux devices make most sense to
cross subsystems. Micro splitting a single driver with aux devices
will make more of a mess than required. Though a good motivating
reason would be if nova-srvio.ko is large.
> >  3) VFIO variant driver to convert the VF into a "VM PF"
with whatever
> >     mediation and enhancement needed
> 
> That should be vGPU only land.
I think it is clear this part should be in a vfio-pci-nova.ko

Then you have two more:

4) A PCI driver in a VM that creates a DRM subsystem device

This is nova-core.ko + nova-drm.ko

5) A VF driver that creates a DRM subsystem device without a VM

Zhi says the device can't do this, but lets assume it could, then I
would expect this to be nova-core.ko + nova-drm.ko, same as #4.

Jason