Linux Virtualization - Oct 2021 - vDPA bus driver selection

On Wed, Oct 27, 2021 at 03:56:16PM +0000, Parav Pandit
wrote:
>Hi Stefano,
>
>> From: Stefano Garzarella <sgarzare at redhat.com>
>> Sent: Wednesday, October 27, 2021 9:17 PM
>> To: Parav Pandit <parav at nvidia.com>
>> Cc: Jason Wang <jasowang at redhat.com>; Michael Tsirkin <mst
at redhat.com>;
>> Linux Virtualization <virtualization at
lists.linux-foundation.org>; Eli Cohen
>> <elic at nvidia.com>
>> Subject: Re: vDPA bus driver selection
>>
>> Hi Parav,
>>
>> On Wed, Oct 27, 2021 at 03:21:15PM +0000, Parav Pandit wrote:
>> >Hi Stefano,
>> >
>> >> From: Stefano Garzarella <sgarzare at redhat.com>
>> >> Sent: Wednesday, October 27, 2021 8:04 PM
>> >>
>> >> Hi folks,
>> >> I was trying to understand if we have a way to specify which
vDPA bus
>> >> driver (e.g. vhost-vdpa, virtio-vdpa) a device should use.
>> >> IIUC we don't have it, and the first registered driver is
used when a
>> >> new device is registered.
>> >>
>> >> I was thinking if it makes sense to extend the management API
to
>> >> specify which bus driver to use for a device. A use case could
be for
>> >> example a single host handling VMs and bare-metal containers,
so we
>> >> would have both virtio-vdpa and vhost-vdpa loaded and we want
to
>> >> attach some devices to VMs through vhost-vdpa and others to
containers
>> through virtio-vdpa.
>> >>
>> >> What do you think?
>> >>
>> >One option is, user keeps the drivers_autoprobe disabled for the
vdpa
>> >bus using,
>> >
>> >$ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac
>> >00:11:22:33:44:55 $ echo 0 > /sys/bus/vdpa/drivers_autoprobe
>> >
>> >And after vdpa device creation, it manually binds to the desired
driver
>> >such as,
>> >
>> >$ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind
>> >Or
>> >$ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind
>>
>> Cool, I didn't know that. This is very useful, but do you think it
might be better
>> to integrate it with the netlink API and specify at creation which bus
driver to
>> use?
>I think it is useful; for vduse case we need the ability to say
"none"
>as well and when nothing specified it should be default driver.
Yep, the default can be the actual behaviour.
>
>More than netlink, I think we need the ability in the core kernel to 
>make this choice.
Okay, but I think we can include that in the vdpa tool.
>I haven't explored what is already available to make that happen.
Me too, I only saw the .match() callback of "struct bus_type" that
could
be used for this purpose.
>If/once its available, I am sure it has more users than just vdpa.
Make sense. I'll spend some time next week.

Thanks,
Stefano

> From: Stefano Garzarella <sgarzare at redhat.com>
> Sent: Thursday, October 28, 2021 3:08 PM
> >> >$ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac
> >> >00:11:22:33:44:55 $ echo 0 >
/sys/bus/vdpa/drivers_autoprobe
> >> >
> >> >And after vdpa device creation, it manually binds to the
desired
> >> >driver such as,
> >> >
> >> >$ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind
> >> >Or
> >> >$ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind
> >>
> >> Cool, I didn't know that. This is very useful, but do you
think it
> >> might be better to integrate it with the netlink API and specify
at
> >> creation which bus driver to use?
> >I think it is useful; for vduse case we need the ability to say
"none"
> >as well and when nothing specified it should be default driver.
> 
> Yep, the default can be the actual behaviour.
> 
> >
> >More than netlink, I think we need the ability in the core kernel to
> >make this choice.
> 
> Okay, but I think we can include that in the vdpa tool.
If functionality and interface exists in other part of the it is hard to wrap
that in vdpa tool that does the duplicate work.
> 
> >I haven't explored what is already available to make that happen.
> 
> Me too, I only saw the .match() callback of "struct bus_type"
that could be used
> for this purpose.
> 
> >If/once its available, I am sure it has more users than just vdpa.
> 
> Make sense. I'll spend some time next week.
Ok. yeah, may be a desired driver can be stored in the vdpa_device that match()
routine can use.
And if that driver is not available,  it returns EPROBE_DEFER; so whenever such
driver is loaded it can be bind.

But I think before wrapping something in vdpa, need to find a reason why the
current interface doesn't solve the problem and also to figure out plumbing.