On Thu, Feb 13, 2020 at 10:58:44PM +0800, Jason Wang wrote:> > On 2020/2/13 ??9:41, Jason Gunthorpe wrote: > > On Thu, Feb 13, 2020 at 11:34:10AM +0800, Jason Wang wrote: > > > > > > You have dev, type or > > > > class to choose from. Type is rarely used and doesn't seem to be used > > > > by vdpa, so class seems the right choice > > > > > > > > Jason > > > Yes, but my understanding is class and bus are mutually exclusive. So we > > > can't add a class to a device which is already attached on a bus. > > While I suppose there are variations, typically 'class' devices are > > user facing things and 'bus' devices are internal facing (ie like a > > PCI device) > > > Though all vDPA devices have the same programming interface, but the > semantic is different. So it looks to me that use bus complies what > class.rst said: > > " > > Each device class defines a set of semantics and a programming interface > that devices of that class adhere to. Device drivers are the > implementation of that programming interface for a particular device on > a particular bus. > > "Here we are talking about the /dev/XX node that provides the programming interface. All the vdpa devices have the same basic chardev interface and discover any semantic variations 'in band'> > So why is this using a bus? VDPA is a user facing object, so the > > driver should create a class vhost_vdpa device directly, and that > > driver should live in the drivers/vhost/ directory. > > This is because we want vDPA to be generic for being used by different > drivers which is not limited to vhost-vdpa. E.g in this series, it allows > vDPA to be used by kernel virtio drivers. And in the future, we will > probably introduce more drivers in the future.I don't see how that connects with using a bus. Every class of virtio traffic is going to need a special HW driver to enable VDPA, that special driver can create the correct vhost side class device.> > For the PCI VF case this driver would bind to a PCI device like > > everything else > > > > For our future SF/ADI cases the driver would bind to some > > SF/ADI/whatever device on a bus. > > All these driver will still be bound to their own bus (PCI or other). And > what the driver needs is to present a vDPA device to virtual vDPA bus on > top.Again, I can't see any reason to inject a 'vdpa virtual bus' on top. That seems like mis-using the driver core. Jason
On Thu, Feb 13, 2020 at 11:05:42AM -0400, Jason Gunthorpe wrote:> On Thu, Feb 13, 2020 at 10:58:44PM +0800, Jason Wang wrote: > > > > On 2020/2/13 ??9:41, Jason Gunthorpe wrote: > > > On Thu, Feb 13, 2020 at 11:34:10AM +0800, Jason Wang wrote: > > > > > > > > You have dev, type or > > > > > class to choose from. Type is rarely used and doesn't seem to be used > > > > > by vdpa, so class seems the right choice > > > > > > > > > > Jason > > > > Yes, but my understanding is class and bus are mutually exclusive. So we > > > > can't add a class to a device which is already attached on a bus. > > > While I suppose there are variations, typically 'class' devices are > > > user facing things and 'bus' devices are internal facing (ie like a > > > PCI device) > > > > > > Though all vDPA devices have the same programming interface, but the > > semantic is different. So it looks to me that use bus complies what > > class.rst said: > > > > " > > > > Each device class defines a set of semantics and a programming interface > > that devices of that class adhere to. Device drivers are the > > implementation of that programming interface for a particular device on > > a particular bus. > > > > " > > Here we are talking about the /dev/XX node that provides the > programming interface. All the vdpa devices have the same basic > chardev interface and discover any semantic variations 'in band' > > > > So why is this using a bus? VDPA is a user facing object, so the > > > driver should create a class vhost_vdpa device directly, and that > > > driver should live in the drivers/vhost/ directory. > > > > This is because we want vDPA to be generic for being used by different > > drivers which is not limited to vhost-vdpa. E.g in this series, it allows > > vDPA to be used by kernel virtio drivers. And in the future, we will > > probably introduce more drivers in the future. > > I don't see how that connects with using a bus. > > Every class of virtio traffic is going to need a special HW driver to > enable VDPA, that special driver can create the correct vhost side > class device.That's just a ton of useless code duplication, and a good chance to have minor variations in implementations confusing userspace. Instead, each device implement the same interface, and then vhost sits on top.> > > For the PCI VF case this driver would bind to a PCI device like > > > everything else > > > > > > For our future SF/ADI cases the driver would bind to some > > > SF/ADI/whatever device on a bus. > > > > All these driver will still be bound to their own bus (PCI or other). And > > what the driver needs is to present a vDPA device to virtual vDPA bus on > > top. > > Again, I can't see any reason to inject a 'vdpa virtual bus' on > top. That seems like mis-using the driver core. > > JasonThat bus is exactly what Greg KH proposed. There are other ways to solve this I guess but this bikeshedding is getting tiring. Come on it's an internal kernel interface, if we feel it was a wrong direction to take we can change our minds later. Main thing is getting UAPI right. -- MST
On Thu, Feb 13, 2020 at 10:41:06AM -0500, Michael S. Tsirkin wrote:> On Thu, Feb 13, 2020 at 11:05:42AM -0400, Jason Gunthorpe wrote: > > On Thu, Feb 13, 2020 at 10:58:44PM +0800, Jason Wang wrote: > > > > > > On 2020/2/13 ??9:41, Jason Gunthorpe wrote: > > > > On Thu, Feb 13, 2020 at 11:34:10AM +0800, Jason Wang wrote: > > > > > > > > > > You have dev, type or > > > > > > class to choose from. Type is rarely used and doesn't seem to be used > > > > > > by vdpa, so class seems the right choice > > > > > > > > > > > > Jason > > > > > Yes, but my understanding is class and bus are mutually exclusive. So we > > > > > can't add a class to a device which is already attached on a bus. > > > > While I suppose there are variations, typically 'class' devices are > > > > user facing things and 'bus' devices are internal facing (ie like a > > > > PCI device) > > > > > > > > > Though all vDPA devices have the same programming interface, but the > > > semantic is different. So it looks to me that use bus complies what > > > class.rst said: > > > > > > " > > > > > > Each device class defines a set of semantics and a programming interface > > > that devices of that class adhere to. Device drivers are the > > > implementation of that programming interface for a particular device on > > > a particular bus. > > > > > > " > > > > Here we are talking about the /dev/XX node that provides the > > programming interface. All the vdpa devices have the same basic > > chardev interface and discover any semantic variations 'in band' > > > > > > So why is this using a bus? VDPA is a user facing object, so the > > > > driver should create a class vhost_vdpa device directly, and that > > > > driver should live in the drivers/vhost/ directory. > > > > > > This is because we want vDPA to be generic for being used by different > > > drivers which is not limited to vhost-vdpa. E.g in this series, it allows > > > vDPA to be used by kernel virtio drivers. And in the future, we will > > > probably introduce more drivers in the future. > > > > I don't see how that connects with using a bus. > > > > Every class of virtio traffic is going to need a special HW driver to > > enable VDPA, that special driver can create the correct vhost side > > class device. > > That's just a ton of useless code duplication, and a good chance > to have minor variations in implementations confusing > userspace.What? Why? This is how almost every user of the driver core works. I don't see how you get any duplication unless the subsystem core is badly done wrong. The 'class' is supposed to provide all the library functions to remove this duplication. Instead of plugging the HW driver in via some bus scheme every subsystem has its own 'ops' that the HW driver provides to the subsystem's class via subsystem_register() This is the *standard* pattern to use the driver core. This is almost there, it just has this extra bus part to convey the HW ops instead of directly.> Instead, each device implement the same interface, and then > vhost sits on top.Sure, but plugging in via ops/etc not via a bus and another struct device.> That bus is exactly what Greg KH proposed. There are other ways > to solve this I guess but this bikeshedding is getting tiring.This discussion was for a different goal, IMHO.> Come on it's an internal kernel interface, if we feel > it was a wrong direction to take we can change our minds later. > Main thing is getting UAPI right.This discussion started because the refcounting has been busted up in every version posted so far. It is not bikeshedding when these bugs are actually being caused by trying to abuse the driver core and shoehorn in a bus that isn't needed when a class is the correct thing to use. Jason
On 2020/2/13 ??11:05, Jason Gunthorpe wrote:> On Thu, Feb 13, 2020 at 10:58:44PM +0800, Jason Wang wrote: >> On 2020/2/13 ??9:41, Jason Gunthorpe wrote: >>> On Thu, Feb 13, 2020 at 11:34:10AM +0800, Jason Wang wrote: >>> >>>>> You have dev, type or >>>>> class to choose from. Type is rarely used and doesn't seem to be used >>>>> by vdpa, so class seems the right choice >>>>> >>>>> Jason >>>> Yes, but my understanding is class and bus are mutually exclusive. So we >>>> can't add a class to a device which is already attached on a bus. >>> While I suppose there are variations, typically 'class' devices are >>> user facing things and 'bus' devices are internal facing (ie like a >>> PCI device) >> >> Though all vDPA devices have the same programming interface, but the >> semantic is different. So it looks to me that use bus complies what >> class.rst said: >> >> " >> >> Each device class defines a set of semantics and a programming interface >> that devices of that class adhere to. Device drivers are the >> implementation of that programming interface for a particular device on >> a particular bus. >> >> " > Here we are talking about the /dev/XX node that provides the > programming interface.I'm confused here, are you suggesting to use class to create char device in vhost-vdpa? That's fine but the comment should go for vhost-vdpa patch.> All the vdpa devices have the same basic > chardev interface and discover any semantic variations 'in band'That's not true, char interface is only used for vhost. Kernel virtio driver does not need char dev but a device on the virtio bus.> >>> So why is this using a bus? VDPA is a user facing object, so the >>> driver should create a class vhost_vdpa device directly, and that >>> driver should live in the drivers/vhost/ directory. >> >> This is because we want vDPA to be generic for being used by different >> drivers which is not limited to vhost-vdpa. E.g in this series, it allows >> vDPA to be used by kernel virtio drivers. And in the future, we will >> probably introduce more drivers in the future. > I don't see how that connects with using a bus.This is demonstrated in the virito-vdpa driver. So if you want to use kernel virito driver for vDPA device, a bus is most straight forward.> > Every class of virtio traffic is going to need a special HW driver to > enable VDPA, that special driver can create the correct vhost side > class device.Are you saying, e.g it's the charge of IFCVF driver to create vhost char dev and other stuffs?> >>> For the PCI VF case this driver would bind to a PCI device like >>> everything else >>> >>> For our future SF/ADI cases the driver would bind to some >>> SF/ADI/whatever device on a bus. >> All these driver will still be bound to their own bus (PCI or other). And >> what the driver needs is to present a vDPA device to virtual vDPA bus on >> top. > Again, I can't see any reason to inject a 'vdpa virtual bus' on > top. That seems like mis-using the driver core.I don't think so. Vhost is not the only programming interface for vDPA. We don't want a device that can only work for userspace drivers and only have a single set of userspace APIs. Thanks> > Jason >
On Fri, Feb 14, 2020 at 11:23:27AM +0800, Jason Wang wrote:> > > Though all vDPA devices have the same programming interface, but the > > > semantic is different. So it looks to me that use bus complies what > > > class.rst said: > > > > > > " > > > > > > Each device class defines a set of semantics and a programming interface > > > that devices of that class adhere to. Device drivers are the > > > implementation of that programming interface for a particular device on > > > a particular bus. > > > > > > " > > Here we are talking about the /dev/XX node that provides the > > programming interface. > > > I'm confused here, are you suggesting to use class to create char device in > vhost-vdpa? That's fine but the comment should go for vhost-vdpa patch.Certainly yes, something creating many char devs should have a class. That makes the sysfs work as expected I suppose this is vhost user? I admit I don't really see how this vhost stuff works, all I see are global misc devices? Very unusual for a new subsystem to be using global misc devices.. I would have expected that a single VDPA device comes out as a single char dev linked to only that VDPA device.> > All the vdpa devices have the same basic > > chardev interface and discover any semantic variations 'in band' > > That's not true, char interface is only used for vhost. Kernel virtio driver > does not need char dev but a device on the virtio bus.Okay, this is fine, but why do you need two busses to accomplish this? Shouldn't the 'struct virito_device' be the plug in point for HW drivers I was talking about - and from there a vhost-user can connect to the struct virtio_device to give it a char dev or a kernel driver can connect to link it to another subsystem? It is easy to see something is going wrong with this design because the drivers/virtio/virtio_vdpa.c mainly contains a bunch of trampoline functions reflecting identical calls from one ops struct to a different ops struct. This suggests the 'vdpa' is some subclass of 'virtio' and it is possibly better to model it by extending 'struct virito_device' to include the vdpa specific stuff. Where does the vhost-user char dev get invovled in with the v2 series? Is that included?> > Every class of virtio traffic is going to need a special HW driver to > > enable VDPA, that special driver can create the correct vhost side > > class device. > > Are you saying, e.g it's the charge of IFCVF driver to create vhost char dev > and other stuffs?No. Jason