thr3ads.net - Linux Virtualization - [PATCH v2] vhost: introduce mdev based hardware backend [Oct 2019]

If this information is useful, please help other people find it:
Share via:

Jason Wang

2019-Oct-28 03:50 UTC

[PATCH v2] vhost: introduce mdev based hardware backend

On 2019/10/28 ??9:58, Tiwei Bie wrote:> On Fri, Oct 25, 2019 at 08:16:26AM -0400, Michael S. Tsirkin wrote:
>> On Fri, Oct 25, 2019 at 05:54:55PM +0800, Jason Wang wrote:
>>> On 2019/10/24 ??6:42, Jason Wang wrote:
>>>> Yes.
>>>>
>>>>
>>>>>  ? And we should try to avoid
>>>>> putting ctrl vq and Rx/Tx vqs in the same DMA space to
prevent
>>>>> guests having the chance to bypass the host (e.g. QEMU) to
>>>>> setup the backend accelerator directly.
>>>>
>>>> That's really good point.? So when "vhost" type
is created, parent
>>>> should assume addr of ctrl_vq is hva.
>>>>
>>>> Thanks
>>>
>>> This works for vhost but not virtio since there's no way for
virtio kernel
>>> driver to differ ctrl_vq with the rest when doing DMA map. One
possible
>>> solution is to provide DMA domain isolation between virtqueues.
Then ctrl vq
>>> can use its dedicated DMA domain for the work.
> It might not be a bad idea to let the parent drivers distinguish
> between virtio-mdev mdevs and vhost-mdev mdevs in ctrl-vq handling
> by mdev's class id.

Yes, that should work, I have something probable better, see below.

>
>>> Anyway, this could be done in the future. We can have a version
first that
>>> doesn't support ctrl_vq.
> +1, thanks
>
>>> Thanks
>> Well no ctrl_vq implies either no offloads, or no XDP (since XDP needs
>> to disable offloads dynamically).
>>
>>          if (!virtio_has_feature(vi->vdev,
VIRTIO_NET_F_CTRL_GUEST_OFFLOADS)
>>              && (virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_TSO4) ||
>>                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_TSO6) ||
>>                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_ECN) ||
>>                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_UFO) ||
>>                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_CSUM))) {
>>                  NL_SET_ERR_MSG_MOD(extack, "Can't set XDP
while host is implementing LRO/CSUM, disable LRO/CSUM first");
>>                  return -EOPNOTSUPP;
>>          }
>>
>> neither is very attractive.
>>
>> So yes ok just for development but we do need to figure out how it will
>> work down the road in production.
> Totally agree.
>
>> So really this specific virtio net device does not support control vq,
>> instead it supports a different transport specific way to send commands
>> to device.
>>
>> Some kind of extension to the transport? Ideas?

So it's basically an issue of isolating DMA domains. Maybe we can start 
with transport API for querying per vq DMA domain/ASID?

- for vhost-mdev, userspace can query the DMA domain for each specific 
virtqueue. For control vq, mdev can return id for software domain, for 
the rest mdev will return id of VFIO domain. Then userspace know that it 
should use different API for preparing the virtqueue, e.g for vq other 
than control vq, it should use VFIO DMA API. The control vq it should 
use hva instead.

- for virito-mdev, we can introduce per-vq DMA device, and route DMA 
mapping request for control vq back to mdev instead of the hardware. (We 
can wrap them into library or helpers to ease the development of vendor 
physical drivers).

Thanks

>>
>>
>> -- 
>> MST

Tiwei Bie

2019-Oct-29 09:57 UTC

head link

[PATCH v2] vhost: introduce mdev based hardware backend

On Mon, Oct 28, 2019 at 11:50:49AM +0800, Jason Wang
wrote:> On 2019/10/28 ??9:58, Tiwei Bie wrote:
> > On Fri, Oct 25, 2019 at 08:16:26AM -0400, Michael S. Tsirkin wrote:
> > > On Fri, Oct 25, 2019 at 05:54:55PM +0800, Jason Wang wrote:
> > > > On 2019/10/24 ??6:42, Jason Wang wrote:
> > > > > Yes.
> > > > > 
> > > > > 
> > > > > >  ? And we should try to avoid
> > > > > > putting ctrl vq and Rx/Tx vqs in the same DMA
space to prevent
> > > > > > guests having the chance to bypass the host (e.g.
QEMU) to
> > > > > > setup the backend accelerator directly.
> > > > > 
> > > > > That's really good point.? So when
"vhost" type is created, parent
> > > > > should assume addr of ctrl_vq is hva.
> > > > > 
> > > > > Thanks
> > > > 
> > > > This works for vhost but not virtio since there's no way
for virtio kernel
> > > > driver to differ ctrl_vq with the rest when doing DMA map.
One possible
> > > > solution is to provide DMA domain isolation between
virtqueues. Then ctrl vq
> > > > can use its dedicated DMA domain for the work.
> > It might not be a bad idea to let the parent drivers distinguish
> > between virtio-mdev mdevs and vhost-mdev mdevs in ctrl-vq handling
> > by mdev's class id.
> 
> 
> Yes, that should work, I have something probable better, see below.
> 
> 
> > 
> > > > Anyway, this could be done in the future. We can have a
version first that
> > > > doesn't support ctrl_vq.
> > +1, thanks
> > 
> > > > Thanks
> > > Well no ctrl_vq implies either no offloads, or no XDP (since XDP
needs
> > > to disable offloads dynamically).
> > > 
> > >          if (!virtio_has_feature(vi->vdev,
VIRTIO_NET_F_CTRL_GUEST_OFFLOADS)
> > >              && (virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_TSO4) ||
> > >                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_TSO6) ||
> > >                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_ECN) ||
> > >                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_UFO) ||
> > >                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_CSUM))) {
> > >                  NL_SET_ERR_MSG_MOD(extack, "Can't set
XDP while host is implementing LRO/CSUM, disable LRO/CSUM first");
> > >                  return -EOPNOTSUPP;
> > >          }
> > > 
> > > neither is very attractive.
> > > 
> > > So yes ok just for development but we do need to figure out how
it will
> > > work down the road in production.
> > Totally agree.
> > 
> > > So really this specific virtio net device does not support
control vq,
> > > instead it supports a different transport specific way to send
commands
> > > to device.
> > > 
> > > Some kind of extension to the transport? Ideas?
> 
> 
> So it's basically an issue of isolating DMA domains. Maybe we can start
with
> transport API for querying per vq DMA domain/ASID?
> 
> - for vhost-mdev, userspace can query the DMA domain for each specific
> virtqueue. For control vq, mdev can return id for software domain, for the
> rest mdev will return id of VFIO domain. Then userspace know that it should
> use different API for preparing the virtqueue, e.g for vq other than
control
> vq, it should use VFIO DMA API. The control vq it should use hva instead.
> 
> - for virito-mdev, we can introduce per-vq DMA device, and route DMA
mapping
> request for control vq back to mdev instead of the hardware. (We can wrap
> them into library or helpers to ease the development of vendor physical
> drivers).
Thanks for this proposal! I'm thinking about it these days.
I think it might be too complicated. I'm wondering whether we
can have something simpler. I will post a RFC patch to show
my idea today.

Thanks,
Tiwei
> 
> Thanks
> 
> 
> > > 
> > > 
> > > -- 
> > > MST
>

Jason Wang

2019-Oct-29 10:48 UTC

head link

[PATCH v2] vhost: introduce mdev based hardware backend

On 2019/10/29 ??5:57, Tiwei Bie wrote:> On Mon, Oct 28, 2019 at 11:50:49AM +0800, Jason Wang wrote:
>> On 2019/10/28 ??9:58, Tiwei Bie wrote:
>>> On Fri, Oct 25, 2019 at 08:16:26AM -0400, Michael S. Tsirkin wrote:
>>>> On Fri, Oct 25, 2019 at 05:54:55PM +0800, Jason Wang wrote:
>>>>> On 2019/10/24 ??6:42, Jason Wang wrote:
>>>>>> Yes.
>>>>>>
>>>>>>
>>>>>>>  ? And we should try to avoid
>>>>>>> putting ctrl vq and Rx/Tx vqs in the same DMA space
to prevent
>>>>>>> guests having the chance to bypass the host (e.g.
QEMU) to
>>>>>>> setup the backend accelerator directly.
>>>>>> That's really good point.? So when
"vhost" type is created, parent
>>>>>> should assume addr of ctrl_vq is hva.
>>>>>>
>>>>>> Thanks
>>>>> This works for vhost but not virtio since there's no
way for virtio kernel
>>>>> driver to differ ctrl_vq with the rest when doing DMA map.
One possible
>>>>> solution is to provide DMA domain isolation between
virtqueues. Then ctrl vq
>>>>> can use its dedicated DMA domain for the work.
>>> It might not be a bad idea to let the parent drivers distinguish
>>> between virtio-mdev mdevs and vhost-mdev mdevs in ctrl-vq handling
>>> by mdev's class id.
>> Yes, that should work, I have something probable better, see below.
>>
>>
>>>>> Anyway, this could be done in the future. We can have a
version first that
>>>>> doesn't support ctrl_vq.
>>> +1, thanks
>>>
>>>>> Thanks
>>>> Well no ctrl_vq implies either no offloads, or no XDP (since
XDP needs
>>>> to disable offloads dynamically).
>>>>
>>>>          if (!virtio_has_feature(vi->vdev,
VIRTIO_NET_F_CTRL_GUEST_OFFLOADS)
>>>>              && (virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_TSO4) ||
>>>>                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_TSO6) ||
>>>>                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_ECN) ||
>>>>                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_UFO) ||
>>>>                  virtio_has_feature(vi->vdev,
VIRTIO_NET_F_GUEST_CSUM))) {
>>>>                  NL_SET_ERR_MSG_MOD(extack, "Can't set
XDP while host is implementing LRO/CSUM, disable LRO/CSUM first");
>>>>                  return -EOPNOTSUPP;
>>>>          }
>>>>
>>>> neither is very attractive.
>>>>
>>>> So yes ok just for development but we do need to figure out how
it will
>>>> work down the road in production.
>>> Totally agree.
>>>
>>>> So really this specific virtio net device does not support
control vq,
>>>> instead it supports a different transport specific way to send
commands
>>>> to device.
>>>>
>>>> Some kind of extension to the transport? Ideas?
>> So it's basically an issue of isolating DMA domains. Maybe we can
start with
>> transport API for querying per vq DMA domain/ASID?
>>
>> - for vhost-mdev, userspace can query the DMA domain for each specific
>> virtqueue. For control vq, mdev can return id for software domain, for
the
>> rest mdev will return id of VFIO domain. Then userspace know that it
should
>> use different API for preparing the virtqueue, e.g for vq other than
control
>> vq, it should use VFIO DMA API. The control vq it should use hva
instead.
>>
>> - for virito-mdev, we can introduce per-vq DMA device, and route DMA
mapping
>> request for control vq back to mdev instead of the hardware. (We can
wrap
>> them into library or helpers to ease the development of vendor physical
>> drivers).
> Thanks for this proposal! I'm thinking about it these days.
> I think it might be too complicated. I'm wondering whether we
> can have something simpler. I will post a RFC patch to show
> my idea today.

Thanks, will check.

Btw, for virtio-mdev, the change should be very minimal, will post an
RFC as well. For vhost-mdev, it could be just a helper to return an ID
for DMA domain like ID_VFIO or ID_HVA.

Or a more straightforward way is to force queues like control vq to use PA.

>
> Thanks,
> Tiwei
>

Maybe Matching Threads

Search for more reasonably related threads

Linux Virtualization - Oct 2019 - [PATCH v2] vhost: introduce mdev based hardware backend

[PATCH v2] vhost: introduce mdev based hardware backend

[PATCH v2] vhost: introduce mdev based hardware backend

[PATCH v2] vhost: introduce mdev based hardware backend

Maybe Matching Threads