Linux Virtualization - Apr 2018 - [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend

> -----Original Message-----
> From: Paolo Bonzini [mailto:pbonzini at redhat.com]
> Sent: Tuesday, April 10, 2018 3:52 PM
> To: Bie, Tiwei <tiwei.bie at intel.com>; Jason Wang <jasowang at
redhat.com>
> Cc: mst at redhat.com; alex.williamson at redhat.com; ddutile at
redhat.com;
> Duyck, Alexander H <alexander.h.duyck at intel.com>; virtio-dev at
lists.oasis-
> open.org; linux-kernel at vger.kernel.org; kvm at vger.kernel.org;
> virtualization at lists.linux-foundation.org; netdev at vger.kernel.org;
Daly, Dan
> <dan.daly at intel.com>; Liang, Cunming <cunming.liang at
intel.com>; Wang,
> Zhihong <zhihong.wang at intel.com>; Tan, Jianfeng <jianfeng.tan
at intel.com>;
> Wang, Xiao W <xiao.w.wang at intel.com>
> Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware
> vhost backend
> 
> On 10/04/2018 06:57, Tiwei Bie wrote:
> >> So you just move the abstraction layer from qemu to kernel, and
you
> >> still need different drivers in kernel for different device
> >> interfaces of accelerators. This looks even more complex than
leaving
> >> it in qemu. As you said, another idea is to implement userspace
vhost
> >> backend for accelerators which seems easier and could co-work with
> >> other parts of qemu without inventing new type of messages.
> >
> > I'm not quite sure. Do you think it's acceptable to add
various vendor
> > specific hardware drivers in QEMU?
> 
> I think so.  We have vendor-specific quirks, and at some point there was an
> idea of using quirks to implement (vendor-specific) live migration support
for
> assigned devices.
Vendor-specific quirks of accessing VGA is a small portion. Other major portions
are still handled by guest driver.

While in this case, when saying various vendor specific drivers in QEMU, it says
QEMU takes over and provides the entire user space device drivers. Some parts
are even not relevant to vhost, they're basic device function enabling.
Moreover, it could be different kinds of devices(network/block/...) under vhost.
No matter # of vendors or # of types, total LOC is not small.

The idea is to avoid introducing these extra complexity out of QEMU, keeping
vhost adapter simple. As vhost protocol is de factor standard, it leverages
kernel device driver to provide the diversity. Changing once in QEMU, then it
supports multi-vendor devices whose drivers naturally providing kernel driver
there.

If QEMU is going to build a user space driver framework there, we're open
mind on that, even leveraging DPDK as the underlay library. Looking forward to
more others' comments from community.

Steve
> 
> Paolo

On Tue, Apr 10, 2018 at 09:23:53AM +0000, Liang, Cunming
wrote:
> 
> 
> > -----Original Message-----
> > From: Paolo Bonzini [mailto:pbonzini at redhat.com]
> > Sent: Tuesday, April 10, 2018 3:52 PM
> > To: Bie, Tiwei <tiwei.bie at intel.com>; Jason Wang <jasowang
at redhat.com>
> > Cc: mst at redhat.com; alex.williamson at redhat.com; ddutile at
redhat.com;
> > Duyck, Alexander H <alexander.h.duyck at intel.com>; virtio-dev
at lists.oasis-
> > open.org; linux-kernel at vger.kernel.org; kvm at vger.kernel.org;
> > virtualization at lists.linux-foundation.org; netdev at
vger.kernel.org; Daly, Dan
> > <dan.daly at intel.com>; Liang, Cunming <cunming.liang at
intel.com>; Wang,
> > Zhihong <zhihong.wang at intel.com>; Tan, Jianfeng
<jianfeng.tan at intel.com>;
> > Wang, Xiao W <xiao.w.wang at intel.com>
> > Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based
hardware
> > vhost backend
> > 
> > On 10/04/2018 06:57, Tiwei Bie wrote:
> > >> So you just move the abstraction layer from qemu to kernel,
and you
> > >> still need different drivers in kernel for different device
> > >> interfaces of accelerators. This looks even more complex than
leaving
> > >> it in qemu. As you said, another idea is to implement
userspace vhost
> > >> backend for accelerators which seems easier and could co-work
with
> > >> other parts of qemu without inventing new type of messages.
> > >
> > > I'm not quite sure. Do you think it's acceptable to add
various vendor
> > > specific hardware drivers in QEMU?
> > 
> > I think so.  We have vendor-specific quirks, and at some point there
was an
> > idea of using quirks to implement (vendor-specific) live migration
support for
> > assigned devices.
> 
> Vendor-specific quirks of accessing VGA is a small portion. Other major
portions are still handled by guest driver.
> 
> While in this case, when saying various vendor specific drivers in QEMU, it
says QEMU takes over and provides the entire user space device drivers. Some
parts are even not relevant to vhost, they're basic device function
enabling. Moreover, it could be different kinds of devices(network/block/...)
under vhost. No matter # of vendors or # of types, total LOC is not small.
> 
> The idea is to avoid introducing these extra complexity out of QEMU,
keeping vhost adapter simple. As vhost protocol is de factor standard, it
leverages kernel device driver to provide the diversity. Changing once in QEMU,
then it supports multi-vendor devices whose drivers naturally providing kernel
driver there.
> 
> If QEMU is going to build a user space driver framework there, we're
open mind on that, even leveraging DPDK as the underlay library. Looking forward
to more others' comments from community.
> 
> Steve
Dependency on a kernel driver is fine IMHO. It's the dependency on a
DPDK backend that makes people unhappy, since the functionality
in question is setup-time only.

As others said, we do not need to go overeboard. A couple of small
vendor-specific quirks in qemu isn't a big deal.
> > 
> > Paolo

> -----Original Message-----
> From: Michael S. Tsirkin [mailto:mst at redhat.com]
> Sent: Tuesday, April 10, 2018 9:36 PM
> To: Liang, Cunming <cunming.liang at intel.com>
> Cc: Paolo Bonzini <pbonzini at redhat.com>; Bie, Tiwei <tiwei.bie
at intel.com>;
> Jason Wang <jasowang at redhat.com>; alex.williamson at redhat.com;
> ddutile at redhat.com; Duyck, Alexander H <alexander.h.duyck at
intel.com>;
> virtio-dev at lists.oasis-open.org; linux-kernel at vger.kernel.org;
> kvm at vger.kernel.org; virtualization at lists.linux-foundation.org;
> netdev at vger.kernel.org; Daly, Dan <dan.daly at intel.com>; Wang,
Zhihong
> <zhihong.wang at intel.com>; Tan, Jianfeng <jianfeng.tan at
intel.com>; Wang, Xiao
> W <xiao.w.wang at intel.com>
> Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware
vhost
> backend
> 
> On Tue, Apr 10, 2018 at 09:23:53AM +0000, Liang, Cunming wrote:
> >
> >
> > > -----Original Message-----
> > > From: Paolo Bonzini [mailto:pbonzini at redhat.com]
> > > Sent: Tuesday, April 10, 2018 3:52 PM
> > > To: Bie, Tiwei <tiwei.bie at intel.com>; Jason Wang
> > > <jasowang at redhat.com>
> > > Cc: mst at redhat.com; alex.williamson at redhat.com; ddutile at
redhat.com;
> > > Duyck, Alexander H <alexander.h.duyck at intel.com>;
> > > virtio-dev at lists.oasis- open.org; linux-kernel at
vger.kernel.org;
> > > kvm at vger.kernel.org; virtualization at
lists.linux-foundation.org;
> > > netdev at vger.kernel.org; Daly, Dan <dan.daly at
intel.com>; Liang,
> > > Cunming <cunming.liang at intel.com>; Wang, Zhihong
> > > <zhihong.wang at intel.com>; Tan, Jianfeng <jianfeng.tan
at intel.com>;
> > > Wang, Xiao W <xiao.w.wang at intel.com>
> > > Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based
> > > hardware vhost backend
> > >
> > > On 10/04/2018 06:57, Tiwei Bie wrote:
> > > >> So you just move the abstraction layer from qemu to
kernel, and
> > > >> you still need different drivers in kernel for different
device
> > > >> interfaces of accelerators. This looks even more complex
than
> > > >> leaving it in qemu. As you said, another idea is to
implement
> > > >> userspace vhost backend for accelerators which seems
easier and
> > > >> could co-work with other parts of qemu without inventing
new type of
> messages.
> > > >
> > > > I'm not quite sure. Do you think it's acceptable to
add various
> > > > vendor specific hardware drivers in QEMU?
> > >
> > > I think so.  We have vendor-specific quirks, and at some point
there
> > > was an idea of using quirks to implement (vendor-specific) live
> > > migration support for assigned devices.
> >
> > Vendor-specific quirks of accessing VGA is a small portion. Other
major portions
> are still handled by guest driver.
> >
> > While in this case, when saying various vendor specific drivers in
QEMU, it says
> QEMU takes over and provides the entire user space device drivers. Some
parts
> are even not relevant to vhost, they're basic device function enabling.
Moreover,
> it could be different kinds of devices(network/block/...) under vhost. No
matter
> # of vendors or # of types, total LOC is not small.
> >
> > The idea is to avoid introducing these extra complexity out of QEMU,
keeping
> vhost adapter simple. As vhost protocol is de factor standard, it leverages
kernel
> device driver to provide the diversity. Changing once in QEMU, then it
supports
> multi-vendor devices whose drivers naturally providing kernel driver there.
> >
> > If QEMU is going to build a user space driver framework there,
we're open mind
> on that, even leveraging DPDK as the underlay library. Looking forward to
more
> others' comments from community.
> >
> > Steve
> 
> Dependency on a kernel driver is fine IMHO. It's the dependency on a
DPDK
> backend that makes people unhappy, since the functionality in question is
setup-
> time only.
Agreed, we don't see dependency on kernel driver is a problem.

mdev based vhost backend (this patch set) is independent with vhost-user
extension patch set. In fact, there're a few vhost-user providers, DPDK
librte_vhost is one of them. FD.IO/VPP and snabbswitch have their own vhost-user
providers. So I can't agree on vhost-user extension patch depends on DPDK
backend. But anyway, that's the topic of another mail thread.
> 
> As others said, we do not need to go overeboard. A couple of small vendor-
> specific quirks in qemu isn't a big deal.
It's quite challenge to identify it's small or not, there's no
uniform metric.

It's only dependent on QEMU itself, that's the obvious benefit. Tradeoff
is to build the entire device driver. We don't object to do that in QEMU,
but wanna make sure to understand the boundary size clearly.
> 
> > >
> > > Paolo

On Tue, Apr 10, 2018 at 09:23:53AM +0000, Liang, Cunming
wrote:
> If QEMU is going to build a user space driver framework there, we're
open mind on that, even leveraging DPDK as the underlay library. Looking forward
to more others' comments from community.
There is already an NVMe VFIO driver in QEMU (see block/nvme.c).  So in
principle there's no reason against userspace drivers in QEMU.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: not available
URL:
<http://lists.linuxfoundation.org/pipermail/virtualization/attachments/20180411/d502646e/attachment.sig>

On 2018?04?10? 17:23, Liang, Cunming wrote:
>
>> -----Original Message-----
>> From: Paolo Bonzini [mailto:pbonzini at redhat.com]
>> Sent: Tuesday, April 10, 2018 3:52 PM
>> To: Bie, Tiwei <tiwei.bie at intel.com>; Jason Wang <jasowang
at redhat.com>
>> Cc: mst at redhat.com; alex.williamson at redhat.com; ddutile at
redhat.com;
>> Duyck, Alexander H <alexander.h.duyck at intel.com>; virtio-dev
at lists.oasis-
>> open.org; linux-kernel at vger.kernel.org; kvm at vger.kernel.org;
>> virtualization at lists.linux-foundation.org; netdev at
vger.kernel.org; Daly, Dan
>> <dan.daly at intel.com>; Liang, Cunming <cunming.liang at
intel.com>; Wang,
>> Zhihong <zhihong.wang at intel.com>; Tan, Jianfeng
<jianfeng.tan at intel.com>;
>> Wang, Xiao W <xiao.w.wang at intel.com>
>> Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based
hardware
>> vhost backend
>>
>> On 10/04/2018 06:57, Tiwei Bie wrote:
>>>> So you just move the abstraction layer from qemu to kernel, and
you
>>>> still need different drivers in kernel for different device
>>>> interfaces of accelerators. This looks even more complex than
leaving
>>>> it in qemu. As you said, another idea is to implement userspace
vhost
>>>> backend for accelerators which seems easier and could co-work
with
>>>> other parts of qemu without inventing new type of messages.
>>> I'm not quite sure. Do you think it's acceptable to add
various vendor
>>> specific hardware drivers in QEMU?
>> I think so.  We have vendor-specific quirks, and at some point there
was an
>> idea of using quirks to implement (vendor-specific) live migration
support for
>> assigned devices.
> Vendor-specific quirks of accessing VGA is a small portion. Other major
portions are still handled by guest driver.
>
> While in this case, when saying various vendor specific drivers in QEMU, it
says QEMU takes over and provides the entire user space device drivers. Some
parts are even not relevant to vhost, they're basic device function
enabling. Moreover, it could be different kinds of devices(network/block/...)
under vhost. No matter # of vendors or # of types, total LOC is not small.
>
> The idea is to avoid introducing these extra complexity out of QEMU,
keeping vhost adapter simple. As vhost protocol is de factor standard, it
leverages kernel device driver to provide the diversity. Changing once in QEMU,
then it supports multi-vendor devices whose drivers naturally providing kernel
driver there.
Let me clarify my question, it's not qemu vs kenrel but userspace vs 
kernel. It could be a library which could be linked to qemu. Doing it in 
userspace have the following obvious advantages:

- attack surface was limited to userspace
- easier to be maintained (compared to kernel driver)
- easier to be extended without introducing new userspace/kernel interfaces
- not tied to a specific operating system

If we want to do it in the kernel, need to consider to unify code 
between mdev device driver and generic driver. For net driver, maybe we 
can even consider to do it on top of exist drivers.
>
> If QEMU is going to build a user space driver framework there, we're
open mind on that, even leveraging DPDK as the underlay library. Looking forward
to more others' comments from community.
I'm doing this now by implementing vhost inside qemu IOThreads. Hope I 
can post RFC in few months.

Thanks
> Steve
>
>> Paolo