Virtualization - Feb 2023 - virtio-fs: adding support for multi-queue

On Tue, Feb 07, 2023 at 02:53:58PM -0500, Vivek Goyal
wrote:
> On Tue, Feb 07, 2023 at 02:45:39PM -0500, Stefan Hajnoczi wrote:
> > On Tue, Feb 07, 2023 at 11:14:46AM +0100, Peter-Jan Gootzen wrote:
> > > Hi,
> > > 
> 
> [cc German]
> 
> > > For my MSc thesis project in collaboration with IBM
> > > (https://github.com/IBM/dpu-virtio-fs) we are looking to improve
the
> > > performance of the virtio-fs driver in high throughput scenarios.
We think
> > > the main bottleneck is the fact that the virtio-fs driver does
not support
> > > multi-queue (while the spec does). A big factor in this is that
our setup on
> > > the virtio-fs device-side (a DPU) does not easily allow multiple
cores to
> > > tend to a single virtio queue.
> 
> This is an interesting limitation in DPU.
Virtqueues are single-consumer queues anyway. Sharing them between
multiple threads would be expensive. I think using multiqueue is natural
and not specific to DPUs.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL:
<http://lists.linuxfoundation.org/pipermail/virtualization/attachments/20230207/c81a9840/attachment.sig>

On Tue, Feb 07, 2023 at 04:32:02PM -0500, Stefan Hajnoczi
wrote:
> On Tue, Feb 07, 2023 at 02:53:58PM -0500, Vivek Goyal wrote:
> > On Tue, Feb 07, 2023 at 02:45:39PM -0500, Stefan Hajnoczi wrote:
> > > On Tue, Feb 07, 2023 at 11:14:46AM +0100, Peter-Jan Gootzen
wrote:
> > > > Hi,
> > > > 
> > 
> > [cc German]
> > 
> > > > For my MSc thesis project in collaboration with IBM
> > > > (https://github.com/IBM/dpu-virtio-fs) we are looking to
improve the
> > > > performance of the virtio-fs driver in high throughput
scenarios. We think
> > > > the main bottleneck is the fact that the virtio-fs driver
does not support
> > > > multi-queue (while the spec does). A big factor in this is
that our setup on
> > > > the virtio-fs device-side (a DPU) does not easily allow
multiple cores to
> > > > tend to a single virtio queue.
> > 
> > This is an interesting limitation in DPU.
> 
> Virtqueues are single-consumer queues anyway. Sharing them between
> multiple threads would be expensive. I think using multiqueue is natural
> and not specific to DPUs.
Can we create multiple threads (a thread pool) on DPU and let these
threads process requests in parallel (While there is only one virt
queue).

So this is what we had done in virtiofsd. One thread is dedicated to
pull the requests from virt queue and then pass the request to thread
pool to process it. And that seems to help with performance in
certain cases.

Is that possible on DPU? That itself can give a nice performance
boost for certain workloads without having to implement multiqueue
actually.

Just curious. I am not opposed to the idea of multiqueue. I am
just curious about the kind of performance gain (if any) it can
provide. And will this be helpful for rust virtiofsd running on
host as well?

Thanks
Vivek