Linux Virtualization - Sep 2019 - [PATCH v3 00/13] virtio-fs: shared file system for virtual machines

On Tue, Sep 03, 2019 at 10:05:02AM +0200, Miklos Szeredi
wrote:
> [Cc:  virtualization at lists.linux-foundation.org, "Michael S.
Tsirkin"
> <mst at redhat.com>, Jason Wang <jasowang at redhat.com>]
> 
> It'd be nice to have an ACK for this from the virtio maintainers.
> 
> Thanks,
> Miklos
Can the patches themselves be posted to the relevant list(s) please?
If possible, please also include "v3" in all patches so they are
easier to find.

I poked at
https://lwn.net/ml/linux-kernel/20190821173742.24574-1-vgoyal at redhat.com/
specifically
https://lwn.net/ml/linux-kernel/20190821173742.24574-12-vgoyal at redhat.com/
and things like:
+	/* TODO lock */
give me pause.

Cleanup generally seems broken to me - what pauses the FS

What about the rest of TODOs in that file?

use of usleep is hacky - can't we do better e.g. with a
completion?

Some typos - e.g. "reuests".

> 
> On Wed, Aug 21, 2019 at 7:38 PM Vivek Goyal <vgoyal at redhat.com>
wrote:
> >
> > Hi,
> >
> > Here are the V3 patches for virtio-fs filesystem. This time I have
> > broken the patch series in two parts. This is first part which does
> > not contain DAX support. Second patch series will contain the patches
> > for DAX support.
> >
> > I have also dropped RFC tag from first patch series as we believe its
> > in good enough shape that it should get a consideration for inclusion
> > upstream.
> >
> > These patches apply on top of 5.3-rc5 kernel and are also available
> > here.
> >
> > https://github.com/rhvgoyal/linux/commits/vivek-5.3-aug-21-2019
> >
> > Patches for V1 and V2 were posted here.
> >
> > https://lwn.net/ml/linux-fsdevel/20181210171318.16998-1-vgoyal at
redhat.com/
> > http://lkml.iu.edu/hypermail/linux/kernel/1905.1/07232.html
> >
> > More information about the project can be found here.
> >
> > https://virtio-fs.gitlab.io
> >
> > Changes from V2
> > ==============> > - Various bug fixes and performance
improvements.
> >
> > HOWTO
> > =====> > We have put instructions on how to use it here.
> >
> > https://virtio-fs.gitlab.io/
> >
> > Some Performance Numbers
> > =======================> > I have basically run bunch of fio
jobs to get a sense of speed of
> > various operations. I wrote a simple wrapper script to run fio jobs
> > 3 times and take their average and report it. These scripts are
available
> > here.
> >
> > https://github.com/rhvgoyal/virtiofs-tests
> >
> > I set up a directory on ramfs on host and exported that directory
inside
> > guest using virtio-9p and virtio-fs and ran tests inside guests. Ran
> > tests with cache=none both for virtio-9p and virtio-fs so that no
caching
> > happens in guest. For virtio-fs, I ran an additional set of tests with
> > dax enabled. Dax is not part of first patch series but I included
> > results here because dax seems to get the maximum performance
advantage
> > and its shows the real potential of virtio-fs.
> >
> > Test Setup
> > -----------
> > - A fedora 28 host with 32G RAM, 2 sockets (6 cores per socket, 2
> >   threads per core)
> >
> > - Using ramfs on host as backing store. 4 fio files of 2G each.
> >
> > - Created a VM with 16 VCPUS and 8GB memory. An 8GB cache window (for
dax
> >   mmap).
> >
> > Test Results
> > ------------
> > - Results in three configurations have been reported. 9p (cache=none),
> >   virtio-fs (cache=none) and virtio-fs (cache=none + dax).
> >
> >   There are other caching modes as well but to me cache=none seemed
most
> >   interesting for now because it does not cache anything in guest
> >   and provides strong coherence. Other modes which provide less strong
> >   coherence and hence are faster are yet to be benchmarked.
> >
> > - Three fio ioengines psync, libaio and mmap have been used.
> >
> > - I/O Workload of randread, radwrite, seqread and seqwrite have been
run.
> >
> > - Each file size is 2G. Block size 4K. iodepth=16
> >
> > - "multi" means same operation was done with 4 jobs and each
job is
> >   operating on a file of size 2G.
> >
> > - Some results are "0 (KiB/s)". That means that particular
operation is
> >   not supported in that configuration.
> >
> > NAME                    I/O Operation           BW(Read/Write)
> >
> > 9p-cache-none           seqread-psync           27(MiB/s)
> > virtiofs-cache-none     seqread-psync           35(MiB/s)
> > virtiofs-dax-cache-none seqread-psync           245(MiB/s)
> >
> > 9p-cache-none           seqread-psync-multi     117(MiB/s)
> > virtiofs-cache-none     seqread-psync-multi     162(MiB/s)
> > virtiofs-dax-cache-none seqread-psync-multi     894(MiB/s)
> >
> > 9p-cache-none           seqread-mmap            24(MiB/s)
> > virtiofs-cache-none     seqread-mmap            0(KiB/s)
> > virtiofs-dax-cache-none seqread-mmap            168(MiB/s)
> >
> > 9p-cache-none           seqread-mmap-multi      115(MiB/s)
> > virtiofs-cache-none     seqread-mmap-multi      0(KiB/s)
> > virtiofs-dax-cache-none seqread-mmap-multi      614(MiB/s)
> >
> > 9p-cache-none           seqread-libaio          26(MiB/s)
> > virtiofs-cache-none     seqread-libaio          139(MiB/s)
> > virtiofs-dax-cache-none seqread-libaio          160(MiB/s)
> >
> > 9p-cache-none           seqread-libaio-multi    129(MiB/s)
> > virtiofs-cache-none     seqread-libaio-multi    142(MiB/s)
> > virtiofs-dax-cache-none seqread-libaio-multi    577(MiB/s)
> >
> > 9p-cache-none           randread-psync          29(MiB/s)
> > virtiofs-cache-none     randread-psync          34(MiB/s)
> > virtiofs-dax-cache-none randread-psync          256(MiB/s)
> >
> > 9p-cache-none           randread-psync-multi    139(MiB/s)
> > virtiofs-cache-none     randread-psync-multi    153(MiB/s)
> > virtiofs-dax-cache-none randread-psync-multi    245(MiB/s)
> >
> > 9p-cache-none           randread-mmap           22(MiB/s)
> > virtiofs-cache-none     randread-mmap           0(KiB/s)
> > virtiofs-dax-cache-none randread-mmap           162(MiB/s)
> >
> > 9p-cache-none           randread-mmap-multi     111(MiB/s)
> > virtiofs-cache-none     randread-mmap-multi     0(KiB/s)
> > virtiofs-dax-cache-none randread-mmap-multi     215(MiB/s)
> >
> > 9p-cache-none           randread-libaio         26(MiB/s)
> > virtiofs-cache-none     randread-libaio         135(MiB/s)
> > virtiofs-dax-cache-none randread-libaio         157(MiB/s)
> >
> > 9p-cache-none           randread-libaio-multi   133(MiB/s)
> > virtiofs-cache-none     randread-libaio-multi   245(MiB/s)
> > virtiofs-dax-cache-none randread-libaio-multi   163(MiB/s)
> >
> > 9p-cache-none           seqwrite-psync          28(MiB/s)
> > virtiofs-cache-none     seqwrite-psync          34(MiB/s)
> > virtiofs-dax-cache-none seqwrite-psync          203(MiB/s)
> >
> > 9p-cache-none           seqwrite-psync-multi    128(MiB/s)
> > virtiofs-cache-none     seqwrite-psync-multi    155(MiB/s)
> > virtiofs-dax-cache-none seqwrite-psync-multi    717(MiB/s)
> >
> > 9p-cache-none           seqwrite-mmap           0(KiB/s)
> > virtiofs-cache-none     seqwrite-mmap           0(KiB/s)
> > virtiofs-dax-cache-none seqwrite-mmap           165(MiB/s)
> >
> > 9p-cache-none           seqwrite-mmap-multi     0(KiB/s)
> > virtiofs-cache-none     seqwrite-mmap-multi     0(KiB/s)
> > virtiofs-dax-cache-none seqwrite-mmap-multi     511(MiB/s)
> >
> > 9p-cache-none           seqwrite-libaio         27(MiB/s)
> > virtiofs-cache-none     seqwrite-libaio         128(MiB/s)
> > virtiofs-dax-cache-none seqwrite-libaio         141(MiB/s)
> >
> > 9p-cache-none           seqwrite-libaio-multi   119(MiB/s)
> > virtiofs-cache-none     seqwrite-libaio-multi   242(MiB/s)
> > virtiofs-dax-cache-none seqwrite-libaio-multi   505(MiB/s)
> >
> > 9p-cache-none           randwrite-psync         27(MiB/s)
> > virtiofs-cache-none     randwrite-psync         34(MiB/s)
> > virtiofs-dax-cache-none randwrite-psync         189(MiB/s)
> >
> > 9p-cache-none           randwrite-psync-multi   137(MiB/s)
> > virtiofs-cache-none     randwrite-psync-multi   150(MiB/s)
> > virtiofs-dax-cache-none randwrite-psync-multi   233(MiB/s)
> >
> > 9p-cache-none           randwrite-mmap          0(KiB/s)
> > virtiofs-cache-none     randwrite-mmap          0(KiB/s)
> > virtiofs-dax-cache-none randwrite-mmap          120(MiB/s)
> >
> > 9p-cache-none           randwrite-mmap-multi    0(KiB/s)
> > virtiofs-cache-none     randwrite-mmap-multi    0(KiB/s)
> > virtiofs-dax-cache-none randwrite-mmap-multi    200(MiB/s)
> >
> > 9p-cache-none           randwrite-libaio        25(MiB/s)
> > virtiofs-cache-none     randwrite-libaio        124(MiB/s)
> > virtiofs-dax-cache-none randwrite-libaio        131(MiB/s)
> >
> > 9p-cache-none           randwrite-libaio-multi  125(MiB/s)
> > virtiofs-cache-none     randwrite-libaio-multi  241(MiB/s)
> > virtiofs-dax-cache-none randwrite-libaio-multi  163(MiB/s)
> >
> > Conclusions
> > ==========> > - In general virtio-fs seems faster than
virtio-9p. Using dax makes it
> >   really interesting.
> >
> > Note:
> >   Right now dax window is 8G and max fio file size is 8G as well (4
> >   files of 2G each). That means everything fits into dax window and no
> >   reclaim is needed. Dax window reclaim logic is slower and if file
> >   size is bigger than dax window size, performance slows down.
> >
> > Description from previous postings
> > =================================> >
> > Design Overview
> > ==============> > With the goal of designing something with
better performance and local file
> > system semantics, a bunch of ideas were proposed.
> >
> > - Use fuse protocol (instead of 9p) for communication between guest
> >   and host. Guest kernel will be fuse client and a fuse server will
> >   run on host to serve the requests.
> >
> > - For data access inside guest, mmap portion of file in QEMU address
> >   space and guest accesses this memory using dax. That way guest page
> >   cache is bypassed and there is only one copy of data (on host). This
> >   will also enable mmap(MAP_SHARED) between guests.
> >
> > - For metadata coherency, there is a shared memory region which
contains
> >   version number associated with metadata and any guest changing
metadata
> >   updates version number and other guests refresh metadata on next
> >   access. This is yet to be implemented.
> >
> > How virtio-fs differs from existing approaches
> > =============================================> > The unique idea
behind virtio-fs is to take advantage of the co-location
> > of the virtual machine and hypervisor to avoid communication
(vmexits).
> >
> > DAX allows file contents to be accessed without communication with the
> > hypervisor. The shared memory region for metadata avoids communication
in
> > the common case where metadata is unchanged.
> >
> > By replacing expensive communication with cheaper shared memory
accesses,
> > we expect to achieve better performance than approaches based on
network
> > file system protocols. In addition, this also makes it easier to
achieve
> > local file system semantics (coherency).
> >
> > These techniques are not applicable to network file system protocols
since
> > the communications channel is bypassed by taking advantage of shared
memory
> > on a local machine. This is why we decided to build virtio-fs rather
than
> > focus on 9P or NFS.
> >
> > Caching Modes
> > ============> > Like virtio-9p, different caching modes are
supported which determine the
> > coherency level as well. The ?cache=FOO? and ?writeback? options
control the
> > level of coherence between the guest and host filesystems.
> >
> > - cache=none
> >   metadata, data and pathname lookup are not cached in guest. They are
always
> >   fetched from host and any changes are immediately pushed to host.
> >
> > - cache=always
> >   metadata, data and pathname lookup are cached in guest and never
expire.
> >
> > - cache=auto
> >   metadata and pathname lookup cache expires after a configured amount
of time
> >   (default is 1 second). Data is cached while the file is open (close
to open
> >   consistency).
> >
> > - writeback/no_writeback
> >   These options control the writeback strategy.  If writeback is
disabled,
> >   then normal writes will immediately be synchronized with the host
fs. If
> >   writeback is enabled, then writes may be cached in the guest until
the file
> >   is closed or an fsync(2) performed. This option has no effect on
mmap-ed
> >   writes or writes going through the DAX mechanism.
> >
> > Thanks
> > Vivek
> >
> > Miklos Szeredi (2):
> >   fuse: delete dentry if timeout is zero
> >   fuse: Use default_file_splice_read for direct IO
> >
> > Stefan Hajnoczi (6):
> >   fuse: export fuse_end_request()
> >   fuse: export fuse_len_args()
> >   fuse: export fuse_get_unique()
> >   fuse: extract fuse_fill_super_common()
> >   fuse: add fuse_iqueue_ops callbacks
> >   virtio_fs: add skeleton virtio_fs.ko module
> >
> > Vivek Goyal (5):
> >   fuse: Export fuse_send_init_request()
> >   Export fuse_dequeue_forget() function
> >   fuse: Separate fuse device allocation and installation in fuse_conn
> >   virtio-fs: Do not provide abort interface in fusectl
> >   init/do_mounts.c: add virtio_fs root fs support
> >
> >  fs/fuse/Kconfig                 |   11 +
> >  fs/fuse/Makefile                |    1 +
> >  fs/fuse/control.c               |    4 +-
> >  fs/fuse/cuse.c                  |    4 +-
> >  fs/fuse/dev.c                   |   89 ++-
> >  fs/fuse/dir.c                   |   26 +-
> >  fs/fuse/file.c                  |   15 +-
> >  fs/fuse/fuse_i.h                |  120 +++-
> >  fs/fuse/inode.c                 |  203 +++---
> >  fs/fuse/virtio_fs.c             | 1061
+++++++++++++++++++++++++++++++
> >  fs/splice.c                     |    3 +-
> >  include/linux/fs.h              |    2 +
> >  include/uapi/linux/virtio_fs.h  |   41 ++
> >  include/uapi/linux/virtio_ids.h |    1 +
> >  init/do_mounts.c                |   10 +
> >  15 files changed, 1462 insertions(+), 129 deletions(-)
> >  create mode 100644 fs/fuse/virtio_fs.c
> >  create mode 100644 include/uapi/linux/virtio_fs.h
Don't the new files need a MAINTAINERS entry?
I think we want virtualization at lists.linux-foundation.org to be
copied.

> >
> > --
> > 2.20.1
> >

On Tue, Sep 03, 2019 at 04:31:38AM -0400, Michael S. Tsirkin wrote:

[..]
> +	/* TODO lock */
> give me pause.
> 
> Cleanup generally seems broken to me - what pauses the FS
I am looking into device removal aspect of it now. Thinking of adding
a reference count to virtiofs device and possibly also a bit flag to
indicate if device is still alive. That way, we should be able to cleanup
device more gracefully.
> 
> What about the rest of TODOs in that file?
I will also take a closer look at TODOs now. Better device cleanup path
might get rid of some of them. Some of them might not be valid anymore.
> 
> use of usleep is hacky - can't we do better e.g. with a
> completion?
Agreed.

Thanks
Vivek

On Tue, Sep 03, 2019 at 10:07:52AM -0400, Vivek Goyal
wrote:
> On Tue, Sep 03, 2019 at 04:31:38AM -0400, Michael S. Tsirkin wrote:
> 
> [..]
> > +	/* TODO lock */
> > give me pause.
> > 
> > Cleanup generally seems broken to me - what pauses the FS
> 
> I am looking into device removal aspect of it now. Thinking of adding
> a reference count to virtiofs device and possibly also a bit flag to
> indicate if device is still alive. That way, we should be able to cleanup
> device more gracefully.
Generally, the way to cleanup things is to first disconnect device from
linux so linux won't send new requests, wait for old ones to finish.



> > 
> > What about the rest of TODOs in that file?
> 
> I will also take a closer look at TODOs now. Better device cleanup path
> might get rid of some of them. Some of them might not be valid anymore.
> 
> > 
> > use of usleep is hacky - can't we do better e.g. with a
> > completion?
> 
> Agreed.
> 
> Thanks
> Vivek