Libguestfs - Aug 2022 - Benchmarks of asynch nbdublk (was: Re: [PATCH libnbd] ublk: Add new nbdublk program)

I have an interesting benchmark to share.  I wanted to find out the
answers to a couple of questions: (1) Is nbdublk stable, and
(2) Is the performance terrible / good / excellent?  I only wanted a
rough impression for these, we can work on optimising it more later.

Since I am developing nbdublk (the client) in a disposable Rawhide VM,
and because I already need to recompile the kernel in that VM to
enable ublk, I have the Linux kernel tree from git around, so I
decided to compare regular VM disk:

  +------------+
  | Rawhide VM |
  |            |
  | compile    |
  | Linux on   |
  | VM disk    |
  +------------+
     |
  +--|--------------------------+
  |  |                   Host   |
  |  V                          |
  | raw format disk             |
  +-----------------------------+

versus nbdublk + ublk:

  +------------+
  | Rawhide VM |
  |            |
  | compile    |
  | Linux on   |
  | /dev/ublkb0|
  | nbdublk    |
  +------------+
     | NBD connection
  +--|--------------------------+
  |  |                   Host   |
  |  V                          |
  | qemu-nbd + qcow2 disk       |
  +-----------------------------+

This is with qemu-7.0.0 and the same version of qemu-nbd.  I did
drop_caches == 3 before both tests.  The .config was identical in both
builds.  Both filesystems use XFS.

With the regular VM disk:

  real 21m12.563s
  user 136m25.367s
  sys  24m11.165s

With nbdublk disk:

  real 26m3.990s
  user 137m16.734s
  sys  24m37.143s

So it's interesting that although nbdublk is a bit slower, it's not
terrible.  In fact this is much better performance than I expected for
a first pass.  (I was honestly expecting it'd be like half the speed).
I'm pretty happy with this performance for now.

Also the compile completed without any errors both times, indicating
that at least for compiling, nbdublk is stable.

However while I was doing these tests I did notice that nbdublk isn't
stable for some operations.  In particular 'git clone' cloning the
Linux git tree from the VM disk to /dev/ublkb0 would sometimes
deadlock or have other problems.  I'm not entirely sure why that is,
given that I didn't get any problems while compiling.  I guess that
git uses the filesystem very aggressively compared to just compiling.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

Alright, so final version for now is here:

  https://gitlab.com/rwmjones/libnbd/-/tree/nbdublk/ublk

I resolved the data corruption problem once I realised that
->handle_event must only retire commands which are on the same queue
(obvious in hindsight).  I ran a parallel job overnight where I did
various heavyweight operations repeatedly -- a bunch of git stuff,
large copies, a big compile etc -- on an nbdublk filesystem.  It
handled everything perfectly.

Before this can go upstream:

 - Does libubdsrv now have a stable API, or could it change?

 - We need to enable the device in the Fedora kernel.

 - We need libubdsrv + ublk tool in Fedora.

 - Possible we need to add tests into the libnbd tree.  I couldn't
   think of a good way to test this that doesn't require root and
   potentially do some horrible stuff to the testing machine.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v