Libguestfs - Mar 2020 - [nbdkit] Serving "temporary disks" over NBD

(This email is largely me just thinking aloud ...  So ideas welcome!)

For the RISC-V builders (http://fedora.riscv.rocks) I'm using an NBD
disk to host the heavily used /var/lib/mock build directory.  This is
because the build machines have only SD-cards as local disk and SD is
both slow and unreliable.

The builders (NBD clients) are set up with an /etc/rc.d/rc.local file
which runs these commands on boot:

  modprobe nbd
  nbd-client -b 512 nbd.server.example.com 10809 /dev/nbd0
  mount /dev/nbd0 /var/lib/mock

The /var/lib/mock directory is essentially temporary and doesn't need
to persist across boots or even mounts.  Ideally every time the client
connects over NBD we'd get a fresh ext4 filesystem.  Also note as
there are multiple clients they must all see a freshly created,
independent view of the disk.

At the moment I'm using nbdkit as the server, but:

(1) There is a separate server and port for each builder (client).
"10809" in the command above is actually a variable port number
depending on the client.

(2) The filesystem is actually persistent because I'm using regular
nbdkit-file-plugin backed by a block device.

(3) The filesystems are very large (100G), much larger than the
available memory on the server.  So they have to be backed by the
server's filesystem rather than being RAM disks.  (This is a
round-about way of saying that nbdkit-memory-plugin cannot be used).

Funnily enough nbd-server despite being very inflexible in many ways
actually makes this particular use-case easy because of the nbd-server -c
option:

 
https://github.com/NetworkBlockDevice/nbd/blob/cb20c16354cccf4698fde74c42f5fb8542b289ae/man/nbd-server.1.in.sgml#L189

Of course I could very easily write a custom plugin just for this
purpose, but could we do something in upstream nbdkit to make it easier?

Currently the cow filter (unlike nbd-server -c) serves identical
content to all clients.  Adding a new option to make it serve a new
COW to each client is possible but not an easy change to make.
Or a new COW filter which works more like nbd-server -c?

An upstream custom plugin which implements this as a special case?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org