similar to: [PATCH nbdkit v2] tmpdisk: Generalize the tmpdisk plugin

v2 was here: https://www.redhat.com/archives/libguestfs/2020-April/msg00075.html In v3: - Add [VAR=VALUE ...] to manual. - Various minor improvements to the manual. - Work (at least, in theory - not tested) with block devices or symlinks. I didn't document this because it's hard to ensure these files or block devices would be cleaned up, so here be dragons. - Remove O_NOCTTY. -

Unfinished (needs tests). This is my attempt to make a "remote tmpfs" plugin as outlined in this prior email: https://www.redhat.com/archives/libguestfs/2020-March/msg00134.html Although it would be possible to construct something a bit like this using existing plugins and filters (perhaps with some new features in those filters) I think it may be nicer to have a dedicated plugin for

Patch 1/2 is uncontroversial. Patch 2/2 is an interesting idea I had to generalize this plugin. It already uses a complete embedded shell script to do most of the work. What if, instead of making special cases for "type" and "label" params, we simply turn any other plugin parameters into script variables? This part of it works fine. However there is another problem which is

v2 was here: https://www.redhat.com/archives/libguestfs/2020-March/msg00154.html v3: - Micro-optimize tmpdir. - Quote $disk in default command shell fragment. - Don't redirect mkfs output to /dev/null. Instead use exec </dev/null >/dev/null before the shell fragment. We may want to do this in other places where we run external shell scripts, or more generally for all

This can be used for creating temporary disks to thin clients, as a kind of "remote tmpfs". See also: https://www.redhat.com/archives/libguestfs/2020-March/msg00134.html --- plugins/data/nbdkit-data-plugin.pod | 1 + plugins/file/nbdkit-file-plugin.pod | 1 + plugins/linuxdisk/nbdkit-linuxdisk-plugin.pod | 7 +- plugins/memory/nbdkit-memory-plugin.pod |

This allows us to be much more flexible about what commands can be used. It also means we do not need to encode any special behaviour for type or label parameters. --- plugins/tmpdisk/nbdkit-tmpdisk-plugin.pod | 91 +++++++++----- plugins/tmpdisk/tmpdisk.c | 147 ++++++++++++++-------- plugins/tmpdisk/default-command.sh.in | 6 + 3 files changed, 164 insertions(+), 80

This allows us to be much more flexible about what commands can be used. It also means we do not need to encode any special behaviour for type or label parameters. --- plugins/tmpdisk/nbdkit-tmpdisk-plugin.pod | 115 ++++++++++---- tests/Makefile.am | 2 + plugins/tmpdisk/tmpdisk.c | 184 ++++++++++++++++------ plugins/tmpdisk/default-command.sh.in |

Needs some work still, see in particular the commit message for patch 3. Rich.

This allows us to be much more flexible about what commands can be used. It also means we do not need to encode any special behaviour for type or label parameters. XXX However one problem which is not addressed here: The $disk passed to the shell script fragment must be updated, not replaced. This is because currently the plugin has a file descriptor opened on this file. If it is deleted by

On 3/17/20 3:53 AM, Richard W.M. Jones wrote: > This can be used for creating temporary disks to thin clients, as a > kind of "remote tmpfs". > > See also: > https://www.redhat.com/archives/libguestfs/2020-March/msg00134.html > --- > +++ b/plugins/tmpdisk/nbdkit-tmpdisk-plugin.pod > @@ -0,0 +1,157 @@ > +=head1 NAME > + > +nbdkit-tmpdisk-plugin - create

This is an RFC only, at the very least it lacks tests. This implements a rather complex new feature in nbdkit-curl-plugin allowing you to specify an external shell script that can be used to fetch an authorization token for services which requires a token or cookie for access, especially if that token must be renewed periodically. The motivation can be seen in the changes to the docs in patch 2.

This creates filesystems on demand. A client simply connects with a desired export name and a new export is created. The export is persistent (until deleted by the server admin), and clients may disconnect and reconnect. In some respects this is similar to the nbdkit-tmpdisk-plugin, or nbdkit-file-plugin with the dir= option. --- plugins/ondemand/nbdkit-ondemand-plugin.pod | 190 ++++++

This plugin now implements efficient zeroing, pre-fetch and extents, which should give quite a performance boost. XXX On the flip side, it no longer ignores flush and FUA (which we can ignore because these are temporary disks), which very likely has a large negative impact on performance. Fixing this would involve generalising fileops a little. --- plugins/tmpdisk/Makefile.am | 2 +

For context see these threads: https://lists.gnu.org/archive/html/qemu-discuss/2020-06/threads.html#00053 https://lists.gnu.org/archive/html/qemu-block/2020-06/threads.html#01496 Rich.

Since the v1 series (0/4, at [1]), I've applied patches 1 and 2, rewritten patch 3 [Forbid NUL in export and context names] into patch 4 here, patch 4 there turned into patch 6 here, and everything else here is new. [1]https://www.redhat.com/archives/libguestfs/2019-September/msg00180.html I don't know if there is a handy reusable function for checking whether a string contains valid

Evolution of this patch series: https://www.redhat.com/archives/libguestfs/2020-July/thread.html#00073 Instead of auth-script, this implements header-script and cookie-script. It can be used for similar purposes but the implementation is somewhat saner. Rich.

For review only, this needs some clean up and more tests. My eyes are going cross-eyed looking at the calculate_offset_of_entry function, so time to take a break ... Rich.

This parameter provided a name for the "default export" -- ie. the one and only export returned to the client by NBD_OPT_LIST. But nbdkit traditionally didn't care what export name the client requested. Since 1.16 plugins have been able to serve different content per export name (and return errors for unknown exports), but the -e option didn't reflect that and only created

On Tue, Mar 17, 2020 at 06:16:49AM -0500, Eric Blake wrote: > >+static void * > >+tmpdisk_open (int readonly) > >+{ > >+ struct handle *h; > >+ CLEANUP_FREE char *disk = NULL; > >+ const char *tmpdir; > >+ > >+ tmpdir = getenv ("TMPDIR"); > >+ if (!tmpdir) > >+ tmpdir = "/var/tmp"; > > Rather than

Turns out Japanese trains are good for coding! In supermin we have a bunch of code to create the libguestfs appliance. It creates it directly using libext2fs (part of e2fsprogs). We can use the same technique to create ext2 virtual disks in nbdkit, which is what this new plugin does. Why a new plugin instead of modifying the floppy plugin? See the 4/4 commit message for an explanation. The