similar to: [PATCH nbdkit v3] tmpdisk: Generalize the tmpdisk plugin.

An evolution of v1 here: https://www.redhat.com/archives/libguestfs/2020-April/msg00035.html I want to generalize the tmpdisk plugin, particularly so you can use commands like ‘qemu-img create’ or even ‘virt-builder’. (Actually virt-builder really works - I tested it - but of course it causes a 30+ second delay when connecting to the server.) You can now use commands such as: nbdkit tmpdisk

Patch 1/2 is uncontroversial. Patch 2/2 is an interesting idea I had to generalize this plugin. It already uses a complete embedded shell script to do most of the work. What if, instead of making special cases for "type" and "label" params, we simply turn any other plugin parameters into script variables? This part of it works fine. However there is another problem which is

Unfinished (needs tests). This is my attempt to make a "remote tmpfs" plugin as outlined in this prior email: https://www.redhat.com/archives/libguestfs/2020-March/msg00134.html Although it would be possible to construct something a bit like this using existing plugins and filters (perhaps with some new features in those filters) I think it may be nicer to have a dedicated plugin for

v2 was here: https://www.redhat.com/archives/libguestfs/2020-March/msg00154.html v3: - Micro-optimize tmpdir. - Quote $disk in default command shell fragment. - Don't redirect mkfs output to /dev/null. Instead use exec </dev/null >/dev/null before the shell fragment. We may want to do this in other places where we run external shell scripts, or more generally for all

This allows us to be much more flexible about what commands can be used. It also means we do not need to encode any special behaviour for type or label parameters. --- plugins/tmpdisk/nbdkit-tmpdisk-plugin.pod | 115 ++++++++++---- tests/Makefile.am | 2 + plugins/tmpdisk/tmpdisk.c | 184 ++++++++++++++++------ plugins/tmpdisk/default-command.sh.in |

This can be used for creating temporary disks to thin clients, as a kind of "remote tmpfs". See also: https://www.redhat.com/archives/libguestfs/2020-March/msg00134.html --- plugins/data/nbdkit-data-plugin.pod | 1 + plugins/file/nbdkit-file-plugin.pod | 1 + plugins/linuxdisk/nbdkit-linuxdisk-plugin.pod | 7 +- plugins/memory/nbdkit-memory-plugin.pod |

This creates filesystems on demand. A client simply connects with a desired export name and a new export is created. The export is persistent (until deleted by the server admin), and clients may disconnect and reconnect. In some respects this is similar to the nbdkit-tmpdisk-plugin, or nbdkit-file-plugin with the dir= option. --- plugins/ondemand/nbdkit-ondemand-plugin.pod | 190 ++++++

Needs some work still, see in particular the commit message for patch 3. Rich.

This allows us to be much more flexible about what commands can be used. It also means we do not need to encode any special behaviour for type or label parameters. --- plugins/tmpdisk/nbdkit-tmpdisk-plugin.pod | 91 +++++++++----- plugins/tmpdisk/tmpdisk.c | 147 ++++++++++++++-------- plugins/tmpdisk/default-command.sh.in | 6 + 3 files changed, 164 insertions(+), 80

For context see these threads: https://lists.gnu.org/archive/html/qemu-discuss/2020-06/threads.html#00053 https://lists.gnu.org/archive/html/qemu-block/2020-06/threads.html#01496 Rich.

For review only, this needs some clean up and more tests. My eyes are going cross-eyed looking at the calculate_offset_of_entry function, so time to take a break ... Rich.

This is an RFC only, at the very least it lacks tests. This implements a rather complex new feature in nbdkit-curl-plugin allowing you to specify an external shell script that can be used to fetch an authorization token for services which requires a token or cookie for access, especially if that token must be renewed periodically. The motivation can be seen in the changes to the docs in patch 2.

This parameter provided a name for the "default export" -- ie. the one and only export returned to the client by NBD_OPT_LIST. But nbdkit traditionally didn't care what export name the client requested. Since 1.16 plugins have been able to serve different content per export name (and return errors for unknown exports), but the -e option didn't reflect that and only created

Evolution of this patch series: https://www.redhat.com/archives/libguestfs/2020-July/thread.html#00073 Instead of auth-script, this implements header-script and cookie-script. It can be used for similar purposes but the implementation is somewhat saner. Rich.

Since the v1 series (0/4, at [1]), I've applied patches 1 and 2, rewritten patch 3 [Forbid NUL in export and context names] into patch 4 here, patch 4 there turned into patch 6 here, and everything else here is new. [1]https://www.redhat.com/archives/libguestfs/2019-September/msg00180.html I don't know if there is a handy reusable function for checking whether a string contains valid

This allows us to be much more flexible about what commands can be used. It also means we do not need to encode any special behaviour for type or label parameters. XXX However one problem which is not addressed here: The $disk passed to the shell script fragment must be updated, not replaced. This is because currently the plugin has a file descriptor opened on this file. If it is deleted by

On 3/17/20 3:53 AM, Richard W.M. Jones wrote: > This can be used for creating temporary disks to thin clients, as a > kind of "remote tmpfs". > > See also: > https://www.redhat.com/archives/libguestfs/2020-March/msg00134.html > --- > +++ b/plugins/tmpdisk/nbdkit-tmpdisk-plugin.pod > @@ -0,0 +1,157 @@ > +=head1 NAME > + > +nbdkit-tmpdisk-plugin - create

This plugin now implements efficient zeroing, pre-fetch and extents, which should give quite a performance boost. XXX On the flip side, it no longer ignores flush and FUA (which we can ignore because these are temporary disks), which very likely has a large negative impact on performance. Fixing this would involve generalising fileops a little. --- plugins/tmpdisk/Makefile.am | 2 +

Since v1: - new patch to add uri_quote() - rebase on top of other recent patches needed while auditing shell_quote() - use uri_quote() instead of shell_quote() for producing $uri Eric Blake (2): common/utils: Add uri_quote and tests captive: Support $uri in --run docs/nbdkit-captive.pod | 8 ++- common/utils/utils.h | 1 + common/utils/test-quotes.c | 108

Turns out Japanese trains are good for coding! In supermin we have a bunch of code to create the libguestfs appliance. It creates it directly using libext2fs (part of e2fsprogs). We can use the same technique to create ext2 virtual disks in nbdkit, which is what this new plugin does. Why a new plugin instead of modifying the floppy plugin? See the 4/4 commit message for an explanation. The