similar to: [PATCH nbdkit] sh: In pwrite, allow scripts to ignore stdin

I hit another transient failure in libnbd CI when a poorly-written eval script did not consume all of stdin during .pwrite. As behaving as a data sink can be a somewhat reasonable feature of a quickly-written sh or eval plugin, we should not be so insistent as treating an EPIPE failure as an immediate return of EIO to the client. Signed-off-by: Eric Blake <eblake at redhat.com> --- I

On Thu, Aug 31, 2023 at 11:12:59AM +0200, Laszlo Ersek wrote: > On 8/31/23 10:02, Richard W.M. Jones wrote: > > > > On Wed, Aug 30, 2023 at 05:21:19PM -0500, Eric Blake wrote: > >> I hit another transient failure in libnbd CI when a poorly-written > >> eval script did not consume all of stdin during .pwrite. As behaving > >> as a data sink can be a

On 8/31/23 00:21, Eric Blake wrote: > I hit another transient failure in libnbd CI when a poorly-written > eval script did not consume all of stdin during .pwrite. As behaving > as a data sink can be a somewhat reasonable feature of a > quickly-written sh or eval plugin, we should not be so insistent as > treating an EPIPE failure as an immediate return of EIO to the client. >

On Thu, Aug 31, 2023 at 10:40:53AM +0200, Laszlo Ersek wrote: > On 8/31/23 00:21, Eric Blake wrote: > > I hit another transient failure in libnbd CI when a poorly-written > > eval script did not consume all of stdin during .pwrite. As behaving > > as a data sink can be a somewhat reasonable feature of a > > quickly-written sh or eval plugin, we should not be so

On Wed, Aug 30, 2023 at 05:21:19PM -0500, Eric Blake wrote: > I hit another transient failure in libnbd CI when a poorly-written > eval script did not consume all of stdin during .pwrite. As behaving > as a data sink can be a somewhat reasonable feature of a > quickly-written sh or eval plugin, we should not be so insistent as > treating an EPIPE failure as an immediate return of

On 8/31/23 10:02, Richard W.M. Jones wrote: > > On Wed, Aug 30, 2023 at 05:21:19PM -0500, Eric Blake wrote: >> I hit another transient failure in libnbd CI when a poorly-written >> eval script did not consume all of stdin during .pwrite. As behaving >> as a data sink can be a somewhat reasonable feature of a >> quickly-written sh or eval plugin, we should not be so

On 8/31/23 10:55, Richard W.M. Jones wrote: > On Thu, Aug 31, 2023 at 10:40:53AM +0200, Laszlo Ersek wrote: >> On 8/31/23 00:21, Eric Blake wrote: >>> I hit another transient failure in libnbd CI when a poorly-written >>> eval script did not consume all of stdin during .pwrite. As behaving >>> as a data sink can be a somewhat reasonable feature of a

On Thu, Aug 31, 2023 at 11:18:27AM +0200, Laszlo Ersek wrote: > On 8/31/23 10:55, Richard W.M. Jones wrote: > > On Thu, Aug 31, 2023 at 10:40:53AM +0200, Laszlo Ersek wrote: > >> On 8/31/23 00:21, Eric Blake wrote: > >>> I hit another transient failure in libnbd CI when a poorly-written > >>> eval script did not consume all of stdin during .pwrite. As

On Thu, Aug 31, 2023 at 12:05:17PM +0200, Laszlo Ersek wrote: > On 8/31/23 11:47, Richard W.M. Jones wrote: > > On Thu, Aug 31, 2023 at 11:12:59AM +0200, Laszlo Ersek wrote: > >> On 8/31/23 10:02, Richard W.M. Jones wrote: > >>> > >>> On Wed, Aug 30, 2023 at 05:21:19PM -0500, Eric Blake wrote: > >>>> I hit another transient failure in libnbd

In the just-added nbdkit-sh-plugin.pod, you documented that unlike other plugins (where introspection or struct member population) can be used to determine which functionalities are supported, the shell plugin has to implement can_write and friends to return a special status 2 to document not supported, and 3 to indicate false, in part because you can't detect if a pwrite function is

This was a rather longer trip around the houses than I anticipated! The basic purpose of the patch series is to set $nbdkit_stdio_safe to "0" or "1" in sh and eval plugin scripts. To do that, I ended up adding a nicer way to manipulate environ lists, and to do that, I ended up adding a whole generic vector implementation which is applicable in a lot of different places.

I'm pushing the first one as blatantly obvious. The second one is also simple enough, but not enough of a bug for me to push tonight. The third is something I noticed while working on sh, but is really more about docs vs. plugins in general. There, we could either change the code to match the docs (breaking backwards behavior for a plugin that set .errno_is_preserved=2) [what my patch did],

With python 3, we have a nicer way to handle socket.error with errno set to EPIPE (or ESHUTDOWN). This is also more correct since in some cases (that I could not reproduce yet with v2v), using e[0] with BrokenPipeError will fail with: >>> OSError(errno.EPIPE, "Broken pipe")[0] Traceback (most recent call last): File "<stdin>", line 1, in <module>

While nbdkit itself must run with SIGPIPE ignored, many applications expect to inherit SIGPIPE in the default state. What's worse, POSIX states that a non-interactive shell script cannot use 'trap' to undo an inherited SIG_IGN on SIGPIPE. I have seen several bug reports over the years of something that works for a developer but fails under a CI environment, where the root cause was

The $tmpdir environment variable is used by the eval and sh plugins to communicate the path to the temporary directory created by the plugin for shell scripts to use for temporary files. Previously we set this environment variable globally (in .load()), but this means the environment variable can be leaked into undesirable places, eg. into the --run script: $ nbdkit sh - --run 'echo

On Thu, Aug 31, 2023 at 10:52:59AM +0100, Richard W.M. Jones wrote: > > > Other plugins (eg written in C) ignore or > > > only partially consume the buffer in pwrite all the time, and that's > > > never a problem. > > > > I don't understand. > > > > https://libguestfs.org/nbdkit-plugin.3.html#pwrite > > > >

This set of patches fixes virt-rescue rather cleanly. In particular the problems with handling ^C are completely fixed. Work still to be done before this can go upstream: - Shutdown doesn't work properly if you exit the shell. At the moment to exit you must do 'reboot -f'. Future improvements: - An escape sequence and escape commands that could be handled by virt-rescue,

--- plugins/eval/nbdkit-eval-plugin.pod | 11 +++-------- plugins/sh/nbdkit-sh-plugin.pod | 18 +++++++++++++++++- plugins/sh/call.c | 8 ++++++-- tests/test-single-sh.sh | 4 ++++ 4 files changed, 30 insertions(+), 11 deletions(-) diff --git a/plugins/eval/nbdkit-eval-plugin.pod b/plugins/eval/nbdkit-eval-plugin.pod index 602fd3aa..a016aa78 100644 ---

This fixes the main issues in virt-rescue and is usable. There are some enhancements which could be made (in follow up work): - An escape sequence and escape commands that could be handled by virt-rescue, eg. to shut down the appliance, mount or unmount filesystems. - `virt-rescue -i' could be implemented cleanly by performing the right API calls before handing control to the

Passing a memoryview we avoid unneeded copy of the original buffer. On the python side memoryview object can be used for slicing, writing to file, or sending to socket. This may break plugins assuming that the they get a bytearray, but good python code should not care about the type of the buffer, only about the behaviour. Testing with a plugin writing to /dev/null shows 2.7x speedup. Real