search for: skip_bytes

...le-plugin(1)> (except not as fast and missing many @@ -60,8 +60,8 @@ features): nbdkit eval \ config='ln -sf "$(realpath "$3")" $tmpdir/file' \ get_size='stat -Lc %s $tmpdir/file' \ - pread='dd if=$tmpdir/file skip=$4 count=$3 iflag=count_bytes,skip_bytes' \ - pwrite='dd of=$tmpdir/file seek=$4 conv=notrunc oflag=seek_bytes' \ + pread='dd bs=1 if=$tmpdir/file skip=$4 count=$3' \ + pwrite='dd bs=1 of=$tmpdir/file seek=$4 conv=notrunc' \ file=disk.img =head1 PARAMETERS diff --git a/plugins/sh/nbdkit-sh-plugin.po...

...handle _be_ the export name: open) printf %s "$3" ;; > + get_size) > + stat -c '%s' "$2" > + ;; at which point, you can simplify to: get_size) echo ${#2} ;; > + pread) > + dd if="$2" skip=$4 count=$3 iflag=skip_bytes,count_bytes > + ;; and this would be something like: pread) echo "$2" | dd skip=$4 count=$3 iflag=skip_bytes,count_bytes ;; or bypass dd by exploiting your knowledge of the client: pread) if test $4.$3 = "0.${#2}"; then printf %s "$2" else...

...Let's put an upper bound on the maximum number of extents we are willing to return (1M extents is an 8M reply chunk). Pre-patch, this can be used to demonstrate the problem: $ nbdkit -f sh - <<\EOF #!/bin/bash size=$((9*1024*1024)) case $1 in get_size) echo $size ;; pread) dd iflag=skip_bytes,count_bytes skip=$4 count=$3 if=/dev/zero || exit 1 ;; can_extents) ;; extents) # Unrealistic in real life, but works to provoke the bug. For a full 9M # query, this produces ~100M for nbdkit to parse, and in turn tries to # produce a 72M reply chunk if we don't cap extents....

The previous incomplete patch was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00049.html based on earlier discussion here: https://www.redhat.com/archives/libguestfs/2019-September/msg00047.html In v2: - The previous patch was incomplete. This version completes it by adding tests and extending nbdkit-sh-plugin. - nbdkit_export_name now returns NULL for error,

...\$( ls /proc/\$\$/fd | wc -w ) -ne \$(($curr_fds + 1)); then + echo "there seem to be leaked fds, curr_fds=$curr_fds" >&2 + ls -l /proc/\$\$/fd >&2 + exit 1 + fi + ) || exit 5 +fi case \$1 in get_size) stat -L -c %s \$f || exit 1 ;; pread) dd iflag=skip_bytes,count_bytes skip=\$4 count=\$3 if=\$f || exit 1 ;; -- 2.20.1

...;<'EOF' +case "$1" in + open) + h=`mktemp $tmpdir/disk-XXXXXX` + echo -n "$3" > $h + echo $h + ;; + get_size) + stat -c '%s' "$2" + ;; + pread) + dd if="$2" skip=$4 count=$3 iflag=skip_bytes,count_bytes + ;; + *) exit 2 ;; +esac +EOF + +# Try to read back various export names from the plugin. +for e in "" "test" "/" "//" " " \ + "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot...

...$4" != true; then + # nbdkit needs a fix to let tlsdummy skip .open when insecure... + # echo 'EINVAL unexpected tls mode' 2>&1; exit 1 + echo insecure; exit 0 + fi + echo $3 ;; + get_size) echo 6 ;; + pread) echo "$2" | dd skip=$4 count=$3 iflag=skip_bytes,count_bytes ;; + *) exit 2; +esac +EOF + +# Plaintext client sees only dummy volume +nbdsh -c ' +import os +h.set_export_name ("hello") +h.connect_unix (os.environ["sock"]) +assert h.get_size () == 512 +assert h.pread (5, 0) == b"dummy" +' + +# Encrypted clien...

...ng the delays may help. + +cat > test-parallel-sh.script <<EOF +#!/usr/bin/env bash +f=test-parallel-sh.data +if ! test -f \$f; then + echo "can't locate test-parallel-sh.data" >&2; exit 5 +fi +case \$1 in + get_size) stat -L -c %s \$f || exit 1 ;; + pread) dd iflag=skip_bytes,count_bytes skip=\$4 count=\$3 if=\$f || exit 1 ;; + pwrite) dd oflag=seek_bytes conv=notrunc seek=\$4 of=\$f || exit 1 ;; + can_write) ;; + *) exit 2 ;; +esac +exit 0 +EOF +chmod +x test-parallel-sh.script + +# With --threads=1, the write should complete first because it was issued first +nbdki...

...e; then - echo 'EINVAL unexpected tls mode' 2>&1; exit 1 - fi - echo $3 ;; + default_export) check "$3"; echo hello ;; + open) check "$4"; echo $3 ;; get_size) echo "$2" | wc -c ;; pread) echo "$2" | dd skip=$4 count=$3 iflag=skip_bytes,count_bytes ;; can_write | can_trim) exit 0 ;; -- 2.28.0

...-parallel-sh.script <<EOF > +#!/usr/bin/env bash > +f=test-parallel-sh.data > +if ! test -f \$f; then > + echo "can't locate test-parallel-sh.data" >&2; exit 5 > +fi > +case \$1 in > + get_size) stat -L -c %s \$f || exit 1 ;; > + pread) dd iflag=skip_bytes,count_bytes skip=\$4 count=\$3 if=\$f || exit 1 ;; > + pwrite) dd oflag=seek_bytes conv=notrunc seek=\$4 of=\$f || exit 1 ;; > + can_write) ;; > + *) exit 2 ;; > +esac > +exit 0 > +EOF > +chmod +x test-parallel-sh.script > + > +# With --threads=1, the write should comp...

This is a major rewrite compared to my v1 series, where I've tried a lot harder to ensure that we still accommodate building on Haiku (although I have not actually yet fired up a Haiku VM to try it for myself). I also managed to make the sh plugin fully parallel, on capable platforms. See also my question on patch 10 on whether I've picked the best naming convention. Eric Blake (17):

...(requires 1.22): + * $ nbdkit -U /tmp/sock sh - <<\EOF + * case $1 in + * list_exports) echo NAMES; echo foo; echo foobar ;; + * open) echo "$3" ;; + * get_size) echo "$2" | wc -c ;; + * pread) echo "$2" | dd skip=$4 count=$3 \ + * iflag=skip_bytes,count_bytes ;; + * *) exit 2 ;; + * esac + * EOF + * $ ./run examples/list-exports /tmp/sock + * [0] foo + * [1] foobar + * Which export to connect to (-1 to quit)? 1 + * Connecting to foobar ... + * /tmp/sock: foobar: size = 7 bytes */ #include <stdio.h> @@ -20,7 +38,...

Patch 3 still needs tests added, but it is at least working from my simple command line tests. Eric Blake (3): server: Implement nbdkit_is_tls for use during .open server: Expose final thread_model to filter's .get_ready tlsdummy: New filter docs/nbdkit-filter.pod | 21 +- docs/nbdkit-plugin.pod | 34 ++- docs/nbdkit-tls.pod

More patches on the way for improving .list_exports signature and adding .export_description, but this is the promised code showing why nbdkit_string_intern is useful. Patch 4 is somewhat RFC: we could either add new API to take the boilerplate from: foo_config(const char *key, const char *value) { if (strcmp (key, "file") == 0) { CLEANUP_FREE char *tmp = nbdkit_realpath (value);

This is a revision of my .default_export work, plus new work on .export_descriptions and a new exportname filter. I think it is now ready to check in. Things I'd still like in 1.22: - the file plugin should implement .list_exports (patch already posted, but it needs rebasing on this series) - the ext2 filter should override .list_exports when in exportname mode - the nbd plugin should be

It's been several weeks since I posted v2 (I got distracted by improving libnbd to better test things, which in turn surfaced some major memory leak problems in nbdsh that are now fixed). Many of the patches are minor rebases from v2, with the biggest changes being fallout from: - patch 2: rename nbdkit_add_default_export to nbdkit_use_default_export - overall: this missed 1.22, so update

Well, I'm not quite done (I still want to get nbdinfo to work on a single nbd connection for all cases when reading the heads of the file is not required), but I'm happy with patches 1-11, and 12-13 show where I'm headed for getting NBD_OPT_INFO to work. Posting now to see if some of the earlier patches are ready to commit while I continue working on the latter half. Eric Blake (13):