search for: count_bytes

While iflag=count_bytes combined with bs > 1 allows for more efficient operation, it is a feature of GNU dd, and not present on other implementations such as BSD. Sticking to just POSIX features makes things more portable. Signed-off-by: Eric Blake <eblake@redhat.com> --- docs/nbdkit-loop.pod |...

On Mon, Aug 17, 2020 at 11:35:39AM -0500, Eric Blake wrote: > While iflag=count_bytes combined with bs > 1 allows for more efficient > operation, it is a feature of GNU dd, and not present on other > implementations such as BSD. Sticking to just POSIX features makes > things more portable. I'm not very convinced by this. Maybe we should persuade the BSD folk to im...

On 12/14/18 4:16 PM, Richard W.M. Jones wrote: > This implements something like a readonly 1MB disk reading as zeroes: > > nbdkit sh script=- <<'EOF' > case "$1" in > get_size) echo 1M ;; > pread) dd if=/dev/zero count=$3 iflag=count_bytes ;; > *) exit 2 ;; > esac > EOF > > Use of "-" is analogous to reading passwords from stdin. > --- > @@ -26,6 +30,19 @@ like this: > You may have to add further C<key=value> arguments to the command > line. > > +=head2 Inline shell sc...

.../18 4:16 PM, Richard W.M. Jones wrote: > >This implements something like a readonly 1MB disk reading as zeroes: > > > >nbdkit sh script=- <<'EOF' > > case "$1" in > > get_size) echo 1M ;; > > pread) dd if=/dev/zero count=$3 iflag=count_bytes ;; > > *) exit 2 ;; > > esac > >EOF > > > >Use of "-" is analogous to reading passwords from stdin. > >--- > > >@@ -26,6 +30,19 @@ like this: > > You may have to add further C<key=value> arguments to the command > > l...

...’, this changes the output to show the hole and zero flags separately. For example: $ ./nbdkit -U - --filter=log sh - logfile=/dev/stdout \ --run 'qemu-img map $nbd' <<'EOF' case "$1" in get_size) echo 1M ;; pread) dd if=/dev/zero count=$3 iflag=count_bytes ;; can_extents) exit 0 ;; extents) echo "0 32K zero" echo "32K 32K hole,zero" echo "64K 983040 " ;; *) exit 2 ;; esac EOF [...] 2019-04-01 11:49:40.818357 connection=1 Extents id=2 offset=0x0 count=0x10...

..._ the export name: open) printf %s "$3" ;; > + get_size) > + stat -c '%s' "$2" > + ;; at which point, you can simplify to: get_size) echo ${#2} ;; > + pread) > + dd if="$2" skip=$4 count=$3 iflag=skip_bytes,count_bytes > + ;; and this would be something like: pread) echo "$2" | dd skip=$4 count=$3 iflag=skip_bytes,count_bytes ;; or bypass dd by exploiting your knowledge of the client: pread) if test $4.$3 = "0.${#2}"; then printf %s "$2" else echo "E...

This implements something like a readonly 1MB disk reading as zeroes: nbdkit sh script=- <<'EOF' case "$1" in get_size) echo 1M ;; pread) dd if=/dev/zero count=$3 iflag=count_bytes ;; *) exit 2 ;; esac EOF Use of "-" is analogous to reading passwords from stdin. --- plugins/sh/nbdkit-sh-plugin.pod | 17 ++++++++++ plugins/sh/sh.c | 55 ++++++++++++++++++++++++++++++++- 2 files changed, 71 insertions(+), 1 deletion(-) diff --git a/plugins/sh/...

...put an upper bound on the maximum number of extents we are willing to return (1M extents is an 8M reply chunk). Pre-patch, this can be used to demonstrate the problem: $ nbdkit -f sh - <<\EOF #!/bin/bash size=$((9*1024*1024)) case $1 in get_size) echo $size ;; pread) dd iflag=skip_bytes,count_bytes skip=$4 count=$3 if=/dev/zero || exit 1 ;; can_extents) ;; extents) # Unrealistic in real life, but works to provoke the bug. For a full 9M # query, this produces ~100M for nbdkit to parse, and in turn tries to # produce a 72M reply chunk if we don't cap extents. for ((i=$4;...

...and count how long it takes. + i=`cat retry-reopen-fail-count` + ((i++)) + echo $i > retry-reopen-fail-count + if [ $i -le 3 ]; then + echo "EIO pread failed" >&2 + exit 1 + else + dd if=/dev/zero count=$3 iflag=count_bytes + fi + ;; + + get_size) echo 512 ;; + *) exit 2 ;; +esac +EOF + +# In this test we should see 4 failures: +# pread FAILS +# retry and wait 2 seconds +# open FAILS +# retry and wait 4 seconds +# open succeeds +# pread FAILS +# retry and wait 8 seconds +# pread FAILS +# retry and...

The previous incomplete patch was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00049.html based on earlier discussion here: https://www.redhat.com/archives/libguestfs/2019-September/msg00047.html In v2: - The previous patch was incomplete. This version completes it by adding tests and extending nbdkit-sh-plugin. - nbdkit_export_name now returns NULL for error,

...ONMENT VARIABLES diff --git a/tests/test-eval.sh b/tests/test-eval.sh index 206c680..4557b02 100755 --- a/tests/test-eval.sh +++ b/tests/test-eval.sh @@ -42,7 +42,9 @@ cleanup_fn rm -f $files nbdkit -U - eval \ get_size='echo 64M' \ pread='dd if=/dev/zero count=$3 iflag=count_bytes' \ + missing='echo "in missing: $@" >> eval.out; exit 2' \ --run 'qemu-img info $nbd' > eval.out cat eval.out grep '67108864 bytes' eval.out +grep 'in missing' eval.out -- 2.24.1

Some feature additions to the shell script plugin allow us to test the export flags field reasonably easily. Rich.

...show the hole and zero flags separately. For example: > > $ ./nbdkit -U - --filter=log sh - logfile=/dev/stdout \ > --run 'qemu-img map $nbd' <<'EOF' > case "$1" in > get_size) echo 1M ;; > pread) dd if=/dev/zero count=$3 iflag=count_bytes ;; > can_extents) exit 0 ;; > extents) > echo "0 32K zero" > echo "32K 32K hole,zero" > echo "64K 983040 " > ;; > *) exit 2 ;; > esac > EOF > [...] > 2019-04-01 11:49:40...

...oc/\$\$/fd | wc -w ) -ne \$(($curr_fds + 1)); then + echo "there seem to be leaked fds, curr_fds=$curr_fds" >&2 + ls -l /proc/\$\$/fd >&2 + exit 1 + fi + ) || exit 5 +fi case \$1 in get_size) stat -L -c %s \$f || exit 1 ;; pread) dd iflag=skip_bytes,count_bytes skip=\$4 count=\$3 if=\$f || exit 1 ;; -- 2.20.1

...OF' +case "$1" in + open) + h=`mktemp $tmpdir/disk-XXXXXX` + echo -n "$3" > $h + echo $h + ;; + get_size) + stat -c '%s' "$2" + ;; + pread) + dd if="$2" skip=$4 count=$3 iflag=skip_bytes,count_bytes + ;; + *) exit 2 ;; +esac +EOF + +# Try to read back various export names from the plugin. +for e in "" "test" "/" "//" " " \ + "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" +do + e...

...og; then fi cat >single-sh.script <<\EOF +if test "$nbdkit_stdio_safe" != "0"; then + echo "unexpected value for nbdkit_stdio_safe ($nbdkit_stdio_safe)" 1>&2 + exit 1 +fi case $1 in get_size) echo 1m ;; pread) dd if=/dev/zero count=$3 iflag=count_bytes ;; -- 2.25.0

.../test-eval.sh b/tests/test-eval.sh > index 206c680..4557b02 100755 > --- a/tests/test-eval.sh > +++ b/tests/test-eval.sh > @@ -42,7 +42,9 @@ cleanup_fn rm -f $files > nbdkit -U - eval \ > get_size='echo 64M' \ > pread='dd if=/dev/zero count=$3 iflag=count_bytes' \ > + missing='echo "in missing: $@" >> eval.out; exit 2' \ > --run 'qemu-img info $nbd' > eval.out > > cat eval.out > grep '67108864 bytes' eval.out > +grep 'in missing' eval.out > -- > 2.24.1 > &...

...true; then + # nbdkit needs a fix to let tlsdummy skip .open when insecure... + # echo 'EINVAL unexpected tls mode' 2>&1; exit 1 + echo insecure; exit 0 + fi + echo $3 ;; + get_size) echo 6 ;; + pread) echo "$2" | dd skip=$4 count=$3 iflag=skip_bytes,count_bytes ;; + *) exit 2; +esac +EOF + +# Plaintext client sees only dummy volume +nbdsh -c ' +import os +h.set_export_name ("hello") +h.connect_unix (os.environ["sock"]) +assert h.get_size () == 512 +assert h.pread (5, 0) == b"dummy" +' + +# Encrypted client sees desir...

v1 was here: https://www.redhat.com/archives/libguestfs/2018-December/thread.html#00123 v2: - Document "-" instead of "script=-" and use it in the test; and verify this also works on FreeBSD; and verify that it doesn't depend on the particular behaviour of our wrapper script and should work with installed nbdkit too. - Fix handling of zero flags parameter. -

...aces around the operators would make this more legible. > + for i in \$(seq \$offset \$length); do > + echo \${i}M \$block_size \$((i%4)) >>$accessfile_full > + echo \${i}M \$block_size \$((i%4)) > + done > + ;; > + pread) dd if=/dev/zero count=\$3 iflag=count_bytes ;; > + can_write) ;; > + pwrite) dd of=/dev/null ;; > + can_trim) ;; > + trim) ;; > + can_zero) ;; > + trim) ;; zero) > + *) exit 2 ;; > +esac > +EOF > + > + > +test_me() { > + num_accesses=$1 > + shift > + > + qemu-io -f raw "$...