Libguestfs - Aug 2023 - [PATCH libnbd] copy/copy-file-to-qcow2-compressed.sh: Skip test for all zeroes disk

Occasionally this test will choose a random seed which results in an
all-zeroes disk.  The test tries to convert this to a compressed qcow2
file, and fails because no compressed clusters are detected in the
resulting file.  This happens because qcow2 stores zero clusters with
a special sparse representation, they are never stored compressed, so
a disk with only zeroes in it will never contain compressed clusters.

To fix this, detect an all-zeroes disk and skip.

Reported-by: Eric Blake
---
 copy/copy-file-to-qcow2-compressed.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/copy/copy-file-to-qcow2-compressed.sh
b/copy/copy-file-to-qcow2-compressed.sh
index 018c8bba2f..2706eadd66 100755
--- a/copy/copy-file-to-qcow2-compressed.sh
+++ b/copy/copy-file-to-qcow2-compressed.sh
@@ -25,6 +25,7 @@ requires $QEMU_NBD --version
 requires nbdkit --exit-with-parent --version
 requires nbdkit sparse-random --dump-plugin
 requires qemu-img --version
+requires nbdinfo --version
 #requires stat --version
 
 # Check the compress driver is supported by this qemu-nbd.
@@ -45,6 +46,15 @@ cleanup_fn rm -f $file1 $file2 $out1 $out2
 size=1G
 seed=$RANDOM
 
+# Occasionally we will choose a seed which results in a completely
+# empty file.  Skip this case.
+if nbdinfo --map --totals -- \
+        [ nbdkit --exit-with-parent sparse-random $size seed=$seed ] |
+    grep -sq '100.0%.*hole,zero'; then
+    echo "$0: bad seed chosen, skipping the test"
+    exit 77
+fi
+
 # Create a compressed qcow2 file1.
 #
 # sparse-random files should compress easily because by default each
-- 
2.41.0

After more testing today I pushed a slightly improved version of this
patch as:

https://gitlab.com/nbdkit/libnbd/-/commit/a333eec17a5ce4b2a4bbbd2f08bf5f4474382074

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On 8/8/23 00:28, Richard W.M. Jones wrote:
> Occasionally this test will choose a random seed which results in an
> all-zeroes disk.  The test tries to convert this to a compressed qcow2
> file, and fails because no compressed clusters are detected in the
> resulting file.  This happens because qcow2 stores zero clusters with
> a special sparse representation, they are never stored compressed, so
> a disk with only zeroes in it will never contain compressed clusters.
> 
> To fix this, detect an all-zeroes disk and skip.
> 
> Reported-by: Eric Blake
> ---
>  copy/copy-file-to-qcow2-compressed.sh | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/copy/copy-file-to-qcow2-compressed.sh
b/copy/copy-file-to-qcow2-compressed.sh
> index 018c8bba2f..2706eadd66 100755
> --- a/copy/copy-file-to-qcow2-compressed.sh
> +++ b/copy/copy-file-to-qcow2-compressed.sh
> @@ -25,6 +25,7 @@ requires $QEMU_NBD --version
>  requires nbdkit --exit-with-parent --version
>  requires nbdkit sparse-random --dump-plugin
>  requires qemu-img --version
> +requires nbdinfo --version
>  #requires stat --version
>  
>  # Check the compress driver is supported by this qemu-nbd.
> @@ -45,6 +46,15 @@ cleanup_fn rm -f $file1 $file2 $out1 $out2
>  size=1G
>  seed=$RANDOM
>  
> +# Occasionally we will choose a seed which results in a completely
> +# empty file.  Skip this case.
> +if nbdinfo --map --totals -- \
> +        [ nbdkit --exit-with-parent sparse-random $size seed=$seed ] |
> +    grep -sq '100.0%.*hole,zero'; then
> +    echo "$0: bad seed chosen, skipping the test"
> +    exit 77
> +fi
> +
>  # Create a compressed qcow2 file1.
>  #
>  # sparse-random files should compress easily because by default each
Nbdkit uses its own PRNG from "common/include/random.h", not the one
from libc. That means the seed=... parameter entirely determines the
random series -- not only does it become time-independent, but also
platform-independent.

And based on that, could we just eliminate "seed=$RANDOM" from this
test
case, and use simply "seed=1" (for example)? What we want is a
sparse-random disk image, generated both times with the same
not-fully-zero contents. seed=1 satisfies that, and makes sure the test
is always "armed" (and, not least, deterministic).

Laszlo

On Mon, Aug 07, 2023 at 11:28:42PM +0100, Richard W.M. Jones
wrote:
> Occasionally this test will choose a random seed which results in an
> all-zeroes disk.  The test tries to convert this to a compressed qcow2
> file, and fails because no compressed clusters are detected in the
> resulting file.  This happens because qcow2 stores zero clusters with
> a special sparse representation, they are never stored compressed, so
> a disk with only zeroes in it will never contain compressed clusters.
> 
> To fix this, detect an all-zeroes disk and skip.
Skipping a stochastic test on the cases where a random number set up
the corner case is still odd; our testsuite passes, but not always
with the same number of tests.  I understand benchmarks wanting to use
stochastic results, but this particular test seems like one where we
aren't buying ourselves any new coverage by using $RANDOM (other than
the fact that we found a corner case where nbdkit's sparse-random can
produce a 1G empty disk), and where a deterministic test better proves
our intent 100% of the time.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org