Richard W.M. Jones
2023-Aug-07 22:28 UTC
[Libguestfs] [PATCH libnbd] copy/copy-file-to-qcow2-compressed.sh: Skip test for all zeroes disk
Occasionally this test will choose a random seed which results in an all-zeroes disk. The test tries to convert this to a compressed qcow2 file, and fails because no compressed clusters are detected in the resulting file. This happens because qcow2 stores zero clusters with a special sparse representation, they are never stored compressed, so a disk with only zeroes in it will never contain compressed clusters. To fix this, detect an all-zeroes disk and skip. Reported-by: Eric Blake --- copy/copy-file-to-qcow2-compressed.sh | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/copy/copy-file-to-qcow2-compressed.sh b/copy/copy-file-to-qcow2-compressed.sh index 018c8bba2f..2706eadd66 100755 --- a/copy/copy-file-to-qcow2-compressed.sh +++ b/copy/copy-file-to-qcow2-compressed.sh @@ -25,6 +25,7 @@ requires $QEMU_NBD --version requires nbdkit --exit-with-parent --version requires nbdkit sparse-random --dump-plugin requires qemu-img --version +requires nbdinfo --version #requires stat --version # Check the compress driver is supported by this qemu-nbd. @@ -45,6 +46,15 @@ cleanup_fn rm -f $file1 $file2 $out1 $out2 size=1G seed=$RANDOM +# Occasionally we will choose a seed which results in a completely +# empty file. Skip this case. +if nbdinfo --map --totals -- \ + [ nbdkit --exit-with-parent sparse-random $size seed=$seed ] | + grep -sq '100.0%.*hole,zero'; then + echo "$0: bad seed chosen, skipping the test" + exit 77 +fi + # Create a compressed qcow2 file1. # # sparse-random files should compress easily because by default each -- 2.41.0
Richard W.M. Jones
2023-Aug-08 09:37 UTC
[Libguestfs] [PATCH libnbd] copy/copy-file-to-qcow2-compressed.sh: Skip test for all zeroes disk
After more testing today I pushed a slightly improved version of this patch as: https://gitlab.com/nbdkit/libnbd/-/commit/a333eec17a5ce4b2a4bbbd2f08bf5f4474382074 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Laszlo Ersek
2023-Aug-08 11:38 UTC
[Libguestfs] [PATCH libnbd] copy/copy-file-to-qcow2-compressed.sh: Skip test for all zeroes disk
On 8/8/23 00:28, Richard W.M. Jones wrote:> Occasionally this test will choose a random seed which results in an > all-zeroes disk. The test tries to convert this to a compressed qcow2 > file, and fails because no compressed clusters are detected in the > resulting file. This happens because qcow2 stores zero clusters with > a special sparse representation, they are never stored compressed, so > a disk with only zeroes in it will never contain compressed clusters. > > To fix this, detect an all-zeroes disk and skip. > > Reported-by: Eric Blake > --- > copy/copy-file-to-qcow2-compressed.sh | 10 ++++++++++ > 1 file changed, 10 insertions(+) > > diff --git a/copy/copy-file-to-qcow2-compressed.sh b/copy/copy-file-to-qcow2-compressed.sh > index 018c8bba2f..2706eadd66 100755 > --- a/copy/copy-file-to-qcow2-compressed.sh > +++ b/copy/copy-file-to-qcow2-compressed.sh > @@ -25,6 +25,7 @@ requires $QEMU_NBD --version > requires nbdkit --exit-with-parent --version > requires nbdkit sparse-random --dump-plugin > requires qemu-img --version > +requires nbdinfo --version > #requires stat --version > > # Check the compress driver is supported by this qemu-nbd. > @@ -45,6 +46,15 @@ cleanup_fn rm -f $file1 $file2 $out1 $out2 > size=1G > seed=$RANDOM > > +# Occasionally we will choose a seed which results in a completely > +# empty file. Skip this case. > +if nbdinfo --map --totals -- \ > + [ nbdkit --exit-with-parent sparse-random $size seed=$seed ] | > + grep -sq '100.0%.*hole,zero'; then > + echo "$0: bad seed chosen, skipping the test" > + exit 77 > +fi > + > # Create a compressed qcow2 file1. > # > # sparse-random files should compress easily because by default eachNbdkit uses its own PRNG from "common/include/random.h", not the one from libc. That means the seed=... parameter entirely determines the random series -- not only does it become time-independent, but also platform-independent. And based on that, could we just eliminate "seed=$RANDOM" from this test case, and use simply "seed=1" (for example)? What we want is a sparse-random disk image, generated both times with the same not-fully-zero contents. seed=1 satisfies that, and makes sure the test is always "armed" (and, not least, deterministic). Laszlo
Eric Blake
2023-Aug-08 12:35 UTC
[Libguestfs] [PATCH libnbd] copy/copy-file-to-qcow2-compressed.sh: Skip test for all zeroes disk
On Mon, Aug 07, 2023 at 11:28:42PM +0100, Richard W.M. Jones wrote:> Occasionally this test will choose a random seed which results in an > all-zeroes disk. The test tries to convert this to a compressed qcow2 > file, and fails because no compressed clusters are detected in the > resulting file. This happens because qcow2 stores zero clusters with > a special sparse representation, they are never stored compressed, so > a disk with only zeroes in it will never contain compressed clusters. > > To fix this, detect an all-zeroes disk and skip.Skipping a stochastic test on the cases where a random number set up the corner case is still odd; our testsuite passes, but not always with the same number of tests. I understand benchmarks wanting to use stochastic results, but this particular test seems like one where we aren't buying ourselves any new coverage by using $RANDOM (other than the fact that we found a corner case where nbdkit's sparse-random can produce a 1G empty disk), and where a deterministic test better proves our intent 100% of the time. -- Eric Blake, Principal Software Engineer Red Hat, Inc. Virtualization: qemu.org | libguestfs.org
Reasonably Related Threads
- [PATCH libnbd] copy/copy-file-to-qcow2-compressed.sh: Skip test for all zeroes disk
- [PATCH nbdkit] tests/test-exportname.sh: Fix test.
- [libnbd PATCH] info: Keep request within 4G bound
- [libnbd PATCH] info: Add support for new 'qemu-nbd -A' qemu:allocation-depth
- [PATCH libnbd] info: Write output atomically.