The nbdkit tests take a long time to run especially on armv7. To try to get an idea of which tests take the most time, and since I happen to have a qemu aarch64 *TCG* guest available, I ran the tests on that guest. A TCG guest is about the worst case for libguestfs, used by some tests, since it uses double-nested TCG, and as you can see below there are tests taking over 6 hours. In fact I didn't complete the run yet. test-cow.sh is still running (after 2+ hours) and there may be further tests to come. Results: 06:43:00 PASS: test-cache.sh 06:37:48 PASS: test-cache-on-read.sh [ test-cow.sh possibly here ] 00:35:41 PASS: test-partitioning3.sh 00:34:29 PASS: test-partitioning5.sh 00:34:03 PASS: test-linuxdisk-copy-out.sh 00:33:58 PASS: test-floppy.sh 00:30:45 PASS: test-partitioning2.sh 00:28:41 PASS: test-linuxdisk.sh 00:28:34 PASS: test-iso.sh 00:02:59 PASS: test-cache-max-size.sh 00:02:58 PASS: test-partitioning6.sh 00:02:55 PASS: test-partitioning4.sh 00:02:51 PASS: test-eflags.sh 00:01:11 PASS: test-export-name.sh 00:01:01 PASS: test-parallel-sh.sh 00:00:54 PASS: test-python.sh 00:00:44 PASS: test-sh-errors.sh 00:00:44 PASS: test-cacheextents.sh 00:00:41 PASS: test-sh-extents.sh 00:00:35 PASS: test-info-base64.sh Ignoring the ones which take mere minutes (those run in seconds on more reasonable hardware), here is a summary of how the tests are implemented: test-cache.sh Gx2 F D test-cache-on-read.sh Gx2 F D test-cow.sh Gx3 F test-partitioning3.sh G test-partitioning5.sh G F test-linuxdisk-copy-out.sh G Dx4 test-floppy.sh G Dx2 test-partitioning2.sh G F test-linuxdisk.sh G D test-iso.sh G D = using libguestfs download or similar APIs F = using libguestfs "fill" APIs which write lots of data G = guestfish xN = runs multiple times So there's quite a strong correlation between multiple runs of guestfish, and using those two APIs, and time taken. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
I pushed a change which should reduce the time taken in the test-cache-* and test-cow.sh tests. Again note this is double TCG which is the worst case for these tests and explains the excessively long times which wouldn't be seen on a normal build system. Updated results: 01:04:18 PASS: test-readahead 01:02:32 PASS: test-cow.sh 00:37:25 PASS: test-floppy.sh 00:34:57 PASS: test-shell 00:34:53 PASS: test-linuxdisk-copy-out.sh 00:33:57 PASS: test-partitioning3.sh 00:33:46 PASS: test-perl 00:33:38 PASS: test-partitioning5.sh 00:33:38 PASS: test-memory 00:33:28 PASS: test-ocaml 00:33:11 PASS: test-lua 00:32:33 PASS: test-tcl 00:32:16 PASS: test-offset 00:31:48 PASS: test-xz 00:31:47 PASS: test-xz-curl 00:30:19 PASS: test-partitioning2.sh 00:28:57 PASS: test-linuxdisk.sh 00:28:46 PASS: test-data 00:28:24 PASS: test-gzip 00:27:57 PASS: test-ext2 00:27:49 PASS: test-curl 00:27:16 PASS: test-iso.sh 00:25:56 PASS: test-nbd [all others < 7 mins] Analysis of what these tests do as in the previous email: test-readahead G Fx2 test-cow.sh Gx2 test-floppy.sh G Dx2 test-shell G test-linuxdisk-copy-out.sh G Dx4 test-partitioning3.sh G test-perl G test-partitioning5.sh G F test-memory G test-ocaml G test-lua G test-tcl G test-offset G test-xz G test-xz-curl G W test-partitioning2.sh G F test-linuxdisk.sh G D test-data G test-gzip G test-ext2 G test-curl G W test-iso.sh G test-nbd G D = using libguestfs download or similar APIs F = using libguestfs "fill" APIs which write lots of data G = guestfish/libguestfs W = curl/webserver xN = runs multiple times test-readahead needs more work. test-cow.sh is obdurately difficult to test. The others are good tests so there's probably not much we can or should do to them. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
I pushed these two patches which together reduce the build time in Koji from 7 hours down to 2 hours - almost all of that spent on the armv7 builder, the other architectures take only a few minutes. https://github.com/libguestfs/nbdkit/commit/fdbd0ef69f0c222d0cdc35d3c48297aff549abce https://github.com/libguestfs/nbdkit/commit/ce8357a883167b75ced16ea9f1488f7933b3e813 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Apparently Analagous Threads
- Re: Long running nbdkit tests
- [PATCH nbdkit v2 1/4] partitioning plugin: Support MBR logical partitions.
- [PATCH nbdkit v2 0/4] Support MBR logical partitions.
- [PATCH nbdkit] partitioning: Support MBR logical partitions.
- [PATCH v2 nbdkit] tests: Add generic requires.