Libguestfs - May 2014 - [PATCH] build: Enable parallel tests.

This patch enables parallel tests.

This has a small performance benefit.

Unfortunately several tests still overwrite common filenames
(eg. 'test.out') which means that if you apply this patch then tests
will fail randomly.

To fix this requires every test to be examined manually to ensure that
it does not use a common filename, eg. changing 'test.out' to
'<testname>.out'.

Rich.

Using 'make check' will (with newish automake) run the tests
in parallel.

This has several consequences:

 - The tests run faster overall (measured: 20 mins -> 14 mins)

 - Failure output is written to a file, not displayed, requiring
   changes to autobuilders.

 - Tests must be modified so they do not overwrite each others files.
---
 .gitignore   |  2 ++
 configure.ac | 18 +-----------------
 2 files changed, 3 insertions(+), 17 deletions(-)

diff --git a/.gitignore b/.gitignore
index 1b9ce89..82d6367 100644
--- a/.gitignore
+++ b/.gitignore
@@ -13,11 +13,13 @@
 *.jar
 *.la
 *.lo
+*.log
 *.o
 *.orig
 *.patch
 *.rej
 *.swp
+*.trs
 
 bindtests.tmp
 cscope.out
diff --git a/configure.ac b/configure.ac
index 358025f..98b7642 100644
--- a/configure.ac
+++ b/configure.ac
@@ -31,23 +31,7 @@ m4_define([libguestfs_release], [13])
 AC_INIT([libguestfs],libguestfs_major.libguestfs_minor.libguestfs_release)
 AC_CONFIG_AUX_DIR([build-aux])
 
-dnl Initialize automake.  automake < 1.12 didn't have serial-tests and
-dnl gives an error if it sees this, but for automake >= 1.13
-dnl serial-tests is required so we have to include it.  Solution is to
-dnl test for the version of automake (by running an external command)
-dnl and provide it if necessary.  Note we have to do this entirely using
-dnl m4 macros since automake queries this macro by running
-dnl 'autoconf --trace'.
-m4_define([serial_tests], [
-    m4_esyscmd([automake --version | head -1 | awk '
-                     {
-                       split ($NF, version, ".");
-                       if (version[1] == 1 && version[2] >= 12)
-                         print "serial-tests";
-                     }'
-    ])
-])
-AM_INIT_AUTOMAKE(foreign serial_tests) dnl NB: Do not [quote] this parameter.
+AM_INIT_AUTOMAKE(foreign) dnl NB: Do not [quote] this parameter.
 
 m4_ifndef([AM_SILENT_RULES], [m4_define([AM_SILENT_RULES],[])])
 AM_SILENT_RULES([yes]) # make --enable-silent-rules the default.
-- 
1.9.0

I had a think about what to do about tests.  I didn't come to any
conclusions :-( but here are some notes anyway.

The problem
-----------

At the moment you can run the tests by doing 'make check'.  This uses
the automake test framework which sucks greatly (like the rest of
automake) but is slightly better than the alternatives (like the rest
of automake).

What we want to do is to make the guests installable.  You would
install a package 'libguestfs-tests', run some script from that
package, and it would run the tests on the installed copy of
libguestfs.

This lets you separate building and testing into two separate tasks,
which is useful for Koji -- the current build + test can take hours.
It would let us do testing in more places, for example, building the
package in Koji and then testing it on baremetal -- currently not
possible with Koji.  We would also like to run the tests after other
packages have changed, especially qemu & the kernel -- tieing the
tests into the build makes this difficult.

Other issues
------------

We still want 'make check' to work, and it should always be possible
to run the tests as non-root even when installed.

Tests have dependencies between them.

It should be possible to run the tests in parallel, where dependencies
permit.

There should be a flexible way to skip tests which doesn't require
every test to parse a $SKIP_* environment variable, which is what
happens now.

The current system keeps many tests in the same directory as the
program being tested, eg. fish/ contains guestfish tests.  That's a
good thing, I think.  At the same time we have a tests/ directory that
has non-specific tests, and I think we should keep that too.

Language binding tests will probably have to stay as they are now,
since the tests use language-specific test harnesses
(eg. rubygem-minitest).

It shouldn't require installing any other software.

It would be nice if tests could generate temporary files without
needing to give them unique names.

Background reading
------------------

I'm quite intrigued by Colin's "InstalledTests" plan for
GNOME:
https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests
There are also some relevant links at the bottom of that page.

He is certainly trying to solve some of the same problems.

Automake installable tests
--------------------------

I think the make the tests installable, we are surely going to need to
write automake rules like:

  testsdir = $(libdir)/guestfs/tests/fish
  fish_testsdir = $(testsdir)/fish
  fish_tests_SCRIPTS = test-a.sh test-add-domain.sh #etc
  fish_tests_DATA = # any data files required for the test

This creates a directory of test scripts, but it doesn't solve how to
run them or how to express dependencies between them.

We could drop a Makefile into the test directory.  How to generate that?

We could have some sort of script as a test harness.

How do we ensure 'make check' still works?  Obvious plan is to have a
check-local rule that does:

  check-local:
    make install testsdir=$(top_builddir)/tmp/check/
    cd $(top_builddir)/tmp/check && run-the-tests

That gets ugly if you want to allow the user to run a single
subdirectory of tests, or to run single tests (something I often want
to do).

It seems like we need to have some kind of program to manage tests.
gnome-desktop-testing-runner is a contender here
(https://git.gnome.org/browse/gnome-desktop-testing/tree/src/gnome-desktop-testing-runner.c)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org