thr3ads.net - Libguestfs - Re: [Libguestfs] [PATCH supermin] build: use a custom test driver [Nov 2015]

If this information is useful, please help other people find it:
Share via:

Pino Toscano

2015-Nov-06 12:24 UTC

[Libguestfs] [PATCH supermin] build: use a custom test driver

Use a custom test driver for running the tests: based on the test-driver
provided by automake, it adds the running time of the test in each .trs
file.
---
 configure.ac        |   1 +
 guestfs-test-driver | 151 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/Makefile.am   |   2 +
 3 files changed, 154 insertions(+)
 create mode 100755 guestfs-test-driver

diff --git a/configure.ac b/configure.ac
index 8cbfa0a..6ff60c6 100644
--- a/configure.ac
+++ b/configure.ac
@@ -19,6 +19,7 @@ dnl Written by Richard W.M. Jones <rjones@redhat.com>
 
 AC_INIT([supermin],[5.1.13])
 AM_INIT_AUTOMAKE(foreign)
+AC_REQUIRE_AUX_FILE([guestfs-test-driver])
 
 dnl Check for basic C environment.
 AC_PROG_CC_STDC
diff --git a/guestfs-test-driver b/guestfs-test-driver
new file mode 100755
index 0000000..09854c2
--- /dev/null
+++ b/guestfs-test-driver
@@ -0,0 +1,151 @@
+#! /bin/sh
+# test-driver - basic testsuite driver script.
+
+scriptversion=2013-07-13.22; # UTC
+
+# Copyright (C) 2011-2015 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# As a special exception to the GNU General Public License, if you
+# distribute this file as part of a program that contains a
+# configuration script generated by Autoconf, you may include it under
+# the same distribution terms that you use for the rest of that program.
+
+# This file is maintained in Automake, please report
+# bugs to <bug-automake@gnu.org> or send patches to
+# <automake-patches@gnu.org>.
+
+# Make unconditional expansion of undefined variables an error.  This
+# helps a lot in preventing typo-related bugs.
+set -u
+
+usage_error ()
+{
+  echo "$0: $*" >&2
+  print_usage >&2
+  exit 2
+}
+
+print_usage ()
+{
+  cat <<END
+Usage:
+  test-driver --test-name=NAME --log-file=PATH --trs-file=PATH
+              [--expect-failure={yes|no}] [--color-tests={yes|no}]
+              [--enable-hard-errors={yes|no}] [--]
+              TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS]
+The '--test-name', '--log-file' and '--trs-file'
options are mandatory.
+END
+}
+
+test_name= # Used for reporting.
+log_file=  # Where to save the output of the test script.
+trs_file=  # Where to save the metadata of the test run.
+expect_failure=no
+color_tests=no
+enable_hard_errors=yes
+while test $# -gt 0; do
+  case $1 in
+  --help) print_usage; exit $?;;
+  --version) echo "test-driver $scriptversion"; exit $?;;
+  --test-name) test_name=$2; shift;;
+  --log-file) log_file=$2; shift;;
+  --trs-file) trs_file=$2; shift;;
+  --color-tests) color_tests=$2; shift;;
+  --expect-failure) expect_failure=$2; shift;;
+  --enable-hard-errors) enable_hard_errors=$2; shift;;
+  --) shift; break;;
+  -*) usage_error "invalid option: '$1'";;
+   *) break;;
+  esac
+  shift
+done
+
+missing_opts+test x"$test_name" = x &&
missing_opts="$missing_opts --test-name"
+test x"$log_file"  = x && missing_opts="$missing_opts
--log-file"
+test x"$trs_file"  = x && missing_opts="$missing_opts
--trs-file"
+if test x"$missing_opts" != x; then
+  usage_error "the following mandatory options are
missing:$missing_opts"
+fi
+
+if test $# -eq 0; then
+  usage_error "missing argument"
+fi
+
+if test $color_tests = yes; then
+  # Keep this in sync with 'lib/am/check.am:$(am__tty_colors)'.
+  red='[0;31m' # Red.
+  grn='[0;32m' # Green.
+  lgn='[1;32m' # Light green.
+  blu='[1;34m' # Blue.
+  mgn='[0;35m' # Magenta.
+  std=''     # No color.
+else
+  red= grn= lgn= blu= mgn= std+fi
+
+do_exit='rm -f $log_file $trs_file; (exit $st); exit $st'
+trap "st=129; $do_exit" 1
+trap "st=130; $do_exit" 2
+trap "st=141; $do_exit" 13
+trap "st=143; $do_exit" 15
+
+# Test script is run here.
+start_t="$(date +'%s')"
+"$@" >$log_file 2>&1
+estatus=$?
+end_t="$(date +'%s')"
+
+if test $enable_hard_errors = no && test $estatus -eq 99; then
+  tweaked_estatus=1
+else
+  tweaked_estatus=$estatus
+fi
+
+case $tweaked_estatus:$expect_failure in
+  0:yes) col=$red res=XPASS recheck=yes gcopy=yes;;
+  0:*)   col=$grn res=PASS  recheck=no  gcopy=no;;
+  77:*)  col=$blu res=SKIP  recheck=no  gcopy=yes;;
+  99:*)  col=$mgn res=ERROR recheck=yes gcopy=yes;;
+  *:yes) col=$lgn res=XFAIL recheck=no  gcopy=yes;;
+  *:*)   col=$red res=FAIL  recheck=yes gcopy=yes;;
+esac
+
+# Report the test outcome and exit status in the logs, so that one can
+# know whether the test passed or failed simply by looking at the
'.log'
+# file, without the need of also peaking into the corresponding '.trs'
+# file (automake bug#11814).
+echo "$res $test_name (exit status: $estatus)" >>$log_file
+
+# Report outcome to console.
+echo "${col}${res}${std}: $test_name"
+
+# Register the test result, and other relevant metadata.
+echo ":test-result: $res" > $trs_file
+echo ":global-test-result: $res" >> $trs_file
+echo ":recheck: $recheck" >> $trs_file
+echo ":copy-in-global-log: $gcopy" >> $trs_file
+echo ":guestfs-time: $(($end_t - $start_t))" >> $trs_file
+
+# Local Variables:
+# mode: shell-script
+# sh-indentation: 2
+# eval: (add-hook 'write-file-hooks 'time-stamp)
+# time-stamp-start: "scriptversion="
+# time-stamp-format: "%:y-%02m-%02d.%02H"
+# time-stamp-time-zone: "UTC"
+# time-stamp-end: "; # UTC"
+# End:
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 4ebc748..49a8edf 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -17,6 +17,8 @@
 #
 # Written by Richard W.M. Jones <rjones@redhat.com>
 
+LOG_DRIVER = env $(SHELL) $(top_srcdir)/guestfs-test-driver
+
 EXTRA_DIST = \
 	automake2junit.ml \
 	$(TESTS)
-- 
2.1.0

Richard W.M. Jones

2015-Nov-06 13:00 UTC

head link

Re: [Libguestfs] [PATCH supermin] build: use a custom test driver

On Fri, Nov 06, 2015 at 01:24:07PM +0100, Pino Toscano
wrote:> Use a custom test driver for running the tests: based on the test-driver
> provided by automake, it adds the running time of the test in each .trs
> file.
> ---
>  configure.ac        |   1 +
>  guestfs-test-driver | 151
++++++++++++++++++++++++++++++++++++++++++++++++++++
supermin-test-driver ..?

I applied this to the supermin tree to try and see how it worked,
but I can't see what it's supposed to do.  I still see the usual
`PASS:' lines in the output.

I don't know whether or not you use emacs, but:
> +# Local Variables:
> +# mode: shell-script
> +# sh-indentation: 2
> +# eval: (add-hook 'write-file-hooks 'time-stamp)
> +# time-stamp-start: "scriptversion="
> +# time-stamp-format: "%:y-%02m-%02d.%02H"
> +# time-stamp-time-zone: "UTC"
> +# time-stamp-end: "; # UTC"
> +# End:
are super annoying.  It even popped up when I opened the reply to this
email message.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Pino Toscano

2015-Nov-06 13:12 UTC

head link

Re: [Libguestfs] [PATCH supermin] build: use a custom test driver

On Friday 06 November 2015 13:00:03 Richard W.M. Jones
wrote:> On Fri, Nov 06, 2015 at 01:24:07PM +0100, Pino Toscano wrote:
> > Use a custom test driver for running the tests: based on the
test-driver
> > provided by automake, it adds the running time of the test in each
.trs
> > file.
> > ---
> >  configure.ac        |   1 +
> >  guestfs-test-driver | 151
++++++++++++++++++++++++++++++++++++++++++++++++++++
> 
> supermin-test-driver ..?
Can do.
> I applied this to the supermin tree to try and see how it worked,
> but I can't see what it's supposed to do.  I still see the usual
> `PASS:' lines in the output.
Yes, that's expected. The difference is just in what gets in the
resulting .trs files, which now have a line like:
:guestfs-time: 1
which indicates how much time each test took; this way, we can parse it
and improve the resulting XML for junit (produced by
tests/automake2junit.ml in CI).

The plan is to apply the same also to libguestfs when switching to
parallel-tests, so we can get proper test results for CI even without
$RUN_OUTPUT_FILE.
> I don't know whether or not you use emacs, but:
> 
> > +# Local Variables:
> > +# mode: shell-script
> > +# sh-indentation: 2
> > +# eval: (add-hook 'write-file-hooks 'time-stamp)
> > +# time-stamp-start: "scriptversion="
> > +# time-stamp-format: "%:y-%02m-%02d.%02H"
> > +# time-stamp-time-zone: "UTC"
> > +# time-stamp-end: "; # UTC"
> > +# End:
> 
> are super annoying.  It even popped up when I opened the reply to this
> email message.
This is part of the test-driver file in upstream automake (which gets
copied in the build tree).  Should I remove them from the local copy?

-- 
Pino Toscano

Maybe Matching Threads

Search for more maybe matching threads

Libguestfs - Nov 2015 - Re: [PATCH supermin] build: use a custom test driver

[Libguestfs] [PATCH supermin] build: use a custom test driver

Re: [Libguestfs] [PATCH supermin] build: use a custom test driver

Re: [Libguestfs] [PATCH supermin] build: use a custom test driver

Maybe Matching Threads