search for: localtestsdir

This converts more of the tests (basically everything under tests/) to use the test harness, revealing some problems which have subsequently been fixed. Rich.

For v6: This is mainly just a rebase, but I have also added tests in the ocaml/ language bindings directory, and for all the OCaml-written virt tools. Rich.

v7: The only changes since v6 are those suggested by Pino in the review of v5.

I should probably start by saying this patch series isn't ready for review yet. This patch series adds a test harness to libguestfs. It allows us to run the tests outside the ordinary 'make check' path in the build tree. In particular, you can use this to run tests when libguestfs has been installed. 'make check' and the other 'make check-*' rules still work. The

Since v4: - More tests have been converted. - Testing local guests fixed. - Drop no-exec-stack test.

This is a more complete patch to add the test harness. The only parts missing now are the language bindings (except OCaml). The language bindings need a bit more thought. At the moment most language binding tests are done through some sort of shell script like perl/run-perl-tests which either runs each test itself or uses some language-specific machinary to run each test. The problem with that

Since v2: - Add perl tests. - Reworked and fixed the tests for virt-builder. - Some further minor bug fixes.

Since v3: - A large number of fixes, especially for running the tests on installed libguestfs. - Fixed EXTRA_DIST rules throughout. - Extra patch 17/17 which is a tidy-up of the generated XML listing guests. Rich.

On Sunday 05 October 2014 14:08:35 Richard W.M. Jones wrote: > +# Install the test harness. > +localtestsdir = $(alltestsdir) > +localtests_SCRIPTS = ../test-harness Shouldn't it rather go to libexec? > +../test-harness: > + rm -f $@ $@-t > + echo 'echo Warning: Install OCaml compiler in order to rebuild the test-harness.' > $@-t > + chmod +x $@-t > +...

On Tue, Oct 21, 2014 at 05:56:14PM +0200, Pino Toscano wrote: > On Sunday 05 October 2014 14:08:35 Richard W.M. Jones wrote: > > +# Install the test harness. > > +localtestsdir = $(alltestsdir) > > +localtests_SCRIPTS = ../test-harness > > Shouldn't it rather go to libexec? No, it's meant to be run by the end user. > > +../test-harness: > > + rm -f $@ $@-t > > + echo 'echo Warning: Install OCaml compiler in order to...

This has got to the stage where it actually works, both for running the tests in-tree and installed. The 'test-harness' script has become rather over-complex in the process however. Rich.

...ize.cmo \ main.cmo +test_harness_objects = \ + types.cmo \ + utils.cmo \ + tests.cmo \ + test_harness.cmo + EXTRA_DIST = $(sources) files-generated.txt OCAMLCFLAGS = -warn-error CDEFLMPSUVYZX-3 -I $(srcdir) -I . -package unix,str noinst_PROGRAM = generator +# Install the test harness. +localtestsdir = $(alltestsdir) +localtests_SCRIPTS = ../test-harness + if HAVE_OCAML generator: $(objects) $(OCAMLFIND) ocamlc $(OCAMLCFLAGS) -linkpkg $^ -o $@ +../test-harness: $(test_harness_objects) + $(OCAMLFIND) ocamlc $(OCAMLCFLAGS) -linkpkg $^ -o $@ + # Dependencies. %.cmi: %.mli $(OCAMLFIND)...

...clean. CLEANFILES = \ @@ -414,6 +425,7 @@ CLEANFILES = \ podwrapper.1 \ qemu-wrapper.sh \ stamp-guestfs-release-notes.pod \ + test-harness.1 \ tmp/disk* \ tmp/run-* \ tmp/valgrind-*.log @@ -423,6 +435,11 @@ clean-local: -rm -rf tmp/guestfs.* -rm -rf tmp/.guestfs-* +# Tests. + +localtestsdir = $(alltestsdir) +localtests_DATA = bindtests + # If you don't want to run all of the tests ('make check') then this # will just run libguestfs-test-tool for a quick check. Note this # is NOT a substitute for proper testing! @@ -541,6 +558,9 @@ check-slow: build-test-guests build-t...

...ize.cmo \ main.cmo +test_harness_objects = \ + types.cmo \ + utils.cmo \ + tests.cmo \ + test_harness.cmo + EXTRA_DIST = $(sources) files-generated.txt OCAMLCFLAGS = -warn-error CDEFLMPSUVYZX-3 -I $(srcdir) -I . -package unix,str noinst_PROGRAM = generator +# Install the test harness. +localtestsdir = $(alltestsdir) +localtests_SCRIPTS = ../test-harness + if HAVE_OCAML generator: $(objects) $(OCAMLFIND) ocamlc $(OCAMLCFLAGS) -linkpkg $^ -o $@ +../test-harness: $(test_harness_objects) + $(OCAMLFIND) ocamlc $(OCAMLCFLAGS) -linkpkg $^ -o $@ + # Dependencies. %.cmi: %.mli $(OCAMLFIND)...

...R A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +localtestsdir = $(alltestsdir)/tests/c-api + +localtests_PROGRAMS = \ + test-add-drive-opts \ + test-backend-settings \ + test-command \ + test-config \ + test-create-handle \ + test-debug-to-file \ + test-environment \ + test-event-string \ + test-just-header \ + test-just-header-cxx \ + test-last-errno \ + tes...

...R A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +localtestsdir = $(alltestsdir)/tests/c-api + +localtests_PROGRAMS = \ + test-add-drive-opts \ + test-add-libvirt-dom \ + test-backend-settings \ + test-command \ + test-config \ + test-create-handle \ + test-debug-to-file \ + test-dlopen \ + test-environment \ + test-event-string \ + test-just-header \ + test-ju...

These are my thoughts on adding a test harness to run tests instead of using automake. The aim of this exercise is to allow us to run the full test suite on an installed copy of libguestfs. Another aim is to allow us to work around all the limitations and problems of automake. The first patch makes an observation that since the ./run script sets up $PATH to contain all the directories