[LLVMdev] QMTest vs. Dejagnu

Vladimir Prus
Mon, 29 Nov 2004 14:10:32 -0000

On Sunday 28 November 2004 00:24, Tanya Lattner wrote:

Just some comments from a QMTest user... Note however, that even with them,
dejagnu looks better.

> Cons of QMTest:
>  1) You have to use the gui to add directories.

I think you're wrong. Manually creating a directory would work, as QMTest
does
not place anything special in directories.

>  2) You have to use the gui to XFAIL a test.

Right.

>  3) It uses something called expectation files that you must load
>  to view which tests XFAIL. There is no way (that I have found) to
>  get a complete list of XFAILs..

Right. This appears to have deep roots in QMTest design, so I don't know how
to make using one file with "RUN" and "XFAIL" lines work --
the allowed
format of expectations file is hardcoded.

>  4) It is also hard to XFAIL across  platforms, because it requires
>  hacking an expectation file for each
> target, which must be done with the gui.

Right and very deplorable. Basically, QMTest is nice tool but it's not clear
how to use it for for cross-platform work like LLVM (or C++ Boost).

>  5) Intermediate output placement can not be controlled.

I don't understand this one. The intermediate output is placed where tests
want it to be placed; QMTest does not create any such output itself.

>  6) The output logs are not as clean.

I think the current version supports dejagnu-style logs.

>  7) Right now we are dependent on a specific version of QMTest.

Yea, that's a big problem.

> I propose that we switch over to using dejagnu by default, renaming
> check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
> for this release or keep it until 1.5.
>
> I'd appreciate your opinion or any feedback you may have.

Looks like dejagnu is better for this task (though I always thought it's
some
rusty tcl-based code). You might want to check on QMTest mailing list first,
in case they have some silver bullets at hand. After all QMTest was supposed
to test gcc, which is also cross-platform.

- Volodya