llvm dev - Oct 2008 - [LLVMdev] 2.4 Pre-release (v1) Available for Testing

On Fri, Oct 10, 2008 at 10:27 PM, Kenneth Boyd <zaimoni at zaimoni.com>
wrote:
> /* snip */
I think a pure C++ llvm test platform would work quite well
personally.  Not only could you test little parts of the API, but you
can also create IR in memory (or load the occasional file, but then
that would not test more of the things, which is kind of the purpose)
and either JIT it to test execution of compiled llvm code, as well as
being able to run various optimization passes in different orders of
different sets and still verify that the code works correctly.  With
such a C++ test platform you do not need to do anything with the
standard output/input as you can read everything directly, do not need
error codes, would just need to make sure everything works as
expected.  Such as create some IR for known complicated problems for
many of the passes and confirm that they always work and nothing
breaks, can even make some that fail now that some optimization pass
should optimize so that would help encourage it to be fixed so that,
that one test would pass.

I see no reason why any shell scripts, DejaGNU, etc... or anything
would be necessary.  If a compiler is capable of compiling llvm, then
it is certainly most capable of creating and handling the tests
through pure C++ as well, as such you would make the tests far more
distributable.  Because many of my programs need to work on both
Windows XP clients and my FreeBSD server, I tend to not handle
anything that will not work on both without too much extra crap, and
things like this have always worked perfectly.  C++ is extremely
capable, and running tests through it tend to execute faster, able to
test more things, can code the tests faster (there are wonderful
libraries that speed such things up, like boost), and it "Just Works"
everywhere.  Relying on posix for example, is just as bad and useless
for multiplatform work as is relying on something like MFC
(consequently, it is reasons like this that I also *HATE* the automake
tools as they create the most convoluted horrible to deal with build
scripts in existence, where-as cmake works wonderfully, and if you
really need to do special crap like automake things do, it is just as
easy to whip up a short C++ program, add its project to your build,
have it build and execute first before building the rest, all easy to
do, and very multiplatform).

I do find it humorous though in how so many people seem to rely on 50
little types of shell scripts spread across 50 different languages
when C++ can do it all, and with the right libraries can do it all in
just about the same short amount of code, all faster, all more
reliable, all easier to use, and by far more multiplatform (as well as
the fact you do not have to memorize 50 languages, especially horrors
like M4 and the makefiles and the other automake tools).

OvermindDL1 wrote:
> On Fri, Oct 10, 2008 at 10:27 PM, Kenneth Boyd <zaimoni at
zaimoni.com> wrote:
>   
>> /* snip */
>>     
>
> I think a pure C++ llvm test platform would work quite well
> personally.  Not only could you test little parts of the API, but you
> can also create IR in memory (or load the occasional file, but then
> that would not test more of the things, which is kind of the purpose)
> and either JIT it to test execution of compiled llvm code, as well as
> being able to run various optimization passes in different orders of
> different sets and still verify that the code works correctly.  With
> such a C++ test platform you do not need to do anything with the
> standard output/input as you can read everything directly, do not need
> error codes, would just need to make sure everything works as
> expected.  Such as create some IR for known complicated problems for
> many of the passes and confirm that they always work and nothing
> breaks, can even make some that fail now that some optimization pass
> should optimize so that would help encourage it to be fixed so that,
> that one test would pass.
>
> I see no reason why any shell scripts, DejaGNU, etc... or anything
> would be necessary.
I don't think anyone is arguing that these are *necessary*, just that 
the time-to-maintain is far less with a test framework that 
automatically picks up all test cases and uses the test cases themselves 
to determine when to run them, and how.
>   If a compiler is capable of compiling llvm, then
> it is certainly most capable of creating and handling the tests
> through pure C++ as well, as such you would make the tests far more
> distributable.
At the absolute minimum, simple counting of the success/failure/internal 
error results requires three different exit codes, and a test driver 
capable of tallying them up and reporting all failures (unexpected 
success, unexpected failure, and internal errors with a designated 
distinct exit code).

That much could be done in C++, although the maintainability of even 
this is much simpler using a Bourne shell script to coordinate the C++ 
test drivers.  If I didn't have to worry about automatically picking up 
the *.ll tests and configuring them via DejaGNU-command comments, 
creating said test driver for the *.ll files would be a rote-work task.

The problem is when you need to inspect stderr/stdout; the C and C++ 
standard libraries are pretty much not there, POSIX only works 
*NIX-side, and Boost generally locks out all but the most compliant or 
major compiliers for interprocess work.

The send-in-the-report feature of the current test framework is also 
mandatory; until that has an alternate, the DejaGNU framework should be 
staying.
> (consequently, it is reasons like this that I also *HATE* the automake
> tools as they create the most convoluted horrible to deal with build
> scripts in existence, where-as cmake works wonderfully, and if you
> really need to do special crap like automake things do, it is just as
> easy to whip up a short C++ program, add its project to your build,
> have it build and execute first before building the rest, all easy to
> do, and very multiplatform).
>   
CMake either works wonderfully (most of the time), or not at all. (My 
platform, MingW32.  CMake incorrectly classifies MingW32 vs CygWin by 
the presence of sh,  throws CygWin fullpaths at MingW32: catastrophic 
failure. Verified with CMake 2.6.1 and CMake 2.6.2, will not bootstrap.)

At least I can repair automake-configure generated makefiles when this 
happens.  CMake is unrepairable (which indicates a serious architectural 
flaw, not repairable merely by adding in enough intelligence to 
distinguish between CygWin and MingW32 with sh).
> I do find it humorous though in how so many people seem to rely on 50
> little types of shell scripts spread across 50 different languages
>   
No need to resort to hyperbole.  Five is credible.

Kenneth

On Sat, Oct 11, 2008 at 7:30 AM, Kenneth Boyd <zaimoni at zaimoni.com>
wrote:
> At the absolute minimum, simple counting of the success/failure/internal
> error results requires three different exit codes, and a test driver
> capable of tallying them up and reporting all failures (unexpected
> success, unexpected failure, and internal errors with a designated
> distinct exit code).
This is what I am not understanding about how you are thinking about
this.  There are a great deal more responses and having a C++ test
program, say boost::test, it can handle a great deal of information,
spitting out exactly what caused various things to fail, it is not
just a matter of running a variety of programs and testing them, those
are more useful if you are testing an external app like llvm-gcc, not
llvm itself, which is a library, not an application, and if it was
treated as such then its testing framework for it and it alone could
become a great deal more powerful while being far easier to use.  It
could quite literally just be a single extra project, each test could
be in its cpp file, no header for the tests needed, and just have
something like boost::test run them all in succession.  If a test is
truly expected to fail in a spectacular way (say, by abort()ing the
application, those really need to be fixed and removed, there are ways
to hook the abort call) those could be made into a simple separate exe
and run by the main boost::test framework as well, nothing special
required.

I use this style testing and it is very simple to use, very powerful,
plenty of information, and it tests llvm as a library, meaning it can
test *all* of it, not just what the mini-apps like llc and such
expose, since they just access specific parts of the library anyway.
No testing error codes, no watching the standard input/output (which
is a horrible way of testing regardless), etc...  And as stated, it is
also very simple for a C++ program to submit its own report, nothing
special there either.

If any mini test apps needed to be used, then you could still have the
main test program watch the standard input/output, that is a simple to
do regardless of the OS you are, but even using something like
boost::interprocess to communicate would be even far more powerful,
and still very simple.  And it seems to support more compilers then
llvm anyway (these are just the tested compilers, it no doubt supports
more just fine, still more then llvm either way):
    * Visual 7.1 Windows XP
    * Visual 8.0 Windows XP
    * GCC 4.1.1 MinGW
    * GCC 3.4.4 Cygwin
    * Intel 9.1 Windows XP
    * GCC 4.1.2 Linux
    * GCC 3.4.3 Solaris 11
    * GCC 4.0 MacOs 10.4.1

DejaGNU can send in reports, so can C++ itself.  It is simple for
boost::filesystem to parse through directories and, say, get all of
the .ll files, of which you can then run through various optimizers
and make sure the converted code is correct, or run through the
interpreter, or run through the JIT to test the JIT, etc...

That does not really surprise me about CMake, but then mingw is not a
primary compiler on Windows, more so is VC++ or Intel, either way a
bug should be submitted to the CMake devs.  I thought you could
override things like that with some command line parameters anyway.
You cannot force it to output mingw files directly?