On Sat, Oct 11, 2008 at 7:30 AM, Kenneth Boyd <zaimoni at zaimoni.com> wrote:> At the absolute minimum, simple counting of the success/failure/internal > error results requires three different exit codes, and a test driver > capable of tallying them up and reporting all failures (unexpected > success, unexpected failure, and internal errors with a designated > distinct exit code).This is what I am not understanding about how you are thinking about this. There are a great deal more responses and having a C++ test program, say boost::test, it can handle a great deal of information, spitting out exactly what caused various things to fail, it is not just a matter of running a variety of programs and testing them, those are more useful if you are testing an external app like llvm-gcc, not llvm itself, which is a library, not an application, and if it was treated as such then its testing framework for it and it alone could become a great deal more powerful while being far easier to use. It could quite literally just be a single extra project, each test could be in its cpp file, no header for the tests needed, and just have something like boost::test run them all in succession. If a test is truly expected to fail in a spectacular way (say, by abort()ing the application, those really need to be fixed and removed, there are ways to hook the abort call) those could be made into a simple separate exe and run by the main boost::test framework as well, nothing special required. I use this style testing and it is very simple to use, very powerful, plenty of information, and it tests llvm as a library, meaning it can test *all* of it, not just what the mini-apps like llc and such expose, since they just access specific parts of the library anyway. No testing error codes, no watching the standard input/output (which is a horrible way of testing regardless), etc... And as stated, it is also very simple for a C++ program to submit its own report, nothing special there either. If any mini test apps needed to be used, then you could still have the main test program watch the standard input/output, that is a simple to do regardless of the OS you are, but even using something like boost::interprocess to communicate would be even far more powerful, and still very simple. And it seems to support more compilers then llvm anyway (these are just the tested compilers, it no doubt supports more just fine, still more then llvm either way): * Visual 7.1 Windows XP * Visual 8.0 Windows XP * GCC 4.1.1 MinGW * GCC 3.4.4 Cygwin * Intel 9.1 Windows XP * GCC 4.1.2 Linux * GCC 3.4.3 Solaris 11 * GCC 4.0 MacOs 10.4.1 DejaGNU can send in reports, so can C++ itself. It is simple for boost::filesystem to parse through directories and, say, get all of the .ll files, of which you can then run through various optimizers and make sure the converted code is correct, or run through the interpreter, or run through the JIT to test the JIT, etc... That does not really surprise me about CMake, but then mingw is not a primary compiler on Windows, more so is VC++ or Intel, either way a bug should be submitted to the CMake devs. I thought you could override things like that with some command line parameters anyway. You cannot force it to output mingw files directly?
I should apologize. I have been trying to keep from attacking the current process that llvm uses for testing (although I do still agree that the current testing framework is more for testing llvm-gcc and not llvm), but I have not quite been so successful. Some basic background. I program for generally just two systems, Windows XP and FreeBSD. These are rather polar opposites, and most 'multi-platform' things seem to be designed for things like multiple platforms of linux, and not actually multiple platforms. As such I have had to find just about everything in every way I can that just makes it simple to compile code for both XP and BSD (even with its usually rather outdated gcc compiler), and it has been wonderfully successful; I can, quite literally, just drop the source without any makefiles and such, and just basically have it build everything in say a source and header directory, using whatever optimizations and such I want, and it just works. I do use CMake, but honestly it is rather rare as I find it generally just easier to compile and link everything myself due to how simple I try to keep everything. I develop from Visual Studio 2005 with the Visual Assist addon (I would drop VC++ in a hearbeat if there was some other IDE, say like Eclipse, that got anywhere near the feature-set that the Visual Assist add-on has, so far nothing is getting close, without Visual Assist though, I would prefer something else, like Eclipse or so, to VC++, but Visual Assist just swings the favor overwhelmingly over), and everything is dumped into an SVN repository running on the BSD box, upon which I will either compile the project through VC++ on Windows, or I will login and sync to the svn on the BSD machine and just run the very simple makefile (which, quite literally, just builds everything together in specific project directories at the optimization and such levels I want, nothing special). I have had no real problems (other then some lacking template support in the older GCC versions that the older BSD's used to use, the newer ones are better though, boost still worked with the older ones, but not all my personal code did). Either way, because of the way I build, I have developed a rather large lack of interest in things that I do not consider truly multi-platform. As such I have found that just doing everything in C++ itself just always works, no hassles. At rare times I will whip up something in python (although I, oddly enough, seem to have to put in more platform specific tests in it then I do in my C++ code), but for any of my bigger projects I just stick with C++ itself, I am after all a C++ programmer first and foremost.
On Oct 11, 2008, at 11:06 AM, OvermindDL1 wrote:> I should apologize. I have been trying to keep from attacking the > current process that llvm uses for testing (although I do still agree > that the current testing framework is more for testing llvm-gcc and > not llvm), but I have not quite been so successful.Ideas and feedback are useful. However, one systematic problem that the LLVM community has is that most of the regular contributors are on unix systems. As you probably know, much open source software works by having people 'scratch their itch'. You (and many other people on windows) are very itchy about this, but you're not producing code. If you have patches to start making specific progress towards your goal, they would be really welcome. Until then, we'll all admit that LLVM could work a lot better on windows, and follow it up with "patches welcome". That said, I am really thrilled about the recent progress towards cmake'ifying llvm, and we do have some very helpful developers on windows (thank you!). There are just a few very specific people who are very willing to complain about lack of feature x, but are apparently unwilling to help make it a reality. -Chris
OvermindDL1 wrote:> On Sat, Oct 11, 2008 at 7:30 AM, Kenneth Boyd <zaimoni at zaimoni.com> wrote: > >> At the absolute minimum, simple counting of the success/failure/internal >> error results requires three different exit codes, and a test driver >> capable of tallying them up and reporting all failures (unexpected >> success, unexpected failure, and internal errors with a designated >> distinct exit code). >> > > This is what I am not understanding about how you are thinking about > this. There are a great deal more responses and having a C++ test > program,Right. Remember, I'm considering what features are actually needed, with a high priority on avoiding duplicate configuration (and obviously a very low priority on actually having anything releasable.) In MSVC and MingW32, abort and assert have the same exit code (3). LLVM relies on assert in debug mode, so being able to distinguish assert from mere failure is important in a useful test suite coordinator for LLVM. An XFAILed test should still be reported as misbehaving if an assert fires. What I'm looking to (re)implement is: * exit code: classify on success/failure/assert|abort [prototype working in Bourne shell script] * if we have an expected success/failure, analyze the stdout/stderr and see if that is what is expected.> say boost::test, it can handle a great deal of information, > spitting out exactly what caused various things to fail, it is not > just a matter of running a variety of programs and testing them, those > are more useful if you are testing an external app like llvm-gcc, not > llvm itself, which is a library, not an application, and if it was > treated as such then its testing framework for it and it alone could > become a great deal more powerful while being far easier to use.In my opinion, both levels of testing (external application and detailed library testing) are pragmatic. The detailed library testing is not really doable in DejaGNU, it should be in C++.> .... > I use this style testing and it is very simple to use, very powerful, > plenty of information, and it tests llvm as a library, meaning it can > test *all* of it, not just what the mini-apps like llc and such > expose, since they just access specific parts of the library anyway. > No testing error codes, no watching the standard input/output (which > is a horrible way of testing regardless), etc... And as stated, it is > also very simple for a C++ program to submit its own report, nothing > special there either. > > If any mini test apps needed to be used, then you could still have the > main test program watch the standard input/output, that is a simple to > do regardless of the OS you are,Controlling both stdin and stdout is very painful in Windows C/C++, and pretty much completely unrelated to any other C/C++ platform. Something like Boost.Interprocess is required to paper over the differences between Windows and POSIX.> That does not really surprise me about CMake, but then mingw is not a > primary compiler on Windows, more so is VC++ or Intel, either way a > bug should be submitted to the CMake devs.I do not want to troll the CMake devteam, so I will not submit the bug report without a full-blown patch. My internal priority for that CMake patch is low, as I need only minimal patching to use the autoconf-generated configure script to build LLVM. Right now it's just llvm.config.in.in that needs patching (the failsafed GenLibDeps.pl script went in recently); I'll float a proper RFC on llvm-commit when I actually schedule time to do a non-hackish patch. (Strategy: abspath from the Cwd module does The Right Thing, but it is standard only from Perl 5.6.0 upwards. Hmm...but even though the *NIX-specific code would work indefinitely far back, there's a use 5.006 which causes the script to fail on anything lower than 5.6.0. The natural flow of control would be to try use Cwd 'abspath'; and if that fails, fall back to the current *NIX-specific code.)> I thought you could > override things like that with some command line parameters anyway. > You cannot force it to output mingw files directly? >Correct. I know which project type will give the correct file paths (MSYS), but since I have sh both CMake 2.6.1 and CMake 2.6.2 categorically refuse to generate the MSYS CMakefile; even the command-line parameter approach fails. If there's nothing to edit, I can't fix it manually. Kenneth
On Sat, Oct 11, 2008 at 1:24 PM, Kenneth Boyd <zaimoni at zaimoni.com> wrote:> /* snip */Actually, my biggest issue with llvm (which highly impacts testing as well) is the use of abort(), which I have resolved in my local copy by just searching for the full word "abort" everywhere and just replacing them with exceptions that relay what happened. Reason this is my biggest issue is that the llvm library is just a part of the full program that will be using it, and I cannot have llvm abort()ing on some bad passed in code when this application needs to have the best possible up-time, I need to send a message back to the node saying that the compile failed, not crash. I can understand using abort in little one-off programs like llc or opt or something that just does one function and ends, but llvm is a library, as such it may be in programs that need to stay persistently up, the use of abort() everywhere is really painful in such circumstances, even an error handler that can be registered by the parent app into the library would work better, although I prefer exceptions for these since they truly are exceptional events. Personally I also do not like asserts either, as that is indicating you are expecting something bad to happen; such a circumstance should have code to handle it, or throw an exception. Which makes me curious, if I submitted a patch that got rid of llvms use of abort()s all over the place, and replaces them with exceptions (the program dies either way if it is unhandled, but with exceptions you at least get a chance to handle it and recover), what is the chance it would be accepted, especially since I see no usage of exceptions at all (even if it passes through the C api, think about it, if an abort occurs, the program dies, if an exception is thrown through the C api, the program still dies, either way it dies, where-as when using the C++ interface you can at least handle it so the parent application need not die). Could always go with registering an error handler with the llvm api, but that would require more code...
Kenneth Boyd <zaimoni at zaimoni.com> writes: [snip]>> That does not really surprise me about CMake, but then mingw is not a >> primary compiler on Windows, more so is VC++ or Intel, either way a >> bug should be submitted to the CMake devs. > I do not want to troll the CMake devteam, so I will not submit the bug > report without a full-blown patch.The CMake devteam is very responsive about bug reports. They will certainly appreciate your bug report, no patch required.> My internal priority for that CMake patch is low, as I need only minimal > patching to use the autoconf-generated configure script to build LLVM. > Right now it's just llvm.config.in.in that needs patching (the failsafed > GenLibDeps.pl script went in recently); I'll float a proper RFC on > llvm-commit when I actually schedule time to do a non-hackish patch.I'm seeing a failure related to circular library references while building LLVM with CMake and MSYS. This didn't happen on the past. Building with the configure script works, so it seems something related to CMake. Do you have any insight on this? [snip]>> I thought you could >> override things like that with some command line parameters anyway. >> You cannot force it to output mingw files directly? >> > Correct. I know which project type will give the correct file paths > (MSYS), but since I have sh both CMake 2.6.1 and CMake 2.6.2 > categorically refuse to generate the MSYS CMakefile; even the > command-line parameter approach fails.Apart from the circular library references mentioned above, CMake works fine with MSYS for me (well, the generated makefiles doesn't work with parallel make (-j x) but that's another issue). [snip] -- Oscar
I've been using gtest (http://code.google.com/p/googletest/) for all of my frontend unit tests and I'm very happy with it. It does all of that automatic test discovery stuff pretty well. I haven't tried the XML test report generation stuff, but it does have that capability. I don't know much about DejaGNU, and from what little I know about it, I'm not sure its worth my time to learn about it. Just reading the intro doc scares me, it feels like autoconf/automake or sendmail, i.e. something bolted together out of several different scripting systems. So, I guess my question would be, what requirements are being satisfied by DejaGNU that wouldn't be satisfied by a C++-based testing framework? OvermindDL1 wrote:> On Sat, Oct 11, 2008 at 7:30 AM, Kenneth Boyd <zaimoni at zaimoni.com> wrote: > >> At the absolute minimum, simple counting of the success/failure/internal >> error results requires three different exit codes, and a test driver >> capable of tallying them up and reporting all failures (unexpected >> success, unexpected failure, and internal errors with a designated >> distinct exit code). >> > > This is what I am not understanding about how you are thinking about > this. There are a great deal more responses and having a C++ test > program, say boost::test, it can handle a great deal of information, > spitting out exactly what caused various things to fail, it is not > just a matter of running a variety of programs and testing them, those > are more useful if you are testing an external app like llvm-gcc, not > llvm itself, which is a library, not an application, and if it was > treated as such then its testing framework for it and it alone could > become a great deal more powerful while being far easier to use. It > could quite literally just be a single extra project, each test could > be in its cpp file, no header for the tests needed, and just have > something like boost::test run them all in succession. If a test is > truly expected to fail in a spectacular way (say, by abort()ing the > application, those really need to be fixed and removed, there are ways > to hook the abort call) those could be made into a simple separate exe > and run by the main boost::test framework as well, nothing special > required. > > I use this style testing and it is very simple to use, very powerful, > plenty of information, and it tests llvm as a library, meaning it can > test *all* of it, not just what the mini-apps like llc and such > expose, since they just access specific parts of the library anyway. > No testing error codes, no watching the standard input/output (which > is a horrible way of testing regardless), etc... And as stated, it is > also very simple for a C++ program to submit its own report, nothing > special there either. > > If any mini test apps needed to be used, then you could still have the > main test program watch the standard input/output, that is a simple to > do regardless of the OS you are, but even using something like > boost::interprocess to communicate would be even far more powerful, > and still very simple. And it seems to support more compilers then > llvm anyway (these are just the tested compilers, it no doubt supports > more just fine, still more then llvm either way): > * Visual 7.1 Windows XP > * Visual 8.0 Windows XP > * GCC 4.1.1 MinGW > * GCC 3.4.4 Cygwin > * Intel 9.1 Windows XP > * GCC 4.1.2 Linux > * GCC 3.4.3 Solaris 11 > * GCC 4.0 MacOs 10.4.1 > > DejaGNU can send in reports, so can C++ itself. It is simple for > boost::filesystem to parse through directories and, say, get all of > the .ll files, of which you can then run through various optimizers > and make sure the converted code is correct, or run through the > interpreter, or run through the JIT to test the JIT, etc... > > That does not really surprise me about CMake, but then mingw is not a > primary compiler on Windows, more so is VC++ or Intel, either way a > bug should be submitted to the CMake devs. I thought you could > override things like that with some command line parameters anyway. > You cannot force it to output mingw files directly? > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > >
On Sun, Oct 12, 2008 at 12:58 AM, Talin <viridia at gmail.com> wrote:> I've been using gtest (http://code.google.com/p/googletest/) for all of > my frontend unit tests and I'm very happy with it. It does all of that > automatic test discovery stuff pretty well. I haven't tried the XML test > report generation stuff, but it does have that capability. > > I don't know much about DejaGNU, and from what little I know about it, > I'm not sure its worth my time to learn about it. Just reading the intro > doc scares me, it feels like autoconf/automake or sendmail, i.e. > something bolted together out of several different scripting systems. > > So, I guess my question would be, what requirements are being satisfied > by DejaGNU that wouldn't be satisfied by a C++-based testing framework?I have not heard of the google testing framework; from reading it is not near as powerful as boost::test but it seems to fit llvm's guidelines quite well. And yes, I am also quite curious what about DejaGNU cannot be done in C++?
Talin wrote:> I've been using gtest (http://code.google.com/p/googletest/) for all of > my frontend unit tests and I'm very happy with it. It does all of that > automatic test discovery stuff pretty well. I haven't tried the XML test > report generation stuff, but it does have that capability. >Ok.> I don't know much about DejaGNU, and from what little I know about it, > I'm not sure its worth my time to learn about it. Just reading the intro > doc scares me, it feels like autoconf/automake or sendmail, i.e. > something bolted together out of several different scripting systems. >Understood. But it's the current test driver system.> So, I guess my question would be, what requirements are being satisfied > by DejaGNU that wouldn't be satisfied by a C++-based testing framework? >With the current system, the full details of how to run the test are read from the test itself. This includes whether the test should fail or succeed (most of the current ones are expected to succeed, as most of the failure cases are actually invariant violations) Kenneth
On Sat, Oct 11, 2008 at 11:58 PM, Talin <viridia at gmail.com> wrote:> I've been using gtest (http://code.google.com/p/googletest/) for all of > my frontend unit tests and I'm very happy with it. It does all of that > automatic test discovery stuff pretty well. I haven't tried the XML test > report generation stuff, but it does have that capability.Justs to chime in: I'm one of the gtest devs. I'd like to add that gtest is very portable, more so than llvm; it even works on WinCE and blackberry. If there are specific features needed by LLVM, depending on what it is I may be able to get it into gtest. Keir> > > I don't know much about DejaGNU, and from what little I know about it, > I'm not sure its worth my time to learn about it. Just reading the intro > doc scares me, it feels like autoconf/automake or sendmail, i.e. > something bolted together out of several different scripting systems. > > So, I guess my question would be, what requirements are being satisfied > by DejaGNU that wouldn't be satisfied by a C++-based testing framework? > > OvermindDL1 wrote: > > On Sat, Oct 11, 2008 at 7:30 AM, Kenneth Boyd <zaimoni at zaimoni.com> > wrote: > > > >> At the absolute minimum, simple counting of the success/failure/internal > >> error results requires three different exit codes, and a test driver > >> capable of tallying them up and reporting all failures (unexpected > >> success, unexpected failure, and internal errors with a designated > >> distinct exit code). > >> > > > > This is what I am not understanding about how you are thinking about > > this. There are a great deal more responses and having a C++ test > > program, say boost::test, it can handle a great deal of information, > > spitting out exactly what caused various things to fail, it is not > > just a matter of running a variety of programs and testing them, those > > are more useful if you are testing an external app like llvm-gcc, not > > llvm itself, which is a library, not an application, and if it was > > treated as such then its testing framework for it and it alone could > > become a great deal more powerful while being far easier to use. It > > could quite literally just be a single extra project, each test could > > be in its cpp file, no header for the tests needed, and just have > > something like boost::test run them all in succession. If a test is > > truly expected to fail in a spectacular way (say, by abort()ing the > > application, those really need to be fixed and removed, there are ways > > to hook the abort call) those could be made into a simple separate exe > > and run by the main boost::test framework as well, nothing special > > required. > > > > I use this style testing and it is very simple to use, very powerful, > > plenty of information, and it tests llvm as a library, meaning it can > > test *all* of it, not just what the mini-apps like llc and such > > expose, since they just access specific parts of the library anyway. > > No testing error codes, no watching the standard input/output (which > > is a horrible way of testing regardless), etc... And as stated, it is > > also very simple for a C++ program to submit its own report, nothing > > special there either. > > > > If any mini test apps needed to be used, then you could still have the > > main test program watch the standard input/output, that is a simple to > > do regardless of the OS you are, but even using something like > > boost::interprocess to communicate would be even far more powerful, > > and still very simple. And it seems to support more compilers then > > llvm anyway (these are just the tested compilers, it no doubt supports > > more just fine, still more then llvm either way): > > * Visual 7.1 Windows XP > > * Visual 8.0 Windows XP > > * GCC 4.1.1 MinGW > > * GCC 3.4.4 Cygwin > > * Intel 9.1 Windows XP > > * GCC 4.1.2 Linux > > * GCC 3.4.3 Solaris 11 > > * GCC 4.0 MacOs 10.4.1 > > > > DejaGNU can send in reports, so can C++ itself. It is simple for > > boost::filesystem to parse through directories and, say, get all of > > the .ll files, of which you can then run through various optimizers > > and make sure the converted code is correct, or run through the > > interpreter, or run through the JIT to test the JIT, etc... > > > > That does not really surprise me about CMake, but then mingw is not a > > primary compiler on Windows, more so is VC++ or Intel, either way a > > bug should be submitted to the CMake devs. I thought you could > > override things like that with some command line parameters anyway. > > You cannot force it to output mingw files directly? > > _______________________________________________ > > LLVM Developers mailing list > > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > > > > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20081012/af7a1aa6/attachment.html>