I've finished adding the -rundejagnu option to the nightly tester script, which was the last step to fully support Dejagnu. I think now is the appropriate time to discuss keeping QMTest or switching to Dejagnu. A lot of work went into using QMTest, so I think we should make this decision carefully and before the 1.4 release. Here are the pros and cons in my eyes, please feel free to add your own opinion. (Note: RUN lines are basically sequences of commands that are put into a bash script and executed. Here is an example: RUN: llvm-as < %s | llvm-dis | llvm-as) Pros of Dejagnu: 1) Adding a test is very quick and easy. In most cases it requires just adding a RUN line to the file. No gui needed. 2) XFAILing a test is also quick and easy. You just add a XFAIL line to a single test with a * for all platforms, or specify part of a target triplet to match. No gui needed. 3) Because you add the RUN and XFAIL lines to each individual test, it's all located in one file. This makes it very easy understand how the test is run and easy to change. 4) You can run just a single sub directory of tests by specifying that directory as the TESTSUITE. (ex: make check-dejagnu TESTSUITE=Regression/CFrontend) 5) All intermediate output is put into an Output directory and easily cleaned up w/ proper Makefile support. 6) It is faster then QMTest (54% faster on the sparc, 20% faster on ppc) 7) The output logs are easy to parse. 8) You can easily search for XFAIL tests. Cons of Dejagnu: 1) There must be a dg.exp file in each subdirectory to run tests in that directory. (easily copied though) Pros of QMTest: 1) Can add names to subdirectories to create testsuites (kinda like a shortcut). 2) Has a gui (some prefer this). Cons of QMTest: 1) You have to use the gui to add directories. 2) You have to use the gui to XFAIL a test. 3) It uses something called expectation files that you must load to view which tests XFAIL. There is no way (that I have found) to get a complete list of XFAILs.. 4) It is also hard to XFAIL across platforms, because it requires hacking an expectation file for each target, which must be done with the gui. 5) Intermediate output placement can not be controlled. 6) The output logs are not as clean. 7) Right now we are dependent on a specific version of QMTest. 8) It has several bugs that can cause it to forkbomb when tools are not built correctly. I propose that we switch over to using dejagnu by default, renaming check-dejagnu to check, and deprecate QMTest. We can either remove qmtest for this release or keep it until 1.5. I'd appreciate your opinion or any feedback you may have. Thanks! -Tanya
Seems pretty clear to me that dejagnu is a better choice. There's one more QMTest issue: we only work with 2.03 as our database doesn't read properly with newer versions. Not having a database seems like a good way to go and having all the necessary things being easily editable in text files will help keep the tests up to date and make XFAILing much easier. I vote for dumping QMTest in 1.4 or at least deprecating it in 1.4 and removing it in 1.5. In any event the "check" target in the test directory should now default to dejagnu. Reid. On Sat, 2004-11-27 at 13:24, Tanya Lattner wrote:> I've finished adding the -rundejagnu option to the nightly tester script, > which was the last step to fully support Dejagnu. I think now is the > appropriate time to discuss keeping QMTest or switching to Dejagnu. A lot > of work went into using QMTest, so I think we should make this decision > carefully and before the 1.4 release. > > Here are the pros and cons in my eyes, please feel free to add your own > opinion. > > (Note: RUN lines are basically sequences of commands that are put into a > bash script and executed. Here is an example: RUN: llvm-as < %s | llvm-dis > | llvm-as) > > Pros of Dejagnu: > 1) Adding a test is very quick and easy. In most cases it requires > just adding a RUN line to the file. No gui needed. > 2) XFAILing a test is also quick and easy. You just add a XFAIL > line to a single test with a * for all platforms, or specify part > of a target triplet to match. No gui needed. > 3) Because you add the RUN and XFAIL lines to each individual > test, it's all located in one file. This makes it very easy > understand how the test is run and easy to change. > 4) You can run just a single sub directory of tests by specifying > that directory as the TESTSUITE. (ex: make check-dejagnu > TESTSUITE=Regression/CFrontend) > 5) All intermediate output is put into an Output directory and > easily cleaned up w/ proper Makefile support. > 6) It is faster then QMTest (54% faster on the sparc, 20% faster > on ppc) > 7) The output logs are easy to parse. > 8) You can easily search for XFAIL tests. > > Cons of Dejagnu: > 1) There must be a dg.exp file in each subdirectory to run tests > in that directory. (easily copied though) > > Pros of QMTest: > 1) Can add names to subdirectories to create testsuites (kinda > like a shortcut). > 2) Has a gui (some prefer this). > > > Cons of QMTest: > 1) You have to use the gui to add directories. > 2) You have to use the gui to XFAIL a test. > 3) It uses something called expectation files that you must load > to view which tests XFAIL. There is no way (that I have found) to > get a complete list of XFAILs.. > 4) It is also hard to XFAIL across platforms, because it requires > hacking an expectation file for each > target, which must be done with the gui. > 5) Intermediate output placement can not be controlled. > 6) The output logs are not as clean. > 7) Right now we are dependent on a specific version of QMTest. > 8) It has several bugs that can cause it to forkbomb when tools > are not built correctly. > > I propose that we switch over to using dejagnu by default, renaming > check-dejagnu to check, and deprecate QMTest. We can either remove qmtest > for this release or keep it until 1.5. > > I'd appreciate your opinion or any feedback you may have. > > Thanks! > -Tanya > > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://mail.cs.uiuc.edu/mailman/listinfo/llvmdev-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20041127/6719158f/attachment.sig>
dejagnu sounds like the better choice to me too. I recommend keeping qmtest for this release and then removing it for 1.5. --Vikram On Nov 27, 2004, at 3:24 PM, Tanya Lattner wrote:> > I've finished adding the -rundejagnu option to the nightly tester > script, > which was the last step to fully support Dejagnu. I think now is the > appropriate time to discuss keeping QMTest or switching to Dejagnu. A > lot > of work went into using QMTest, so I think we should make this decision > carefully and before the 1.4 release. > > Here are the pros and cons in my eyes, please feel free to add your own > opinion. > > (Note: RUN lines are basically sequences of commands that are put into > a > bash script and executed. Here is an example: RUN: llvm-as < %s | > llvm-dis > | llvm-as) > > Pros of Dejagnu: > 1) Adding a test is very quick and easy. In most cases it requires > just adding a RUN line to the file. No gui needed. > 2) XFAILing a test is also quick and easy. You just add a XFAIL > line to a single test with a * for all platforms, or specify part > of a target triplet to match. No gui needed. > 3) Because you add the RUN and XFAIL lines to each individual > test, it's all located in one file. This makes it very easy > understand how the test is run and easy to change. > 4) You can run just a single sub directory of tests by specifying > that directory as the TESTSUITE. (ex: make check-dejagnu > TESTSUITE=Regression/CFrontend) > 5) All intermediate output is put into an Output directory and > easily cleaned up w/ proper Makefile support. > 6) It is faster then QMTest (54% faster on the sparc, 20% faster > on ppc) > 7) The output logs are easy to parse. > 8) You can easily search for XFAIL tests. > > Cons of Dejagnu: > 1) There must be a dg.exp file in each subdirectory to run tests > in that directory. (easily copied though) > > Pros of QMTest: > 1) Can add names to subdirectories to create testsuites (kinda > like a shortcut). > 2) Has a gui (some prefer this). > > > Cons of QMTest: > 1) You have to use the gui to add directories. > 2) You have to use the gui to XFAIL a test. > 3) It uses something called expectation files that you must load > to view which tests XFAIL. There is no way (that I have found) to > get a complete list of XFAILs.. > 4) It is also hard to XFAIL across platforms, because it requires > hacking an expectation file for each > target, which must be done with the gui. > 5) Intermediate output placement can not be controlled. > 6) The output logs are not as clean. > 7) Right now we are dependent on a specific version of QMTest. > 8) It has several bugs that can cause it to forkbomb when tools > are not built correctly. > > I propose that we switch over to using dejagnu by default, renaming > check-dejagnu to check, and deprecate QMTest. We can either remove > qmtest > for this release or keep it until 1.5. > > I'd appreciate your opinion or any feedback you may have. > > Thanks! > -Tanya > > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://mail.cs.uiuc.edu/mailman/listinfo/llvmdev >--Vikram http://www.cs.uiuc.edu/~vadve http://llvm.cs.uiuc.edu/
On Sunday 28 November 2004 00:24, Tanya Lattner wrote: Just some comments from a QMTest user... Note however, that even with them, dejagnu looks better.> Cons of QMTest: > 1) You have to use the gui to add directories.I think you're wrong. Manually creating a directory would work, as QMTest does not place anything special in directories.> 2) You have to use the gui to XFAIL a test.Right.> 3) It uses something called expectation files that you must load > to view which tests XFAIL. There is no way (that I have found) to > get a complete list of XFAILs..Right. This appears to have deep roots in QMTest design, so I don't know how to make using one file with "RUN" and "XFAIL" lines work -- the allowed format of expectations file is hardcoded.> 4) It is also hard to XFAIL across platforms, because it requires > hacking an expectation file for each > target, which must be done with the gui.Right and very deplorable. Basically, QMTest is nice tool but it's not clear how to use it for for cross-platform work like LLVM (or C++ Boost).> 5) Intermediate output placement can not be controlled.I don't understand this one. The intermediate output is placed where tests want it to be placed; QMTest does not create any such output itself.> 6) The output logs are not as clean.I think the current version supports dejagnu-style logs.> 7) Right now we are dependent on a specific version of QMTest.Yea, that's a big problem.> I propose that we switch over to using dejagnu by default, renaming > check-dejagnu to check, and deprecate QMTest. We can either remove qmtest > for this release or keep it until 1.5. > > I'd appreciate your opinion or any feedback you may have.Looks like dejagnu is better for this task (though I always thought it's some rusty tcl-based code). You might want to check on QMTest mailing list first, in case they have some silver bullets at hand. After all QMTest was supposed to test gcc, which is also cross-platform. - Volodya
Tanya Lattner wrote:> I've finished adding the -rundejagnu option to the nightly tester script, > which was the last step to fully support Dejagnu. I think now is the > appropriate time to discuss keeping QMTest or switching to Dejagnu. A lot > of work went into using QMTest, so I think we should make this decision > carefully and before the 1.4 release. > > Here are the pros and cons in my eyes, please feel free to add your own > opinion. > > (Note: RUN lines are basically sequences of commands that are put into a > bash script and executed. Here is an example: RUN: llvm-as < %s | llvm-dis > | llvm-as) > > Pros of Dejagnu: > 1) Adding a test is very quick and easy. In most cases it requires > just adding a RUN line to the file. No gui needed. > 2) XFAILing a test is also quick and easy. You just add a XFAIL > line to a single test with a * for all platforms, or specify part > of a target triplet to match. No gui needed. > 3) Because you add the RUN and XFAIL lines to each individual > test, it's all located in one file. This makes it very easy > understand how the test is run and easy to change. > 4) You can run just a single sub directory of tests by specifying > that directory as the TESTSUITE. (ex: make check-dejagnu > TESTSUITE=Regression/CFrontend) > 5) All intermediate output is put into an Output directory and > easily cleaned up w/ proper Makefile support. > 6) It is faster then QMTest (54% faster on the sparc, 20% faster > on ppc) > 7) The output logs are easy to parse. > 8) You can easily search for XFAIL tests. > > Cons of Dejagnu: > 1) There must be a dg.exp file in each subdirectory to run tests > in that directory. (easily copied though)Can these be located in the source tree and found when running the tests in the object tree? If so, that would save us from having to copy these over into the object tree via the configure script, which would be very nice. What other packages does DejaGnu require, and are most people likely to have those packages installed?> > Pros of QMTest: > 1) Can add names to subdirectories to create testsuites (kinda > like a shortcut). > 2) Has a gui (some prefer this).I agree that having a command line tool or being able to edit information inside the test file would be ideal. The GUI is painful.> > > Cons of QMTest: > 1) You have to use the gui to add directories. > 2) You have to use the gui to XFAIL a test. > 3) It uses something called expectation files that you must load > to view which tests XFAIL. There is no way (that I have found) to > get a complete list of XFAILs.. > 4) It is also hard to XFAIL across platforms, because it requires > hacking an expectation file for each > target, which must be done with the gui.QMTest could use the same expectations file for all platforms; I eventually chose not to do that because I need some tests to pass on some platforms and fail on others. The problem, I think, is being able to XFAIL across platforms and still be easily able to XFAIL/XPASS tests for specific platforms (i.e. test1 fails on all platforms, but test2 fails on Sparc only).> 5) Intermediate output placement can not be controlled.Actually, this is not a QMTest issue as much as it is the test code that I wrote. The testing code I wrote (qmtest.py) puts all temporary files into a temporary directory in the object tree. I think I did this because QMTest runs all tests from the $OBJ_ROOT/test directory instead of recursively cd'ing into all of the test directories like the old Makefile system did.> 6) The output logs are not as clean.FWIW, QMTest can generate an HTML output file.> 7) Right now we are dependent on a specific version of QMTest.This is due to the fact that QMTest changed their coding interfaces in newer versions,so our QMTest python code no longer works.> 8) It has several bugs that can cause it to forkbomb when tools > are not built correctly.I don't know if this is a coding error on my part or a bug in whatever Python library I'm using to execute the program. In any event, the current implementation should be fixable; it just hasn't gotten high priority on my list yet.> > I propose that we switch over to using dejagnu by default, renaming > check-dejagnu to check, and deprecate QMTest. We can either remove qmtest > for this release or keep it until 1.5.Despite my feedback on QMTest, I am getting the impression that Dejagnu is the easier tool to use (although I don't know for sure as I've not used it). Using QMTest is only good if we upgrade to a newer version, and even then, it may not get us the features we want (like not using the GUI, easy XFAIL'ing, etc). Since Dejagnu is already in the tree, I don't see much point in upgrading to a newer version of QMTest. The only real downside to Dejagnu might be external software dependencies. Regarding the upgrade path, it would be nice to have only one test system in place for LLVM 1.4. However, if there isn't enough time, I think we should just stick with QMTest and then remove it for LLVM 1.5.> > I'd appreciate your opinion or any feedback you may have. > > Thanks! > -Tanya > > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://mail.cs.uiuc.edu/mailman/listinfo/llvmdev-- John T. -- ********************************************************************* * John T. Criswell Email: criswell at uiuc.edu * * Research Programmer * * University of Illinois at Urbana-Champaign * * * * "It's today!" said Piglet. "My favorite day," said Pooh. * *********************************************************************
> > Cons of QMTest: > > 1) You have to use the gui to add directories. > > I think you're wrong. Manually creating a directory would work, as QMTest does > not place anything special in directories.I may have worded this poorly, but I think you have to use the gui to add new directories or tests, or specific tests. Otherwise, it does not know what to do with those files. I get lots of warnings/errors from QMTest when it comes across a tests thats not in the database (says something like "could not load test".> > 5) Intermediate output placement can not be controlled. > > I don't understand this one. The intermediate output is placed where tests > want it to be placed; QMTest does not create any such output itself.This could be how we have written our python files.. but right now if you run a tool and it generates some output.. where that output is placed is not controlled. So it makes it difficult to clean up this intermediate output.> > I propose that we switch over to using dejagnu by default, renaming > > check-dejagnu to check, and deprecate QMTest. We can either remove qmtest > > for this release or keep it until 1.5. > > > > I'd appreciate your opinion or any feedback you may have. > > Looks like dejagnu is better for this task (though I always thought it's some > rusty tcl-based code). You might want to check on QMTest mailing list first, > in case they have some silver bullets at hand. After all QMTest was supposed > to test gcc, which is also cross-platform.Thanks for your feedback!!! -Tanya
> > Cons of Dejagnu: > > 1) There must be a dg.exp file in each subdirectory to run tests > > in that directory. (easily copied though) > > Can these be located in the source tree and found when running the tests > in the object tree? If so, that would save us from having to copy these > over into the object tree via the configure script, which would be very > nice.They do not have to be copied over. I run objdir != srcdir and dejagnu works fine. The trickey part is that you can not specify a particular testsuite if you have not run the complete suite of tests (only applies to when objdir != srcdir). This is because dejagnu wants to place the sum file in that directory which won't exist until you run the complete suite. There is a bugzilla bug on this, but its really not that big of a deal to run the Regression/Features tests once (in my opinion).> What other packages does DejaGnu require, and are most people likely to > have those packages installed?tcl (which most people will have installed) and expect are the only requirements to install Dejagnu.> > Cons of QMTest: > > 1) You have to use the gui to add directories. > > 2) You have to use the gui to XFAIL a test. > > 3) It uses something called expectation files that you must load > > to view which tests XFAIL. There is no way (that I have found) to > > get a complete list of XFAILs.. > > 4) It is also hard to XFAIL across platforms, because it requires > > hacking an expectation file for each > > target, which must be done with the gui. > > QMTest could use the same expectations file for all platforms; I > eventually chose not to do that because I need some tests to pass on > some platforms and fail on others. > > The problem, I think, is being able to XFAIL across platforms and still > be easily able to XFAIL/XPASS tests for specific platforms (i.e. test1 > fails on all platforms, but test2 fails on Sparc only). > > > 5) Intermediate output placement can not be controlled. > > Actually, this is not a QMTest issue as much as it is the test code that > I wrote. The testing code I wrote (qmtest.py) puts all temporary files > into a temporary directory in the object tree. I think I did this > because QMTest runs all tests from the $OBJ_ROOT/test directory instead > of recursively cd'ing into all of the test directories like the old > Makefile system did.There are some tests that generate some intermediate output that are not getting placed in the temp directory. I'm not sure why that is.. but it must be something that just wasn't noticed til now.> > > 6) The output logs are not as clean. > > FWIW, QMTest can generate an HTML output file.That is a good point.> > I propose that we switch over to using dejagnu by default, renaming > > check-dejagnu to check, and deprecate QMTest. We can either remove qmtest > > for this release or keep it until 1.5. > > Despite my feedback on QMTest, I am getting the impression that Dejagnu > is the easier tool to use (although I don't know for sure as I've not > used it). Using QMTest is only good if we upgrade to a newer version, > and even then, it may not get us the features we want (like not using > the GUI, easy XFAIL'ing, etc). Since Dejagnu is already in the tree, I > don't see much point in upgrading to a newer version of QMTest. > > The only real downside to Dejagnu might be external software dependencies.I actually think that more people will have tcl installed then python.. but thats just my personal opinion.> Regarding the upgrade path, it would be nice to have only one test > system in place for LLVM 1.4. However, if there isn't enough time, I > think we should just stick with QMTest and then remove it for LLVM 1.5.The only downside I see to keeping it in for 1.4 is that you (since you are the one that updates QMTest) will have to update the expectation files for the tests (there have been several new ones added). But otherwise, its easy to keep it in the tree. -Tanya