Hi all, as you might have seen, I just did some restructuring on the testing documentation. Since the changes are only documentation and mostly text movement, I took the liberty of committing without posting a patch first. Was that appropriate?. The incentive for this commit was that I've spent the last week trying to get a grip on the test suite with moderate success. I think this restructuring will help new users to get started with the test suite faster. Or, more directly, I wanted to add some documentation on comparing test results, but couldn't find a decent spot in the testing documentation structure. There is still one issue I'd like to discuss here. The test suite documentation is a bit confusing in terminology. There are three import concepts: All of the testing stuff in llvm, the llvm/test DejaGNU part, and the test-suite part. Currently, the first is called the "LLVM test suite" (which is the subject of the testing documentation). The second is called the DejaGNU tests, code fragment tests or llvm/test. The last one is the most confusing, it's called the whole program tests, test-suite (named after the svn directory it is in, easily confused with the entire test suite), llvm-test (named after the directory in which you should place it, easily confused with llvm/test). I propose a few changes here. First, to use the "whole program tests", you should checkout the "test-suite" svn module into a "projects/llvm-test" directory. This would be way more logical if you could just checkout into a "projects/test-suite" directory. Furthermore, I would no longer refer to all of llvm's testing stuff as the "LLVM test suite", but use the term "LLVM test architecture" or "framework" or something similar (suggestions?). Since the testing stuff is composed of two distinct parts, I'm not sure the term "suite" is appropriate here. If people think that "test suite" is still very much appropriate here, I think that renaming the "test-suite" svn module would be in order. Lastly, I would refer to the "whole program tests" as the "Test suite", since it is a suite of testing programs that can be run as a whole. This will also make sure that the name properly matches the svn module that the "whole program tests" are housing in. After these changes, I think the testing framework should be a lot easier to describe and document and more accessible to new users. Any comments/questions/objections? Gr. Matthijs -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20080523/b9784129/attachment.sig>
On May 23, 2008, at 5:06 AM, Matthijs Kooijman wrote:> Hi all, > > as you might have seen, I just did some restructuring on the testing > documentation. Since the changes are only documentation and mostly > text > movement, I took the liberty of committing without posting a patch > first. Was > that appropriate?.Yep, if you consider it to be obvious, it can be committed without approval.> The incentive for this commit was that I've spent the last week > trying to get > a grip on the test suite with moderate success. I think this > restructuring > will help new users to get started with the test suite faster. Or, > more > directly, I wanted to add some documentation on comparing test > results, but > couldn't find a decent spot in the testing documentation structure.Awesome, we welcome improvements, obviously there are many to make here :)> There is still one issue I'd like to discuss here. The test suite > documentation is a bit confusing in terminology. There are three > import > concepts: All of the testing stuff in llvm, the llvm/test DejaGNU > part, and > the test-suite part.Ok.> Currently, the first is called the "LLVM test suite" (which is the > subject of > the testing documentation). The second is called the DejaGNU tests, > code > fragment tests or llvm/test. The last one is the most confusing, > it's called > the whole program tests, test-suite (named after the svn directory > it is in, > easily confused with the entire test suite), llvm-test (named after > the > directory in which you should place it, easily confused with llvm/ > test).llvm-test was the name of the CVS module. When we moved to SVN, it got renamed to test-suite, which I consider to be very confusing. We should probably recommend that people check it out into test-suite, and refer to it as that. Alternatively, change the svn module name to something more useful.> Furthermore, I would no longer refer to all of llvm's testing stuff > as the > "LLVM test suite", but use the term "LLVM test architecture" or > "framework" or > something similar (suggestions?).LLVM testing infrastructure?> Since the testing stuff is composed of two > distinct parts, I'm not sure the term "suite" is appropriate here. > If people > think that "test suite" is still very much appropriate here, I think > that > renaming the "test-suite" svn module would be in order.Right. It is somewhat complicated by the fact that "llvm-test" is really just a collection of programs with some makefile drivers. How about calling it "program-tests" or "llvm-program-tests" or something like that?> Lastly, I would refer to the "whole program tests" as the "Test > suite", since > it is a suite of testing programs that can be run as a whole. This > will also > make sure that the name properly matches the svn module that the > "whole > program tests" are housing in.Yeah, we definitely need a better name :)> After these changes, I think the testing framework should be a lot > easier to > describe and document and more accessible to new users.Sounds great to me, -Chris
> > >> Currently, the first is called the "LLVM test suite" (which is the >> subject of >> the testing documentation). The second is called the DejaGNU tests, >> code >> fragment tests or llvm/test. The last one is the most confusing, >> it's called >> the whole program tests, test-suite (named after the svn directory >> it is in, >> easily confused with the entire test suite), llvm-test (named after >> the >> directory in which you should place it, easily confused with llvm/ >> test). > > llvm-test was the name of the CVS module. When we moved to SVN, it > got renamed to test-suite, which I consider to be very confusing. We > should probably recommend that people check it out into test-suite, > and refer to it as that. Alternatively, change the svn module name to > something more useful. >I think at one time we did try to allow people to check it out as test-suite, but there is something in the Makefiles that require it to be llvm-test (I don't know exactly what though). So this is something that needs to be fixed. It would be great to get it fixed. -Tanya>> Furthermore, I would no longer refer to all of llvm's testing stuff >> as the >> "LLVM test suite", but use the term "LLVM test architecture" or >> "framework" or >> something similar (suggestions?). > > LLVM testing infrastructure? > >> Since the testing stuff is composed of two >> distinct parts, I'm not sure the term "suite" is appropriate here. >> If people >> think that "test suite" is still very much appropriate here, I think >> that >> renaming the "test-suite" svn module would be in order. > > Right. It is somewhat complicated by the fact that "llvm-test" is > really just a collection of programs with some makefile drivers. How > about calling it "program-tests" or "llvm-program-tests" or something > like that? > >> Lastly, I would refer to the "whole program tests" as the "Test >> suite", since >> it is a suite of testing programs that can be run as a whole. This >> will also >> make sure that the name properly matches the svn module that the >> "whole >> program tests" are housing in. > > Yeah, we definitely need a better name :) > >> After these changes, I think the testing framework should be a lot >> easier to >> describe and document and more accessible to new users. > > Sounds great to me, > > -Chris > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20080523/ec693d20/attachment.html>
Hi All, I've finally implemented my proposed changes to the testing documentation. I think I fixed up the naming everywhere, but I did only a single pass over the document, so I might have missed something. Review welcome :-) It is now recommended to put the test suite in "projects/test-suite" instead of "projects/llvm-test". I've also updated the configure script and projects/Makefile to reflect this. I haven't run autoconf, since I don't have the right version, so someone should probably do that to check my manual changes to configure. Also, it would perhaps be useful to add a deprecation warning when using llvm-test, but my autoconf magic is insufficient (plus I couldn't really test that due to no autoconf-2.6). Gr. Matthijs -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20080624/26ee13ff/attachment.sig>
Great! To really finish this work, the nightly tester script needs to be updated. Its currently full of llvm-test references. Would you be willing to modify this as well? Thanks, Tanya On Tue, 24 Jun 2008, Matthijs Kooijman wrote:> Hi All, > > I've finally implemented my proposed changes to the testing documentation. I > think I fixed up the naming everywhere, but I did only a single pass over the > document, so I might have missed something. Review welcome :-) > > It is now recommended to put the test suite in "projects/test-suite" instead > of "projects/llvm-test". I've also updated the configure script and > projects/Makefile to reflect this. I haven't run autoconf, since I don't have > the right version, so someone should probably do that to check my manual > changes to configure. Also, it would perhaps be useful to add a deprecation > warning when using llvm-test, but my autoconf magic is insufficient (plus I > couldn't really test that due to no autoconf-2.6). > > Gr. > > Matthijs >