First, let me ask if there's a page that documents the testing process in full. If there is, just give me a link to that and save yourself the time of answering this. (I did Google it, all I found was docs on writing tests for LLVM.) Sorry if this is way off, but I have no idea what to download and unpack to start testing. I know it's stuff from here: http://llvm.org/pre-releases/3.3/rc1/. I've built LLVM and Clang before, for my own use. IIRC, I put clang in llvm/tools/ and just ran ./configure and make. So what I need to know is: which of those files do I need, and which llvm subdirectories the go in? And I do this for both 3.3rc1 and 3.2, then run: utils/release/test-release.sh --release 3.2 in the llvm 3.2 directory, and utils/release/test-release.sh --release 3.3 --rc 1 in the llvm 3.3rc1 directory, right? For clarification, by llvm 3.x directory, I mean the extracted llvm 3.x tarball and all the 3.x sub-tools extracted in their respective subdirectories. Then what after that? Sorry for all the extra clarification. This is the first testing I've ever done. I really like LLVM and accompanying tools. I'd like to become more involved in the LLVM project, but I don't think I have the necessary skills yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130509/b06982a6/attachment.html>
Hi Tyler, On 05/09/2013 07:03 AM, Tyler Hardin wrote:> First, let me ask if there's a page that documents the testing process in > full. If there is, just give me a link to that and save yourself the time > of answering this. (I did Google it, all I found was docs on writing tests > for LLVM.) > > Sorry if this is way off, but I have no idea what to download and unpack to > start testing. I know it's stuff from here: > http://llvm.org/pre-releases/3.3/rc1/. > > I've built LLVM and Clang before, for my own use. IIRC, I put clang in > llvm/tools/ and just ran ./configure and make. So what I need to know is: > which of those files do I need, and which llvm subdirectories the go in? > > And I do this for both 3.3rc1 and 3.2, then run: > utils/release/test-release.sh --release 3.2 in the llvm 3.2 directory, and > utils/release/test-release.sh --release 3.3 --rc 1 in the llvm 3.3rc1 > directory, right? > > For clarification, by llvm 3.x directory, I mean the extracted llvm 3.x > tarball and all the 3.x sub-tools extracted in their respective > subdirectories. > > Then what after that? > > Sorry for all the extra clarification. This is the first testing I've ever > done. I really like LLVM and accompanying tools. I'd like to become more > involved in the LLVM project, but I don't think I have the necessary skills > yet. >Disclaimer: This is also my very first testing session, but I think that I've figured it out ... at least the procedure makes sense to me ;) The full procedure for getting the nightly test results of *a single RC* is as follows: 1. Get the sources for LLVM + Clang from SVN (i.e. checkout from the specific tags, e.g. RELEASE_33/rc1) 2. Run the script from utils/release/test-release.sh with appropriate arguments, e.g. to build 3.3 RC1: test-release.sh -release 3.3 -rc 1 Adapt -build-dir and -j as needed 3. Once 2 is finished, download the llvm-test-suite, install LNT (if not done already) and run it, don't forget to set --cc and --cxx to the newly compiled clang/clang++ binaries. When this step is finished, you'll get a directory containing a log file (report.json) Now for the important: you have to do this for 3.2final *and* 3.3rc1m so simply repeat steps 1 to 3. Once you have the result.json files for both releases, you can compare them with the mentioned findRegressions-{nightly,simple}.py scripts. I hope that clarifies some things. Cheers, Sebastian -- Mit freundlichen Grüßen / Kind regards Sebastian Dreßler Zuse Institute Berlin (ZIB) Takustraße 7 D-14195 Berlin-Dahlem Germany dressler at zib.de Phone: +49 30 84185-261 http://www.zib.de/
Sorry, I forgot to mention, that, if you download the sources yourself, you should append "-no-checkout" to test-release.sh. Cheers, Sebastian On 05/09/2013 08:39 AM, Sebastian Dreßler wrote:> Hi Tyler, > > On 05/09/2013 07:03 AM, Tyler Hardin wrote: >> First, let me ask if there's a page that documents the testing process in >> full. If there is, just give me a link to that and save yourself the time >> of answering this. (I did Google it, all I found was docs on writing tests >> for LLVM.) >> >> Sorry if this is way off, but I have no idea what to download and unpack to >> start testing. I know it's stuff from here: >> http://llvm.org/pre-releases/3.3/rc1/. >> >> I've built LLVM and Clang before, for my own use. IIRC, I put clang in >> llvm/tools/ and just ran ./configure and make. So what I need to know is: >> which of those files do I need, and which llvm subdirectories the go in? >> >> And I do this for both 3.3rc1 and 3.2, then run: >> utils/release/test-release.sh --release 3.2 in the llvm 3.2 directory, and >> utils/release/test-release.sh --release 3.3 --rc 1 in the llvm 3.3rc1 >> directory, right? >> >> For clarification, by llvm 3.x directory, I mean the extracted llvm 3.x >> tarball and all the 3.x sub-tools extracted in their respective >> subdirectories. >> >> Then what after that? >> >> Sorry for all the extra clarification. This is the first testing I've ever >> done. I really like LLVM and accompanying tools. I'd like to become more >> involved in the LLVM project, but I don't think I have the necessary skills >> yet. >> > > Disclaimer: This is also my very first testing session, but I think that > I've figured it out ... at least the procedure makes sense to me ;) > > > The full procedure for getting the nightly test results of *a single RC* > is as follows: > > 1. Get the sources for LLVM + Clang from SVN (i.e. checkout from the > specific tags, e.g. RELEASE_33/rc1) > > 2. Run the script from utils/release/test-release.sh with appropriate > arguments, e.g. to build 3.3 RC1: > > test-release.sh -release 3.3 -rc 1 > > Adapt -build-dir and -j as needed > > 3. Once 2 is finished, download the llvm-test-suite, install LNT (if not > done already) and run it, don't forget to set --cc and --cxx to the > newly compiled clang/clang++ binaries. When this step is finished, > you'll get a directory containing a log file (report.json) > > > Now for the important: you have to do this for 3.2final *and* 3.3rc1m so > simply repeat steps 1 to 3. Once you have the result.json files for both > releases, you can compare them with the mentioned > findRegressions-{nightly,simple}.py scripts. > > > I hope that clarifies some things. > > Cheers, > Sebastian >-- Mit freundlichen Grüßen / Kind regards Sebastian Dreßler Zuse Institute Berlin (ZIB) Takustraße 7 D-14195 Berlin-Dahlem Germany dressler at zib.de Phone: +49 30 84185-261 http://www.zib.de/
Hi, On 05/09/2013 08:39 AM, Sebastian Dreßler wrote:> Now for the important: you have to do this for 3.2final *and* 3.3rc1m so > simply repeat steps 1 to 3. Once you have the result.json files for both > releases, you can compare them with the mentioned > findRegressions-{nightly,simple}.py scripts.sorry, I'll have to correct this. It's not the result.json the script needs. I think it's "test.log" but this gives me unexpected output: findRegressions-nightly.py writes "ERROR!" in particular. Maybe someone could confirm that it is "test.log" or correct me? Cheers, Sebastian -- Mit freundlichen Grüßen / Kind regards Sebastian Dreßler Zuse Institute Berlin (ZIB) Takustraße 7 D-14195 Berlin-Dahlem Germany dressler at zib.de Phone: +49 30 84185-261 http://www.zib.de/