Hi All,
I am working on a bug in clang. I already have a fix for it and I am going
through the "LLVM Testing Infrastructure Guide" to make sure I
haven't
broken anything else. I have few questions regarding the test suite
infrastructure that I hope someone can answer.
1. I checked out and built llvm, clang, and test-suite from svn tip. When I
run llvm/test *without* my changes on x86, I see 3 failures. Is it expected?
I was expecting svn tip on x86 to be clean. Is there a nightly regression
result available somewhere that I can check for broken tests. Here is the
report from my log:
Failing Tests (3):
Clang :: Analysis/iterators.cpp
Clang :: Driver/hello.c
Clang :: Preprocessor/macro_paste_c_block_comment.c
Expected Passes : 8757
Expected Failures : 69
Unsupported Tests : 551
Unexpected Failures: 3
2. I see clang has its own test suite under clang/test. I assume when I run
llvm/test, the test infrastructure also runs clang/test.
3. For running projects/test-suite, the instruction guide directs to build
llvm-gcc. Why is llvm-gcc required? Why is clang not being used?
4. I see many compile errors when running projects/test-suite. Is it
expected?
5. I used "make -i" to ignore compile erros and execute all tests
under
projects/test-suite. I finally get a report something like this:
Program |
GCCAS Bytecode LLC compile LLC-BETA compile JIT codegen | GCC CBE
LLC LLC-BETA JIT | GCC/CBE GCC/LLC
GMultiSource/Applications/Burg/burg |
5.5484 115468 2.9722 * 3.0922 | 0.0100 *
0.0000 * 3.2400 | n/a - n
MultiSource/Applications/ClamAV/clamscan |
41.8306 1335184 34.4382 * 18.6692 | 0.1200 *
0.1300 * 19.5200 | n/a 0.92 n
I was also expecting to see a list of passing/failing test cases similar to
llvm/test runs. Is there a make target to do so?
I apologies for asking so many questions. I followed directions from the
"LLVM Testing Infrastructure Guide" but it doesn't address all
these issues.
I will appreciate your help to resolve these.
Thanks,
Sundeep
On Jul 27, 2011, at 3:00 PM, Sundeep wrote:> Hi All, > > I am working on a bug in clang. I already have a fix for it and I am going > through the "LLVM Testing Infrastructure Guide" to make sure I haven't > broken anything else. I have few questions regarding the test suite > infrastructure that I hope someone can answer. > > 1. I checked out and built llvm, clang, and test-suite from svn tip. When I > run llvm/test *without* my changes on x86, I see 3 failures. Is it expected? > I was expecting svn tip on x86 to be clean. Is there a nightly regression > result available somewhere that I can check for broken tests. Here is the > report from my log: > > Failing Tests (3): > Clang :: Analysis/iterators.cpp > Clang :: Driver/hello.c > Clang :: Preprocessor/macro_paste_c_block_comment.c > > Expected Passes : 8757 > Expected Failures : 69 > Unsupported Tests : 551 > Unexpected Failures: 3 >Because you're living life on the bleeding edge (i.e., pulling code from the svn repository) it's possible for this to happen from time to time. A fix is probably already in the works. One way to ensure you don't introduce a new error is to maintain a clean (i.e., unmodified version of llvm/clang), which you can use as a baseline. Run regressions on both the clean and modified versions to see if you cause any additional failures.> 2. I see clang has its own test suite under clang/test. I assume when I run > llvm/test, the test infrastructure also runs clang/test.How are you invoking the tests? I typically do 'make check-all' from my build directory, which runs both regression suites.> 3. For running projects/test-suite, the instruction guide directs to build > llvm-gcc. Why is llvm-gcc required? Why is clang not being used?Very possible the documentation needs updating.> 4. I see many compile errors when running projects/test-suite. Is it > expected?No.> 5. I used "make -i" to ignore compile erros and execute all tests under > projects/test-suite. I finally get a report something like this: > > Program | > GCCAS Bytecode LLC compile LLC-BETA compile JIT codegen | GCC CBE > LLC LLC-BETA JIT | GCC/CBE GCC/LLC > GMultiSource/Applications/Burg/burg | > 5.5484 115468 2.9722 * 3.0922 | 0.0100 * > 0.0000 * 3.2400 | n/a - n > MultiSource/Applications/ClamAV/clamscan | > 41.8306 1335184 34.4382 * 18.6692 | 0.1200 * > 0.1300 * 19.5200 | n/a 0.92 n > > I was also expecting to see a list of passing/failing test cases similar to > llvm/test runs. Is there a make target to do so? >You might try 'make -i report' Chad> I apologies for asking so many questions. I followed directions from the > "LLVM Testing Infrastructure Guide" but it doesn't address all these issues. > I will appreciate your help to resolve these. > > Thanks, > Sundeep > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > 1. I checked out and built llvm, clang, and test-suite from svn tip. > When I > > run llvm/test *without* my changes on x86, I see 3 failures. Is it > expected? > > I was expecting svn tip on x86 to be clean. Is there a nightly > regression > > result available somewhere that I can check for broken tests. Here is > the > > report from my log: > > > > Failing Tests (3): > > Clang :: Analysis/iterators.cpp > > Clang :: Driver/hello.c > > Clang :: Preprocessor/macro_paste_c_block_comment.c > > > > Expected Passes : 8757 > > Expected Failures : 69 > > Unsupported Tests : 551 > > Unexpected Failures: 3 > > > > Because you're living life on the bleeding edge (i.e., pulling code > from the svn repository) it's possible for this to happen from time to > time. A fix is probably already in the works. One way to ensure you > don't introduce a new error is to maintain a clean (i.e., unmodified > version of llvm/clang), which you can use as a baseline. Run > regressions on both the clean and modified versions to see if you cause > any additional failures.Makes sense.> > 2. I see clang has its own test suite under clang/test. I assume when > I run > > llvm/test, the test infrastructure also runs clang/test. > > How are you invoking the tests? I typically do 'make check-all' from > my build directory, which runs both regression suites.Yes, I am doing "make check-all".> > 3. For running projects/test-suite, the instruction guide directs to > build > > llvm-gcc. Why is llvm-gcc required? Why is clang not being used? > > Very possible the documentation needs updating.I see. Do you run projects/test-suite with clang?> > 4. I see many compile errors when running projects/test-suite. Is it > > expected? > > No.I will try to debug. Do you have a log file from your previous runs that I can use? Thanks Chad. I appreciate it. -Sundeep