I've run the tests from clang-tests/trunk/gcc-4_2-testsuite on a Ubuntu
x86-64 Linux box with the following results:
=== gcc Summary ==
# of expected passes 29946
# of unexpected failures 9938
# of unexpected successes 29
# of expected failures 28
# of unresolved testcases 1451
# of untested testcases 273
# of unsupported tests 811
pid is 4456 -4456
output is clang version 2.9 (trunk 113841)
Target: x86_64-unknown-linux-gnu
Thread model: posix
status 0
/home/john/llvm/Release+Asserts/bin/clang version 2.9 (trunk 113841)
This seems to be an improvement over the results I had previously when using
the i386 configuration. However, it sounded like you had more success on a
Darwin box, with only 1 unexpected failure (mine, which should be fixed
now). Should I expect that with this configuration?
I found that I had to exit most other running applications to be able to run
the tests without running out of memory.
I've enclosed my hacked makefile, hacked to change the configuration and add
the -v (verbose) flag.
I have a 6 mb log file from this run if needed.
~/gcc-4_2-testsuite$ find . -print | wc
36392 36392 1687901
Some output from the first few failed tests:
Testing gcc.apple/5490617.c
doing compile
pid is 17502 -17502
output is status 0
FAIL: gcc.apple/5490617.c scan-assembler .lcomm __my_endbss
Testing gcc.apple/4104248.c
doing compile
pid is 17445 -17445
close result is 17445 exp8 0 1
output is ilp3217400.c:2: error: array size is negative
status 1
Testing gcc.apple/4641942.c
doing compile
pid is 17481 -17481
output is status 0
FAIL: gcc.apple/4641942.c (test for warnings, line 20)
Testing gcc.apple/5597292.c
doing compile
pid is 17507 -17507
close result is 17507 exp8 0 1
output is /home/john/gcc-4_2-testsuite/src/gcc.apple/5597292.c:7: error:
fields must have a constant size: 'variable length array in structure'
extension will never be supported
status 1
compiler exited with status 1
FAIL: gcc.apple/5597292.c (test for excess errors)
Testing gcc.apple/6286881.c
doing compile
pid is 17572 -17572
close result is 17572 exp8 0 1
output is /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: warning: ^
has lower precedence than !=; != will be evaluated first [-Wparentheses]
/home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: note: place
parentheses around the ^ expression to evaluate it first
/home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: note: place
parentheses around the != expression to silence this warning
/home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: error: address
expression must be an lvalue or a function designator
status 1
compiler exited with status 1
FAIL: gcc.apple/6286881.c (test for errors, line 8)
etc.
-John
--
John Thompson
John.Thompson.JTSoftware at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20100922/223ffa63/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Makefile
Type: application/octet-stream
Size: 2872 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20100922/223ffa63/attachment.obj>
On Sep 22, 2010, at 11:32 AMPDT, John Thompson wrote:> 've run the tests from clang-tests/trunk/gcc-4_2-testsuite on a Ubuntu x86-64 Linux box with the following results: > > === gcc Summary ==> > # of expected passes 29946 > # of unexpected failures 9938 > # of unexpected successes 29 > # of expected failures 28 > # of unresolved testcases 1451 > # of untested testcases 273 > # of unsupported tests 811 > pid is 4456 -4456 > output is clang version 2.9 (trunk 113841)For reference, here are the results from the last time I ran it with llvm-gcc, about a month ago: # of expected passes 39621 # of unexpected failures 1112 # of unexpected successes 2 # of expected failures 59 # of unresolved testcases 80 # of untested testcases 280 # of unsupported tests 1073 Many of the 9000 extra failures are probably llvm-gcc/clang differences that are nothing to do with you, though. The best way to get useful info out of this is to run it with and without the patch you're testing, and compare. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20100922/5a34c5f2/attachment.html>
On Sep 22, 2010, at 1:04 PMPDT, Dale Johannesen wrote:> > Many of the 9000 extra failures are probably llvm-gcc/clang differences that are nothing to do with you, though. The best way to get useful info out of this is to run it with and without the patch you're testing, and compare.btw, the main log file that you're quoting from does not generally have enough info to reproduce failures. There is a more detailed log at <objdir>/gcc/testsuite/gcc/gcc.log . Beware, it is destroyed every time you rerun the test suite. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20100922/6e4b6565/attachment.html>
Hi John, On Wed, Sep 22, 2010 at 11:32 AM, John Thompson <john.thompson.jtsoftware at gmail.com> wrote:> I've run the tests from clang-tests/trunk/gcc-4_2-testsuite on a Ubuntu > x86-64 Linux box with the following results: > > === gcc Summary ==> > # of expected passes 29946 > # of unexpected failures 9938 > # of unexpected successes 29 > # of expected failures 28 > # of unresolved testcases 1451 > # of untested testcases 273 > # of unsupported tests 811 > pid is 4456 -4456 > output is clang version 2.9 (trunk 113841) > Target: x86_64-unknown-linux-gnu > Thread model: posix > status 0 > /home/john/llvm/Release+Asserts/bin/clang version 2.9 (trunk 113841) > > This seems to be an improvement over the results I had previously when using > the i386 configuration. However, it sounded like you had more success on a > Darwin box, with only 1 unexpected failure (mine, which should be fixed > now). Should I expect that with this configuration?No, that is unfortunately not the case. What I do for testing is keep arounds results (see the 'make save-results' utility target) and compare the diffs from run to run. This is a real pain, but the best tool we have at the moment. - Daniel> I found that I had to exit most other running applications to be able to run > the tests without running out of memory. > > I've enclosed my hacked makefile, hacked to change the configuration and add > the -v (verbose) flag. > > I have a 6 mb log file from this run if needed. > > ~/gcc-4_2-testsuite$ find . -print | wc > 36392 36392 1687901 > > Some output from the first few failed tests: > > Testing gcc.apple/5490617.c > doing compile > pid is 17502 -17502 > output is status 0 > FAIL: gcc.apple/5490617.c scan-assembler .lcomm __my_endbss > > Testing gcc.apple/4104248.c > doing compile > pid is 17445 -17445 > close result is 17445 exp8 0 1 > output is ilp3217400.c:2: error: array size is negative > status 1 > > Testing gcc.apple/4641942.c > doing compile > pid is 17481 -17481 > output is status 0 > FAIL: gcc.apple/4641942.c (test for warnings, line 20) > > Testing gcc.apple/5597292.c > doing compile > pid is 17507 -17507 > close result is 17507 exp8 0 1 > output is /home/john/gcc-4_2-testsuite/src/gcc.apple/5597292.c:7: error: > fields must have a constant size: 'variable length array in structure' > extension will never be supported > status 1 > compiler exited with status 1 > FAIL: gcc.apple/5597292.c (test for excess errors) > > Testing gcc.apple/6286881.c > doing compile > pid is 17572 -17572 > close result is 17572 exp8 0 1 > output is /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: warning: ^ > has lower precedence than !=; != will be evaluated first [-Wparentheses] > /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: note: place > parentheses around the ^ expression to evaluate it first > /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: note: place > parentheses around the != expression to silence this warning > /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: error: address > expression must be an lvalue or a function designator > status 1 > compiler exited with status 1 > FAIL: gcc.apple/6286881.c (test for errors, line 8) > > etc. > > -John > > -- > John Thompson > John.Thompson.JTSoftware at gmail.com > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > >
All, Here's a bit of old shell script I've used for comparing test results from various GCC testsuite runs. It's nothing fancy, but does cut down a bit of the noise and makes it a bit more convenient. Just point it at two directories containing the *.sum result files. YMMV and all that. -------------- next part -------------- A non-text attachment was scrubbed... Name: compare_test_results.sh Type: application/octet-stream Size: 1475 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20101004/f9a250ab/attachment.obj> -------------- next part -------------- -Jim On Oct 2, 2010, at 9:18 AM, Daniel Dunbar wrote:> Hi John, > > On Wed, Sep 22, 2010 at 11:32 AM, John Thompson > <john.thompson.jtsoftware at gmail.com> wrote: >> I've run the tests from clang-tests/trunk/gcc-4_2-testsuite on a Ubuntu >> x86-64 Linux box with the following results: >> >> === gcc Summary ==>> >> # of expected passes 29946 >> # of unexpected failures 9938 >> # of unexpected successes 29 >> # of expected failures 28 >> # of unresolved testcases 1451 >> # of untested testcases 273 >> # of unsupported tests 811 >> pid is 4456 -4456 >> output is clang version 2.9 (trunk 113841) >> Target: x86_64-unknown-linux-gnu >> Thread model: posix >> status 0 >> /home/john/llvm/Release+Asserts/bin/clang version 2.9 (trunk 113841) >> >> This seems to be an improvement over the results I had previously when using >> the i386 configuration. However, it sounded like you had more success on a >> Darwin box, with only 1 unexpected failure (mine, which should be fixed >> now). Should I expect that with this configuration? > > No, that is unfortunately not the case. > > What I do for testing is keep arounds results (see the 'make > save-results' utility target) and compare the diffs from run to run. > This is a real pain, but the best tool we have at the moment. > > - Daniel > >> I found that I had to exit most other running applications to be able to run >> the tests without running out of memory. >> >> I've enclosed my hacked makefile, hacked to change the configuration and add >> the -v (verbose) flag. >> >> I have a 6 mb log file from this run if needed. >> >> ~/gcc-4_2-testsuite$ find . -print | wc >> 36392 36392 1687901 >> >> Some output from the first few failed tests: >> >> Testing gcc.apple/5490617.c >> doing compile >> pid is 17502 -17502 >> output is status 0 >> FAIL: gcc.apple/5490617.c scan-assembler .lcomm __my_endbss >> >> Testing gcc.apple/4104248.c >> doing compile >> pid is 17445 -17445 >> close result is 17445 exp8 0 1 >> output is ilp3217400.c:2: error: array size is negative >> status 1 >> >> Testing gcc.apple/4641942.c >> doing compile >> pid is 17481 -17481 >> output is status 0 >> FAIL: gcc.apple/4641942.c (test for warnings, line 20) >> >> Testing gcc.apple/5597292.c >> doing compile >> pid is 17507 -17507 >> close result is 17507 exp8 0 1 >> output is /home/john/gcc-4_2-testsuite/src/gcc.apple/5597292.c:7: error: >> fields must have a constant size: 'variable length array in structure' >> extension will never be supported >> status 1 >> compiler exited with status 1 >> FAIL: gcc.apple/5597292.c (test for excess errors) >> >> Testing gcc.apple/6286881.c >> doing compile >> pid is 17572 -17572 >> close result is 17572 exp8 0 1 >> output is /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: warning: ^ >> has lower precedence than !=; != will be evaluated first [-Wparentheses] >> /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: note: place >> parentheses around the ^ expression to evaluate it first >> /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: note: place >> parentheses around the != expression to silence this warning >> /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: error: address >> expression must be an lvalue or a function designator >> status 1 >> compiler exited with status 1 >> FAIL: gcc.apple/6286881.c (test for errors, line 8) >> >> etc. >> >> -John >> >> -- >> John Thompson >> John.Thompson.JTSoftware at gmail.com >> >> >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >> >> > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev