> > 1. I checked out and built llvm, clang, and test-suite from svn tip. > When I > > run llvm/test *without* my changes on x86, I see 3 failures. Is it > expected? > > I was expecting svn tip on x86 to be clean. Is there a nightly > regression > > result available somewhere that I can check for broken tests. Here is > the > > report from my log: > > > > Failing Tests (3): > > Clang :: Analysis/iterators.cpp > > Clang :: Driver/hello.c > > Clang :: Preprocessor/macro_paste_c_block_comment.c > > > > Expected Passes : 8757 > > Expected Failures : 69 > > Unsupported Tests : 551 > > Unexpected Failures: 3 > > > > Because you're living life on the bleeding edge (i.e., pulling code > from the svn repository) it's possible for this to happen from time to > time. A fix is probably already in the works. One way to ensure you > don't introduce a new error is to maintain a clean (i.e., unmodified > version of llvm/clang), which you can use as a baseline. Run > regressions on both the clean and modified versions to see if you cause > any additional failures.Makes sense.> > 2. I see clang has its own test suite under clang/test. I assume when > I run > > llvm/test, the test infrastructure also runs clang/test. > > How are you invoking the tests? I typically do 'make check-all' from > my build directory, which runs both regression suites.Yes, I am doing "make check-all".> > 3. For running projects/test-suite, the instruction guide directs to > build > > llvm-gcc. Why is llvm-gcc required? Why is clang not being used? > > Very possible the documentation needs updating.I see. Do you run projects/test-suite with clang?> > 4. I see many compile errors when running projects/test-suite. Is it > > expected? > > No.I will try to debug. Do you have a log file from your previous runs that I can use? Thanks Chad. I appreciate it. -Sundeep
On Jul 28, 2011, at 8:18 AM, Sundeep wrote:>>> 1. I checked out and built llvm, clang, and test-suite from svn tip. >> When I >>> run llvm/test *without* my changes on x86, I see 3 failures. Is it >> expected? >>> I was expecting svn tip on x86 to be clean. Is there a nightly >> regression >>> result available somewhere that I can check for broken tests. Here is >> the >>> report from my log: >>> >>> Failing Tests (3): >>> Clang :: Analysis/iterators.cpp >>> Clang :: Driver/hello.c >>> Clang :: Preprocessor/macro_paste_c_block_comment.c >>> >>> Expected Passes : 8757 >>> Expected Failures : 69 >>> Unsupported Tests : 551 >>> Unexpected Failures: 3 >>> >> >> Because you're living life on the bleeding edge (i.e., pulling code >> from the svn repository) it's possible for this to happen from time to >> time. A fix is probably already in the works. One way to ensure you >> don't introduce a new error is to maintain a clean (i.e., unmodified >> version of llvm/clang), which you can use as a baseline. Run >> regressions on both the clean and modified versions to see if you cause >> any additional failures. > > Makes sense. > >>> 2. I see clang has its own test suite under clang/test. I assume when >> I run >>> llvm/test, the test infrastructure also runs clang/test. >> >> How are you invoking the tests? I typically do 'make check-all' from >> my build directory, which runs both regression suites. > > Yes, I am doing "make check-all". > >>> 3. For running projects/test-suite, the instruction guide directs to >> build >>> llvm-gcc. Why is llvm-gcc required? Why is clang not being used? >> >> Very possible the documentation needs updating. > > I see. Do you run projects/test-suite with clang? >>From your build/project/test-suite directory, try running:make ARCH=x86_64 CC_UNDER_TEST_IS_CLANG=1 CC_UNDER_TEST_TARGET_IS_X86_64=1 DISABLE_CBE=1 DISABLE_JIT=1 ENABLE_HASHED_PROGRAM_OUTPUT=1 ENABLE_OPTIMIZED=1 LD_ENV_OVERRIDES="env DYLD_LIBRARY_PATH=/path/to/your/build/Release+Asserts/lib" LLC_OPTFLAGS=-O3 LLI_OPTFLAGS=-O3 OPTFLAGS=-O3 SMALL_PROBLEM_SIZE=1 TARGET_CC="" TARGET_CXX="" TARGET_FLAGS="-arch x86_64" TARGET_LLVMGCC=/path/to/your/build/Release+Asserts/bin/clang TARGET_LLVMGXX=/path/to/your/build/Release+Asserts/bin/clang++ TEST=simple USE_REFERENCE_OUTPUT=1 report ***Note the ENABLE_OPTIMIZED=1 flag requires that you run configure with '--enable-optimized' at config time (i.e., ../llvm/configure --enable-optimized). You may have to fiddle with the options, but I think this will get you going in the right direction.>>> 4. I see many compile errors when running projects/test-suite. Is it >>> expected? >> >> No. > > I will try to debug. Do you have a log file from your previous runs that I > can use? >I don't have any on hand, nor do I think they would be of help. Try the above make command a see what happens. For the most part everything should PASS.> Thanks Chad. I appreciate it. >You're welcome Sundeep.> -Sundeep >
Hi Chad, I finally got it to work. I think "LLVM Testing Infrastructure Guide" should be updated to reflect the latest changes (use clang instead of llvm-gcc and make command etc.). I ran into problems with using system headers and startup files (crt*.o). So I had to build and install llvm-gcc, next build clang using headers from llvm-gcc install dir, and then use clang to run regressions under projects/test-suite. It works beautifully. Anyways, I wanted to thank you for all your help. I really appreciate it. -Sundeep> -----Original Message----- > From: Chad Rosier [mailto:mcrosier at apple.com] > Sent: Thursday, July 28, 2011 12:37 PM > To: Sundeep > Cc: cfe-dev at cs.uiuc.edu; 'LLVM Dev' > Subject: Re: [LLVMdev] LLVM / CLANG Test Infrastructure Question > > > On Jul 28, 2011, at 8:18 AM, Sundeep wrote: > > >>> 1. I checked out and built llvm, clang, and test-suite from svn > tip. > >> When I > >>> run llvm/test *without* my changes on x86, I see 3 failures. Is it > >> expected? > >>> I was expecting svn tip on x86 to be clean. Is there a nightly > >> regression > >>> result available somewhere that I can check for broken tests. Here > is > >> the > >>> report from my log: > >>> > >>> Failing Tests (3): > >>> Clang :: Analysis/iterators.cpp > >>> Clang :: Driver/hello.c > >>> Clang :: Preprocessor/macro_paste_c_block_comment.c > >>> > >>> Expected Passes : 8757 > >>> Expected Failures : 69 > >>> Unsupported Tests : 551 > >>> Unexpected Failures: 3 > >>> > >> > >> Because you're living life on the bleeding edge (i.e., pulling code > >> from the svn repository) it's possible for this to happen from time > to > >> time. A fix is probably already in the works. One way to ensure > you > >> don't introduce a new error is to maintain a clean (i.e., unmodified > >> version of llvm/clang), which you can use as a baseline. Run > >> regressions on both the clean and modified versions to see if you > cause > >> any additional failures. > > > > Makes sense. > > > >>> 2. I see clang has its own test suite under clang/test. I assume > when > >> I run > >>> llvm/test, the test infrastructure also runs clang/test. > >> > >> How are you invoking the tests? I typically do 'make check-all' > from > >> my build directory, which runs both regression suites. > > > > Yes, I am doing "make check-all". > > > >>> 3. For running projects/test-suite, the instruction guide directs > to > >> build > >>> llvm-gcc. Why is llvm-gcc required? Why is clang not being used? > >> > >> Very possible the documentation needs updating. > > > > I see. Do you run projects/test-suite with clang? > > > > From your build/project/test-suite directory, try running: > > make ARCH=x86_64 CC_UNDER_TEST_IS_CLANG=1 > CC_UNDER_TEST_TARGET_IS_X86_64=1 DISABLE_CBE=1 DISABLE_JIT=1 > ENABLE_HASHED_PROGRAM_OUTPUT=1 ENABLE_OPTIMIZED=1 LD_ENV_OVERRIDES="env > DYLD_LIBRARY_PATH=/path/to/your/build/Release+Asserts/lib" > LLC_OPTFLAGS=-O3 LLI_OPTFLAGS=-O3 OPTFLAGS=-O3 SMALL_PROBLEM_SIZE=1 > TARGET_CC="" TARGET_CXX="" TARGET_FLAGS="-arch x86_64" > TARGET_LLVMGCC=/path/to/your/build/Release+Asserts/bin/clang > TARGET_LLVMGXX=/path/to/your/build/Release+Asserts/bin/clang++ > TEST=simple USE_REFERENCE_OUTPUT=1 report > > ***Note the ENABLE_OPTIMIZED=1 flag requires that you run configure > with '--enable-optimized' at config time (i.e., ../llvm/configure -- > enable-optimized). > > You may have to fiddle with the options, but I think this will get you > going in the right direction. > > >>> 4. I see many compile errors when running projects/test-suite. Is > it > >>> expected? > >> > >> No. > > > > I will try to debug. Do you have a log file from your previous runs > that I > > can use? > > > > I don't have any on hand, nor do I think they would be of help. Try > the above make command a see what happens. For the most part > everything should PASS. > > > Thanks Chad. I appreciate it. > > > > You're welcome Sundeep. > > > -Sundeep > >