search for: reference_output

...endian=no config.log:ENDIAN='little' config.status:s, at ENDIAN@,|#_!!_#|little,g configure.log:checking whether byte ordering is bigendian... no grep: External: Is a directory Makefile.config:ENDIAN := little === So far, so good. Makefile.programs: -if [ -f "$(PROJ_SRC_DIR)/$*.reference_output.$(ENDIAN)-endian.$(REFERENCE_OUTPUT_KEY)" ]; then \ Makefile.programs: cp $(PROJ_SRC_DIR)/$*.reference_output.$(ENDIAN)-endian.$(REFERENCE_OUTPUT_KEY) $@; \ Makefile.programs: elif [ -f "$(PROJ_SRC_DIR)/$*.reference_output.$(ENDIAN)-endian" ]; then \ Makefile.programs:...

...nce Clang is using the same libraries. Also, if the "native" result is generated by GCC, than all problems go away, since the result will be target dependent (or rather, library dependent). Is there a way to turn on the dynamic generation of the native file instead of copying it from the reference_output? cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130312/c9fb4761/attachment.html>

On 12 March 2013 16:21, Duncan Sands <baldrick at free.fr> wrote: > IIRC the reference output is not used by default. You have to put > USE_REFERENCE_OUTPUT := 1 > in the Makefile in order to make use of the reference output. As > BenchmarkGame doesn't have this, are you sure the reference output > is causing the problem? > That was my initial assumption, too. But if I just run that test, the Makefile doesn't use GCC at all and on...

...ou have sources also in the $LLVM_SRC_ROOT/projects/test-suite/External/SPEC/CINT2000/164.gzip? The following is content of above directory in my case. I have copied the CINT2000 sources in this directory manually. $ls -1 $LLVM_SRC_ROOT/projects/test-suite/External/SPEC/CINT2000/164.gzip 164.gzip.reference_output 164.gzip.reference_output.small compile_info compile_parms exec_info_input1 exec_info_input2 exec_info_input3 exec_info_input4 exec_info_input5 exec_info_input6 exec_info_input7 input1 Makefile make_src_164.gzip output1 output2 output3 output4 output5 output6 output7 patched_src src Does this look...

Hi Renato, > The test is initializing srand(1), so in theory, it shouldn't be different > between compilers, since Clang is using the same libraries. If Clang and GCC disagree on the same source, same machine and with the same libraries, that certainly is odd. But it doesn't make checking against the output of a particular libc's RNG any better an idea in general. Cheers.

...ame libraries. > > Also, if the "native" result is generated by GCC, than all problems go away, > since the result will be target dependent (or rather, library dependent). Is > there a way to turn on the dynamic generation of the native file instead of > copying it from the reference_output? IIRC the reference output is not used by default. You have to put USE_REFERENCE_OUTPUT := 1 in the Makefile in order to make use of the reference output. As BenchmarkGame doesn't have this, are you sure the reference output is causing the problem? Ciao, Duncan.

...ecause there's *always* an error bar. The output of O0, O1, O2, O3, Ofast, Os, Oz should all be within the boundaries of an average and its associated error bar. By understanding what's the *expected* output and its associated error range we can accurately predict what will be the correct reference_output and the tolerance for each individual test. Your solution 2 "works" because you're doing the matching yourself, in the code, and for that, you pay the penalty of running it twice. But it's not easy to control the tolerance, nor it's stable for all platforms where we don't...

...e the case. > We still do see cross-system discrepancies sometimes because of differences in denormal handling, but on the same system that should be consistent (aside, perhaps, from compiler-level constant-folding issues). But the test-suite doesn't run on a single system, nor it has one reference_output for each system. cheers, --renato

Hi All, I am trying to configure llvm so with test-suite enabled. I have downloaded the test-suite. Spec 2000 benchmarks sources are plugged in speccpu2000 at $LLVM_SRC_ROOT/projects/test-suite-externals/speccpu2000/benchspec e.g.: 164.gzip dir is $LLVM_SRC_ROOT/projects/test-suite-externals/speccpu2000/benchspec/164.gzip My configure command is: ../configure --disable-optimized

Hi Daya, > checking for spec2000 benchmark sources... no, not found in > $LLVM_SRC_ROOT/projects/test-suite-externals/speccpu2000/benchspec I have 164.gzip at $LLVM_SRC_ROOT/projects/test-suite/External/SPEC/CINT2000/164.gzip and it seems to work. Ciao, Duncan.

Hi Renato, On 12/03/13 17:33, Renato Golin wrote: > On 12 March 2013 16:21, Duncan Sands <baldrick at free.fr > <mailto:baldrick at free.fr>> wrote: > > IIRC the reference output is not used by default. You have to put > USE_REFERENCE_OUTPUT := 1 > in the Makefile in order to make use of the reference output. As > BenchmarkGame doesn't have this, are you sure the reference output > is causing the problem? > > > That was my initial assumption, too. But if I just run that test, the Makefile > doesn...

Ah, thank you, that makes sense. Is there any chance you can point me toward some documentation for how to add tests to the test-suite? I've found a fair bit on how to run the tests, but nothing on how to add new ones. Is it just as simple as adding testcase.c and testscase.reference_output in the appropriate directory? Thanks! On 01/02/2018 02:50 PM, Nemanja Ivanovic wrote: > Typically the "end-to-end" test cases reside in `projects/test-suite`. > All the test cases there build executable binaries from source, run > those and compare the output (as well as th...

...(though not perfect) result. If you sort clamscan.out-nat and clamscan.out-simple, you'll see that they're identical. I'm not sure if there is an option to run the program in sequential mode (if there is, the test will pass), but if not, the simplest way to fix the diff is to sort the reference_output and the program's output on execution. Adding "| sort" to the RUN_OPTIONS didn't have the desired effect (that was a long shot, anyway), does anyone know of a way to sort the output of a program on those Makefiles? Maybe a rule that can be intermediary of the final rule (I'm...

...$LLVM_SRC_ROOT/projects/test-suite/External/SPEC/CINT2000/164.gzip? > > The following is content of above directory in my case. I have copied the > CINT2000 sources in this directory manually. > > $ls -1 $LLVM_SRC_ROOT/projects/test-suite/External/SPEC/CINT2000/164.gzip > 164.gzip.reference_output > 164.gzip.reference_output.small > compile_info > compile_parms > exec_info_input1 > exec_info_input2 > exec_info_input3 > exec_info_input4 > exec_info_input5 > exec_info_input6 > exec_info_input7 > input1 > Makefile > make_src_164.gzip > output1 > outp...

...marks/LivermoreLoops/lloops.execution_time (2 of 2) When I use the option to only run this test: --only-test SingleSource/Benchmarks/LivermoreLoops I can see the binary and if I execute, it runs (still missing bits and bobs, but ignore that for now). I've seen some files with extension "reference_output", does that have anything to do with it? Since it "fails" on "compile_time", I'm guessing I'm not quite there yet, though. If you want to reproduce it, just add "LivermoreLoops" to the Benchmarks' Makefile and untar the attached to it. -- cheers, --...

...mes because of >> > differences in denormal handling, but on the same system that >> > should be consistent (aside, perhaps, from compiler-level >> > constant-folding issues). >> >> But the test-suite doesn't run on a single system, nor it has one >> reference_output for each system. > > I agree and understand, and we may need a tolerance in practice to deal with differences from denormal handling, etc. However, if Sebastian is seeing differences on the same system, we should understand why. Is he running on an ARM Darwin system, or an x86 using fp80 arit...

...thank you, that makes sense. > > Is there any chance you can point me toward some documentation for how to > add tests to the test-suite? I've found a fair bit on how to run the tests, > but nothing on how to add new ones. Is it just as simple as adding > testcase.c and testscase.reference_output in the appropriate directory? > > Thanks! > > On 01/02/2018 02:50 PM, Nemanja Ivanovic wrote: > > Typically the "end-to-end" test cases reside in `projects/test-suite`. All > the test cases there build executable binaries from source, run those and > compare the ou...

...est/build/SingleSource/Regression/C/Regression-C-2003-06-16-InvalidInitializer cd /òpt/llvm-test/build/SingleSource/Regression/C ; /òpt/llvm-test/build/tools/fpcmp /òpt/llvm-test/build/SingleSource/Regression/C/Output/Regression-C-2003-06-16-InvalidInitializer.test.out 2003-06-16-InvalidInitializer.reference_output + cd /òpt/llvm-test/build/SingleSource/Regression/C + /òpt/llvm-test/build/tools/fpcmp /òpt/llvm-test/build/SingleSource/Regression/C/Output/Regression-C-2003-06-16-InvalidInitializer.test.out 2003-06-16-InvalidInitializer.reference_output /òpt/llvm-test/build/tools/fpcmp: error: unable to open ...

Typically the "end-to-end" test cases reside in `projects/test-suite`. All the test cases there build executable binaries from source, run those and compare the output (as well as the return code) to what's expected. You should probably put your test cases in there. Please note also that there are subcategories there - there are single source file and multiple source file programs.

...cord the outputs, ignore the differences. 3. Collate each platofrm's output for each test and see how different they are To make it easier to compare, in the past, I've used this trick: 1. Run in one platform, ex. x86_64, ignored the reference 2. Copy the output of those tests back to the reference_output 3. Run on a different platform, tweaking the tolerance until it "passes" 4. Run on yet another platform, making sure you don't need to tweak the tolerance yet again If the tolerance is "too high" for that test, we can further discuss how to change it to make it better. If n...