search for: use_reference_output

On 12 March 2013 16:21, Duncan Sands <baldrick at free.fr> wrote: > IIRC the reference output is not used by default. You have to put > USE_REFERENCE_OUTPUT := 1 > in the Makefile in order to make use of the reference output. As > BenchmarkGame doesn't have this, are you sure the reference output > is causing the problem? > That was my initial assumption, too. But if I just run that test, the Makefile doesn't use GCC at all and on...

Hi Renato, On 12/03/13 17:33, Renato Golin wrote: > On 12 March 2013 16:21, Duncan Sands <baldrick at free.fr > <mailto:baldrick at free.fr>> wrote: > > IIRC the reference output is not used by default. You have to put > USE_REFERENCE_OUTPUT := 1 > in the Makefile in order to make use of the reference output. As > BenchmarkGame doesn't have this, are you sure the reference output > is causing the problem? > > > That was my initial assumption, too. But if I just run that test, the Makefile > doesn...

...() then there is no reason to assume somehow the benchmark is worse if it always used the FreeBSD one, say, as opposed to a platform specific one. - Daniel I think that, in this particular case, having bugs in GCC is far less > problematic than assuming fixed outputs. > > I've tried USE_REFERENCE_OUTPUT := 0 on the Makefile, but the test.log > still prints it as 1 (and fails). > > cheers, > --renato > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130312/4bcc2d5e/attachment.html>

...all problems go away, > since the result will be target dependent (or rather, library dependent). Is > there a way to turn on the dynamic generation of the native file instead of > copying it from the reference_output? IIRC the reference output is not used by default. You have to put USE_REFERENCE_OUTPUT := 1 in the Makefile in order to make use of the reference output. As BenchmarkGame doesn't have this, are you sure the reference output is causing the problem? Ciao, Duncan.

...ng RNG that is good enough for the tests' purposes, fast enough not to steal the benchmarks' hot spots and does not use target/library-specific code is not trivial. I think that, in this particular case, having bugs in GCC is far less problematic than assuming fixed outputs. I've tried USE_REFERENCE_OUTPUT := 0 on the Makefile, but the test.log still prints it as 1 (and fails). cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130312/5852fa34/attachment.html>

I use variants on the following command: make -k TEST=simple ARCH=ARM TARGET_FLAGS="-arch armv7 -isysroot path-to-sdk -ccc-install-dir path-to-binutils" USE_REFERENCE_OUTPUT=1 REMOTE_HOST=localhost REMOTE_USER=root REMOTE_CLIENT=ssh REMOTE_PORT=10022 DISABLE_JIT=1 DISABLE_CBE=1 SMALL_PROBLEM_SIZE=1 TARGET_CC=path-to-clang TARGET_CXX=path-to-clang++ report That assumes key-based ssh authentication (so you don't get password prompted for every execution) and that th...

The issues seems to be this line in the tools Makefile timeit-target: timeit.c $(LD_ENV_OVERRIDES) $(LCC) -o $@ $< $(LDFLAGS) $(CFLAGS) $(TARGET_FLAGS) -O3 It should not add target flags if we are simulating the target on the host. On 06/06/2013 06:59 PM, reed kotler wrote: > I want to get lnt to use qemu for the execution. > > In that case, RHOST= is not set. > > But I

...stating the timing. > - Have the tests run for fixed iterations, instead of doing their own > adaptive run Yes, that's rubbish. That was needed to compare the results based on CPU specific features, but we don't need that. > - Produce reference output files, so it works with USE_REFERENCE_OUTPUT=1 Is this a simple diff or do you compare the numerical results by value+-stdev? cheers, --renato

...the CPU ID stuff, this isn't useful and adds messiness. - Have the test just produce output that can be compared, instead of including its own check routines - Have the tests run for fixed iterations, instead of doing their own adaptive run - Produce reference output files, so it works with USE_REFERENCE_OUTPUT=1 The kernels themselves are really trivial, so it would be ideal if it was split up to be one-test-per file with minimal other stuff in the test other than setup and output. - Daniel On Tue, Nov 6, 2012 at 1:53 PM, Renato Golin <rengolin at systemcall.org>wrote: > Ok, I got the benc...

I want to get lnt to use qemu for the execution. In that case, RHOST= is not set. But I change the Arch because I am going to run in cross mode. Then I'm setting RUNUNDER to be a script which runs qemu. In this case it builds timeit-target as a Mips which fails because this is running on x86. ~/mysandbox/bin/lnt runtest nt --sandbox ~/mysandbox --cc /local/llvmpb_a/install/bin/clang

On 12 March 2013 14:24, Tim Northover <t.p.northover at gmail.com> wrote: > Could be worth a try. But if that thing really is generating random > numbers I'm not sure replacing one genuine cast-iron random number > with another is the best solution long-term. > The test is initializing srand(1), so in theory, it shouldn't be different between compilers, since Clang is

.../lib" LLC_OPTFLAGS=-O3 LLI_OPTFLAGS=-O3 OPTFLAGS=-O3 SMALL_PROBLEM_SIZE=1 TARGET_CC="" TARGET_CXX="" TARGET_FLAGS="-arch x86_64" TARGET_LLVMGCC=/path/to/your/build/Release+Asserts/bin/clang TARGET_LLVMGXX=/path/to/your/build/Release+Asserts/bin/clang++ TEST=simple USE_REFERENCE_OUTPUT=1 report ***Note the ENABLE_OPTIMIZED=1 flag requires that you run configure with '--enable-optimized' at config time (i.e., ../llvm/configure --enable-optimized). You may have to fiddle with the options, but I think this will get you going in the right direction. >>> 4. I see m...

Ok, I got the benchmark to work on test-suite, but it's not printing details for each run (or execution wouldn't work). I had to comment out the printf lines, but nothing more than that. I'm not sure how individual timings would have to be extracted, but the program produces output via text file, which can be used for comparison. Also, it does check the results and does report if they

On Tue, Mar 12, 2013 at 9:19 AM, Renato Golin <renato.golin at linaro.org>wrote: > On 12 March 2013 15:28, Hal Finkel <hfinkel at anl.gov> wrote: > >> Can't we just paste in a RNG so that we'll get the same output on all >> systems (and can still use the reference output)? >> > > We can, though other tests suffer from the same issue. Would be good

> > 1. I checked out and built llvm, clang, and test-suite from svn tip. > When I > > run llvm/test *without* my changes on x86, I see 3 failures. Is it > expected? > > I was expecting svn tip on x86 to be clean. Is there a nightly > regression > > result available somewhere that I can check for broken tests. Here is > the > > report from my log: > >

Hi Raul, I've successfully run them over ssh using LNT without too much difficulty. There is documentation on getting started with the LNT system here: http://llvm.org/docs/lnt/quickstart.html. It doesn't specifically mention remote execution, but if you give the lnt command a "--help" option then there are a few reasonably self-explanatory options described (--remote,

...e tests run for fixed iterations, instead of doing their own > > adaptive run > > Yes, that's rubbish. That was needed to compare the results based on > CPU specific features, but we don't need that. > > > > - Produce reference output files, so it works with > USE_REFERENCE_OUTPUT=1 > > Is this a simple diff or do you compare the numerical results by > value+-stdev? > We have support for value + tolerance. You can set FP_TOLERANCE=.0001 or so in the Makefile to tune the limit. For example see SingleSource/Benchmarks/Misc/Makefile. - Daniel cheers, > --rena...

...ich are only a few lines long: http://wiki.osdev.org/Random_Number_Generator -Hal > > - Daniel > > > > > > > > I think that, in this particular case, having bugs in GCC is far less > problematic than assuming fixed outputs. > > > I've tried USE_REFERENCE_OUTPUT := 0 on the Makefile, but the > test.log still prints it as 1 (and fails). > > > cheers, > --renato >

Hello, is there any option to run LLVM unit tests on ARM from your desktop? I mean, the code is properly cross-compiled on your desktop and you want to run the unit tests automatically on an ARM board. Chrood might work, but I am looking for some option like <runremote.sh --host XXX --user ....>. I have seen something related to it in projects/test-suite/Makefile.programs (REMOTE_HOST,