search for: benchmarkgam

On 12 March 2013 16:21, Duncan Sands <baldrick at free.fr> wrote: > IIRC the reference output is not used by default. You have to put > USE_REFERENCE_OUTPUT := 1 > in the Makefile in order to make use of the reference output. As > BenchmarkGame doesn't have this, are you sure the reference output > is causing the problem? > That was my initial assumption, too. But if I just run that test, the Makefile doesn't use GCC at all and only copies the reference_output to the out-nat file. I then copied a "good" output t...

...pendent). Is > there a way to turn on the dynamic generation of the native file instead of > copying it from the reference_output? IIRC the reference output is not used by default. You have to put USE_REFERENCE_OUTPUT := 1 in the Makefile in order to make use of the reference output. As BenchmarkGame doesn't have this, are you sure the reference output is causing the problem? Ciao, Duncan.

...3 16:21, Duncan Sands <baldrick at free.fr > <mailto:baldrick at free.fr>> wrote: > > IIRC the reference output is not used by default. You have to put > USE_REFERENCE_OUTPUT := 1 > in the Makefile in order to make use of the reference output. As > BenchmarkGame doesn't have this, are you sure the reference output > is causing the problem? > > > That was my initial assumption, too. But if I just run that test, the Makefile > doesn't use GCC at all and only copies the reference_output to the out-nat file. if you look at the firs...

On 12 March 2013 14:24, Tim Northover <t.p.northover at gmail.com> wrote: > Could be worth a try. But if that thing really is generating random > numbers I'm not sure replacing one genuine cast-iron random number > with another is the best solution long-term. > The test is initializing srand(1), so in theory, it shouldn't be different between compilers, since Clang is

On Mar 12, 2013, at 7:36 AM, Tim Northover <t.p.northover at gmail.com> wrote: > Hi Renato, > >> The test is initializing srand(1), so in theory, it shouldn't be different >> between compilers, since Clang is using the same libraries. > > If Clang and GCC disagree on the same source, same machine and with > the same libraries, that certainly is odd. But it

On 12 March 2013 14:53, Marshall Clow <mclow.lists at gmail.com> wrote: > I agree; I'm pretty sure that the only guarantee is that for a given > implementation of stand, if you initialize it with the same seed, you get > the same sequence. > > There is no "correct" sequence. > I'm not suggesting a correct sequence, I'm just looking for a way to turn

Hi folks, I'm investigating the LNT failures on our bot and found that I cannot reproduce BenchmarkGame pass. I've compiled it with GCC, Clang on both ARM and x86_64, with -O3 or with the arguments that the test-suite passes to it and all I can get is the result below: Found duplicate: 420094 Found duplicate: 341335 Found duplicate: 150397 Found duplicate: 157527 Found duplicate: 269724 But n...

...o Golin" <renato.golin at linaro.org> > Cc: "Hal Finkel" <hfinkel at anl.gov>, "Marshall Clow" <mclow.lists at gmail.com>, "LLVM Dev" <llvmdev at cs.uiuc.edu> > Sent: Tuesday, March 12, 2013 12:30:12 PM > Subject: Re: [LLVMdev] LNT BenchmarkGame > > > > > > > > On Tue, Mar 12, 2013 at 10:23 AM, Renato Golin < > renato.golin at linaro.org > wrote: > > > > > On 12 March 2013 16:48, Daniel Dunbar < daniel at zuster.org > wrote: > > > > > > > The forme...

...iginal Message ----- > From: "Renato Golin" <renato.golin at linaro.org> > To: "Marshall Clow" <mclow.lists at gmail.com> > Cc: "LLVM Dev" <llvmdev at cs.uiuc.edu> > Sent: Tuesday, March 12, 2013 10:22:41 AM > Subject: Re: [LLVMdev] LNT BenchmarkGame > > > > On 12 March 2013 14:53, Marshall Clow < mclow.lists at gmail.com > > wrote: > > > > > > > I agree; I'm pretty sure that the only guarantee is that for a given > implementation of stand, if you initialize it with the same seed, >...

Hi Renato, > The test is initializing srand(1), so in theory, it shouldn't be different > between compilers, since Clang is using the same libraries. If Clang and GCC disagree on the same source, same machine and with the same libraries, that certainly is odd. But it doesn't make checking against the output of a particular libc's RNG any better an idea in general. Cheers.

On 12 March 2013 15:28, Hal Finkel <hfinkel at anl.gov> wrote: > Can't we just paste in a RNG so that we'll get the same output on all > systems (and can still use the reference output)? > We can, though other tests suffer from the same issue. Would be good to have a solution to all of them without pasting the same code on all of them. I really thought that the native

On 12 March 2013 16:48, Daniel Dunbar <daniel at zuster.org> wrote: > The former mode is historically what the test suite did, the latter mode > is substantially faster (and independent of bugs in the native CC). > Yes, I agree this is better for many cases, but not for all. Implementing RNG that is good enough for the tests' purposes, fast enough not to steal the

On 12 March 2013 19:21, Hal Finkel <hfinkel at anl.gov> wrote: > +1 > > There are a couple of example implementations here which are only a few > lines long: > http://wiki.osdev.org/Random_Number_Generator I was going to rant about the quality of simple LCGs but it seems that *all* standard implementations rely on that, so the argument of using a standard library against a

On Mar 12, 2013, at 12:48, Renato Golin <renato.golin at linaro.org> wrote: On 12 March 2013 19:21, Hal Finkel <hfinkel at anl.gov> wrote: > +1 > > There are a couple of example implementations here which are only a few > lines long: > http://wiki.osdev.org/Random_Number_Generator I was going to rant about the quality of simple LCGs but it seems that *all* standard

> Is there any chance that the results are not being checked correctly? Any > other ideas? I think I vaguely convinced myself that the infrastructure didn't actually check whether tests it classified as benchmarks passed or failed. Not sure I had any good evidence for it other than things like you're seeing. > I'm tempted to just change the reference output and see what >

On Tue, Mar 12, 2013 at 10:23 AM, Renato Golin <renato.golin at linaro.org>wrote: > On 12 March 2013 16:48, Daniel Dunbar <daniel at zuster.org> wrote: > >> The former mode is historically what the test suite did, the latter mode >> is substantially faster (and independent of bugs in the native CC). >> > > Yes, I agree this is better for many cases, but

Jakob Stoklund Olesen <stoklund at 2pi.dk> writes: > +10.0% SingleSource/Benchmarks/CoyoteBench/huffbench > +12.0% SingleSource/Benchmarks/McGill/chomp > +18.0% SingleSource/Benchmarks/BenchmarkGame/n-body > +45.5% SingleSource/Benchmarks/BenchmarkGame/puzzle > +10.0% SingleSource/Benchmarks/Shootout/heapsort > +10.5% MultiSource/Benchmarks/Trimaran/enc-3des/enc-3des > +10.9% SingleSource/Benchmarks/Shootout-C++/heapsort > +11.7% MultiSource/Benchmarks/Ptrdi...

...MultiSource/Applications/obsequi MultiSource/Benchmarks/MiBench/automotive-bitcount MultiSource/Benchmarks/MiBench/telecomm-FFT MultiSource/Benchmarks/Olden/voronoi MultiSource/Benchmarks/Ptrdist/anagram SingleSource/Benchmarks/BenchmarkGame Everything is built off trunk. Has anyone else seen these failures and found a fix? Perhaps I'm missing a dependency? There doesn't appear to be a linux machine on llvm.org/perf to compare with either. paul

On Tue, Mar 12, 2013 at 9:19 AM, Renato Golin <renato.golin at linaro.org>wrote: > On 12 March 2013 15:28, Hal Finkel <hfinkel at anl.gov> wrote: > >> Can't we just paste in a RNG so that we'll get the same output on all >> systems (and can still use the reference output)? >> > > We can, though other tests suffer from the same issue. Would be good

...time (497 of 986) FAIL: MultiSource/Benchmarks/MiBench/telecomm-FFT/telecomm-fft.execution_time (498 of 986) FAIL: MultiSource/Benchmarks/Olden/voronoi/voronoi.execution_time (499 of 986) FAIL: MultiSource/Benchmarks/Ptrdist/anagram/anagram.execution_time (500 of 986) FAIL: SingleSource/Benchmarks/BenchmarkGame/puzzle.execution_time (501 of 986) Notice how the test numbers are consecutive (494-501). They all pass when Burg passes. paul On 2013-01-20 2:50 PM, "Redmond, Paul" <paul.redmond at intel.com> wrote: >Hi, > >I figured out how to resolve the failures. I noticed that M...