similar to: [LLVMdev] LNT BenchmarkGame

Hi Renato, On 12/03/13 15:33, Renato Golin wrote: > On 12 March 2013 14:24, Tim Northover <t.p.northover at gmail.com > <mailto:t.p.northover at gmail.com>> wrote: > > Could be worth a try. But if that thing really is generating random > numbers I'm not sure replacing one genuine cast-iron random number > with another is the best solution long-term.

On 12 March 2013 16:21, Duncan Sands <baldrick at free.fr> wrote: > IIRC the reference output is not used by default. You have to put > USE_REFERENCE_OUTPUT := 1 > in the Makefile in order to make use of the reference output. As > BenchmarkGame doesn't have this, are you sure the reference output > is causing the problem? > That was my initial assumption, too. But

On 12 March 2013 14:24, Tim Northover <t.p.northover at gmail.com> wrote: > Could be worth a try. But if that thing really is generating random > numbers I'm not sure replacing one genuine cast-iron random number > with another is the best solution long-term. > The test is initializing srand(1), so in theory, it shouldn't be different between compilers, since Clang is

> Is there any chance that the results are not being checked correctly? Any > other ideas? I think I vaguely convinced myself that the infrastructure didn't actually check whether tests it classified as benchmarks passed or failed. Not sure I had any good evidence for it other than things like you're seeing. > I'm tempted to just change the reference output and see what >

----- Original Message ----- > From: "Renato Golin" <renato.golin at linaro.org> > To: "Marshall Clow" <mclow.lists at gmail.com> > Cc: "LLVM Dev" <llvmdev at cs.uiuc.edu> > Sent: Tuesday, March 12, 2013 10:22:41 AM > Subject: Re: [LLVMdev] LNT BenchmarkGame > > > > On 12 March 2013 14:53, Marshall Clow < mclow.lists

On 12 March 2013 15:28, Hal Finkel <hfinkel at anl.gov> wrote: > Can't we just paste in a RNG so that we'll get the same output on all > systems (and can still use the reference output)? > We can, though other tests suffer from the same issue. Would be good to have a solution to all of them without pasting the same code on all of them. I really thought that the native

On 12 March 2013 16:48, Daniel Dunbar <daniel at zuster.org> wrote: > The former mode is historically what the test suite did, the latter mode > is substantially faster (and independent of bugs in the native CC). > Yes, I agree this is better for many cases, but not for all. Implementing RNG that is good enough for the tests' purposes, fast enough not to steal the

On Tue, Mar 12, 2013 at 10:23 AM, Renato Golin <renato.golin at linaro.org>wrote: > On 12 March 2013 16:48, Daniel Dunbar <daniel at zuster.org> wrote: > >> The former mode is historically what the test suite did, the latter mode >> is substantially faster (and independent of bugs in the native CC). >> > > Yes, I agree this is better for many cases, but

Hi Renato, On 12/03/13 17:33, Renato Golin wrote: > On 12 March 2013 16:21, Duncan Sands <baldrick at free.fr > <mailto:baldrick at free.fr>> wrote: > > IIRC the reference output is not used by default. You have to put > USE_REFERENCE_OUTPUT := 1 > in the Makefile in order to make use of the reference output. As > BenchmarkGame doesn't

On Mar 12, 2013, at 7:36 AM, Tim Northover <t.p.northover at gmail.com> wrote: > Hi Renato, > >> The test is initializing srand(1), so in theory, it shouldn't be different >> between compilers, since Clang is using the same libraries. > > If Clang and GCC disagree on the same source, same machine and with > the same libraries, that certainly is odd. But it

Hi Renato, > The test is initializing srand(1), so in theory, it shouldn't be different > between compilers, since Clang is using the same libraries. If Clang and GCC disagree on the same source, same machine and with the same libraries, that certainly is odd. But it doesn't make checking against the output of a particular libc's RNG any better an idea in general. Cheers.

----- Original Message ----- > From: "Daniel Dunbar" <daniel at zuster.org> > To: "Renato Golin" <renato.golin at linaro.org> > Cc: "Hal Finkel" <hfinkel at anl.gov>, "Marshall Clow" <mclow.lists at gmail.com>, "LLVM Dev" <llvmdev at cs.uiuc.edu> > Sent: Tuesday, March 12, 2013 12:30:12 PM > Subject: Re:

On Tue, Mar 12, 2013 at 9:19 AM, Renato Golin <renato.golin at linaro.org>wrote: > On 12 March 2013 15:28, Hal Finkel <hfinkel at anl.gov> wrote: > >> Can't we just paste in a RNG so that we'll get the same output on all >> systems (and can still use the reference output)? >> > > We can, though other tests suffer from the same issue. Would be good

On 12 March 2013 14:53, Marshall Clow <mclow.lists at gmail.com> wrote: > I agree; I'm pretty sure that the only guarantee is that for a given > implementation of stand, if you initialize it with the same seed, you get > the same sequence. > > There is no "correct" sequence. > I'm not suggesting a correct sequence, I'm just looking for a way to turn

On 12 March 2013 19:21, Hal Finkel <hfinkel at anl.gov> wrote: > +1 > > There are a couple of example implementations here which are only a few > lines long: > http://wiki.osdev.org/Random_Number_Generator I was going to rant about the quality of simple LCGs but it seems that *all* standard implementations rely on that, so the argument of using a standard library against a

On Mar 12, 2013, at 12:48, Renato Golin <renato.golin at linaro.org> wrote: On 12 March 2013 19:21, Hal Finkel <hfinkel at anl.gov> wrote: > +1 > > There are a couple of example implementations here which are only a few > lines long: > http://wiki.osdev.org/Random_Number_Generator I was going to rant about the quality of simple LCGs but it seems that *all* standard

Hi Renato, I've investigated a few of these for AArch64 recently, and some of the results will be applicable in the 32-bit world too. > MultiSource/Benchmarks/McCat/12-IOtest/iotest.execution_time This is because of disagreement between ABIs over whether "char" is signed. ARM says no, x86 says yes. >

Hi Folks, Looks like our LNT ARM buildbot with the vectorizer is running and producing good results. There are only 11 failures: FAIL: MultiSource/Applications/Burg/burg.execution_time (1 of 1104) FAIL: MultiSource/Applications/ClamAV/clamscan.execution_time (2 of 1104) FAIL: MultiSource/Applications/lemon/lemon.execution_time (3 of 1104) FAIL:

Hi Renato, I noticed the bot yesterday. Thanks for working on this! On Feb 19, 2013, at 7:45 AM, Renato Golin <renato.golin at linaro.org> wrote: > Hi Folks, > > Looks like our LNT ARM buildbot with the vectorizer is running and producing good results. Do you have a base run with vectorization turned off? So we could see where we are degrading things? When you say good

While porting my backends to llvm-3.2, I found a few places where the optimizers could have performed better. I believe the mainstream targets can also benefits from my tweaks. But before upstreaming my changes, I would like to quantify their merits on other applications --- not just my domain specific codes. In a word, it seemed the right time for me to start using LNT :) I followed the