On Tue, Mar 12, 2013 at 10:23 AM, Renato Golin <renato.golin at linaro.org>wrote:> On 12 March 2013 16:48, Daniel Dunbar <daniel at zuster.org> wrote: > >> The former mode is historically what the test suite did, the latter mode >> is substantially faster (and independent of bugs in the native CC). >> > > Yes, I agree this is better for many cases, but not for all. Implementing > RNG that is good enough for the tests' purposes, fast enough not to steal > the benchmarks' hot spots and does not use target/library-specific code is > not trivial. >This is not true, all one needs to do is replace existing srand(), rand() with some specific platforms version (and those are usually very simple RNGs). If the code is already using srand()/rand() then there is no reason to assume somehow the benchmark is worse if it always used the FreeBSD one, say, as opposed to a platform specific one. - Daniel I think that, in this particular case, having bugs in GCC is far less> problematic than assuming fixed outputs. > > I've tried USE_REFERENCE_OUTPUT := 0 on the Makefile, but the test.log > still prints it as 1 (and fails). > > cheers, > --renato >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130312/4bcc2d5e/attachment.html>
On 12 March 2013 17:30, Daniel Dunbar <daniel at zuster.org> wrote:> If the code is already using srand()/rand() then there is no reason to > assume somehow the benchmark is worse if it always used the FreeBSD one, > say, as opposed to a platform specific one. >I'm not convinced that running GCC on library-specific tests will be worse than pasting library code inside each test that has a library problem. In theory, it should just work if we manage to disable USE_REFERENCE_OUTPUT for those particular tests. cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130312/f0ad1173/attachment.html>
----- Original Message -----> From: "Daniel Dunbar" <daniel at zuster.org> > To: "Renato Golin" <renato.golin at linaro.org> > Cc: "Hal Finkel" <hfinkel at anl.gov>, "Marshall Clow" <mclow.lists at gmail.com>, "LLVM Dev" <llvmdev at cs.uiuc.edu> > Sent: Tuesday, March 12, 2013 12:30:12 PM > Subject: Re: [LLVMdev] LNT BenchmarkGame > > > > > > > > On Tue, Mar 12, 2013 at 10:23 AM, Renato Golin < > renato.golin at linaro.org > wrote: > > > > > On 12 March 2013 16:48, Daniel Dunbar < daniel at zuster.org > wrote: > > > > > > > The former mode is historically what the test suite did, the latter > mode is substantially faster (and independent of bugs in the native > CC). > > > > Yes, I agree this is better for many cases, but not for all. > Implementing RNG that is good enough for the tests' purposes, fast > enough not to steal the benchmarks' hot spots and does not use > target/library-specific code is not trivial. > > > This is not true, all one needs to do is replace existing srand(), > rand() with some specific platforms version (and those are usually > very simple RNGs). If the code is already using srand()/rand() then > there is no reason to assume somehow the benchmark is worse if it > always used the FreeBSD one, say, as opposed to a platform specific > one.+1 There are a couple of example implementations here which are only a few lines long: http://wiki.osdev.org/Random_Number_Generator -Hal> > - Daniel > > > > > > > > I think that, in this particular case, having bugs in GCC is far less > problematic than assuming fixed outputs. > > > I've tried USE_REFERENCE_OUTPUT := 0 on the Makefile, but the > test.log still prints it as 1 (and fails). > > > cheers, > --renato >
> There are a couple of example implementations here which are only a few lines long: > http://wiki.osdev.org/Random_Number_GeneratorYep, if the writers cared about the RNG in use they wouldn't have relied on the standard library to get it right. They're notoriously bad for anything but toys. Tim.
On 12 March 2013 19:21, Hal Finkel <hfinkel at anl.gov> wrote:> +1 > > There are a couple of example implementations here which are only a few > lines long: > http://wiki.osdev.org/Random_Number_GeneratorI was going to rant about the quality of simple LCGs but it seems that *all* standard implementations rely on that, so the argument of using a standard library against a one-liner is void. ;) Now, on to the practical issues: is there a place where this can be added once and used everywhere in the LNT, or should we just add that function to all tests that use it? cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130312/1b461a42/attachment.html>
Hi, | There are a couple of example implementations here which are only a few lines long: | http://wiki.osdev.org/Random_Number_Generator As Tim mentioned originally using libc rand argues that the quality of the random number generator won't affect benchmarks results, the only possible issue in using one of those "not random, just looks it" generators I can see is if any benchmarks were added in future which are multithreaded/parallelised. In addition to the non-determinisim of using a single global variable seed, its cache line might be hammered enough that it actually contributes significantly to the benchmark run time (relative to a per-core RNG). But adding parallel tests seems unlikely. Cheers, Dave