llvm dev - Nov 2012 - [LLVMdev] YA Vectorization Benchmark

On 6 November 2012 22:28, Daniel Dunbar <daniel at zuster.org>
wrote:
> Is it possible instead to refactor the tests so that each binary
corresponds
> to one test? For example, look at how Hal went about integrating TSVC:
It should be possible. I'll have to understand better what the
preamble does to make sure I'm not stripping out important stuff, but
also what to copy to each kernel's initialization.

Also, I don't know how the timing functions perform across platforms.
I'd have to implement a decent enough timing system, platform
independent, to factor out the initialization step.

> Other things that I would *like* before integrating it:
>  - Rip out the CPU ID stuff, this isn't useful and adds messiness.
Absolutely, that is meaningless.

>  - Have the test just produce output that can be compared, instead of
> including its own check routines
I can make it print numbers in order, is that good enough for the
comparison routines?

If I got it right, the tests self-validates the results, so at least
we know it executed correctly in the end. I can make it produce "OK"
and "FAIL" either way with some numbers stating the timing.

>  - Have the tests run for fixed iterations, instead of doing their own
> adaptive run
Yes, that's rubbish. That was needed to compare the results based on
CPU specific features, but we don't need that.

>  - Produce reference output files, so it works with USE_REFERENCE_OUTPUT=1
Is this a simple diff or do you compare the numerical results by value+-stdev?


cheers,
--renato

On Wed, Nov 7, 2012 at 12:39 AM, Renato Golin <rengolin at
systemcall.org>wrote:
> On 6 November 2012 22:28, Daniel Dunbar <daniel at zuster.org> wrote:
> > Is it possible instead to refactor the tests so that each binary
> corresponds
> > to one test? For example, look at how Hal went about integrating TSVC:
>
> It should be possible. I'll have to understand better what the
> preamble does to make sure I'm not stripping out important stuff, but
> also what to copy to each kernel's initialization.
>
> Also, I don't know how the timing functions perform across platforms.
> I'd have to implement a decent enough timing system, platform
> independent, to factor out the initialization step.
>
The way we handle timing of all the other tests is just by timing the
executable. This isn't perfect, but its what we use everywhere else so we
should stick with keeping it outside the tests.

>
>
> > Other things that I would *like* before integrating it:
> >  - Rip out the CPU ID stuff, this isn't useful and adds messiness.
>
> Absolutely, that is meaningless.
>
>
> >  - Have the test just produce output that can be compared, instead of
> > including its own check routines
>
> I can make it print numbers in order, is that good enough for the
> comparison routines?
>
Yup.

> If I got it right, the tests self-validates the results, so at least
> we know it executed correctly in the end. I can make it produce
"OK"
> and "FAIL" either way with some numbers stating the timing.
>
Yeah, we can get rid of that, the way we check all the other tests is by
comparing to reference outputs or output from a reference binary (compiled
by the system compiler).

> >  - Have the tests run for fixed iterations, instead of doing their own
> > adaptive run
>
> Yes, that's rubbish. That was needed to compare the results based on
> CPU specific features, but we don't need that.
>
>
> >  - Produce reference output files, so it works with
> USE_REFERENCE_OUTPUT=1
>
> Is this a simple diff or do you compare the numerical results by
> value+-stdev?
>
We have support for value + tolerance. You can set FP_TOLERANCE=.0001 or so
in the Makefile to tune the limit. For example
see SingleSource/Benchmarks/Misc/Makefile.

 - Daniel

cheers,
> --renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20121107/7ee1705a/attachment.html>

Daniel, Nadav, Hal,

So, after some painstakingly boring re-formatting, I've split the 24
kernels into 24 files (and left a horrible header file with code in
it, which I'll clean up later).

Since we're taking times in the benchmark tool, and we're trying to
assert the quality of the FP approximation by the vectorization, I'll
try to come up with a reasonable watermark for each test. Maybe adding
the values of all elements in the result vector, or something, and
printing that value, using FP_TOLERANCE to detect errors in the
vectorized code.

Does this seem sensible?

cheers,
--renato