llvm dev - May 2021 - LLVM LNT floating point performance tests on X86 - using the llvm-test-suite benchmarks

Using march=native as suggested by @Florian Hahn<mailto:florian_hahn at
apple.com> I can see 20 failing tests, thanks!

I looked at the source for
SingleSource/Benchmarks/Polybench/linear-algebra/kernels/trmm which is one of
the failing tests and noticed that there is #pragma STDC FP_CONTRACT OFF in many
places, but not in the function kernel_trmm

Perhaps this function was overlooked for that pragma because the FMA is
syntactically not obvious due to the += operation?  So I added the pragma and
reran that test, and now that test is passing.

Would it be acceptable to modify trmm with this change?

From: Florian Hahn <florian_hahn at apple.com>
Sent: Wednesday, May 19, 2021 12:25 PM
To: Blower, Melanie I <melanie.blower at intel.com>; llvm-dev <llvm-dev
at lists.llvm.org>
Cc: Johannes Doerfert <johannesdoerfert at gmail.com>; spatel+llvm at
rotateright.com; hal.finkel.llvm at gmail.com
Subject: Re: LLVM LNT floating point performance tests on X86 - using the
llvm-test-suite benchmarks




On May 19, 2021, at 17:08, Blower, Melanie I <melanie.blower at
intel.com<mailto:melanie.blower at intel.com>> wrote:

What I'm trying to do is to determine whether the patch I'm submitting
is going to cause benchmarking problems that force the patch to be
reverted--since that happened the last time I committed the patch (several
months ago).


IIUC the problem with the patch was not runtime performance, but unexpected
results causing some tests/benchmarks to fail.

I think you might need to select a CPU on X86 to cause the mis-compares. Below
the commands I used to reproduce the failure of
MultiSource/Applications/oggenc/oggenc. Note `-march=native`. Without that, the
test passes. There are a couple of other failures as well.

> cmake -G Ninja \
    -DCMAKE_C_COMPILER=/path/to/bin/clang \
    -DCMAKE_C_FLAGS="-O3 -march=native" -DCMAKE_CXX_FLAGS="-O3
-march=native" \
    Path/to/llvm-test-suite
> ninja MultiSource/Applications/oggenc/oggenc
> llvm-lit MultiSource/Applications/oggenc/
-- Testing: 1 tests, 1 workers --
FAIL: test-suite :: MultiSource/Applications/oggenc/oggenc.test (1 of 1)
********************
Failed Tests (1):
  test-suite :: MultiSource/Applications/oggenc/oggenc.test
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210519/d0788f07/attachment.html>

On 5/19/21 1:54 PM, Blower, Melanie I wrote:
> Using march=native as suggested by @Florian Hahn<mailto:florian_hahn at
apple.com> I can see 20 failing tests, thanks!
>
> I looked at the source for
SingleSource/Benchmarks/Polybench/linear-algebra/kernels/trmm which is one of
the failing tests and noticed that there is #pragma STDC FP_CONTRACT OFF in many
places, but not in the function kernel_trmm
>
> Perhaps this function was overlooked for that pragma because the FMA is
syntactically not obvious due to the += operation?  So I added the pragma and
reran that test, and now that test is passing.
>
> Would it be acceptable to modify trmm with this change?
Yes.

>
> From: Florian Hahn <florian_hahn at apple.com>
> Sent: Wednesday, May 19, 2021 12:25 PM
> To: Blower, Melanie I <melanie.blower at intel.com>; llvm-dev
<llvm-dev at lists.llvm.org>
> Cc: Johannes Doerfert <johannesdoerfert at gmail.com>; spatel+llvm at
rotateright.com; hal.finkel.llvm at gmail.com
> Subject: Re: LLVM LNT floating point performance tests on X86 - using the
llvm-test-suite benchmarks
>
>
>
>
> On May 19, 2021, at 17:08, Blower, Melanie I <melanie.blower at
intel.com<mailto:melanie.blower at intel.com>> wrote:
>
> What I'm trying to do is to determine whether the patch I'm
submitting is going to cause benchmarking problems that force the patch to be
reverted--since that happened the last time I committed the patch (several
months ago).
>
>
> IIUC the problem with the patch was not runtime performance, but unexpected
results causing some tests/benchmarks to fail.
>
> I think you might need to select a CPU on X86 to cause the mis-compares.
Below the commands I used to reproduce the failure of
MultiSource/Applications/oggenc/oggenc. Note `-march=native`. Without that, the
test passes. There are a couple of other failures as well.
>
>
>> cmake -G Ninja \
>      -DCMAKE_C_COMPILER=/path/to/bin/clang \
>      -DCMAKE_C_FLAGS="-O3 -march=native"
-DCMAKE_CXX_FLAGS="-O3 -march=native" \
>      Path/to/llvm-test-suite
>
>> ninja MultiSource/Applications/oggenc/oggenc
>> llvm-lit MultiSource/Applications/oggenc/
> -- Testing: 1 tests, 1 workers --
> FAIL: test-suite :: MultiSource/Applications/oggenc/oggenc.test (1 of 1)
> ********************
> Failed Tests (1):
>    test-suite :: MultiSource/Applications/oggenc/oggenc.test
>