llvm dev - May 2021 - LLVM LNT floating point performance tests on X86 - using the llvm-test-suite benchmarks

> On May 19, 2021, at 17:08, Blower, Melanie I <melanie.blower at
intel.com> wrote:
> 
> What I'm trying to do is to determine whether the patch I'm
submitting is going to cause benchmarking problems that force the patch to be
reverted--since that happened the last time I committed the patch (several
months ago).

IIUC the problem with the patch was not runtime performance, but unexpected
results causing some tests/benchmarks to fail.

I think you might need to select a CPU on X86 to cause the mis-compares. Below
the commands I used to reproduce the failure of
MultiSource/Applications/oggenc/oggenc. Note `-march=native`. Without that, the
test passes. There are a couple of other failures as well.

> cmake -G Ninja \                
    -DCMAKE_C_COMPILER=/path/to/bin/clang \
    -DCMAKE_C_FLAGS="-O3 -march=native" -DCMAKE_CXX_FLAGS="-O3
-march=native" \
    Path/to/llvm-test-suite
> ninja MultiSource/Applications/oggenc/oggenc
> llvm-lit MultiSource/Applications/oggenc/
-- Testing: 1 tests, 1 workers --
FAIL: test-suite :: MultiSource/Applications/oggenc/oggenc.test (1 of 1)
********************
Failed Tests (1):
  test-suite :: MultiSource/Applications/oggenc/oggenc.test
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210519/bd158497/attachment.html>

> IIUC the problem with the patch was not runtime performance, but unexpected
results causing some tests/benchmarks to fail.
I believe we saw several problems. I may be merging issues with a couple of
different variations of this change in my mind, but we definitely saw
performance regressions on an LNT bot running on a Broadwell-based system at
some point due to different loop unrolling behavior when the llvm.fmuladd
intrinsic was used.

There were, as you note, test failures reported because of the change Melanie is
working with. I think those were caused by the tests being intolerant of
variations in floating point results (such as would be expected when FMA is
used) and not being very easy to update. Any suggestions on how to handle that
would also be helpful.

Finally, there was a build failure with aarch64 when this change was made. That
seems to have been caused by a problem in the aarch64 backend that was exposed
by this change. The associated bug (https://bugs.llvm.org/show_bug.cgi?id=44892)
has since been marked as fixed.

-Andy

From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Florian
Hahn via llvm-dev
Sent: Wednesday, May 19, 2021 9:25 AM
To: Blower, Melanie I <melanie.blower at intel.com>; llvm-dev <llvm-dev
at lists.llvm.org>
Cc: spatel+llvm at rotateright.com
Subject: Re: [llvm-dev] LLVM LNT floating point performance tests on X86 - using
the llvm-test-suite benchmarks




On May 19, 2021, at 17:08, Blower, Melanie I <melanie.blower at
intel.com<mailto:melanie.blower at intel.com>> wrote:

What I'm trying to do is to determine whether the patch I'm submitting
is going to cause benchmarking problems that force the patch to be
reverted--since that happened the last time I committed the patch (several
months ago).


IIUC the problem with the patch was not runtime performance, but unexpected
results causing some tests/benchmarks to fail.

I think you might need to select a CPU on X86 to cause the mis-compares. Below
the commands I used to reproduce the failure of
MultiSource/Applications/oggenc/oggenc. Note `-march=native`. Without that, the
test passes. There are a couple of other failures as well.

> cmake -G Ninja \
    -DCMAKE_C_COMPILER=/path/to/bin/clang \
    -DCMAKE_C_FLAGS="-O3 -march=native" -DCMAKE_CXX_FLAGS="-O3
-march=native" \
    Path/to/llvm-test-suite
> ninja MultiSource/Applications/oggenc/oggenc
> llvm-lit MultiSource/Applications/oggenc/
-- Testing: 1 tests, 1 workers --
FAIL: test-suite :: MultiSource/Applications/oggenc/oggenc.test (1 of 1)
********************
Failed Tests (1):
  test-suite :: MultiSource/Applications/oggenc/oggenc.test
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210519/6277adc6/attachment-0001.html>

Using march=native as suggested by @Florian Hahn<mailto:florian_hahn at
apple.com> I can see 20 failing tests, thanks!

I looked at the source for
SingleSource/Benchmarks/Polybench/linear-algebra/kernels/trmm which is one of
the failing tests and noticed that there is #pragma STDC FP_CONTRACT OFF in many
places, but not in the function kernel_trmm

Perhaps this function was overlooked for that pragma because the FMA is
syntactically not obvious due to the += operation?  So I added the pragma and
reran that test, and now that test is passing.

Would it be acceptable to modify trmm with this change?

From: Florian Hahn <florian_hahn at apple.com>
Sent: Wednesday, May 19, 2021 12:25 PM
To: Blower, Melanie I <melanie.blower at intel.com>; llvm-dev <llvm-dev
at lists.llvm.org>
Cc: Johannes Doerfert <johannesdoerfert at gmail.com>; spatel+llvm at
rotateright.com; hal.finkel.llvm at gmail.com
Subject: Re: LLVM LNT floating point performance tests on X86 - using the
llvm-test-suite benchmarks




On May 19, 2021, at 17:08, Blower, Melanie I <melanie.blower at
intel.com<mailto:melanie.blower at intel.com>> wrote:

What I'm trying to do is to determine whether the patch I'm submitting
is going to cause benchmarking problems that force the patch to be
reverted--since that happened the last time I committed the patch (several
months ago).


IIUC the problem with the patch was not runtime performance, but unexpected
results causing some tests/benchmarks to fail.

I think you might need to select a CPU on X86 to cause the mis-compares. Below
the commands I used to reproduce the failure of
MultiSource/Applications/oggenc/oggenc. Note `-march=native`. Without that, the
test passes. There are a couple of other failures as well.

> cmake -G Ninja \
    -DCMAKE_C_COMPILER=/path/to/bin/clang \
    -DCMAKE_C_FLAGS="-O3 -march=native" -DCMAKE_CXX_FLAGS="-O3
-march=native" \
    Path/to/llvm-test-suite
> ninja MultiSource/Applications/oggenc/oggenc
> llvm-lit MultiSource/Applications/oggenc/
-- Testing: 1 tests, 1 workers --
FAIL: test-suite :: MultiSource/Applications/oggenc/oggenc.test (1 of 1)
********************
Failed Tests (1):
  test-suite :: MultiSource/Applications/oggenc/oggenc.test
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210519/d0788f07/attachment.html>