llvm dev - Oct 2016 - Matthias` suggestion for "test-suite" tests that are broken at "-Ofast" and are difficult to "repair"

On 10/11/2016 at 4:15 PM, Matthias Braun wrote:
> I don't find it surprising that some applications do not work properly
with -ffast-math and I think we
> have to accept that fact.  I think it is valid to skip those tests in the
test-suite when a fast math
> flag combination is used (after making sure there is no easy way to make
the test more robust).
> I would add a "TEST_SUITE_FAST_MATH" variable in the cmake build
(possibly scanning CFLAGS for -ffast-math/-Ofast
>  to determine the default value) and then adding if(NOT
TEST_SUITE_FAST_MATH) ... endif()
around the benchmark.



I accept Matthias` proposal, and I propose that the above be used for wherever
"really fixing"
the FP problems with a test is beyond a reasonable effort+time level.  This
email is to check
acceptance [or lack thereof] of the above.

Regards,

Abe

----- Original Message -----
> From: "Abe Skolnik" <a.skolnik at samsung.com>
> To: "Matthias Braun" <mbraun at apple.com>
> Cc: "Hal Finkel" <hfinkel at anl.gov>, "Renato
Golin" <renato.golin at linaro.org>, "llvm-dev"
<llvm-dev at lists.llvm.org>
> Sent: Wednesday, October 12, 2016 11:00:39 AM
> Subject: Matthias` suggestion for "test-suite" tests that are
broken at "-Ofast" and are difficult to "repair"
> 
> On 10/11/2016 at 4:15 PM, Matthias Braun wrote:
> 
> > I don't find it surprising that some applications do not work
> > properly with -ffast-math and I think we
> > have to accept that fact.  I think it is valid to skip those tests
> > in the test-suite when a fast math
> > flag combination is used (after making sure there is no easy way to
> > make the test more robust).
> 
> > I would add a "TEST_SUITE_FAST_MATH" variable in the cmake
build
> > (possibly scanning CFLAGS for -ffast-math/-Ofast
> >  to determine the default value) and then adding if(NOT
> >  TEST_SUITE_FAST_MATH) ... endif()
> around the benchmark.
> 
> 
> 
> I accept Matthias` proposal, and I propose that the above be used for
> wherever "really fixing"
> the FP problems with a test is beyond a reasonable effort+time level.
>  This email is to check
> acceptance [or lack thereof] of the above.
This makes sense to me. I think we should set this up so that the tests are
enabled by default, and then we specifically disable those tests for which
we've determined that is the best practical solution.

 -Hal
>
> Regards,
> 
> Abe
> 
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

On 12 October 2016 at 17:09, Hal Finkel <hfinkel at anl.gov>
wrote:
>> I accept Matthias` proposal, and I propose that the above be used for
>> wherever "really fixing"
>> the FP problems with a test is beyond a reasonable effort+time level.
>>  This email is to check
>> acceptance [or lack thereof] of the above.
>
> This makes sense to me. I think we should set this up so that the tests are
enabled by default, and then we specifically disable those tests for which
we've determined that is the best practical solution.
With -ffast-math, all bets are off, and it makes no sense to check
against good values, I agree.

But the benchmarking mode could be run with -ffast-math, and you'd
still want to make sure it works to some level. I mean, we run SPEC
with -ffast-math and it's not that bad, so it should be doable.

But from a test point of view, what sense does it make to run with
-ffast-math if you'll discard all "problematic" FP tests? Why run
the
tests at all in that config if they're not meaningful?

I'm not against marking them off for fast-math, I'm just honestly
trying to understand what the point of it is... simply put: "what is
the meaning of saying all integer and some non-previously-failing FP
tests `pass` with -ffast-math"?

cheers,
--renato