llvm dev - Oct 2016 - [test-suite] making polybench/symm succeed with "-Ofast" and "-ffp-contract=on"

On 12 October 2016 at 15:05, Hal Finkel <hfinkel at anl.gov>
wrote:
> This is something we need to understand. No, there's not always an
error bar. With FMA formation and without non-IEEE-compliant optimizations (i.e.
fast-math), the optimized answer should be identical to the non-optimized
answer.
What about architectures that this is never respected, like Darwin?

In the general case, indeed, optimisation levels should not change the
IEEE representation and the tests should be deterministic.

But we can't guarantee this will always be the case.

> We still do see cross-system discrepancies sometimes because of differences
in denormal handling, but on the same system that should be consistent (aside,
perhaps, from compiler-level constant-folding issues).
But the test-suite doesn't run on a single system, nor it has one
reference_output for each system.

cheers,
--renato

----- Original Message -----
> From: "Renato Golin" <renato.golin at linaro.org>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "Sebastian Paul Pop" <s.pop at samsung.com>,
"llvm-dev" <llvm-dev at lists.llvm.org>, "Matthias
Braun"
> <matze at braunis.de>, "Clang Dev" <cfe-dev at
lists.llvm.org>, "nd" <nd at arm.com>, "Abe
Skolnik" <a.skolnik at samsung.com>,
> "Sebastian Pop" <sebpop.llvm at gmail.com>
> Sent: Wednesday, October 12, 2016 9:16:39 AM
> Subject: Re: [test-suite] making polybench/symm succeed with
"-Ofast" and "-ffp-contract=on"
> 
> On 12 October 2016 at 15:05, Hal Finkel <hfinkel at anl.gov> wrote:
> > This is something we need to understand. No, there's not always an
> > error bar. With FMA formation and without non-IEEE-compliant
> > optimizations (i.e. fast-math), the optimized answer should be
> > identical to the non-optimized answer.
> 
> What about architectures that this is never respected, like Darwin?
> 
> In the general case, indeed, optimisation levels should not change
> the
> IEEE representation and the tests should be deterministic.
> 
> But we can't guarantee this will always be the case.
> 
> 
> > We still do see cross-system discrepancies sometimes because of
> > differences in denormal handling, but on the same system that
> > should be consistent (aside, perhaps, from compiler-level
> > constant-folding issues).
> 
> But the test-suite doesn't run on a single system, nor it has one
> reference_output for each system.
I agree and understand, and we may need a tolerance in practice to deal with
differences from denormal handling, etc. However, if Sebastian is seeing
differences on the same system, we should understand why. Is he running on an
ARM Darwin system, or an x86 using fp80 arithmetic, etc.? (Senastian?)

 -Hal
> 
> cheers,
> --renato
> 
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

On 12 October 2016 at 15:28, Hal Finkel <hfinkel at anl.gov>
wrote:
> I agree and understand, and we may need a tolerance in practice to deal
with differences from denormal handling, etc. However, if Sebastian is seeing
differences on the same system, we should understand why. Is he running on an
ARM Darwin system, or an x86 using fp80 arithmetic, etc.? (Senastian?)
Oh, I didn't mean that. But since we will run the same tests on Darwin
and x86-fp80, the results need to take that into account from the
beginning.

Though, I recall Sebastian was using -Ofast, which does turn
-ffast-math, so the same idea applies, no?

cheers,
--renato

On Wed, Oct 12, 2016 at 10:28 AM, Hal Finkel <hfinkel at anl.gov>
wrote:
> ----- Original Message -----
>> From: "Renato Golin" <renato.golin at linaro.org>
>> To: "Hal Finkel" <hfinkel at anl.gov>
>> Cc: "Sebastian Paul Pop" <s.pop at samsung.com>,
"llvm-dev" <llvm-dev at lists.llvm.org>, "Matthias
Braun"
>> <matze at braunis.de>, "Clang Dev" <cfe-dev at
lists.llvm.org>, "nd" <nd at arm.com>, "Abe
Skolnik" <a.skolnik at samsung.com>,
>> "Sebastian Pop" <sebpop.llvm at gmail.com>
>> Sent: Wednesday, October 12, 2016 9:16:39 AM
>> Subject: Re: [test-suite] making polybench/symm succeed with
"-Ofast" and "-ffp-contract=on"
>>
>> On 12 October 2016 at 15:05, Hal Finkel <hfinkel at anl.gov>
wrote:
>> > This is something we need to understand. No, there's not
always an
>> > error bar. With FMA formation and without non-IEEE-compliant
>> > optimizations (i.e. fast-math), the optimized answer should be
>> > identical to the non-optimized answer.
>>
>> What about architectures that this is never respected, like Darwin?
>>
>> In the general case, indeed, optimisation levels should not change
>> the
>> IEEE representation and the tests should be deterministic.
>>
>> But we can't guarantee this will always be the case.
>>
>>
>> > We still do see cross-system discrepancies sometimes because of
>> > differences in denormal handling, but on the same system that
>> > should be consistent (aside, perhaps, from compiler-level
>> > constant-folding issues).
>>
>> But the test-suite doesn't run on a single system, nor it has one
>> reference_output for each system.
>
> I agree and understand, and we may need a tolerance in practice to deal
with differences from denormal handling, etc. However, if Sebastian is seeing
differences on the same system, we should understand why. Is he running on an
ARM Darwin system, or an x86 using fp80 arithmetic,
My dev machine is an x86_64-linux.  This is where I ran all my reported results.
How do I determine whether I am using fp80 arithmetic?

Thanks,
Sebastian