llvm dev - Sep 2016 - [cfe-dev] a proposed script to help with test-suite programs that output _lots_ of FP numbers

----- Original Message -----
> From: "Sebastian Pop via cfe-dev" <cfe-dev at
lists.llvm.org>
> To: "Renato Golin" <renato.golin at linaro.org>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>,
"Sebastian Paul Pop" <s.pop at samsung.com>, "Abe
Skolnik"
> <a.skolnik at samsung.com>, "cfe-dev" <cfe-dev at
lists.llvm.org>
> Sent: Thursday, September 29, 2016 4:26:23 PM
> Subject: Re: [cfe-dev] [llvm-dev] a proposed script to help with test-suite
programs that output _lots_ of FP numbers
> 
> On Thu, Sep 29, 2016 at 2:05 PM, Renato Golin
> <renato.golin at linaro.org> wrote:
> > On 29 September 2016 at 19:21, Sebastian Pop
> > <sebpop.llvm at gmail.com> wrote:
> >> Cumulating errors is a bad idea.
> >> As others have suggested, please prepare a patch that disables
> >> fp-contract on those testcases.
> >
> > No, please, let's not disable things just because they fail.
> >
> > If the test is not meaningful or if the results are not good,
let's
> > just change the test in a meaningful way that can work with any FP
> > optimisation without changing meaning.
> >
> > If it does change meaning, it's a bug and we *want* to catch.
> 
> These tests are checking the results against a "golden file"
output
> computed with fp-contract=off.
> IMHO the most sensible solution is to continue checking those tests
> with the same flag
> as at the time when the reference output has been recorded.
We don't want to lose the more-stringent test coverage just because that no
longer might be the default mode. We'll also want, where practical, some
looser test mode that will work regardless of the FP contraction setting. Both
are important.

 -Hal
> 
> Thanks,
> Sebastian
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> 
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

On Thu, Sep 29, 2016 at 4:32 PM, Hal Finkel <hfinkel at anl.gov>
wrote:
> We don't want to lose the more-stringent test coverage just because
that no longer might be the default mode. We'll also want, where practical,
some looser test mode that will work regardless of the FP contraction setting.
Both are important.
What about recording two golden files with fp-cotract=on and off, then
diff the output against both, and if none is similar enough error.
I see two problems with recording the output of fp-contract=on:
- based on the target there may be more or less contractions done,
- and also the results may vary over time as the compiler gets better
at folding.

----- Original Message -----
> From: "Sebastian Pop" <sebpop.llvm at gmail.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>,
"Sebastian Paul Pop" <s.pop at samsung.com>, "Abe
Skolnik"
> <a.skolnik at samsung.com>, "cfe-dev" <cfe-dev at
lists.llvm.org>, "Renato Golin" <renato.golin at linaro.org>
> Sent: Thursday, September 29, 2016 4:41:23 PM
> Subject: Re: [cfe-dev] [llvm-dev] a proposed script to help with test-suite
programs that output _lots_ of FP numbers
> 
> On Thu, Sep 29, 2016 at 4:32 PM, Hal Finkel <hfinkel at anl.gov>
wrote:
> > We don't want to lose the more-stringent test coverage just
because
> > that no longer might be the default mode. We'll also want, where
> > practical, some looser test mode that will work regardless of the
> > FP contraction setting. Both are important.
> 
> What about recording two golden files with fp-cotract=on and off,
> then
> diff the output against both, and if none is similar enough error.
> I see two problems with recording the output of fp-contract=on:
> - based on the target there may be more or less contractions done,
> - and also the results may vary over time as the compiler gets better
> at folding.
Recording the output with -ffp-contract=on to use as a reference output does not
seem useful, exactly for the reasons you specify. We should generate the
reference outputs with -ffp-contract=off as we do now. That's the only good
target-independent configuration. Maybe we could do this:

 - Have a reference output with -ffp-contract=off
 - Run program compiled with -ffp-contract=off and the default
 - Compare the former against the reference output for validation
 - Compare the latter (the default build) against the former with fpcmp with
some tolerance

This also has the advantage that we still only need the hashed reference outputs
in the repository.

 -Hal
> 
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

[On Thu, Sep 29, 2016 at 4:32 PM, Hal Finkel <hfinkel at anl.gov>
wrote:]
> We don't want to lose the more-stringent test coverage just because
that no longer might be the default mode.
>  We'll also want, where practical, some looser test mode that will work
regardless of the FP
contraction setting.
>  Both are important.
[On 09/29/2016 04:41 PM, Sebastian Pop wrote:]
> What about recording two golden files with fp-contract=on and off,
> then diff the output against both, and if none is similar enough error.
> I see two problems with recording the output of fp-contract=on:
> - based on the target there may be more or less contractions done,
> - and also the results may vary over time as the compiler gets better
> at folding.
Yes, separate "golden files" with FP contraction on [or
"fast"] is problematic.

However, if I/we can get compression working, then in some sense we can
"have our cake and eat
it too": have small reference files in the repository and in downloads, and
have any
precision/tolerance we like [including requiring exact matching] at test time.

If all the big outputs compress very nicely, as does the one from 
"MultiSource/Benchmarks/VersaBench/beamformer", then we can probably
do this across all tests
the output of which depends on FP computations.

-- Abe

On 29 September 2016 at 22:32, Hal Finkel <hfinkel at anl.gov>
wrote:
> We don't want to lose the more-stringent test coverage just because
that no longer might be the default mode. We'll also want, where practical,
some looser test mode that will work regardless of the FP contraction setting.
Both are important.
That's why Abe proposed running both modes. I think this is the only
long term solution.

On Thu, Sep 29, 2016 at 5:04 PM, Renato Golin <renato.golin at linaro.org>
wrote:
> On 29 September 2016 at 22:32, Hal Finkel <hfinkel at anl.gov> wrote:
>> We don't want to lose the more-stringent test coverage just because
that no longer might be the default mode. We'll also want, where practical,
some looser test mode that will work regardless of the FP contraction setting.
Both are important.
>
> That's why Abe proposed running both modes. I think this is the only
> long term solution.
So we do agree that currently the testsuite runs only one of the
modes, that is fp-contract=off.
The test-suite does not run the other mode: fp-contract=on, that is a
new feature, i.e., more testing.

If we agree on these two points, then let's move forward, commit the
patch that Abe has just posted to add CFLAGS += -ffp-contract=off
and then in a second stage add an extra step for the new mode fp-contract=on.

Thanks,
Sebastian