Hal Finkel via llvm-dev
2016-Sep-29 21:55 UTC
[llvm-dev] [cfe-dev] a proposed script to help with test-suite programs that output _lots_ of FP numbers
----- Original Message -----> From: "Sebastian Pop" <sebpop.llvm at gmail.com> > To: "Hal Finkel" <hfinkel at anl.gov> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>, "Sebastian Paul Pop" <s.pop at samsung.com>, "Abe Skolnik" > <a.skolnik at samsung.com>, "cfe-dev" <cfe-dev at lists.llvm.org>, "Renato Golin" <renato.golin at linaro.org> > Sent: Thursday, September 29, 2016 4:41:23 PM > Subject: Re: [cfe-dev] [llvm-dev] a proposed script to help with test-suite programs that output _lots_ of FP numbers > > On Thu, Sep 29, 2016 at 4:32 PM, Hal Finkel <hfinkel at anl.gov> wrote: > > We don't want to lose the more-stringent test coverage just because > > that no longer might be the default mode. We'll also want, where > > practical, some looser test mode that will work regardless of the > > FP contraction setting. Both are important. > > What about recording two golden files with fp-cotract=on and off, > then > diff the output against both, and if none is similar enough error. > I see two problems with recording the output of fp-contract=on: > - based on the target there may be more or less contractions done, > - and also the results may vary over time as the compiler gets better > at folding.Recording the output with -ffp-contract=on to use as a reference output does not seem useful, exactly for the reasons you specify. We should generate the reference outputs with -ffp-contract=off as we do now. That's the only good target-independent configuration. Maybe we could do this: - Have a reference output with -ffp-contract=off - Run program compiled with -ffp-contract=off and the default - Compare the former against the reference output for validation - Compare the latter (the default build) against the former with fpcmp with some tolerance This also has the advantage that we still only need the hashed reference outputs in the repository. -Hal>-- Hal Finkel Lead, Compiler Technology and Programming Languages Leadership Computing Facility Argonne National Laboratory
Abe Skolnik via llvm-dev
2016-Sep-29 22:01 UTC
[llvm-dev] [cfe-dev] a proposed script to help with test-suite programs that output _lots_ of FP numbers
["Hal Finkel" <hfinkel at anl.gov> wrote:]> Recording the output with -ffp-contract=on to use as a reference output does not seem useful, > exactly for the reasons you specify. We should generate the reference outputs with -ffp-contract=off as we do now. > That's the only good target-independent configuration. Maybe we could do this:> - Have a reference output with -ffp-contract=off > - Run program compiled with -ffp-contract=off and the default > - Compare the former against the reference output for validation > - Compare the latter (the default build) against the former with fpcmp with some tolerance> This also has the advantage that we still only need the hashed reference outputs in the repository.Ah. Interesting idea. So please tell me if I have understood you correctly. I think you [Hal] are suggesting something like this: 1) compile the program with FP fusion off, run program, capture output and save it, hash it and compare against reference hash. 2) if comparison against reference hash says "not equal", fail the test and stop [stop testing this particular subtest, that is] 3) compile the program with FP fusion on/"fast", capture the output, compare using "fpcmp" and some positive tolerance against the output of the non-fusion build of the same source code; fail only if outside the tolerance limit[s] Is that right? Regards, Abe
Hal Finkel via llvm-dev
2016-Sep-29 22:03 UTC
[llvm-dev] [cfe-dev] a proposed script to help with test-suite programs that output _lots_ of FP numbers
----- Original Message -----> From: "Abe Skolnik" <a.skolnik at samsung.com> > To: "Hal Finkel" <hfinkel at anl.gov>, "Sebastian Pop" <sebpop.llvm at gmail.com> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>, "Sebastian Paul Pop" <s.pop at samsung.com>, "cfe-dev" > <cfe-dev at lists.llvm.org>, "Renato Golin" <renato.golin at linaro.org> > Sent: Thursday, September 29, 2016 5:01:10 PM > Subject: Re: [cfe-dev] [llvm-dev] a proposed script to help with test-suite programs that output _lots_ of FP numbers > > ["Hal Finkel" <hfinkel at anl.gov> wrote:] > > > Recording the output with -ffp-contract=on to use as a reference > > output does not seem useful, > > exactly for the reasons you specify. We should generate the > > reference outputs with -ffp-contract=off as we do now. > > That's the only good target-independent configuration. Maybe we > > could do this: > > > - Have a reference output with -ffp-contract=off > > - Run program compiled with -ffp-contract=off and the default > > - Compare the former against the reference output for validation > > - Compare the latter (the default build) against the former with > > fpcmp with some tolerance > > > This also has the advantage that we still only need the hashed > > reference outputs in the repository. > > Ah. Interesting idea. So please tell me if I have understood you > correctly. I think you > [Hal] are suggesting something like this: > > 1) compile the program with FP fusion off, run program, capture > output and save it, hash it > and compare against reference hash. > > 2) if comparison against reference hash says "not equal", fail the > test and stop [stop > testing this particular subtest, that is] > > 3) compile the program with FP fusion on/"fast", capture the > output, > compare using "fpcmp" and some positive tolerance against the > output of the non-fusion > build of the same source code; > fail only if outside the tolerance limit[s] > > > Is that right?Correct. Thanks again, Hal> > Regards, > > Abe > >-- Hal Finkel Lead, Compiler Technology and Programming Languages Leadership Computing Facility Argonne National Laboratory
Matthias Braun via llvm-dev
2016-Sep-29 22:05 UTC
[llvm-dev] [cfe-dev] a proposed script to help with test-suite programs that output _lots_ of FP numbers
My (limited) understanding so far was that -fp-contract=on does not give you deterministic results because depending on optimization levels (and architecture support) you could get more or less multiply-add instructions, so recording a fp-contract=on reference file seems pointless to me... - Matthias> On Sep 29, 2016, at 3:01 PM, Abe Skolnik via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > ["Hal Finkel" <hfinkel at anl.gov> wrote:] > >> Recording the output with -ffp-contract=on to use as a reference output does not seem useful, >> exactly for the reasons you specify. We should generate the reference outputs with -ffp-contract=off as we do now. >> That's the only good target-independent configuration. Maybe we could do this: > >> - Have a reference output with -ffp-contract=off >> - Run program compiled with -ffp-contract=off and the default >> - Compare the former against the reference output for validation >> - Compare the latter (the default build) against the former with fpcmp with some tolerance > >> This also has the advantage that we still only need the hashed reference outputs in the repository. > > Ah. Interesting idea. So please tell me if I have understood you correctly. I think you [Hal] are suggesting something like this: > > 1) compile the program with FP fusion off, run program, capture output and save it, hash it and compare against reference hash. > > 2) if comparison against reference hash says "not equal", fail the test and stop [stop testing this particular subtest, that is] > > 3) compile the program with FP fusion on/"fast", capture the output, > compare using "fpcmp" and some positive tolerance against the output of the non-fusion build of the same source code; > fail only if outside the tolerance limit[s] > > > Is that right? > > Regards, > > Abe > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev