Hal Finkel via llvm-dev
2016-Sep-29 23:35 UTC
[llvm-dev] [cfe-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs and more-optimized builds that may have different outputs due to FP optimizations
----- Original Message -----> From: "Matthias Braun via cfe-dev" <cfe-dev at lists.llvm.org> > To: "Abe Skolnik" <a.skolnik at samsung.com> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>, "cfe-dev" <cfe-dev at lists.llvm.org> > Sent: Thursday, September 29, 2016 6:20:09 PM > Subject: Re: [cfe-dev] [llvm-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs > and more-optimized builds that may have different outputs due to FP optimizations > > > > On Sep 29, 2016, at 3:59 PM, Abe Skolnik <a.skolnik at samsung.com> > > wrote: > > > > Dear all, > > > > I would like some help, please, with implementing Hal`s excellent > > suggestion, which I have reworded as below. Hal has confirmed a > > previous version of my rewording as a correct interpretation. [I > > made minor changes since then, e.g. for grammar.] > > > > [Abe wrote:] > > > >>> I think you [Hal] are suggesting something like this: > > > >>> 1) compile the program with FP fusion off, > >>> run the program, capture the output and save it, > >>> hash it and compare it against the reference hash. > > > >>> 2) if comparison against the reference hash says "not equal", > >>> fail the test and stop [i.e. stop testing this particular > >>> subtest] > > > >>> 3) compile the program with FP fusion on/"fast", capture the > >>> output, > >>> compare it using "fpcmp" and some positive tolerance against > >>> the > >>> output of the non-fusion build of the same source code; > >>> fail only if outside the tolerance limit[s] > > Currently the test-suite works by first building the executable and > then running them on a set of inputs. Having a multi-step thingy > does not fit that. > > Having said that you could possibly just build two variants first and > use different comparison steps for each. That at least fits the > model but is still a precedent and requires some deeper test-suite > buildsystem hacking.I think makes sense. We can have the fp-contract-off target and the fp-contract-default target, and use set_target_properties to add -ffp-contract=off to the compile_flags for the former. Then we just need to add the output from the default target as a generated file, and add a dependency on it to the fp-contract-default comparison step. Does that sound right? Are the buildbots still using the makefile-based system, or are they all on the cmake-based system now? -Hal> > - Matthias > _______________________________________________ > cfe-dev mailing list > cfe-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev >-- Hal Finkel Lead, Compiler Technology and Programming Languages Leadership Computing Facility Argonne National Laboratory
Matthias Braun via llvm-dev
2016-Sep-29 23:42 UTC
[llvm-dev] [cfe-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs and more-optimized builds that may have different outputs due to FP optimizations
> On Sep 29, 2016, at 4:35 PM, Hal Finkel <hfinkel at anl.gov> wrote: > > ----- Original Message ----- >> From: "Matthias Braun via cfe-dev" <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> >> To: "Abe Skolnik" <a.skolnik at samsung.com <mailto:a.skolnik at samsung.com>> >> Cc: "llvm-dev" <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>, "cfe-dev" <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> >> Sent: Thursday, September 29, 2016 6:20:09 PM >> Subject: Re: [cfe-dev] [llvm-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs >> and more-optimized builds that may have different outputs due to FP optimizations >> >> >>> On Sep 29, 2016, at 3:59 PM, Abe Skolnik <a.skolnik at samsung.com> >>> wrote: >>> >>> Dear all, >>> >>> I would like some help, please, with implementing Hal`s excellent >>> suggestion, which I have reworded as below. Hal has confirmed a >>> previous version of my rewording as a correct interpretation. [I >>> made minor changes since then, e.g. for grammar.] >>> >>> [Abe wrote:] >>> >>>>> I think you [Hal] are suggesting something like this: >>> >>>>> 1) compile the program with FP fusion off, >>>>> run the program, capture the output and save it, >>>>> hash it and compare it against the reference hash. >>> >>>>> 2) if comparison against the reference hash says "not equal", >>>>> fail the test and stop [i.e. stop testing this particular >>>>> subtest] >>> >>>>> 3) compile the program with FP fusion on/"fast", capture the >>>>> output, >>>>> compare it using "fpcmp" and some positive tolerance against >>>>> the >>>>> output of the non-fusion build of the same source code; >>>>> fail only if outside the tolerance limit[s] >> >> Currently the test-suite works by first building the executable and >> then running them on a set of inputs. Having a multi-step thingy >> does not fit that. >> >> Having said that you could possibly just build two variants first and >> use different comparison steps for each. That at least fits the >> model but is still a precedent and requires some deeper test-suite >> buildsystem hacking. > > I think makes sense. We can have the fp-contract-off target and the fp-contract-default target, and use set_target_properties to add -ffp-contract=off to the compile_flags for the former. Then we just need to add the output from the default target as a generated file, and add a dependency on it to the fp-contract-default comparison step. Does that sound right?in a cmake build that would be the right thing to do. The test-suite typically uses a bunch of wrappers above all that so the cmakefiles look more like the old makefiles did... You can probably just duplicate everything in the current beam/CMakeLists.txt and just set a different name for the "PROG" variable and adjust the CFLAGS as wanted. However that needs some careful testing to make sure those two targets are indeed completely independent and don't override each others file.> > Are the buildbots still using the makefile-based system, or are they all on the cmake-based system now?The scripts in the zorg repository appear to be using "lnt runtest nt" (= makefile system) rather than "lnt runtest test-suite" (=cmake/lit test-suite). So the majority of bots is still on the makefiles I assume :-( - Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160929/d0b3048e/attachment.html>
Hal Finkel via llvm-dev
2016-Oct-03 02:56 UTC
[llvm-dev] [cfe-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs and more-optimized builds that may have different outputs due to FP optimizations
----- Original Message -----> From: "Matthias Braun" <mbraun at apple.com> > To: "Hal Finkel" <hfinkel at anl.gov> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>, "cfe-dev" > <cfe-dev at lists.llvm.org>, "Abe Skolnik" <a.skolnik at samsung.com> > Sent: Thursday, September 29, 2016 6:42:55 PM > Subject: Re: [cfe-dev] [llvm-dev] improving test-suite`s FP subtests > to be able to compare both exact-match outputs and more-optimized > builds that may have different outputs due to FP optimizations> > On Sep 29, 2016, at 4:35 PM, Hal Finkel < hfinkel at anl.gov > wrote: >> > ----- Original Message ----- >> > > From: "Matthias Braun via cfe-dev" < cfe-dev at lists.llvm.org > > > > > > > To: "Abe Skolnik" < a.skolnik at samsung.com > > > > > > > Cc: "llvm-dev" < llvm-dev at lists.llvm.org >, "cfe-dev" < > > > cfe-dev at lists.llvm.org > > > > > > > Sent: Thursday, September 29, 2016 6:20:09 PM > > > > > > Subject: Re: [cfe-dev] [llvm-dev] improving test-suite`s FP > > > subtests > > > to be able to compare both exact-match outputs > > > > > > and more-optimized builds that may have different outputs due to > > > FP > > > optimizations > > >> > > > On Sep 29, 2016, at 3:59 PM, Abe Skolnik < > > > > a.skolnik at samsung.com > > > > > > > > > > > > > > > wrote: > > > > > >> > > > Dear all, > > > > > >> > > > I would like some help, please, with implementing Hal`s > > > > excellent > > > > > > > > > > suggestion, which I have reworded as below. Hal has confirmed a > > > > > > > > > > previous version of my rewording as a correct interpretation. > > > > [I > > > > > > > > > > made minor changes since then, e.g. for grammar.] > > > > > >> > > > [Abe wrote:] > > > > > >> > > > > > I think you [Hal] are suggesting something like this: > > > > > > > > > > > > > > >> > > > > > 1) compile the program with FP fusion off, > > > > > > > > > > > > > > > > > > > > > run the program, capture the output and save it, > > > > > > > > > > > > > > > > > > > > > hash it and compare it against the reference hash. > > > > > > > > > > > > > > >> > > > > > 2) if comparison against the reference hash says "not > > > > > > equal", > > > > > > > > > > > > > > > > > > > > > fail the test and stop [i.e. stop testing this particular > > > > > > > > > > > > > > > > > > > > > subtest] > > > > > > > > > > > > > > >> > > > > > 3) compile the program with FP fusion on/"fast", capture > > > > > > the > > > > > > > > > > > > > > > > > > > > > output, > > > > > > > > > > > > > > > > > > > > > compare it using "fpcmp" and some positive tolerance > > > > > > against > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > output of the non-fusion build of the same source code; > > > > > > > > > > > > > > > > > > > > > fail only if outside the tolerance limit[s] > > > > > > > > > > > > > > >> > > Currently the test-suite works by first building the executable > > > and > > > > > > then running them on a set of inputs. Having a multi-step thingy > > > > > > does not fit that. > > >> > > Having said that you could possibly just build two variants first > > > and > > > > > > use different comparison steps for each. That at least fits the > > > > > > model but is still a precedent and requires some deeper > > > test-suite > > > > > > buildsystem hacking. > > >> > I think makes sense. We can have the fp-contract-off target and the > > fp-contract-default target, and use set_target_properties to add > > -ffp-contract=off to the compile_flags for the former. Then we just > > need to add the output from the default target as a generated file, > > and add a dependency on it to the fp-contract-default comparison > > step. Does that sound right? >> in a cmake build that would be the right thing to do. The test-suite > typically uses a bunch of wrappers above all that so the cmakefiles > look more like the old makefiles did... You can probably just > duplicate everything in the current beam/CMakeLists.txt and just set > a different name for the "PROG" variable and adjust the CFLAGS as > wanted. However that needs some careful testing to make sure those > two targets are indeed completely independent and don't override > each others file.> > Are the buildbots still using the makefile-based system, or are > > they > > all on the cmake-based system now? >> The scripts in the zorg repository appear to be using "lnt runtest > nt" (= makefile system) rather than "lnt runtest test-suite" > (=cmake/lit test-suite). So the majority of bots is still on the > makefiles I assume :-(Is there any reason we should not start transitioning that now? -Hal> - Matthias-- Hal Finkel Lead, Compiler Technology and Programming Languages Leadership Computing Facility Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161002/3fb9c191/attachment.html>