llvm dev - Sep 2016 - [cfe-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs and more-optimized builds that may have different outputs due to FP optimizations

----- Original Message -----
> From: "Matthias Braun via cfe-dev" <cfe-dev at
lists.llvm.org>
> To: "Abe Skolnik" <a.skolnik at samsung.com>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>,
"cfe-dev" <cfe-dev at lists.llvm.org>
> Sent: Thursday, September 29, 2016 6:20:09 PM
> Subject: Re: [cfe-dev] [llvm-dev] improving test-suite`s FP subtests to be
able to compare both exact-match outputs
> and more-optimized builds that may have different outputs due to FP
optimizations
> 
> 
> > On Sep 29, 2016, at 3:59 PM, Abe Skolnik <a.skolnik at
samsung.com>
> > wrote:
> > 
> > Dear all,
> > 
> > I would like some help, please, with implementing Hal`s excellent
> > suggestion, which I have reworded as below.  Hal has confirmed a
> > previous version of my rewording as a correct interpretation.  [I
> > made minor changes since then, e.g. for grammar.]
> > 
> > [Abe wrote:]
> > 
> >>> I think you [Hal] are suggesting something like this:
> > 
> >>>   1) compile the program with FP fusion off,
> >>>      run the program, capture the output and save it,
> >>>      hash it and compare it against the reference hash.
> > 
> >>>   2) if comparison against the reference hash says "not
equal",
> >>>      fail the test and stop [i.e. stop testing this particular
> >>>      subtest]
> > 
> >>>   3) compile the program with FP fusion on/"fast",
capture the
> >>>   output,
> >>>      compare it using "fpcmp" and some positive
tolerance against
> >>>      the
> >>>      output of the non-fusion build of the same source code;
> >>>      fail only if outside the tolerance limit[s]
> 
> Currently the test-suite works by first building the executable and
> then running them on a set of inputs. Having a multi-step thingy
> does not fit that.
> 
> Having said that you could possibly just build two variants first and
> use different comparison steps for each. That at least fits the
> model but is still a precedent and requires some deeper test-suite
> buildsystem hacking.
I think makes sense. We can have the fp-contract-off target and the
fp-contract-default target, and use set_target_properties to add
-ffp-contract=off to the compile_flags for the former. Then we just need to add
the output from the default target as a generated file, and add a dependency on
it to the fp-contract-default comparison step. Does that sound right?

Are the buildbots still using the makefile-based system, or are they all on the
cmake-based system now?

 -Hal
> 
> - Matthias
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> 
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

> On Sep 29, 2016, at 4:35 PM, Hal Finkel <hfinkel at anl.gov> wrote:
> 
> ----- Original Message -----
>> From: "Matthias Braun via cfe-dev" <cfe-dev at
lists.llvm.org <mailto:cfe-dev at lists.llvm.org>>
>> To: "Abe Skolnik" <a.skolnik at samsung.com
<mailto:a.skolnik at samsung.com>>
>> Cc: "llvm-dev" <llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org>>, "cfe-dev" <cfe-dev
at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>>
>> Sent: Thursday, September 29, 2016 6:20:09 PM
>> Subject: Re: [cfe-dev] [llvm-dev] improving test-suite`s FP subtests to
be able to compare both exact-match outputs
>> and more-optimized builds that may have different outputs due to FP
optimizations
>> 
>> 
>>> On Sep 29, 2016, at 3:59 PM, Abe Skolnik <a.skolnik at
samsung.com>
>>> wrote:
>>> 
>>> Dear all,
>>> 
>>> I would like some help, please, with implementing Hal`s excellent
>>> suggestion, which I have reworded as below.  Hal has confirmed a
>>> previous version of my rewording as a correct interpretation.  [I
>>> made minor changes since then, e.g. for grammar.]
>>> 
>>> [Abe wrote:]
>>> 
>>>>> I think you [Hal] are suggesting something like this:
>>> 
>>>>>  1) compile the program with FP fusion off,
>>>>>     run the program, capture the output and save it,
>>>>>     hash it and compare it against the reference hash.
>>> 
>>>>>  2) if comparison against the reference hash says "not
equal",
>>>>>     fail the test and stop [i.e. stop testing this
particular
>>>>>     subtest]
>>> 
>>>>>  3) compile the program with FP fusion on/"fast",
capture the
>>>>>  output,
>>>>>     compare it using "fpcmp" and some positive
tolerance against
>>>>>     the
>>>>>     output of the non-fusion build of the same source code;
>>>>>     fail only if outside the tolerance limit[s]
>> 
>> Currently the test-suite works by first building the executable and
>> then running them on a set of inputs. Having a multi-step thingy
>> does not fit that.
>> 
>> Having said that you could possibly just build two variants first and
>> use different comparison steps for each. That at least fits the
>> model but is still a precedent and requires some deeper test-suite
>> buildsystem hacking.
> 
> I think makes sense. We can have the fp-contract-off target and the
fp-contract-default target, and use set_target_properties to add
-ffp-contract=off to the compile_flags for the former. Then we just need to add
the output from the default target as a generated file, and add a dependency on
it to the fp-contract-default comparison step. Does that sound right?
in a cmake build that would be the right thing to do. The test-suite typically
uses a bunch of wrappers above all that so the cmakefiles look more like the old
makefiles did... You can probably just duplicate everything in the current
beam/CMakeLists.txt and just set a different name for the "PROG"
variable and adjust the CFLAGS as wanted. However that needs some careful
testing to make sure those two targets are indeed completely independent and
don't override each others file.
> 
> Are the buildbots still using the makefile-based system, or are they all on
the cmake-based system now?
The scripts in the zorg repository appear to be using "lnt runtest nt"
(= makefile system) rather than "lnt runtest test-suite" (=cmake/lit
test-suite). So the majority of bots is still on the makefiles I assume :-(

- Matthias
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160929/d0b3048e/attachment.html>

----- Original Message -----
> From: "Matthias Braun" <mbraun at apple.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>,
"cfe-dev"
> <cfe-dev at lists.llvm.org>, "Abe Skolnik" <a.skolnik at
samsung.com>
> Sent: Thursday, September 29, 2016 6:42:55 PM
> Subject: Re: [cfe-dev] [llvm-dev] improving test-suite`s FP subtests
> to be able to compare both exact-match outputs and more-optimized
> builds that may have different outputs due to FP optimizations
> > On Sep 29, 2016, at 4:35 PM, Hal Finkel < hfinkel at anl.gov >
wrote:
> 
> > ----- Original Message -----
> 
> > > From: "Matthias Braun via cfe-dev" < cfe-dev at
lists.llvm.org >
> > 
> 
> > > To: "Abe Skolnik" < a.skolnik at samsung.com >
> > 
> 
> > > Cc: "llvm-dev" < llvm-dev at lists.llvm.org >,
"cfe-dev" <
> > > cfe-dev at lists.llvm.org >
> > 
> 
> > > Sent: Thursday, September 29, 2016 6:20:09 PM
> > 
> 
> > > Subject: Re: [cfe-dev] [llvm-dev] improving test-suite`s FP
> > > subtests
> > > to be able to compare both exact-match outputs
> > 
> 
> > > and more-optimized builds that may have different outputs due to
> > > FP
> > > optimizations
> > 
> 
> > > > On Sep 29, 2016, at 3:59 PM, Abe Skolnik <
> > > > a.skolnik at samsung.com
> > > > >
> > > 
> > 
> 
> > > > wrote:
> > > 
> > 
> 
> > > > Dear all,
> > > 
> > 
> 
> > > > I would like some help, please, with implementing Hal`s
> > > > excellent
> > > 
> > 
> 
> > > > suggestion, which I have reworded as below. Hal has
confirmed a
> > > 
> > 
> 
> > > > previous version of my rewording as a correct
interpretation.
> > > > [I
> > > 
> > 
> 
> > > > made minor changes since then, e.g. for grammar.]
> > > 
> > 
> 
> > > > [Abe wrote:]
> > > 
> > 
> 
> > > > > > I think you [Hal] are suggesting something like
this:
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > 1) compile the program with FP fusion off,
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > run the program, capture the output and save it,
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > hash it and compare it against the reference hash.
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > 2) if comparison against the reference hash says
"not
> > > > > > equal",
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > fail the test and stop [i.e. stop testing this
particular
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > subtest]
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > 3) compile the program with FP fusion
on/"fast", capture
> > > > > > the
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > output,
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > compare it using "fpcmp" and some
positive tolerance
> > > > > > against
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > the
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > output of the non-fusion build of the same source
code;
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > fail only if outside the tolerance limit[s]
> > > > > 
> > > > 
> > > 
> > 
> 
> > > Currently the test-suite works by first building the executable
> > > and
> > 
> 
> > > then running them on a set of inputs. Having a multi-step thingy
> > 
> 
> > > does not fit that.
> > 
> 
> > > Having said that you could possibly just build two variants first
> > > and
> > 
> 
> > > use different comparison steps for each. That at least fits the
> > 
> 
> > > model but is still a precedent and requires some deeper
> > > test-suite
> > 
> 
> > > buildsystem hacking.
> > 
> 
> > I think makes sense. We can have the fp-contract-off target and the
> > fp-contract-default target, and use set_target_properties to add
> > -ffp-contract=off to the compile_flags for the former. Then we just
> > need to add the output from the default target as a generated file,
> > and add a dependency on it to the fp-contract-default comparison
> > step. Does that sound right?
> 
> in a cmake build that would be the right thing to do. The test-suite
> typically uses a bunch of wrappers above all that so the cmakefiles
> look more like the old makefiles did... You can probably just
> duplicate everything in the current beam/CMakeLists.txt and just set
> a different name for the "PROG" variable and adjust the CFLAGS as
> wanted. However that needs some careful testing to make sure those
> two targets are indeed completely independent and don't override
> each others file.
> > Are the buildbots still using the makefile-based system, or are
> > they
> > all on the cmake-based system now?
> 
> The scripts in the zorg repository appear to be using "lnt runtest
> nt" (= makefile system) rather than "lnt runtest test-suite"
> (=cmake/lit test-suite). So the majority of bots is still on the
> makefiles I assume :-(
Is there any reason we should not start transitioning that now? 

-Hal 
> - Matthias
-- 

Hal Finkel 
Lead, Compiler Technology and Programming Languages 
Leadership Computing Facility 
Argonne National Laboratory 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161002/3fb9c191/attachment.html>