llvm dev - Sep 2016 - [cfe-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs and more-optimized builds that may have different outputs due to FP optimizations

Dear all,

I would like some help, please, with implementing Hal`s excellent suggestion,
which I have
reworded as below.  Hal has confirmed a previous version of my rewording as a
correct
interpretation.  [I made minor changes since then, e.g. for grammar.]

[Abe wrote:]
>> I think you [Hal] are suggesting something like this:
>>    1) compile the program with FP fusion off,
>>       run the program, capture the output and save it,
>>       hash it and compare it against the reference hash.
>>    2) if comparison against the reference hash says "not
equal",
>>       fail the test and stop [i.e. stop testing this particular
subtest]
>>    3) compile the program with FP fusion on/"fast", capture
the output,
>>       compare it using "fpcmp" and some positive tolerance
against the
>>       output of the non-fusion build of the same source code;
>>       fail only if outside the tolerance limit[s]
>> Is that right?
[Hal wrote:]
> Correct.

As of now, I do not understand how to make a single directory and its contained
source code
compile to more than one program and trigger more than one test run.  I think I
understand
Makefiles well enough to make that happen, but I`m pretty sure I _don`t_
understand either {
[1] CMake in general or [2] test-suite`s use of it specifically } well enough to
do this
without any help.  Maybe the "solution" is to have new directories
with symlinks to the
existing source code?

The help in question could be as minimal as "look here" with a Web
link, if the link in
question would guide me towards enlightenment.  ;-)

Regards,

Abe

> On Sep 29, 2016, at 3:59 PM, Abe Skolnik <a.skolnik at samsung.com>
wrote:
> 
> Dear all,
> 
> I would like some help, please, with implementing Hal`s excellent
suggestion, which I have reworded as below.  Hal has confirmed a previous
version of my rewording as a correct interpretation.  [I made minor changes
since then, e.g. for grammar.]
> 
> [Abe wrote:]
> 
>>> I think you [Hal] are suggesting something like this:
> 
>>>   1) compile the program with FP fusion off,
>>>      run the program, capture the output and save it,
>>>      hash it and compare it against the reference hash.
> 
>>>   2) if comparison against the reference hash says "not
equal",
>>>      fail the test and stop [i.e. stop testing this particular
subtest]
> 
>>>   3) compile the program with FP fusion on/"fast",
capture the output,
>>>      compare it using "fpcmp" and some positive tolerance
against the
>>>      output of the non-fusion build of the same source code;
>>>      fail only if outside the tolerance limit[s]
Currently the test-suite works by first building the executable and then running
them on a set of inputs. Having a multi-step thingy does not fit that.

Having said that you could possibly just build two variants first and use
different comparison steps for each. That at least fits the model but is still a
precedent and requires some deeper test-suite buildsystem hacking.

- Matthias

----- Original Message -----
> From: "Matthias Braun via cfe-dev" <cfe-dev at
lists.llvm.org>
> To: "Abe Skolnik" <a.skolnik at samsung.com>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>,
"cfe-dev" <cfe-dev at lists.llvm.org>
> Sent: Thursday, September 29, 2016 6:20:09 PM
> Subject: Re: [cfe-dev] [llvm-dev] improving test-suite`s FP subtests to be
able to compare both exact-match outputs
> and more-optimized builds that may have different outputs due to FP
optimizations
> 
> 
> > On Sep 29, 2016, at 3:59 PM, Abe Skolnik <a.skolnik at
samsung.com>
> > wrote:
> > 
> > Dear all,
> > 
> > I would like some help, please, with implementing Hal`s excellent
> > suggestion, which I have reworded as below.  Hal has confirmed a
> > previous version of my rewording as a correct interpretation.  [I
> > made minor changes since then, e.g. for grammar.]
> > 
> > [Abe wrote:]
> > 
> >>> I think you [Hal] are suggesting something like this:
> > 
> >>>   1) compile the program with FP fusion off,
> >>>      run the program, capture the output and save it,
> >>>      hash it and compare it against the reference hash.
> > 
> >>>   2) if comparison against the reference hash says "not
equal",
> >>>      fail the test and stop [i.e. stop testing this particular
> >>>      subtest]
> > 
> >>>   3) compile the program with FP fusion on/"fast",
capture the
> >>>   output,
> >>>      compare it using "fpcmp" and some positive
tolerance against
> >>>      the
> >>>      output of the non-fusion build of the same source code;
> >>>      fail only if outside the tolerance limit[s]
> 
> Currently the test-suite works by first building the executable and
> then running them on a set of inputs. Having a multi-step thingy
> does not fit that.
> 
> Having said that you could possibly just build two variants first and
> use different comparison steps for each. That at least fits the
> model but is still a precedent and requires some deeper test-suite
> buildsystem hacking.
I think makes sense. We can have the fp-contract-off target and the
fp-contract-default target, and use set_target_properties to add
-ffp-contract=off to the compile_flags for the former. Then we just need to add
the output from the default target as a generated file, and add a dependency on
it to the fp-contract-default comparison step. Does that sound right?

Are the buildbots still using the makefile-based system, or are they all on the
cmake-based system now?

 -Hal
> 
> - Matthias
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> 
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory