llvm dev - Dec 2012 - [LLVMdev] [cfe-dev] RFC: Change tests to run with fixed (not-host dependent) triple

I'm ok with this in principle, but how about with the nuance that some tests
(eg test/codegen) explicitly opt into march=native?

-Chris

On Nov 30, 2012, at 12:16 PM, Daniel Dunbar <daniel at zuster.org> wrote:
> Hi all,
> 
> We consistently see test failures arising because by default many of our
tests run in a mode where the tool (clang or llc) pick host-dependent behavior.
This makes it easy for developers to write tests that pass on their system, but
fail for other developers.
> 
> There is some utility in this behavior, as it gives us (unintended) testing
coverage of some things, but overall I think it is a net loss for productivity.
> 
> I propose:
> 
> a. We change the test suite to run in such a way that all tools default to
an "unknown" host triple.
> 
> b. If someone feels there is missing coverage in some area, we add
increased tests for that area (which get run on all platforms).
> 
> c. If there is some reason that running with an "unknown" host
triple is undesirable, I propose that we set the default test triple to be
"x86_64-pc-linux-gnu", and require deviations to be specified.
> 
> Comments?
> 
>  - Daniel
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

My thoughts:

On Sat, Dec 1, 2012 at 9:06 AM, Chandler Carruth <chandlerc at
google.com>wrote:
>  On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <clattner at
apple.com>wrote:
>
>> I'm ok with this in principle, but how about with the nuance that
some
>> tests (eg test/codegen) explicitly opt into march=native?
>>
>
> I'd really like the default behavior to be something that forces the
test
> to either be independent of the targeted triple, or explicitly set a
> target. I like the default being unknown.
>
To state the obvious, there's two different things that go on in tests: the
specific thing being tested and things that aren't being tested but just
need to be done to provide enough "context" for what's being
tested. I was
involved in a long slog trying to fix up a lot of the ARM regression test
failues (using my work email address). Here's some "roughly right"
statistics:

There were probably about 25--30 bugs where the issue was behaviour that
FileCheck regexps didn't account for (mostly due to ABI issues). There the
prevailing opinion seemed to be keep simple FileCheck tests but that tests
should be run with a specific triple; however that triple shouldn't always
be x86_64 (because that's a bit special). There've been about 5-10 tests
where the test was testing something that was architecture specific without
secifying they needed it (eg, testing for specific x86_64 machine
optimizations without doing that); again the upshot has been to have these
requirements specified explicitly. There have been 5-10 JIT code tests
where "support" code pasted from, eg, lli into a test hadn't been
updated
when the JIT core was changed. The one definite bug that was there was in
devirtualisation (which probably lowered ok but failed the module
verifier). This bug was definitely not visible due to the general set of
ARM failures that were basically issues with the tests.

So on the one hand, I'd love it if the tests were constructed in such a way
that the "fuzz" of ABI differences didn't need to be considered.
On the
other hand, if the devirtualization test had been run only using an x86_64
triple the issue wouldn't have come to light as quickly. That seems to me
to be the crux of the problem: LLVM (and especially Clang) is only _mostly_
target independent, and getting the smaller set of target dependent
elements wrong breaks compilation just as much as a generic bug so finding
these things as early as possible seems desirable.

>   I wonder, would the ability to run the entire test suite with all of
> the 'default' triples (that lit sets to unknown in normal runs)
instead set
> to the host, or to a specific triple maybe be a useful extra form of
> checking? This would let both humans and build bots find bugs and
> discrepancies specific to a particular target.
>
Dreaming here: I wonder if one could come up with some set of meta-regexps
that describe the annoying stuff like ABI differences so that if the above
were done, one could try t separate the test failures into "test regreps
written implicitly assuming different system, failures look due to correct
system dependent stuff being generated" and "test is failing on this
system
in a non-understood way". That sounds too tricky to be reliable, but I
don't know...

>   We could even have a common test target that build bots use which runs
> all the tests both in the default, and in the host-triple mode so that we
> force people to converge on target independent tests or explicit triples.
>
A reasonable idea, except I'll reiterate: it assumes there _is_ completely
target independent behaviour in non-trivial test code. If it's the case
that there's not really it might be a situation where biting the bullet and
trying to put a wide ranging set of triples randomly throughout the test
suite and hoping to catch stuff that way is the best idea.
>    --
>
cheers, dave tweed__________________________
high-performance computing and machine vision expert: david.tweed at gmail.com
"while having code so boring anyone can maintain it, use Python." --
attempted insult seen on slashdot
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20121201/35f50df2/attachment.html>

On Sat, Dec 1, 2012 at 1:06 AM, Chandler Carruth <chandlerc at
google.com>wrote:
> On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <clattner at
apple.com> wrote:
>
>> I'm ok with this in principle, but how about with the nuance that
some
>> tests (eg test/codegen) explicitly opt into march=native?
>>
>
> I'd really like the default behavior to be something that forces the
test
> to either be independent of the targeted triple, or explicitly set a
> target. I like the default being unknown.
>
> I wonder, would the ability to run the entire test suite with all of the
> 'default' triples (that lit sets to unknown in normal runs) instead
set to
> the host, or to a specific triple maybe be a useful extra form of checking?
> This would let both humans and build bots find bugs and discrepancies
> specific to a particular target.
>
I'd prefer not to do this universally (i.e., run the whole test suite that
way), but what I do think would be useful is to add enough test suite
support to be able to easily run the same test on multiple triples (or even
all configured ones).

My primary goal is to have the tests that individual developers be
equivalent (independent of the target they are running on).

 - Daniel

We could even have a common test target that build bots use which runs
all
> the tests both in the default, and in the host-triple mode so that we force
> people to converge on target independent tests or explicit triples.
>
>
>>
>> -Chris
>>
>> On Nov 30, 2012, at 12:16 PM, Daniel Dunbar <daniel at
zuster.org> wrote:
>>
>> > Hi all,
>> >
>> > We consistently see test failures arising because by default many
of
>> our tests run in a mode where the tool (clang or llc) pick
host-dependent
>> behavior. This makes it easy for developers to write tests that pass on
>> their system, but fail for other developers.
>> >
>> > There is some utility in this behavior, as it gives us
(unintended)
>> testing coverage of some things, but overall I think it is a net loss
for
>> productivity.
>> >
>> > I propose:
>> >
>> > a. We change the test suite to run in such a way that all tools
default
>> to an "unknown" host triple.
>> >
>> > b. If someone feels there is missing coverage in some area, we add
>> increased tests for that area (which get run on all platforms).
>> >
>> > c. If there is some reason that running with an
"unknown" host triple
>> is undesirable, I propose that we set the default test triple to be
>> "x86_64-pc-linux-gnu", and require deviations to be
specified.
>> >
>> > Comments?
>> >
>> >  - Daniel
>> >
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20121203/886f8041/attachment.html>