[LLVMdev] [cfe-dev] RFC: Change tests to run with fixed (not-host dependent) triple

Daniel Dunbar
Mon, 03 Dec 2012 19:51:32 -0000

On Mon, Dec 3, 2012 at 10:56 AM, David Blaikie <dblaikie at gmail.com>
wrote:

> On Mon, Dec 3, 2012 at 10:47 AM, Daniel Dunbar <daniel at zuster.org>
wrote:
> > On Sat, Dec 1, 2012 at 1:06 AM, Chandler Carruth <chandlerc at
google.com>
> > wrote:
> >>
> >> On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <clattner at
apple.com>
> wrote:
> >>>
> >>> I'm ok with this in principle, but how about with the
nuance that some
> >>> tests (eg test/codegen) explicitly opt into march=native?
> >>
> >>
> >> I'd really like the default behavior to be something that
forces the
> test
> >> to either be independent of the targeted triple, or explicitly set
a
> target.
> >> I like the default being unknown.
> >>
> >> I wonder, would the ability to run the entire test suite with all
of the
> >> 'default' triples (that lit sets to unknown in normal
runs) instead set
> to
> >> the host, or to a specific triple maybe be a useful extra form of
> checking?
> >> This would let both humans and build bots find bugs and
discrepancies
> >> specific to a particular target.
> >
> >
> > I'd prefer not to do this universally (i.e., run the whole test
suite
> that
> > way),
>
> I'm not entirely sure which distinction you're drawing here when
you
> say "universally". I assume we're specifically talking about
only
> those tests in the lit-based regression suite that don't already have
> an explicit triple. Only these tests would get the default
("unknown")
> triple and be affected by some lit-parameter to override the default
> with any other triple or possibly a list of triples.
>
> Icing would be to have that list be autodetected from the supported
> targets in the current build.
>

What I want is for this to not be "implicit" behavior. Many tests have
no
value being run with multiple triples.

What I want is for the author of the test to explicitly declare what they
are trying to do. If a test is useful on multiple triples, then they should
write something that says so, and the test suite will take the extra time
to do it.

My opinion on tests is that they are significantly better when they are
written with an explicit idea of what coverage they are trying to get (and
hopefully a comment explaining that too).

This would not be the default (the default would just be to run
them
> once, with the "unknown" triple) but would be an easily
accessible
> target for developers and would be the target that any respectable
> buildbot would use. (without enforcement by bots the feature would be
> useless to users since the results would not be clean).
>
> > but what I do think would be useful is to add enough test suite
> > support to be able to easily run the same test on multiple triples (or
> even
> > all configured ones).
> >
> > My primary goal is to have the tests that individual developers be
> > equivalent (independent of the target they are running on).
>
> Did you a word? ("have the tests that individual developers <write?
> run?> be equivalent")
>

Hah. Yes, "tests that individual developers write".

 - Daniel


> - David
>
> >
> >  - Daniel
> >
> >> We could even have a common test target that build bots use which
runs
> all
> >> the tests both in the default, and in the host-triple mode so that
we
> force
> >> people to converge on target independent tests or explicit
triples.
> >>
> >>>
> >>>
> >>> -Chris
> >>>
> >>> On Nov 30, 2012, at 12:16 PM, Daniel Dunbar <daniel at
zuster.org> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > We consistently see test failures arising because by
default many of
> >>> > our tests run in a mode where the tool (clang or llc)
pick
> host-dependent
> >>> > behavior. This makes it easy for developers to write
tests that pass
> on
> >>> > their system, but fail for other developers.
> >>> >
> >>> > There is some utility in this behavior, as it gives us
(unintended)
> >>> > testing coverage of some things, but overall I think it
is a net
> loss for
> >>> > productivity.
> >>> >
> >>> > I propose:
> >>> >
> >>> > a. We change the test suite to run in such a way that all
tools
> default
> >>> > to an "unknown" host triple.
> >>> >
> >>> > b. If someone feels there is missing coverage in some
area, we add
> >>> > increased tests for that area (which get run on all
platforms).
> >>> >
> >>> > c. If there is some reason that running with an
"unknown" host triple
> >>> > is undesirable, I propose that we set the default test
triple to be
> >>> > "x86_64-pc-linux-gnu", and require deviations
to be specified.
> >>> >
> >>> > Comments?
> >>> >
> >>> >  - Daniel
> >>> >
> >>> > _______________________________________________
> >>> > LLVM Developers mailing list
> >>> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> >>> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> >>> _______________________________________________
> >>> cfe-dev mailing list
> >>> cfe-dev at cs.uiuc.edu
> >>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >>
> >>
> >
> >
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20121203/65605da5/attachment.html>