llvm dev - Dec 2012 - [LLVMdev] [cfe-dev] RFC: Change tests to run with fixed (not-host dependent) triple

On Sat, Dec 1, 2012 at 1:06 AM, Chandler Carruth <chandlerc at
google.com>wrote:
> On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <clattner at
apple.com> wrote:
>
>> I'm ok with this in principle, but how about with the nuance that
some
>> tests (eg test/codegen) explicitly opt into march=native?
>>
>
> I'd really like the default behavior to be something that forces the
test
> to either be independent of the targeted triple, or explicitly set a
> target. I like the default being unknown.
>
> I wonder, would the ability to run the entire test suite with all of the
> 'default' triples (that lit sets to unknown in normal runs) instead
set to
> the host, or to a specific triple maybe be a useful extra form of checking?
> This would let both humans and build bots find bugs and discrepancies
> specific to a particular target.
>
I'd prefer not to do this universally (i.e., run the whole test suite that
way), but what I do think would be useful is to add enough test suite
support to be able to easily run the same test on multiple triples (or even
all configured ones).

My primary goal is to have the tests that individual developers be
equivalent (independent of the target they are running on).

 - Daniel

We could even have a common test target that build bots use which runs
all
> the tests both in the default, and in the host-triple mode so that we force
> people to converge on target independent tests or explicit triples.
>
>
>>
>> -Chris
>>
>> On Nov 30, 2012, at 12:16 PM, Daniel Dunbar <daniel at
zuster.org> wrote:
>>
>> > Hi all,
>> >
>> > We consistently see test failures arising because by default many
of
>> our tests run in a mode where the tool (clang or llc) pick
host-dependent
>> behavior. This makes it easy for developers to write tests that pass on
>> their system, but fail for other developers.
>> >
>> > There is some utility in this behavior, as it gives us
(unintended)
>> testing coverage of some things, but overall I think it is a net loss
for
>> productivity.
>> >
>> > I propose:
>> >
>> > a. We change the test suite to run in such a way that all tools
default
>> to an "unknown" host triple.
>> >
>> > b. If someone feels there is missing coverage in some area, we add
>> increased tests for that area (which get run on all platforms).
>> >
>> > c. If there is some reason that running with an
"unknown" host triple
>> is undesirable, I propose that we set the default test triple to be
>> "x86_64-pc-linux-gnu", and require deviations to be
specified.
>> >
>> > Comments?
>> >
>> >  - Daniel
>> >
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20121203/886f8041/attachment.html>

On Mon, Dec 3, 2012 at 10:47 AM, Daniel Dunbar <daniel at zuster.org>
wrote:
> On Sat, Dec 1, 2012 at 1:06 AM, Chandler Carruth <chandlerc at
google.com>
> wrote:
>>
>> On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <clattner at
apple.com> wrote:
>>>
>>> I'm ok with this in principle, but how about with the nuance
that some
>>> tests (eg test/codegen) explicitly opt into march=native?
>>
>>
>> I'd really like the default behavior to be something that forces
the test
>> to either be independent of the targeted triple, or explicitly set a
target.
>> I like the default being unknown.
>>
>> I wonder, would the ability to run the entire test suite with all of
the
>> 'default' triples (that lit sets to unknown in normal runs)
instead set to
>> the host, or to a specific triple maybe be a useful extra form of
checking?
>> This would let both humans and build bots find bugs and discrepancies
>> specific to a particular target.
>
>
> I'd prefer not to do this universally (i.e., run the whole test suite
that
> way),
I'm not entirely sure which distinction you're drawing here when you
say "universally". I assume we're specifically talking about only
those tests in the lit-based regression suite that don't already have
an explicit triple. Only these tests would get the default ("unknown")
triple and be affected by some lit-parameter to override the default
with any other triple or possibly a list of triples.

Icing would be to have that list be autodetected from the supported
targets in the current build.

This would not be the default (the default would just be to run them
once, with the "unknown" triple) but would be an easily accessible
target for developers and would be the target that any respectable
buildbot would use. (without enforcement by bots the feature would be
useless to users since the results would not be clean).
> but what I do think would be useful is to add enough test suite
> support to be able to easily run the same test on multiple triples (or even
> all configured ones).
>
> My primary goal is to have the tests that individual developers be
> equivalent (independent of the target they are running on).
Did you a word? ("have the tests that individual developers <write?
run?> be equivalent")

- David
>
>  - Daniel
>
>> We could even have a common test target that build bots use which runs
all
>> the tests both in the default, and in the host-triple mode so that we
force
>> people to converge on target independent tests or explicit triples.
>>
>>>
>>>
>>> -Chris
>>>
>>> On Nov 30, 2012, at 12:16 PM, Daniel Dunbar <daniel at
zuster.org> wrote:
>>>
>>> > Hi all,
>>> >
>>> > We consistently see test failures arising because by default
many of
>>> > our tests run in a mode where the tool (clang or llc) pick
host-dependent
>>> > behavior. This makes it easy for developers to write tests
that pass on
>>> > their system, but fail for other developers.
>>> >
>>> > There is some utility in this behavior, as it gives us
(unintended)
>>> > testing coverage of some things, but overall I think it is a
net loss for
>>> > productivity.
>>> >
>>> > I propose:
>>> >
>>> > a. We change the test suite to run in such a way that all
tools default
>>> > to an "unknown" host triple.
>>> >
>>> > b. If someone feels there is missing coverage in some area, we
add
>>> > increased tests for that area (which get run on all
platforms).
>>> >
>>> > c. If there is some reason that running with an
"unknown" host triple
>>> > is undesirable, I propose that we set the default test triple
to be
>>> > "x86_64-pc-linux-gnu", and require deviations to be
specified.
>>> >
>>> > Comments?
>>> >
>>> >  - Daniel
>>> >
>>> > _______________________________________________
>>> > LLVM Developers mailing list
>>> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>>
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>

On Mon, Dec 3, 2012 at 10:56 AM, David Blaikie <dblaikie at gmail.com>
wrote:
> On Mon, Dec 3, 2012 at 10:47 AM, Daniel Dunbar <daniel at zuster.org>
wrote:
> > On Sat, Dec 1, 2012 at 1:06 AM, Chandler Carruth <chandlerc at
google.com>
> > wrote:
> >>
> >> On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <clattner at
apple.com>
> wrote:
> >>>
> >>> I'm ok with this in principle, but how about with the
nuance that some
> >>> tests (eg test/codegen) explicitly opt into march=native?
> >>
> >>
> >> I'd really like the default behavior to be something that
forces the
> test
> >> to either be independent of the targeted triple, or explicitly set
a
> target.
> >> I like the default being unknown.
> >>
> >> I wonder, would the ability to run the entire test suite with all
of the
> >> 'default' triples (that lit sets to unknown in normal
runs) instead set
> to
> >> the host, or to a specific triple maybe be a useful extra form of
> checking?
> >> This would let both humans and build bots find bugs and
discrepancies
> >> specific to a particular target.
> >
> >
> > I'd prefer not to do this universally (i.e., run the whole test
suite
> that
> > way),
>
> I'm not entirely sure which distinction you're drawing here when
you
> say "universally". I assume we're specifically talking about
only
> those tests in the lit-based regression suite that don't already have
> an explicit triple. Only these tests would get the default
("unknown")
> triple and be affected by some lit-parameter to override the default
> with any other triple or possibly a list of triples.
>
> Icing would be to have that list be autodetected from the supported
> targets in the current build.
>
What I want is for this to not be "implicit" behavior. Many tests have
no
value being run with multiple triples.

What I want is for the author of the test to explicitly declare what they
are trying to do. If a test is useful on multiple triples, then they should
write something that says so, and the test suite will take the extra time
to do it.

My opinion on tests is that they are significantly better when they are
written with an explicit idea of what coverage they are trying to get (and
hopefully a comment explaining that too).

This would not be the default (the default would just be to run
them
> once, with the "unknown" triple) but would be an easily
accessible
> target for developers and would be the target that any respectable
> buildbot would use. (without enforcement by bots the feature would be
> useless to users since the results would not be clean).
>
> > but what I do think would be useful is to add enough test suite
> > support to be able to easily run the same test on multiple triples (or
> even
> > all configured ones).
> >
> > My primary goal is to have the tests that individual developers be
> > equivalent (independent of the target they are running on).
>
> Did you a word? ("have the tests that individual developers <write?
> run?> be equivalent")
>
Hah. Yes, "tests that individual developers write".

 - Daniel

> - David
>
> >
> >  - Daniel
> >
> >> We could even have a common test target that build bots use which
runs
> all
> >> the tests both in the default, and in the host-triple mode so that
we
> force
> >> people to converge on target independent tests or explicit
triples.
> >>
> >>>
> >>>
> >>> -Chris
> >>>
> >>> On Nov 30, 2012, at 12:16 PM, Daniel Dunbar <daniel at
zuster.org> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > We consistently see test failures arising because by
default many of
> >>> > our tests run in a mode where the tool (clang or llc)
pick
> host-dependent
> >>> > behavior. This makes it easy for developers to write
tests that pass
> on
> >>> > their system, but fail for other developers.
> >>> >
> >>> > There is some utility in this behavior, as it gives us
(unintended)
> >>> > testing coverage of some things, but overall I think it
is a net
> loss for
> >>> > productivity.
> >>> >
> >>> > I propose:
> >>> >
> >>> > a. We change the test suite to run in such a way that all
tools
> default
> >>> > to an "unknown" host triple.
> >>> >
> >>> > b. If someone feels there is missing coverage in some
area, we add
> >>> > increased tests for that area (which get run on all
platforms).
> >>> >
> >>> > c. If there is some reason that running with an
"unknown" host triple
> >>> > is undesirable, I propose that we set the default test
triple to be
> >>> > "x86_64-pc-linux-gnu", and require deviations
to be specified.
> >>> >
> >>> > Comments?
> >>> >
> >>> >  - Daniel
> >>> >
> >>> > _______________________________________________
> >>> > LLVM Developers mailing list
> >>> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> >>> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> >>> _______________________________________________
> >>> cfe-dev mailing list
> >>> cfe-dev at cs.uiuc.edu
> >>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >>
> >>
> >
> >
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20121203/65605da5/attachment.html>