llvm dev - Dec 2012 - [LLVMdev] RFC: Change tests to run with fixed (not-host dependent) triple

Hi all,

We consistently see test failures arising because by default many of our
tests run in a mode where the tool (clang or llc) pick host-dependent
behavior. This makes it easy for developers to write tests that pass on
their system, but fail for other developers.

There is some utility in this behavior, as it gives us (unintended) testing
coverage of some things, but overall I think it is a net loss for
productivity.

I propose:

a. We change the test suite to run in such a way that all tools default to
an "unknown" host triple.

b. If someone feels there is missing coverage in some area, we add
increased tests for that area (which get run on all platforms).

c. If there is some reason that running with an "unknown" host triple
is
undesirable, I propose that we set the default test triple to be
"x86_64-pc-linux-gnu", and require deviations to be specified.

Comments?

 - Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20121130/492b4cf4/attachment.html>

I'm ok with this in principle, but how about with the nuance that some tests
(eg test/codegen) explicitly opt into march=native?

-Chris

On Nov 30, 2012, at 12:16 PM, Daniel Dunbar <daniel at zuster.org> wrote:
> Hi all,
> 
> We consistently see test failures arising because by default many of our
tests run in a mode where the tool (clang or llc) pick host-dependent behavior.
This makes it easy for developers to write tests that pass on their system, but
fail for other developers.
> 
> There is some utility in this behavior, as it gives us (unintended) testing
coverage of some things, but overall I think it is a net loss for productivity.
> 
> I propose:
> 
> a. We change the test suite to run in such a way that all tools default to
an "unknown" host triple.
> 
> b. If someone feels there is missing coverage in some area, we add
increased tests for that area (which get run on all platforms).
> 
> c. If there is some reason that running with an "unknown" host
triple is undesirable, I propose that we set the default test triple to be
"x86_64-pc-linux-gnu", and require deviations to be specified.
> 
> Comments?
> 
>  - Daniel
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <clattner at apple.com>
wrote:
> I'm ok with this in principle, but how about with the nuance that some
> tests (eg test/codegen) explicitly opt into march=native?
>
I'd really like the default behavior to be something that forces the test
to either be independent of the targeted triple, or explicitly set a
target. I like the default being unknown.

I wonder, would the ability to run the entire test suite with all of the
'default' triples (that lit sets to unknown in normal runs) instead set
to
the host, or to a specific triple maybe be a useful extra form of checking?
This would let both humans and build bots find bugs and discrepancies
specific to a particular target.

We could even have a common test target that build bots use which runs all
the tests both in the default, and in the host-triple mode so that we force
people to converge on target independent tests or explicit triples.

>
> -Chris
>
> On Nov 30, 2012, at 12:16 PM, Daniel Dunbar <daniel at zuster.org>
wrote:
>
> > Hi all,
> >
> > We consistently see test failures arising because by default many of
our
> tests run in a mode where the tool (clang or llc) pick host-dependent
> behavior. This makes it easy for developers to write tests that pass on
> their system, but fail for other developers.
> >
> > There is some utility in this behavior, as it gives us (unintended)
> testing coverage of some things, but overall I think it is a net loss for
> productivity.
> >
> > I propose:
> >
> > a. We change the test suite to run in such a way that all tools
default
> to an "unknown" host triple.
> >
> > b. If someone feels there is missing coverage in some area, we add
> increased tests for that area (which get run on all platforms).
> >
> > c. If there is some reason that running with an "unknown"
host triple is
> undesirable, I propose that we set the default test triple to be
> "x86_64-pc-linux-gnu", and require deviations to be specified.
> >
> > Comments?
> >
> >  - Daniel
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20121201/4788bba6/attachment.html>

Apart from what has already being said, which I agree mostly, I have
some specific comments:

> a. We change the test suite to run in such a way that all tools default to
> an "unknown" host triple.
The assumptions will be different and I believe the refactoring of the
tests to make them pass on all currently passing architectures will
not be trivial. Good luck! ;)

> b. If someone feels there is missing coverage in some area, we add
increased
> tests for that area (which get run on all platforms).
If you mean adding "unknown" triple tests as much as possible, I
agree.

> c. If there is some reason that running with an "unknown" host
triple is
> undesirable, I propose that we set the default test triple to be
> "x86_64-pc-linux-gnu", and require deviations to be specified.
Here, I don't agree. I don't see why one platform should be the
default over another.

A similar concept, but less prejudicial is: If the tests require
platform-specific features, platform-specific tests should be added.
*Only* those added will be assumed tested (for the sakes of
validation). Other architectures can add similar tests on their own
triples, if necessary / desired.

In a nutshell, "unknown" defaults to nothing at all, whether running
on Intel, ARM, MIPS or whatever. If you default "unknown" to the host
architecture, or to a specific architecture, the benefits of having an
"unknown" target is void, IMO.


--
cheers,
--renato

http://systemcall.org/

On Mon, Dec 3, 2012 at 1:37 AM, Renato Golin <rengolin at systemcall.org>
wrote:
> Apart from what has already being said, which I agree mostly, I have
> some specific comments:
>
>
>> a. We change the test suite to run in such a way that all tools default
to
>> an "unknown" host triple.
>
> The assumptions will be different and I believe the refactoring of the
> tests to make them pass on all currently passing architectures will
> not be trivial. Good luck! ;)
I'm not sure what you mean by "refactoring of the tests to make them
pass on all currently passing architectures" - if we force tests
without specified triples to use an "unknown" triple, the tests would
need to be refactored, but why would they need to be refactored to
account for all currently passing architectures? They should behave
exactly the same (since they won't inherit the host triple that they
do now) on all test machines/architectures, right?
>> b. If someone feels there is missing coverage in some area, we add
increased
>> tests for that area (which get run on all platforms).
>
> If you mean adding "unknown" triple tests as much as possible, I
agree.
I believe this statement was about the reverse: if certain
target-specific coverage is required it would have to be done by
writing a test that specifies /that/ target triple, not by (as is
sometimes the case currently) writing a test that picks up the host
triple & then running it on a machine with the desired triple to get
coverage for.
>> c. If there is some reason that running with an "unknown"
host triple is
>> undesirable, I propose that we set the default test triple to be
>> "x86_64-pc-linux-gnu", and require deviations to be
specified.
>
> Here, I don't agree. I don't see why one platform should be the
> default over another.
Because we would need/want a default of some kind. The argument here
is: "If we can't choose some agnostic default for all tests, we should
choose a non-agnostic default" - the only alternative position is that
we don't choose a default but instead force every test to specify an
arbitrary triple. I don't think this is substantially more valuable,
though it is the current state of affairs among the tests that do have
triples specified (that they are "random" in the sense that
they're
usually whatever architecture the developer is working on at the time
- so we have lots of linux ones, lots of darwin ones, and a smattering
of ARM)
> A similar concept, but less prejudicial is: If the tests require
> platform-specific features, platform-specific tests should be added.
> *Only* those added will be assumed tested (for the sakes of
> validation). Other architectures can add similar tests on their own
> triples, if necessary / desired.
Yes, that's the intent. This clause (c) was only applicable to the
approach overall (ie: if there are no (or no substantial) tests that
can be expressed with an unknown triple then we can't really use that
as a default - we'd have to pick some other default, or have no
default at all & force every test to specify a triple). For specific
tests that need to test platform-specific things, that is what Daniel
was saying in (b) - add coverage for that area by writing a test with
an explicit triple.
> In a nutshell, "unknown" defaults to nothing at all, whether
running
> on Intel, ARM, MIPS or whatever. If you default "unknown" to the
host
> architecture, or to a specific architecture, the benefits of having an
> "unknown" target is void, IMO.
>
>
> --
> cheers,
> --renato
>
> http://systemcall.org/
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

On 3 December 2012 17:21, David Tweed <david.tweed at arm.com>
wrote:
> Just a point here: the reason I'd mildly prefer not to have a default
that
> avoids as much target dependent stuff as possible is that it's
generally
> going to have a higher probability of passing even if something is
"wrong"
> in the sense that, eg, if the return type of some thing is ABI mandated to
> be void, then you could be getting the type from the wrong place and still
> pass since all places give the same result; if you happen to be doing this
> on an ABI where the where the value differs depending where it is obtained
> from then you'll probably catch errors that are "latent
errors" for the
> simpler ABI. (Yes, this is a thinly disguised version of the
> devirtualisation issue, but I do think this phenomenon may apply elsewhere:
> the less complicated an ABI the more chance that something "slightly
wrong"
> will actually pass.) Those are my thoughts anyway.
I suppose I had that feeling, but couldn't put my finger on it, and to
be honest, I still can't. It feels like we're jumping from one
undefined state to another, but I think the new state is more strict
than the previous, so it is better than what we've got.

However, it'd be good it whoever is changing this could test on a few
other architectures before committing. Or, maybe even preferred, pass
the changes around so that people that work every day on other
architectures (like David Tweed) can have a look and see if it makes
sense on them, too.


--
cheers,
--renato

http://systemcall.org/

On Mon, Dec 3, 2012 at 9:21 AM, David Tweed <david.tweed at arm.com>
wrote:
> -----Original Message-----
> From: cfe-dev-bounces at cs.uiuc.edu [mailto:cfe-dev-bounces at
cs.uiuc.edu] On
> Behalf Of David Blaikie
> Sent: 03 December 2012 16:41
> To: Renato Golin
> Cc: LLVM Developers Mailing List; cfe-dev
> Subject: Re: [cfe-dev] [LLVMdev] RFC: Change tests to run with fixed
> (not-host dependent) triple
>
>>> c. If there is some reason that running with an "unknown"
host triple is
>>> undesirable, I propose that we set the default test triple to be
>>> "x86_64-pc-linux-gnu", and require deviations to be
specified.
>>
>> Here, I don't agree. I don't see why one platform should be the
>> default over another.
>
> | Because we would need/want a default of some kind. The argument here
> | is: "If we can't choose some agnostic default for all tests, we
should
> | choose a non-agnostic default" - the only alternative position is
that
> | we don't choose a default but instead force every test to specify an
> | arbitrary triple. I don't think this is substantially more valuable,
> | though it is the current state of affairs among the tests that do have
> | triples specified (that they are "random" in the sense that
they're
> | usually whatever architecture the developer is working on at the time
> | - so we have lots of linux ones, lots of darwin ones, and a smattering
> | of ARM)
>
> Just a point here: the reason I'd mildly prefer not to have a default
that
> avoids as much target dependent stuff as possible is that it's
generally
> going to have a higher probability of passing even if something is
"wrong"
> in the sense that, eg, if the return type of some thing is ABI mandated to
> be void, then you could be getting the type from the wrong place and still
> pass since all places give the same result;
Sorry, I think I lost track of this example around here. What do you
mean by "getting the type from the wrong place"?
What sort of solution are you proposing?

What I think we're discussing here is:
All tests without a triple be written in such a way as to pass for any
triple (& there would be test infrastructure to help ensure this) &
that the default would be a fixed "unknown" triple, or an arbitrary
(but constant/singular) concrete triple.

What is it you have in mind?
>  if you happen to be doing this
> on an ABI where the where the value differs depending where it is obtained
> from then you'll probably catch errors that are "latent
errors" for the
> simpler ABI. (Yes, this is a thinly disguised version of the
> devirtualisation issue, but I do think this phenomenon may apply elsewhere:
> the less complicated an ABI the more chance that something "slightly
wrong"
> will actually pass.) Those are my thoughts anyway.
>
> Regards,
> Dave
>
>
>

The example I was giving was a "thinly generalized" version of the
devirtualisation issue: what was happening was that a destructor, which on
ARM ABI is returns "this", was being devirtualised and a function with
the
return type of the base class prototype was being added whereas it ought to
have been the return type of the actual function being inlined. (Devirt
can't do general covariant returns, but I gather that no-one actually uses
destructor return types anyway.) On x86_64 the return type of a destructor
is always void, so getting the type from the base class prototype gives the
same result. I know it's a matter of semantics: was that code actually
"a
latent bug" on x86_64 when it would always be guaranteed to produce the
right code (on that platform)?

I know this is a ridiculously specific example, and I don't know if similar
effects can happen elsewhere, I just have this feeling that picking the
simplest platform it's more likely for stuff to pass even if there's a
minor latent issue. But I can't back that up with any facts.

As for proposed solutions, I can't say I really have anything that counts
as a solution. I think I need to see what "unknown" actually implies
in
terms of behaviour/code-paths, it's just that if it's equivalent to
"the
simplest, most vanilla target" it may be less effective in finding issues.
But again, that's just a feeling.

Regards,
Dave


On Mon, Dec 3, 2012 at 6:59 PM, David Blaikie <dblaikie at gmail.com>
wrote:
> On Mon, Dec 3, 2012 at 9:21 AM, David Tweed <david.tweed at arm.com>
wrote:
> > -----Original Message-----
> > From: cfe-dev-bounces at cs.uiuc.edu [mailto:cfe-dev-bounces at
cs.uiuc.edu]
> On
> > Behalf Of David Blaikie
> > Sent: 03 December 2012 16:41
> > To: Renato Golin
> > Cc: LLVM Developers Mailing List; cfe-dev
> > Subject: Re: [cfe-dev] [LLVMdev] RFC: Change tests to run with fixed
> > (not-host dependent) triple
> >
> >>> c. If there is some reason that running with an
"unknown" host triple
> is
> >>> undesirable, I propose that we set the default test triple to
be
> >>> "x86_64-pc-linux-gnu", and require deviations to be
specified.
> >>
> >> Here, I don't agree. I don't see why one platform should
be the
> >> default over another.
> >
> > | Because we would need/want a default of some kind. The argument here
> > | is: "If we can't choose some agnostic default for all
tests, we should
> > | choose a non-agnostic default" - the only alternative position
is that
> > | we don't choose a default but instead force every test to
specify an
> > | arbitrary triple. I don't think this is substantially more
valuable,
> > | though it is the current state of affairs among the tests that do
have
> > | triples specified (that they are "random" in the sense
that they're
> > | usually whatever architecture the developer is working on at the
time
> > | - so we have lots of linux ones, lots of darwin ones, and a
smattering
> > | of ARM)
> >
> > Just a point here: the reason I'd mildly prefer not to have a
default
> that
> > avoids as much target dependent stuff as possible is that it's
generally
> > going to have a higher probability of passing even if something is
> "wrong"
> > in the sense that, eg, if the return type of some thing is ABI
mandated
> to
> > be void, then you could be getting the type from the wrong place and
> still
> > pass since all places give the same result;
>
> Sorry, I think I lost track of this example around here. What do you
> mean by "getting the type from the wrong place"?
> What sort of solution are you proposing?
>
> What I think we're discussing here is:
> All tests without a triple be written in such a way as to pass for any
> triple (& there would be test infrastructure to help ensure this) &
> that the default would be a fixed "unknown" triple, or an
arbitrary
> (but constant/singular) concrete triple.
>
> What is it you have in mind?
>
> >  if you happen to be doing this
> > on an ABI where the where the value differs depending where it is
> obtained
> > from then you'll probably catch errors that are "latent
errors" for the
> > simpler ABI. (Yes, this is a thinly disguised version of the
> > devirtualisation issue, but I do think this phenomenon may apply
> elsewhere:
> > the less complicated an ABI the more chance that something
"slightly
> wrong"
> > will actually pass.) Those are my thoughts anyway.
> >
> > Regards,
> > Dave
> >
> >
> >
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>


-- 
cheers, dave tweed__________________________
high-performance computing and machine vision expert: david.tweed at gmail.com
"while having code so boring anyone can maintain it, use Python." --
attempted insult seen on slashdot
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20121204/632cef51/attachment.html>