llvm dev - Sep 2015 - Testing "normal" cross-compilers versus GPU backends

> On Sep 3, 2015, at 11:23 AM, Robinson, Paul <Paul_Robinson at
playstation.sony.com> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: Tom Stellard [mailto:tom at stellard.net]
>> Sent: Thursday, September 03, 2015 7:31 AM
>> To: Mehdi Amini
>> Cc: Robinson, Paul; llvm-dev at lists.llvm.org; NAKAMURA Takumi
>> Subject: Re: Testing "normal" cross-compilers versus GPU
backends
>> 
>> On Thu, Sep 03, 2015 at 02:07:54AM -0700, Mehdi Amini wrote:
>>> 
>>>> On Sep 3, 2015, at 12:18 AM, Robinson, Paul
>> <Paul_Robinson at playstation.sony.com> wrote:
>>>> 
>>>> 
>>>> 
>>>>> -----Original Message-----
>>>>> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
>>>>> Sent: Wednesday, September 02, 2015 7:10 PM
>>>>> To: Robinson, Paul
>>>>> Cc: llvm-dev at lists.llvm.org; tom at stellard.net;
NAKAMURA Takumi
>>>>> Subject: Re: Testing "normal" cross-compilers
versus GPU backends
>>>>> 
>>>>> Hi Paul,
>>>>> 
>>>>> Thanks for the summary!
>>>>> 
>>>>>> On Sep 2, 2015, at 5:44 PM, Robinson, Paul
>>>>> <Paul_Robinson at playstation.sony.com> wrote:
>>>>>> 
>>>>>> This note arose from
>> https://urldefense.proofpoint.com/v2/url?u=http-
>>>>>
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
>>>>> 
>>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
>>>>>
o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but
>> the
>>>>> reviewers
>>>>>> felt that we needed a broader audience, because the
proposed patch
>>>>>> really didn't solve the entire problem and we had
no better ideas.
>>>>>> 
>>>>>> Mehdi Amini needs to build LLVM with just a GPU
backend, and still
>> have
>>>>>> "ninja check" Just Work.  Commits
r243958-243960 tried to accomplish
>>>>>> that; however they are too big a hammer, and cause much
simpler
>> cross
>>>>>> environments (like mine) to exclude big chunks of very
useful tests
>>>>>> (including my favorite, DebugInfo).
>>>>>> 
>>>>>> FYI, my main cross environment is building on X86
Windows but using
>>>>>> a default target triple for PS4 (which is also X86).
>>>>>> 
>>>>>> I experimented with building LLVM with just the ARM
backend (running
>> on
>>>>>> an X86 workstation) and setting the default triple to
some ARM
>> value.
>>>>>> "ninja check" worked fine (without
Mehdi's series of commits), so
>> the
>>>>>> normal kind of cross-compiler environment seems to be
happy with how
>>>>>> things were set up originally.
>>>>>> 
>>>>>> Mehdi reports building LLVM with the X86 and AMDGPU
backends,
>> setting
>>>>>> the default triple to "amdgcn--amdhsa", and
getting 200-some
>> failures.
>>>>>> 
>>>>>> (This does make me wonder about AMDGPU testing in
general; how does
>> that
>>>>>> work?  The only places I see lit checks for AMDGPU are
in the usual
>>>>>> target-dependent places.)
>>>>> 
>>>>> I don’t understand this interrogation about how do you do
testing in
>>>>> general. The same way you don’t process tests/CodeGen/X86/*
with the
>> ARM
>>>>> backend, you can’t process any random IR through these
backends.
>>>> 
>>>> You said you had 200+ failures with AMDGPU.  Are the AMD folks
simply
>>>> tolerating the 200 failures, and you don't want to?  I
should hope
>> there
>>>> is more to it than that.
>>> 
>>> Well, I don’t know, they might just run `ninja check` with the
default
>> triple set to X86?
>>> (which I would consider being working around a buggy test suite)
>>> 
>> 
>> I always enable AMDGPU and X86 when I build, so I've never run into
>> this problem.
>> 
>> -Tom
> 
> Tom, presumably your default target triple is X86-ish?  And the only
> tests to exercise the AMDGPU backend are those that explicitly specify
> a triple for AMDGPU?
This is how we used to do it as well, and I assume this is how most of backends
out of the main CPUs are dealing with the situation.
This is what I consider a bug and was trying to solve (admittedly not in an
optimal way).
> Mehdi, assuming that's what Tom does, your stated goal was to be able
to
> run tests *without* including the X86 backend, so Tom's solution
won't
> work for you (restating just for confirmation).
Yes, or alternatively I expect the test suite to pass whatever the default
triple is.
> Krzysztof suggested much the same thing that I think you are currently
> doing, which is deliberately configure a default triple but exclude the
> corresponding backend.  
You and Takumi were considering this as an unsupported configuration before, and
I tend to agree with that (this is the configuration I’m using for our tests but
it was not intentional to leave the default triple unset).
> I expect we can detect that situation in lit.cfg
> and exclude tests on that basis, rather than 'native'.  It would
solve
> the problem for my case (host triple != target triple, although the arch
> parts of the triple do match) and the "normal" cross-compiler
case (e.g.
> host = X86, backend + target triple = ARM).
> 
> I'm going to play around with that and see what I can do to make it
work.
> 
>> 
>> 
>>>> 
>>>>> 
>>>>> IMO, the problem is in general about tests that are written
without
>>>>> specifying a triple, that will be executed with the default
triple.
>>>>> 
>>>>> Most of these tests were written with X86 (or ARM) in mind,
and there
>> is
>>>>> no guarantee that they will behave as intended with every
possible
>> triple.
>>>>> The DataLayout for instance has to be the one from the
target, and is
>> not
>>>>> portable.
>>>>> I think a "portable backend test” is pretty rare in
general.
>>>> 
>>>> It depends on what the test is trying to do.  I'm sure it
is quite
>> common
>>>> for IR tests to behave essentially the same way regardless of
target.
>>> 
>>> IR tests != backend test (I may miss your point here, it’s late…).
> 
> Right, sorry, lost focus for a moment there... nevertheless it is still
> the case that many tests exercise functionality that is not particularly
> target-centric and these should be run for any target that actually 
> supports that functionality.  For example, the DebugInfo tests should
> be run for any target that supports emitting debug info.
I’m not sure that “debug info” support is all or nothing. 
As an extreme example, I know targets that support debug info but do not support
function calls, what if your "debug info” test involves these?

Also, I’m not a DebugInfo expert, but when a front-end generated them, aren’t
they dependent on the DataLayout? Hence the target?
It means that many DebugInfo test could fail with a backend that would have a
significantly different DataLayout and won’t expect some stuff the way they are
played down.

>  Whether a
> target supports debug info is orthogonal to its native-ness. As written
> the top-level DebugInfo tests should work fine as long as the default
> triple's backend is included, and that backend supports debug info.
> 
> If your backend doesn't support debug info, then it's reasonable to
> exclude those from your testing; but we can't do that at the cost of
> excluding those from testing other backends that *do* support the feature,
> even if that testing runs in a cross-compiler environment.
> 
> In this particular example, we'd be setting things up so that DebugInfo
> is excluded for the wrong reason (not based on some more abstract idea
> of the feature-set of the target) but as Krzysztof observes, getting
> a feature-oriented set of conditions would be a huge task.
Agree: the predicate I wrote is not correct, and I don’t see a trivial “good”
solution. This is why any compromise or intermediate solution that is good
enough to support whatever use-case people here have, including your, seems
appropriate to implement.

Let me know I can help coding something.

— 
Mehdi

> 
>>> 
>>> 
>>>> We have lots of tests (the ones you chose to mark
"native") that had
>> been
>>>> working fine with ARM, X86, PPC, SPARC, and whatever.
>>>> 
>>>> The fact that they don't work with your backend is
different from
>> saying
>>>> those tests can't possibly work for any cross-compiler.
>>> 
>>> I believe this is far from what I said, or at least from what I had
in
>> mind.
>>> 
>>>> But the latter is what your patch implemented, and it is
preventing
>> useful testing.
>>> 
>>> I don’t disagree with that statement, and I agree that it should be
>> fixed.
>>> It doesn’t mean that I think the previous situation was better
though.
>>> The current over-conservative state seems at least more correct
than
>> having only a subset of the targets that can pass `ninja check`.
>>> 
>>> Are you attending the LLVM bay area social tonight by any chance?
> 
> Unfortunately no, I have a prior commitment.
> --paulr
> 
>>> 
>>> Best,
>>> 
>>> —
>>> Mehdi
>>> 
>>> 
>>> 
>>>> --paulr
>>>> 
>>>>> 
>>>>> You can run all the tests by setting the default triple to
X86 and
>>>>> compiling in the X86 backend, but that’s just a trick to
make the
>> tests
>>>>> happy.
>>>>> Alternatively, and this is what I tried to do, blacklisting
these
>> tests
>>>>> that “pretends” to be “portable” but are not.
>>>>> 
>>>>> 
>>>>>> Mehdi's solution was:
>>>>>> - In lit.cfg, change the existing "native"
feature definition from
>>>>>> "host-triple == target-triple" to also check
"and the corresponding
>>>>>> backend is included."(**)
>>>>> 
>>>>> I agree that we could remove the condition on the host
backend
>> included
>>>>> and deemed it a unsupported, but the build system needs to
reject
>> this
>>>>> configuration.
>>>>> 
>>>>> 
>>>>>> - Make piles of tests that seemed inapplicable to GPUs
depend on the
>>>>>> "native" feature (through REQUIRES: or in the
lit.local.cfg).
>>>>> 
>>>>> Nitpick: the GPU is just an example, any other backends can
be
>> affected.
>>>>> It seems that these test are “lying” about the target they
will be
>> able to
>>>>> run on (like if they would run on “anything”).
>>>>> 
>>>>> 
>>>>>> - Build LLVM with just the GPU backend, and not set a
target triple
>>>>>> (i.e., it's set to the host triple, typically an
X86-something).(*)
>>>>>> Thus lit.cfg sees matching host and target triples, but
the X86
>>>>>> backend is missing, and so the "native"
feature is not set.
>>>>>> 
>>>>>> [The "native" feature was invented to
identify environments where
>>>>>> JIT would work properly. The "host-triple ==
target-triple"
>> condition
>>>>>> isn't exactly right, but it works well enough.]
>>>>>> 
>>>>>> The major problem is that these new "native"
dependencies are
>> incorrect.
>>>>>> For example the DebugInfo tests don't really
require it; they work
>> fine
>>>>>> as long as the default triple has the corresponding
backend
>> included,
>>>>>> as my ARM-on-X86 experiment demonstrated.
>>>>> 
>>>>> Are they are guarantee’d to work with a default triple set
to any of
>> the
>>>>> possible (in-tree) backend?
>>>>> (I don’t know enough about these tests, which is
definitively why I
>>>>> included them in the “big hammer” solution)
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> —
>>>>> Mehdi
>>>

> -----Original Message-----
> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
> Sent: Thursday, September 03, 2015 3:26 PM
> To: Robinson, Paul
> Cc: Tom Stellard; llvm-dev at lists.llvm.org; NAKAMURA Takumi
> Subject: Re: Testing "normal" cross-compilers versus GPU backends
> 
> 
> > On Sep 3, 2015, at 11:23 AM, Robinson, Paul
> <Paul_Robinson at playstation.sony.com> wrote:
> >
> >
> >
> >> -----Original Message-----
> >> From: Tom Stellard [mailto:tom at stellard.net]
> >> Sent: Thursday, September 03, 2015 7:31 AM
> >> To: Mehdi Amini
> >> Cc: Robinson, Paul; llvm-dev at lists.llvm.org; NAKAMURA Takumi
> >> Subject: Re: Testing "normal" cross-compilers versus GPU
backends
> >>
> >> On Thu, Sep 03, 2015 at 02:07:54AM -0700, Mehdi Amini wrote:
> >>>
> >>>> On Sep 3, 2015, at 12:18 AM, Robinson, Paul
> >> <Paul_Robinson at playstation.sony.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
> >>>>> Sent: Wednesday, September 02, 2015 7:10 PM
> >>>>> To: Robinson, Paul
> >>>>> Cc: llvm-dev at lists.llvm.org; tom at stellard.net;
NAKAMURA Takumi
> >>>>> Subject: Re: Testing "normal"
cross-compilers versus GPU backends
> >>>>>
> >>>>> Hi Paul,
> >>>>>
> >>>>> Thanks for the summary!
> >>>>>
> >>>>>> On Sep 2, 2015, at 5:44 PM, Robinson, Paul
> >>>>> <Paul_Robinson at playstation.sony.com> wrote:
> >>>>>>
> >>>>>> This note arose from
> >> https://urldefense.proofpoint.com/v2/url?u=http-
> >>>>>
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
> >>>>>
> >>
>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
> >>>>>
o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but
> >> the
> >>>>> reviewers
> >>>>>> felt that we needed a broader audience, because
the proposed patch
> >>>>>> really didn't solve the entire problem and we
had no better ideas.
> >>>>>>
> >>>>>> Mehdi Amini needs to build LLVM with just a GPU
backend, and still
> >> have
> >>>>>> "ninja check" Just Work.  Commits
r243958-243960 tried to
> accomplish
> >>>>>> that; however they are too big a hammer, and cause
much simpler
> >> cross
> >>>>>> environments (like mine) to exclude big chunks of
very useful tests
> >>>>>> (including my favorite, DebugInfo).
> >>>>>>
> >>>>>> FYI, my main cross environment is building on X86
Windows but using
> >>>>>> a default target triple for PS4 (which is also
X86).
> >>>>>>
> >>>>>> I experimented with building LLVM with just the
ARM backend
> (running
> >> on
> >>>>>> an X86 workstation) and setting the default triple
to some ARM
> >> value.
> >>>>>> "ninja check" worked fine (without
Mehdi's series of commits), so
> >> the
> >>>>>> normal kind of cross-compiler environment seems to
be happy with
> how
> >>>>>> things were set up originally.
> >>>>>>
> >>>>>> Mehdi reports building LLVM with the X86 and
AMDGPU backends,
> >> setting
> >>>>>> the default triple to "amdgcn--amdhsa",
and getting 200-some
> >> failures.
> >>>>>>
> >>>>>> (This does make me wonder about AMDGPU testing in
general; how does
> >> that
> >>>>>> work?  The only places I see lit checks for AMDGPU
are in the usual
> >>>>>> target-dependent places.)
> >>>>>
> >>>>> I don’t understand this interrogation about how do you
do testing in
> >>>>> general. The same way you don’t process
tests/CodeGen/X86/* with the
> >> ARM
> >>>>> backend, you can’t process any random IR through these
backends.
> >>>>
> >>>> You said you had 200+ failures with AMDGPU.  Are the AMD
folks simply
> >>>> tolerating the 200 failures, and you don't want to?  I
should hope
> >> there
> >>>> is more to it than that.
> >>>
> >>> Well, I don’t know, they might just run `ninja check` with the
default
> >> triple set to X86?
> >>> (which I would consider being working around a buggy test
suite)
> >>>
> >>
> >> I always enable AMDGPU and X86 when I build, so I've never run
into
> >> this problem.
> >>
> >> -Tom
> >
> > Tom, presumably your default target triple is X86-ish?  And the only
> > tests to exercise the AMDGPU backend are those that explicitly specify
> > a triple for AMDGPU?
> 
> This is how we used to do it as well, and I assume this is how most of
> backends out of the main CPUs are dealing with the situation.
> This is what I consider a bug and was trying to solve (admittedly not in
> an optimal way).
> 
> > Mehdi, assuming that's what Tom does, your stated goal was to be
able to
> > run tests *without* including the X86 backend, so Tom's solution
won't
> > work for you (restating just for confirmation).
> 
> Yes, or alternatively I expect the test suite to pass whatever the default
> triple is.
Um, that may be asking a bit much...
> 
> > Krzysztof suggested much the same thing that I think you are currently
> > doing, which is deliberately configure a default triple but exclude
the
> > corresponding backend.
> 
> You and Takumi were considering this as an unsupported configuration
> before, and I tend to agree with that (this is the configuration I’m using
> for our tests but it was not intentional to leave the default triple
> unset).
Right, intuitively it doesn't make sense.  Is it actually useful to build a 
GPU compiler that will crash unless you ask it to generate GPU code? Seems
to me it should default to producing GPU code.

> 
> > I expect we can detect that situation in lit.cfg
> > and exclude tests on that basis, rather than 'native'.  It
would solve
> > the problem for my case (host triple != target triple, although the
arch
> > parts of the triple do match) and the "normal"
cross-compiler case (e.g.
> > host = X86, backend + target triple = ARM).
> >
> > I'm going to play around with that and see what I can do to make
it
> work.
> >
> >>
> >>
> >>>>
> >>>>>
> >>>>> IMO, the problem is in general about tests that are
written without
> >>>>> specifying a triple, that will be executed with the
default triple.
> >>>>>
> >>>>> Most of these tests were written with X86 (or ARM) in
mind, and
> there
> >> is
> >>>>> no guarantee that they will behave as intended with
every possible
> >> triple.
> >>>>> The DataLayout for instance has to be the one from the
target, and
> is
> >> not
> >>>>> portable.
> >>>>> I think a "portable backend test” is pretty rare
in general.
> >>>>
> >>>> It depends on what the test is trying to do.  I'm sure
it is quite
> >> common
> >>>> for IR tests to behave essentially the same way regardless
of target.
> >>>
> >>> IR tests != backend test (I may miss your point here, it’s
late…).
> >
> > Right, sorry, lost focus for a moment there... nevertheless it is
still
> > the case that many tests exercise functionality that is not
particularly
> > target-centric and these should be run for any target that actually
> > supports that functionality.  For example, the DebugInfo tests should
> > be run for any target that supports emitting debug info.
> 
> I’m not sure that “debug info” support is all or nothing.
> As an extreme example, I know targets that support debug info but do not
> support function calls, what if your "debug info” test involves these?
Then as part of getting the test suite to work for you, you would need to
disable that particular test for your target.  It sounds like this kind of
thing is exactly what the Hexagon folks did, and it seems quite reasonable.
(And in fact I see two DebugInfo tests marked XFAIL: hexagon.)
> 
> Also, I’m not a DebugInfo expert, but when a front-end generated them,
> aren’t they dependent on the DataLayout? Hence the target?
Not really. DebugInfo tests primarily care what the DWARF description
looks like, not so much what the generated code looks like, and there
are 100 or so DebugInfo tests that work across lots of targets.

Many DebugInfo tests are in target-dependent directories as artifacts of
how the tests are implemented, rather than because they are truly
target-dependent. The ones I've tripped over tend to be target-specific
because they generate assembler and the assembler syntax varies too much
across targets to be worth making the CHECKs tolerant enough.

There are fine details that would be DataLayout dependent, but 99% of
the DebugInfo tests aren't checking things that are at such a low level.
> It means that many DebugInfo test could fail with a backend that would
> have a significantly different DataLayout and won’t expect some stuff the
> way they are played down.
> 
> 
> >  Whether a
> > target supports debug info is orthogonal to its native-ness. As
written
> > the top-level DebugInfo tests should work fine as long as the default
> > triple's backend is included, and that backend supports debug
info.
> >
> > If your backend doesn't support debug info, then it's
reasonable to
> > exclude those from your testing; but we can't do that at the cost
of
> > excluding those from testing other backends that *do* support the
> feature,
> > even if that testing runs in a cross-compiler environment.
> >
> > In this particular example, we'd be setting things up so that
DebugInfo
> > is excluded for the wrong reason (not based on some more abstract idea
> > of the feature-set of the target) but as Krzysztof observes, getting
> > a feature-oriented set of conditions would be a huge task.
> 
> Agree: the predicate I wrote is not correct, and I don’t see a trivial
> “good” solution. This is why any compromise or intermediate solution that
> is good enough to support whatever use-case people here have, including
> your, seems appropriate to implement.
> 
> Let me know I can help coding something.
I have experimented with implementing the thing Takumi and I think should
be a configuration error. :-)  Basically it takes the same kind of approach
that I did in D12506, except it checks for the existence of the target that
matches the default triple. If that target exists then 'llc' with no
triple
will succeed, and it looks like the bulk of the tests that you disabled are
in that category.  I'm not especially happy about this tactic, though.

The Hexagon precedent is interesting; Krzysztof said they set the default
triple, and didn't have to xfail all that much stuff.  Searching the tree,
I find exactly 7 individual tests marked XFAIL: hexagon, plus it disables 
all of ExecutionEngine, and turns off the 'object-emission' feature.

I'm curious if you would try setting the default triple to match your
target, and see what /kinds/ of tests fail.  The raw number is much less
interesting than in the categories.

Thanks,
--paulr
> 
> —
> Mehdi
> 
> 
> >
> >>>
> >>>
> >>>> We have lots of tests (the ones you chose to mark
"native") that had
> >> been
> >>>> working fine with ARM, X86, PPC, SPARC, and whatever.
> >>>>
> >>>> The fact that they don't work with your backend is
different from
> >> saying
> >>>> those tests can't possibly work for any
cross-compiler.
> >>>
> >>> I believe this is far from what I said, or at least from what
I had in
> >> mind.
> >>>
> >>>> But the latter is what your patch implemented, and it is
preventing
> >> useful testing.
> >>>
> >>> I don’t disagree with that statement, and I agree that it
should be
> >> fixed.
> >>> It doesn’t mean that I think the previous situation was better
though.
> >>> The current over-conservative state seems at least more
correct than
> >> having only a subset of the targets that can pass `ninja check`.
> >>>
> >>> Are you attending the LLVM bay area social tonight by any
chance?
> >
> > Unfortunately no, I have a prior commitment.
> > --paulr
> >
> >>>
> >>> Best,
> >>>
> >>> —
> >>> Mehdi
> >>>
> >>>
> >>>
> >>>> --paulr
> >>>>
> >>>>>
> >>>>> You can run all the tests by setting the default
triple to X86 and
> >>>>> compiling in the X86 backend, but that’s just a trick
to make the
> >> tests
> >>>>> happy.
> >>>>> Alternatively, and this is what I tried to do,
blacklisting these
> >> tests
> >>>>> that “pretends” to be “portable” but are not.
> >>>>>
> >>>>>
> >>>>>> Mehdi's solution was:
> >>>>>> - In lit.cfg, change the existing
"native" feature definition from
> >>>>>> "host-triple == target-triple" to also
check "and the corresponding
> >>>>>> backend is included."(**)
> >>>>>
> >>>>> I agree that we could remove the condition on the host
backend
> >> included
> >>>>> and deemed it a unsupported, but the build system
needs to reject
> >> this
> >>>>> configuration.
> >>>>>
> >>>>>
> >>>>>> - Make piles of tests that seemed inapplicable to
GPUs depend on
> the
> >>>>>> "native" feature (through REQUIRES: or
in the lit.local.cfg).
> >>>>>
> >>>>> Nitpick: the GPU is just an example, any other
backends can be
> >> affected.
> >>>>> It seems that these test are “lying” about the target
they will be
> >> able to
> >>>>> run on (like if they would run on “anything”).
> >>>>>
> >>>>>
> >>>>>> - Build LLVM with just the GPU backend, and not
set a target triple
> >>>>>> (i.e., it's set to the host triple, typically
an X86-something).(*)
> >>>>>> Thus lit.cfg sees matching host and target
triples, but the X86
> >>>>>> backend is missing, and so the "native"
feature is not set.
> >>>>>>
> >>>>>> [The "native" feature was invented to
identify environments where
> >>>>>> JIT would work properly. The "host-triple ==
target-triple"
> >> condition
> >>>>>> isn't exactly right, but it works well
enough.]
> >>>>>>
> >>>>>> The major problem is that these new
"native" dependencies are
> >> incorrect.
> >>>>>> For example the DebugInfo tests don't really
require it; they work
> >> fine
> >>>>>> as long as the default triple has the
corresponding backend
> >> included,
> >>>>>> as my ARM-on-X86 experiment demonstrated.
> >>>>>
> >>>>> Are they are guarantee’d to work with a default triple
set to any of
> >> the
> >>>>> possible (in-tree) backend?
> >>>>> (I don’t know enough about these tests, which is
definitively why I
> >>>>> included them in the “big hammer” solution)
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> —
> >>>>> Mehdi
> >>>

On 9/3/2015 7:56 PM, Robinson, Paul via llvm-dev wrote:
> Right, intuitively it doesn't make sense.  Is it actually useful to
build a
> GPU compiler that will crash unless you ask it to generate GPU code? Seems
> to me it should default to producing GPU code.
If the lack of the default triple was to become a feature, the compiler 
could simply request specific arch/cpu settings and exit gracefully. 
The default triple could be set to something that explicitly states the 
absence of a particular default target (e.g. triple=none, or something 
like that).  This may not be useful for popular targets (that are used 
as hosts), but it might be for embedded targets that come in a variety 
of different flavors.  The default choice may be a lot less obvious in 
those cases.  As an added bonus, if users have to explicitly specify 
target details, they would be immune to potential changes of the 
defaults if they were to upgrade the compiler.

-Krzysztof

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, 
hosted by The Linux Foundation

> On Sep 3, 2015, at 5:56 PM, Robinson, Paul <Paul_Robinson at
playstation.sony.com> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
>> Sent: Thursday, September 03, 2015 3:26 PM
>> To: Robinson, Paul
>> Cc: Tom Stellard; llvm-dev at lists.llvm.org; NAKAMURA Takumi
>> Subject: Re: Testing "normal" cross-compilers versus GPU
backends
>> 
>> 
>>> On Sep 3, 2015, at 11:23 AM, Robinson, Paul
>> <Paul_Robinson at playstation.sony.com> wrote:
>>> 
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: Tom Stellard [mailto:tom at stellard.net]
>>>> Sent: Thursday, September 03, 2015 7:31 AM
>>>> To: Mehdi Amini
>>>> Cc: Robinson, Paul; llvm-dev at lists.llvm.org; NAKAMURA Takumi
>>>> Subject: Re: Testing "normal" cross-compilers versus
GPU backends
>>>> 
>>>> On Thu, Sep 03, 2015 at 02:07:54AM -0700, Mehdi Amini wrote:
>>>>> 
>>>>>> On Sep 3, 2015, at 12:18 AM, Robinson, Paul
>>>> <Paul_Robinson at playstation.sony.com> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
>>>>>>> Sent: Wednesday, September 02, 2015 7:10 PM
>>>>>>> To: Robinson, Paul
>>>>>>> Cc: llvm-dev at lists.llvm.org; tom at
stellard.net; NAKAMURA Takumi
>>>>>>> Subject: Re: Testing "normal"
cross-compilers versus GPU backends
>>>>>>> 
>>>>>>> Hi Paul,
>>>>>>> 
>>>>>>> Thanks for the summary!
>>>>>>> 
>>>>>>>> On Sep 2, 2015, at 5:44 PM, Robinson, Paul
>>>>>>> <Paul_Robinson at playstation.sony.com>
wrote:
>>>>>>>> 
>>>>>>>> This note arose from
>>>> https://urldefense.proofpoint.com/v2/url?u=http-
>>>>>>>
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
>>>>>>> 
>>>> 
>>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
>>>>>>>
o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but
>>>> the
>>>>>>> reviewers
>>>>>>>> felt that we needed a broader audience, because
the proposed patch
>>>>>>>> really didn't solve the entire problem and
we had no better ideas.
>>>>>>>> 
>>>>>>>> Mehdi Amini needs to build LLVM with just a GPU
backend, and still
>>>> have
>>>>>>>> "ninja check" Just Work.  Commits
r243958-243960 tried to
>> accomplish
>>>>>>>> that; however they are too big a hammer, and
cause much simpler
>>>> cross
>>>>>>>> environments (like mine) to exclude big chunks
of very useful tests
>>>>>>>> (including my favorite, DebugInfo).
>>>>>>>> 
>>>>>>>> FYI, my main cross environment is building on
X86 Windows but using
>>>>>>>> a default target triple for PS4 (which is also
X86).
>>>>>>>> 
>>>>>>>> I experimented with building LLVM with just the
ARM backend
>> (running
>>>> on
>>>>>>>> an X86 workstation) and setting the default
triple to some ARM
>>>> value.
>>>>>>>> "ninja check" worked fine (without
Mehdi's series of commits), so
>>>> the
>>>>>>>> normal kind of cross-compiler environment seems
to be happy with
>> how
>>>>>>>> things were set up originally.
>>>>>>>> 
>>>>>>>> Mehdi reports building LLVM with the X86 and
AMDGPU backends,
>>>> setting
>>>>>>>> the default triple to
"amdgcn--amdhsa", and getting 200-some
>>>> failures.
>>>>>>>> 
>>>>>>>> (This does make me wonder about AMDGPU testing
in general; how does
>>>> that
>>>>>>>> work?  The only places I see lit checks for
AMDGPU are in the usual
>>>>>>>> target-dependent places.)
>>>>>>> 
>>>>>>> I don’t understand this interrogation about how do
you do testing in
>>>>>>> general. The same way you don’t process
tests/CodeGen/X86/* with the
>>>> ARM
>>>>>>> backend, you can’t process any random IR through
these backends.
>>>>>> 
>>>>>> You said you had 200+ failures with AMDGPU.  Are the
AMD folks simply
>>>>>> tolerating the 200 failures, and you don't want to?
I should hope
>>>> there
>>>>>> is more to it than that.
>>>>> 
>>>>> Well, I don’t know, they might just run `ninja check` with
the default
>>>> triple set to X86?
>>>>> (which I would consider being working around a buggy test
suite)
>>>>> 
>>>> 
>>>> I always enable AMDGPU and X86 when I build, so I've never
run into
>>>> this problem.
>>>> 
>>>> -Tom
>>> 
>>> Tom, presumably your default target triple is X86-ish?  And the
only
>>> tests to exercise the AMDGPU backend are those that explicitly
specify
>>> a triple for AMDGPU?
>> 
>> This is how we used to do it as well, and I assume this is how most of
>> backends out of the main CPUs are dealing with the situation.
>> This is what I consider a bug and was trying to solve (admittedly not
in
>> an optimal way).
>> 
>>> Mehdi, assuming that's what Tom does, your stated goal was to
be able to
>>> run tests *without* including the X86 backend, so Tom's
solution won't
>>> work for you (restating just for confirmation).
>> 
>> Yes, or alternatively I expect the test suite to pass whatever the
default
>> triple is.
> 
> Um, that may be asking a bit much…
It already works for >95% of the suite :)
The rest is working (and useful) only for “some” (unspecified) cross compiler
configuration.
> 
>> 
>>> Krzysztof suggested much the same thing that I think you are
currently
>>> doing, which is deliberately configure a default triple but exclude
the
>>> corresponding backend.
>> 
>> You and Takumi were considering this as an unsupported configuration
>> before, and I tend to agree with that (this is the configuration I’m
using
>> for our tests but it was not intentional to leave the default triple
>> unset).
> 
> Right, intuitively it doesn't make sense.  Is it actually useful to
build a
> GPU compiler that will crash unless you ask it to generate GPU code? Seems
> to me it should default to producing GPU code.
Correct me if I’m wrong:

You’re viewing this from the “clang” point of view. A default triple is needed
because the command line interface does not require to specify it.

I see LLVM as a library or compiler *framework* in the first place, and clang is
just a use case as another.

When you build a compiler using LLVM as a library: 1) it does not have to be a
command line compiler, 2) the interface does not have to make optional the
target selection

Most GPU compilers are embedded in the driver (they compile shaders on-demand
during host program execution). The driver can detect the hardware and
initialize LLVM with the right triple.

We build LLVM as a shared library, we then build multiple compiler that will
link to this library to CodeGen to various backend. The compiler is responsible
to select and initialize the appropriate backend, we *never* rely on the default
triple, and I don’t even see how we could.

You could also see LLVM as a system library that can have multiple clients, each
client responsible of its own initialization.


>>> I expect we can detect that situation in lit.cfg
>>> and exclude tests on that basis, rather than 'native'.  It
would solve
>>> the problem for my case (host triple != target triple, although the
arch
>>> parts of the triple do match) and the "normal"
cross-compiler case (e.g.
>>> host = X86, backend + target triple = ARM).
>>> 
>>> I'm going to play around with that and see what I can do to
make it
>> work.
>>> 
>>>> 
>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> IMO, the problem is in general about tests that are
written without
>>>>>>> specifying a triple, that will be executed with the
default triple.
>>>>>>> 
>>>>>>> Most of these tests were written with X86 (or ARM)
in mind, and
>> there
>>>> is
>>>>>>> no guarantee that they will behave as intended with
every possible
>>>> triple.
>>>>>>> The DataLayout for instance has to be the one from
the target, and
>> is
>>>> not
>>>>>>> portable.
>>>>>>> I think a "portable backend test” is pretty
rare in general.
>>>>>> 
>>>>>> It depends on what the test is trying to do.  I'm
sure it is quite
>>>> common
>>>>>> for IR tests to behave essentially the same way
regardless of target.
>>>>> 
>>>>> IR tests != backend test (I may miss your point here, it’s
late…).
>>> 
>>> Right, sorry, lost focus for a moment there... nevertheless it is
still
>>> the case that many tests exercise functionality that is not
particularly
>>> target-centric and these should be run for any target that actually
>>> supports that functionality.  For example, the DebugInfo tests
should
>>> be run for any target that supports emitting debug info.
>> 
>> I’m not sure that “debug info” support is all or nothing.
>> As an extreme example, I know targets that support debug info but do
not
>> support function calls, what if your "debug info” test involves
these?
> 
> Then as part of getting the test suite to work for you, you would need to
> disable that particular test for your target.  It sounds like this kind of
> thing is exactly what the Hexagon folks did, and it seems quite reasonable.
> (And in fact I see two DebugInfo tests marked XFAIL: hexagon.)
It seems conceptually wrong to me, for the reason I already exposed.
It should go the other way (whitelist instead of blacklist)
> 
>> 
>> Also, I’m not a DebugInfo expert, but when a front-end generated them,
>> aren’t they dependent on the DataLayout? Hence the target?
> 
> Not really. DebugInfo tests primarily care what the DWARF description
> looks like, not so much what the generated code looks like,
My question is less will the CHECK matches than “will the backend be able to
generate code with invalid debug information (ex: pointer size, etc.) or just
crash?”

> and there
> are 100 or so DebugInfo tests that work across lots of targets.
> 
> Many DebugInfo tests are in target-dependent directories as artifacts of
> how the tests are implemented, rather than because they are truly
> target-dependent. The ones I've tripped over tend to be target-specific
> because they generate assembler and the assembler syntax varies too much
> across targets to be worth making the CHECKs tolerant enough.
> 
> There are fine details that would be DataLayout dependent, but 99% of
> the DebugInfo tests aren't checking things that are at such a low
level.
> 
>> It means that many DebugInfo test could fail with a backend that would
>> have a significantly different DataLayout and won’t expect some stuff
the
>> way they are played down.
>> 
>> 
>>> Whether a
>>> target supports debug info is orthogonal to its native-ness. As
written
>>> the top-level DebugInfo tests should work fine as long as the
default
>>> triple's backend is included, and that backend supports debug
info.
>>> 
>>> If your backend doesn't support debug info, then it's
reasonable to
>>> exclude those from your testing; but we can't do that at the
cost of
>>> excluding those from testing other backends that *do* support the
>> feature,
>>> even if that testing runs in a cross-compiler environment.
>>> 
>>> In this particular example, we'd be setting things up so that
DebugInfo
>>> is excluded for the wrong reason (not based on some more abstract
idea
>>> of the feature-set of the target) but as Krzysztof observes,
getting
>>> a feature-oriented set of conditions would be a huge task.
>> 
>> Agree: the predicate I wrote is not correct, and I don’t see a trivial
>> “good” solution. This is why any compromise or intermediate solution
that
>> is good enough to support whatever use-case people here have, including
>> your, seems appropriate to implement.
>> 
>> Let me know I can help coding something.
> 
> I have experimented with implementing the thing Takumi and I think should
> be a configuration error. :-)  Basically it takes the same kind of approach
> that I did in D12506, except it checks for the existence of the target that
> matches the default triple. If that target exists then 'llc' with
no triple
> will succeed, and it looks like the bulk of the tests that you disabled are
> in that category.  I'm not especially happy about this tactic, though.
Why aren’t you happy about that?
> 
> The Hexagon precedent is interesting; Krzysztof said they set the default
> triple, and didn't have to xfail all that much stuff.  Searching the
tree,
> I find exactly 7 individual tests marked XFAIL: hexagon, plus it disables 
> all of ExecutionEngine, and turns off the 'object-emission'
feature.
> 
> I'm curious if you would try setting the default triple to match your
> target, and see what /kinds/ of tests fail.  The raw number is much less
> interesting than in the categories.
Failing tests attached, let me know which ones you’d like me to investigate.

— 
Mehdi