llvm dev - Sep 2015 - Testing "normal" cross-compilers versus GPU backends

On Thu, Sep 03, 2015 at 02:07:54AM -0700, Mehdi Amini
wrote:
> 
> > On Sep 3, 2015, at 12:18 AM, Robinson, Paul <Paul_Robinson at
playstation.sony.com> wrote:
> > 
> > 
> > 
> >> -----Original Message-----
> >> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
> >> Sent: Wednesday, September 02, 2015 7:10 PM
> >> To: Robinson, Paul
> >> Cc: llvm-dev at lists.llvm.org; tom at stellard.net; NAKAMURA
Takumi
> >> Subject: Re: Testing "normal" cross-compilers versus GPU
backends
> >> 
> >> Hi Paul,
> >> 
> >> Thanks for the summary!
> >> 
> >>> On Sep 2, 2015, at 5:44 PM, Robinson, Paul
> >> <Paul_Robinson at playstation.sony.com> wrote:
> >>> 
> >>> This note arose from
https://urldefense.proofpoint.com/v2/url?u=http-
> >>
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
> >>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
> >>
o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but the
> >> reviewers
> >>> felt that we needed a broader audience, because the proposed
patch
> >>> really didn't solve the entire problem and we had no
better ideas.
> >>> 
> >>> Mehdi Amini needs to build LLVM with just a GPU backend, and
still have
> >>> "ninja check" Just Work.  Commits r243958-243960
tried to accomplish
> >>> that; however they are too big a hammer, and cause much
simpler cross
> >>> environments (like mine) to exclude big chunks of very useful
tests
> >>> (including my favorite, DebugInfo).
> >>> 
> >>> FYI, my main cross environment is building on X86 Windows but
using
> >>> a default target triple for PS4 (which is also X86).
> >>> 
> >>> I experimented with building LLVM with just the ARM backend
(running on
> >>> an X86 workstation) and setting the default triple to some ARM
value.
> >>> "ninja check" worked fine (without Mehdi's
series of commits), so the
> >>> normal kind of cross-compiler environment seems to be happy
with how
> >>> things were set up originally.
> >>> 
> >>> Mehdi reports building LLVM with the X86 and AMDGPU backends,
setting
> >>> the default triple to "amdgcn--amdhsa", and getting
200-some failures.
> >>> 
> >>> (This does make me wonder about AMDGPU testing in general; how
does that
> >>> work?  The only places I see lit checks for AMDGPU are in the
usual
> >>> target-dependent places.)
> >> 
> >> I don’t understand this interrogation about how do you do testing
in
> >> general. The same way you don’t process tests/CodeGen/X86/* with
the ARM
> >> backend, you can’t process any random IR through these backends.
> > 
> > You said you had 200+ failures with AMDGPU.  Are the AMD folks simply
> > tolerating the 200 failures, and you don't want to?  I should hope
there
> > is more to it than that.
> 
> Well, I don’t know, they might just run `ninja check` with the default
triple set to X86?
> (which I would consider being working around a buggy test suite)
> 
I always enable AMDGPU and X86 when I build, so I've never run into
this problem.

-Tom

> > 
> >> 
> >> IMO, the problem is in general about tests that are written
without
> >> specifying a triple, that will be executed with the default
triple.
> >> 
> >> Most of these tests were written with X86 (or ARM) in mind, and
there is
> >> no guarantee that they will behave as intended with every possible
triple.
> >> The DataLayout for instance has to be the one from the target, and
is not
> >> portable.
> >> I think a "portable backend test” is pretty rare in general.
> > 
> > It depends on what the test is trying to do.  I'm sure it is quite
common
> > for IR tests to behave essentially the same way regardless of target.
> 
> IR tests != backend test (I may miss your point here, it’s late…).
> 
> 
> > We have lots of tests (the ones you chose to mark "native")
that had been
> > working fine with ARM, X86, PPC, SPARC, and whatever.
> > 
> > The fact that they don't work with your backend is different from
saying
> > those tests can't possibly work for any cross-compiler.
> 
> I believe this is far from what I said, or at least from what I had in
mind.
> 
> >  But the latter is what your patch implemented, and it is preventing
useful testing.
> 
> I don’t disagree with that statement, and I agree that it should be fixed.
> It doesn’t mean that I think the previous situation was better though. 
> The current over-conservative state seems at least more correct than having
only a subset of the targets that can pass `ninja check`.
> 
> Are you attending the LLVM bay area social tonight by any chance?
> 
> Best,
> 
> — 
> Mehdi
> 
> 
> 
> > --paulr
> > 
> >> 
> >> You can run all the tests by setting the default triple to X86 and
> >> compiling in the X86 backend, but that’s just a trick to make the
tests
> >> happy.
> >> Alternatively, and this is what I tried to do, blacklisting these
tests
> >> that “pretends” to be “portable” but are not.
> >> 
> >> 
> >>> Mehdi's solution was:
> >>> - In lit.cfg, change the existing "native" feature
definition from
> >>> "host-triple == target-triple" to also check
"and the corresponding
> >>> backend is included."(**)
> >> 
> >> I agree that we could remove the condition on the host backend
included
> >> and deemed it a unsupported, but the build system needs to reject
this
> >> configuration.
> >> 
> >> 
> >>> - Make piles of tests that seemed inapplicable to GPUs depend
on the
> >>> "native" feature (through REQUIRES: or in the
lit.local.cfg).
> >> 
> >> Nitpick: the GPU is just an example, any other backends can be
affected.
> >> It seems that these test are “lying” about the target they will be
able to
> >> run on (like if they would run on “anything”).
> >> 
> >> 
> >>> - Build LLVM with just the GPU backend, and not set a target
triple
> >>> (i.e., it's set to the host triple, typically an
X86-something).(*)
> >>> Thus lit.cfg sees matching host and target triples, but the
X86
> >>> backend is missing, and so the "native" feature is
not set.
> >>> 
> >>> [The "native" feature was invented to identify
environments where
> >>> JIT would work properly. The "host-triple ==
target-triple" condition
> >>> isn't exactly right, but it works well enough.]
> >>> 
> >>> The major problem is that these new "native"
dependencies are incorrect.
> >>> For example the DebugInfo tests don't really require it;
they work fine
> >>> as long as the default triple has the corresponding backend
included,
> >>> as my ARM-on-X86 experiment demonstrated.
> >> 
> >> Are they are guarantee’d to work with a default triple set to any
of the
> >> possible (in-tree) backend?
> >> (I don’t know enough about these tests, which is definitively why
I
> >> included them in the “big hammer” solution)
> >> 
> >> Thanks,
> >> 
> >> —
> >> Mehdi
>

> -----Original Message-----
> From: Tom Stellard [mailto:tom at stellard.net]
> Sent: Thursday, September 03, 2015 7:31 AM
> To: Mehdi Amini
> Cc: Robinson, Paul; llvm-dev at lists.llvm.org; NAKAMURA Takumi
> Subject: Re: Testing "normal" cross-compilers versus GPU backends
> 
> On Thu, Sep 03, 2015 at 02:07:54AM -0700, Mehdi Amini wrote:
> >
> > > On Sep 3, 2015, at 12:18 AM, Robinson, Paul
> <Paul_Robinson at playstation.sony.com> wrote:
> > >
> > >
> > >
> > >> -----Original Message-----
> > >> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
> > >> Sent: Wednesday, September 02, 2015 7:10 PM
> > >> To: Robinson, Paul
> > >> Cc: llvm-dev at lists.llvm.org; tom at stellard.net; NAKAMURA
Takumi
> > >> Subject: Re: Testing "normal" cross-compilers
versus GPU backends
> > >>
> > >> Hi Paul,
> > >>
> > >> Thanks for the summary!
> > >>
> > >>> On Sep 2, 2015, at 5:44 PM, Robinson, Paul
> > >> <Paul_Robinson at playstation.sony.com> wrote:
> > >>>
> > >>> This note arose from
> https://urldefense.proofpoint.com/v2/url?u=http-
> > >>
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
> > >>
>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
> > >>
o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but
> the
> > >> reviewers
> > >>> felt that we needed a broader audience, because the
proposed patch
> > >>> really didn't solve the entire problem and we had no
better ideas.
> > >>>
> > >>> Mehdi Amini needs to build LLVM with just a GPU backend,
and still
> have
> > >>> "ninja check" Just Work.  Commits
r243958-243960 tried to accomplish
> > >>> that; however they are too big a hammer, and cause much
simpler
> cross
> > >>> environments (like mine) to exclude big chunks of very
useful tests
> > >>> (including my favorite, DebugInfo).
> > >>>
> > >>> FYI, my main cross environment is building on X86 Windows
but using
> > >>> a default target triple for PS4 (which is also X86).
> > >>>
> > >>> I experimented with building LLVM with just the ARM
backend (running
> on
> > >>> an X86 workstation) and setting the default triple to
some ARM
> value.
> > >>> "ninja check" worked fine (without Mehdi's
series of commits), so
> the
> > >>> normal kind of cross-compiler environment seems to be
happy with how
> > >>> things were set up originally.
> > >>>
> > >>> Mehdi reports building LLVM with the X86 and AMDGPU
backends,
> setting
> > >>> the default triple to "amdgcn--amdhsa", and
getting 200-some
> failures.
> > >>>
> > >>> (This does make me wonder about AMDGPU testing in
general; how does
> that
> > >>> work?  The only places I see lit checks for AMDGPU are in
the usual
> > >>> target-dependent places.)
> > >>
> > >> I don’t understand this interrogation about how do you do
testing in
> > >> general. The same way you don’t process tests/CodeGen/X86/*
with the
> ARM
> > >> backend, you can’t process any random IR through these
backends.
> > >
> > > You said you had 200+ failures with AMDGPU.  Are the AMD folks
simply
> > > tolerating the 200 failures, and you don't want to?  I should
hope
> there
> > > is more to it than that.
> >
> > Well, I don’t know, they might just run `ninja check` with the default
> triple set to X86?
> > (which I would consider being working around a buggy test suite)
> >
> 
> I always enable AMDGPU and X86 when I build, so I've never run into
> this problem.
> 
> -Tom
Tom, presumably your default target triple is X86-ish?  And the only
tests to exercise the AMDGPU backend are those that explicitly specify
a triple for AMDGPU?

Mehdi, assuming that's what Tom does, your stated goal was to be able to
run tests *without* including the X86 backend, so Tom's solution won't
work for you (restating just for confirmation).

Krzysztof suggested much the same thing that I think you are currently
doing, which is deliberately configure a default triple but exclude the
corresponding backend.  I expect we can detect that situation in lit.cfg
and exclude tests on that basis, rather than 'native'.  It would solve
the problem for my case (host triple != target triple, although the arch
parts of the triple do match) and the "normal" cross-compiler case
(e.g.
host = X86, backend + target triple = ARM).

I'm going to play around with that and see what I can do to make it work.
> 
> 
> > >
> > >>
> > >> IMO, the problem is in general about tests that are written
without
> > >> specifying a triple, that will be executed with the default
triple.
> > >>
> > >> Most of these tests were written with X86 (or ARM) in mind,
and there
> is
> > >> no guarantee that they will behave as intended with every
possible
> triple.
> > >> The DataLayout for instance has to be the one from the
target, and is
> not
> > >> portable.
> > >> I think a "portable backend test” is pretty rare in
general.
> > >
> > > It depends on what the test is trying to do.  I'm sure it is
quite
> common
> > > for IR tests to behave essentially the same way regardless of
target.
> >
> > IR tests != backend test (I may miss your point here, it’s late…).
Right, sorry, lost focus for a moment there... nevertheless it is still
the case that many tests exercise functionality that is not particularly
target-centric and these should be run for any target that actually 
supports that functionality.  For example, the DebugInfo tests should
be run for any target that supports emitting debug info.  Whether a
target supports debug info is orthogonal to its native-ness. As written
the top-level DebugInfo tests should work fine as long as the default
triple's backend is included, and that backend supports debug info.

If your backend doesn't support debug info, then it's reasonable to
exclude those from your testing; but we can't do that at the cost of
excluding those from testing other backends that *do* support the feature,
even if that testing runs in a cross-compiler environment.

In this particular example, we'd be setting things up so that DebugInfo
is excluded for the wrong reason (not based on some more abstract idea
of the feature-set of the target) but as Krzysztof observes, getting
a feature-oriented set of conditions would be a huge task.
> >
> >
> > > We have lots of tests (the ones you chose to mark
"native") that had
> been
> > > working fine with ARM, X86, PPC, SPARC, and whatever.
> > >
> > > The fact that they don't work with your backend is different
from
> saying
> > > those tests can't possibly work for any cross-compiler.
> >
> > I believe this is far from what I said, or at least from what I had in
> mind.
> >
> > >  But the latter is what your patch implemented, and it is
preventing
> useful testing.
> >
> > I don’t disagree with that statement, and I agree that it should be
> fixed.
> > It doesn’t mean that I think the previous situation was better though.
> > The current over-conservative state seems at least more correct than
> having only a subset of the targets that can pass `ninja check`.
> >
> > Are you attending the LLVM bay area social tonight by any chance?
Unfortunately no, I have a prior commitment.
--paulr
> >
> > Best,
> >
> > —
> > Mehdi
> >
> >
> >
> > > --paulr
> > >
> > >>
> > >> You can run all the tests by setting the default triple to
X86 and
> > >> compiling in the X86 backend, but that’s just a trick to make
the
> tests
> > >> happy.
> > >> Alternatively, and this is what I tried to do, blacklisting
these
> tests
> > >> that “pretends” to be “portable” but are not.
> > >>
> > >>
> > >>> Mehdi's solution was:
> > >>> - In lit.cfg, change the existing "native"
feature definition from
> > >>> "host-triple == target-triple" to also check
"and the corresponding
> > >>> backend is included."(**)
> > >>
> > >> I agree that we could remove the condition on the host
backend
> included
> > >> and deemed it a unsupported, but the build system needs to
reject
> this
> > >> configuration.
> > >>
> > >>
> > >>> - Make piles of tests that seemed inapplicable to GPUs
depend on the
> > >>> "native" feature (through REQUIRES: or in the
lit.local.cfg).
> > >>
> > >> Nitpick: the GPU is just an example, any other backends can
be
> affected.
> > >> It seems that these test are “lying” about the target they
will be
> able to
> > >> run on (like if they would run on “anything”).
> > >>
> > >>
> > >>> - Build LLVM with just the GPU backend, and not set a
target triple
> > >>> (i.e., it's set to the host triple, typically an
X86-something).(*)
> > >>> Thus lit.cfg sees matching host and target triples, but
the X86
> > >>> backend is missing, and so the "native" feature
is not set.
> > >>>
> > >>> [The "native" feature was invented to identify
environments where
> > >>> JIT would work properly. The "host-triple ==
target-triple"
> condition
> > >>> isn't exactly right, but it works well enough.]
> > >>>
> > >>> The major problem is that these new "native"
dependencies are
> incorrect.
> > >>> For example the DebugInfo tests don't really require
it; they work
> fine
> > >>> as long as the default triple has the corresponding
backend
> included,
> > >>> as my ARM-on-X86 experiment demonstrated.
> > >>
> > >> Are they are guarantee’d to work with a default triple set to
any of
> the
> > >> possible (in-tree) backend?
> > >> (I don’t know enough about these tests, which is definitively
why I
> > >> included them in the “big hammer” solution)
> > >>
> > >> Thanks,
> > >>
> > >> —
> > >> Mehdi
> >

On Thu, Sep 03, 2015 at 06:23:23PM +0000, Robinson, Paul
wrote:
> 
> 
> > -----Original Message-----
> > From: Tom Stellard [mailto:tom at stellard.net]
> > Sent: Thursday, September 03, 2015 7:31 AM
> > To: Mehdi Amini
> > Cc: Robinson, Paul; llvm-dev at lists.llvm.org; NAKAMURA Takumi
> > Subject: Re: Testing "normal" cross-compilers versus GPU
backends
> > 
> > On Thu, Sep 03, 2015 at 02:07:54AM -0700, Mehdi Amini wrote:
> > >
> > > > On Sep 3, 2015, at 12:18 AM, Robinson, Paul
> > <Paul_Robinson at playstation.sony.com> wrote:
> > > >
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
> > > >> Sent: Wednesday, September 02, 2015 7:10 PM
> > > >> To: Robinson, Paul
> > > >> Cc: llvm-dev at lists.llvm.org; tom at stellard.net;
NAKAMURA Takumi
> > > >> Subject: Re: Testing "normal" cross-compilers
versus GPU backends
> > > >>
> > > >> Hi Paul,
> > > >>
> > > >> Thanks for the summary!
> > > >>
> > > >>> On Sep 2, 2015, at 5:44 PM, Robinson, Paul
> > > >> <Paul_Robinson at playstation.sony.com> wrote:
> > > >>>
> > > >>> This note arose from
> > https://urldefense.proofpoint.com/v2/url?u=http-
> > > >>
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
> > > >>
> >
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
> > > >>
o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but
> > the
> > > >> reviewers
> > > >>> felt that we needed a broader audience, because the
proposed patch
> > > >>> really didn't solve the entire problem and we
had no better ideas.
> > > >>>
> > > >>> Mehdi Amini needs to build LLVM with just a GPU
backend, and still
> > have
> > > >>> "ninja check" Just Work.  Commits
r243958-243960 tried to accomplish
> > > >>> that; however they are too big a hammer, and cause
much simpler
> > cross
> > > >>> environments (like mine) to exclude big chunks of
very useful tests
> > > >>> (including my favorite, DebugInfo).
> > > >>>
> > > >>> FYI, my main cross environment is building on X86
Windows but using
> > > >>> a default target triple for PS4 (which is also X86).
> > > >>>
> > > >>> I experimented with building LLVM with just the ARM
backend (running
> > on
> > > >>> an X86 workstation) and setting the default triple
to some ARM
> > value.
> > > >>> "ninja check" worked fine (without
Mehdi's series of commits), so
> > the
> > > >>> normal kind of cross-compiler environment seems to
be happy with how
> > > >>> things were set up originally.
> > > >>>
> > > >>> Mehdi reports building LLVM with the X86 and AMDGPU
backends,
> > setting
> > > >>> the default triple to "amdgcn--amdhsa",
and getting 200-some
> > failures.
> > > >>>
> > > >>> (This does make me wonder about AMDGPU testing in
general; how does
> > that
> > > >>> work?  The only places I see lit checks for AMDGPU
are in the usual
> > > >>> target-dependent places.)
> > > >>
> > > >> I don’t understand this interrogation about how do you
do testing in
> > > >> general. The same way you don’t process
tests/CodeGen/X86/* with the
> > ARM
> > > >> backend, you can’t process any random IR through these
backends.
> > > >
> > > > You said you had 200+ failures with AMDGPU.  Are the AMD
folks simply
> > > > tolerating the 200 failures, and you don't want to?  I
should hope
> > there
> > > > is more to it than that.
> > >
> > > Well, I don’t know, they might just run `ninja check` with the
default
> > triple set to X86?
> > > (which I would consider being working around a buggy test suite)
> > >
> > 
> > I always enable AMDGPU and X86 when I build, so I've never run
into
> > this problem.
> > 
> > -Tom
> 
> Tom, presumably your default target triple is X86-ish?  And the only
> tests to exercise the AMDGPU backend are those that explicitly specify
> a triple for AMDGPU?
> 
Correct, the default triple is x86-ish, and all the AMDGPU backend tests
specify a triple.

-Tom
> Mehdi, assuming that's what Tom does, your stated goal was to be able
to
> run tests *without* including the X86 backend, so Tom's solution
won't
> work for you (restating just for confirmation).
> 
> Krzysztof suggested much the same thing that I think you are currently
> doing, which is deliberately configure a default triple but exclude the
> corresponding backend.  I expect we can detect that situation in lit.cfg
> and exclude tests on that basis, rather than 'native'.  It would
solve
> the problem for my case (host triple != target triple, although the arch
> parts of the triple do match) and the "normal" cross-compiler
case (e.g.
> host = X86, backend + target triple = ARM).
> 
> I'm going to play around with that and see what I can do to make it
work.
> 
> > 
> > 
> > > >
> > > >>
> > > >> IMO, the problem is in general about tests that are
written without
> > > >> specifying a triple, that will be executed with the
default triple.
> > > >>
> > > >> Most of these tests were written with X86 (or ARM) in
mind, and there
> > is
> > > >> no guarantee that they will behave as intended with
every possible
> > triple.
> > > >> The DataLayout for instance has to be the one from the
target, and is
> > not
> > > >> portable.
> > > >> I think a "portable backend test” is pretty rare in
general.
> > > >
> > > > It depends on what the test is trying to do.  I'm sure
it is quite
> > common
> > > > for IR tests to behave essentially the same way regardless
of target.
> > >
> > > IR tests != backend test (I may miss your point here, it’s
late…).
> 
> Right, sorry, lost focus for a moment there... nevertheless it is still
> the case that many tests exercise functionality that is not particularly
> target-centric and these should be run for any target that actually 
> supports that functionality.  For example, the DebugInfo tests should
> be run for any target that supports emitting debug info.  Whether a
> target supports debug info is orthogonal to its native-ness. As written
> the top-level DebugInfo tests should work fine as long as the default
> triple's backend is included, and that backend supports debug info.
> 
> If your backend doesn't support debug info, then it's reasonable to
> exclude those from your testing; but we can't do that at the cost of
> excluding those from testing other backends that *do* support the feature,
> even if that testing runs in a cross-compiler environment.
> 
> In this particular example, we'd be setting things up so that DebugInfo
> is excluded for the wrong reason (not based on some more abstract idea
> of the feature-set of the target) but as Krzysztof observes, getting
> a feature-oriented set of conditions would be a huge task.
> 
> > >
> > >
> > > > We have lots of tests (the ones you chose to mark
"native") that had
> > been
> > > > working fine with ARM, X86, PPC, SPARC, and whatever.
> > > >
> > > > The fact that they don't work with your backend is
different from
> > saying
> > > > those tests can't possibly work for any cross-compiler.
> > >
> > > I believe this is far from what I said, or at least from what I
had in
> > mind.
> > >
> > > >  But the latter is what your patch implemented, and it is
preventing
> > useful testing.
> > >
> > > I don’t disagree with that statement, and I agree that it should
be
> > fixed.
> > > It doesn’t mean that I think the previous situation was better
though.
> > > The current over-conservative state seems at least more correct
than
> > having only a subset of the targets that can pass `ninja check`.
> > >
> > > Are you attending the LLVM bay area social tonight by any chance?
> 
> Unfortunately no, I have a prior commitment.
> --paulr
> 
> > >
> > > Best,
> > >
> > > —
> > > Mehdi
> > >
> > >
> > >
> > > > --paulr
> > > >
> > > >>
> > > >> You can run all the tests by setting the default triple
to X86 and
> > > >> compiling in the X86 backend, but that’s just a trick to
make the
> > tests
> > > >> happy.
> > > >> Alternatively, and this is what I tried to do,
blacklisting these
> > tests
> > > >> that “pretends” to be “portable” but are not.
> > > >>
> > > >>
> > > >>> Mehdi's solution was:
> > > >>> - In lit.cfg, change the existing "native"
feature definition from
> > > >>> "host-triple == target-triple" to also
check "and the corresponding
> > > >>> backend is included."(**)
> > > >>
> > > >> I agree that we could remove the condition on the host
backend
> > included
> > > >> and deemed it a unsupported, but the build system needs
to reject
> > this
> > > >> configuration.
> > > >>
> > > >>
> > > >>> - Make piles of tests that seemed inapplicable to
GPUs depend on the
> > > >>> "native" feature (through REQUIRES: or in
the lit.local.cfg).
> > > >>
> > > >> Nitpick: the GPU is just an example, any other backends
can be
> > affected.
> > > >> It seems that these test are “lying” about the target
they will be
> > able to
> > > >> run on (like if they would run on “anything”).
> > > >>
> > > >>
> > > >>> - Build LLVM with just the GPU backend, and not set
a target triple
> > > >>> (i.e., it's set to the host triple, typically an
X86-something).(*)
> > > >>> Thus lit.cfg sees matching host and target triples,
but the X86
> > > >>> backend is missing, and so the "native"
feature is not set.
> > > >>>
> > > >>> [The "native" feature was invented to
identify environments where
> > > >>> JIT would work properly. The "host-triple ==
target-triple"
> > condition
> > > >>> isn't exactly right, but it works well enough.]
> > > >>>
> > > >>> The major problem is that these new
"native" dependencies are
> > incorrect.
> > > >>> For example the DebugInfo tests don't really
require it; they work
> > fine
> > > >>> as long as the default triple has the corresponding
backend
> > included,
> > > >>> as my ARM-on-X86 experiment demonstrated.
> > > >>
> > > >> Are they are guarantee’d to work with a default triple
set to any of
> > the
> > > >> possible (in-tree) backend?
> > > >> (I don’t know enough about these tests, which is
definitively why I
> > > >> included them in the “big hammer” solution)
> > > >>
> > > >> Thanks,
> > > >>
> > > >> —
> > > >> Mehdi
> > >

> On Sep 3, 2015, at 11:23 AM, Robinson, Paul <Paul_Robinson at
playstation.sony.com> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: Tom Stellard [mailto:tom at stellard.net]
>> Sent: Thursday, September 03, 2015 7:31 AM
>> To: Mehdi Amini
>> Cc: Robinson, Paul; llvm-dev at lists.llvm.org; NAKAMURA Takumi
>> Subject: Re: Testing "normal" cross-compilers versus GPU
backends
>> 
>> On Thu, Sep 03, 2015 at 02:07:54AM -0700, Mehdi Amini wrote:
>>> 
>>>> On Sep 3, 2015, at 12:18 AM, Robinson, Paul
>> <Paul_Robinson at playstation.sony.com> wrote:
>>>> 
>>>> 
>>>> 
>>>>> -----Original Message-----
>>>>> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
>>>>> Sent: Wednesday, September 02, 2015 7:10 PM
>>>>> To: Robinson, Paul
>>>>> Cc: llvm-dev at lists.llvm.org; tom at stellard.net;
NAKAMURA Takumi
>>>>> Subject: Re: Testing "normal" cross-compilers
versus GPU backends
>>>>> 
>>>>> Hi Paul,
>>>>> 
>>>>> Thanks for the summary!
>>>>> 
>>>>>> On Sep 2, 2015, at 5:44 PM, Robinson, Paul
>>>>> <Paul_Robinson at playstation.sony.com> wrote:
>>>>>> 
>>>>>> This note arose from
>> https://urldefense.proofpoint.com/v2/url?u=http-
>>>>>
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
>>>>> 
>>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
>>>>>
o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but
>> the
>>>>> reviewers
>>>>>> felt that we needed a broader audience, because the
proposed patch
>>>>>> really didn't solve the entire problem and we had
no better ideas.
>>>>>> 
>>>>>> Mehdi Amini needs to build LLVM with just a GPU
backend, and still
>> have
>>>>>> "ninja check" Just Work.  Commits
r243958-243960 tried to accomplish
>>>>>> that; however they are too big a hammer, and cause much
simpler
>> cross
>>>>>> environments (like mine) to exclude big chunks of very
useful tests
>>>>>> (including my favorite, DebugInfo).
>>>>>> 
>>>>>> FYI, my main cross environment is building on X86
Windows but using
>>>>>> a default target triple for PS4 (which is also X86).
>>>>>> 
>>>>>> I experimented with building LLVM with just the ARM
backend (running
>> on
>>>>>> an X86 workstation) and setting the default triple to
some ARM
>> value.
>>>>>> "ninja check" worked fine (without
Mehdi's series of commits), so
>> the
>>>>>> normal kind of cross-compiler environment seems to be
happy with how
>>>>>> things were set up originally.
>>>>>> 
>>>>>> Mehdi reports building LLVM with the X86 and AMDGPU
backends,
>> setting
>>>>>> the default triple to "amdgcn--amdhsa", and
getting 200-some
>> failures.
>>>>>> 
>>>>>> (This does make me wonder about AMDGPU testing in
general; how does
>> that
>>>>>> work?  The only places I see lit checks for AMDGPU are
in the usual
>>>>>> target-dependent places.)
>>>>> 
>>>>> I don’t understand this interrogation about how do you do
testing in
>>>>> general. The same way you don’t process tests/CodeGen/X86/*
with the
>> ARM
>>>>> backend, you can’t process any random IR through these
backends.
>>>> 
>>>> You said you had 200+ failures with AMDGPU.  Are the AMD folks
simply
>>>> tolerating the 200 failures, and you don't want to?  I
should hope
>> there
>>>> is more to it than that.
>>> 
>>> Well, I don’t know, they might just run `ninja check` with the
default
>> triple set to X86?
>>> (which I would consider being working around a buggy test suite)
>>> 
>> 
>> I always enable AMDGPU and X86 when I build, so I've never run into
>> this problem.
>> 
>> -Tom
> 
> Tom, presumably your default target triple is X86-ish?  And the only
> tests to exercise the AMDGPU backend are those that explicitly specify
> a triple for AMDGPU?
This is how we used to do it as well, and I assume this is how most of backends
out of the main CPUs are dealing with the situation.
This is what I consider a bug and was trying to solve (admittedly not in an
optimal way).
> Mehdi, assuming that's what Tom does, your stated goal was to be able
to
> run tests *without* including the X86 backend, so Tom's solution
won't
> work for you (restating just for confirmation).
Yes, or alternatively I expect the test suite to pass whatever the default
triple is.
> Krzysztof suggested much the same thing that I think you are currently
> doing, which is deliberately configure a default triple but exclude the
> corresponding backend.  
You and Takumi were considering this as an unsupported configuration before, and
I tend to agree with that (this is the configuration I’m using for our tests but
it was not intentional to leave the default triple unset).
> I expect we can detect that situation in lit.cfg
> and exclude tests on that basis, rather than 'native'.  It would
solve
> the problem for my case (host triple != target triple, although the arch
> parts of the triple do match) and the "normal" cross-compiler
case (e.g.
> host = X86, backend + target triple = ARM).
> 
> I'm going to play around with that and see what I can do to make it
work.
> 
>> 
>> 
>>>> 
>>>>> 
>>>>> IMO, the problem is in general about tests that are written
without
>>>>> specifying a triple, that will be executed with the default
triple.
>>>>> 
>>>>> Most of these tests were written with X86 (or ARM) in mind,
and there
>> is
>>>>> no guarantee that they will behave as intended with every
possible
>> triple.
>>>>> The DataLayout for instance has to be the one from the
target, and is
>> not
>>>>> portable.
>>>>> I think a "portable backend test” is pretty rare in
general.
>>>> 
>>>> It depends on what the test is trying to do.  I'm sure it
is quite
>> common
>>>> for IR tests to behave essentially the same way regardless of
target.
>>> 
>>> IR tests != backend test (I may miss your point here, it’s late…).
> 
> Right, sorry, lost focus for a moment there... nevertheless it is still
> the case that many tests exercise functionality that is not particularly
> target-centric and these should be run for any target that actually 
> supports that functionality.  For example, the DebugInfo tests should
> be run for any target that supports emitting debug info.
I’m not sure that “debug info” support is all or nothing. 
As an extreme example, I know targets that support debug info but do not support
function calls, what if your "debug info” test involves these?

Also, I’m not a DebugInfo expert, but when a front-end generated them, aren’t
they dependent on the DataLayout? Hence the target?
It means that many DebugInfo test could fail with a backend that would have a
significantly different DataLayout and won’t expect some stuff the way they are
played down.

>  Whether a
> target supports debug info is orthogonal to its native-ness. As written
> the top-level DebugInfo tests should work fine as long as the default
> triple's backend is included, and that backend supports debug info.
> 
> If your backend doesn't support debug info, then it's reasonable to
> exclude those from your testing; but we can't do that at the cost of
> excluding those from testing other backends that *do* support the feature,
> even if that testing runs in a cross-compiler environment.
> 
> In this particular example, we'd be setting things up so that DebugInfo
> is excluded for the wrong reason (not based on some more abstract idea
> of the feature-set of the target) but as Krzysztof observes, getting
> a feature-oriented set of conditions would be a huge task.
Agree: the predicate I wrote is not correct, and I don’t see a trivial “good”
solution. This is why any compromise or intermediate solution that is good
enough to support whatever use-case people here have, including your, seems
appropriate to implement.

Let me know I can help coding something.

— 
Mehdi

> 
>>> 
>>> 
>>>> We have lots of tests (the ones you chose to mark
"native") that had
>> been
>>>> working fine with ARM, X86, PPC, SPARC, and whatever.
>>>> 
>>>> The fact that they don't work with your backend is
different from
>> saying
>>>> those tests can't possibly work for any cross-compiler.
>>> 
>>> I believe this is far from what I said, or at least from what I had
in
>> mind.
>>> 
>>>> But the latter is what your patch implemented, and it is
preventing
>> useful testing.
>>> 
>>> I don’t disagree with that statement, and I agree that it should be
>> fixed.
>>> It doesn’t mean that I think the previous situation was better
though.
>>> The current over-conservative state seems at least more correct
than
>> having only a subset of the targets that can pass `ninja check`.
>>> 
>>> Are you attending the LLVM bay area social tonight by any chance?
> 
> Unfortunately no, I have a prior commitment.
> --paulr
> 
>>> 
>>> Best,
>>> 
>>> —
>>> Mehdi
>>> 
>>> 
>>> 
>>>> --paulr
>>>> 
>>>>> 
>>>>> You can run all the tests by setting the default triple to
X86 and
>>>>> compiling in the X86 backend, but that’s just a trick to
make the
>> tests
>>>>> happy.
>>>>> Alternatively, and this is what I tried to do, blacklisting
these
>> tests
>>>>> that “pretends” to be “portable” but are not.
>>>>> 
>>>>> 
>>>>>> Mehdi's solution was:
>>>>>> - In lit.cfg, change the existing "native"
feature definition from
>>>>>> "host-triple == target-triple" to also check
"and the corresponding
>>>>>> backend is included."(**)
>>>>> 
>>>>> I agree that we could remove the condition on the host
backend
>> included
>>>>> and deemed it a unsupported, but the build system needs to
reject
>> this
>>>>> configuration.
>>>>> 
>>>>> 
>>>>>> - Make piles of tests that seemed inapplicable to GPUs
depend on the
>>>>>> "native" feature (through REQUIRES: or in the
lit.local.cfg).
>>>>> 
>>>>> Nitpick: the GPU is just an example, any other backends can
be
>> affected.
>>>>> It seems that these test are “lying” about the target they
will be
>> able to
>>>>> run on (like if they would run on “anything”).
>>>>> 
>>>>> 
>>>>>> - Build LLVM with just the GPU backend, and not set a
target triple
>>>>>> (i.e., it's set to the host triple, typically an
X86-something).(*)
>>>>>> Thus lit.cfg sees matching host and target triples, but
the X86
>>>>>> backend is missing, and so the "native"
feature is not set.
>>>>>> 
>>>>>> [The "native" feature was invented to
identify environments where
>>>>>> JIT would work properly. The "host-triple ==
target-triple"
>> condition
>>>>>> isn't exactly right, but it works well enough.]
>>>>>> 
>>>>>> The major problem is that these new "native"
dependencies are
>> incorrect.
>>>>>> For example the DebugInfo tests don't really
require it; they work
>> fine
>>>>>> as long as the default triple has the corresponding
backend
>> included,
>>>>>> as my ARM-on-X86 experiment demonstrated.
>>>>> 
>>>>> Are they are guarantee’d to work with a default triple set
to any of
>> the
>>>>> possible (in-tree) backend?
>>>>> (I don’t know enough about these tests, which is
definitively why I
>>>>> included them in the “big hammer” solution)
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> —
>>>>> Mehdi
>>>