llvm dev - Sep 2015 - Testing "normal" cross-compilers versus GPU backends

This note arose from http://reviews.llvm.org/D12506 but the reviewers
felt that we needed a broader audience, because the proposed patch
really didn't solve the entire problem and we had no better ideas.

Mehdi Amini needs to build LLVM with just a GPU backend, and still have
"ninja check" Just Work.  Commits r243958-243960 tried to accomplish
that; however they are too big a hammer, and cause much simpler cross
environments (like mine) to exclude big chunks of very useful tests
(including my favorite, DebugInfo).

FYI, my main cross environment is building on X86 Windows but using
a default target triple for PS4 (which is also X86).

I experimented with building LLVM with just the ARM backend (running on
an X86 workstation) and setting the default triple to some ARM value.
"ninja check" worked fine (without Mehdi's series of commits), so
the
normal kind of cross-compiler environment seems to be happy with how 
things were set up originally.

Mehdi reports building LLVM with the X86 and AMDGPU backends, setting
the default triple to "amdgcn--amdhsa", and getting 200-some failures.

(This does make me wonder about AMDGPU testing in general; how does that
work?  The only places I see lit checks for AMDGPU are in the usual
target-dependent places.)


Mehdi's solution was:
- In lit.cfg, change the existing "native" feature definition from
  "host-triple == target-triple" to also check "and the
corresponding
  backend is included."(**)
- Make piles of tests that seemed inapplicable to GPUs depend on the
  "native" feature (through REQUIRES: or in the lit.local.cfg).
- Build LLVM with just the GPU backend, and not set a target triple
  (i.e., it's set to the host triple, typically an X86-something).(*)
  Thus lit.cfg sees matching host and target triples, but the X86
  backend is missing, and so the "native" feature is not set.

[The "native" feature was invented to identify environments where
JIT would work properly. The "host-triple == target-triple" condition
isn't exactly right, but it works well enough.]

The major problem is that these new "native" dependencies are
incorrect.
For example the DebugInfo tests don't really require it; they work fine
as long as the default triple has the corresponding backend included,
as my ARM-on-X86 experiment demonstrated.

So, we need some solution that Mehdi can use in a GPU-backend-testing 
workflow, but that does not so drastically interfere with normal 
cross-compiler testing.  Does anybody have any suggestions?

Thanks,
--paulr


(*) This configuration seems inherently wrong; the default target triple
ought to require having the corresponding backend included, I think.
(**) Which makes this modification superfluous.

Hi Paul,

Thanks for the summary!
> On Sep 2, 2015, at 5:44 PM, Robinson, Paul <Paul_Robinson at
playstation.sony.com> wrote:
> 
> This note arose from
https://urldefense.proofpoint.com/v2/url?u=http-3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8eo6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=
but the reviewers
> felt that we needed a broader audience, because the proposed patch
> really didn't solve the entire problem and we had no better ideas.
> 
> Mehdi Amini needs to build LLVM with just a GPU backend, and still have
> "ninja check" Just Work.  Commits r243958-243960 tried to
accomplish
> that; however they are too big a hammer, and cause much simpler cross
> environments (like mine) to exclude big chunks of very useful tests
> (including my favorite, DebugInfo).
> 
> FYI, my main cross environment is building on X86 Windows but using
> a default target triple for PS4 (which is also X86).
> 
> I experimented with building LLVM with just the ARM backend (running on
> an X86 workstation) and setting the default triple to some ARM value.
> "ninja check" worked fine (without Mehdi's series of
commits), so the
> normal kind of cross-compiler environment seems to be happy with how 
> things were set up originally.
> 
> Mehdi reports building LLVM with the X86 and AMDGPU backends, setting
> the default triple to "amdgcn--amdhsa", and getting 200-some
failures.
> 
> (This does make me wonder about AMDGPU testing in general; how does that
> work?  The only places I see lit checks for AMDGPU are in the usual
> target-dependent places.)
I don’t understand this interrogation about how do you do testing in general.
The same way you don’t process tests/CodeGen/X86/* with the ARM backend, you
can’t process any random IR through these backends.

IMO, the problem is in general about tests that are written without specifying a
triple, that will be executed with the default triple.

Most of these tests were written with X86 (or ARM) in mind, and there is no
guarantee that they will behave as intended with every possible triple.
The DataLayout for instance has to be the one from the target, and is not
portable.
I think a "portable backend test” is pretty rare in general.

You can run all the tests by setting the default triple to X86 and compiling in
the X86 backend, but that’s just a trick to make the tests happy.
Alternatively, and this is what I tried to do, blacklisting these tests that
“pretends” to be “portable” but are not.

> Mehdi's solution was:
> - In lit.cfg, change the existing "native" feature definition
from
>  "host-triple == target-triple" to also check "and the
corresponding
>  backend is included."(**)
I agree that we could remove the condition on the host backend included and
deemed it a unsupported, but the build system needs to reject this
configuration.

> - Make piles of tests that seemed inapplicable to GPUs depend on the
>  "native" feature (through REQUIRES: or in the lit.local.cfg).
Nitpick: the GPU is just an example, any other backends can be affected.
It seems that these test are “lying” about the target they will be able to run
on (like if they would run on “anything”).

> - Build LLVM with just the GPU backend, and not set a target triple
>  (i.e., it's set to the host triple, typically an X86-something).(*)
>  Thus lit.cfg sees matching host and target triples, but the X86
>  backend is missing, and so the "native" feature is not set.
> 
> [The "native" feature was invented to identify environments where
> JIT would work properly. The "host-triple == target-triple"
condition
> isn't exactly right, but it works well enough.]
> 
> The major problem is that these new "native" dependencies are
incorrect.
> For example the DebugInfo tests don't really require it; they work fine
> as long as the default triple has the corresponding backend included,
> as my ARM-on-X86 experiment demonstrated.
Are they are guarantee’d to work with a default triple set to any of the
possible (in-tree) backend?
(I don’t know enough about these tests, which is definitively why I included
them in the “big hammer” solution)

Thanks,

— 
Mehdi

> -----Original Message-----
> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
> Sent: Wednesday, September 02, 2015 7:10 PM
> To: Robinson, Paul
> Cc: llvm-dev at lists.llvm.org; tom at stellard.net; NAKAMURA Takumi
> Subject: Re: Testing "normal" cross-compilers versus GPU backends
> 
> Hi Paul,
> 
> Thanks for the summary!
> 
> > On Sep 2, 2015, at 5:44 PM, Robinson, Paul
> <Paul_Robinson at playstation.sony.com> wrote:
> >
> > This note arose from https://urldefense.proofpoint.com/v2/url?u=http-
>
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
> o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but
the
> reviewers
> > felt that we needed a broader audience, because the proposed patch
> > really didn't solve the entire problem and we had no better ideas.
> >
> > Mehdi Amini needs to build LLVM with just a GPU backend, and still
have
> > "ninja check" Just Work.  Commits r243958-243960 tried to
accomplish
> > that; however they are too big a hammer, and cause much simpler cross
> > environments (like mine) to exclude big chunks of very useful tests
> > (including my favorite, DebugInfo).
> >
> > FYI, my main cross environment is building on X86 Windows but using
> > a default target triple for PS4 (which is also X86).
> >
> > I experimented with building LLVM with just the ARM backend (running
on
> > an X86 workstation) and setting the default triple to some ARM value.
> > "ninja check" worked fine (without Mehdi's series of
commits), so the
> > normal kind of cross-compiler environment seems to be happy with how
> > things were set up originally.
> >
> > Mehdi reports building LLVM with the X86 and AMDGPU backends, setting
> > the default triple to "amdgcn--amdhsa", and getting 200-some
failures.
> >
> > (This does make me wonder about AMDGPU testing in general; how does
that
> > work?  The only places I see lit checks for AMDGPU are in the usual
> > target-dependent places.)
> 
> I don’t understand this interrogation about how do you do testing in
> general. The same way you don’t process tests/CodeGen/X86/* with the ARM
> backend, you can’t process any random IR through these backends.
You said you had 200+ failures with AMDGPU.  Are the AMD folks simply
tolerating the 200 failures, and you don't want to?  I should hope there
is more to it than that.
> 
> IMO, the problem is in general about tests that are written without
> specifying a triple, that will be executed with the default triple.
> 
> Most of these tests were written with X86 (or ARM) in mind, and there is
> no guarantee that they will behave as intended with every possible triple.
> The DataLayout for instance has to be the one from the target, and is not
> portable.
> I think a "portable backend test” is pretty rare in general.
It depends on what the test is trying to do.  I'm sure it is quite common
for IR tests to behave essentially the same way regardless of target.
We have lots of tests (the ones you chose to mark "native") that had
been
working fine with ARM, X86, PPC, SPARC, and whatever.

The fact that they don't work with your backend is different from saying
those tests can't possibly work for any cross-compiler.  But the latter
is what your patch implemented, and it is preventing useful testing.
--paulr
> 
> You can run all the tests by setting the default triple to X86 and
> compiling in the X86 backend, but that’s just a trick to make the tests
> happy.
> Alternatively, and this is what I tried to do, blacklisting these tests
> that “pretends” to be “portable” but are not.
> 
> 
> > Mehdi's solution was:
> > - In lit.cfg, change the existing "native" feature
definition from
> >  "host-triple == target-triple" to also check "and the
corresponding
> >  backend is included."(**)
> 
> I agree that we could remove the condition on the host backend included
> and deemed it a unsupported, but the build system needs to reject this
> configuration.
> 
> 
> > - Make piles of tests that seemed inapplicable to GPUs depend on the
> >  "native" feature (through REQUIRES: or in the
lit.local.cfg).
> 
> Nitpick: the GPU is just an example, any other backends can be affected.
> It seems that these test are “lying” about the target they will be able to
> run on (like if they would run on “anything”).
> 
> 
> > - Build LLVM with just the GPU backend, and not set a target triple
> >  (i.e., it's set to the host triple, typically an
X86-something).(*)
> >  Thus lit.cfg sees matching host and target triples, but the X86
> >  backend is missing, and so the "native" feature is not set.
> >
> > [The "native" feature was invented to identify environments
where
> > JIT would work properly. The "host-triple == target-triple"
condition
> > isn't exactly right, but it works well enough.]
> >
> > The major problem is that these new "native" dependencies
are incorrect.
> > For example the DebugInfo tests don't really require it; they work
fine
> > as long as the default triple has the corresponding backend included,
> > as my ARM-on-X86 experiment demonstrated.
> 
> Are they are guarantee’d to work with a default triple set to any of the
> possible (in-tree) backend?
> (I don’t know enough about these tests, which is definitively why I
> included them in the “big hammer” solution)
> 
> Thanks,
> 
> —
> Mehdi

The Hexagon compiler (that we ship) only comes as a cross-compiler, and 
we don't include any other targets in it.  We set the default target 
triple to hexagon-..., and the "make check" works.  (I think we may
have
had to XFAIL some tests, but it was not widespread.)

The problem in the AMDGPU case seems to be that there are some features 
that the backed does not support (will not support?), and any test that 
relies on them will fail.  In theory, each target could have a 
corresponding set of limitations, and that set could be different for 
each such target.  The theoretical solution would then be to identify 
the set of all features and annotate each test (that does not specify 
the target triple) to list all the required ones.  Each target could 
mark some features as "unsupported" and then only the fully supported 
tests would run.  In practice this is a lot of work and I have no idea 
if this problem is common enough to warrant it.

If this is specific to AMDGPU, then maybe the solution could also be 
specific?  For example, the tests that are not expected to work on that 
target could just be XFAILed with a comment explaining it.  It's ugly, 
but it's quick.

-Krzysztof


On 9/2/2015 7:44 PM, Robinson, Paul via llvm-dev wrote:
> This note arose from http://reviews.llvm.org/D12506 but the reviewers
> felt that we needed a broader audience, because the proposed patch
> really didn't solve the entire problem and we had no better ideas.
>
> Mehdi Amini needs to build LLVM with just a GPU backend, and still have
> "ninja check" Just Work.  Commits r243958-243960 tried to
accomplish
> that; however they are too big a hammer, and cause much simpler cross
> environments (like mine) to exclude big chunks of very useful tests
> (including my favorite, DebugInfo).
>
> FYI, my main cross environment is building on X86 Windows but using
> a default target triple for PS4 (which is also X86).
>
> I experimented with building LLVM with just the ARM backend (running on
> an X86 workstation) and setting the default triple to some ARM value.
> "ninja check" worked fine (without Mehdi's series of
commits), so the
> normal kind of cross-compiler environment seems to be happy with how
> things were set up originally.
>
> Mehdi reports building LLVM with the X86 and AMDGPU backends, setting
> the default triple to "amdgcn--amdhsa", and getting 200-some
failures.
>
> (This does make me wonder about AMDGPU testing in general; how does that
> work?  The only places I see lit checks for AMDGPU are in the usual
> target-dependent places.)
>
>
> Mehdi's solution was:
> - In lit.cfg, change the existing "native" feature definition
from
>    "host-triple == target-triple" to also check "and the
corresponding
>    backend is included."(**)
> - Make piles of tests that seemed inapplicable to GPUs depend on the
>    "native" feature (through REQUIRES: or in the lit.local.cfg).
> - Build LLVM with just the GPU backend, and not set a target triple
>    (i.e., it's set to the host triple, typically an X86-something).(*)
>    Thus lit.cfg sees matching host and target triples, but the X86
>    backend is missing, and so the "native" feature is not set.
>
> [The "native" feature was invented to identify environments where
> JIT would work properly. The "host-triple == target-triple"
condition
> isn't exactly right, but it works well enough.]
>
> The major problem is that these new "native" dependencies are
incorrect.
> For example the DebugInfo tests don't really require it; they work fine
> as long as the default triple has the corresponding backend included,
> as my ARM-on-X86 experiment demonstrated.
>
> So, we need some solution that Mehdi can use in a GPU-backend-testing
> workflow, but that does not so drastically interfere with normal
> cross-compiler testing.  Does anybody have any suggestions?
>
> Thanks,
> --paulr
>
>
> (*) This configuration seems inherently wrong; the default target triple
> ought to require having the corresponding backend included, I think.
> (**) Which makes this modification superfluous.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, 
hosted by The Linux Foundation

On 9/2/2015 7:44 PM, Robinson, Paul via llvm-dev wrote:
> - Build LLVM with just the GPU backend, and not set a target triple
>    (i.e., it's set to the host triple, typically an X86-something).(*)
>    Thus lit.cfg sees matching host and target triples, but the X86
>    backend is missing, and so the "native" feature is not set.
>
> (*) This configuration seems inherently wrong; the default target triple
> ought to require having the corresponding backend included, I think.
I'm not very familiar with the "native" feature, so I'm not
sure what
the impact is there, but it seems like setting the default target triple 
to something that does not exist could be a way for a target to require 
that the proper triple is provided explicitly.  This way any test that 
does not explicitly contain a matching triple would be ignored on that 
target.
This would still exclude large number of potentially useful tests, but 
maybe it's a limitation of the testing that the target would be willing 
to tolerate?

-Krzysztof

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, 
hosted by The Linux Foundation

Relatedly, I'd really like it if there was some way to run all of those
generic tests for *more* than just the default triple. It's quite
unfortunate that you cannot easily run, say, the DebugInfo tests for both X86
and Sparc at one go.

It'd be great if there was some sort of 'list of triples to run the
generic tests on' build option. If that existed, it'd solve both my wish
(to be able to run the tests for multiple such triples -- ideally I think
it'd *default* to running them on each configured-in target!), and
Mehdi's desire to not run them at all.

On Fri, Sep 4, 2015 at 7:37 AM, James Y Knight via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Relatedly, I'd really like it if there was some way to run all of those
> generic tests for *more* than just the default triple. It's quite
> unfortunate that you cannot easily run, say, the DebugInfo tests for both
> X86 and Sparc at one go.
>
> It'd be great if there was some sort of 'list of triples to run the
> generic tests on' build option. If that existed, it'd solve both my
wish
> (to be able to run the tests for multiple such triples -- ideally I think
> it'd *default* to running them on each configured-in target!),

(making it the default would make check-all a bit slow, I would imagine -
but yes, having an easy option "ninja check-all-llvm-targets" or
something
like that could be handy (maybe this could generalize to Clang tests too,
not sure - many of them don't have a fixed triple and it'd be nice to be
able to reproduce failures from an apple buildbot on linux, etc))

> and Mehdi's desire to not run them at all.
>
Just by setting the set of targets to empty? Makes sense.

> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150904/61878e8e/attachment.html>