llvm dev - Sep 2015 - Testing "normal" cross-compilers versus GPU backends

> -----Original Message-----
> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
> Sent: Wednesday, September 02, 2015 7:10 PM
> To: Robinson, Paul
> Cc: llvm-dev at lists.llvm.org; tom at stellard.net; NAKAMURA Takumi
> Subject: Re: Testing "normal" cross-compilers versus GPU backends
> 
> Hi Paul,
> 
> Thanks for the summary!
> 
> > On Sep 2, 2015, at 5:44 PM, Robinson, Paul
> <Paul_Robinson at playstation.sony.com> wrote:
> >
> > This note arose from https://urldefense.proofpoint.com/v2/url?u=http-
>
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
> o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but
the
> reviewers
> > felt that we needed a broader audience, because the proposed patch
> > really didn't solve the entire problem and we had no better ideas.
> >
> > Mehdi Amini needs to build LLVM with just a GPU backend, and still
have
> > "ninja check" Just Work.  Commits r243958-243960 tried to
accomplish
> > that; however they are too big a hammer, and cause much simpler cross
> > environments (like mine) to exclude big chunks of very useful tests
> > (including my favorite, DebugInfo).
> >
> > FYI, my main cross environment is building on X86 Windows but using
> > a default target triple for PS4 (which is also X86).
> >
> > I experimented with building LLVM with just the ARM backend (running
on
> > an X86 workstation) and setting the default triple to some ARM value.
> > "ninja check" worked fine (without Mehdi's series of
commits), so the
> > normal kind of cross-compiler environment seems to be happy with how
> > things were set up originally.
> >
> > Mehdi reports building LLVM with the X86 and AMDGPU backends, setting
> > the default triple to "amdgcn--amdhsa", and getting 200-some
failures.
> >
> > (This does make me wonder about AMDGPU testing in general; how does
that
> > work?  The only places I see lit checks for AMDGPU are in the usual
> > target-dependent places.)
> 
> I don’t understand this interrogation about how do you do testing in
> general. The same way you don’t process tests/CodeGen/X86/* with the ARM
> backend, you can’t process any random IR through these backends.
You said you had 200+ failures with AMDGPU.  Are the AMD folks simply
tolerating the 200 failures, and you don't want to?  I should hope there
is more to it than that.
> 
> IMO, the problem is in general about tests that are written without
> specifying a triple, that will be executed with the default triple.
> 
> Most of these tests were written with X86 (or ARM) in mind, and there is
> no guarantee that they will behave as intended with every possible triple.
> The DataLayout for instance has to be the one from the target, and is not
> portable.
> I think a "portable backend test” is pretty rare in general.
It depends on what the test is trying to do.  I'm sure it is quite common
for IR tests to behave essentially the same way regardless of target.
We have lots of tests (the ones you chose to mark "native") that had
been
working fine with ARM, X86, PPC, SPARC, and whatever.

The fact that they don't work with your backend is different from saying
those tests can't possibly work for any cross-compiler.  But the latter
is what your patch implemented, and it is preventing useful testing.
--paulr
> 
> You can run all the tests by setting the default triple to X86 and
> compiling in the X86 backend, but that’s just a trick to make the tests
> happy.
> Alternatively, and this is what I tried to do, blacklisting these tests
> that “pretends” to be “portable” but are not.
> 
> 
> > Mehdi's solution was:
> > - In lit.cfg, change the existing "native" feature
definition from
> >  "host-triple == target-triple" to also check "and the
corresponding
> >  backend is included."(**)
> 
> I agree that we could remove the condition on the host backend included
> and deemed it a unsupported, but the build system needs to reject this
> configuration.
> 
> 
> > - Make piles of tests that seemed inapplicable to GPUs depend on the
> >  "native" feature (through REQUIRES: or in the
lit.local.cfg).
> 
> Nitpick: the GPU is just an example, any other backends can be affected.
> It seems that these test are “lying” about the target they will be able to
> run on (like if they would run on “anything”).
> 
> 
> > - Build LLVM with just the GPU backend, and not set a target triple
> >  (i.e., it's set to the host triple, typically an
X86-something).(*)
> >  Thus lit.cfg sees matching host and target triples, but the X86
> >  backend is missing, and so the "native" feature is not set.
> >
> > [The "native" feature was invented to identify environments
where
> > JIT would work properly. The "host-triple == target-triple"
condition
> > isn't exactly right, but it works well enough.]
> >
> > The major problem is that these new "native" dependencies
are incorrect.
> > For example the DebugInfo tests don't really require it; they work
fine
> > as long as the default triple has the corresponding backend included,
> > as my ARM-on-X86 experiment demonstrated.
> 
> Are they are guarantee’d to work with a default triple set to any of the
> possible (in-tree) backend?
> (I don’t know enough about these tests, which is definitively why I
> included them in the “big hammer” solution)
> 
> Thanks,
> 
> —
> Mehdi

> On Sep 3, 2015, at 12:18 AM, Robinson, Paul <Paul_Robinson at
playstation.sony.com> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
>> Sent: Wednesday, September 02, 2015 7:10 PM
>> To: Robinson, Paul
>> Cc: llvm-dev at lists.llvm.org; tom at stellard.net; NAKAMURA Takumi
>> Subject: Re: Testing "normal" cross-compilers versus GPU
backends
>> 
>> Hi Paul,
>> 
>> Thanks for the summary!
>> 
>>> On Sep 2, 2015, at 5:44 PM, Robinson, Paul
>> <Paul_Robinson at playstation.sony.com> wrote:
>>> 
>>> This note arose from
https://urldefense.proofpoint.com/v2/url?u=http-
>>
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
>>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
>> o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e= 
but the
>> reviewers
>>> felt that we needed a broader audience, because the proposed patch
>>> really didn't solve the entire problem and we had no better
ideas.
>>> 
>>> Mehdi Amini needs to build LLVM with just a GPU backend, and still
have
>>> "ninja check" Just Work.  Commits r243958-243960 tried to
accomplish
>>> that; however they are too big a hammer, and cause much simpler
cross
>>> environments (like mine) to exclude big chunks of very useful tests
>>> (including my favorite, DebugInfo).
>>> 
>>> FYI, my main cross environment is building on X86 Windows but using
>>> a default target triple for PS4 (which is also X86).
>>> 
>>> I experimented with building LLVM with just the ARM backend
(running on
>>> an X86 workstation) and setting the default triple to some ARM
value.
>>> "ninja check" worked fine (without Mehdi's series of
commits), so the
>>> normal kind of cross-compiler environment seems to be happy with
how
>>> things were set up originally.
>>> 
>>> Mehdi reports building LLVM with the X86 and AMDGPU backends,
setting
>>> the default triple to "amdgcn--amdhsa", and getting
200-some failures.
>>> 
>>> (This does make me wonder about AMDGPU testing in general; how does
that
>>> work?  The only places I see lit checks for AMDGPU are in the usual
>>> target-dependent places.)
>> 
>> I don’t understand this interrogation about how do you do testing in
>> general. The same way you don’t process tests/CodeGen/X86/* with the
ARM
>> backend, you can’t process any random IR through these backends.
> 
> You said you had 200+ failures with AMDGPU.  Are the AMD folks simply
> tolerating the 200 failures, and you don't want to?  I should hope
there
> is more to it than that.
Well, I don’t know, they might just run `ninja check` with the default triple
set to X86?
(which I would consider being working around a buggy test suite)
> 
>> 
>> IMO, the problem is in general about tests that are written without
>> specifying a triple, that will be executed with the default triple.
>> 
>> Most of these tests were written with X86 (or ARM) in mind, and there
is
>> no guarantee that they will behave as intended with every possible
triple.
>> The DataLayout for instance has to be the one from the target, and is
not
>> portable.
>> I think a "portable backend test” is pretty rare in general.
> 
> It depends on what the test is trying to do.  I'm sure it is quite
common
> for IR tests to behave essentially the same way regardless of target.
IR tests != backend test (I may miss your point here, it’s late…).

> We have lots of tests (the ones you chose to mark "native") that
had been
> working fine with ARM, X86, PPC, SPARC, and whatever.
> 
> The fact that they don't work with your backend is different from
saying
> those tests can't possibly work for any cross-compiler.
I believe this is far from what I said, or at least from what I had in mind.
>  But the latter is what your patch implemented, and it is preventing useful
testing.
I don’t disagree with that statement, and I agree that it should be fixed.
It doesn’t mean that I think the previous situation was better though. 
The current over-conservative state seems at least more correct than having only
a subset of the targets that can pass `ninja check`.

Are you attending the LLVM bay area social tonight by any chance?

Best,

— 
Mehdi


> --paulr
> 
>> 
>> You can run all the tests by setting the default triple to X86 and
>> compiling in the X86 backend, but that’s just a trick to make the tests
>> happy.
>> Alternatively, and this is what I tried to do, blacklisting these tests
>> that “pretends” to be “portable” but are not.
>> 
>> 
>>> Mehdi's solution was:
>>> - In lit.cfg, change the existing "native" feature
definition from
>>> "host-triple == target-triple" to also check "and
the corresponding
>>> backend is included."(**)
>> 
>> I agree that we could remove the condition on the host backend included
>> and deemed it a unsupported, but the build system needs to reject this
>> configuration.
>> 
>> 
>>> - Make piles of tests that seemed inapplicable to GPUs depend on
the
>>> "native" feature (through REQUIRES: or in the
lit.local.cfg).
>> 
>> Nitpick: the GPU is just an example, any other backends can be
affected.
>> It seems that these test are “lying” about the target they will be able
to
>> run on (like if they would run on “anything”).
>> 
>> 
>>> - Build LLVM with just the GPU backend, and not set a target triple
>>> (i.e., it's set to the host triple, typically an
X86-something).(*)
>>> Thus lit.cfg sees matching host and target triples, but the X86
>>> backend is missing, and so the "native" feature is not
set.
>>> 
>>> [The "native" feature was invented to identify
environments where
>>> JIT would work properly. The "host-triple ==
target-triple" condition
>>> isn't exactly right, but it works well enough.]
>>> 
>>> The major problem is that these new "native" dependencies
are incorrect.
>>> For example the DebugInfo tests don't really require it; they
work fine
>>> as long as the default triple has the corresponding backend
included,
>>> as my ARM-on-X86 experiment demonstrated.
>> 
>> Are they are guarantee’d to work with a default triple set to any of
the
>> possible (in-tree) backend?
>> (I don’t know enough about these tests, which is definitively why I
>> included them in the “big hammer” solution)
>> 
>> Thanks,
>> 
>> —
>> Mehdi

On Thu, Sep 03, 2015 at 02:07:54AM -0700, Mehdi Amini
wrote:
> 
> > On Sep 3, 2015, at 12:18 AM, Robinson, Paul <Paul_Robinson at
playstation.sony.com> wrote:
> > 
> > 
> > 
> >> -----Original Message-----
> >> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
> >> Sent: Wednesday, September 02, 2015 7:10 PM
> >> To: Robinson, Paul
> >> Cc: llvm-dev at lists.llvm.org; tom at stellard.net; NAKAMURA
Takumi
> >> Subject: Re: Testing "normal" cross-compilers versus GPU
backends
> >> 
> >> Hi Paul,
> >> 
> >> Thanks for the summary!
> >> 
> >>> On Sep 2, 2015, at 5:44 PM, Robinson, Paul
> >> <Paul_Robinson at playstation.sony.com> wrote:
> >>> 
> >>> This note arose from
https://urldefense.proofpoint.com/v2/url?u=http-
> >>
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
> >>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
> >>
o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but the
> >> reviewers
> >>> felt that we needed a broader audience, because the proposed
patch
> >>> really didn't solve the entire problem and we had no
better ideas.
> >>> 
> >>> Mehdi Amini needs to build LLVM with just a GPU backend, and
still have
> >>> "ninja check" Just Work.  Commits r243958-243960
tried to accomplish
> >>> that; however they are too big a hammer, and cause much
simpler cross
> >>> environments (like mine) to exclude big chunks of very useful
tests
> >>> (including my favorite, DebugInfo).
> >>> 
> >>> FYI, my main cross environment is building on X86 Windows but
using
> >>> a default target triple for PS4 (which is also X86).
> >>> 
> >>> I experimented with building LLVM with just the ARM backend
(running on
> >>> an X86 workstation) and setting the default triple to some ARM
value.
> >>> "ninja check" worked fine (without Mehdi's
series of commits), so the
> >>> normal kind of cross-compiler environment seems to be happy
with how
> >>> things were set up originally.
> >>> 
> >>> Mehdi reports building LLVM with the X86 and AMDGPU backends,
setting
> >>> the default triple to "amdgcn--amdhsa", and getting
200-some failures.
> >>> 
> >>> (This does make me wonder about AMDGPU testing in general; how
does that
> >>> work?  The only places I see lit checks for AMDGPU are in the
usual
> >>> target-dependent places.)
> >> 
> >> I don’t understand this interrogation about how do you do testing
in
> >> general. The same way you don’t process tests/CodeGen/X86/* with
the ARM
> >> backend, you can’t process any random IR through these backends.
> > 
> > You said you had 200+ failures with AMDGPU.  Are the AMD folks simply
> > tolerating the 200 failures, and you don't want to?  I should hope
there
> > is more to it than that.
> 
> Well, I don’t know, they might just run `ninja check` with the default
triple set to X86?
> (which I would consider being working around a buggy test suite)
> 
I always enable AMDGPU and X86 when I build, so I've never run into
this problem.

-Tom

> > 
> >> 
> >> IMO, the problem is in general about tests that are written
without
> >> specifying a triple, that will be executed with the default
triple.
> >> 
> >> Most of these tests were written with X86 (or ARM) in mind, and
there is
> >> no guarantee that they will behave as intended with every possible
triple.
> >> The DataLayout for instance has to be the one from the target, and
is not
> >> portable.
> >> I think a "portable backend test” is pretty rare in general.
> > 
> > It depends on what the test is trying to do.  I'm sure it is quite
common
> > for IR tests to behave essentially the same way regardless of target.
> 
> IR tests != backend test (I may miss your point here, it’s late…).
> 
> 
> > We have lots of tests (the ones you chose to mark "native")
that had been
> > working fine with ARM, X86, PPC, SPARC, and whatever.
> > 
> > The fact that they don't work with your backend is different from
saying
> > those tests can't possibly work for any cross-compiler.
> 
> I believe this is far from what I said, or at least from what I had in
mind.
> 
> >  But the latter is what your patch implemented, and it is preventing
useful testing.
> 
> I don’t disagree with that statement, and I agree that it should be fixed.
> It doesn’t mean that I think the previous situation was better though. 
> The current over-conservative state seems at least more correct than having
only a subset of the targets that can pass `ninja check`.
> 
> Are you attending the LLVM bay area social tonight by any chance?
> 
> Best,
> 
> — 
> Mehdi
> 
> 
> 
> > --paulr
> > 
> >> 
> >> You can run all the tests by setting the default triple to X86 and
> >> compiling in the X86 backend, but that’s just a trick to make the
tests
> >> happy.
> >> Alternatively, and this is what I tried to do, blacklisting these
tests
> >> that “pretends” to be “portable” but are not.
> >> 
> >> 
> >>> Mehdi's solution was:
> >>> - In lit.cfg, change the existing "native" feature
definition from
> >>> "host-triple == target-triple" to also check
"and the corresponding
> >>> backend is included."(**)
> >> 
> >> I agree that we could remove the condition on the host backend
included
> >> and deemed it a unsupported, but the build system needs to reject
this
> >> configuration.
> >> 
> >> 
> >>> - Make piles of tests that seemed inapplicable to GPUs depend
on the
> >>> "native" feature (through REQUIRES: or in the
lit.local.cfg).
> >> 
> >> Nitpick: the GPU is just an example, any other backends can be
affected.
> >> It seems that these test are “lying” about the target they will be
able to
> >> run on (like if they would run on “anything”).
> >> 
> >> 
> >>> - Build LLVM with just the GPU backend, and not set a target
triple
> >>> (i.e., it's set to the host triple, typically an
X86-something).(*)
> >>> Thus lit.cfg sees matching host and target triples, but the
X86
> >>> backend is missing, and so the "native" feature is
not set.
> >>> 
> >>> [The "native" feature was invented to identify
environments where
> >>> JIT would work properly. The "host-triple ==
target-triple" condition
> >>> isn't exactly right, but it works well enough.]
> >>> 
> >>> The major problem is that these new "native"
dependencies are incorrect.
> >>> For example the DebugInfo tests don't really require it;
they work fine
> >>> as long as the default triple has the corresponding backend
included,
> >>> as my ARM-on-X86 experiment demonstrated.
> >> 
> >> Are they are guarantee’d to work with a default triple set to any
of the
> >> possible (in-tree) backend?
> >> (I don’t know enough about these tests, which is definitively why
I
> >> included them in the “big hammer” solution)
> >> 
> >> Thanks,
> >> 
> >> —
> >> Mehdi
>