llvm dev - Sep 2015 - Testing "normal" cross-compilers versus GPU backends

On 09/04/2015 10:25 AM, Robinson, Paul via llvm-dev
wrote:
>
>> -----Original Message-----
>> From: James Y Knight [mailto:jyknight at google.com]
>> Sent: Friday, September 04, 2015 10:12 AM
>> To: Mehdi Amini; Mehdi Amini via llvm-dev
>> Cc: Robinson, Paul
>> Subject: Re: [llvm-dev] Testing "normal" cross-compilers
versus GPU
>> backends
>>
>>>>> The Hexagon precedent is interesting; Krzysztof said they
set the
>> default
>>>>> triple, and didn't have to xfail all that much stuff. 
Searching the
>> tree,
>>>>> I find exactly 7 individual tests marked XFAIL: hexagon,
plus it
>> disables
>>>>> all of ExecutionEngine, and turns off the
'object-emission' feature.
>>>>>
>>>>> I'm curious if you would try setting the default triple
to match your
>>>>> target, and see what /kinds/ of tests fail.  The raw number
is much
>> less
>>>>> interesting than in the categories.
>>>> Failing tests attached, let me know which ones you’d like me to
>> investigate.
>>> Tests:
>>> <failing_tests>
>>>
>>> (note I forced enable the “native” feature on this run)
>> So, just looking at the first one in that list for an example, running:
>> llc -mtriple amdgcn test/CodeGen/Generic/2002-04-14-
>> UnexpectedUnsignedType.ll
>> you get an error of:
>> error: unsupported call to function bar in foo
>>
>> ..because apparently AMDGPU simply doesn't support function calls.
At all.
>>
>> That is a rather unusual feature to be missing, and I'd imagine is
causing
>> a fair number of the other nominally-generic tests to fail too...
> Hmm I'm told GPUs as a class typically don't support calls
(everything has
> to be inlined).
>
> Perhaps the more appropriate direction here is to invent a "gpu"
config flag
> and mark some things XFAIL: gpu.
> --paulr
Mehdi - Could you list a couple of the features that show up in generic 
tests which your target doesn't support by design?  After reading 
through the discussion to this point, I don't really have a feel for 
that.   Additionally, do you have any sense for how common that set is 
across GPU targets?

Paul - If we introduce the type of flag your talking about, we should 
probably have a discussion about which classes of backends LLVM supports 
and try to create a *documented* list of feature expectations somewhere.

p.s. Maybe I missed something, but why did simply using "XFAIL:
AMDGPU"
get rejected?  That seems like the easiest fix.

Philip

Hi Philip,

> On Sep 8, 2015, at 9:07 AM, Philip Reames <listmail at
philipreames.com> wrote:
> 
> On 09/04/2015 10:25 AM, Robinson, Paul via llvm-dev wrote:
>> 
>>> -----Original Message-----
>>> From: James Y Knight [mailto:jyknight at google.com]
>>> Sent: Friday, September 04, 2015 10:12 AM
>>> To: Mehdi Amini; Mehdi Amini via llvm-dev
>>> Cc: Robinson, Paul
>>> Subject: Re: [llvm-dev] Testing "normal" cross-compilers
versus GPU
>>> backends
>>> 
>>>>>> The Hexagon precedent is interesting; Krzysztof said
they set the
>>> default
>>>>>> triple, and didn't have to xfail all that much
stuff.  Searching the
>>> tree,
>>>>>> I find exactly 7 individual tests marked XFAIL:
hexagon, plus it
>>> disables
>>>>>> all of ExecutionEngine, and turns off the
'object-emission' feature.
>>>>>> 
>>>>>> I'm curious if you would try setting the default
triple to match your
>>>>>> target, and see what /kinds/ of tests fail.  The raw
number is much
>>> less
>>>>>> interesting than in the categories.
>>>>> Failing tests attached, let me know which ones you’d like
me to
>>> investigate.
>>>> Tests:
>>>> <failing_tests>
>>>> 
>>>> (note I forced enable the “native” feature on this run)
>>> So, just looking at the first one in that list for an example,
running:
>>> llc -mtriple amdgcn test/CodeGen/Generic/2002-04-14-
>>> UnexpectedUnsignedType.ll
>>> you get an error of:
>>> error: unsupported call to function bar in foo
>>> 
>>> ..because apparently AMDGPU simply doesn't support function
calls. At all.
>>> 
>>> That is a rather unusual feature to be missing, and I'd imagine
is causing
>>> a fair number of the other nominally-generic tests to fail too...
>> Hmm I'm told GPUs as a class typically don't support calls
(everything has
>> to be inlined).
>> 
>> Perhaps the more appropriate direction here is to invent a
"gpu" config flag
>> and mark some things XFAIL: gpu.
>> --paulr
> Mehdi - Could you list a couple of the features that show up in generic
tests which your target doesn't support by design?  After reading through
the discussion to this point, I don't really have a feel for that.  
Additionally, do you have any sense for how common that set is across GPU
targets?
This is what I was referring to in my last email:
"To be “correct”, tests should “require” (in the lit sense) features like
“function call”, etc. However this is a significant amount of work with not that
much added value.
This is what Takumi did to disable one test that check for “debug_frame” which
are not emitted on Windows.”

I could try to survey all the failing tests and abstract the “features” they are
using, but this does not seem like a small enough task that I can commit to an
ETA now or have an estimate about when it’ll come on top of my stack.

This is why I tried to offer an alternative with http://reviews.llvm.org/D12660
; to identify the tests that rely on the default triple using a generic
default_triple feature.

> Paul - If we introduce the type of flag your talking about, we should
probably have a discussion about which classes of backends LLVM supports and try
to create a *documented* list of feature expectations somewhere.
> 
> p.s. Maybe I missed something, but why did simply using "XFAIL:
AMDGPU" get rejected?  That seems like the easiest fix.
You haven't missed anything IIUC.
If we were to document the list of feature expectations from the backends, the
“default_triple” would be applied to the tests that makes use of these features.

— 
Mehdi

> > Mehdi - Could you list a couple of the features that show up in
generic
> tests which your target doesn't support by design?  After reading
through
> the discussion to this point, I don't really have a feel for that.
> Additionally, do you have any sense for how common that set is across GPU
> targets?
> 
> This is what I was referring to in my last email:
> "To be “correct”, tests should “require” (in the lit sense) features
like
> “function call”, etc. However this is a significant amount of work with
> not that much added value.
> This is what Takumi did to disable one test that check for “debug_frame”
> which are not emitted on Windows.”
> 
> I could try to survey all the failing tests and abstract the “features”
> they are using, but this does not seem like a small enough task that I can
> commit to an ETA now or have an estimate about when it’ll come on top of
> my stack.
> 
> This is why I tried to offer an alternative with
> http://reviews.llvm.org/D12660 ; to identify the tests that rely on the
> default triple using a generic default_triple feature.
> 
> 
> > Paul - If we introduce the type of flag your talking about, we should
> probably have a discussion about which classes of backends LLVM supports
> and try to create a *documented* list of feature expectations somewhere.
> >
> > p.s. Maybe I missed something, but why did simply using "XFAIL:
AMDGPU"
> get rejected?  That seems like the easiest fix.
> 
> You haven't missed anything IIUC.
> If we were to document the list of feature expectations from the backends,
> the “default_triple” would be applied to the tests that makes use of these
> features.
> 
> —
> Mehdi
Hmmm "default_triple" is certainly not an expectation or feature of
any
target; rather, it is an expectation of certain tools when used in certain
ways (mainly: llc or llvm-mc without an explicit triple) and therefore it
is an expectation of how certain tests are _written_.  This has little to
do with the _features_ verified by those tests.

In particular, configuring a GPU-only LLVM with a default triple for that 
GPU will still fail all the same tests that Mehdi tripped over.  Inventing
"default_triple" lets us trick the test suite into skipping the tests
that
the GPU target would fail anyway.

This is not to say "default_triple" is a bad thing; it lets us support
a
kind of configuration that Mehdi has persuaded me is valid, and I'm happy
to have it.  But it is not a target-feature-based solution.

It is not impossible that if we defined a "GPU" feature set, we might
find
that it would exclude many of the same tests that "default_triple"
excludes.
But that would be essentially by chance, not because "default_triple"
has
any inherent meaning for targets.
--paulr

On 09/08/2015 04:04 PM, Mehdi Amini wrote:
> Hi Philip,
>
>
>> On Sep 8, 2015, at 9:07 AM, Philip Reames <listmail at
philipreames.com> wrote:
>>
>> On 09/04/2015 10:25 AM, Robinson, Paul via llvm-dev wrote:
>>>> -----Original Message-----
>>>> From: James Y Knight [mailto:jyknight at google.com]
>>>> Sent: Friday, September 04, 2015 10:12 AM
>>>> To: Mehdi Amini; Mehdi Amini via llvm-dev
>>>> Cc: Robinson, Paul
>>>> Subject: Re: [llvm-dev] Testing "normal"
cross-compilers versus GPU
>>>> backends
>>>>
>>>>>>> The Hexagon precedent is interesting; Krzysztof
said they set the
>>>> default
>>>>>>> triple, and didn't have to xfail all that much
stuff.  Searching the
>>>> tree,
>>>>>>> I find exactly 7 individual tests marked XFAIL:
hexagon, plus it
>>>> disables
>>>>>>> all of ExecutionEngine, and turns off the
'object-emission' feature.
>>>>>>>
>>>>>>> I'm curious if you would try setting the
default triple to match your
>>>>>>> target, and see what /kinds/ of tests fail.  The
raw number is much
>>>> less
>>>>>>> interesting than in the categories.
>>>>>> Failing tests attached, let me know which ones you’d
like me to
>>>> investigate.
>>>>> Tests:
>>>>> <failing_tests>
>>>>>
>>>>> (note I forced enable the “native” feature on this run)
>>>> So, just looking at the first one in that list for an example,
running:
>>>> llc -mtriple amdgcn test/CodeGen/Generic/2002-04-14-
>>>> UnexpectedUnsignedType.ll
>>>> you get an error of:
>>>> error: unsupported call to function bar in foo
>>>>
>>>> ..because apparently AMDGPU simply doesn't support function
calls. At all.
>>>>
>>>> That is a rather unusual feature to be missing, and I'd
imagine is causing
>>>> a fair number of the other nominally-generic tests to fail
too...
>>> Hmm I'm told GPUs as a class typically don't support calls
(everything has
>>> to be inlined).
>>>
>>> Perhaps the more appropriate direction here is to invent a
"gpu" config flag
>>> and mark some things XFAIL: gpu.
>>> --paulr
>> Mehdi - Could you list a couple of the features that show up in generic
tests which your target doesn't support by design?  After reading through
the discussion to this point, I don't really have a feel for that.  
Additionally, do you have any sense for how common that set is across GPU
targets?
> This is what I was referring to in my last email:
> "To be “correct”, tests should “require” (in the lit sense) features
like “function call”, etc. However this is a significant amount of work with not
that much added value.
> This is what Takumi did to disable one test that check for “debug_frame”
which are not emitted on Windows.”
>
> I could try to survey all the failing tests and abstract the “features”
they are using, but this does not seem like a small enough task that I can
commit to an ETA now or have an estimate about when it’ll come on top of my
stack.
>
> This is why I tried to offer an alternative with
http://reviews.llvm.org/D12660 ; to identify the tests that rely on the default
triple using a generic default_triple feature.
Mehdi - You're answer focuses on implementation details and a complete 
list.  That's not what I was getting at.  The *only* concrete feature 
mentioned so far has been function calls.  All I'm looking for is this a 
problem specific to function calls?  Or are there a couple of other 
known features that your backend just doesn't support?  This is purely 
so that I can have a chance of understanding the discussion.  I'm not 
asking for an exhaustive list or anything.
>
>
>> Paul - If we introduce the type of flag your talking about, we should
probably have a discussion about which classes of backends LLVM supports and try
to create a *documented* list of feature expectations somewhere.
>>
>> p.s. Maybe I missed something, but why did simply using "XFAIL:
AMDGPU" get rejected?  That seems like the easiest fix.
> You haven't missed anything IIUC.
> If we were to document the list of feature expectations from the backends,
the “default_triple” would be applied to the tests that makes use of these
features.
How many tests would you have to add XFAILs to if we went with the 
simple approach?  Ballpark number is sufficient; I just want an idea of 
the magnitude.
>
> —
> Mehdi
>