llvm dev - Jul 2016 - [RFC] Make Lanai backend non-experimental

On 19 July 2016 at 17:04, Martin J. O'Riordan via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Presumably if my out-of-tree backend was to be pushed to LLVM, it too would
be considered experimental.
Yes. Though, not all out-of-tree back-ends end up upstream for
different reasons.

A few basic rules to get accepted are if:
 * the target exists and can be easily purchased / emulated for
investigating problems,
 * there are official documents / specs published by the project /
company that maintains the targets,
 * there is a reasonable community maintaining the rest of the system
(firmware, OS, other tools, etc),
 * enough people commit themselves to maintain the LLVM back-end to
avoid bit-rot,
 * the back-end is free of contentious features that would mean
breaking every other target.

For example:
 * a PIC back-end ticks many of the boxes, but it doesn't have a
community around it and would very quickly bit-rot,
 * the CHERI back-end has an active community and interested
developers, but it would disrupt the IR too much,
 * the BPF back-end has been added mostly as-is and is actively
maintained, so gained the status of non-experimental.

The main difference between experimental and official is that they get
built by default. So, if you don't specify them at CMake time, they
will be built and their tests will execute, which is a big win for the
back-end maintainers (as they get free testing), but it's also a big
burden on other developers if their tests are not independent enough
and keep breaking due to unrelated changes.

Under those conditions, it's as easy to remove a back-end that is too
much trouble and no one cares about, as it to add a new experimental
back-end or promote it to official. It all depends on the community
around it and how much they value the LLVM back-end.

>  But what is involved in making a particular target, feature or back-end
experimental other than simply agreement?  Are there particular build processes
or '-Dmacroname's to add, or how does somebody like me edit my
out-of-tree sources so that it would obey the "is experimental" rules?
I'm assuming you already have an implementation, for example, in
GitHub. The easiest way to get things started is to point people at
the repository and make sure it works on trunk.

The second step, then, would be to merge that into the tree and
propose to include it as an experimental target. It will take a few
months until the process is complete and then you'll be able to check
it out and work straight from LLVM's repository.

You'll have to change the CMake to force it to be built
(-DLLVM_TARGETS_TO_BUILD) until such time that you have shown that you
care about the target and that it hasn't failed (best if you have a
buildbot showing the history always green).

Once all that is done, you can do as Jacques did and request it to be
built by default (aka. non-experimental, aka. official back-end).

After that change, though, you're permanently enlisted to maintain it
even more intensely. Not just your buildbot will break, but other
people's too, and they're probably a lot faster than yours, so you
need to keep an eye on them, especially if they break due to unrelated
commits.

That is the contract that you have to sign to have your target
supported, and if you break that contract (let it bit-rot, disable
tests that should be passing, remove parts of the code, or just
neglects bug reports), your back-end *will* be removed.

Hope that helps,
--renato

On 19 Jul 2016, at 17:42, Renato Golin via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> * the CHERI back-end has an active community and interested
> developers, but it would disrupt the IR too much,
The IR disruptions for CHERI aren’t that major (the main thing is that we allow
alloca’s in different address spaces, but that’s not too difficult to upstream),
but it does fail to tick one box that you didn’t list:

* The ISA is stable and mature.

The CHERI ISA is now evolving a lot less quickly than it was a year ago, but we
do still make changes that break legacy code and so it would not be helpful to
have a compiler with a release cycle that is completely decoupled from our FPGA
/ QEMU releases.  It would be annoying for everyone upstream to have something
in tree where the ISA is a moving target.

A few things that we’ve done are definitely ‘research quality’ code and wouldn’t
meet upstream’s code quality standards either.

David

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3719 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160719/a86bb9ea/attachment-0001.bin>

On 19 July 2016 at 17:52, David Chisnall <david.chisnall at cl.cam.ac.uk>
wrote:
> The IR disruptions for CHERI aren’t that major (the main thing is that we
allow alloca’s in different address spaces, but that’s not too difficult to
upstream),
I remember things like inttoptr / ptrtoint were major pains to you and
could be even experienced in the front-end (constant folding, etc).
Has that changed recently?

> but it does fail to tick one box that you didn’t list:
>
> * The ISA is stable and mature.
Indeed! That was part of the "documents published", but I agree, one
can "publish" anything these days. :)

> A few things that we’ve done are definitely ‘research quality’ code and
wouldn’t meet upstream’s code quality standards either.
Some "research quality" internals can be seen in any back-end if you
look closely enough, but having most of it being flaky is indeed, a
big problem that will show up in breakages for unrelated changes.

cheers,
--renato

Hi Renato
> On Jul 19, 2016, at 9:42 AM, Renato Golin via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> A few basic rules to get accepted are if:
> * the target exists and can be easily purchased / emulated for
> investigating problems,
> * there are official documents / specs published by the project /
> company that maintains the targets,
> * there is a reasonable community maintaining the rest of the system
> (firmware, OS, other tools, etc),
> * enough people commit themselves to maintain the LLVM back-end to
> avoid bit-rot,
> * the back-end is free of contentious features that would mean
> breaking every other target.
This is an excellent list.  We should probably have something like it in the
docs if we don’t already.

I have no problem with Lanai in LLVM (non-experimentally) if it meets most of
these criteria to some degree.  Obviously that doesn’t mean every target in LLVM
needs to have detailed microarchitecture documents, but a basic ISA should be
publicly available.  Personally I also think the ability to at least simulate
the code is very important too.

I tried googling it but mostly found news reports about the existence of the
backend, not documentation or anything else.

I think the Lanai community have done an excellent job meeting the last 2
points.  However, I’d really appreciate more information on the first 3.  I
think meeting each of them (even minimally) is important to leave experimental
status.

Thanks,
Pete
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160719/c501ead1/attachment.html>

> On Jul 19, 2016, at 09:52, David Chisnall via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> The IR disruptions for CHERI aren’t that major (the main thing is that we
allow alloca’s in different address spaces, but that’s not too difficult to
upstream), but it does fail to tick one box that you didn’t list:
We would like this feature upstreamed

-Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160719/e3864c09/attachment.html>

> On Jul 19, 2016, at 12:57 PM, Pete Cooper via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi Renato
>> On Jul 19, 2016, at 9:42 AM, Renato Golin via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> A few basic rules to get accepted are if:
>> * the target exists and can be easily purchased / emulated for
>> investigating problems,
>> * there are official documents / specs published by the project /
>> company that maintains the targets,
>> * there is a reasonable community maintaining the rest of the system
>> (firmware, OS, other tools, etc),
>> * enough people commit themselves to maintain the LLVM back-end to
>> avoid bit-rot,
>> * the back-end is free of contentious features that would mean
>> breaking every other target.
> This is an excellent list.  We should probably have something like it in
the docs if we don’t already.
Agree with Pete.

I’d add (because the community will have to maintain it somehow):
- good code quality and documentation
- good testing (small tests, documented, with good coverage).

— 
Mehdi

> 
> I have no problem with Lanai in LLVM (non-experimentally) if it meets most
of these criteria to some degree.  Obviously that doesn’t mean every target in
LLVM needs to have detailed microarchitecture documents, but a basic ISA should
be publicly available.  Personally I also think the ability to at least simulate
the code is very important too.
> 
> I tried googling it but mostly found news reports about the existence of
the backend, not documentation or anything else.
> 
> I think the Lanai community have done an excellent job meeting the last 2
points.  However, I’d really appreciate more information on the first 3.  I
think meeting each of them (even minimally) is important to leave experimental
status.
> 
> Thanks,
> Pete
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160719/8d45bd52/attachment.html>

On Tue, Jul 19, 2016 at 9:42 AM Renato Golin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 19 July 2016 at 17:04, Martin J. O'Riordan via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > Presumably if my out-of-tree backend was to be pushed to LLVM, it too
> would be considered experimental.
>
> Yes. Though, not all out-of-tree back-ends end up upstream for
> different reasons.
>
> A few basic rules to get accepted are if:
>  * the target exists and can be easily purchased / emulated for
> investigating problems,
>  * there are official documents / specs published by the project /
> company that maintains the targets,
>
Most of your list makes a lot of sense to me Renato, but these two items
don't really.

I think a large number of targets in tree would be *extremely* hard to
purchase and directly investigate problems. As examples:

- Hexagon
- SystemZ
- XCore

I'm sure it is theoretically possible for me to acquire such a system and
test on it, but the amount of time and energy it would take make it a
practical impossibility.

Even for fairly pedestrian backends such as AArch64 and PowerPC, we
routinely have people ask active developers on those backends to do the
triaging of issues. I've never seen this be a problem in practice, and
I'm
not convinced that it is something we should actually require to have an
upstream backend.


I also worry that we have backends for targets that don't have any (real)
published ISA spec. I'm not sure there is a formal "spec" per-se
for BPF.
And perhaps there is a spec for AMDGPU or Hexagon, but even when I've asked
developers on those backends about problems, they've not pointed be at the
published ISA documentation.


And I don't think any of these backends are bad or should be removed. I
rather like several of them. =] I just think that sufficient community
involvement makes the availability of hardware/emulator "nice to have"
but
not essential, and it makes the ISA documentation substantially less
important.


The rest of your list below though, I think is a great list.

>  * there is a reasonable community maintaining the rest of the system
> (firmware, OS, other tools, etc),
>  * enough people commit themselves to maintain the LLVM back-end to
> avoid bit-rot,
>  * the back-end is free of contentious features that would mean
> breaking every other target.
>
> For example:
>  * a PIC back-end ticks many of the boxes, but it doesn't have a
> community around it and would very quickly bit-rot,
>  * the CHERI back-end has an active community and interested
> developers, but it would disrupt the IR too much,
>  * the BPF back-end has been added mostly as-is and is actively
> maintained, so gained the status of non-experimental.
>
> The main difference between experimental and official is that they get
> built by default. So, if you don't specify them at CMake time, they
> will be built and their tests will execute, which is a big win for the
> back-end maintainers (as they get free testing), but it's also a big
> burden on other developers if their tests are not independent enough
> and keep breaking due to unrelated changes.
>
> Under those conditions, it's as easy to remove a back-end that is too
> much trouble and no one cares about, as it to add a new experimental
> back-end or promote it to official. It all depends on the community
> around it and how much they value the LLVM back-end.
>
>
> >  But what is involved in making a particular target, feature or
back-end
> experimental other than simply agreement?  Are there particular build
> processes or '-Dmacroname's to add, or how does somebody like me
edit my
> out-of-tree sources so that it would obey the "is experimental"
rules?
>
> I'm assuming you already have an implementation, for example, in
> GitHub. The easiest way to get things started is to point people at
> the repository and make sure it works on trunk.
>
> The second step, then, would be to merge that into the tree and
> propose to include it as an experimental target. It will take a few
> months until the process is complete and then you'll be able to check
> it out and work straight from LLVM's repository.
>
> You'll have to change the CMake to force it to be built
> (-DLLVM_TARGETS_TO_BUILD) until such time that you have shown that you
> care about the target and that it hasn't failed (best if you have a
> buildbot showing the history always green).
>
> Once all that is done, you can do as Jacques did and request it to be
> built by default (aka. non-experimental, aka. official back-end).
>
> After that change, though, you're permanently enlisted to maintain it
> even more intensely. Not just your buildbot will break, but other
> people's too, and they're probably a lot faster than yours, so you
> need to keep an eye on them, especially if they break due to unrelated
> commits.
>
> That is the contract that you have to sign to have your target
> supported, and if you break that contract (let it bit-rot, disable
> tests that should be passing, remove parts of the code, or just
> neglects bug reports), your back-end *will* be removed.
>
> Hope that helps,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160725/be299207/attachment.html>

Hi Chandler,

I think you have good points. Maybe we could make some hard-lined
rules and others as "nice-to-have".

The biggest problem is the community behind it, outside of LLVM. If
the community is strong, and they care about LLVM support, than we can
keep their back-ends in tree and not have to worry about them.

IIRC, our *only* rule for a long time has been "keep up or give up".
Now, how do we know when a target is good for moving out of
experimental? And what is the meaning of experimental anyway?

Maybe we should separate the discussion from the actual change. So, if
you could comment on the review D22753 about which of the points you'd
consider mandatory and which are nice-to-have, that'd probably be
easiest.

About the Lanai back-end being official, I have no reservations. But I
was the only one to say anything, so I'll wait for others to have
their say on the review before I put my approval.

Just a few unrelated comments below... :)


On 25 July 2016 at 19:46, Chandler Carruth <chandlerc at google.com>
wrote:
> Even for fairly pedestrian backends such as AArch64 and PowerPC, we
> routinely have people ask active developers on those backends to do the
> triaging of issues.
I think that's most for specific environment than anything else.

Really, you can get sub-$100 AArch64 1-day delivery and a *very*
decent open source emulator (QEMU), both system and user level.

> And I don't think any of these backends are bad or should be removed. I
> rather like several of them. =] I just think that sufficient community
> involvement makes the availability of hardware/emulator "nice to
have" but
> not essential, and it makes the ISA documentation substantially less
> important.
I also don't think emulators replace the ISA in any way, but I get
what you mean. Being nice-to-have, either one would be better than
nothing, but having both would be really optimal.

Between the two, I'd personally prefer the ISA (if it was correct and
complete) than a simulator. But that's me.

cheers,
--renato

On Jul 25, 2016, at 11:46 AM, Chandler Carruth via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> A few basic rules to get accepted are if:
>  * the target exists and can be easily purchased / emulated for
> investigating problems,
>  * there are official documents / specs published by the project /
> company that maintains the targets,
> 
> Most of your list makes a lot of sense to me Renato, but these two items
don't really.
> 
> I think a large number of targets in tree would be *extremely* hard to
purchase and directly investigate problems. As examples:
> 
> - Hexagon
> - SystemZ
> - XCore
> 
> I'm sure it is theoretically possible for me to acquire such a system
and test on it, but the amount of time and energy it would take make it a
practical impossibility.

I agree, but it isn’t completely unreasonable to expect a simulator to be
available.

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160725/8b9110e3/attachment.html>

On Mon, Jul 25, 2016 at 11:46 AM, Chandler Carruth via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Tue, Jul 19, 2016 at 9:42 AM Renato Golin via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> On 19 July 2016 at 17:04, Martin J. O'Riordan via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>> > Presumably if my out-of-tree backend was to be pushed to LLVM, it
too
>> would be considered experimental.
>>
>> Yes. Though, not all out-of-tree back-ends end up upstream for
>> different reasons.
>>
>> A few basic rules to get accepted are if:
>>  * the target exists and can be easily purchased / emulated for
>> investigating problems,
>>  * there are official documents / specs published by the project /
>> company that maintains the targets,
>>
>
> Most of your list makes a lot of sense to me Renato, but these two items
> don't really.
>
> I think a large number of targets in tree would be *extremely* hard to
> purchase and directly investigate problems. As examples:
>
> - Hexagon
> - SystemZ
> - XCore
>
> I'm sure it is theoretically possible for me to acquire such a system
and
> test on it, but the amount of time and energy it would take make it a
> practical impossibility.
>
> Even for fairly pedestrian backends such as AArch64 and PowerPC, we
> routinely have people ask active developers on those backends to do the
> triaging of issues. I've never seen this be a problem in practice, and
I'm
> not convinced that it is something we should actually require to have an
> upstream backend.
>
As I mentioned in the original thread "[llvm-dev] [RFC] Lanai
backend", it
seems to me that the "can I run the code coming out of the backend?"
property is more about enabling LLVM's users to do interesting things with
the backend, rather than about LLVM's internal maintenance.

http://lists.llvm.org/pipermail/llvm-dev/2016-February/095214.html

I'm not sure how that affects the decision.

My gut (and I'm actually not sure if I exactly consider this a pro or a con
personally) is that is sounds Stallman-esque to require backends to answer
"yes" to "can I run the code coming out of the backend?". If
you squint it
is sort of like an anti-tivoization requirement, which would be a very
ironic requirement for a BSD-licensed project.

One example that personally convinces me that "can I run the code coming
out of the backend?" should not be a requirement is the following: if Apple
is willing to develop their mobile GPU backend(s?) upstream, but still keep
their actual driver stack and hardware locked down, I think it would be a
net win for the community to bring it into tree.

-- Sean Silva

>
>
> I also worry that we have backends for targets that don't have any
(real)
> published ISA spec. I'm not sure there is a formal "spec"
per-se for BPF.
> And perhaps there is a spec for AMDGPU or Hexagon, but even when I've
asked
> developers on those backends about problems, they've not pointed be at
the
> published ISA documentation.
>
>
> And I don't think any of these backends are bad or should be removed. I
> rather like several of them. =] I just think that sufficient community
> involvement makes the availability of hardware/emulator "nice to
have" but
> not essential, and it makes the ISA documentation substantially less
> important.
>
>
> The rest of your list below though, I think is a great list.
>
>
>>  * there is a reasonable community maintaining the rest of the system
>> (firmware, OS, other tools, etc),
>>  * enough people commit themselves to maintain the LLVM back-end to
>> avoid bit-rot,
>>  * the back-end is free of contentious features that would mean
>> breaking every other target.
>>
>> For example:
>>  * a PIC back-end ticks many of the boxes, but it doesn't have a
>> community around it and would very quickly bit-rot,
>>  * the CHERI back-end has an active community and interested
>> developers, but it would disrupt the IR too much,
>>  * the BPF back-end has been added mostly as-is and is actively
>> maintained, so gained the status of non-experimental.
>>
>> The main difference between experimental and official is that they get
>> built by default. So, if you don't specify them at CMake time, they
>> will be built and their tests will execute, which is a big win for the
>> back-end maintainers (as they get free testing), but it's also a
big
>> burden on other developers if their tests are not independent enough
>> and keep breaking due to unrelated changes.
>>
>> Under those conditions, it's as easy to remove a back-end that is
too
>> much trouble and no one cares about, as it to add a new experimental
>> back-end or promote it to official. It all depends on the community
>> around it and how much they value the LLVM back-end.
>>
>>
>> >  But what is involved in making a particular target, feature or
>> back-end experimental other than simply agreement?  Are there
particular
>> build processes or '-Dmacroname's to add, or how does somebody
like me edit
>> my out-of-tree sources so that it would obey the "is
experimental" rules?
>>
>> I'm assuming you already have an implementation, for example, in
>> GitHub. The easiest way to get things started is to point people at
>> the repository and make sure it works on trunk.
>>
>> The second step, then, would be to merge that into the tree and
>> propose to include it as an experimental target. It will take a few
>> months until the process is complete and then you'll be able to
check
>> it out and work straight from LLVM's repository.
>>
>> You'll have to change the CMake to force it to be built
>> (-DLLVM_TARGETS_TO_BUILD) until such time that you have shown that you
>> care about the target and that it hasn't failed (best if you have a
>> buildbot showing the history always green).
>>
>> Once all that is done, you can do as Jacques did and request it to be
>> built by default (aka. non-experimental, aka. official back-end).
>>
>> After that change, though, you're permanently enlisted to maintain
it
>> even more intensely. Not just your buildbot will break, but other
>> people's too, and they're probably a lot faster than yours, so
you
>> need to keep an eye on them, especially if they break due to unrelated
>> commits.
>>
>> That is the contract that you have to sign to have your target
>> supported, and if you break that contract (let it bit-rot, disable
>> tests that should be passing, remove parts of the code, or just
>> neglects bug reports), your back-end *will* be removed.
>>
>> Hope that helps,
>> --renato
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160725/519a2e51/attachment.html>

On 7/25/2016 1:46 PM, Chandler Carruth via llvm-dev
wrote:
> And perhaps there is a spec for AMDGPU or Hexagon, but even when I've
> asked developers on those backends about problems, they've not pointed
> be at the published ISA documentation.
The Hexagon ISA is now available at
https://developer.qualcomm.com/software/hexagon-dsp-sdk/tools

It's the "Programmer's Reference Manual". You need to create
an account
to get it, but it's there now.

-Krzysztof

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, 
hosted by The Linux Foundation