llvm dev - Mar 2014 - [LLVMdev] Contributing the Apple ARM64 compiler backend

All,

Attached below are the patches that make up the Apple ARM64 compiler backend (in
addition to compiler_rt, libc++ and lldb support), and we'd like to start
the process of integrating them into mainline LLVM.  We and ARM have discussed a
general approach to integrating them into mainline and look forward to working
through this with the community at large.

First a bit of context to help jump-start the discussion. The ARM64 backend is a
complete production quality implementation for ARM’s 64-bit architecture,
AArch64. It supports both iOS and Linux as target platforms. In the source tree,
the backend co-exists with the AArch64 backend without conflicts.

As a first step, we'd like to add this as a peer of the AArch64 backend. 
While having two AArch64 backends in the tree long term doesn’t make sense, this
approach worked well when the Sparc backends were merged and for AMDIL. This
will allow us to work quickly towards merging the ports to create a backend that
is truly the best of both. The public AArch64 backend has attracted an excellent
core group of contributors all of whom have done great work and we are very much
looking forward to collaborating with them and the rest of the community to make
this happen.

Getting to that idillic end-state of one AArch64 backend in the tree which
encompasses the best of both is going to take a fair bit of work and a lot of
discussion amongst the interested parties. To get there, I think a rough roadmap
like the following makes sense:

1. Import the ARM64 backend into the public tree so it’s easily accessible for
everyone to investigate.
2. Test it. Benchmark it. Explore it. Get data for the community to work with
about the state of the back end. ARM has some excellent data that will help
guide us here.
3. Identify the core backend to build on and to merge features and tuning from
the other to. The data I have seen so far leads me to believe ARM64 is the
correct choice here, but that’s a decision primarily for the contributors above.
4. Clean up the codebase (C++11-ify it, for example), fix any regressions and
test failures identified in benchmarking.

This will give us a backend that is a superset of both ARM64 and AArch64 in
terms both of performance and functionality. We can then consolidate to a single
backend, named AArch64 for consistency with the current public tree.

Regards,
Jim Grosbach

-------------- next part --------------
A non-text attachment was scrubbed...
Name: arm64-libc++.patch.gz
Type: application/x-gzip
Size: 1249 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140328/ecaab42b/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: arm64-compiler-rt.patch.gz
Type: application/x-gzip
Size: 2240 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140328/ecaab42b/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: arm64-llvm.patch.gz
Type: application/x-gzip
Size: 705735 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140328/ecaab42b/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: arm64-clang.patch.gz
Type: application/x-gzip
Size: 109888 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140328/ecaab42b/attachment-0003.bin>

First and foremost, I'd like to say that I think this is a really great
roadmap.

On Fri, Mar 28, 2014 at 12:35 PM, James Grosbach <grosbach at apple.com>
wrote:
> 1. Import the ARM64 backend into the public tree so it’s easily accessible
> for everyone to investigate.

I support this, and would ask that this be done essentially right away. I
don't think patches are going to be reasonable ways to evaluate this. I
have two specific reasons for thinking this is a good thing to do
essentially "right away":

1) It comes from existing long-time and very experienced community members,
so I have no concerns over either coding convention adherence or long term
maintenance problems.

2) This is complex enough and requires enough study that a patch series is
completely infeasible.

I'd like to hear from the existing AArch64 backend maintainers if they have
any concerns about getting #1 done right away?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140328/022ad589/attachment.html>

On 28 March 2014 19:35, James Grosbach <grosbach at apple.com>
wrote:
> Attached below are the patches that make up the Apple ARM64 compiler
backend (in addition to compiler_rt, libc++ and lldb support), and we'd like
to start the process of integrating them into mainline LLVM.  We and ARM have
discussed a general approach to integrating them into mainline and look forward
to working through this with the community at large.
W00t!! With libraries and debugger support, nice!

> 2. Test it. Benchmark it. Explore it. Get data for the community to work
with about the state of the back end. ARM has some excellent data that will help
guide us here.
Count me in. Would that be a different triple? arm64-linux-gnu?

> 3. Identify the core backend to build on and to merge features and tuning
from the other to. The data I have seen so far leads me to believe ARM64 is the
correct choice here, but that's a decision primarily for the contributors
above.
Either way, we'll have to move features form one side to the other. I
don't see bad choices ahead, just hard work.

> This will give us a backend that is a superset of both ARM64 and AArch64 in
terms both of performance and functionality. We can then consolidate to a single
backend, named AArch64 for consistency with the current public tree.
Do you have an idea of how much work would be involved and a
finger-in-the-air estimate of time?

I don't think we'll be able to move everything and remove one of the
back-ends in a single release cycle, so it doesn't make sense to wait
for 3.5 to go out before the merge, since it won't be ready by the
time 3.6 leaves the door. So, now is as good as any time to start
this. If people are concerned with confusion, we may just ship one of
the back-ends on the final binaries, whichever is in better shape.

Thanks Jim, glad to see things are finally rolling out. Interesting times ahead!

cheers,
--renato

Hi Chandler,
> I'd like to hear from the existing AArch64 backend maintainers if they
have
> any concerns about getting #1 done right away?
I agree (with my AArch64 maintainer hat on).

The goal of merging these two backends will hopefully do a lot to make
sure the eventual code is better than what exists now in either
individual directory; and doing the real work towards that will be
*much* easier when both are in-tree.

Tim.

Hi Renato,
>> 2. Test it. Benchmark it. Explore it. Get data for the community to
work with about the state of the back end. ARM has some excellent data that will
help guide us here.
>
> Count me in. Would that be a different triple? arm64-linux-gnu?
Yep. "--target arm64-linux-gnu" should behave basically the same as
aarch64-linux-gnu, except ending up in the ARM64 backend. Complete
with the usual issues of finding your toolchain's includes & libs;
we've not solved that.

Cheers.

Tim.

Patches welcome, indeed!

Joe

On Mar 28, 2014, at 3:35 PM, James Grosbach <grosbach at apple.com> wrote:
> All,
> 
> Attached below are the patches that make up the Apple ARM64 compiler
backend (in addition to compiler_rt, libc++ and lldb support), and we'd like
to start the process of integrating them into mainline LLVM.  We and ARM have
discussed a general approach to integrating them into mainline and look forward
to working through this with the community at large.
> 
> First a bit of context to help jump-start the discussion. The ARM64 backend
is a complete production quality implementation for ARM’s 64-bit architecture,
AArch64. It supports both iOS and Linux as target platforms. In the source tree,
the backend co-exists with the AArch64 backend without conflicts.
> 
> As a first step, we'd like to add this as a peer of the AArch64
backend.  While having two AArch64 backends in the tree long term doesn’t make
sense, this approach worked well when the Sparc backends were merged and for
AMDIL. This will allow us to work quickly towards merging the ports to create a
backend that is truly the best of both. The public AArch64 backend has attracted
an excellent core group of contributors all of whom have done great work and we
are very much looking forward to collaborating with them and the rest of the
community to make this happen.
> 
> Getting to that idillic end-state of one AArch64 backend in the tree which
encompasses the best of both is going to take a fair bit of work and a lot of
discussion amongst the interested parties. To get there, I think a rough roadmap
like the following makes sense:
> 
> 1. Import the ARM64 backend into the public tree so it’s easily accessible
for everyone to investigate.
> 2. Test it. Benchmark it. Explore it. Get data for the community to work
with about the state of the back end. ARM has some excellent data that will help
guide us here.
> 3. Identify the core backend to build on and to merge features and tuning
from the other to. The data I have seen so far leads me to believe ARM64 is the
correct choice here, but that’s a decision primarily for the contributors above.
> 4. Clean up the codebase (C++11-ify it, for example), fix any regressions
and test failures identified in benchmarking.
> 
> This will give us a backend that is a superset of both ARM64 and AArch64 in
terms both of performance and functionality. We can then consolidate to a single
backend, named AArch64 for consistency with the current public tree.
> 
> Regards,
> Jim Grosbach
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140328/f2e8e753/attachment.sig>

On 28 March 2014 20:36, Chandler Carruth <chandlerc at google.com>
wrote:
> 1) It comes from existing long-time and very experienced community members,
> so I have no concerns over either coding convention adherence or long term
> maintenance problems.
>
> 2) This is complex enough and requires enough study that a patch series is
> completely infeasible.
Completely agree, plus:

3) You guys probably have that back-end in parallel internally up and
running already. With tests.

I vote for incremental post-commit review and testing.

cheers,
--renato

>
> Getting to that idillic end-state of one AArch64 backend in the tree which
encompasses the best of both is going to take a fair bit of work and a lot of
discussion amongst the interested parties. To get there, I think a rough roadmap
like the following makes sense:
>
> 1. Import the ARM64 backend into the public tree so it's easily
accessible for everyone to investigate.
> 2. Test it. Benchmark it. Explore it. Get data for the community to work
with about the state of the back end. ARM has some excellent data that will help
guide us here.
> 3. Identify the core backend to build on and to merge features and tuning
from the other to. The data I have seen so far leads me to believe ARM64 is the
correct choice here, but that's a decision primarily for the contributors
above.
> 4. Clean up the codebase (C++11-ify it, for example), fix any regressions
and test failures identified in benchmarking.
>
Sounds great. It'd be good to get the target independent bits applied
separately (hopefully after the arm64 backend is committed so that
tests are more obvious), but I'm in favor of committing this as soon
as you can type svn ci with the appropriate splits for patches.

-eric

My suggestions:

1. Do #1 below as you suggested as quickly as feasible.
2. String up and flog any and all executives and lawyers who forced the world to
wait since September for this. Perhaps beat them with a big stick labeled
"Technical Debt."
3. A few minor review things that popped out at me:

libc++: _LIBCPP_NONUNIQUE_RTTI_BIT needs a block comment to explain why.
llvm:
  include/LLVM/MC/MCAsmInfo.h: ^meast^means
  Linker Optimization Hints seem to be documented in Radar. Perhaps a bit more
commenting in here?
  I also saw a few 80-column violations scroll by, and several first line
comment length issues as well.

I look forward to arm64 and the new relocations landing in lld soon as well. :-)

Alex

On Mar 28, 2014, at 12:35 PM, James Grosbach wrote:
> All,
> 
> Attached below are the patches that make up the Apple ARM64 compiler
backend (in addition to compiler_rt, libc++ and lldb support), and we'd like
to start the process of integrating them into mainline LLVM.  We and ARM have
discussed a general approach to integrating them into mainline and look forward
to working through this with the community at large.
> 
> First a bit of context to help jump-start the discussion. The ARM64 backend
is a complete production quality implementation for ARM’s 64-bit architecture,
AArch64. It supports both iOS and Linux as target platforms. In the source tree,
the backend co-exists with the AArch64 backend without conflicts.
> 
> As a first step, we'd like to add this as a peer of the AArch64
backend.  While having two AArch64 backends in the tree long term doesn’t make
sense, this approach worked well when the Sparc backends were merged and for
AMDIL. This will allow us to work quickly towards merging the ports to create a
backend that is truly the best of both. The public AArch64 backend has attracted
an excellent core group of contributors all of whom have done great work and we
are very much looking forward to collaborating with them and the rest of the
community to make this happen.
> 
> Getting to that idillic end-state of one AArch64 backend in the tree which
encompasses the best of both is going to take a fair bit of work and a lot of
discussion amongst the interested parties. To get there, I think a rough roadmap
like the following makes sense:
> 
> 1. Import the ARM64 backend into the public tree so it’s easily accessible
for everyone to investigate.
> 2. Test it. Benchmark it. Explore it. Get data for the community to work
with about the state of the back end. ARM has some excellent data that will help
guide us here.
> 3. Identify the core backend to build on and to merge features and tuning
from the other to. The data I have seen so far leads me to believe ARM64 is the
correct choice here, but that’s a decision primarily for the contributors above.
> 4. Clean up the codebase (C++11-ify it, for example), fix any regressions
and test failures identified in benchmarking.
> 
> This will give us a backend that is a superset of both ARM64 and AArch64 in
terms both of performance and functionality. We can then consolidate to a single
backend, named AArch64 for consistency with the current public tree.
> 
> Regards,
> Jim Grosbach
> 
>
<arm64-libc++.patch.gz><arm64-compiler-rt.patch.gz><arm64-llvm.patch.gz><arm64-clang.patch.gz>_______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On Fri, Mar 28, 2014 at 12:35 PM, James Grosbach <grosbach at apple.com>
wrote:
> All,
>
> Attached below are the patches that make up the Apple ARM64 compiler
backend (in addition to compiler_rt, libc++ and lldb support), and we'd like
to start the process of integrating them into mainline LLVM.  We and ARM have
discussed a general approach to integrating them into mainline and look forward
to working through this with the community at large.
>
> First a bit of context to help jump-start the discussion. The ARM64 backend
is a complete production quality implementation for ARM's 64-bit
architecture, AArch64. It supports both iOS and Linux as target platforms. In
the source tree, the backend co-exists with the AArch64 backend without
conflicts.
>
> As a first step, we'd like to add this as a peer of the AArch64
backend.  While having two AArch64 backends in the tree long term doesn't
make sense, this approach worked well when the Sparc backends were merged and
for AMDIL. This will allow us to work quickly towards merging the ports to
create a backend that is truly the best of both. The public AArch64 backend has
attracted an excellent core group of contributors all of whom have done great
work and we are very much looking forward to collaborating with them and the
rest of the community to make this happen.
>
> Getting to that idillic end-state of one AArch64 backend in the tree which
encompasses the best of both is going to take a fair bit of work and a lot of
discussion amongst the interested parties. To get there, I think a rough roadmap
like the following makes sense:
>
> 1. Import the ARM64 backend into the public tree so it's easily
accessible for everyone to investigate.
For the record I've reviewed the lib/Basic stuff (non-neon,
non-mangling) code for arm64 for clang and that's OK to commit.

-eric

On 03/28/2014 11:27 PM, Alex Rosenberg wrote:
> My suggestions:
>
> 1. Do #1 below as you suggested as quickly as feasible.
> 2. String up and flog any and all executives and lawyers who forced the
world to wait since September for this. Perhaps beat them with a big stick
labeled "Technical Debt."
> 3. A few minor review things that popped out at me:
>
> libc++: _LIBCPP_NONUNIQUE_RTTI_BIT needs a block comment to explain why.
> llvm:
>    include/LLVM/MC/MCAsmInfo.h: ^meast^means
>    Linker Optimization Hints seem to be documented in Radar. Perhaps a bit
more commenting in here?
>    I also saw a few 80-column violations scroll by, and several first line
comment length issues as well.
Thanks to clang-format such style issues should be a no-brainer. It 
should definitely be run before integration.

Tobias

> 1. Import the ARM64 backend into the public tree so it's easily
> accessible for everyone to investigate.
> 2. Test it. Benchmark it. Explore it. Get data for the community to
> work with about the state of the back end. ARM has some excellent data
> that will help guide us here.
> 3. Identify the core backend to build on and to merge features and
> tuning from the other to. The data I have seen so far leads me to
> believe ARM64 is the correct choice here, but that's a decision
> primarily for the contributors above.
> 4. Clean up the codebase (C++11-ify it, for example), fix any
> regressions and test failures identified in benchmarking.
>
> This will give us a backend that is a superset of both ARM64 and
> AArch64 in terms both of performance and functionality. We can then
> consolidate to a single backend, named AArch64 for consistency with the
> current public tree.
Our MC Hammer[0] testing on this shows that the ARM64 backend has around a
4% failure rate overall, one criteria for a successful merge would certainly
be to retain the level architectural correctness that is currently present
in the AArch64 backend.

Looking at the failures that are present in the ARM64 backend, it doesn't
look like it would be too much work to fixup the MC layer to get this
testsuite passing.

[0] http://llvm.org/devmtg/2012-04-12/Slides/Richard_Barton.pdf

Regards,
Bradley Smith

Hi all,

Firstly thanks so much to Apple for open sourcing this and Tim for going
through the effort of committing it!

Along with Bradley I've been looking at this today from a perspective of
working out how best to get this merge completed. The one sentence summary
is "I think we should use ARM64 as a base".

My view on the backends is that the ARM64 backend is more performant but has
correctness issues. What we want in the end is a performant, correct
compiler. I think the best solution is to use ARM64 as the base for the
merge, because detecting and fixing correctness issues is significantly
easier than detecting and fixing performance issues.

The big pain issues I see merging from ARM64 to AArch64 are:
1.	Apple have created a fairly complete scheduling model already for
ARM64, and we'd have to merge the partial? model in AArch64 and theirs. We
risk regressing performance on Apple's targets here, and we can't
determine
ourselves whether we have or not. This is not ideal.
2.	Porting over the DAG-to-DAG optimizations and any other
optimizations that rely on the tablegen layout will be very tricky.
3.	The conditional compare pass is fairly comprehensive - we'd have to
port that over or rewrite it and that would be a lot of work.
4.	A very quick analysis last night indicated that ARM64 has
implemented just under half of the optimizations we discovered opportunities
for in SPEC and EEMBC. That's a fairly comprehensive number of
optimizations, and they won't all be easy to port.

The big pain issues I see going the other way, from AArch64 to ARM64 are:
1.	Functional regressions. These are fairly easy to detect - we have a
bunch of test suites and codegen faults are easy to spot (incorrect
results). I've spent the day looking at the MC Hammer failures, and there
aren't many very bad ones. Certainly none that are horrendous to fix.
2.	Performance on A53. But isn't it really just the scheduling model
that needs updating? There are no A53-specific optimizations in
Target/AArch64 that I know of.

Cheers,

James
> -----Original Message-----
> From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at
cs.uiuc.edu]
> On Behalf Of Bradley Smith
> Sent: 31 March 2014 18:19
> To: LLVM Developers Mailing List
> Subject: Re: [LLVMdev] Contributing the Apple ARM64 compiler backend
> 
> > 1. Import the ARM64 backend into the public tree so it's easily
> > accessible for everyone to investigate.
> > 2. Test it. Benchmark it. Explore it. Get data for the community to
> > work with about the state of the back end. ARM has some excellent data
> > that will help guide us here.
> > 3. Identify the core backend to build on and to merge features and
> > tuning from the other to. The data I have seen so far leads me to
> > believe ARM64 is the correct choice here, but that's a decision
> > primarily for the contributors above.
> > 4. Clean up the codebase (C++11-ify it, for example), fix any
> > regressions and test failures identified in benchmarking.
> >
> > This will give us a backend that is a superset of both ARM64 and
> > AArch64 in terms both of performance and functionality. We can then
> > consolidate to a single backend, named AArch64 for consistency with
the
> > current public tree.
> 
> Our MC Hammer[0] testing on this shows that the ARM64 backend has
> around a
> 4% failure rate overall, one criteria for a successful merge would
certainly
> be to retain the level architectural correctness that is currently present
> in the AArch64 backend.
> 
> Looking at the failures that are present in the ARM64 backend, it
doesn't
> look like it would be too much work to fixup the MC layer to get this
> testsuite passing.
> 
> [0] http://llvm.org/devmtg/2012-04-12/Slides/Richard_Barton.pdf
> 
> Regards,
> Bradley Smith
> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

> Our MC Hammer[0] testing on this shows that the ARM64 backend has
> around a
> 4% failure rate overall, one criteria for a successful merge would
> certainly
> be to retain the level architectural correctness that is currently
> present
> in the AArch64 backend.
> 
> Looking at the failures that are present in the ARM64 backend, it
> doesn't
> look like it would be too much work to fixup the MC layer to get this
> testsuite passing.
> 
> [0] http://llvm.org/devmtg/2012-04-12/Slides/Richard_Barton.pdf
We have spent a few days going through these failures in order to get a
better idea of how hard it would be to fix them. So far we have managed to
get the failure rate down to ~0.1% with 37 patches, although many of them
aren't clean enough to be pushed upstream just yet.

We have also done some benchmarking on an A57 SMM[0] of ARM64 vs the current
AArch64 backend (using just -O3), the results seem to favour the ARM64
backend. Coremark is ~10% faster and ~14% smaller, EEMBC on average is 4%
faster (with outliers of ~20% in either direction) and 10% smaller,
GeekBench is on average 27% faster.

It should be noted however that the A57 SMM board had a somewhat unrealistic
memory latency hence these numbers may not exactly reflect real hardware.

[0]
http://www.arm.com/products/tools/development-boards/versatile-express/soft-
macro-models.php

Regards,
Bradley Smith