llvm dev - Feb 2017 - Problems using Clang with LLD on embedded ARM

Hi,

I stopped into IRC to ask about a problem I've been having using Clang in
conjunction with LLD to compile and link for an embedded project on
Cortex-M ARM processor.

First, I am able to separately compile with a call to clang and link with a
call to lld, but I cannot use clang to link using lld using the
-fuse-ld=lld flag. I have the output from `clang -v -fuse-ld=lld -target
arm-none-eabi main.c -o main` here [1], where calling `clang -target
arm-none-eabi -c main.c -o main.o` and `ld.lld main.o -o main` work fine.
Doing the compiling and linking for my host computer works just fine using
-fuse-ld=lld, and I verified that it is using LLD by checking the .comments
section in the generated elf.

Second, combining LTO with specifying any specific arm architecture seems
to break linking with LLD. That is, I compile the object files with LTO
(either -flto or -flto=thin) and, in addition to specifying `-target
arm-none-eabi`, I specify an architecture specific flag (such as
-march=armv7e-m, -mcpu=cortex-m4, or -mthumb, or instead I use -target
armv7em-none-eabi), then while calling lld to link I get an error like:
ld.lld: error: main.o: could not infer e_machine from bitcode target triple
thumbv7em-none--eabi
LLD 4.0.0

I am working on Linux and these two problems seem to exist on both 4.0rc2
and the current head.

I am really happy with the progress LLVM has made, with the upcoming 4.0
release, I have managed to compile and link programs for my embedded
projects without any of the gcc arm toolchain! There are still some
shortcomings with the linker scripts compared to ld that make those a bit
trickier to work with, even with the added MEMORY section parsing in the
current head. Other than the above, the only other bother I've hit is
having to re-link with `--oformat binary` to get raw binary output, since
there I haven't found an objcopy equivalent in llvm. I can add feature
requests for these missing features, as well as the bug reports for the
above two issues, on the bugzilla if that is appropriate.

Thanks for the help,

Sean

[1] https://gist.github.com/flacjacket/35d92ea619af96352d5ac06844f2f1b5
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170225/2f6aa864/attachment.html>

On Sat, Feb 25, 2017 at 9:38 PM, Sean Vig via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi,
>
> I stopped into IRC to ask about a problem I've been having using Clang
in
> conjunction with LLD to compile and link for an embedded project on
> Cortex-M ARM processor.
>
> First, I am able to separately compile with a call to clang and link with
> a call to lld, but I cannot use clang to link using lld using the
> -fuse-ld=lld flag. I have the output from `clang -v -fuse-ld=lld -target
> arm-none-eabi main.c -o main` here [1], where calling `clang -target
> arm-none-eabi -c main.c -o main.o` and `ld.lld main.o -o main` work fine.
> Doing the compiling and linking for my host computer works just fine using
> -fuse-ld=lld, and I verified that it is using LLD by checking the .comments
> section in the generated elf.
>
For that triple, Clang seems to be calling into GCC driver
(/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize
-fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd
selects ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it
doesn't work like that apparently).

@Renato: Do you know why Clang is forwarding to GCC like this for linking?
If we need to live with this arrangement for a long time we should
definitely invest in teaching GCC about -fuse-ld=lld

@Davide: is there any chance you could revive that GCC patch to add support
for LLD?

(though it does seem like we still fail H.J.'s test case in
https://bugs.llvm.org//show_bug.cgi?id=28414 has not yet been fixed. I
further reduced it. Seems to be related to symbol versioning. We don't
recognize that `.symver bar, foo@@VERS` is a definition of `foo`.
)

> Second, combining LTO with specifying any specific arm architecture seems
> to break linking with LLD. That is, I compile the object files with LTO
> (either -flto or -flto=thin) and, in addition to specifying `-target
> arm-none-eabi`, I specify an architecture specific flag (such as
> -march=armv7e-m, -mcpu=cortex-m4, or -mthumb, or instead I use -target
> armv7em-none-eabi), then while calling lld to link I get an error like:
> ld.lld: error: main.o: could not infer e_machine from bitcode target
> triple thumbv7em-none--eabi
> LLD 4.0.0
>
If you are building your own LLD, can you see if the attached patch fixes
it for you? If not, can you add `--reproduce /tmp/repro.tar` to your link
command line and upload repro.tar somewhere for us?

@Renato, do you know if there is any way for us to
implement getBitcodeMachineKind in a sane way so that we don't fill in a
long tail of cases of inferring EM_* from the triple? It seems that the
EM_* choices are buried very deep in the backend, so it might not be
feasible :(


>
> I am working on Linux and these two problems seem to exist on both 4.0rc2
> and the current head.
>
> I am really happy with the progress LLVM has made, with the upcoming 4.0
> release, I have managed to compile and link programs for my embedded
> projects without any of the gcc arm toolchain! There are still some
> shortcomings with the linker scripts compared to ld that make those a bit
> trickier to work with, even with the added MEMORY section parsing in the
> current head.
>
What kind of issues have you been running into?

Linker scripts are certainly one area that we have been slowly filling in
the long tail of compatibility, it would be great if you filed bugs with
any issues you encounter! If the files are small enough and you can share
them, it can be as simple as just adding `--reproduce /tmp/repro.tar` to
your link command line (or setting LLD_REPRODUCE=/tmp/repro.tar in the
environment) and attach repro.tar to a bug report whenever you encounter an
issue (even if repro.tar is larger, you can try hosting on google drive /
dropbox etc., compressing with xz also often helps). Or if it is a parsing
problem just attaching the problematic linker script to the bug report
might be enough.


> Other than the above, the only other bother I've hit is having to
re-link
> with `--oformat binary` to get raw binary output, since there I haven't
> found an objcopy equivalent in llvm. I can add feature requests for these
> missing features, as well as the bug reports for the above two issues, on
> the bugzilla if that is appropriate.
>

Regarding objcopy, that is something that we have wanted to write for a
long time (but it has never quite been a priority). As objcopy supports a
bewildering set of different options, it would be useful for you to file a
bug about what specific options/features you need. We don't really have a
bugzilla component for objcopy yet, so you can just throw it under llvm-nm
(another one of the binutils, close enough) in
https://bugs.llvm.org/enter_bug.cgi?product=tools

-- Sean Silva

>
> Thanks for the help,
>
> Sean
>
> [1] https://gist.github.com/flacjacket/35d92ea619af96352d5ac06844f2f1b5
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170226/dafdf386/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: thumb.patch
Type: text/x-patch
Size: 371 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170226/dafdf386/attachment.bin>

Sean Silva via llvm-dev <llvm-dev at lists.llvm.org> writes:
> For that triple, Clang seems to be calling into GCC driver
> (/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize
> -fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd
> selects ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it
> doesn't work like that apparently).
Gosh, that is an old issue. I was sure we had converted to used ld
everywhere and deleted the old code for using the gcc driver as a
linker.

Doing that is probably the correct way to fix this issue. Having gcc
itelf support -fuse-ld=lld would be an awesome independent improvement.

Cheers,
Rafael

On Sun, Feb 26, 2017 at 3:10 AM, Sean Silva <chisophugis at gmail.com>
wrote:
>
>
> On Sat, Feb 25, 2017 at 9:38 PM, Sean Vig via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>>
>> Hi,
>>
>> I stopped into IRC to ask about a problem I've been having using
Clang in
>> conjunction with LLD to compile and link for an embedded project on
Cortex-M
>> ARM processor.
>>
>> First, I am able to separately compile with a call to clang and link
with
>> a call to lld, but I cannot use clang to link using lld using the
>> -fuse-ld=lld flag. I have the output from `clang -v -fuse-ld=lld
-target
>> arm-none-eabi main.c -o main` here [1], where calling `clang -target
>> arm-none-eabi -c main.c -o main.o` and `ld.lld main.o -o main` work
fine.
>> Doing the compiling and linking for my host computer works just fine
using
>> -fuse-ld=lld, and I verified that it is using LLD by checking the
.comments
>> section in the generated elf.
>
>
> For that triple, Clang seems to be calling into GCC driver
> (/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize
> -fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd
selects
> ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it
doesn't
> work like that apparently).
>
> @Renato: Do you know why Clang is forwarding to GCC like this for linking?
> If we need to live with this arrangement for a long time we should
> definitely invest in teaching GCC about -fuse-ld=lld
>
> @Davide: is there any chance you could revive that GCC patch to add support
> for LLD?
I can probably find some time to revive that support next week
(sorry it's GDC week this week and I won't likely have time
for anything compiler-related).

--
David

Hi Sean

On Sun, Feb 26, 2017 at 5:10 AM, Sean Silva <chisophugis at gmail.com>
wrote:
>
> For that triple, Clang seems to be calling into GCC driver
> (/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize
> -fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd
> selects ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it
> doesn't work like that apparently).
>
> @Renato: Do you know why Clang is forwarding to GCC like this for linking?
> If we need to live with this arrangement for a long time we should
> definitely invest in teaching GCC about -fuse-ld=lld
>
> @Davide: is there any chance you could revive that GCC patch to add
> support for LLD?
>
> (though it does seem like we still fail H.J.'s test case in
> https://bugs.llvm.org//show_bug.cgi?id=28414 has not yet been fixed. I
> further reduced it. Seems to be related to symbol versioning. We don't
> recognize that `.symver bar, foo@@VERS` is a definition of `foo`.
> )
>
>
> If you are building your own LLD, can you see if the attached patch fixes
> it for you? If not, can you add `--reproduce /tmp/repro.tar` to your link
> command line and upload repro.tar somewhere for us?
>
Thanks! I applied that patch and it links with -flto now, but it looks like
there are now some other problems. First, it seems to be dropping things
that I try to KEEP in the linker script, even when I don't use
--gc-sections, namely my interrupt vector table. Second, when using thin
LTO, it starts trying to link symbols __aeabi_memclr4 and __aeabi_uidiv,
which I don't think it should since I am using -ffreestanding -fno-builtins
when compiling (with full lto or no lto, it does not try to link these
symbols).

I just sent an email to register for a bugzilla account, I can report there
with a minimally reproducible example for these problems.

>
> @Renato, do you know if there is any way for us to
> implement getBitcodeMachineKind in a sane way so that we don't fill in
a
> long tail of cases of inferring EM_* from the triple? It seems that the
> EM_* choices are buried very deep in the backend, so it might not be
> feasible :(
>
> What kind of issues have you been running into?
>
> Linker scripts are certainly one area that we have been slowly filling in
> the long tail of compatibility, it would be great if you filed bugs with
> any issues you encounter! If the files are small enough and you can share
> them, it can be as simple as just adding `--reproduce /tmp/repro.tar` to
> your link command line (or setting LLD_REPRODUCE=/tmp/repro.tar in the
> environment) and attach repro.tar to a bug report whenever you encounter an
> issue (even if repro.tar is larger, you can try hosting on google drive /
> dropbox etc., compressing with xz also often helps). Or if it is a parsing
> problem just attaching the problematic linker script to the bug report
> might be enough.
>
I haven't hit anything that looked like a bug, just features that aren't
implemented, or that work slightly differently between ld linker scripts.
I'd be happy to make additional feature requests for those on the bug
tracker, too.


Regarding objcopy, that is something that we have wanted to write for
a
> long time (but it has never quite been a priority). As objcopy supports a
> bewildering set of different options, it would be useful for you to file a
> bug about what specific options/features you need. We don't really have
a
> bugzilla component for objcopy yet, so you can just throw it under llvm-nm
> (another one of the binutils, close enough) in https://bugs.llvm.org/enter
> _bug.cgi?product=tools
>
Sounds good.

Thanks for the patch and the help.

Sean


>
> -- Sean Silva
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170226/fd795532/attachment.html>

On 26 February 2017 at 11:10, Sean Silva <chisophugis at gmail.com>
wrote:
> For that triple, Clang seems to be calling into GCC driver
> (/usr/bin/arm-none-eabi-gcc) for linking and GCC doesn't recognize
> -fuse-ld=lld (supposedly -fuse-ld=gold selects ld.gold, -fuse-ld=bfd
selects
> ld.bfd and you would expect -fuse-lld=lld to select ld.lld, but it
doesn't
> work like that apparently).
This is really odd, but since LLD is not officially supported on ARM
yet (Peter is almost there!), we haven't got around using it for
anything serious.

It smells like yet-another clang driver kerfuffle when cross-compiling.

cheers,
--renato