When using the ARM cross compiler we've run into an issue with LTO and optimized libraries. Consider you have an optimized library opt.a, which contains a version of memcpy. Compiling with LTO (something like), clang myTest.c opt.a -flto -o myTest causes myTest.c to get compiled to bitcode. Then the bitcode gets passed to the linker. The linker looks through the bitcode (via the gold plugin) searching for symbols it needs to resolve. But any memcpy() calls in myTest.c got converted to llvm.memcpy() calls in the bitcode. Any symbols that start with 'llvm.' get ignored in LTOModule.cpp. So the linker doesn't know that it needs memcpy. Now once the linker processes opt.a it doesn't know that it needs memcpy, so the linker doesn't keep the definition of memcpy from opt.a. After the (full) compilation of myTest is complete, the linker winds up trying to pull memcpy from libc. But once it hits opt.a to process again, it sees a multiple definition issue. It appears that optimized libraries and LTO don't currently mix well. Does anyone have any insight about how this should be accomplished? Daniel -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140609/85389d85/attachment.html>
Daniel, This has been a recurring issue with LTO on darwin too. To fix most issues, I added an enhancement to the linker whenever it was dealing with LTO. After the LTO codegen was done, if any new undefined symbols appeared, the link made another pass looking for definitions in dylibs (DSO). This solved almost all issues because memcpy and friends are implemented in libSystem.dylib on darwin. The remaining cases were for static programs (embedded, EFI, etc) which don’t use dylibs. For them, the current work around is to add -U as needed on the linker command line (e.g. -U _memcpy) which forces the linker to retain a copy of that runtime utility function. -Nick On Jun 9, 2014, at 1:26 PM, Daniel Stewart <stewartd at codeaurora.org> wrote:> When using the ARM cross compiler we’ve run into an issue with LTO and optimized libraries. > > Consider you have an optimized library opt.a, which contains a version of memcpy. > > Compiling with LTO (something like), > clang myTest.c opt.a –flto –o myTest > > causes myTest.c to get compiled to bitcode. > > Then the bitcode gets passed to the linker. The linker looks through the bitcode (via the gold plugin) searching for symbols it needs to resolve. But any memcpy() calls in myTest.c got converted to llvm.memcpy() calls in the bitcode. Any symbols that start with ‘llvm.’ get ignored in LTOModule.cpp. So the linker doesn’t know that it needs memcpy. Now once the linker processes opt.a it doesn’t know that it needs memcpy, so the linker doesn’t keep the definition of memcpy from opt.a. > > After the (full) compilation of myTest is complete, the linker winds up trying to pull memcpy from libc. But once it hits opt.a to process again, it sees a multiple definition issue. > > It appears that optimized libraries and LTO don’t currently mix well. Does anyone have any insight about how this should be accomplished? > > Daniel > > -- > Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140609/940f53f4/attachment.html>
On 9 June 2014 16:26, Daniel Stewart <stewartd at codeaurora.org> wrote:> When using the ARM cross compiler we’ve run into an issue with LTO and > optimized libraries. > > > > Consider you have an optimized library opt.a, which contains a version of > memcpy. > > > > Compiling with LTO (something like), > > clang myTest.c opt.a –flto –o myTest > > > > causes myTest.c to get compiled to bitcode. > > > > Then the bitcode gets passed to the linker. The linker looks through the > bitcode (via the gold plugin) searching for symbols it needs to resolve. But > any memcpy() calls in myTest.c got converted to llvm.memcpy() calls in the > bitcode. Any symbols that start with ‘llvm.’ get ignored in LTOModule.cpp. > So the linker doesn’t know that it needs memcpy. Now once the linker > processes opt.a it doesn’t know that it needs memcpy, so the linker doesn’t > keep the definition of memcpy from opt.a. > > > > After the (full) compilation of myTest is complete, the linker winds up > trying to pull memcpy from libc. But once it hits opt.a to process again, it > sees a multiple definition issue. > > > > It appears that optimized libraries and LTO don’t currently mix well. Does > anyone have any insight about how this should be accomplished? >I don't think gold has an option for doing a full second pass. What is has is the ADD_INPUT_LIBRARY callback which should cause it to look at the provided library again. It is probably a good idea to try to find a convenient way to use it from llvm. If the performance impact is not too horrible, it might make sense to change gold-plugin.cpp to use that callback for every .a that it sees that contains at least one native object file. Cheers, Rafael
I had tried to use the add_input _library as
(*add_input_library)("opt.a");
but got a seg fault every time. I'm not sure exactly how to use it,
apparently.
I also noticed that in gold_plugin.cpp:onload(), that add_input_library is
defined as
case LDPT_ADD_INPUT_LIBRARY:
add_input_library = tv->tv_u.tv_add_input_file;
break;
Seems odd that add_input_library doesn't actually use the
tv_add_input_library. I changed mine locally to point to add_input_library, but
still got a crash.
Am I using the callback correctly?
-----Original Message-----
From: Rafael Espíndola [mailto:rafael.espindola at gmail.com]
Sent: Monday, June 09, 2014 5:51 PM
To: Daniel Stewart
Cc: LLVM Developers Mailing List
Subject: Re: [LLVMdev] LTO and Optimized libraries don't mix
On 9 June 2014 16:26, Daniel Stewart <stewartd at codeaurora.org>
wrote:> When using the ARM cross compiler we’ve run into an issue with LTO and
> optimized libraries.
>
>
>
> Consider you have an optimized library opt.a, which contains a version
> of memcpy.
>
>
>
> Compiling with LTO (something like),
>
> clang myTest.c opt.a –flto –o myTest
>
>
>
> causes myTest.c to get compiled to bitcode.
>
>
>
> Then the bitcode gets passed to the linker. The linker looks through
> the bitcode (via the gold plugin) searching for symbols it needs to
> resolve. But any memcpy() calls in myTest.c got converted to
> llvm.memcpy() calls in the bitcode. Any symbols that start with ‘llvm.’ get
ignored in LTOModule.cpp.
> So the linker doesn’t know that it needs memcpy. Now once the linker
> processes opt.a it doesn’t know that it needs memcpy, so the linker
> doesn’t keep the definition of memcpy from opt.a.
>
>
>
> After the (full) compilation of myTest is complete, the linker winds
> up trying to pull memcpy from libc. But once it hits opt.a to process
> again, it sees a multiple definition issue.
>
>
>
> It appears that optimized libraries and LTO don’t currently mix well.
> Does anyone have any insight about how this should be accomplished?
>
I don't think gold has an option for doing a full second pass. What is has
is the ADD_INPUT_LIBRARY callback which should cause it to look at the provided
library again.
It is probably a good idea to try to find a convenient way to use it from llvm.
If the performance impact is not too horrible, it might make sense to change
gold-plugin.cpp to use that callback for every .a that it sees that contains at
least one native object file.
Cheers,
Rafael
Actually I used it as
add_input_library("opt"), without the .a, as the documents say use it
as you would with a -l. But the result is still the same. I also get a crash
when I try to use the add_input_file() as well.
-----Original Message-----
From: Daniel Stewart [mailto:stewartd at codeaurora.org]
Sent: Tuesday, June 10, 2014 8:13 AM
To: 'Rafael Espíndola'
Cc: 'LLVM Developers Mailing List'
Subject: RE: [LLVMdev] LTO and Optimized libraries don't mix
I had tried to use the add_input _library as
(*add_input_library)("opt.a");
but got a seg fault every time. I'm not sure exactly how to use it,
apparently.
I also noticed that in gold_plugin.cpp:onload(), that add_input_library is
defined as
case LDPT_ADD_INPUT_LIBRARY:
add_input_library = tv->tv_u.tv_add_input_file; break;
Seems odd that add_input_library doesn't actually use the
tv_add_input_library. I changed mine locally to point to add_input_library, but
still got a crash.
Am I using the callback correctly?
-----Original Message-----
From: Rafael Espíndola [mailto:rafael.espindola at gmail.com]
Sent: Monday, June 09, 2014 5:51 PM
To: Daniel Stewart
Cc: LLVM Developers Mailing List
Subject: Re: [LLVMdev] LTO and Optimized libraries don't mix
On 9 June 2014 16:26, Daniel Stewart <stewartd at codeaurora.org>
wrote:> When using the ARM cross compiler we’ve run into an issue with LTO and
> optimized libraries.
>
>
>
> Consider you have an optimized library opt.a, which contains a version
> of memcpy.
>
>
>
> Compiling with LTO (something like),
>
> clang myTest.c opt.a –flto –o myTest
>
>
>
> causes myTest.c to get compiled to bitcode.
>
>
>
> Then the bitcode gets passed to the linker. The linker looks through
> the bitcode (via the gold plugin) searching for symbols it needs to
> resolve. But any memcpy() calls in myTest.c got converted to
> llvm.memcpy() calls in the bitcode. Any symbols that start with ‘llvm.’ get
ignored in LTOModule.cpp.
> So the linker doesn’t know that it needs memcpy. Now once the linker
> processes opt.a it doesn’t know that it needs memcpy, so the linker
> doesn’t keep the definition of memcpy from opt.a.
>
>
>
> After the (full) compilation of myTest is complete, the linker winds
> up trying to pull memcpy from libc. But once it hits opt.a to process
> again, it sees a multiple definition issue.
>
>
>
> It appears that optimized libraries and LTO don’t currently mix well.
> Does anyone have any insight about how this should be accomplished?
>
I don't think gold has an option for doing a full second pass. What is has
is the ADD_INPUT_LIBRARY callback which should cause it to look at the provided
library again.
It is probably a good idea to try to find a convenient way to use it from llvm.
If the performance impact is not too horrible, it might make sense to change
gold-plugin.cpp to use that callback for every .a that it sees that contains at
least one native object file.
Cheers,
Rafael