llvm dev - Apr 2020 - Building libjpeg-turbo with LTO

Hi,

I have tried to build libjpeg-turbo
<https://github.com/libjpeg-turbo/libjpeg-turbo> with LTO in LLVM, using
both clangbut get many errors in lld that look like the following:

ld: error: undefined symbol: jpeg_std_error
>>> referenced by jcstest.c:76
>>>               lto.tmp:(main)
ld: error: undefined symbol: jpeg_CreateCompress
>>> referenced by jcstest.c:86
>>>               lto.tmp:(main)
ld: error: undefined symbol: jpeg_set_defaults
>>> referenced by jcstest.c:88
>>>               lto.tmp:(main)
ld: error: undefined symbol: jpeg_default_colorspace
>>> referenced by jcstest.c:90
>>>               lto.tmp:(main)
>>> referenced by jcstest.c:114
>>>               lto.tmp:(main)
This only occurs when compiling with the -flto flag. Has anyone been able
to build libjpeg-turbo with LTO? Are there any modifications I need to make
to the makefile or other configuration in order to do so? Thanks for your
help!

Best,
Shishir Jessu
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200408/4cca8811/attachment.html>

To correct a typo: I am using both clang 6.0.0, and a local build of clang
10.0.0, and each result in the same error.

Best,
Shishir Jessu

On Wed, Apr 8, 2020 at 12:22 PM Shishir V Jessu <shishir.jessu at
utexas.edu>
wrote:
> Hi,
>
> I have tried to build libjpeg-turbo
> <https://github.com/libjpeg-turbo/libjpeg-turbo> with LTO in LLVM,
using
> both clangbut get many errors in lld that look like the following:
>
> ld: error: undefined symbol: jpeg_std_error
> >>> referenced by jcstest.c:76
> >>>               lto.tmp:(main)
>
> ld: error: undefined symbol: jpeg_CreateCompress
> >>> referenced by jcstest.c:86
> >>>               lto.tmp:(main)
>
> ld: error: undefined symbol: jpeg_set_defaults
> >>> referenced by jcstest.c:88
> >>>               lto.tmp:(main)
>
> ld: error: undefined symbol: jpeg_default_colorspace
> >>> referenced by jcstest.c:90
> >>>               lto.tmp:(main)
> >>> referenced by jcstest.c:114
> >>>               lto.tmp:(main)
>
> This only occurs when compiling with the -flto flag. Has anyone been able
> to build libjpeg-turbo with LTO? Are there any modifications I need to make
> to the makefile or other configuration in order to do so? Thanks for your
> help!
>
> Best,
> Shishir Jessu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200408/09118d8c/attachment.html>

Are the object files for jcstest.c and the source files defining these
symbols being directly LTO linked together, or are the defs first LTO
linked into a shared library? It would be helpful to see the build commands
involved.
Teresa

On Wed, Apr 8, 2020 at 10:25 AM Shishir V Jessu via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> To correct a typo: I am using both clang 6.0.0, and a local build of clang
> 10.0.0, and each result in the same error.
>
> Best,
> Shishir Jessu
>
> On Wed, Apr 8, 2020 at 12:22 PM Shishir V Jessu <shishir.jessu at
utexas.edu>
> wrote:
>
>> Hi,
>>
>> I have tried to build libjpeg-turbo
>> <https://github.com/libjpeg-turbo/libjpeg-turbo> with LTO in
LLVM, using
>> both clangbut get many errors in lld that look like the following:
>>
>> ld: error: undefined symbol: jpeg_std_error
>> >>> referenced by jcstest.c:76
>> >>>               lto.tmp:(main)
>>
>> ld: error: undefined symbol: jpeg_CreateCompress
>> >>> referenced by jcstest.c:86
>> >>>               lto.tmp:(main)
>>
>> ld: error: undefined symbol: jpeg_set_defaults
>> >>> referenced by jcstest.c:88
>> >>>               lto.tmp:(main)
>>
>> ld: error: undefined symbol: jpeg_default_colorspace
>> >>> referenced by jcstest.c:90
>> >>>               lto.tmp:(main)
>> >>> referenced by jcstest.c:114
>> >>>               lto.tmp:(main)
>>
>> This only occurs when compiling with the -flto flag. Has anyone been
able
>> to build libjpeg-turbo with LTO? Are there any modifications I need to
make
>> to the makefile or other configuration in order to do so? Thanks for
your
>> help!
>>
>> Best,
>> Shishir Jessu
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200408/1569e317/attachment.html>

Adding a couple of lld folks.

I helped Shishir debug this, the link line looked like:
   /home/sjessu/build/bin/clang -O0 -flto -o jcstest jcstest.o
 ./.libs/libjpeg.a
and the issue was that libjpeg.a was created with the system ar instead of
llvm-ar. It worked when recreating libjpeg.a with llvm-ar.

I noticed that the lld code has some special handling for the case
when there is a missing symbol table, which often happens with system ar
created archives containing bitcode. I noticed that the lld code will
sometimes emit an error, but actually contains a special hack to handle
archives containing *only* bitcode objects, so that they are handled
correctly even when there is no symbol table because it was created with
the system ar. Unfortunately, in this case it neither gave an error nor did
the special handling, because libjpeg.a also contains some native objects
and thus had a non-zero symbol table. I created a version of libjpeg.a
using the system library and containing only the bitcode objects, and
confirmed it links fine with lld (the native objects weren't needed in this
case). BTW this is the code in ELF/Driver.cpp LinkerDriver::addFile.

Would it be possible to extend the hack in lld to handle cases like this
with some bitcode objects and some non-bitcode objects, so that the bitcode
objects are not simply ignored?

Thanks,
Teresa

On Wed, Apr 8, 2020 at 10:25 AM Shishir V Jessu via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> To correct a typo: I am using both clang 6.0.0, and a local build of clang
> 10.0.0, and each result in the same error.
>
> Best,
> Shishir Jessu
>
> On Wed, Apr 8, 2020 at 12:22 PM Shishir V Jessu <shishir.jessu at
utexas.edu>
> wrote:
>
>> Hi,
>>
>> I have tried to build libjpeg-turbo
>> <https://github.com/libjpeg-turbo/libjpeg-turbo> with LTO in
LLVM, using
>> both clangbut get many errors in lld that look like the following:
>>
>> ld: error: undefined symbol: jpeg_std_error
>> >>> referenced by jcstest.c:76
>> >>>               lto.tmp:(main)
>>
>> ld: error: undefined symbol: jpeg_CreateCompress
>> >>> referenced by jcstest.c:86
>> >>>               lto.tmp:(main)
>>
>> ld: error: undefined symbol: jpeg_set_defaults
>> >>> referenced by jcstest.c:88
>> >>>               lto.tmp:(main)
>>
>> ld: error: undefined symbol: jpeg_default_colorspace
>> >>> referenced by jcstest.c:90
>> >>>               lto.tmp:(main)
>> >>> referenced by jcstest.c:114
>> >>>               lto.tmp:(main)
>>
>> This only occurs when compiling with the -flto flag. Has anyone been
able
>> to build libjpeg-turbo with LTO? Are there any modifications I need to
make
>> to the makefile or other configuration in order to do so? Thanks for
your
>> help!
>>
>> Best,
>> Shishir Jessu
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200408/dd8ad06b/attachment.html>