llvm dev - Jun 2019 - Running distributed thinLTO without thin archives.

Thanks!

Question about the final link step:

Do I provide all the object files to the link step, i.e. something like:
clang++ -o thinlto main-native.o lib/lib-native.o src/lib-native.o

Do I need to provide --start-lib markers on that final link step as well?

Tanoy

On Tue, Jun 18, 2019 at 10:37 AM Teresa Johnson <tejohnson at google.com>
wrote:
> Hi Tanoy,
>
> You can't use distributed ThinLTO with archives (thin or not), at least
> not today. The reason is that we need to be able to identify specific
> bitcode object files to import from in the backends, and that logic does
> not know how to deal with objects within archives. We do distributed
> ThinLTO in our builds but don't use .a files, rather, we use
> --start-lib/--end-lib around the files that would be in the same archive
> when performing the thin link. I.e. if you change your thin link to be:
>
> clang++ -flto=thin -o index -O3
>  -Wl,-plugin-opt,thinlto-index-only=thinlto.objects
> -Wl,-plugin-opt,thinlto-emit-imports-files main.o --start-lib lib/lib.o
> src/lib.o --end-lib
>
> things should work.
>
> Note you also need to do the ThinLTO backend compile for each of the
> archive constituents anyway, e.g. something like:
>   clang++ -c -x ir lib/lib.o -O3 -flto=thin -o lib/lib-native.o
> -fthinlto-index=lib/lib.o.thinlto.bc
> etc
>
> HTH,
> Teresa
>
> On Mon, Jun 17, 2019 at 2:46 PM Tanoy Sinha via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> I'm trying to run distributed ThinLTO without thin archives.
>> When I do, I get an error in the optimizer when clang tries to open a
>> nonexistent file:
>>
>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c main.cpp -o main.o
>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c lib/lib.cpp -o
lib/lib.o
>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c src/lib.cpp -o
src/lib.o
>> llvm-ar -format gnu qcs lib.a lib/lib.o src/lib.o
>> clang++ -flto=thin -o index -O3
>>  -Wl,-plugin-opt,thinlto-index-only=thinlto.objects
>> -Wl,-plugin-opt,thinlto-emit-imports-files main.o lib.a
>> clang++ -c -x ir main.o -O3 -flto=thin -o main-native.o
>> -fthinlto-index=main.o.thinlto.bc
>> Error loading imported file 'lib.a.llvm.2596.lib.cpp': No such
file or
>> directory
>>
>> In this case, gold has registered the modules within my archive with
>> ThinLTO.
>> The string "lib.a.llvm.2596.lib.cpp" is generated with the
archive in
>> question, plus an offset indicating where in the archive the particular
>> object file is.
>> Unfortunately, when the optimizer tries to include the proper modules,
>> it's naively looking for a bitcode file with the name of the string
>> provided, but there's obviously no
"lib.a.llvm.2596.lib.cpp" for it to open.
>>
>> Has anyone else tried to get clang to understand distributed ThinLTO
when
>> using non thin archives?
>> Is there some way to get clang to understand these out of the box?
>>
>> I'm actually a little confused about the ".cpp" in
>> "lib.a.llvm.2596.lib.cpp".
>> Seems like it should be a ".o"?
>>
>> It didn't seem like there was anything out of the box that
supported
>> this.
>> I was looking at having clang actually read in the archive file and
>> register the correct bitcode module.
>> I wanted to run it by the list to get some second opinions before I
>> started that.
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
> --
> Teresa Johnson |  Software Engineer |  tejohnson at google.com |
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190618/82113c6c/attachment.html>

On Tue, Jun 18, 2019 at 1:40 PM Tanoy Sinha <tsinha at gmail.com> wrote:
> Thanks!
>
> Question about the final link step:
>
> Do I provide all the object files to the link step, i.e. something like:
> clang++ -o thinlto main-native.o lib/lib-native.o src/lib-native.o
>
> Do I need to provide --start-lib markers on that final link step as well?
>
No and No. After the thin link the linker has already done its symbol
resolution, and using --start-lib/--end-lib in the final link can muck with
that. However, the list of files the linker selected in the right order is
emitted in the argument given to thinlto-index-only (thinlto.objects in
your case below). You can pass that to the native link via the "@"
option:
  clang++ -o thinlto @thinlto.objects
however you need to deal with the fact that this file contains the original
bitcode names (not the names you gave it in your backend step like
main-native.o, etc)
There are 2 options for correcting the names:
1) Manually rename in thinlto.objects
2) Use the thinlto_prefix_replace=oldprefix;newprefix plugin option, to
replace the old path prefix of the input bitcode files with a new path
prefix. In your case the old prefix is "", so you could do something
like
"-Wl,-plugin-opt,thinlto_prefix_replace=:native/". This should do 2
things:
1) the generated .thinlto.bc index files and the .imports files will be put
under a "native/" subdirectory; 2) the paths in thinlto.objects should
also
have the "native/" prefix. If you use that prefix in your LTO backend
clang
invocations (e.g. -o native/main.o instead of main-native.o), then
thinlto.objects can just be passed directly to the final link via "@"
without any modification.

Note that if your thin link included any already native files/libraries,
those still need to be passed as the thinlto.objects only includes those
that were originally bitcode.

Teresa

> Tanoy
>
> On Tue, Jun 18, 2019 at 10:37 AM Teresa Johnson <tejohnson at
google.com>
> wrote:
>
>> Hi Tanoy,
>>
>> You can't use distributed ThinLTO with archives (thin or not), at
least
>> not today. The reason is that we need to be able to identify specific
>> bitcode object files to import from in the backends, and that logic
does
>> not know how to deal with objects within archives. We do distributed
>> ThinLTO in our builds but don't use .a files, rather, we use
>> --start-lib/--end-lib around the files that would be in the same
archive
>> when performing the thin link. I.e. if you change your thin link to be:
>>
>> clang++ -flto=thin -o index -O3
>>  -Wl,-plugin-opt,thinlto-index-only=thinlto.objects
>> -Wl,-plugin-opt,thinlto-emit-imports-files main.o --start-lib lib/lib.o
>> src/lib.o --end-lib
>>
>> things should work.
>>
>> Note you also need to do the ThinLTO backend compile for each of the
>> archive constituents anyway, e.g. something like:
>>   clang++ -c -x ir lib/lib.o -O3 -flto=thin -o lib/lib-native.o
>> -fthinlto-index=lib/lib.o.thinlto.bc
>> etc
>>
>> HTH,
>> Teresa
>>
>> On Mon, Jun 17, 2019 at 2:46 PM Tanoy Sinha via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> I'm trying to run distributed ThinLTO without thin archives.
>>> When I do, I get an error in the optimizer when clang tries to open
a
>>> nonexistent file:
>>>
>>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c main.cpp -o main.o
>>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c lib/lib.cpp -o
lib/lib.o
>>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c src/lib.cpp -o
src/lib.o
>>> llvm-ar -format gnu qcs lib.a lib/lib.o src/lib.o
>>> clang++ -flto=thin -o index -O3
>>>  -Wl,-plugin-opt,thinlto-index-only=thinlto.objects
>>> -Wl,-plugin-opt,thinlto-emit-imports-files main.o lib.a
>>> clang++ -c -x ir main.o -O3 -flto=thin -o main-native.o
>>> -fthinlto-index=main.o.thinlto.bc
>>> Error loading imported file 'lib.a.llvm.2596.lib.cpp': No
such file or
>>> directory
>>>
>>> In this case, gold has registered the modules within my archive
with
>>> ThinLTO.
>>> The string "lib.a.llvm.2596.lib.cpp" is generated with
the archive in
>>> question, plus an offset indicating where in the archive the
particular
>>> object file is.
>>> Unfortunately, when the optimizer tries to include the proper
modules,
>>> it's naively looking for a bitcode file with the name of the
string
>>> provided, but there's obviously no
"lib.a.llvm.2596.lib.cpp" for it to open.
>>>
>>> Has anyone else tried to get clang to understand distributed
ThinLTO
>>> when using non thin archives?
>>> Is there some way to get clang to understand these out of the box?
>>>
>>> I'm actually a little confused about the ".cpp" in
>>> "lib.a.llvm.2596.lib.cpp".
>>> Seems like it should be a ".o"?
>>>
>>> It didn't seem like there was anything out of the box that
supported
>>> this.
>>> I was looking at having clang actually read in the archive file and
>>> register the correct bitcode module.
>>> I wanted to run it by the list to get some second opinions before I
>>> started that.
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
>>
>> --
>> Teresa Johnson |  Software Engineer |  tejohnson at google.com |
>>
>
-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190618/a2e9b4d1/attachment.html>

Thanks very much!

One more question:

If I wanted to implement archive support for distributed ThinLTO, what all
would I need to do?

I know I need to pull out the bitcode module during the optimizer step,
which I've looked at.

What else would I need to change?

On Tue, Jun 18, 2019 at 4:58 PM Teresa Johnson <tejohnson at google.com>
wrote:
>
> On Tue, Jun 18, 2019 at 1:40 PM Tanoy Sinha <tsinha at gmail.com>
wrote:
>
>> Thanks!
>>
>> Question about the final link step:
>>
>> Do I provide all the object files to the link step, i.e. something
like:
>> clang++ -o thinlto main-native.o lib/lib-native.o src/lib-native.o
>>
>
>> Do I need to provide --start-lib markers on that final link step as
well?
>>
>
> No and No. After the thin link the linker has already done its symbol
> resolution, and using --start-lib/--end-lib in the final link can muck with
> that. However, the list of files the linker selected in the right order is
> emitted in the argument given to thinlto-index-only (thinlto.objects in
> your case below). You can pass that to the native link via the
"@" option:
>   clang++ -o thinlto @thinlto.objects
> however you need to deal with the fact that this file contains the
> original bitcode names (not the names you gave it in your backend step like
> main-native.o, etc)
> There are 2 options for correcting the names:
> 1) Manually rename in thinlto.objects
> 2) Use the thinlto_prefix_replace=oldprefix;newprefix plugin option, to
> replace the old path prefix of the input bitcode files with a new path
> prefix. In your case the old prefix is "", so you could do
something like
> "-Wl,-plugin-opt,thinlto_prefix_replace=:native/". This should do
2 things:
> 1) the generated .thinlto.bc index files and the .imports files will be put
> under a "native/" subdirectory; 2) the paths in thinlto.objects
should also
> have the "native/" prefix. If you use that prefix in your LTO
backend clang
> invocations (e.g. -o native/main.o instead of main-native.o), then
> thinlto.objects can just be passed directly to the final link via
"@"
> without any modification.
>
> Note that if your thin link included any already native files/libraries,
> those still need to be passed as the thinlto.objects only includes those
> that were originally bitcode.
>
> Teresa
>
>
>> Tanoy
>>
>> On Tue, Jun 18, 2019 at 10:37 AM Teresa Johnson <tejohnson at
google.com>
>> wrote:
>>
>>> Hi Tanoy,
>>>
>>> You can't use distributed ThinLTO with archives (thin or not),
at least
>>> not today. The reason is that we need to be able to identify
specific
>>> bitcode object files to import from in the backends, and that logic
does
>>> not know how to deal with objects within archives. We do
distributed
>>> ThinLTO in our builds but don't use .a files, rather, we use
>>> --start-lib/--end-lib around the files that would be in the same
archive
>>> when performing the thin link. I.e. if you change your thin link to
be:
>>>
>>> clang++ -flto=thin -o index -O3
>>>  -Wl,-plugin-opt,thinlto-index-only=thinlto.objects
>>> -Wl,-plugin-opt,thinlto-emit-imports-files main.o --start-lib
lib/lib.o
>>> src/lib.o --end-lib
>>>
>>> things should work.
>>>
>>> Note you also need to do the ThinLTO backend compile for each of
the
>>> archive constituents anyway, e.g. something like:
>>>   clang++ -c -x ir lib/lib.o -O3 -flto=thin -o lib/lib-native.o
>>> -fthinlto-index=lib/lib.o.thinlto.bc
>>> etc
>>>
>>> HTH,
>>> Teresa
>>>
>>> On Mon, Jun 17, 2019 at 2:46 PM Tanoy Sinha via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> I'm trying to run distributed ThinLTO without thin
archives.
>>>> When I do, I get an error in the optimizer when clang tries to
open a
>>>> nonexistent file:
>>>>
>>>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c main.cpp -o
main.o
>>>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c lib/lib.cpp -o
lib/lib.o
>>>> clang++ -flto=thin -Xclang -fno-lto-unit -O3 -c src/lib.cpp -o
src/lib.o
>>>> llvm-ar -format gnu qcs lib.a lib/lib.o src/lib.o
>>>> clang++ -flto=thin -o index -O3
>>>>  -Wl,-plugin-opt,thinlto-index-only=thinlto.objects
>>>> -Wl,-plugin-opt,thinlto-emit-imports-files main.o lib.a
>>>> clang++ -c -x ir main.o -O3 -flto=thin -o main-native.o
>>>> -fthinlto-index=main.o.thinlto.bc
>>>> Error loading imported file 'lib.a.llvm.2596.lib.cpp':
No such file or
>>>> directory
>>>>
>>>> In this case, gold has registered the modules within my archive
with
>>>> ThinLTO.
>>>> The string "lib.a.llvm.2596.lib.cpp" is generated
with the archive in
>>>> question, plus an offset indicating where in the archive the
particular
>>>> object file is.
>>>> Unfortunately, when the optimizer tries to include the proper
modules,
>>>> it's naively looking for a bitcode file with the name of
the string
>>>> provided, but there's obviously no
"lib.a.llvm.2596.lib.cpp" for it to open.
>>>>
>>>> Has anyone else tried to get clang to understand distributed
ThinLTO
>>>> when using non thin archives?
>>>> Is there some way to get clang to understand these out of the
box?
>>>>
>>>> I'm actually a little confused about the ".cpp"
in
>>>> "lib.a.llvm.2596.lib.cpp".
>>>> Seems like it should be a ".o"?
>>>>
>>>> It didn't seem like there was anything out of the box that
supported
>>>> this.
>>>> I was looking at having clang actually read in the archive file
and
>>>> register the correct bitcode module.
>>>> I wanted to run it by the list to get some second opinions
before I
>>>> started that.
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
>>>
>>> --
>>> Teresa Johnson |  Software Engineer |  tejohnson at google.com |
>>>
>>
>
> --
> Teresa Johnson |  Software Engineer |  tejohnson at google.com |
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190618/941ec36a/attachment.html>