llvm dev - Apr 2021 - Supporting Regular and Thin LTO with a Single LTO Bitcode Format

We're using regular LTO for our Clang toolchain build because we don't
mind
spending more resources to squeeze out as much performance as possible.
However, when looking into our build performance, I've noticed that we only
spent about 1/3 of the total build time in building distribution
components, the rest is spent on building unit tests and tools that are
only used by lit tests. For the latter, we don't care about the
performance, so it'd be nice to avoid doing regular LTO to speed up the
build.

The idea I had would be to use a single LTO bitcode format for all
translation units, and then decide only at link time whether to use regular
LTO for distribution components or ThinLTO for everything else.

After doing some research, I found the "Supporting Regular and Thin LTO
with a Single LTO Bitcode Format
<https://www.snsystems.com/technology/tech-blog/supporting-regular-and-thin-lto-with-a-single-lto-bitcode-format>"
talk presented by Matthew Voss at LLVM Developers’ Meeting 2019 which does
exactly what I described, but it seems like this was only implemented
downstream.

Has there been any progress on upstreaming the implementation? Is there any
way to do what I described using the in-tree LTO implementation?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210413/3a39018d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3996 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210413/3a39018d/attachment.bin>

+Matthew and Teresa for any context they might have

High level sounds like a reasonable thing to me, for what it's worth.

On Tue, Apr 13, 2021 at 2:19 PM Petr Hosek via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> We're using regular LTO for our Clang toolchain build because we
don't mind spending more resources to squeeze out as much performance as
possible. However, when looking into our build performance, I've noticed
that we only spent about 1/3 of the total build time in building distribution
components, the rest is spent on building unit tests and tools that are only
used by lit tests. For the latter, we don't care about the performance, so
it'd be nice to avoid doing regular LTO to speed up the build.
>
> The idea I had would be to use a single LTO bitcode format for all
translation units, and then decide only at link time whether to use regular LTO
for distribution components or ThinLTO for everything else.
>
> After doing some research, I found the "Supporting Regular and Thin
LTO with a Single LTO Bitcode Format" talk presented by Matthew Voss at
LLVM Developers’ Meeting 2019 which does exactly what I described, but it seems
like this was only implemented downstream.
>
> Has there been any progress on upstreaming the implementation? Is there any
way to do what I described using the in-tree LTO implementation?
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

We also have this problem, and we are considering using fembed-bitcode:
https://crbug.com/1196260

This way, we wouldn't do any kind of LTO for test binaries, we'd just
link
native objects and discard the bitcode sections. ThinLTO has the advantage
that it may do less code generation of non-prevailing inline functions, but
it is also more complex overall. It is too early for me to say which of
these approaches is best, but we'll keep in touch.

On Tue, Apr 13, 2021 at 2:19 PM Petr Hosek via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> We're using regular LTO for our Clang toolchain build because we
don't
> mind spending more resources to squeeze out as much performance as
> possible. However, when looking into our build performance, I've
noticed
> that we only spent about 1/3 of the total build time in building
> distribution components, the rest is spent on building unit tests and tools
> that are only used by lit tests. For the latter, we don't care about
the
> performance, so it'd be nice to avoid doing regular LTO to speed up the
> build.
>
> The idea I had would be to use a single LTO bitcode format for all
> translation units, and then decide only at link time whether to use regular
> LTO for distribution components or ThinLTO for everything else.
>
> After doing some research, I found the "Supporting Regular and Thin
LTO
> with a Single LTO Bitcode Format
>
<https://www.snsystems.com/technology/tech-blog/supporting-regular-and-thin-lto-with-a-single-lto-bitcode-format>"
> talk presented by Matthew Voss at LLVM Developers’ Meeting 2019 which does
> exactly what I described, but it seems like this was only implemented
> downstream.
>
> Has there been any progress on upstreaming the implementation? Is there
> any way to do what I described using the in-tree LTO implementation?
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210414/73e6a2ad/attachment.html>