llvm dev - Sep 2020 - LTO'd / PGO'd Pre-built Release Binary

A compiler built with LTO / PGO can significantly improve build times. Locally I
see ~10% reduction in compilation times of Debug or Release versions of
clang/llvm using an LTO'd clang compiler. This is not really news of course
(e.g. recently also reconfirmed in
https://linuxplumbersconf.org/event/7/contributions/802/), but that made me
curious if we can and/or should enable LTO and/or PGO to create our prebuilt
binaries?

The disadvantage for our release people will be obvious: that will take a lot
more time and resources. But would it be worth to make this investment?

Cheers,
Sjoerd.


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200907/40da8714/attachment.html>

Sjoerd,

I emailed llvm-dev a while ago suggesting that we at least turn on LTO. I
really didn't get much feedback on that unfortunately, I think most people
get clang via their distro or compile it themselves.

I did look into what's needed: the release script need to build the first
and second stage as it does now but for the final stage we would need to
adapt it to send in a compatible linker (gold or lld) and fix ranlib/ar
invocations.

I think the most complicated part is that the release testers are the one
responsible for building the final binaries and there is a lot of different
systems that might require some tweaking.

I have had a dream of some CI service being able to build PGO/LTO optimized
binaries (at least for Linux, windows and Mac) for each release - but I
think we would need a cloud vendor sponsoring that CPU time (as a reference
our toolchain build takes ~5 hrs for something similar).

Thanks,
Tobias

On Mon, Sep 7, 2020, 14:50 Sjoerd Meijer via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> A compiler built with LTO / PGO can significantly improve build times.
> Locally I see ~10% reduction in compilation times of Debug or Release
> versions of clang/llvm using an LTO'd clang compiler. This is not
really
> news of course (e.g. recently also reconfirmed in
> https://linuxplumbersconf.org/event/7/contributions/802/), but that made
> me curious if we can and/or should enable LTO and/or PGO to create our
> prebuilt binaries?
>
> The disadvantage for our release people will be obvious: that will take a
> lot more time and resources. But would it be worth to make this investment?
>
> Cheers,
> Sjoerd.
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200907/7de6dd82/attachment-0001.html>

> I emailed llvm-dev a while ago suggesting that we at least turn on LTO.
Ah, sorry, missed that. Otherwise I would have replied to/revived that thread.
> I really didn't get much feedback on that unfortunately, I think most
people get clang via their distro or compile it themselves.
Well, that's what I was wondering, wasn't sure if it is worth
doing....but I guess we upload these binaries for a reason and guess these
binaries are picked up by some people? Even if not, it would be good to set an
example (and for testing)?
> I think the most complicated part is that the release testers are the one
responsible for building the final binaries and there is a lot of different
systems that might require some tweaking.
Oh yeah, definitely. I see that it's easy to suggest this, and that it's
a lot of work for the people who have to do this.

Cheers.
________________________________
From: Tobias Hieta <tobias at plexapp.com>
Sent: 07 September 2020 14:56
To: Sjoerd Meijer <Sjoerd.Meijer at arm.com>
Cc: llvm-dev <llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] LTO'd / PGO'd Pre-built Release Binary

Sjoerd,

I emailed llvm-dev a while ago suggesting that we at least turn on LTO. I really
didn't get much feedback on that unfortunately, I think most people get
clang via their distro or compile it themselves.

I did look into what's needed: the release script need to build the first
and second stage as it does now but for the final stage we would need to adapt
it to send in a compatible linker (gold or lld) and fix ranlib/ar invocations.

I think the most complicated part is that the release testers are the one
responsible for building the final binaries and there is a lot of different
systems that might require some tweaking.

I have had a dream of some CI service being able to build PGO/LTO optimized
binaries (at least for Linux, windows and Mac) for each release - but I think we
would need a cloud vendor sponsoring that CPU time (as a reference our toolchain
build takes ~5 hrs for something similar).

Thanks,
Tobias

On Mon, Sep 7, 2020, 14:50 Sjoerd Meijer via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
A compiler built with LTO / PGO can significantly improve build times. Locally I
see ~10% reduction in compilation times of Debug or Release versions of
clang/llvm using an LTO'd clang compiler. This is not really news of course
(e.g. recently also reconfirmed in
https://linuxplumbersconf.org/event/7/contributions/802/), but that made me
curious if we can and/or should enable LTO and/or PGO to create our prebuilt
binaries?

The disadvantage for our release people will be obvious: that will take a lot
more time and resources. But would it be worth to make this investment?

Cheers,
Sjoerd.


_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200907/d0849ff0/attachment.html>