via llvm-dev
2021-Apr-14 19:31 UTC
[llvm-dev] Consider making build directories of the buildbots available
Ah, apologies, I think I wasn't clear. I am talking about making object files available in this manner so that people can download them and compile just their changes without having to compile all of LLVM, thus reducing the barrier to entry. As far as I am aware, the releases on Github do not contain object files. Nabeel Omer From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Neil Nelson via llvm-dev Sent: 14 April 2021 20:05 To: llvm-dev at lists.llvm.org Subject: Re: [llvm-dev] Consider making build directories of the buildbots available Perhaps these pages may help. https://github.com/llvm/llvm-project/releases https://github.com/llvm/llvm-project/releases/tag/llvmorg-12.0.0-rc5 Neil Nelson On 4/14/21 12:53 PM, via llvm-dev wrote: Hi LLVM devs, As you are already aware, performing a clean build of LLVM requires considerable computing resources. This presents a barrier to entry for people who do not have access to large computers. Since the buildbots already regularly compile the LLVM codebase, making tarballs of their build directories available on a public facing server will dramatically reduce the barrier to entry. Is this something that the community is willing to consider? Thanks, Nabeel Omer _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210414/43e86172/attachment.html>
Alexandre Ganea via llvm-dev
2021-Apr-14 20:53 UTC
[llvm-dev] Consider making build directories of the buildbots available
That's an interesting idea. There are several issues to consider:
1. The build needs to be deterministic, if we want to share .OBJ files among
LLVM developers. In essence, that means fully implementing [1]. I'm not sure
how much of the LLVM codebase supports that.
Just the toolchain part is challenging to implement. We would need to ensure all
users are using precisely the same compiler, the same linker, same cmake, same
platform SDKs, etc. in order to expect good cache hits.
2. There's the security consideration. If anyone is to pull on the cached
.OBJ files, you need to "trust" these .OBJ files in the first place.
That means maybe restraining the list of cache "publishers" to the
LLVM github group, and signing the .OBJs with a private key or something along
those lines.
3. There's a file size consideration. My build folder for only { llvm,
clang, lld } is about 40 GB. When using the ThinLTO cache, that goes over 100
GB. Still, build artefacts compress quite well (3:1 at least), and you'd
probably pay the network price only once, then cache hits would be incremental.
I think the same idea could apply for distributing the compilation. It'd be
interesting to have a public LLVM distributed compilation service. But how can
we trust it? Even if we only compile on "trusted" machines,
there's still a risk of attack. That's probably why these caching/remote
compilation systems are only used inside an organization, which can guarantee
trust (somehow). Since you're at Sony, perhaps you have access to a internal
SN-DBS pool? [2]
[1] https://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html
[2] https://www.snsystems.com/tech-blog/2014/01/06/building-with-the-network/
De : llvm-dev <llvm-dev-bounces at lists.llvm.org> De la part de via
llvm-dev
Envoyé : April 14, 2021 3:31 PM
À : llvm-dev at lists.llvm.org
Objet : Re: [llvm-dev] Consider making build directories of the buildbots
available
Ah, apologies, I think I wasn't clear. I am talking about making object
files available in this manner so that people can download them and compile just
their changes without having to compile all of LLVM, thus reducing the barrier
to entry. As far as I am aware, the releases on Github do not contain object
files.
Nabeel Omer
From: llvm-dev <llvm-dev-bounces at lists.llvm.org<mailto:llvm-dev-bounces
at lists.llvm.org>> On Behalf Of Neil Nelson via llvm-dev
Sent: 14 April 2021 20:05
To: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
Subject: Re: [llvm-dev] Consider making build directories of the buildbots
available
Perhaps these pages may help.
https://github.com/llvm/llvm-project/releases
https://github.com/llvm/llvm-project/releases/tag/llvmorg-12.0.0-rc5
Neil Nelson
On 4/14/21 12:53 PM, via llvm-dev wrote:
Hi LLVM devs,
As you are already aware, performing a clean build of LLVM requires considerable
computing resources. This presents a barrier to entry for people who do not have
access to large computers. Since the buildbots already regularly compile the
LLVM codebase, making tarballs of their build directories available on a public
facing server will dramatically reduce the barrier to entry. Is this something
that the community is willing to consider?
Thanks,
Nabeel Omer
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210414/17d7cee8/attachment-0001.html>