llvm dev - Oct 2020 - [RFC] A vision for building the runtimes

> On Oct 23, 2020, at 02:25, Dominik Montada <dominik.montada at
hightec-rt.com> wrote:
> 
> Hi Louis,
> 
> big +1 from me. I recently set up the runtimes for our downstream toolchain
and to be honest it was quite a pain to get everything working. Due to the
complexity we ended up creating our own CMake cache to make the configuration
easier. We also tried to use the existing toolchain build. But since it tries to
build the runtimes immediately after it built the compiler we cannot use it, as
we have to build our C library first. So having a "unified standalone"
approach sounds just like something that would make this use-case a lot easier.
Glad to see support!
> 
> Do you already have an idea of how a multilib build would look like with
your proposed setup?
Can you define what you mean by a multilib build? Do you mean building for
several architectures at once? Do you produce the libs for each architecture in
different directories?

If you're talking about what I'm thinking of, I believe the simplest and
most CMake-friendly way of doing it would be to have multiple CMake invocations.
We can hide those behind a "driver" build like what the Runtimes build
does, for example, but the libc++/libc++abi/libunwind builds themselves
wouldn't be aware they're being built for multiple archs.
> Also, are you planning of including compiler-rt in this as well or is this
strictly meant for libc++, libc++abi and libunwind?
For now, libc++, libc++abi and libunwind. Those are different because it makes
sense to ship them alongside the compiler, or not. For example, at Apple we ship
those as system libraries, not alongside our compiler. The compiler-rt build is
also significantly more complicated.

Louis
> 
> Thanks for doing this!
> 
> Dominik
> 
> Am 23.10.20 um 00:31 schrieb Louis Dionne via llvm-dev:
>> Hi folks,
>> 
>> The topic of how to build the runtimes has been brought up several
times in the past, sometimes by me, sometimes by others. All in all, building
the runtimes is fairly complex: there's several ways of doing it, and they
don't always work. This complexity leads to several problems, the first one
being that I, as a maintainer, can't currently guarantee a good experience
for people trying to build the library. What follows is a concrete proposal to
make things better within a reasonable time frame.
>> 
>> 
>> The current state of things
>> -------------------------------------
>> 
>> The runtimes (libc++, libc++abi and libunwind) currently support the
following ways of being built:
>> 
>> - Monorepo
>>     This is the "easy" and most common way to build the
runtimes. It builds the runtimes as subprojects of LLVM (with
LLVM_ENABLE_PROJECTS), with the same compiler that's used to build LLVM.
>> 
>>     However, this is mostly wrong, since it encourages users to build
libc++ using the system compiler, not the just-built compiler. Since a recent
compiler is necessary for correctness (e.g. the RTTI for fundamental types
generated in libc++abi), this is a real issue.
>> 
>>     This also requires the whole LLVM CMake machinery to work on the
platform where one wants to build the runtimes. This doesn't really work on
most embedded platforms, so they just can't use the monorepo build.
>> 
>>     This also suffers from issues like the fact that LLVM sets several
global variables/flags that subprojects inherit. While it may make sense for
some subprojects to use these defaults (e.g. the default C++ Standard), it is
actively harmful for the runtimes, which need to have a tight control over these
things.
>> 
>> - Standalone
>>     Each runtime project also supports a Standalone build. This is a
build where the root CMakeLists.txt is the one from the project itself. This is
nice because it's lightweight, and it doesn't require all the LLVM CMake
setup to work, which solves problems for embedded platforms.
>> 
>>     Before the monorepo era, this type of build also made sense cause
you could build one runtime without checking out the other ones, however that is
not true anymore (the runtimes all share code that requires them to be
co-located in the monorepo even if you're just building one of them).
>> 
>>     This type of build has the significant downside that we need to tie
together the various runtime projects using CMake variables. For example, we
have to tell libc++abi where to find the libc++ headers, and we have to tell
libc++ where to find the library for libc++abi. This leads to a plethora of
CMake options that are brittle and add a lot of complexity
(LIBCXX_CXX_ABI_INTREE, LIBCXXABI_LIBCXX_INCLUDES, etc.).
>> 
>> - The llvm/runtimes build
>>     I like to call this the Toolchain build instead, cause that's
really what it does. It builds the runtimes using the just-built toolchain, and
with the goal of including those runtimes in the toolchain. It's more of a
driver for the individual builds than a build configuration itself. It's
currently built on top of the Standalone builds -- it builds the toolchain and
then builds the various runtimes individually, stringing them together as
required.
>> 
>> 
>> Proposal
>> --------------
>> 
>> My goal with this proposal is to achieve:
>> 1. Decoupling from the top-level LLVM CMake setup (which doesn't
work, see above)
>> 2. A simple build that works everywhere, including embedded platforms
>> 3. Remove the need to manually tie together the various runtimes (as in
the Standalone builds)
>> 
>> My proposal is basically to have a "Unified Standalone" build
for all the runtimes. It would look similar to a Monorepo build in essence (i.e.
you'd have a single CMake invocation where you would specify the flags for
all runtime projects at once), but it wouldn't be using the top-level LLVM
CMake setup [1]. Specifically:
>> 
>> 1. Add a `runtimes/CMakeLists.txt` file that includes the runtimes
subprojects that are requested through -DLLVM_ENABLE_PROJECTS (open to
bikeshed), and sets up minimal stuff like the `llvm-lit` wrapper and Python, but
none of the harmful stuff that's done by the top-level LLVM CMake.
>> 2. Deprecate the old individual Standalone builds for this new
"Unified Standalone build".
>> 3. Users migrate to the new Unified Standalone build. Users include the
current "Runtimes" build, some places in compiler-rt, and various
organizations.
>> 4. Remove support for the old individual Standalone builds.
>> 
>> As a second step, we should also:
>> 5. Deprecate the current Monorepo build in favor of either the Unified
Standalone build (for those who just wish to build the runtimes), or the current
Runtimes (aka Toolchain) build, for those who wish to build a toolchain.
>> 6. Let users migrate to either
>> 7. Remove support for the Monorepo build (e.g. make it an error to try
and build one of the runtimes through the current Monorepo build).
>> 
>> At the end of this transition, we'd hence have a single way to
build all the runtimes, and a "driver" to build them with the
just-built toolchain.
>> 
>> Moving towards a single CMake invocation for the Standalone builds is
the key element of this proposal that will make everything simpler, and remove
the need to setup a bunch of things manually. It will also make it easier to
start sharing more code across the various runtimes [2].
>> 
>> I have already written the CMakeLists.txt for the "Unified
Standalone" build, and I've been using it to build libc++ and libc++abi
at Apple. It is incredibly simple, and it works well so far.
>> 
>> I'm looking forward to your comments,
>> Louis
>> 
>> [1] If you're wondering what that would look like:
>> 
>>     $ mkdir <monorepo-root>/build
>>     $ cd <monorepo-root>/build
>>     $ cmake ../runtimes
-DLLVM_ENABLE_PROJECTS="libcxx;libcxxabi;libunwind" \
>>         -C <path-to-your-cache-if-desired> \
>>         -DLIBCXX_ENABLE_XXX \
>>         -DLIBCXXABI_ENABLE_XXX \
>>         <other options>
>>     $ ninja install-cxx install-cxxabi
>> 
>> [2] If you're wondering, I'm not proposing to remove being able
to build libc++ against other ABI libraries, or any such thing. The Unified
Standalone build would retain the same amount of flexibility as today.
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> -- 
> ----------------------------------------------------------------------
> Dominik Montada                   Email: dominik.montada at hightec-rt.com
> HighTec EDV-Systeme GmbH          Phone: +49 681 92613 19
> Europaallee 19                    Fax:   +49-681-92613-26
> D-66113 Saarbrücken               WWW: http://www.hightec-rt.com
<http://www.hightec-rt.com/>
> 
> Managing Director: Vera Strothmann
> Register Court: Saarbrücken, HRB 10445, VAT ID: DE 138344222
> 
> This e-mail may contain confidential and/or privileged information. If
> you are not the intended recipient please notify the sender immediately
> and destroy this e-mail. Any unauthorised copying, disclosure or
> distribution of the material in this e-mail is strictly forbidden.
> ---
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201023/31798ba2/attachment-0001.html>

Am 23.10.20 um 16:30 schrieb Louis Dionne:
>
>
>> On Oct 23, 2020, at 02:25, Dominik Montada 
>> <dominik.montada at hightec-rt.com 
>> <mailto:dominik.montada at hightec-rt.com>> wrote:
>>
>> Hi Louis,
>>
>> big +1 from me. I recently set up the runtimes for our downstream 
>> toolchain and to be honest it was quite a pain to get everything 
>> working. Due to the complexity we ended up creating our own CMake 
>> cache to make the configuration easier. We also tried to use the 
>> existing toolchain build. But since it tries to build the runtimes 
>> immediately after it built the compiler we cannot use it, as we have 
>> to build our C library first. So having a "unified
standalone"
>> approach sounds just like something that would make this use-case a 
>> lot easier.
>
> Glad to see support!
>
>>
>> Do you already have an idea of how a multilib build would look like 
>> with your proposed setup?
>
> Can you define what you mean by a multilib build? Do you mean building 
> for several architectures at once? Do you produce the libs for each 
> architecture in different directories?
Yes, but also more than that: our target has different architectures, so 
we build each library for each architecture. But on top of that, we also 
build the libraries with different configurations, e.g. with or without 
exceptions. The Fuchsia toolchain is doing something similar IIRC. We 
produce these libraries in different directories per architecture and 
per configuration, so for example:

target/arch1/except
target/arch1/noexcept
target/arch2/except
target/arch2/noexcept
>
> If you're talking about what I'm thinking of, I believe the
simplest
> and most CMake-friendly way of doing it would be to have multiple 
> CMake invocations. We can hide those behind a "driver" build like
what
> the Runtimes build does, for example, but the 
> libc++/libc++abi/libunwind builds themselves wouldn't be aware
they're
> being built for multiple archs.
With the use-case described above, we simply do one CMake invocation for 
each library configuration.
>
>> Also, are you planning of including compiler-rt in this as well or is 
>> this strictly meant for libc++, libc++abi and libunwind?
>
> For now, libc++, libc++abi and libunwind. Those are different because 
> it makes sense to ship them alongside the compiler, or not. For 
> example, at Apple we ship those as system libraries, not alongside our 
> compiler. The compiler-rt build is also significantly more complicated.
>
> Louis
>
>>
>> Thanks for doing this!
>>
>> Dominik
>>
>> Am 23.10.20 um 00:31 schrieb Louis Dionne via llvm-dev:
>>> Hi folks,
>>>
>>> The topic of how to build the runtimes has been brought up several 
>>> times in the past, sometimes by me, sometimes by others. All in
all,
>>> building the runtimes is fairly complex: there's several ways
of
>>> doing it, and they don't always work. This complexity leads to 
>>> several problems, the first one being that I, as a maintainer,
can't
>>> currently guarantee a good experience for people trying to build
the
>>> library. What follows is a concrete proposal to make things better 
>>> within a reasonable time frame.
>>>
>>>
>>> The current state of things
>>> -------------------------------------
>>>
>>> The runtimes (libc++, libc++abi and libunwind) currently support
the
>>> following ways of being built:
>>>
>>> - Monorepo
>>>     This is the "easy" and most common way to build the
runtimes. It
>>> builds the runtimes as subprojects of LLVM (with 
>>> LLVM_ENABLE_PROJECTS), with the same compiler that's used to
build LLVM.
>>>
>>>     However, this is mostly wrong, since it encourages users to 
>>> build libc++ using the system compiler, not the just-built
compiler.
>>> Since a recent compiler is necessary for correctness (e.g. the RTTI
>>> for fundamental types generated in libc++abi), this is a real
issue.
>>>
>>>     This also requires the whole LLVM CMake machinery to work on
the
>>> platform where one wants to build the runtimes. This doesn't
really
>>> work on most embedded platforms, so they just can't use the
monorepo
>>> build.
>>>
>>>     This also suffers from issues like the fact that LLVM sets 
>>> several global variables/flags that subprojects inherit. While it 
>>> may make sense for some subprojects to use these defaults (e.g. the
>>> default C++ Standard), it is actively harmful for the runtimes, 
>>> which need to have a tight control over these things.
>>>
>>> - Standalone
>>>     Each runtime project also supports a Standalone build. This is
a
>>> build where the root CMakeLists.txt is the one from the project 
>>> itself. This is nice because it's lightweight, and it
doesn't
>>> require all the LLVM CMake setup to work, which solves problems for
>>> embedded platforms.
>>>
>>>     Before the monorepo era, this type of build also made sense 
>>> cause you could build one runtime without checking out the other 
>>> ones, however that is not true anymore (the runtimes all share code
>>> that requires them to be co-located in the monorepo even if
you're
>>> just building one of them).
>>>
>>>     This type of build has the significant downside that we need to
>>> tie together the various runtime projects using CMake variables.
For
>>> example, we have to tell libc++abi where to find the libc++
headers,
>>> and we have to tell libc++ where to find the library for libc++abi.
>>> This leads to a plethora of CMake options that are brittle and add
a
>>> lot of complexity (LIBCXX_CXX_ABI_INTREE,
LIBCXXABI_LIBCXX_INCLUDES,
>>> etc.).
>>>
>>> - The llvm/runtimes build
>>>     I like to call this the Toolchain build instead, cause
that's
>>> really what it does. It builds the runtimes using the just-built 
>>> toolchain, and with the goal of including those runtimes in the 
>>> toolchain. It's more of a driver for the individual builds than
a
>>> build configuration itself. It's currently built on top of the 
>>> Standalone builds -- it builds the toolchain and then builds the 
>>> various runtimes individually, stringing them together as required.
>>>
>>>
>>> Proposal
>>> --------------
>>>
>>> My goal with this proposal is to achieve:
>>> 1. Decoupling from the top-level LLVM CMake setup (which
doesn't
>>> work, see above)
>>> 2. A simple build that works everywhere, including embedded
platforms
>>> 3. Remove the need to manually tie together the various runtimes
(as
>>> in the Standalone builds)
>>>
>>> My proposal is basically to have a "Unified Standalone"
build for
>>> all the runtimes. It would look similar to a Monorepo build in 
>>> essence (i.e. you'd have a single CMake invocation where you
would
>>> specify the flags for all runtime projects at once), but it
wouldn't
>>> be using the top-level LLVM CMake setup [1]. Specifically:
>>>
>>> 1. Add a `runtimes/CMakeLists.txt` file that includes the runtimes 
>>> subprojects that are requested through -DLLVM_ENABLE_PROJECTS (open
>>> to bikeshed), and sets up minimal stuff like the `llvm-lit` wrapper
>>> and Python, but none of the harmful stuff that's done by the 
>>> top-level LLVM CMake.
>>> 2. Deprecate the old individual Standalone builds for this new 
>>> "Unified Standalone build".
>>> 3. Users migrate to the new Unified Standalone build. Users include
>>> the current "Runtimes" build, some places in compiler-rt,
and
>>> various organizations.
>>> 4. Remove support for the old individual Standalone builds.
>>>
>>> As a second step, we should also:
>>> 5. Deprecate the current Monorepo build in favor of either the 
>>> Unified Standalone build (for those who just wish to build the 
>>> runtimes), or the current Runtimes (aka Toolchain) build, for those
>>> who wish to build a toolchain.
>>> 6. Let users migrate to either
>>> 7. Remove support for the Monorepo build (e.g. make it an error to 
>>> try and build one of the runtimes through the current Monorepo
build).
>>>
>>> At the end of this transition, we'd hence have a single way to
build
>>> all the runtimes, and a "driver" to build them with the
just-built
>>> toolchain.
>>>
>>> Moving towards a single CMake invocation for the Standalone builds 
>>> is the key element of this proposal that will make everything 
>>> simpler, and remove the need to setup a bunch of things manually.
It
>>> will also make it easier to start sharing more code across the 
>>> various runtimes [2].
>>>
>>> I have already written the CMakeLists.txt for the "Unified 
>>> Standalone" build, and I've been using it to build libc++
and
>>> libc++abi at Apple. It is incredibly simple, and it works well so
far.
>>>
>>> I'm looking forward to your comments,
>>> Louis
>>>
>>> [1] If you're wondering what that would look like:
>>>
>>>     $ mkdir <monorepo-root>/build
>>>     $ cd <monorepo-root>/build
>>>     $ cmake ../runtimes 
>>> -DLLVM_ENABLE_PROJECTS="libcxx;libcxxabi;libunwind" \
>>>         -C <path-to-your-cache-if-desired> \
>>>         -DLIBCXX_ENABLE_XXX \
>>>         -DLIBCXXABI_ENABLE_XXX \
>>>         <other options>
>>>     $ ninja install-cxx install-cxxabi
>>>
>>> [2] If you're wondering, I'm not proposing to remove being
able to
>>> build libc++ against other ABI libraries, or any such thing. The 
>>> Unified Standalone build would retain the same amount of
flexibility
>>> as today.
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>> --
>> ----------------------------------------------------------------------
>> Dominik Montada Email: dominik.montada at hightec-rt.com 
>> <mailto:dominik.montada at hightec-rt.com>
>> HighTec EDV-Systeme GmbH          Phone: +49 681 92613 19
>> Europaallee 19                    Fax:   +49-681-92613-26
>> D-66113 Saarbrücken               WWW:http://www.hightec-rt.com 
>> <http://www.hightec-rt.com/>
>>
>> Managing Director: Vera Strothmann
>> Register Court: Saarbrücken, HRB 10445, VAT ID: DE 138344222
>>
>> This e-mail may contain confidential and/or privileged information. If
>> you are not the intended recipient please notify the sender immediately
>> and destroy this e-mail. Any unauthorised copying, disclosure or
>> distribution of the material in this e-mail is strictly forbidden.
>> ---
>
-- 
----------------------------------------------------------------------
Dominik Montada                   Email: dominik.montada at hightec-rt.com
HighTec EDV-Systeme GmbH          Phone: +49 681 92613 19
Europaallee 19                    Fax:   +49-681-92613-26
D-66113 Saarbrücken               WWW: http://www.hightec-rt.com

Managing Director: Vera Strothmann
Register Court: Saarbrücken, HRB 10445, VAT ID: DE 138344222

This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient please notify the sender immediately
and destroy this e-mail. Any unauthorised copying, disclosure or
distribution of the material in this e-mail is strictly forbidden.
---
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201023/941a5882/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 6822 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201023/941a5882/attachment-0001.bin>

> On Oct 23, 2020, at 12:36, Dominik Montada <dominik.montada at
hightec-rt.com> wrote:
> 
> 
> 
> Am 23.10.20 um 16:30 schrieb Louis Dionne:
>> 
>> 
>>> On Oct 23, 2020, at 02:25, Dominik Montada <dominik.montada at
hightec-rt.com <mailto:dominik.montada at hightec-rt.com>> wrote:
>>> 
>>> Hi Louis,
>>> 
>>> big +1 from me. I recently set up the runtimes for our downstream
toolchain and to be honest it was quite a pain to get everything working. Due to
the complexity we ended up creating our own CMake cache to make the
configuration easier. We also tried to use the existing toolchain build. But
since it tries to build the runtimes immediately after it built the compiler we
cannot use it, as we have to build our C library first. So having a
"unified standalone" approach sounds just like something that would
make this use-case a lot easier.
>> 
>> Glad to see support!
>> 
>>> 
>>> Do you already have an idea of how a multilib build would look like
with your proposed setup?
>> 
>> Can you define what you mean by a multilib build? Do you mean building
for several architectures at once? Do you produce the libs for each architecture
in different directories?
> Yes, but also more than that: our target has different architectures, so we
build each library for each architecture. But on top of that, we also build the
libraries with different configurations, e.g. with or without exceptions. The
Fuchsia toolchain is doing something similar IIRC. We produce these libraries in
different directories per architecture and per configuration, so for example:
> 
> target/arch1/except
> target/arch1/noexcept
> target/arch2/except
> target/arch2/noexcept
> 
>> 
>> If you're talking about what I'm thinking of, I believe the
simplest and most CMake-friendly way of doing it would be to have multiple CMake
invocations. We can hide those behind a "driver" build like what the
Runtimes build does, for example, but the libc++/libc++abi/libunwind builds
themselves wouldn't be aware they're being built for multiple archs.
> With the use-case described above, we simply do one CMake invocation for
each library configuration.
Yes, I think one CMake invocation per configuration is the way to go. Unlike
other build systems, it seems like CMake wasn't really designed with
multi-configuration builds in mind.

Now, one can wrap these multiple CMake invocations in a single one by using a
"driving" CMakeLists.txt and calls to ExternalProject_add if so
desired, but it's really another layer on top.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201023/7d7fe683/attachment.html>