llvm dev - Oct 2021 - Upcoming change with how libc++, libc++abi and libunwind are being built

Hi Louis,

This sounds like a huge simplification, thank you!

With this change, will we still be able to use LLVM_ENABLE_PROJECTS to 
build libc++, libc++abi and libunwind? IIUC, you are disabling this 
option entirely, right? Will CMake warn us about the 
incorrect/deprecated usage? Should the LLVM documentation be updated [1]?

Thank you,
-Andrzej

[1] https://llvm.org/docs/CMake.html

On 11/10/2021 17:17, Louis Dionne via llvm-dev wrote:
> Hi folks,
> 
> This message is both a heads up and a request for comments on an 
> upcoming change we want to make to the way we build 
> libc++/libc++abi/libunwind.
> 
> For a long time now, those three projects have supported various ways to 
> be built:
> 
> 1. The Project build, where one would use <root>/llvm as the root of
the
> CMake invocation and pass LLVM_ENABLE_PROJECTS.
> 2. The Standalone build, where one would perform one CMake invocation 
> per project being built, and fill in various CMake variables to tell 
> each project where to find bits of the other projects.
> 3. The "Runtimes" or "Bootstrapping" build, where one
would use
> <root>/llvm as the root of the CMake invocation and pass 
> LLVM_ENABLE_RUNTIMES. This would result in Clang being built, and then 
> the runtimes being built with that fresh Clang.
> 4. The "Unified Standalone" build, where one would use 
> <root>/libcxx/utils/ci/runtimes as the root of the CMake invocation
and
> pass LLVM_ENABLE_PROJECTS. This was introduced to work around the fact 
> that the Project build doesn't work on several embedded platforms, and
I
> think it is only used at Apple.
> 
> Through the years, this has gotten rather confusing and difficult to 
> support. We currently support all of those on a best-effort basis and we 
> do have build bots checking each of them, however the complexity is 
> often getting in the way of providing proper support and improving the 
> build system of those projects. We've been trying to migrate to a 
> simpler way to build the runtimes that works for all use cases (notably 
> embedded platforms) for a long time, and we recently got to a point 
> where this is possible (thanks mainly to @phosek and @mstorsjo). To 
> summarize what we're aiming for, we'll now have two ways of
building
> libc++/lib++abi/libunwind:
> 
> 1. The "Default" build, where the CMake invocation is rooted at 
> <root>/runtimes and one uses LLVM_ENABLE_RUNTIMES.
> 2. The "Bootstrapping" build, where the CMake invocation is
rooted at
> <root>/llvm and one uses LLVM_ENABLE_RUNTIMES (this is the current 
> "Bootstrapping" build unchanged).
> 
> If you have been using the "Project", "Standalone" or
"Unified
> Standalone" builds, you should be moving to the new
"Default" build.
> This should be really easy to achieve, basically you can just change 
> LLVM_ENABLE_PROJECTS to LLVM_ENABLE_RUNTIMES, perform a single 
> invocation of CMake instead of multiple invocations, and pass all the 
> LIBCXX_FOO, LIBCXXABI_FOO and LIBUNWIND_FOO options to that single CMake 
> invocation. See the release note in the PR linked below for more details.
> 
> We are aiming to deprecate all the old ways to build as soon as 
> possible, and to keep supporting them until after the LLVM 14 release, 
> at which point we would like to remove support for those and get started 
> with making improvements. The review at https://reviews.llvm.org/D111356 
> <https://reviews.llvm.org/D111356> enshrines this deprecation in the 
> documentation. For technical discussion on this proposed change, please 
> comment on that review to keep everything in one place.
> 
> Cheers,
> Louis
> 
> Questions you might be asking yourself:
> 
> Q: Why doesn't the Project build work for embedded platforms?
> A: A lot of CMake code in <root>/llvm expects that we're building
for a
> target with the usual capabilities for a build host, but that isn't the
> case for several embedded targets. While it doesn't make sense to build
> LLVM or Clang for those embedded targets (which would never be used as 
> build hosts), it makes sense to build the runtimes for those targets if 
> you are cross-compiling.
> 
> Q: Currently, we use the Standalone build and are not required to have 
> all of the Monorepo available. Does this change our ability to do that?
> A: Not in the mid term. Building any of the runtimes already requires 
> all the runtimes to be co-located. The only thing you could get away 
> with previously was not having the part in <root>/llvm. With the 
> "Default" build, this will be possible as soon as we move a few
CMake
> helpers from <root>/llvm/cmake to <root>/cmake. Then,
you'll only have
> to have <root>/runtimes, <root>/cmake and the projects you are
building.
> 
> Q: What's the difference between the "Runtimes build" and the
> "Bootstrapping build"?
> A: None. We're trying to move away from the term "Runtimes
build"
> because it's waaaay too confusing, and doesn't describe the fact
that
> we're really building the runtimes with a just-built Clang.
> 
> Q: What's up with naming the new default build the "Default
build"?
> A: The name sucks, but we had to pick one and the "Runtimes
build" was
> already taken for the build that does a bootstrap. Suggestions are 
> welcome -- basically what this performs is a normal build of the 
> runtimes using your system compiler, like you used to do with 
> LLVM_ENABLE_PROJECTS (but it's better).
> 
> Q: What are the benefits of this migration beyond simplifying the code 
> for maintainers?
> A: Reducing the number of ways we can build the runtimes means that all 
> the workflows for building the runtimes will become simpler for users 
> (especially vendors). Furthermore, the new "Default" build is as
simple
> as the old Project build, but it is faster to configure and works on 
> embedded platforms. In the near future, it should also be possible to 
> perform the equivalent of an old "Project build" without having
all of
> the monorepo available.
> 
> Q: Does this affect other "runtimes" projects like compiler-rt?
> A: Not for the time being. We will probably attempt to move those over 
> in the future as well, however for the time being I can only confirm 
> that this works fine for libc++, libc++abi and libunwind, so that's 
> where my deprecation efforts are focused.
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

On Thu, Oct 28, 2021 at 4:22 AM Andrzej Warzynski <andrzej.warzynski at
arm.com>
wrote:
> Hi Louis,
>
> This sounds like a huge simplification, thank you!
>
> With this change, will we still be able to use LLVM_ENABLE_PROJECTS to
> build libc++, libc++abi and libunwind? IIUC, you are disabling this
> option entirely, right? Will CMake warn us about the
> incorrect/deprecated usage? Should the LLVM documentation be updated [1]?
>
Hi,

The goal is to deprecate and then entirely remove building the runtimes
with LLVM_ENABLE_PROJECTS. Libc++'s own documentation has already been
updated, but I will update the documentation you linked to and add a
warning in the CMake -- thanks for the suggestion!

Cheers,
Louis


>
> Thank you,
> -Andrzej
>
> [1] https://llvm.org/docs/CMake.html
>
> On 11/10/2021 17:17, Louis Dionne via llvm-dev wrote:
> > Hi folks,
> >
> > This message is both a heads up and a request for comments on an
> > upcoming change we want to make to the way we build
> > libc++/libc++abi/libunwind.
> >
> > For a long time now, those three projects have supported various ways
to
> > be built:
> >
> > 1. The Project build, where one would use <root>/llvm as the
root of the
> > CMake invocation and pass LLVM_ENABLE_PROJECTS.
> > 2. The Standalone build, where one would perform one CMake invocation
> > per project being built, and fill in various CMake variables to tell
> > each project where to find bits of the other projects.
> > 3. The "Runtimes" or "Bootstrapping" build, where
one would use
> > <root>/llvm as the root of the CMake invocation and pass
> > LLVM_ENABLE_RUNTIMES. This would result in Clang being built, and then
> > the runtimes being built with that fresh Clang.
> > 4. The "Unified Standalone" build, where one would use
> > <root>/libcxx/utils/ci/runtimes as the root of the CMake
invocation and
> > pass LLVM_ENABLE_PROJECTS. This was introduced to work around the fact
> > that the Project build doesn't work on several embedded platforms,
and I
> > think it is only used at Apple.
> >
> > Through the years, this has gotten rather confusing and difficult to
> > support. We currently support all of those on a best-effort basis and
we
> > do have build bots checking each of them, however the complexity is
> > often getting in the way of providing proper support and improving the
> > build system of those projects. We've been trying to migrate to a
> > simpler way to build the runtimes that works for all use cases
(notably
> > embedded platforms) for a long time, and we recently got to a point
> > where this is possible (thanks mainly to @phosek and @mstorsjo). To
> > summarize what we're aiming for, we'll now have two ways of
building
> > libc++/lib++abi/libunwind:
> >
> > 1. The "Default" build, where the CMake invocation is rooted
at
> > <root>/runtimes and one uses LLVM_ENABLE_RUNTIMES.
> > 2. The "Bootstrapping" build, where the CMake invocation is
rooted at
> > <root>/llvm and one uses LLVM_ENABLE_RUNTIMES (this is the
current
> > "Bootstrapping" build unchanged).
> >
> > If you have been using the "Project", "Standalone"
or "Unified
> > Standalone" builds, you should be moving to the new
"Default" build.
> > This should be really easy to achieve, basically you can just change
> > LLVM_ENABLE_PROJECTS to LLVM_ENABLE_RUNTIMES, perform a single
> > invocation of CMake instead of multiple invocations, and pass all the
> > LIBCXX_FOO, LIBCXXABI_FOO and LIBUNWIND_FOO options to that single
CMake
> > invocation. See the release note in the PR linked below for more
details.
> >
> > We are aiming to deprecate all the old ways to build as soon as
> > possible, and to keep supporting them until after the LLVM 14 release,
> > at which point we would like to remove support for those and get
started
> > with making improvements. The review at
https://reviews.llvm.org/D111356
> > <https://reviews.llvm.org/D111356> enshrines this deprecation in
the
> > documentation. For technical discussion on this proposed change,
please
> > comment on that review to keep everything in one place.
> >
> > Cheers,
> > Louis
> >
> > Questions you might be asking yourself:
> >
> > Q: Why doesn't the Project build work for embedded platforms?
> > A: A lot of CMake code in <root>/llvm expects that we're
building for a
> > target with the usual capabilities for a build host, but that
isn't the
> > case for several embedded targets. While it doesn't make sense to
build
> > LLVM or Clang for those embedded targets (which would never be used as
> > build hosts), it makes sense to build the runtimes for those targets
if
> > you are cross-compiling.
> >
> > Q: Currently, we use the Standalone build and are not required to have
> > all of the Monorepo available. Does this change our ability to do
that?
> > A: Not in the mid term. Building any of the runtimes already requires
> > all the runtimes to be co-located. The only thing you could get away
> > with previously was not having the part in <root>/llvm. With the
> > "Default" build, this will be possible as soon as we move a
few CMake
> > helpers from <root>/llvm/cmake to <root>/cmake. Then,
you'll only have
> > to have <root>/runtimes, <root>/cmake and the projects you
are building.
> >
> > Q: What's the difference between the "Runtimes build"
and the
> > "Bootstrapping build"?
> > A: None. We're trying to move away from the term "Runtimes
build"
> > because it's waaaay too confusing, and doesn't describe the
fact that
> > we're really building the runtimes with a just-built Clang.
> >
> > Q: What's up with naming the new default build the "Default
build"?
> > A: The name sucks, but we had to pick one and the "Runtimes
build" was
> > already taken for the build that does a bootstrap. Suggestions are
> > welcome -- basically what this performs is a normal build of the
> > runtimes using your system compiler, like you used to do with
> > LLVM_ENABLE_PROJECTS (but it's better).
> >
> > Q: What are the benefits of this migration beyond simplifying the code
> > for maintainers?
> > A: Reducing the number of ways we can build the runtimes means that
all
> > the workflows for building the runtimes will become simpler for users
> > (especially vendors). Furthermore, the new "Default" build
is as simple
> > as the old Project build, but it is faster to configure and works on
> > embedded platforms. In the near future, it should also be possible to
> > perform the equivalent of an old "Project build" without
having all of
> > the monorepo available.
> >
> > Q: Does this affect other "runtimes" projects like
compiler-rt?
> > A: Not for the time being. We will probably attempt to move those over
> > in the future as well, however for the time being I can only confirm
> > that this works fine for libc++, libc++abi and libunwind, so
that's
> > where my deprecation efforts are focused.
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211028/43dd1b30/attachment.html>