llvm dev - Apr 2020 - Clarifying the supported ways to build libc++, libc++abi and libunwind

[Cross-post to llvm-dev to make sure everybody relevant sees this]

Hi,

I'm currently trying to simplify the libc++/libc++abi/libunwind build
systems and testing setup. In doing so, I am encountering issues related to
"unusual" ways of building them. By unusual, I just mean "not the
usual monorepo build with LLVM_ENABLE_PROJECTS". I would like to pin down
what the set of supported use cases for building the runtime libraries are. In
particular, the world I would like to live in is one where the only way to build
libc++/libc++abi/libunwind is:

    $ mkdir build
    $ cd build
    $ cmake <monorepo-root>/llvm
-DLLVM_ENABLE_PROJECTS=libcxx;libcxxabi;libunwind <options>
    $ ninja -C build install-{cxx,cxxabi,unwind}

The "runtimes" build would be built on top of this -- it would be just
a driver for building these libraries using documented options against the
just-built Clang. I think it already does so in essence, however if I'm not
mistaken it uses the "Standalone build" and it definitely sets some
magic and undocumented CMake variables (like HAVE_LIBCXXABI) that we have to be
really careful not to break.

So, to better understand what people use today, I have some questions. I know
the answer to some of those, but I want to see what others have to say:

1. What is a "Standalone build"? What does it enable that a normal
monorepo build can't?
2. What is the "Runtimes" build? How does it work, what is it used
for, and what does it expect from libc++/libc++abi/libunwind?
3. Are there other "hidden" ways to build the runtime libraries?

Cheers,
Louis

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200408/2062a647/attachment.html>

On 04/08/2020 08:46 AM, Louis Dionne via llvm-dev wrote:
> [Cross-post to llvm-dev to make sure everybody relevant sees this]
> 
> Hi,
> 
> I'm currently trying to simplify the libc++/libc++abi/libunwind build
systems and testing setup. In doing so, I am encountering issues related to
"unusual" ways of building them. By unusual, I just mean "not the
usual monorepo build with LLVM_ENABLE_PROJECTS". I would like to pin down
what the set of supported use cases for building the runtime libraries are. In
particular, the world I would like to live in is one where the only way to build
libc++/libc++abi/libunwind is:
> 
>     $ mkdir build
>     $ cd build
>     $ cmake <monorepo-root>/llvm
-DLLVM_ENABLE_PROJECTS=libcxx;libcxxabi;libunwind <options>
>     $ ninja -C build install-{cxx,cxxabi,unwind}
> 
> The "runtimes" build would be built on top of this -- it would be
just a driver for building these libraries using documented options against the
just-built Clang. I think it already does so in essence, however if I'm not
mistaken it uses the "Standalone build" and it definitely sets some
magic and undocumented CMake variables (like HAVE_LIBCXXABI) that we have to be
really careful not to break.
> 
> So, to better understand what people use today, I have some questions. I
know the answer to some of those, but I want to see what others have to say:
> 
> 1. What is a "Standalone build"? What does it enable that a
normal monorepo build can't?
We use stand-alone builds in Fedora.  How we build is essentially like this:


$ mkdir build
$ cd build
$ cmake <monorepo-root>/libcxx <options>
$ ninja -C build
$ ninja -C build install

The main advantages of building this way is that you don't need the full
source tarball, just the libcxx source, and also it ensures that the
default targets only build the parts that we need.

For reference, the spec files we use for building these projects can be found
here:
https://src.fedoraproject.org/rpms/libcxx/blob/master/f/libcxx.spec
https://src.fedoraproject.org/rpms/libcxxabi/blob/master/f/libcxxabi.spec

-Tom
> 2. What is the "Runtimes" build? How does it work, what is it
used for, and what does it expect from libc++/libc++abi/libunwind?
> 3. Are there other "hidden" ways to build the runtime libraries?
> 
> Cheers,
> Louis
> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

Hello Louis,

I earlier this week converted our toolchain scripts from a monorepo
build for libc++ to a "standalone build" where I build libunwind,
libc++abi and libc++ in succession instead of one single build. The
only reason I converted to this is because it's the only sane way to
build libc++ for a system without a existing C++ runtime. The problem
is actually not that libc++ depends on a existing C++ runtime - but
rather that CMake testing does.

When running a monorepo build on a system without a C++ runtime it
will fail a lot of tests from HandleLLVMOptions, while the standalone
builds for unwind, libc++abi and libc++ is much simpler and have less
tests I need to force a value for. This way I don't need first add a
c++ runtime.

My scripts are an adoption of mstorsjo's llvm-mingw (his script to
build libc++ for mingw can be seen here:
https://github.com/mstorsjo/llvm-mingw/blob/master/build-libcxx.sh).

I am all for simplifying this build on the other hand - building
libc++ and knowing which of the three should be static or dynamic and
when to use the USE_STATIC_UNWINDER etc is very complex and required a
lot of iterations to get it to work and I am still not sure if I did
the best thing (I ended up with static libc++abi but shared unwinder).

Thanks,
Tobias

On Wed, Apr 8, 2020 at 5:46 PM Louis Dionne via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> [Cross-post to llvm-dev to make sure everybody relevant sees this]
>
> Hi,
>
> I'm currently trying to simplify the libc++/libc++abi/libunwind build
systems and testing setup. In doing so, I am encountering issues related to
"unusual" ways of building them. By unusual, I just mean "not the
usual monorepo build with LLVM_ENABLE_PROJECTS". I would like to pin down
what the set of supported use cases for building the runtime libraries are. In
particular, the world I would like to live in is one where the only way to build
libc++/libc++abi/libunwind is:
>
>     $ mkdir build
>     $ cd build
>     $ cmake <monorepo-root>/llvm
-DLLVM_ENABLE_PROJECTS=libcxx;libcxxabi;libunwind <options>
>     $ ninja -C build install-{cxx,cxxabi,unwind}
>
> The "runtimes" build would be built on top of this -- it would be
just a driver for building these libraries using documented options against the
just-built Clang. I think it already does so in essence, however if I'm not
mistaken it uses the "Standalone build" and it definitely sets some
magic and undocumented CMake variables (like HAVE_LIBCXXABI) that we have to be
really careful not to break.
>
> So, to better understand what people use today, I have some questions. I
know the answer to some of those, but I want to see what others have to say:
>
> 1. What is a "Standalone build"? What does it enable that a
normal monorepo build can't?
> 2. What is the "Runtimes" build? How does it work, what is it
used for, and what does it expect from libc++/libc++abi/libunwind?
> 3. Are there other "hidden" ways to build the runtime libraries?
>
> Cheers,
> Louis
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

(CCing Chris and Petr, who’ve done the most work on the runtimes build)

At least for me on Linux, using LLVM_ENABLE_PROJECTS is actually the unusual way
of building libc++; I use LLVM_ENABLE_RUNTIMES. The reason is, my host compiler
is often gcc, but I want to build, test, and ship libc++ with the clang I just
built.

The runtimes build is when you use LLVM_ENABLE_RUNTIMES. It sets up the build of
all runtimes (compiler-rt, libc++, libc++abi, libunwind, etc.) as a CMake
ExternalProject which depends on the build of clang and other toolchain tools.
In other words, if I run the following:

cmake -DLLVM_ENABLE_PROJECTS=clang
-DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi' path/to/my/llvm-project/llvm
ninja cxx

The build system will automatically build clang and other toolchain tools (e.g.
llvm-ar), run the ExternalProject configuration with e.g. CMAKE_C_COMPILER and
CMAKE_CXX_COMPILER set to the just-built clang, and then build libc++ with that
configuration (so with the just-built clang). It’s a pretty convenient workflow
for my setup. It also takes care of e.g. automatically rebuilding libc++ if you
make changes to clang and then run `ninja cxx` again.

As for why the runtimes build use the “standalone build” setup, it’s because
there’s a separate CMake configuration happening for the runtimes in this setup
(which is necessary in order to be able to configure them to use the just-built
toolchain), so e.g. clang isn’t available as an in-tree target. See
https://reviews.llvm.org/D62410 for more details. Your top-level CMakeLists.txt
in the runtimes build is llvm/runtimes/CMakeLists.txt and not
libcxx/CMakeLists.txt (as it would be in a fully standalone build), but it’s
also not llvm/CMakeLists.txt (as it would be with LLVM_ENABLE_PROJECTS).

At the CMake round table at the dev meeting last October, we’d discussed the
runtimes builds, and Chris had advanced that there should be two supported ways
to build the runtimes:

  1.  If you’re building as part of the LLVM build and using
LLVM_ENABLE_RUNTIMES
  2.  If you’re building fully standalone (as in you’re pointing CMake to
libc++’s source directory, so that your top-level CMakeLists is
libcxx/CMakeLists.txt)

I agree with that position. In particular, I think LLVM_ENABLE_PROJECTS is
definitely the wrong thing to use for compiler-rt, which is strongly tied to
your just-built compiler. I think it’s arguably the wrong thing to use for
libc++/libc++abi/libunwind as well, where you either want to use your just-built
Clang (which LLVM_ENABLE_RUNTIMES was made for), or a different compiler (in
which case you’d do a fully standalone build), but silently using your host
compiler for them is probably not what you want. (The LLVM_ENABLE_PROJECTS
workflow for libc++/libc++abi/libunwind probably works out better on macOS,
where your host compiler is a recent Clang anyway so the difference between it
and a just-built Clang aren’t as marked, but I’ve had issues even on macOS in
the past where using the host compiler via LLVM_ENABLE_PROJECTS gave me weird
libc++ test errors, and LLVM_ENABLE_RUNTIMES just worked.)

What do you think?

From: llvm-dev <llvm-dev-bounces at lists.llvm.org> on behalf of Louis
Dionne via llvm-dev <llvm-dev at lists.llvm.org>
Reply-To: Louis Dionne <ldionne at apple.com>
Date: Wednesday, April 8, 2020 at 8:46 AM
To: Libc++ Dev <libcxx-dev at lists.llvm.org>
Cc: llvm-dev <llvm-dev at lists.llvm.org>
Subject: [llvm-dev] Clarifying the supported ways to build libc++, libc++abi and
libunwind

[Cross-post to llvm-dev to make sure everybody relevant sees this]

Hi,

I'm currently trying to simplify the libc++/libc++abi/libunwind build
systems and testing setup. In doing so, I am encountering issues related to
"unusual" ways of building them. By unusual, I just mean "not the
usual monorepo build with LLVM_ENABLE_PROJECTS". I would like to pin down
what the set of supported use cases for building the runtime libraries are. In
particular, the world I would like to live in is one where the only way to build
libc++/libc++abi/libunwind is:

    $ mkdir build
    $ cd build
    $ cmake <monorepo-root>/llvm
-DLLVM_ENABLE_PROJECTS=libcxx;libcxxabi;libunwind <options>
    $ ninja -C build install-{cxx,cxxabi,unwind}

The "runtimes" build would be built on top of this -- it would be just
a driver for building these libraries using documented options against the
just-built Clang. I think it already does so in essence, however if I'm not
mistaken it uses the "Standalone build" and it definitely sets some
magic and undocumented CMake variables (like HAVE_LIBCXXABI) that we have to be
really careful not to break.

So, to better understand what people use today, I have some questions. I know
the answer to some of those, but I want to see what others have to say:

1. What is a "Standalone build"? What does it enable that a normal
monorepo build can't?
2. What is the "Runtimes" build? How does it work, what is it used
for, and what does it expect from libc++/libc++abi/libunwind?
3. Are there other "hidden" ways to build the runtime libraries?

Cheers,
Louis

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200408/2342d6f6/attachment.html>

Thanks Shoaib for a great summary. To summarize this as an answer to Louis'
questions:

1. What is a "Standalone build"? What does it enable that a normal
monorepo
build can't?

This means building any of the runtimes separately, where the runtime's
CMakeLists.txt (e.g. path/to/my/llvm-project/libcxx/CMakeLists.txt) is the
top-level one. The reason for using this variant is the ability to build
individual runtimes without the rest of LLVM (e.g. building only libc++)
which is not something that the monorepo build can do.

2. What is the "Runtimes" build? How does it work, what is it used
for, and
what does it expect from libc++/libc++abi/libunwind?

This means building runtimes with the just-built Clang. The reason why this
is complicated is because at the point when you run CMake for runtimes
(i.e. where path/to/my/llvm-project/llvm/runtimes/CMakeLists.txt is the
top-level one), you may not have a fully working toolchain yet.

For example, in case of Fuchsia, our sysroot contains only libc, so when
we're doing the runtimes build, and one of the runtimes tries to
use check_cxx_compiler_flag, which behind the scenes tries to compile and
link a small C++ binary, that check is going to fail not because the flag
isn't supported, but because we don't have libc++ yet (that's what
we're
trying to build right now). So we need to be really careful and avoid
introducing cycles e.g. where libc++ CMake build depends C++ standard
library (even if that dependency is not explicit).

Note that this is going to become significantly easier after we upgrade
CMake because 3.6 introduced
https://cmake.org/cmake/help/v3.6/variable/CMAKE_TRY_COMPILE_TARGET_TYPE.html
which
allows building static archives instead of executables when running check_*
functions which can help and break these cycles.

3. Are there other "hidden" ways to build the runtime libraries?

Not "hidden", but there's the default way which most developers
use where
you use the same host compiler to build both Clang and your runtimes. I
think this mode should go away because it's too fragile: it silently relies
on your host compiler and the Clang you just built using the same ABI,
which isn't guaranteed (unless you're using a multi-stage build) and
I'm
surprised we haven't yet seen issues due to this (or maybe we did and
people just aren't aware of the problem).

(Why do we need HAVE_* flags?)

It's because in the runtimes build, there's no guarantee about the order
in
which runtimes are being built, so you cannot use e.g. if (TARGET
cxxabi_shared) from within libc++ build because you don't have any
guarantee whether libc++abi has already been processed. So instead,
runtimes build sets these flags for each runtime being built (that is each
runtime specified in -DLLVM_ENABLE_RUNTIMES=) and you can at least check
whether that runtime is being built at all (e.g. you can check from within
libc++ whether libc++abi is also being built as part of the runtimes build).

There's more discussion related to this in
https://reviews.llvm.org/D68833, once
we update CMake to >=3.11 we'll eliminate all HAVE_* variables and
replace
them with generator expressions which is a much better solution.

On Wed, Apr 8, 2020 at 1:21 PM Shoaib Meenai <smeenai at fb.com> wrote:
> (CCing Chris and Petr, who’ve done the most work on the runtimes build)
>
>
>
> At least for me on Linux, using LLVM_ENABLE_PROJECTS is actually the
> unusual way of building libc++; I use LLVM_ENABLE_RUNTIMES. The reason is,
> my host compiler is often gcc, but I want to build, test, and ship libc++
> with the clang I just built.
>
>
>
> The runtimes build is when you use LLVM_ENABLE_RUNTIMES. It sets up the
> build of all runtimes (compiler-rt, libc++, libc++abi, libunwind, etc.) as
> a CMake ExternalProject which depends on the build of clang and other
> toolchain tools. In other words, if I run the following:
>
>
>
> cmake -DLLVM_ENABLE_PROJECTS=clang
> -DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi'
path/to/my/llvm-project/llvm
>
> ninja cxx
>
>
>
> The build system will automatically build clang and other toolchain tools
> (e.g. llvm-ar), run the ExternalProject configuration with e.g.
> CMAKE_C_COMPILER and CMAKE_CXX_COMPILER set to the just-built clang, and
> then build libc++ with that configuration (so with the just-built clang).
> It’s a pretty convenient workflow for my setup. It also takes care of e.g.
> automatically rebuilding libc++ if you make changes to clang and then run
> `ninja cxx` again.
>
>
>
> As for why the runtimes build use the “standalone build” setup, it’s
> because there’s a separate CMake configuration happening for the runtimes
> in this setup (which is necessary in order to be able to configure them to
> use the just-built toolchain), so e.g. clang isn’t available as an in-tree
> target. See https://reviews.llvm.org/D62410 for more details. Your
> top-level CMakeLists.txt in the runtimes build is
> llvm/runtimes/CMakeLists.txt and not libcxx/CMakeLists.txt (as it would be
> in a fully standalone build), but it’s also not llvm/CMakeLists.txt (as it
> would be with LLVM_ENABLE_PROJECTS).
>
>
>
> At the CMake round table at the dev meeting last October, we’d discussed
> the runtimes builds, and Chris had advanced that there should be two
> supported ways to build the runtimes:
>
>    1. If you’re building as part of the LLVM build and using
>    LLVM_ENABLE_RUNTIMES
>    2. If you’re building fully standalone (as in you’re pointing CMake to
>    libc++’s source directory, so that your top-level CMakeLists is
>    libcxx/CMakeLists.txt)
>
>
>
> I agree with that position. In particular, I think LLVM_ENABLE_PROJECTS is
> definitely the wrong thing to use for compiler-rt, which is strongly tied
> to your just-built compiler. I think it’s arguably the wrong thing to use
> for libc++/libc++abi/libunwind as well, where you either want to use your
> just-built Clang (which LLVM_ENABLE_RUNTIMES was made for), or a different
> compiler (in which case you’d do a fully standalone build), but silently
> using your host compiler for them is probably not what you want. (The
> LLVM_ENABLE_PROJECTS workflow for libc++/libc++abi/libunwind probably works
> out better on macOS, where your host compiler is a recent Clang anyway so
> the difference between it and a just-built Clang aren’t as marked, but I’ve
> had issues even on macOS in the past where using the host compiler via
> LLVM_ENABLE_PROJECTS gave me weird libc++ test errors, and
> LLVM_ENABLE_RUNTIMES just worked.)
>
>
>
> What do you think?
>
>
>
> *From: *llvm-dev <llvm-dev-bounces at lists.llvm.org> on behalf of
Louis
> Dionne via llvm-dev <llvm-dev at lists.llvm.org>
> *Reply-To: *Louis Dionne <ldionne at apple.com>
> *Date: *Wednesday, April 8, 2020 at 8:46 AM
> *To: *Libc++ Dev <libcxx-dev at lists.llvm.org>
> *Cc: *llvm-dev <llvm-dev at lists.llvm.org>
> *Subject: *[llvm-dev] Clarifying the supported ways to build libc++,
> libc++abi and libunwind
>
>
>
> [Cross-post to llvm-dev to make sure everybody relevant sees this]
>
>
>
> Hi,
>
>
>
> I'm currently trying to simplify the libc++/libc++abi/libunwind build
> systems and testing setup. In doing so, I am encountering issues related to
> "unusual" ways of building them. By unusual, I just mean
"not the usual
> monorepo build with LLVM_ENABLE_PROJECTS". I would like to pin down
what
> the set of supported use cases for building the runtime libraries are. In
> particular, the world I would like to live in is one where the only way to
> build libc++/libc++abi/libunwind is:
>
>
>
>     $ mkdir build
>
>     $ cd build
>
>     $ cmake <monorepo-root>/llvm
> -DLLVM_ENABLE_PROJECTS=libcxx;libcxxabi;libunwind <options>
>
>     $ ninja -C build install-{cxx,cxxabi,unwind}
>
>
>
> The "runtimes" build would be built on top of this -- it would be
just a
> driver for building these libraries using documented options against the
> just-built Clang. I think it already does so in essence, however if I'm
not
> mistaken it uses the "Standalone build" and it definitely sets
some magic
> and undocumented CMake variables (like HAVE_LIBCXXABI) that we have to be
> really careful not to break.
>
>
>
> So, to better understand what people use today, I have some questions. I
> know the answer to some of those, but I want to see what others have to
say:
>
>
>
> 1. What is a "Standalone build"? What does it enable that a
normal
> monorepo build can't?
>
> 2. What is the "Runtimes" build? How does it work, what is it
used for,
> and what does it expect from libc++/libc++abi/libunwind?
>
> 3. Are there other "hidden" ways to build the runtime libraries?
>
>
>
> Cheers,
>
> Louis
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200408/f2525929/attachment.html>