llvm dev - Nov 2015 - [RFC] Strategies for Bootstrapping Compiler-RT builtins

In the effort to flesh out the CMake build system a problematic issue has come
up, and I’d like some feedback on how to best handle it.

For reference this issue has been reported by a few users, one proposed patches
that don’t really address the underlying problem here:
http://reviews.llvm.org/D13131

The problem comes when bootstrapping a cross-compiler toolchain. In order to
have a cross-compiling toolchain that can build a “hello world” application you
need four basic components:

(1) clang
(2) ld
(3) libclang_rt (builtins)
(4) runtime libraries

Today building this toolchain with CMake is impossible because you cannot
configure the cross-compiled builtins. The failure is a result of CMake’s
try_compile function always testing a full compile then link operation. When
bootstrapping a cross-compiler this will always fail because linking even the
simplest applications fails when you don’t have libclang_rt prebuilt.

So, how do we fix this? I have a couple ideas, and am open to more.

(1) Roll our own CMake checks

We could roll our own replacement to try_compile and the various check macros
that we need. In my opinion this is probably the right solution, but it does
have downsides.

The big downside is that when bootstrapping compiler-rt it will need to build
differently. In particular, it is probable that a bootstrap build of compiler-rt
will not be able to perform the necessary checks to build the runtimes, so when
bootstrapping we’ll need to disable building all the runtime libraries. We can
probably find clever ways to hide a bunch of the complexity here, but it is not
going to be clean.

(2) Provide a way to bootstrap the builtins without CMake

Another alternative would be to provide a way to bootstrap the builtin libraries
without CMake. The builtin libraries are actually very simple to compile. It is
possible to roll a custom build script for use only bootstrapping the builtins
that could run on any platform and just get to a functional compiler. The
biggest downside here is that bootstrapping on all supported platforms with all
supported compilers is actually a non-trivial matrix, and supporting and
maintaining that could be a real pain. This is my least favorite option.

(3) Split the builtins and the runtime libraries

This is the most complicated approach, but I also think it is the best approach.
One of the underlying problems here is that the builtin libraries and the
runtime libraries have very different requirements for building. The builtins
really only require a functional compiler and archiver, and the runtime
libraries require a full linker + runtime libraries (libc & libcxx). These
additional build-time requirements actually make things very complicated because
when bootstrapping a cross toolchain compiler-rt needs to build in two different
places in the build order; once before libcxx, and once after.

I believe that the cleanest solution to this problem is going to be to separate
the builtins and the sanitizers. Doing this and rolling our own CMake checks
would allow us to have a fully CMake solution for building a cross-targeting
toolchain. We might even be able to get support for try_compile checks that
don’t link from CMake which would allow us to get rid of the hand-rolled checks
in the future (I have already started a thread on the cmake-developers list).

Logistically this solution could take many forms. We could break compiler-rt out
into two repositories, which would be a huge undertaking, or we could leave it
as a single repository and have the builtins be able to build as a sub-project.
I think we can make it work such that compiler-rt can be built either from the
top-level directory to build it all, or from the builtins sub directory to
support bootstrapping cross-compilers.

Either way, supporting this approach will require significant cleanup and
refactoring because we’ll need to separate out the build system functionality
into three categories: things that apply to builtins, things that apply to
runtimes, things that apply to both. That separation will need to be somewhat
clearly maintained so that we can prevent inadvertent stream crossing, because
that is almost always bad.

Thoughts? Additional suggestions?

Thanks,
-Chris

Chris Bieneman via llvm-dev <llvm-dev at lists.llvm.org>
writes:
> In the effort to flesh out the CMake build system a problematic issue
> has come up, and I’d like some feedback on how to best handle it.
>
> For reference this issue has been reported by a few users, one
> proposed patches that don’t really address the underlying problem
> here:
> http://reviews.llvm.org/D13131
>
> The problem comes when bootstrapping a cross-compiler toolchain. In
> order to have a cross-compiling toolchain that can build a “hello
> world” application you need four basic components:
>
> (1) clang
> (2) ld
> (3) libclang_rt (builtins)
> (4) runtime libraries
>
> Today building this toolchain with CMake is impossible because you
> cannot configure the cross-compiled builtins. The failure is a result
> of CMake’s try_compile function always testing a full compile then
> link operation. When bootstrapping a cross-compiler this will always
> fail because linking even the simplest applications fails when you
> don’t have libclang_rt prebuilt.
>
> So, how do we fix this? I have a couple ideas, and am open to more.
>
> (1) Roll our own CMake checks
>
> We could roll our own replacement to try_compile and the various check
> macros that we need. In my opinion this is probably the right
> solution, but it does have downsides.
>
> The big downside is that when bootstrapping compiler-rt it will need
> to build differently. In particular, it is probable that a bootstrap
> build of compiler-rt will not be able to perform the necessary checks
> to build the runtimes, so when bootstrapping we’ll need to disable
> building all the runtime libraries. We can probably find clever ways
> to hide a bunch of the complexity here, but it is not going to be
> clean.
>
> (2) Provide a way to bootstrap the builtins without CMake
>
> Another alternative would be to provide a way to bootstrap the builtin
> libraries without CMake. The builtin libraries are actually very
> simple to compile. It is possible to roll a custom build script for
> use only bootstrapping the builtins that could run on any platform and
> just get to a functional compiler. The biggest downside here is that
> bootstrapping on all supported platforms with all supported compilers
> is actually a non-trivial matrix, and supporting and maintaining that
> could be a real pain. This is my least favorite option.
>
> (3) Split the builtins and the runtime libraries
>
> This is the most complicated approach, but I also think it is the best
> approach. One of the underlying problems here is that the builtin
> libraries and the runtime libraries have very different requirements
> for building. The builtins really only require a functional compiler
> and archiver, and the runtime libraries require a full linker +
> runtime libraries (libc & libcxx). These additional build-time
> requirements actually make things very complicated because when
> bootstrapping a cross toolchain compiler-rt needs to build in two
> different places in the build order; once before libcxx, and once
> after.
>
> I believe that the cleanest solution to this problem is going to be to
> separate the builtins and the sanitizers.
To be clear, you mean split the builtins (as in lib/builtins) and all of
the runtime libraries (as in everything else under lib: sanitizers,
profiling runtimes, etc). Correct?
> Doing this and rolling our own CMake checks would allow us to have a
> fully CMake solution for building a cross-targeting toolchain. We
> might even be able to get support for try_compile checks that don’t
> link from CMake which would allow us to get rid of the hand-rolled
> checks in the future (I have already started a thread on the
> cmake-developers list).
>
> Logistically this solution could take many forms. We could break
> compiler-rt out into two repositories, which would be a huge
> undertaking, or we could leave it as a single repository and have the
> builtins be able to build as a sub-project. I think we can make it
> work such that compiler-rt can be built either from the top-level
> directory to build it all, or from the builtins sub directory to
> support bootstrapping cross-compilers.
>
> Either way, supporting this approach will require significant cleanup
> and refactoring because we’ll need to separate out the build system
> functionality into three categories: things that apply to builtins,
> things that apply to runtimes, things that apply to both. That
> separation will need to be somewhat clearly maintained so that we can
> prevent inadvertent stream crossing, because that is almost always
> bad.
>
> Thoughts? Additional suggestions?
ISTM that (3) essentially includes the work from (1), but ends up in a
more maintainable state. (3) feels like the right approach to me, as (1)
will be error prone and hard to maintain, and (2) is kludgy and complex
in a way that could scare people away from contributing.

On Nov 2, 2015, at 11:10 AM, Chris Bieneman via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Today building this toolchain with CMake is impossible because you cannot
configure the cross-compiled builtins. The failure is a result of CMake’s
try_compile function always testing a full compile then link operation. When
bootstrapping a cross-compiler this will always fail because linking even the
simplest applications fails when you don’t have libclang_rt prebuilt.
> 
…

Are there side-effects from try_compile that we need?  Can we disable it?


Jim Rowan
jmr at codeaurora.org
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the
Linux Foundation

> Today building this toolchain with CMake is impossible because you cannot
configure the cross-compiled builtins. The failure is a result of CMake’s
try_compile function always testing a full compile then link operation. When
bootstrapping a cross-compiler this will always fail because linking even the
simplest applications fails when you don’t have libclang_rt prebuilt.
The situation is much worse, because it requires not only compiler-rt,
but also all the runtime libraries (libc, libc++, libc++abi) to be
available.
> This is the most complicated approach, but I also think it is the best
approach.
This won't work due to dependencies on libc / libc++ / libc++abi.

I think either (1) should be implemented. Or (4) - make the libraries
just to be "shimmed" on fly - proving empty libs for the configure
time just to pacify cmake.

-- 
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University

> On Nov 2, 2015, at 9:42 AM, Justin Bogner <mail at justinbogner.com>
wrote:
> 
> Chris Bieneman via llvm-dev <llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org>> writes:
>> In the effort to flesh out the CMake build system a problematic issue
>> has come up, and I’d like some feedback on how to best handle it.
>> 
>> For reference this issue has been reported by a few users, one
>> proposed patches that don’t really address the underlying problem
>> here:
>> http://reviews.llvm.org/D13131
>> 
>> The problem comes when bootstrapping a cross-compiler toolchain. In
>> order to have a cross-compiling toolchain that can build a “hello
>> world” application you need four basic components:
>> 
>> (1) clang
>> (2) ld
>> (3) libclang_rt (builtins)
>> (4) runtime libraries
>> 
>> Today building this toolchain with CMake is impossible because you
>> cannot configure the cross-compiled builtins. The failure is a result
>> of CMake’s try_compile function always testing a full compile then
>> link operation. When bootstrapping a cross-compiler this will always
>> fail because linking even the simplest applications fails when you
>> don’t have libclang_rt prebuilt.
>> 
>> So, how do we fix this? I have a couple ideas, and am open to more.
>> 
>> (1) Roll our own CMake checks
>> 
>> We could roll our own replacement to try_compile and the various check
>> macros that we need. In my opinion this is probably the right
>> solution, but it does have downsides.
>> 
>> The big downside is that when bootstrapping compiler-rt it will need
>> to build differently. In particular, it is probable that a bootstrap
>> build of compiler-rt will not be able to perform the necessary checks
>> to build the runtimes, so when bootstrapping we’ll need to disable
>> building all the runtime libraries. We can probably find clever ways
>> to hide a bunch of the complexity here, but it is not going to be
>> clean.
>> 
>> (2) Provide a way to bootstrap the builtins without CMake
>> 
>> Another alternative would be to provide a way to bootstrap the builtin
>> libraries without CMake. The builtin libraries are actually very
>> simple to compile. It is possible to roll a custom build script for
>> use only bootstrapping the builtins that could run on any platform and
>> just get to a functional compiler. The biggest downside here is that
>> bootstrapping on all supported platforms with all supported compilers
>> is actually a non-trivial matrix, and supporting and maintaining that
>> could be a real pain. This is my least favorite option.
>> 
>> (3) Split the builtins and the runtime libraries
>> 
>> This is the most complicated approach, but I also think it is the best
>> approach. One of the underlying problems here is that the builtin
>> libraries and the runtime libraries have very different requirements
>> for building. The builtins really only require a functional compiler
>> and archiver, and the runtime libraries require a full linker +
>> runtime libraries (libc & libcxx). These additional build-time
>> requirements actually make things very complicated because when
>> bootstrapping a cross toolchain compiler-rt needs to build in two
>> different places in the build order; once before libcxx, and once
>> after.
>> 
>> I believe that the cleanest solution to this problem is going to be to
>> separate the builtins and the sanitizers.
> 
> To be clear, you mean split the builtins (as in lib/builtins) and all of
> the runtime libraries (as in everything else under lib: sanitizers,
> profiling runtimes, etc). Correct?
Yes. lib/builtins is the part that causes all the problems.

-Chris
> 
>> Doing this and rolling our own CMake checks would allow us to have a
>> fully CMake solution for building a cross-targeting toolchain. We
>> might even be able to get support for try_compile checks that don’t
>> link from CMake which would allow us to get rid of the hand-rolled
>> checks in the future (I have already started a thread on the
>> cmake-developers list).
>> 
>> Logistically this solution could take many forms. We could break
>> compiler-rt out into two repositories, which would be a huge
>> undertaking, or we could leave it as a single repository and have the
>> builtins be able to build as a sub-project. I think we can make it
>> work such that compiler-rt can be built either from the top-level
>> directory to build it all, or from the builtins sub directory to
>> support bootstrapping cross-compilers.
>> 
>> Either way, supporting this approach will require significant cleanup
>> and refactoring because we’ll need to separate out the build system
>> functionality into three categories: things that apply to builtins,
>> things that apply to runtimes, things that apply to both. That
>> separation will need to be somewhat clearly maintained so that we can
>> prevent inadvertent stream crossing, because that is almost always
>> bad.
>> 
>> Thoughts? Additional suggestions?
> 
> ISTM that (3) essentially includes the work from (1), but ends up in a
> more maintainable state. (3) feels like the right approach to me, as (1)
> will be error prone and hard to maintain, and (2) is kludgy and complex
> in a way that could scare people away from contributing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151102/b1901044/attachment.html>

> On Nov 2, 2015, at 9:47 AM, Rowan, Jim <jmr at codeaurora.org> wrote:
> 
> 
> On Nov 2, 2015, at 11:10 AM, Chris Bieneman via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
>> 
>> Today building this toolchain with CMake is impossible because you
cannot configure the cross-compiled builtins. The failure is a result of CMake’s
try_compile function always testing a full compile then link operation. When
bootstrapping a cross-compiler this will always fail because linking even the
simplest applications fails when you don’t have libclang_rt prebuilt.
>> 
> 
> …
> 
> Are there side-effects from try_compile that we need?  Can we disable it?
We use try_compile to determine the capabilities of the toolchain you are
building with. It is very important that we be able to make these calls in some
form to figure out supported compiler flags, architectures, etc…

-Chris
> 
> 
> Jim Rowan
> jmr at codeaurora.org
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted
by the Linux Foundation
> 
> 
>

> On Nov 2, 2015, at 10:10 AM, Anton Korobeynikov <anton at
korobeynikov.info> wrote:
> 
>> Today building this toolchain with CMake is impossible because you
cannot configure the cross-compiled builtins. The failure is a result of CMake’s
try_compile function always testing a full compile then link operation. When
bootstrapping a cross-compiler this will always fail because linking even the
simplest applications fails when you don’t have libclang_rt prebuilt.
> The situation is much worse, because it requires not only compiler-rt,
> but also all the runtime libraries (libc, libc++, libc++abi) to be
> available.
> 
>> This is the most complicated approach, but I also think it is the best
approach.
> This won't work due to dependencies on libc / libc++ / libc++abi.
That is not entirely correct. Only some of the builtins have dependencies on
libc, and none of them should have dependencies on libc++ or libc++abi. I
believe we can construct a working partial builtin library without libc, to make
bootstrapping work.

-Chris
> 
> I think either (1) should be implemented. Or (4) - make the libraries
> just to be "shimmed" on fly - proving empty libs for the
configure
> time just to pacify cmake.
> 
> -- 
> With best regards, Anton Korobeynikov
> Faculty of Mathematics and Mechanics, Saint Petersburg State University

> The problem comes when bootstrapping a cross-compiler toolchain. In order
to
> have a cross-compiling toolchain that can build a “hello world” application
you
> need four basic components:
>
> (1) clang
> (2) ld
> (3) libclang_rt (builtins)
> (4) runtime libraries
>
> Today building this toolchain with CMake is impossible because you cannot
> configure the cross-compiled builtins. The failure is a result of CMake’s
> try_compile function always testing a full compile then link operation.
When
> bootstrapping a cross-compiler this will always fail because linking even
the
> simplest applications fails when you don’t have libclang_rt prebuilt.
In my case, I had to use the CMAKE_{C,CXX}_COMPILER_FORCED variables and
guestimate the appropriate C/C++ flags in order to avoid CMake getting in the
way with the various checks. My scripts build clang & lld, then install the
linux's kernel & Musl's C library headers and finally build
compiler-rt's
builtins library (by setting COMPILER_RT_BUILD_SANITIZERS=Off). The next steps
build Musl, libunwind, libcxx and libcxxabi. This way I'm able to
"use" CMake
for every LLVM project.

What isn't clear to me is whether you intend to extend your work to
libunwind,
libcxx & libcxxabi because they do require a working C++ compiler too. Also,
my understanding is that the same is true for the sanitizers libraries. 
> So, how do we fix this? I have a couple ideas, and am open to more.
>
> ...
>
> (3) Split the builtins and the runtime libraries
>
> This is the most complicated approach, but I also think it is the best
> approach. One of the underlying problems here is that the builtin libraries
> and the runtime libraries have very different requirements for building.
The
> builtins really only require a functional compiler and archiver, and the
> runtime libraries require a full linker + runtime libraries (libc &
libcxx).
> These additional build-time requirements actually make things very
> complicated because when bootstrapping a cross toolchain compiler-rt needs
to
> build in two different places in the build order; once before libcxx, and
> once after.
IMHO, from the options that you've mentioned, (3) is the cleanest one. 
Conceptually, as you mentioned the builtins library does not belong with the
sanitizers and it would make sense splitting them in separate repositories.
Would it be too difficult to create an initial split given that we already
provide/support the COMPILER_RT_BUILD_BUILTINS and COMPILER_RT_BUILD_SANITIZERS
options?
> Either way, supporting this approach will require significant cleanup and
> refactoring because we’ll need to separate out the build system
functionality
> into three categories: things that apply to builtins, things that apply to
> runtimes, things that apply to both.
I don't have enought experience with CMake so I don't know how feasible
this is,
but for the last two categories it would be nice to keep in mind that the user
might have a C library other than Glibc.

- Vasileios

Hi Chris - Many thanks for airing all this.   I'm now hopeful for an
end to my own hacks and false starts trying to fix these same
problems.  My response is coming from the perspective of an
out-of-tree target without binutils or libgcc support.

On Mon, Nov 2, 2015 at 9:10 AM, Chris Bieneman via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> (3) Split the built-ins and the runtime libraries
+1.  Eliminating unnecessary entanglement is good engineering and
helps especially in the overwhelming early days of new target
development.
> I believe that the cleanest solution to this problem is going to be to
separate the built-ins and the sanitizers.
+1, again for sensible modularity and being merciful on new targets.
> Logistically this solution could take many forms. We could break
compiler-rt out into two repositories, which would be a huge undertaking, or we
could leave it as a single repository and have the built-ins be able to build as
a sub-project. I think we can make it work such that compiler-rt can be built
either from the top-level directory to build it all, or from the built-ins sub
directory to support bootstrapping cross-compilers.
I'm not sure if this jibes with your statement, but IMHO, the
refactored built-ins should move into the LLVM repo.  In this spirit,
I believe gcc includes libgcc in-tree.  Being both a small library and
intimately tied to LLVM, the complexity of a new separate built-ins
repo is a little dubious.  The sanitizers are a separate can of worms.

Thanks,
-steve

Hi Chris,

Thanks for taking the time to raise this issue.
I know I have been a bit pushy about getting it solved.

I think either (1) should be implemented. Or (4) - make the
libraries
> just to be "shimmed" on fly - proving empty libs for the
configure
> time just to pacify cmake.
Unfortunately option 4 here wont work for some targets.
The specific one I am dealing with is mingw-w64.
The mingw-w64-crt depends on some functions internally in
libgcc/compiler-rt to even build a return 0 from main empty executable.
So this would still fail at link time.
This could have been a solution if we were dealing with linux targets only.

In my case, I had to use the CMAKE_{C,CXX}_COMPILER_FORCED variables
and
> guestimate the appropriate C/C++ flags in order to avoid CMake getting in
> the
> way with the various checks. My scripts build clang & lld, then install
the
> linux's kernel & Musl's C library headers and finally build
compiler-rt's
> builtins library (by setting COMPILER_RT_BUILD_SANITIZERS=Off). The next
> steps
> build Musl, libunwind, libcxx and libcxxabi. This way I'm able to
"use"
> CMake
> for every LLVM project.

This is exactly how I am working around this atm for the mingw-w64 target.
Do you have your scripts somewhere I would like to compare?

IMHO, from the options that you've mentioned, (3) is the cleanest
one.
> Conceptually, as you mentioned the builtins library does not belong with
> the
> sanitizers and it would make sense splitting them in separate repositories.
> Would it be too difficult to create an initial split given that we already
> provide/support the COMPILER_RT_BUILD_BUILTINS and
> COMPILER_RT_BUILD_SANITIZERS
> options?
The issue seems to be that even if we set COMPILER_RT_BUILD_SANITIZERS to
off it still does all the checks.
Maybe it would be possible from within the cmakelists of compiler-rt to
seperate them out more to avoid this when we are just building the builtins?
This seems like a decent first step forward regardless if it is agreed
to separate out into 2 different projects.

I'm not sure if this jibes with your statement, but IMHO,
the
> refactored built-ins should move into the LLVM repo.  In this spirit,
> I believe gcc includes libgcc in-tree.  Being both a small library and
> intimately tied to LLVM, the complexity of a new separate built-ins
> repo is a little dubious.  The sanitizers are a separate can of worms.

This was something I also was thinking about.
There are complexities here though because unlike gcc, llvm is multi target
based.

- Martell


On Mon, Nov 2, 2015 at 9:01 PM, Steve King via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi Chris - Many thanks for airing all this.   I'm now hopeful for an
> end to my own hacks and false starts trying to fix these same
> problems.  My response is coming from the perspective of an
> out-of-tree target without binutils or libgcc support.
>
> On Mon, Nov 2, 2015 at 9:10 AM, Chris Bieneman via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > (3) Split the built-ins and the runtime libraries
>
> +1.  Eliminating unnecessary entanglement is good engineering and
> helps especially in the overwhelming early days of new target
> development.
>
> > I believe that the cleanest solution to this problem is going to be to
> separate the built-ins and the sanitizers.
>
> +1, again for sensible modularity and being merciful on new targets.
>
> > Logistically this solution could take many forms. We could break
> compiler-rt out into two repositories, which would be a huge undertaking,
> or we could leave it as a single repository and have the built-ins be able
> to build as a sub-project. I think we can make it work such that
> compiler-rt can be built either from the top-level directory to build it
> all, or from the built-ins sub directory to support bootstrapping
> cross-compilers.
>
> I'm not sure if this jibes with your statement, but IMHO, the
> refactored built-ins should move into the LLVM repo.  In this spirit,
> I believe gcc includes libgcc in-tree.  Being both a small library and
> intimately tied to LLVM, the complexity of a new separate built-ins
> repo is a little dubious.  The sanitizers are a separate can of worms.
>
> Thanks,
> -steve
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151102/b3aa6629/attachment.html>

> On Nov 2, 2015, at 11:41 AM, Vasileios Kalintiris <Vasileios.Kalintiris
at imgtec.com> wrote:
> 
>> The problem comes when bootstrapping a cross-compiler toolchain. In
order to
>> have a cross-compiling toolchain that can build a “hello world”
application you
>> need four basic components:
>> 
>> (1) clang
>> (2) ld
>> (3) libclang_rt (builtins)
>> (4) runtime libraries
>> 
>> Today building this toolchain with CMake is impossible because you
cannot
>> configure the cross-compiled builtins. The failure is a result of
CMake’s
>> try_compile function always testing a full compile then link operation.
When
>> bootstrapping a cross-compiler this will always fail because linking
even the
>> simplest applications fails when you don’t have libclang_rt prebuilt.
> 
> In my case, I had to use the CMAKE_{C,CXX}_COMPILER_FORCED variables and
> guestimate the appropriate C/C++ flags in order to avoid CMake getting in
the
> way with the various checks. My scripts build clang & lld, then install
the
> linux's kernel & Musl's C library headers and finally build
compiler-rt's
> builtins library (by setting COMPILER_RT_BUILD_SANITIZERS=Off). The next
steps
> build Musl, libunwind, libcxx and libcxxabi. This way I'm able to
"use" CMake
> for every LLVM project.
Setting the *_FORCED variables is roughly equivalent to setting the *_WORKS
variables. I’d really like it if we didn’t need to set those.
> 
> What isn't clear to me is whether you intend to extend your work to
libunwind,
> libcxx & libcxxabi because they do require a working C++ compiler too.
Also,
> my understanding is that the same is true for the sanitizers libraries. 
I have a separate set of patches I’m working on now to generalize our use of
ExternalProject so we can build libcxx, libcxxabi, libunwind, and anything else
that comes along with just-built compilers to fix this.
> 
>> So, how do we fix this? I have a couple ideas, and am open to more.
>> 
>> ...
>> 
>> (3) Split the builtins and the runtime libraries
>> 
>> This is the most complicated approach, but I also think it is the best
>> approach. One of the underlying problems here is that the builtin
libraries
>> and the runtime libraries have very different requirements for
building. The
>> builtins really only require a functional compiler and archiver, and
the
>> runtime libraries require a full linker + runtime libraries (libc &
libcxx).
>> These additional build-time requirements actually make things very
>> complicated because when bootstrapping a cross toolchain compiler-rt
needs to
>> build in two different places in the build order; once before libcxx,
and
>> once after.
> 
> IMHO, from the options that you've mentioned, (3) is the cleanest one. 
> Conceptually, as you mentioned the builtins library does not belong with
the
> sanitizers and it would make sense splitting them in separate repositories.
> Would it be too difficult to create an initial split given that we already
> provide/support the COMPILER_RT_BUILD_BUILTINS and
COMPILER_RT_BUILD_SANITIZERS
> options?
I think there is a lot of work involved here. We’ll need to generalize and split
up some of the configuration content (stuff like config-ix.cmake) into the bits
that apply everywhere, the bits that are needed only for builtins, and the bits
that are only needed for sanitizers. I’m willing to start tackling that work if
this is what the community thinks the right approach is.
> 
>> Either way, supporting this approach will require significant cleanup
and
>> refactoring because we’ll need to separate out the build system
functionality
>> into three categories: things that apply to builtins, things that apply
to
>> runtimes, things that apply to both.
> 
> I don't have enought experience with CMake so I don't know how
feasible this is,
> but for the last two categories it would be nice to keep in mind that the
user
> might have a C library other than Glibc.
I’m certainly not going to expect the user has Glibc. I do all my work on Darwin
platforms and we don’t have Glibc.

-Chris
> 
> - Vasileios

> On Nov 2, 2015, at 12:01 PM, Steve King <steve at metrokings.com>
wrote:
> 
> Hi Chris - Many thanks for airing all this.   I'm now hopeful for an
> end to my own hacks and false starts trying to fix these same
> problems.  My response is coming from the perspective of an
> out-of-tree target without binutils or libgcc support.
> 
> On Mon, Nov 2, 2015 at 9:10 AM, Chris Bieneman via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> (3) Split the built-ins and the runtime libraries
> 
> +1.  Eliminating unnecessary entanglement is good engineering and
> helps especially in the overwhelming early days of new target
> development.
> 
>> I believe that the cleanest solution to this problem is going to be to
separate the built-ins and the sanitizers.
> 
> +1, again for sensible modularity and being merciful on new targets.
> 
>> Logistically this solution could take many forms. We could break
compiler-rt out into two repositories, which would be a huge undertaking, or we
could leave it as a single repository and have the built-ins be able to build as
a sub-project. I think we can make it work such that compiler-rt can be built
either from the top-level directory to build it all, or from the built-ins sub
directory to support bootstrapping cross-compilers.
> 
> I'm not sure if this jibes with your statement, but IMHO, the
> refactored built-ins should move into the LLVM repo.  In this spirit,
> I believe gcc includes libgcc in-tree.  Being both a small library and
> intimately tied to LLVM, the complexity of a new separate built-ins
> repo is a little dubious.  The sanitizers are a separate can of worms.
Sadly, I believe there are licensing reasons why the builtins can’t be in the
LLVM repo. Specifically the LLVM license has an attribution clause that the
compiler-rt license deliberately doesn’t have. The LLVM attribution clause if
applied to the builtins would mean anyone who builds an application that links
the builtins would need to distribute their software with a notice saying it
includes LLVM code.

-Chris
> 
> Thanks,
> -steve