llvm dev - Dec 2017 - RFC: Default path for cross-compiled runtimes

Today, there're two different locations for runtimes files within
Clang's
installation:

compiler-rt:
  headers: $prefix/lib/clang/$version/include(/sanitizer)
  libraries: $prefix/lib/clang/$version/lib/$os/libclang_rt.$name-$arch.$ext

libc++, libc++abi, libunwind:
  headers: $prefix/include/c++/v1
  libraries: $prefix/lib/$name.$ext

The scheme used by libc++, libc++abi, libunwind doesn't support targets
other than the host which is a problem when cross-compiling. In our
toolchain, we would like to build runtimes for all host and target
platforms we support, e.g. a single toolchain will have runtimes for x86_64
and aarch64 Linux, Fuchsia and Windows. All you need to provide is the
target triple and the sysroot. While this is possible with builtins,
sanitizers and other compiler-rt runtimes, it's not possible with libc++,
libc++abi, libunwind.

Our proposal is to move both compiler-rt and libc++, libc++abi, libunwind
into a new location that would support cross-compilation and unify the
layout:

headers: $prefix/lib/clang/$version/include(/$triple)(/c++/v1)
libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext

This means that for compiler-rt, the main difference would be moving the
runtime libraries to an target specific subdirectory rather than including
the architecture in the library name; for libc++, libc++abi, libunwind,
both headers and libraries will be moved to a new, target specific location.

In terms of implementation, we'll need to modify the Clang driver to use
this location when looking for runtimes and C++ libraries and headers. This
should be a non-intrusive change: we'll modify the driver to look into the
new location in addition to the existing ones, so if the new path doesn't
exist, the driver will simply fallback to the existing behavior. When this
is done, we need to modify the CMake build to install files into the new
location.

This layout would be only used when runtimes are built as part of the
llvm/runtimes tree or using LLVM_ENABLE_RUNTIMES in the monorepo layout
(because this setup supports cross-compiling runtimes). When built as part
of LLVM or standalone, libc++, libc++abi, libunwind would still install
their files to $prefix/include and $prefix/lib as today.

Once the overall scheme is agreed upon, we could also consider
de-duplicating C++ headers across targets, by moving shared headers into a
common directory, with only varying subset in include/$triple.

To give an example, for x86_64 and aarch64 Linux, this would look like:

$prefix/lib/clang/6.0.0/include/sanitizer
$prefix/lib/clang/6.0.0/include/c++/v1
$prefix/lib/clang/6.0.0/include/x86_64-linux-gnu/c++/v1/__config
...
$prefix/lib/clang/6.0.0/x86_64-linux-gnu/lib/libclang_rt.asan.so
$prefix/lib/clang/6.0.0/x86_64-linux-gnu/lib/libc++.so
...
$prefix/lib/clang/6.0.0/aarch64-linux-gnu/lib/libclang_rt.asan.so
$prefix/lib/clang/6.0.0/aarch64-linux-gnu/lib/libc++.so
...
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171219/5c9ac661/attachment.html>

On Tue, Dec 19, 2017 at 04:15:53PM +0000, Petr Hosek via cfe-dev
wrote:
> The scheme used by libc++, libc++abi, libunwind doesn't support targets
> other than the host which is a problem when cross-compiling. In our
> toolchain, we would like to build runtimes for all host and target
> platforms we support, e.g. a single toolchain will have runtimes for x86_64
> and aarch64 Linux, Fuchsia and Windows. All you need to provide is the
> target triple and the sysroot. While this is possible with builtins,
> sanitizers and other compiler-rt runtimes, it's not possible with
libc++,
> libc++abi, libunwind.
I don't understand what you mean. Normally, $prefix is part of sysroot
and there is no problem here.

Joerg

On 12/19/17 9:15 AM, Petr Hosek via llvm-dev wrote:
> Today, there're two different locations for runtimes files within 
> Clang's installation:
> 
> compiler-rt:
>    headers: $prefix/lib/clang/$version/include(/sanitizer)
>    libraries: 
> $prefix/lib/clang/$version/lib/$os/libclang_rt.$name-$arch.$ext
> 
> libc++, libc++abi, libunwind:
>    headers: $prefix/include/c++/v1
>    libraries: $prefix/lib/$name.$ext
> 
> The scheme used by libc++, libc++abi, libunwind doesn't support targets
> other than the host which is a problem when cross-compiling.
Yes, it does: --sysroot
What's currently missing is a standardized naming for a) where the 
sysroots live, and b) what they're named. Host libraries should continue 
to live in $prefix/{include, lib, lib32, lib64} as appropriate, whereas 
target libraries should live in something like: 
$prefix/clang-runtimes/$triple/$multilib/{usr/include, usr/lib, etc.}

  In our
> toolchain, we would like to build runtimes for all host and target 
> platforms we support, e.g. a single toolchain will have runtimes for 
> x86_64 and aarch64 Linux, Fuchsia and Windows. All you need to provide 
> is the target triple and the sysroot. While this is possible with 
> builtins, sanitizers and other compiler-rt runtimes, it's not possible 
> with libc++, libc++abi, libunwind.
> 
> Our proposal is to move both compiler-rt and libc++, libc++abi, 
> libunwind into a new location that would support cross-compilation and 
> unify the layout:
> 
> headers: $prefix/lib/clang/$version/include(/$triple)(/c++/v1)
> libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
I don't think it's a good idea to tie all the runtimes to the compiler 
like that. It makes sense for the builtins to live there since they are 
heavily coupled with the specific compiler version, but I'm not 
convinced libc++/libc++abi/libunwind should.
> 
> This means that for compiler-rt, the main difference would be moving the 
> runtime libraries to an target specific subdirectory rather than 
> including the architecture in the library name; for libc++, libc++abi, 
> libunwind, both headers and libraries will be moved to a new, target 
> specific location.
> 
> In terms of implementation, we'll need to modify the Clang driver to
use
> this location when looking for runtimes and C++ libraries and headers. 
> This should be a non-intrusive change: we'll modify the driver to look 
> into the new location in addition to the existing ones, so if the new 
> path doesn't exist, the driver will simply fallback to the existing 
> behavior. When this is done, we need to modify the CMake build to 
> install files into the new location.
> 
> This layout would be only used when runtimes are built as part of the 
> llvm/runtimes tree or using LLVM_ENABLE_RUNTIMES in the monorepo layout 
> (because this setup supports cross-compiling runtimes). When built as 
> part of LLVM or standalone, libc++, libc++abi, libunwind would still 
> install their files to $prefix/include and $prefix/lib as today.
> 
> Once the overall scheme is agreed upon, we could also consider 
> de-duplicating C++ headers across targets, by moving shared headers into 
> a common directory, with only varying subset in include/$triple.
At the very least the __config_site headers cannot be de-duplicated.

Beyond that, I strongly disagree with de-duplicating them in general 
because it will break --sysroot= unless you add a bunch of symlinks... 
which don't exist on every host platform.


Jon
> 
> To give an example, for x86_64 and aarch64 Linux, this would look like:
> 
> $prefix/lib/clang/6.0.0/include/sanitizer
> $prefix/lib/clang/6.0.0/include/c++/v1
> $prefix/lib/clang/6.0.0/include/x86_64-linux-gnu/c++/v1/__config
> ...
> $prefix/lib/clang/6.0.0/x86_64-linux-gnu/lib/libclang_rt.asan.so 
> <http://libclang_rt.asan.so>
> $prefix/lib/clang/6.0.0/x86_64-linux-gnu/lib/libc++.so
> ...
> $prefix/lib/clang/6.0.0/aarch64-linux-gnu/lib/libclang_rt.asan.so 
> <http://libclang_rt.asan.so>
> $prefix/lib/clang/6.0.0/aarch64-linux-gnu/lib/libc++.so
> ...
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
-- 
Jon Roelofs
jonathan at codesourcery.com
CodeSourcery / Mentor Embedded / Siemens

On Tue, Dec 19, 2017 at 8:33 AM Jonathan Roelofs <jonathan at
codesourcery.com>
wrote:
> On 12/19/17 9:15 AM, Petr Hosek via llvm-dev wrote:
> > Today, there're two different locations for runtimes files within
> > Clang's installation:
> >
> > compiler-rt:
> >    headers: $prefix/lib/clang/$version/include(/sanitizer)
> >    libraries:
> > $prefix/lib/clang/$version/lib/$os/libclang_rt.$name-$arch.$ext
> >
> > libc++, libc++abi, libunwind:
> >    headers: $prefix/include/c++/v1
> >    libraries: $prefix/lib/$name.$ext
> >
> > The scheme used by libc++, libc++abi, libunwind doesn't support
targets
> > other than the host which is a problem when cross-compiling.
>
> Yes, it does: --sysroot>
What if my sysroot doesn't contains C++ library, or even if it does I may
still want to use libc++ shipped with the toolchain e.g. because the one
that's part of the sysroot doesn't support C++17?

I don't like the "build libc++ separately and then put it inside your
sysroot" solution for several reasons, most importantly because the sysroot
typically considered read-only e.g. I cannot modify Xcode's sysroot
replacing whatever libc++ is already there, but I can use libSystem.dylib
from Xcode's sysroot with my own libc++ version. In our case also we ship
the toolchain separately from the sysroot at different frequencies because
they come from different source and system libraries change far less often
than libc++.

What this means in practice is that when cross-compiling you have to do
something like:

clang++ --target=<arch>-<vendor>-<os>
--sysroot=/path/to/sysroot -stdlib=libc++ -nostdinc++
-I<libc++-install-prefix>/include/c++/v1
-L<libc++-install-prefix>/lib

This is even when the C++ library for your target is part of your Clang
toolchain so the driver arguably should know how to find it without you
having to duplicate the driver logic.

> What's currently missing is a standardized naming for a) where the
> sysroots live, and b) what they're named. Host libraries should
continue
> to live in $prefix/{include, lib, lib32, lib64} as appropriate, whereas
> target libraries should live in something like:
> $prefix/clang-runtimes/$triple/$multilib/{usr/include, usr/lib, etc.}
>
I don't care too much about what the path is going to be. However, it'd
be
nice if we could unify the paths between compiler-rt runtimes and libc++. I
always assumed that $prefix/lib/clang/$version was the path for runtimes
hence suggesting it.

>   In our
> > toolchain, we would like to build runtimes for all host and target
> > platforms we support, e.g. a single toolchain will have runtimes for
> > x86_64 and aarch64 Linux, Fuchsia and Windows. All you need to provide
> > is the target triple and the sysroot. While this is possible with
> > builtins, sanitizers and other compiler-rt runtimes, it's not
possible
> > with libc++, libc++abi, libunwind.
> >
> > Our proposal is to move both compiler-rt and libc++, libc++abi,
> > libunwind into a new location that would support cross-compilation and
> > unify the layout:
> >
> > headers: $prefix/lib/clang/$version/include(/$triple)(/c++/v1)
> > libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
>
> I don't think it's a good idea to tie all the runtimes to the
compiler
> like that. It makes sense for the builtins to live there since they are
> heavily coupled with the specific compiler version, but I'm not
> convinced libc++/libc++abi/libunwind should.
>
They may be less tied than sanitizers because they don't rely on compiler
instrumentation, but there's still some dependency, e.g. in order to use
coroutines in libc++ I need a version of Clang that has the appropriate
intrinsics.

I could build libc++ against an older version of Clang which will disable
features not supported by that Clang version, but that version is again
tied to that version of Clang.

> > This means that for compiler-rt, the main difference would be moving
the
> > runtime libraries to an target specific subdirectory rather than
> > including the architecture in the library name; for libc++, libc++abi,
> > libunwind, both headers and libraries will be moved to a new, target
> > specific location.
> >
> > In terms of implementation, we'll need to modify the Clang driver
to use
> > this location when looking for runtimes and C++ libraries and headers.
> > This should be a non-intrusive change: we'll modify the driver to
look
> > into the new location in addition to the existing ones, so if the new
> > path doesn't exist, the driver will simply fallback to the
existing
> > behavior. When this is done, we need to modify the CMake build to
> > install files into the new location.
> >
> > This layout would be only used when runtimes are built as part of the
> > llvm/runtimes tree or using LLVM_ENABLE_RUNTIMES in the monorepo
layout
> > (because this setup supports cross-compiling runtimes). When built as
> > part of LLVM or standalone, libc++, libc++abi, libunwind would still
> > install their files to $prefix/include and $prefix/lib as today.
> >
> > Once the overall scheme is agreed upon, we could also consider
> > de-duplicating C++ headers across targets, by moving shared headers
into
> > a common directory, with only varying subset in include/$triple.
>
> At the very least the __config_site headers cannot be de-duplicated.
>
I think it should be sufficient to have separate __config headers for
separate targets, that's really the only thing that differs.

> Beyond that, I strongly disagree with de-duplicating them in general
> because it will break --sysroot= unless you add a bunch of symlinks...
> which don't exist on every host platform.

I don't think you need symlinks, all you need is to put both include/c++/v1
and include/$triple/c++/v1 to your include paths, where the former contains
all the common headers while the latter contains the target specific
__config. This is exactly what GNU "multi-arch" layout does.

> >
> > To give an example, for x86_64 and aarch64 Linux, this would look
like:
> >
> > $prefix/lib/clang/6.0.0/include/sanitizer
> > $prefix/lib/clang/6.0.0/include/c++/v1
> > $prefix/lib/clang/6.0.0/include/x86_64-linux-gnu/c++/v1/__config
> > ...
> > $prefix/lib/clang/6.0.0/x86_64-linux-gnu/lib/libclang_rt.asan.so
> > <http://libclang_rt.asan.so>
> > $prefix/lib/clang/6.0.0/x86_64-linux-gnu/lib/libc++.so
> > ...
> > $prefix/lib/clang/6.0.0/aarch64-linux-gnu/lib/libclang_rt.asan.so
> > <http://libclang_rt.asan.so>
> > $prefix/lib/clang/6.0.0/aarch64-linux-gnu/lib/libc++.so
> > ...
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
>
> --
> Jon Roelofs
> jonathan at codesourcery.com
> CodeSourcery / Mentor Embedded / Siemens
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171219/afb043de/attachment-0001.html>