llvm dev - Mar 2019 - Trying to create a pure LLVM toolchain on musl based distribution

Hello,

I'm trying to create a pure LLVM toolchain (that will not depend on GNU 
and produce GNU-free code too) on a musl based distribution.

For now, I use gcc to bootstrap and build all LLVM components. I do it 
individually because I was running out of space and memory trying to 
build all using LLVM_ENABLE_PROJECTS. Also, I don't want to create a 
all-in-one package. Then, once I'm able to build program with clang, 
I'll rebuild all using clang instead of gcc.

# LLVM

I've built LLVM using the following configuration:

     -DCMAKE_BUILD_TYPE=Release
     -DBUILD_SHARED_LIBS=On
     -DCMAKE_INSTALL_PREFIX=/usr
     -DLLVM_HOST_TRIPLE=x86_64-linux-musl
     -DLLVM_ENABLE_RTTI=On

# compiler-rt

I've needed to disable xray/sanitizers otherwise it didn't build on
musl.

     -DCMAKE_BUILD_TYPE=Release
     -DCMAKE_INSTALL_PREFIX=/usr
     -DCOMPILER_RT_BUILD_BUILTINS=On
     -DCOMPILER_RT_BUILD_SANITIZERS=Off
     -DCOMPILER_RT_BUILD_XRAY=Off
     -DCOMPILER_RT_USE_BUILTINS_LIBRARY=On

# libc++

     -DBUILD_SHARED_LIBS=On
     -DCMAKE_BUILD_TYPE=Release
     -DCMAKE_INSTALL_PREFIX=/usr
     -DLIBCXX_HAS_MUSL_LIBC=On
     -DLIBCXX_HAS_GCC_S_LIB=Off
     -DLIBCXX_ENABLE_STATIC=Off

# libc++abi

     -DBUILD_SHARED_LIBS=On
     -DCMAKE_BUILD_TYPE=Release
     -DCMAKE_INSTALL_PREFIX=/usr
     -DLIBCXXABI_ENABLE_STATIC=Off

# libunwind

     -DBUILD_SHARED_LIBS=On
     -DCMAKE_BUILD_TYPE=Release
     -DCMAKE_INSTALL_PREFIX=/usr
     -DLIBUNWIND_ENABLE_STATIC=Off
     -DLIBUNWIND_USE_COMPILER_RT=On

# clang

And finally clang:

     -DBUILD_SHARED_LIBS=On
     -DCMAKE_BUILD_TYPE=Release
     -DCMAKE_INSTALL_PREFIX=/usr
     -DCLANG_DEFAULT_CXX_STDLIB=libc++
     -DCLANG_DEFAULT_LINKER=lld
     -DCLANG_DEFAULT_RTLIB=compiler-rt
     -DCLANG_VENDOR="Vanilla Linux"

Everything built fine, now the problem is the location of compiler-rt 
libraries which are not the correct place where clang/ld.lld searches. 
As the following output shows:

clang -v /tmp/test.c
Vanilla Linux clang version 8.0.0 (tags/RELEASE_800/final) (based on 
LLVM 8.0.0)
Target: x86_64-unknown-linux-musl
Thread model: posix
InstalledDir: /usr/bin
  "/usr/bin/clang-8" -cc1 -triple x86_64-unknown-linux-musl -emit-obj 
-mrelax-all -disable-free -disable-llvm-verifier -discard-value-names 
-main-file-name test.c -mrelocation-model pic -pic-level 2 -pic-is-pie 
-mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose 
-mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu 
x86-64 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir 
/usr/lib/clang/8.0.0 -internal-isystem /usr/local/include 
-internal-isystem /usr/lib/clang/8.0.0/include -internal-externc-isystem 
/include -internal-externc-isystem /usr/include -fdebug-compilation-dir 
/usr/src/vanilla -ferror-limit 19 -fmessage-length 184 
-fobjc-runtime=gcc -fdiagnostics-show-option -o /tmp/test-1d99ce.o -x c 
/tmp/test.c -faddrsig
clang -cc1 version 8.0.0 based upon LLVM 8.0.0 default target 
x86_64-linux-musl
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
  /usr/local/include
  /usr/lib/clang/8.0.0/include
  /usr/include
End of search list.
  "/usr/bin/ld.lld" -pie --eh-frame-hdr -m elf_x86_64 -dynamic-linker 
/lib/ld-musl-x86_64.so.1 -o a.out /usr/bin/../lib/Scrt1.o 
/usr/bin/../lib/crti.o crtbeginS.o -L/usr/bin/../lib -L/lib -L/usr/lib 
/tmp/test-1d99ce.o 
/usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a -lc 
/usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a crtendS.o 
/usr/bin/../lib/crtn.o
ld.lld: error: cannot open crtbeginS.o: No such file or directory
ld.lld: error: cannot open 
/usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a: No such 
file or directory
ld.lld: error: cannot open 
/usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a: No such 
file or directory
ld.lld: error: cannot open crtendS.o: No such file or directory
clang-8: error: linker command failed with exit code 1 (use -v to see 
invocation)

compiler-rt installed files under the following directories hierarchy 
instead:

usr/lib/linux/libclang_rt.builtins-x86_64.a
usr/lib/linux/libclang_rt.fuzzer-x86_64.a
usr/lib/linux/libclang_rt.fuzzer_no_main-x86_64.a
usr/lib/linux/libclang_rt.profile-x86_64.a

What did I misconfigured? Is it clang/lld or compiler-rt? Also do I need 
to do something else regarding crtbeginS.o and crtendS.o that are not 
provided by musl?

Regards,

-- 
David

Hello David,

I don't know much about the specifics of Musl, so I'm responding
generally.

As I understand it, clang expects to find the compiler-rt libraries
relative to the resource directory, which you can find out the
location of with clang --print-resource-dir . By default it is
lib/clang/9.0.0 assuming you are building from master. I think that
-DCMAKE_INSTALL_PREFIX=/usr has broken that assumption. I think that
you would either need to take out the CMAKE_INSTALL_PREFIX or change
the location of the resource directory, which I think that you can
alter at build time.

I don't think that there is a way of preventing clang adding
crtbeginS.o and crtendS.o without --nostdlib or --nostartfiles but
using these options will also not add the other crt*.o files that you
may be expecting.

There is a line in lib/Driver/Toolchains/Gnu.cpp which the Linux driver uses:
const bool HasCRTBeginEndFiles       ToolChain.getTriple().hasEnvironment() ||
      (ToolChain.getTriple().getVendor() != llvm::Triple::MipsTechnologies);
It looks like Musl may need to be included there?

Peter

On Mon, 25 Mar 2019 at 12:52, David Demelier via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> Hello,
>
> I'm trying to create a pure LLVM toolchain (that will not depend on GNU
> and produce GNU-free code too) on a musl based distribution.
>
> For now, I use gcc to bootstrap and build all LLVM components. I do it
> individually because I was running out of space and memory trying to
> build all using LLVM_ENABLE_PROJECTS. Also, I don't want to create a
> all-in-one package. Then, once I'm able to build program with clang,
> I'll rebuild all using clang instead of gcc.
>
> # LLVM
>
> I've built LLVM using the following configuration:
>
>      -DCMAKE_BUILD_TYPE=Release
>      -DBUILD_SHARED_LIBS=On
>      -DCMAKE_INSTALL_PREFIX=/usr
>      -DLLVM_HOST_TRIPLE=x86_64-linux-musl
>      -DLLVM_ENABLE_RTTI=On
>
> # compiler-rt
>
> I've needed to disable xray/sanitizers otherwise it didn't build on
musl.
>
>      -DCMAKE_BUILD_TYPE=Release
>      -DCMAKE_INSTALL_PREFIX=/usr
>      -DCOMPILER_RT_BUILD_BUILTINS=On
>      -DCOMPILER_RT_BUILD_SANITIZERS=Off
>      -DCOMPILER_RT_BUILD_XRAY=Off
>      -DCOMPILER_RT_USE_BUILTINS_LIBRARY=On
>
> # libc++
>
>      -DBUILD_SHARED_LIBS=On
>      -DCMAKE_BUILD_TYPE=Release
>      -DCMAKE_INSTALL_PREFIX=/usr
>      -DLIBCXX_HAS_MUSL_LIBC=On
>      -DLIBCXX_HAS_GCC_S_LIB=Off
>      -DLIBCXX_ENABLE_STATIC=Off
>
> # libc++abi
>
>      -DBUILD_SHARED_LIBS=On
>      -DCMAKE_BUILD_TYPE=Release
>      -DCMAKE_INSTALL_PREFIX=/usr
>      -DLIBCXXABI_ENABLE_STATIC=Off
>
> # libunwind
>
>      -DBUILD_SHARED_LIBS=On
>      -DCMAKE_BUILD_TYPE=Release
>      -DCMAKE_INSTALL_PREFIX=/usr
>      -DLIBUNWIND_ENABLE_STATIC=Off
>      -DLIBUNWIND_USE_COMPILER_RT=On
>
> # clang
>
> And finally clang:
>
>      -DBUILD_SHARED_LIBS=On
>      -DCMAKE_BUILD_TYPE=Release
>      -DCMAKE_INSTALL_PREFIX=/usr
>      -DCLANG_DEFAULT_CXX_STDLIB=libc++
>      -DCLANG_DEFAULT_LINKER=lld
>      -DCLANG_DEFAULT_RTLIB=compiler-rt
>      -DCLANG_VENDOR="Vanilla Linux"
>
> Everything built fine, now the problem is the location of compiler-rt
> libraries which are not the correct place where clang/ld.lld searches.
> As the following output shows:
>
> clang -v /tmp/test.c
> Vanilla Linux clang version 8.0.0 (tags/RELEASE_800/final) (based on
> LLVM 8.0.0)
> Target: x86_64-unknown-linux-musl
> Thread model: posix
> InstalledDir: /usr/bin
>   "/usr/bin/clang-8" -cc1 -triple x86_64-unknown-linux-musl
-emit-obj
> -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names
> -main-file-name test.c -mrelocation-model pic -pic-level 2 -pic-is-pie
> -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose
> -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu
> x86-64 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir
> /usr/lib/clang/8.0.0 -internal-isystem /usr/local/include
> -internal-isystem /usr/lib/clang/8.0.0/include -internal-externc-isystem
> /include -internal-externc-isystem /usr/include -fdebug-compilation-dir
> /usr/src/vanilla -ferror-limit 19 -fmessage-length 184
> -fobjc-runtime=gcc -fdiagnostics-show-option -o /tmp/test-1d99ce.o -x c
> /tmp/test.c -faddrsig
> clang -cc1 version 8.0.0 based upon LLVM 8.0.0 default target
> x86_64-linux-musl
> ignoring nonexistent directory "/include"
> #include "..." search starts here:
> #include <...> search starts here:
>   /usr/local/include
>   /usr/lib/clang/8.0.0/include
>   /usr/include
> End of search list.
>   "/usr/bin/ld.lld" -pie --eh-frame-hdr -m elf_x86_64
-dynamic-linker
> /lib/ld-musl-x86_64.so.1 -o a.out /usr/bin/../lib/Scrt1.o
> /usr/bin/../lib/crti.o crtbeginS.o -L/usr/bin/../lib -L/lib -L/usr/lib
> /tmp/test-1d99ce.o
> /usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a -lc
> /usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a crtendS.o
> /usr/bin/../lib/crtn.o
> ld.lld: error: cannot open crtbeginS.o: No such file or directory
> ld.lld: error: cannot open
> /usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a: No such
> file or directory
> ld.lld: error: cannot open
> /usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a: No such
> file or directory
> ld.lld: error: cannot open crtendS.o: No such file or directory
> clang-8: error: linker command failed with exit code 1 (use -v to see
> invocation)
>
> compiler-rt installed files under the following directories hierarchy
> instead:
>
> usr/lib/linux/libclang_rt.builtins-x86_64.a
> usr/lib/linux/libclang_rt.fuzzer-x86_64.a
> usr/lib/linux/libclang_rt.fuzzer_no_main-x86_64.a
> usr/lib/linux/libclang_rt.profile-x86_64.a
>
> What did I misconfigured? Is it clang/lld or compiler-rt? Also do I need
> to do something else regarding crtbeginS.o and crtendS.o that are not
> provided by musl?
>
> Regards,
>
> --
> David
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Le 25/03/2019 à 14:41, Peter Smith a écrit :
> Hello David,
> 
> I don't know much about the specifics of Musl, so I'm responding
generally.
> 
> As I understand it, clang expects to find the compiler-rt libraries
> relative to the resource directory, which you can find out the
> location of with clang --print-resource-dir . By default it is
> lib/clang/9.0.0 assuming you are building from master. I think that
> -DCMAKE_INSTALL_PREFIX=/usr has broken that assumption. I think that
> you would either need to take out the CMAKE_INSTALL_PREFIX or change
> the location of the resource directory, which I think that you can
> alter at build time.
> 
You're right, I've just checked both packages in Alpine and Arch and 
they indeed move the /usr/lib/linux directory under the clang's one. 
I've fixed that it worked, now I still have to find the solution 
regarding crtbeginS.so and crtendS.so.

Thanks :)
> I don't think that there is a way of preventing clang adding
> crtbeginS.o and crtendS.o without --nostdlib or --nostartfiles but
> using these options will also not add the other crt*.o files that you
> may be expecting.
> 
> There is a line in lib/Driver/Toolchains/Gnu.cpp which the Linux driver
uses:
> const bool HasCRTBeginEndFiles >       
ToolChain.getTriple().hasEnvironment() ||
>        (ToolChain.getTriple().getVendor() !=
llvm::Triple::MipsTechnologies);
> It looks like Musl may need to be included there?
I'll have a look, I don't know much the lowest part of the toolchain 
build process so I don't understand very much what's going on there. 
Hopefully someone that has better low-level knowledges and use musl will 
be able to guide me for this.

What can I tell is that musl provide those files:

usr/lib/Scrt1.o
usr/lib/crt1.o
usr/lib/crti.o
usr/lib/crtn.o
usr/lib/ld-musl-x86_64.so.1

Regards,

-- 
David

On 3/25/19 8:52 AM, David Demelier via llvm-dev wrote:
> Hello,
> 
> I'm trying to create a pure LLVM toolchain (that will not depend on GNU
> and produce GNU-free code too) on a musl based distribution.
> 
> # LLVM
> # compiler-rt
> # libc++
> # libc++abi
> # libunwind
> # clang
David,

You may be interested to know that Zig (https://ziglang.org/) ships with
such a toolchain already. It cross compiles components (musl, libunwind,
etc) for the target lazily from source, and supports these libc targets:

  aarch64_be-linux-gnu
  aarch64_be-linux-musl
  aarch64-linux-gnu
  aarch64-linux-musleabi
  armeb-linux-gnueabi
  armeb-linux-gnueabihf
  armeb-linux-musleabi
  armeb-linux-musleabihf
  arm-linux-gnueabi
  arm-linux-gnueabihf
  arm-linux-musleabi
  arm-linux-musleabihf
  i386-linux-gnu
  i386-linux-musl
  mips64el-linux-gnuabi64
  mips64el-linux-gnuabin32
  mips64el-linux-musl
  mips64-linux-gnuabi64
  mips64-linux-gnuabin32
  mips64-linux-musl
  mipsel-linux-gnu
  mipsel-linux-musl
  mips-linux-gnu
  mips-linux-musl
  powerpc64le-linux-gnu
  powerpc64le-linux-musl
  powerpc64-linux-gnu
  powerpc64-linux-musl
  powerpc-linux-gnu
  powerpc-linux-musl
  riscv32-linux-musl
  riscv64-linux-gnu
  riscv64-linux-musl
  s390x-linux-gnu
  s390x-linux-musl
  sparc-linux-gnu
  sparcv9-linux-gnu
  x86_64-linux-gnu
  x86_64-linux-gnux32
  x86_64-linux-musl

Here's an example of using Zig to compile hello-world.c for
aarch64-linux-gnu:

andy at xps ~/m/z/hello> cat hello.c
#include <stdio.h>

int main(int argc, char **argv) {
    printf("what up dawg\n");
    return 0;
}
andy at xps ~/m/z/hello> zig build-exe --c-source hello.c --library c
-target aarch64v8-linux-gnu
andy at xps ~/m/z/hello> file hello
hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV),
dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for
GNU/Linux 2.0.0, with debug_info, not stripped

And now for x86_64-linux-musl:

andy at xps ~/m/z/hello> zig build-exe --c-source hello.c --library c
-target x86_64-linux-musl
andy at xps ~/m/z/hello> ./hello
what up dawg
andy at xps ~/m/z/hello> ldd ./hello
	not a dynamic executable

This works on any OS including Windows and macOS since it has no system
dependencies.

This feature is new and not well tested yet, but if it piques your
interest, you can play with this functionality using pre-built binaries
available on https://ziglang.org/download/ or you can build Zig master
branch from source.

Regards,
Andrew

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190325/64be6724/attachment.sig>