llvm dev - Dec 2015 - difference with autotools, cmake and ninja building methods

2015-12-01 1:20 GMT+08:00 Chris Bieneman <beanz at
apple.com>:
>
> On Nov 29, 2015, at 9:09 PM, 慕冬亮 <mudongliangabcd at gmail.com>
wrote:
>
> 2015-11-30 12:58 GMT+08:00 Chris Bieneman <beanz at apple.com>:
>
> The autotools build system is officially deprecated and will be removed in
a
> future release.
>
> CMake is the recommended configuration system, but it is only a
> configuration system. It generates build files for multiple different build
> systems. One of the most popular build systems is Ninja. You cannot build
> LLVM with Ninja without using CMake, but CMake doesn’t require Ninja. You
> can use CMake to generate Makefiles as well as Xcode and Visual Studio
> projects, and more.
>
> I often used cmake to compile llvm and clang. But I found a boring
> problem: when I update all the repositories - llvm,clang,compiler-rt
> and try to recompile, but I can't compile source code successfully
>
>
> Do you mean when you update the repositories you can’t compile llvm, clang,
> and compiler-rt? Do you have a specific error? Keep in mind the llvm,
clang,
> and compiler-rt repositories need to be kept in sync. You can’t update one
> without updating them all.
>
I always "svn update" three repositories. I don't have a specific
error.
> and
> it takes me much time to recompile.
>
>
> The clang compiler is a quite large source project, depending on your
> hardware it may take a while to compile. Ninja is better than any other
tool
> that I’m aware of at minimizing the work required during incremental
builds.
But I have a problem about compiling with Ninja. I did compilation in
Debian stable and testing.
There are always errors when I do "ninja" command.

FAILED: : && /usr/bin/c++   -fPIC -fvisibility-inlines-hidden -Wall -W
-Wno-unused-parameter -Wwrite-strings -Wcast-qual
-Wno-missing-field-initializers -pedantic -Wno-long-long
-Wno-maybe-uninitialized -Wnon-virtual-dtor -Wno-comment -std=c++11
-ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual
-fno-strict-aliasing  -Wl,-allow-shlib-undefined  -Wl,--export-dynamic
 -Wl,-O3 tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o
tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o
tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o  -o
bin/clang-3.8  lib/libLLVMAArch64CodeGen.a
lib/libLLVMAArch64AsmPrinter.a lib/libLLVMAArch64AsmParser.a
lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a
lib/libLLVMAArch64Disassembler.a lib/libLLVMAMDGPUCodeGen.a
lib/libLLVMAMDGPUAsmPrinter.a lib/libLLVMAMDGPUAsmParser.a
lib/libLLVMAMDGPUDesc.a lib/libLLVMAMDGPUInfo.a
lib/libLLVMARMCodeGen.a lib/libLLVMARMAsmPrinter.a
lib/libLLVMARMAsmParser.a lib/libLLVMARMDesc.a lib/libLLVMARMInfo.a
lib/libLLVMARMDisassembler.a lib/libLLVMBPFCodeGen.a
lib/libLLVMBPFAsmPrinter.a lib/libLLVMBPFDesc.a lib/libLLVMBPFInfo.a
lib/libLLVMCppBackendCodeGen.a lib/libLLVMCppBackendInfo.a
lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a
lib/libLLVMHexagonDesc.a lib/libLLVMHexagonInfo.a
lib/libLLVMHexagonDisassembler.a lib/libLLVMMipsCodeGen.a
lib/libLLVMMipsAsmPrinter.a lib/libLLVMMipsAsmParser.a
lib/libLLVMMipsDesc.a lib/libLLVMMipsInfo.a
lib/libLLVMMipsDisassembler.a lib/libLLVMMSP430CodeGen.a
lib/libLLVMMSP430AsmPrinter.a lib/libLLVMMSP430Desc.a
lib/libLLVMMSP430Info.a lib/libLLVMNVPTXCodeGen.a
lib/libLLVMNVPTXAsmPrinter.a lib/libLLVMNVPTXDesc.a
lib/libLLVMNVPTXInfo.a lib/libLLVMPowerPCCodeGen.a
lib/libLLVMPowerPCAsmPrinter.a lib/libLLVMPowerPCAsmParser.a
lib/libLLVMPowerPCDesc.a lib/libLLVMPowerPCInfo.a
lib/libLLVMPowerPCDisassembler.a lib/libLLVMSparcCodeGen.a
lib/libLLVMSparcAsmPrinter.a lib/libLLVMSparcAsmParser.a
lib/libLLVMSparcDesc.a lib/libLLVMSparcInfo.a
lib/libLLVMSparcDisassembler.a lib/libLLVMSystemZCodeGen.a
lib/libLLVMSystemZAsmPrinter.a lib/libLLVMSystemZAsmParser.a
lib/libLLVMSystemZDesc.a lib/libLLVMSystemZInfo.a
lib/libLLVMSystemZDisassembler.a lib/libLLVMX86CodeGen.a
lib/libLLVMX86AsmPrinter.a lib/libLLVMX86AsmParser.a
lib/libLLVMX86Desc.a lib/libLLVMX86Info.a lib/libLLVMX86Disassembler.a
lib/libLLVMXCoreCodeGen.a lib/libLLVMXCoreAsmPrinter.a
lib/libLLVMXCoreDesc.a lib/libLLVMXCoreInfo.a
lib/libLLVMXCoreDisassembler.a lib/libLLVMAnalysis.a
lib/libLLVMCodeGen.a lib/libLLVMCore.a lib/libLLVMipo.a
lib/libLLVMInstCombine.a lib/libLLVMInstrumentation.a lib/libLLVMMC.a
lib/libLLVMMCParser.a lib/libLLVMObjCARCOpts.a lib/libLLVMOption.a
lib/libLLVMScalarOpts.a lib/libLLVMSupport.a
lib/libLLVMTransformUtils.a lib/libLLVMVectorize.a lib/libclangBasic.a
lib/libclangCodeGen.a lib/libclangDriver.a lib/libclangFrontend.a
lib/libclangFrontendTool.a lib/libLLVMAArch64Desc.a
lib/libLLVMAArch64AsmPrinter.a lib/libLLVMAArch64Info.a
lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUAsmPrinter.a
lib/libLLVMAMDGPUUtils.a lib/libLLVMARMDesc.a
lib/libLLVMARMAsmPrinter.a lib/libLLVMARMInfo.a
lib/libLLVMBPFAsmPrinter.a lib/libLLVMHexagonDesc.a
lib/libLLVMHexagonInfo.a lib/libLLVMMipsAsmPrinter.a
lib/libLLVMMipsInfo.a lib/libLLVMMSP430AsmPrinter.a
lib/libLLVMNVPTXAsmPrinter.a lib/libLLVMPowerPCAsmPrinter.a
lib/libLLVMPowerPCInfo.a lib/libLLVMSparcAsmPrinter.a
lib/libLLVMSparcInfo.a lib/libLLVMSystemZDesc.a
lib/libLLVMSystemZAsmPrinter.a lib/libLLVMSystemZInfo.a
lib/libLLVMX86CodeGen.a lib/libLLVMX86Desc.a
lib/libLLVMX86AsmPrinter.a lib/libLLVMX86Utils.a lib/libLLVMX86Info.a
lib/libLLVMXCoreAsmPrinter.a lib/libLLVMAsmPrinter.a
lib/libLLVMSelectionDAG.a lib/libLLVMCodeGen.a lib/libLLVMXCoreInfo.a
lib/libLLVMMCDisassembler.a lib/libclangCodeGen.a lib/libLLVMipo.a
lib/libLLVMVectorize.a lib/libLLVMInstrumentation.a
lib/libLLVMObjCARCOpts.a lib/libLLVMScalarOpts.a
lib/libLLVMInstCombine.a lib/libLLVMTarget.a lib/libLLVMBitWriter.a
lib/libLLVMIRReader.a lib/libLLVMAsmParser.a lib/libLLVMLinker.a
lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a
lib/libLLVMProfileData.a lib/libLLVMObject.a
lib/libclangRewriteFrontend.a lib/libclangARCMigrate.a
lib/libclangStaticAnalyzerFrontend.a lib/libclangFrontend.a
lib/libclangDriver.a lib/libLLVMOption.a lib/libclangParse.a
lib/libLLVMMCParser.a lib/libclangSerialization.a
lib/libLLVMBitReader.a lib/libclangSema.a lib/libclangEdit.a
lib/libclangStaticAnalyzerCheckers.a lib/libclangStaticAnalyzerCore.a
lib/libclangAnalysis.a lib/libclangAST.a lib/libclangRewrite.a
lib/libclangLex.a lib/libclangBasic.a lib/libLLVMCore.a
lib/libLLVMMC.a lib/libLLVMSupport.a -lrt -ldl -ltinfo -lpthread -lm
-Wl,-rpath,"\$ORIGIN/../lib" && :
collect2: fatal error: ld terminated with signal 9 [Killed]
compilation terminated.
[2706/2983] Linking CXX executable bin/clang-check
ninja: build stopped: subcommand failed.
>
> -Chris
>
> Does anyone has advice about this
> problem?
>
> This page has the LLVM project’s documentation on using CMake:
> http://llvm.org/docs/CMake.html
>
> -Chris
> ile.
> On Nov 29, 2015, at 7:52 PM, 慕冬亮 via llvm-dev <llvm-dev at
lists.llvm.org>
> wrote:
>
>   When I see one book about llvm and choose the building method
> between autotools, cmake, and ninja building methods, I was confused.
> Is there any link about this content?
>   Thanks for reply.
>
> --
> My best regards to you.
>
>    No System Is Safe!
>    mudongliang
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
>
>
> --
> My best regards to you.
>
>     No System Is Safe!
>     mudongliang
>
>


-- 
My best regards to you.

     No System Is Safe!
     mudongliang

On 1 Dec 2015, at 10:03, 慕冬亮 via llvm-dev <llvm-dev at lists.llvm.org>
wrote:
> 
> collect2: fatal error: ld terminated with signal 9 [Killed]
> compilation terminated.
> [2706/2983] Linking CXX executable bin/clang-check
> ninja: build stopped: subcommand failed.
This looks like the linker is running out of memory.  This is a huge link job
and BFD linker will consume at least one GB of RAM, possibly far more depending
on your build config.  If you’re on a 32-bit platform, then you won’t be able to
do this with a debug build.

The big speedup if you’re using BFD ld is to enable shared library support. 
This is a particularly big win if you’re doing incremental builds a lot, because
the extra startup time overhead is likely to be far less than the 4-5 minutes of
extra time spent linking, even on a fast machine.

If you do want to do static linking, then your best bet is to do ninja -k 10,
which should let it skip over the link jobs that fail, but keep compiling the
sources, then ninja -j1, which will make it do the remaining steps one at a
time.  In general, the defaults for ninja work well for LLVM/Clang if you have
at least one GB of RAM per core.

On a modern laptop, you should be able to do a clean build in 5-10 minutes, but
BFD ld can be a bottleneck.  It would be nice if CMake had an option to stick
-fuse-ld=gold (or -fuse-ld=lld) in all of the correct places, but when I’ve
tried poking the link flags to add this myself I’ve found weird linking errors
that I haven’t had time to debug.

David

On 1 December 2015 at 10:17, David Chisnall via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> On 1 Dec 2015, at 10:03, 慕冬亮 via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>>
>> collect2: fatal error: ld terminated with signal 9 [Killed]
>> compilation terminated.
>> [2706/2983] Linking CXX executable bin/clang-check
>> ninja: build stopped: subcommand failed.
>
> This looks like the linker is running out of memory.  This is a huge link
job and BFD linker will consume at least one GB of RAM, possibly far more
depending on your build config.  If you’re on a 32-bit platform, then you won’t
be able to do this with a debug build.
>
> The big speedup if you’re using BFD ld is to enable shared library support.
This is a particularly big win if you’re doing incremental builds a lot, because
the extra startup time overhead is likely to be far less than the 4-5 minutes of
extra time spent linking, even on a fast machine.
>
> If you do want to do static linking, then your best bet is to do ninja -k
10, which should let it skip over the link jobs that fail, but keep compiling
the sources, then ninja -j1, which will make it do the remaining steps one at a
time.  In general, the defaults for ninja work well for LLVM/Clang if you have
at least one GB of RAM per core.
>
> On a modern laptop, you should be able to do a clean build in 5-10 minutes,
but BFD ld can be a bottleneck.  It would be nice if CMake had an option to
stick -fuse-ld=gold (or -fuse-ld=lld) in all of the correct places, but when
I’ve tried poking the link flags to add this myself I’ve found weird linking
errors that I haven’t had time to debug.
I seem to have had success with       -DCMAKE_C_FLAGS="-fuse-ld=gold"
-DCMAKE_CXX_FLAGS="-fuse-ld=gold", but YMMV.

Alex

David Chisnall via llvm-dev <llvm-dev at lists.llvm.org>
writes:
> On 1 Dec 2015, at 10:03, 慕冬亮 via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>> 
>> collect2: fatal error: ld terminated with signal 9 [Killed]
>> compilation terminated.
>> [2706/2983] Linking CXX executable bin/clang-check
>> ninja: build stopped: subcommand failed.
>
> This looks like the linker is running out of memory.  This is a huge
> link job and BFD linker will consume at least one GB of RAM, possibly
> far more depending on your build config.  If you’re on a 32-bit
> platform, then you won’t be able to do this with a debug build.
>
> The big speedup if you’re using BFD ld is to enable shared library
> support.  This is a particularly big win if you’re doing incremental
> builds a lot, because the extra startup time overhead is likely to be
> far less than the 4-5 minutes of extra time spent linking, even on a
> fast machine.
>
> If you do want to do static linking, then your best bet is to do ninja
> -k 10, which should let it skip over the link jobs that fail, but keep
> compiling the sources, then ninja -j1, which will make it do the
> remaining steps one at a time.  In general, the defaults for ninja
> work well for LLVM/Clang if you have at least one GB of RAM per core.
There's a cmake parameter `-DLLVM_PARALLEL_LINK_JOBS=1` that will limit
the number of concurrent link jobs. That makes for a better experience
than running ninja twice :)
> On a modern laptop, you should be able to do a clean build in 5-10
> minutes, but BFD ld can be a bottleneck.  It would be nice if CMake
> had an option to stick -fuse-ld=gold (or -fuse-ld=lld) in all of the
> correct places, but when I’ve tried poking the link flags to add this
> myself I’ve found weird linking errors that I haven’t had time to
> debug.
>
> David
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On Tue, Dec 1, 2015 at 2:17 AM, David Chisnall via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 1 Dec 2015, at 10:03, 慕冬亮 via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> >
> > collect2: fatal error: ld terminated with signal 9 [Killed]
> > compilation terminated.
> > [2706/2983] Linking CXX executable bin/clang-check
> > ninja: build stopped: subcommand failed.
>
> This looks like the linker is running out of memory.  This is a huge link
> job and BFD linker will consume at least one GB of RAM, possibly far more
> depending on your build config.  If you’re on a 32-bit platform, then you
> won’t be able to do this with a debug build.
>
With Fission (AKA Split Dwarf) you might be able to manage it (check for
the BLAH_BLAH_SPLIT_DWARF option in the CMake config)

> The big speedup if you’re using BFD ld is to enable shared library
> support.  This is a particularly big win if you’re doing incremental builds
> a lot, because the extra startup time overhead is likely to be far less
> than the 4-5 minutes of extra time spent linking, even on a fast machine.
>
If you plan to run the regression tests with that build, you'll probably
lose the time you gain by linking - the startup of all the llvm tools for
all those little tests adds up. A "make check-all" of clang/llvm, at
least
on my machine (32 core, 32GB of RAM, running of an SSD) is slower (even if
you just touch one cc file - causing maximal linking, minimal compiling,
etc) with a shared library build, unfortunately.

>
> If you do want to do static linking, then your best bet is to do ninja -k
> 10, which should let it skip over the link jobs that fail, but keep
> compiling the sources, then ninja -j1, which will make it do the remaining
> steps one at a time.  In general, the defaults for ninja work well for
> LLVM/Clang if you have at least one GB of RAM per core.
>
I think there's a ninja flag for the number of link jobs you want to run in
parallel, separate from the usual parallelism.

> On a modern laptop, you should be able to do a clean build in 5-10
> minutes, but BFD ld can be a bottleneck.  It would be nice if CMake had an
> option to stick -fuse-ld=gold (or -fuse-ld=lld) in all of the correct
> places, but when I’ve tried poking the link flags to add this myself I’ve
> found weird linking errors that I haven’t had time to debug.
>
> David
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151201/04ad7e8b/attachment.html>