Tom Stellard via llvm-dev
2020-Apr-01 21:10 UTC
[llvm-dev] LLD issue on a massively parallel build machine
On 04/01/2020 01:47 PM, Itaru Kitayama via llvm-dev wrote:> Thanks for the heads up the supercomputer > Is down for maintenance this week. > I’ll give it a try when it gets back. >This doesn't look to me like it's necessarily an lld issue. Trying to build all the sub-projects without limiting the number of ninja jobs is almost guaranteed to run out of memory on a machine like this with such a low memory:core ratio. -Tom> On Thu, Apr 2, 2020 at 1:49 Fangrui Song <maskray at google.com <mailto:maskray at google.com>> wrote: > > On 2020-03-29, Nemanja Ivanovic via llvm-dev wrote: > >Glad you got it working. > >My suggestion about LLVM_ENABLE_THREADS didn't work because you didn't > >apply it when building the build linker. > > > >When you don't have the ability to rebuild the build compiler, this doesn't > >apply. In those cases, I end up doing a dirty hack where I use a wrapper > >script with something like: > > > ><path-to-build-compiler> "$@" -Wl,--no-threads > > > >To invoke the build compiler. That way you still get full parallelism while > >compiling and linking, but you prevent LLD from using threads. > > > https://reviews.llvm.org/D76885 (committed yesterday) changed > --no-threads to --threads={1,2,...} (--no-threads is used rarely (GNU ld > does not support it) so I'd rather not keep the option for > compatibility). > > You may try -Wl,--threads=1 > > >On Sat., Mar. 28, 2020, 4:19 p.m. Itaru Kitayama, <itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> > >wrote: > > > >> Good news, I was able to use up to 37 cores for LLVM build with LLD. > >> The build speed, did not measure precisely though, is comparable to the > >> build with GNU ld case. > >> > >> Thank you all for your help! > >> > >> On Sun, Mar 29, 2020 at 5:04 AM Alex Brachet-Mialot < > >> alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>> wrote: > >> > >>> Yes unfortunately that would limit you to 4 cores. > >>> > >>> There’s no easy way to use lld with —no-threads with our build system at > >>> the moment unfortunately. > >>> > >>> I also had just been using ld.gold, but will switch now that numactl > >>> works for me. > >>> > >>> On Sat, Mar 28, 2020 at 3:58 PM Itaru Kitayama <itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> > >>> wrote: > >>> > >>>> That is slowing down the build visibly, for the speed I should stick > >>>> with ld at the moment. > >>>> > >>>> > >>>> On Sun, Mar 29, 2020 at 4:42 AM Alexandre Ganea < > >>>> alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com>> wrote: > >>>> > >>>>> Would `taskset -c 0-3 ninja check-all -j4` work? > >>>>> > >>>>> > >>>>> > >>>>> *De :* Itaru Kitayama <itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> > >>>>> *Envoyé :* March 28, 2020 3:37 PM > >>>>> *À :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>> > >>>>> *Cc :* Alexandre Ganea <alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com>>; LLVM Dev < > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>; Nemanja Ivanovic <nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com>> > >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel build > >>>>> machine > >>>>> > >>>>> > >>>>> > >>>>> $ free -g > >>>>> total used free shared buff/cache > >>>>> available > >>>>> Mem: 376 149 20 1 207 > >>>>> 225 > >>>>> Swap: 3 0 3 > >>>>> > >>>>> > >>>>> > >>>>> Too small swap size for a 72-core login machine? > >>>>> > >>>>> > >>>>> > >>>>> On Sun, Mar 29, 2020 at 4:28 AM Alex Brachet-Mialot < > >>>>> alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>> wrote: > >>>>> > >>>>> Enable threads is for building llvm with or without thread support. To > >>>>> my knowledge it has no effect on the build system. > >>>>> > >>>>> > >>>>> > >>>>> Like was discussed above you could try limiting the number of > >>>>> processes created (assuming your lld was after that commit) with numactl or > >>>>> taskset. > >>>>> > >>>>> > >>>>> > >>>>> In your case though it looked like you were running out of memory. You > >>>>> could try increasing your swap space? > >>>>> > >>>>> > >>>>> > >>>>> Alex > >>>>> > >>>>> > >>>>> > >>>>> On Sat, Mar 28, 2020 at 3:15 PM Itaru Kitayama < > >>>>> itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> wrote: > >>>>> > >>>>> Hi, > >>>>> > >>>>> My configuration is below: > >>>>> > >>>>> cmake -GNinja -DLLVM_ENABLE_LLD=ON -DLLVM_ENABLE_THREADS=OFF > >>>>> -DLLVM_ENABLE_LIBCXX=ON -DCMAKE_BUILD_TYPE=Release > >>>>> -DGCC_INSTALL_PREFIX=/home/usr4/c74014i/opt/gcc-7.5.0/ > >>>>> -DLIBOMPTARGET_ENABLE_DEBUG=ON > >>>>> -DCMAKE_INSTALL_PREFIX=$HOME/opt/clang/${today} -DCMAKE_C_COMPILER=clang > >>>>> -DCMAKE_C_FLAGS="" -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_FLAGS="" > >>>>> -DLLVM_ENABLE_PROJECTS="clang;openmp;lld;libcxx;libcxxabi;compiler-rt;libunwind" > >>>>> -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_60 > >>>>> -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=60 -DLLVM_TARGETS_TO_BUILD="all" > >>>>> /tmp/llvm-project/llvm > >>>>> > >>>>> > >>>>> > >>>>> but did not help my case: > >>>>> > >>>>> > >>>>> > >>>>> [...] > >>>>> > >>>>> [4802/5201] Linking CXX executable bin/lld > >>>>> FAILED: bin/lld > >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++ -stdlib=libc++ > >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time > >>>>> -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter > >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic > >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default > >>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor > >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections > >>>>> -O3 -DNDEBUG -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics > >>>>> -Wl,-allow-shlib-undefined -Wl,--export-dynamic -Wl,-O3 > >>>>> tools/lld/tools/lld/CMakeFiles/lld.dir/lld.cpp.o -o bin/lld > >>>>> -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.a -lpthread > >>>>> lib/liblldCommon.a lib/liblldCOFF.a lib/liblldDriver.a lib/liblldELF.a > >>>>> lib/liblldMinGW.a lib/liblldWasm.a lib/liblldMachO.a > >>>>> lib/liblldReaderWriter.a lib/liblldYAML.a lib/liblldCore.a > >>>>> lib/liblldCOFF.a lib/libLLVMDebugInfoPDB.a lib/libLLVMLibDriver.a > >>>>> lib/libLLVMWindowsManifest.a /usr/lib64/libxml2.so lib/liblldCommon.a > >>>>> lib/libLLVMOption.a lib/libLLVMAArch64CodeGen.a > >>>>> lib/libLLVMAArch64AsmParser.a lib/libLLVMAArch64Disassembler.a > >>>>> lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUCodeGen.a > >>>>> lib/libLLVMMIRParser.a lib/libLLVMAMDGPUAsmParser.a > >>>>> lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUDesc.a > >>>>> lib/libLLVMAMDGPUInfo.a lib/libLLVMAMDGPUUtils.a lib/libLLVMARMCodeGen.a > >>>>> lib/libLLVMARMAsmParser.a lib/libLLVMARMDisassembler.a > >>>>> lib/libLLVMARMDesc.a lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a > >>>>> lib/libLLVMAVRCodeGen.a lib/libLLVMAVRAsmParser.a lib/libLLVMAVRDesc.a > >>>>> lib/libLLVMAVRDisassembler.a lib/libLLVMAVRInfo.a > >>>>> lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a lib/libLLVMBPFDesc.a > >>>>> lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a > >>>>> lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a > >>>>> lib/libLLVMHexagonDisassembler.a lib/libLLVMHexagonDesc.a > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a > >>>>> lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiDisassembler.a > >>>>> lib/libLLVMLanaiDesc.a lib/libLLVMLanaiInfo.a lib/libLLVMMipsCodeGen.a > >>>>> lib/libLLVMMipsAsmParser.a lib/libLLVMMipsDesc.a > >>>>> lib/libLLVMMipsDisassembler.a lib/libLLVMMipsInfo.a > >>>>> lib/libLLVMMSP430CodeGen.a lib/libLLVMMSP430AsmParser.a > >>>>> lib/libLLVMMSP430Desc.a lib/libLLVMMSP430Disassembler.a > >>>>> lib/libLLVMMSP430Info.a lib/libLLVMNVPTXCodeGen.a lib/libLLVMNVPTXDesc.a > >>>>> lib/libLLVMNVPTXInfo.a lib/libLLVMPowerPCCodeGen.a > >>>>> lib/libLLVMPowerPCAsmParser.a lib/libLLVMPowerPCDesc.a > >>>>> lib/libLLVMPowerPCDisassembler.a lib/libLLVMPowerPCInfo.a > >>>>> lib/libLLVMRISCVCodeGen.a lib/libLLVMRISCVAsmParser.a > >>>>> lib/libLLVMRISCVDesc.a lib/libLLVMRISCVDisassembler.a > >>>>> lib/libLLVMRISCVInfo.a lib/libLLVMRISCVUtils.a lib/libLLVMSparcCodeGen.a > >>>>> lib/libLLVMSparcAsmParser.a lib/libLLVMSparcDesc.a > >>>>> lib/libLLVMSparcDisassembler.a lib/libLLVMSparcInfo.a > >>>>> lib/libLLVMSystemZCodeGen.a lib/libLLVMSystemZAsmParser.a > >>>>> lib/libLLVMSystemZDisassembler.a lib/libLLVMSystemZDesc.a > >>>>> lib/libLLVMSystemZInfo.a lib/libLLVMWebAssemblyCodeGen.a > >>>>> lib/libLLVMWebAssemblyAsmParser.a lib/libLLVMWebAssemblyDisassembler.a > >>>>> lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a > >>>>> lib/libLLVMX86CodeGen.a lib/libLLVMCFGuard.a lib/libLLVMGlobalISel.a > >>>>> lib/libLLVMX86AsmParser.a lib/libLLVMX86Desc.a > >>>>> lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a lib/libLLVMX86Utils.a > >>>>> lib/libLLVMXCoreCodeGen.a lib/libLLVMAsmPrinter.a > >>>>> lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a > >>>>> lib/libLLVMXCoreDesc.a lib/libLLVMXCoreDisassembler.a > >>>>> lib/libLLVMMCDisassembler.a lib/libLLVMXCoreInfo.a lib/libLLVMLTO.a > >>>>> lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a lib/libLLVMCodeGen.a > >>>>> lib/libLLVMTarget.a lib/libLLVMCoroutines.a lib/libLLVMipo.a > >>>>> lib/libLLVMBitWriter.a lib/libLLVMScalarOpts.a lib/libLLVMVectorize.a > >>>>> lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a > >>>>> lib/libLLVMLinker.a lib/libLLVMFrontendOpenMP.a lib/libLLVMIRReader.a > >>>>> lib/libLLVMAsmParser.a lib/libLLVMInstrumentation.a > >>>>> lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a lib/libLLVMObject.a > >>>>> lib/libLLVMBitReader.a lib/libLLVMMCParser.a lib/libLLVMMC.a > >>>>> lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a > >>>>> lib/libLLVMTextAPI.a lib/libLLVMProfileData.a lib/libLLVMCore.a > >>>>> lib/libLLVMBinaryFormat.a lib/libLLVMRemarks.a > >>>>> lib/libLLVMBitstreamReader.a lib/libLLVMSupport.a -lz -lrt -ldl > >>>>> -ltinfo -lpthread -lm lib/libLLVMDemangle.a && cd > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E > >>>>> create_symlink lld /tmp/build/202003290405/./bin/lld-link && cd > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E > >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld.lld && cd > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E > >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld64.lld && cd > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E > >>>>> create_symlink lld /tmp/build/202003290405/./bin/wasm-ld > >>>>> terminating with uncaught exception of type std::__1::system_error: > >>>>> thread constructor failed: Resource temporarily unavailable > >>>>> PLEASE submit a bug report to https://bugs.llvm.org/ and include the > >>>>> crash backtrace. > >>>>> #0 0x00000000007924f4 PrintStackTraceSignalHandler(void*) > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7924f4) > >>>>> #1 0x00000000007902ce llvm::sys::RunSignalHandlers() > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7902ce) > >>>>> #2 0x0000000000792bd5 SignalHandler(int) > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x792bd5) > >>>>> #3 0x00007fddd54bb630 __restore_rt (/lib64/libpthread.so.0+0xf630) > >>>>> #4 0x00007fddd3b40377 raise > >>>>> /usr/src/debug/glibc-2.17-c758a686/signal/../nptl/sysdeps/unix/sysv/linux/raise.c:55:0 > >>>>> #5 0x00007fddd3b41a68 abort > >>>>> /usr/src/debug/glibc-2.17-c758a686/stdlib/abort.c:92:0 > >>>>> #6 0x00007fddd411ae8b > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2be8b) > >>>>> #7 0x00007fddd40fdb9f demangling_terminate_handler() > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0xeb9f) > >>>>> #8 0x00007fddd411a0a3 std::__terminate(void (*)()) > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2b0a3) > >>>>> #9 0x00007fddd411ce06 > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2de06) > >>>>> #10 0x00007fddd411cd9f > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2dd9f) > >>>>> #11 0x00007fddd43bded4 std::__1::__throw_system_error(int, char const*) > >>>>> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++.so.1+0x90ed4) > >>>>> #12 0x0000000002a7a316 void* > >>>>> std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, > >>>>> std::__1::default_delete<std::__1::__thread_struct> >, > >>>>> llvm::parallel::detail::(anonymous > >>>>> namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::'lambda'()> > >>>>> >(void*) (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x2a7a316) > >>>>> #13 0x00007fddd54b3ea5 start_thread > >>>>> /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307:0 > >>>>> #14 0x00007fddd3c088cd clone > >>>>> /usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113:0 > >>>>> clang-11: error: unable to execute command: Aborted > >>>>> clang-11: error: linker command failed due to signal (use -v to see > >>>>> invocation) > >>>>> > >>>>> > >>>>> > >>>>> On Sun, Mar 29, 2020 at 1:44 AM Nemanja Ivanovic < > >>>>> nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com>> wrote: > >>>>> > >>>>> The discussion in https://reviews.llvm.org/D76240 > >>>>> > >>>>> may be of interest to the participants of this thread. Specifically, > >>>>> Fangrui's planned work on a command line option to limit threads for LLD. > >>>>> Until that becomes a reality, you might want to use the CMake variable > >>>>> -DLLVM_ENABLE_THREADS=OFF when building the LLD that you are using to > >>>>> prevent parallelism. > >>>>> > >>>>> > >>>>> > >>>>> On Sat, Mar 28, 2020 at 12:32 PM Alexandre Ganea via llvm-dev < > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: > >>>>> > >>>>> Alex : > >>>>> > >>>>> Can you please try `numactl` or `taskset` after > >>>>> https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74 > >>>>> ? > >>>>> > >>>>> > >>>>> > >>>>> There was a tiny bug in how *sched_getaffinity*() was used, see: > >>>>> https://reviews.llvm.org/D75153#1942336 > >>>>> > >>>>> > >>>>> > >>>>> *De :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>> > >>>>> *Envoyé :* March 28, 2020 12:11 PM > >>>>> *À :* Itaru Kitayama <itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> > >>>>> *Cc :* LLVM Dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>; Alexandre Ganea < > >>>>> alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com>>; Fāng-ruì Sòng <maskray at google.com <mailto:maskray at google.com>> > >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel build > >>>>> machine > >>>>> > >>>>> > >>>>> > >>>>> I also run into problems with lld on machines with lots of cores (208), > >>>>> but I run into rlimits for number of processes I can create, something > >>>>> outside of my control. > >>>>> > >>>>> > >>>>> > >>>>> Unfortunately things like numactl or taskset do not limit the > >>>>> concurrency of lld currently. > >>>>> > >>>>> > >>>>> > >>>>> On Sat, Mar 28, 2020 at 4:10 AM Itaru Kitayama via llvm-dev < > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: > >>>>> > >>>>> Hi, > >>>>> > >>>>> On a 1296-core Intel machine with 376 GB, setting > >>>>> > >>>>> > >>>>> > >>>>> -DLLVM_PARALLEL_LINK_JOB=1 > >>>>> > >>>>> > >>>>> > >>>>> does not help (switching back to ld scales) see: > >>>>> > >>>>> > >>>>> > >>>>> [5085/5201] Linking CXX executable bin/clang-11 > >>>>> FAILED: bin/clang-11 > >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++ -stdlib=libc++ > >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time > >>>>> -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter > >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic > >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default > >>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor > >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections > >>>>> -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG > >>>>> -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics > >>>>> -Wl,-allow-shlib-undefined -Wl,--export-dynamic -Wl,-O3 > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1gen_reproducer_main.cpp.o > >>>>> -o bin/clang-11 -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMAArch64CodeGen.a > >>>>> lib/libLLVMAArch64AsmParser.a lib/libLLVMAArch64Desc.a > >>>>> lib/libLLVMAArch64Disassembler.a lib/libLLVMAArch64Info.a > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUCodeGen.a > >>>>> lib/libLLVMAMDGPUAsmParser.a lib/libLLVMAMDGPUDesc.a > >>>>> lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUInfo.a > >>>>> lib/libLLVMAMDGPUUtils.a lib/libLLVMARMCodeGen.a > >>>>> lib/libLLVMARMAsmParser.a lib/libLLVMARMDesc.a > >>>>> lib/libLLVMARMDisassembler.a lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a > >>>>> lib/libLLVMAVRCodeGen.a lib/libLLVMAVRAsmParser.a lib/libLLVMAVRDesc.a > >>>>> lib/libLLVMAVRDisassembler.a lib/libLLVMAVRInfo.a > >>>>> lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a lib/libLLVMBPFDesc.a > >>>>> lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a > >>>>> lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a > >>>>> lib/libLLVMHexagonDesc.a lib/libLLVMHexagonDisassembler.a > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a > >>>>> lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiDesc.a > >>>>> lib/libLLVMLanaiDisassembler.a lib/libLLVMLanaiInfo.a > >>>>> lib/libLLVMMipsCodeGen.a lib/libLLVMMipsAsmParser.a > >>>>> lib/libLLVMMipsDesc.a lib/libLLVMMipsDisassembler.a > >>>>> lib/libLLVMMipsInfo.a lib/libLLVMMSP430CodeGen.a > >>>>> lib/libLLVMMSP430AsmParser.a lib/libLLVMMSP430Desc.a > >>>>> lib/libLLVMMSP430Disassembler.a lib/libLLVMMSP430Info.a > >>>>> lib/libLLVMNVPTXCodeGen.a lib/libLLVMNVPTXDesc.a lib/libLLVMNVPTXInfo.a > >>>>> lib/libLLVMPowerPCCodeGen.a lib/libLLVMPowerPCAsmParser.a > >>>>> lib/libLLVMPowerPCDesc.a lib/libLLVMPowerPCDisassembler.a > >>>>> lib/libLLVMPowerPCInfo.a lib/libLLVMRISCVCodeGen.a > >>>>> lib/libLLVMRISCVAsmParser.a lib/libLLVMRISCVDesc.a > >>>>> lib/libLLVMRISCVDisassembler.a lib/libLLVMRISCVInfo.a > >>>>> lib/libLLVMRISCVUtils.a lib/libLLVMSparcCodeGen.a > >>>>> lib/libLLVMSparcAsmParser.a lib/libLLVMSparcDesc.a > >>>>> lib/libLLVMSparcDisassembler.a lib/libLLVMSparcInfo.a > >>>>> lib/libLLVMSystemZCodeGen.a lib/libLLVMSystemZAsmParser.a > >>>>> lib/libLLVMSystemZDesc.a lib/libLLVMSystemZDisassembler.a > >>>>> lib/libLLVMSystemZInfo.a lib/libLLVMWebAssemblyCodeGen.a > >>>>> lib/libLLVMWebAssemblyAsmParser.a lib/libLLVMWebAssemblyDesc.a > >>>>> lib/libLLVMWebAssemblyDisassembler.a lib/libLLVMWebAssemblyInfo.a > >>>>> lib/libLLVMX86CodeGen.a lib/libLLVMX86AsmParser.a lib/libLLVMX86Desc.a > >>>>> lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a lib/libLLVMX86Utils.a > >>>>> lib/libLLVMXCoreCodeGen.a lib/libLLVMXCoreDesc.a > >>>>> lib/libLLVMXCoreDisassembler.a lib/libLLVMXCoreInfo.a > >>>>> lib/libLLVMAnalysis.a lib/libLLVMCodeGen.a lib/libLLVMCore.a > >>>>> lib/libLLVMipo.a lib/libLLVMAggressiveInstCombine.a > >>>>> lib/libLLVMInstCombine.a lib/libLLVMInstrumentation.a lib/libLLVMMC.a > >>>>> lib/libLLVMMCParser.a lib/libLLVMObjCARCOpts.a lib/libLLVMOption.a > >>>>> lib/libLLVMScalarOpts.a lib/libLLVMSupport.a lib/libLLVMTransformUtils.a > >>>>> lib/libLLVMVectorize.a -lpthread lib/libclangBasic.a > >>>>> lib/libclangCodeGen.a lib/libclangDriver.a lib/libclangFrontend.a > >>>>> lib/libclangFrontendTool.a lib/libclangSerialization.a > >>>>> lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMMIRParser.a lib/libLLVMAMDGPUDesc.a > >>>>> lib/libLLVMAMDGPUInfo.a lib/libLLVMAMDGPUUtils.a lib/libLLVMARMDesc.a > >>>>> lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a lib/libLLVMHexagonDesc.a > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiDesc.a lib/libLLVMLanaiInfo.a > >>>>> lib/libLLVMSystemZDesc.a lib/libLLVMSystemZInfo.a > >>>>> lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a > >>>>> lib/libLLVMCFGuard.a lib/libLLVMGlobalISel.a lib/libLLVMAsmPrinter.a > >>>>> lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a > >>>>> lib/libLLVMMCDisassembler.a lib/libclangCodeGen.a lib/libLLVMCoverage.a > >>>>> lib/libLLVMLTO.a lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a > >>>>> lib/libLLVMCodeGen.a lib/libLLVMTarget.a lib/libLLVMCoroutines.a > >>>>> lib/libLLVMipo.a lib/libLLVMInstrumentation.a lib/libLLVMVectorize.a > >>>>> lib/libLLVMBitWriter.a lib/libLLVMIRReader.a lib/libLLVMAsmParser.a > >>>>> lib/libLLVMLinker.a lib/libLLVMScalarOpts.a > >>>>> lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a > >>>>> lib/libclangRewriteFrontend.a lib/libclangARCMigrate.a > >>>>> lib/libclangStaticAnalyzerFrontend.a lib/libclangStaticAnalyzerCheckers.a > >>>>> lib/libclangStaticAnalyzerCore.a lib/libclangCrossTU.a > >>>>> lib/libclangIndex.a lib/libclangFrontend.a lib/libclangDriver.a > >>>>> lib/libLLVMOption.a lib/libclangParse.a lib/libclangSerialization.a > >>>>> lib/libclangSema.a lib/libclangAnalysis.a lib/libclangASTMatchers.a > >>>>> lib/libclangEdit.a lib/libclangFormat.a lib/libclangToolingInclusions.a > >>>>> lib/libclangToolingCore.a lib/libclangAST.a lib/libLLVMFrontendOpenMP.a > >>>>> lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a > >>>>> lib/libLLVMProfileData.a lib/libLLVMObject.a lib/libLLVMMCParser.a > >>>>> lib/libLLVMBitReader.a lib/libLLVMTextAPI.a lib/libclangRewrite.a > >>>>> lib/libclangLex.a lib/libclangBasic.a lib/libLLVMCore.a > >>>>> lib/libLLVMRemarks.a lib/libLLVMBitstreamReader.a lib/libLLVMMC.a > >>>>> lib/libLLVMBinaryFormat.a lib/libLLVMDebugInfoCodeView.a > >>>>> lib/libLLVMDebugInfoMSF.a lib/libLLVMSupport.a -lz -lrt -ldl -ltinfo > >>>>> -lpthread -lm lib/libLLVMDemangle.a && : > >>>>> ld.lld: error: failed to open bin/clang-11: Cannot allocate memory > >>>>> > >>>>> _______________________________________________ > >>>>> LLVM Developers mailing list > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >>>>> > >>>>> _______________________________________________ > >>>>> LLVM Developers mailing list > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >>>>> > >>>>> > > >_______________________________________________ > >LLVM Developers mailing list > >llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > >https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >
Itaru Kitayama via llvm-dev
2020-Apr-01 22:17 UTC
[llvm-dev] LLD issue on a massively parallel build machine
Tom, Then what ratio do you think it’s minimal? On Thu, Apr 2, 2020 at 6:11 Tom Stellard <tstellar at redhat.com> wrote:> On 04/01/2020 01:47 PM, Itaru Kitayama via llvm-dev wrote: > > Thanks for the heads up the supercomputer > > Is down for maintenance this week. > > I’ll give it a try when it gets back. > > > > This doesn't look to me like it's necessarily an lld issue. Trying > to build all the sub-projects without limiting the number of ninja jobs > is almost guaranteed to run out of memory on a machine like this with such > a low memory:core ratio. > > -Tom > > > On Thu, Apr 2, 2020 at 1:49 Fangrui Song <maskray at google.com <mailto: > maskray at google.com>> wrote: > > > > On 2020-03-29, Nemanja Ivanovic via llvm-dev wrote: > > >Glad you got it working. > > >My suggestion about LLVM_ENABLE_THREADS didn't work because you > didn't > > >apply it when building the build linker. > > > > > >When you don't have the ability to rebuild the build compiler, this > doesn't > > >apply. In those cases, I end up doing a dirty hack where I use a > wrapper > > >script with something like: > > > > > ><path-to-build-compiler> "$@" -Wl,--no-threads > > > > > >To invoke the build compiler. That way you still get full > parallelism while > > >compiling and linking, but you prevent LLD from using threads. > > > > > > https://reviews.llvm.org/D76885 (committed yesterday) changed > > --no-threads to --threads={1,2,...} (--no-threads is used rarely > (GNU ld > > does not support it) so I'd rather not keep the option for > > compatibility). > > > > You may try -Wl,--threads=1 > > > > >On Sat., Mar. 28, 2020, 4:19 p.m. Itaru Kitayama, < > itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> > > >wrote: > > > > > >> Good news, I was able to use up to 37 cores for LLVM build with > LLD. > > >> The build speed, did not measure precisely though, is comparable > to the > > >> build with GNU ld case. > > >> > > >> Thank you all for your help! > > >> > > >> On Sun, Mar 29, 2020 at 5:04 AM Alex Brachet-Mialot < > > >> alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>> > wrote: > > >> > > >>> Yes unfortunately that would limit you to 4 cores. > > >>> > > >>> There’s no easy way to use lld with —no-threads with our build > system at > > >>> the moment unfortunately. > > >>> > > >>> I also had just been using ld.gold, but will switch now that > numactl > > >>> works for me. > > >>> > > >>> On Sat, Mar 28, 2020 at 3:58 PM Itaru Kitayama < > itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> > > >>> wrote: > > >>> > > >>>> That is slowing down the build visibly, for the speed I should > stick > > >>>> with ld at the moment. > > >>>> > > >>>> > > >>>> On Sun, Mar 29, 2020 at 4:42 AM Alexandre Ganea < > > >>>> alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com>> > wrote: > > >>>> > > >>>>> Would `taskset -c 0-3 ninja check-all -j4` work? > > >>>>> > > >>>>> > > >>>>> > > >>>>> *De :* Itaru Kitayama <itaru.kitayama at gmail.com <mailto: > itaru.kitayama at gmail.com>> > > >>>>> *Envoyé :* March 28, 2020 3:37 PM > > >>>>> *À :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com > <mailto:alexbrachetmialot at gmail.com>> > > >>>>> *Cc :* Alexandre Ganea <alexandre.ganea at ubisoft.com <mailto: > alexandre.ganea at ubisoft.com>>; LLVM Dev < > > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>; > Nemanja Ivanovic <nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com > >> > > >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel > build > > >>>>> machine > > >>>>> > > >>>>> > > >>>>> > > >>>>> $ free -g > > >>>>> total used free shared > buff/cache > > >>>>> available > > >>>>> Mem: 376 149 20 1 > 207 > > >>>>> 225 > > >>>>> Swap: 3 0 3 > > >>>>> > > >>>>> > > >>>>> > > >>>>> Too small swap size for a 72-core login machine? > > >>>>> > > >>>>> > > >>>>> > > >>>>> On Sun, Mar 29, 2020 at 4:28 AM Alex Brachet-Mialot < > > >>>>> alexbrachetmialot at gmail.com <mailto: > alexbrachetmialot at gmail.com>> wrote: > > >>>>> > > >>>>> Enable threads is for building llvm with or without thread > support. To > > >>>>> my knowledge it has no effect on the build system. > > >>>>> > > >>>>> > > >>>>> > > >>>>> Like was discussed above you could try limiting the number of > > >>>>> processes created (assuming your lld was after that commit) > with numactl or > > >>>>> taskset. > > >>>>> > > >>>>> > > >>>>> > > >>>>> In your case though it looked like you were running out of > memory. You > > >>>>> could try increasing your swap space? > > >>>>> > > >>>>> > > >>>>> > > >>>>> Alex > > >>>>> > > >>>>> > > >>>>> > > >>>>> On Sat, Mar 28, 2020 at 3:15 PM Itaru Kitayama < > > >>>>> itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> > wrote: > > >>>>> > > >>>>> Hi, > > >>>>> > > >>>>> My configuration is below: > > >>>>> > > >>>>> cmake -GNinja -DLLVM_ENABLE_LLD=ON -DLLVM_ENABLE_THREADS=OFF > > >>>>> -DLLVM_ENABLE_LIBCXX=ON -DCMAKE_BUILD_TYPE=Release > > >>>>> -DGCC_INSTALL_PREFIX=/home/usr4/c74014i/opt/gcc-7.5.0/ > > >>>>> -DLIBOMPTARGET_ENABLE_DEBUG=ON > > >>>>> -DCMAKE_INSTALL_PREFIX=$HOME/opt/clang/${today} > -DCMAKE_C_COMPILER=clang > > >>>>> -DCMAKE_C_FLAGS="" -DCMAKE_CXX_COMPILER=clang++ > -DCMAKE_CXX_FLAGS="" > > >>>>> > -DLLVM_ENABLE_PROJECTS="clang;openmp;lld;libcxx;libcxxabi;compiler-rt;libunwind" > > >>>>> -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_60 > > >>>>> -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=60 > -DLLVM_TARGETS_TO_BUILD="all" > > >>>>> /tmp/llvm-project/llvm > > >>>>> > > >>>>> > > >>>>> > > >>>>> but did not help my case: > > >>>>> > > >>>>> > > >>>>> > > >>>>> [...] > > >>>>> > > >>>>> [4802/5201] Linking CXX executable bin/lld > > >>>>> FAILED: bin/lld > > >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++ > -stdlib=libc++ > > >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time > > >>>>> -Werror=unguarded-availability-new -Wall -Wextra > -Wno-unused-parameter > > >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers > -pedantic > > >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default > > >>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor > > >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections > -fdata-sections > > >>>>> -O3 -DNDEBUG -stdlib=libc++ -fuse-ld=lld > -Wl,--color-diagnostics > > >>>>> -Wl,-allow-shlib-undefined -Wl,--export-dynamic -Wl,-O3 > > >>>>> tools/lld/tools/lld/CMakeFiles/lld.dir/lld.cpp.o -o bin/lld > > >>>>> -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.a -lpthread > > >>>>> lib/liblldCommon.a lib/liblldCOFF.a lib/liblldDriver.a > lib/liblldELF.a > > >>>>> lib/liblldMinGW.a lib/liblldWasm.a lib/liblldMachO.a > > >>>>> lib/liblldReaderWriter.a lib/liblldYAML.a lib/liblldCore.a > > >>>>> lib/liblldCOFF.a lib/libLLVMDebugInfoPDB.a > lib/libLLVMLibDriver.a > > >>>>> lib/libLLVMWindowsManifest.a /usr/lib64/libxml2.so > lib/liblldCommon.a > > >>>>> lib/libLLVMOption.a lib/libLLVMAArch64CodeGen.a > > >>>>> lib/libLLVMAArch64AsmParser.a > lib/libLLVMAArch64Disassembler.a > > >>>>> lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a > > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUCodeGen.a > > >>>>> lib/libLLVMMIRParser.a lib/libLLVMAMDGPUAsmParser.a > > >>>>> lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUDesc.a > > >>>>> lib/libLLVMAMDGPUInfo.a lib/libLLVMAMDGPUUtils.a > lib/libLLVMARMCodeGen.a > > >>>>> lib/libLLVMARMAsmParser.a lib/libLLVMARMDisassembler.a > > >>>>> lib/libLLVMARMDesc.a lib/libLLVMARMInfo.a > lib/libLLVMARMUtils.a > > >>>>> lib/libLLVMAVRCodeGen.a lib/libLLVMAVRAsmParser.a > lib/libLLVMAVRDesc.a > > >>>>> lib/libLLVMAVRDisassembler.a lib/libLLVMAVRInfo.a > > >>>>> lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a > lib/libLLVMBPFDesc.a > > >>>>> lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a > > >>>>> lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a > > >>>>> lib/libLLVMHexagonDisassembler.a lib/libLLVMHexagonDesc.a > > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a > > >>>>> lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiDisassembler.a > > >>>>> lib/libLLVMLanaiDesc.a lib/libLLVMLanaiInfo.a > lib/libLLVMMipsCodeGen.a > > >>>>> lib/libLLVMMipsAsmParser.a lib/libLLVMMipsDesc.a > > >>>>> lib/libLLVMMipsDisassembler.a lib/libLLVMMipsInfo.a > > >>>>> lib/libLLVMMSP430CodeGen.a lib/libLLVMMSP430AsmParser.a > > >>>>> lib/libLLVMMSP430Desc.a lib/libLLVMMSP430Disassembler.a > > >>>>> lib/libLLVMMSP430Info.a lib/libLLVMNVPTXCodeGen.a > lib/libLLVMNVPTXDesc.a > > >>>>> lib/libLLVMNVPTXInfo.a lib/libLLVMPowerPCCodeGen.a > > >>>>> lib/libLLVMPowerPCAsmParser.a lib/libLLVMPowerPCDesc.a > > >>>>> lib/libLLVMPowerPCDisassembler.a lib/libLLVMPowerPCInfo.a > > >>>>> lib/libLLVMRISCVCodeGen.a lib/libLLVMRISCVAsmParser.a > > >>>>> lib/libLLVMRISCVDesc.a lib/libLLVMRISCVDisassembler.a > > >>>>> lib/libLLVMRISCVInfo.a lib/libLLVMRISCVUtils.a > lib/libLLVMSparcCodeGen.a > > >>>>> lib/libLLVMSparcAsmParser.a lib/libLLVMSparcDesc.a > > >>>>> lib/libLLVMSparcDisassembler.a lib/libLLVMSparcInfo.a > > >>>>> lib/libLLVMSystemZCodeGen.a lib/libLLVMSystemZAsmParser.a > > >>>>> lib/libLLVMSystemZDisassembler.a lib/libLLVMSystemZDesc.a > > >>>>> lib/libLLVMSystemZInfo.a lib/libLLVMWebAssemblyCodeGen.a > > >>>>> lib/libLLVMWebAssemblyAsmParser.a > lib/libLLVMWebAssemblyDisassembler.a > > >>>>> lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a > > >>>>> lib/libLLVMX86CodeGen.a lib/libLLVMCFGuard.a > lib/libLLVMGlobalISel.a > > >>>>> lib/libLLVMX86AsmParser.a lib/libLLVMX86Desc.a > > >>>>> lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a > lib/libLLVMX86Utils.a > > >>>>> lib/libLLVMXCoreCodeGen.a lib/libLLVMAsmPrinter.a > > >>>>> lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a > > >>>>> lib/libLLVMXCoreDesc.a lib/libLLVMXCoreDisassembler.a > > >>>>> lib/libLLVMMCDisassembler.a lib/libLLVMXCoreInfo.a > lib/libLLVMLTO.a > > >>>>> lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a > lib/libLLVMCodeGen.a > > >>>>> lib/libLLVMTarget.a lib/libLLVMCoroutines.a lib/libLLVMipo.a > > >>>>> lib/libLLVMBitWriter.a lib/libLLVMScalarOpts.a > lib/libLLVMVectorize.a > > >>>>> lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a > > >>>>> lib/libLLVMLinker.a lib/libLLVMFrontendOpenMP.a > lib/libLLVMIRReader.a > > >>>>> lib/libLLVMAsmParser.a lib/libLLVMInstrumentation.a > > >>>>> lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a > lib/libLLVMObject.a > > >>>>> lib/libLLVMBitReader.a lib/libLLVMMCParser.a lib/libLLVMMC.a > > >>>>> lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a > > >>>>> lib/libLLVMTextAPI.a lib/libLLVMProfileData.a > lib/libLLVMCore.a > > >>>>> lib/libLLVMBinaryFormat.a lib/libLLVMRemarks.a > > >>>>> lib/libLLVMBitstreamReader.a lib/libLLVMSupport.a -lz > -lrt -ldl > > >>>>> -ltinfo -lpthread -lm lib/libLLVMDemangle.a && cd > > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && > > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E > > >>>>> create_symlink lld /tmp/build/202003290405/./bin/lld-link && cd > > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && > > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E > > >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld.lld && cd > > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && > > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E > > >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld64.lld && cd > > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && > > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E > > >>>>> create_symlink lld /tmp/build/202003290405/./bin/wasm-ld > > >>>>> terminating with uncaught exception of type > std::__1::system_error: > > >>>>> thread constructor failed: Resource temporarily unavailable > > >>>>> PLEASE submit a bug report to https://bugs.llvm.org/ and > include the > > >>>>> crash backtrace. > > >>>>> #0 0x00000000007924f4 PrintStackTraceSignalHandler(void*) > > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7924f4) > > >>>>> #1 0x00000000007902ce llvm::sys::RunSignalHandlers() > > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7902ce) > > >>>>> #2 0x0000000000792bd5 SignalHandler(int) > > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x792bd5) > > >>>>> #3 0x00007fddd54bb630 __restore_rt > (/lib64/libpthread.so.0+0xf630) > > >>>>> #4 0x00007fddd3b40377 raise > > >>>>> > /usr/src/debug/glibc-2.17-c758a686/signal/../nptl/sysdeps/unix/sysv/linux/raise.c:55:0 > > >>>>> #5 0x00007fddd3b41a68 abort > > >>>>> /usr/src/debug/glibc-2.17-c758a686/stdlib/abort.c:92:0 > > >>>>> #6 0x00007fddd411ae8b > > >>>>> > (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2be8b) > > >>>>> #7 0x00007fddd40fdb9f demangling_terminate_handler() > > >>>>> > (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0xeb9f) > > >>>>> #8 0x00007fddd411a0a3 std::__terminate(void (*)()) > > >>>>> > (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2b0a3) > > >>>>> #9 0x00007fddd411ce06 > > >>>>> > (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2de06) > > >>>>> #10 0x00007fddd411cd9f > > >>>>> > (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2dd9f) > > >>>>> #11 0x00007fddd43bded4 std::__1::__throw_system_error(int, > char const*) > > >>>>> > (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++.so.1+0x90ed4) > > >>>>> #12 0x0000000002a7a316 void* > > >>>>> > std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, > > >>>>> std::__1::default_delete<std::__1::__thread_struct> >, > > >>>>> llvm::parallel::detail::(anonymous > > >>>>> > namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::'lambda'()> > > >>>>> >(void*) > (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x2a7a316) > > >>>>> #13 0x00007fddd54b3ea5 start_thread > > >>>>> /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307:0 > > >>>>> #14 0x00007fddd3c088cd clone > > >>>>> > /usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113:0 > > >>>>> clang-11: error: unable to execute command: Aborted > > >>>>> clang-11: error: linker command failed due to signal (use -v > to see > > >>>>> invocation) > > >>>>> > > >>>>> > > >>>>> > > >>>>> On Sun, Mar 29, 2020 at 1:44 AM Nemanja Ivanovic < > > >>>>> nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com>> > wrote: > > >>>>> > > >>>>> The discussion in https://reviews.llvm.org/D76240 > > >>>>> > > >>>>> may be of interest to the participants of this thread. > Specifically, > > >>>>> Fangrui's planned work on a command line option to limit > threads for LLD. > > >>>>> Until that becomes a reality, you might want to use the CMake > variable > > >>>>> -DLLVM_ENABLE_THREADS=OFF when building the LLD that you are > using to > > >>>>> prevent parallelism. > > >>>>> > > >>>>> > > >>>>> > > >>>>> On Sat, Mar 28, 2020 at 12:32 PM Alexandre Ganea via llvm-dev < > > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> > wrote: > > >>>>> > > >>>>> Alex : > > >>>>> > > >>>>> Can you please try `numactl` or `taskset` after > > >>>>> > https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74 > > >>>>> ? > > >>>>> > > >>>>> > > >>>>> > > >>>>> There was a tiny bug in how *sched_getaffinity*() was used, > see: > > >>>>> https://reviews.llvm.org/D75153#1942336 > > >>>>> > > >>>>> > > >>>>> > > >>>>> *De :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com > <mailto:alexbrachetmialot at gmail.com>> > > >>>>> *Envoyé :* March 28, 2020 12:11 PM > > >>>>> *À :* Itaru Kitayama <itaru.kitayama at gmail.com <mailto: > itaru.kitayama at gmail.com>> > > >>>>> *Cc :* LLVM Dev <llvm-dev at lists.llvm.org <mailto: > llvm-dev at lists.llvm.org>>; Alexandre Ganea < > > >>>>> alexandre.ganea at ubisoft.com <mailto: > alexandre.ganea at ubisoft.com>>; Fāng-ruì Sòng <maskray at google.com <mailto: > maskray at google.com>> > > >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel > build > > >>>>> machine > > >>>>> > > >>>>> > > >>>>> > > >>>>> I also run into problems with lld on machines with lots of > cores (208), > > >>>>> but I run into rlimits for number of processes I can create, > something > > >>>>> outside of my control. > > >>>>> > > >>>>> > > >>>>> > > >>>>> Unfortunately things like numactl or taskset do not limit the > > >>>>> concurrency of lld currently. > > >>>>> > > >>>>> > > >>>>> > > >>>>> On Sat, Mar 28, 2020 at 4:10 AM Itaru Kitayama via llvm-dev < > > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> > wrote: > > >>>>> > > >>>>> Hi, > > >>>>> > > >>>>> On a 1296-core Intel machine with 376 GB, setting > > >>>>> > > >>>>> > > >>>>> > > >>>>> -DLLVM_PARALLEL_LINK_JOB=1 > > >>>>> > > >>>>> > > >>>>> > > >>>>> does not help (switching back to ld scales) see: > > >>>>> > > >>>>> > > >>>>> > > >>>>> [5085/5201] Linking CXX executable bin/clang-11 > > >>>>> FAILED: bin/clang-11 > > >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++ > -stdlib=libc++ > > >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time > > >>>>> -Werror=unguarded-availability-new -Wall -Wextra > -Wno-unused-parameter > > >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers > -pedantic > > >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default > > >>>>> -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor > > >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections > -fdata-sections > > >>>>> -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 > -DNDEBUG > > >>>>> -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics > > >>>>> -Wl,-allow-shlib-undefined -Wl,--export-dynamic -Wl,-O3 > > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o > > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o > > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o > > >>>>> > tools/clang/tools/driver/CMakeFiles/clang.dir/cc1gen_reproducer_main.cpp.o > > >>>>> -o bin/clang-11 -Wl,-rpath,"\$ORIGIN/../lib" > lib/libLLVMAArch64CodeGen.a > > >>>>> lib/libLLVMAArch64AsmParser.a lib/libLLVMAArch64Desc.a > > >>>>> lib/libLLVMAArch64Disassembler.a lib/libLLVMAArch64Info.a > > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUCodeGen.a > > >>>>> lib/libLLVMAMDGPUAsmParser.a lib/libLLVMAMDGPUDesc.a > > >>>>> lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUInfo.a > > >>>>> lib/libLLVMAMDGPUUtils.a lib/libLLVMARMCodeGen.a > > >>>>> lib/libLLVMARMAsmParser.a lib/libLLVMARMDesc.a > > >>>>> lib/libLLVMARMDisassembler.a lib/libLLVMARMInfo.a > lib/libLLVMARMUtils.a > > >>>>> lib/libLLVMAVRCodeGen.a lib/libLLVMAVRAsmParser.a > lib/libLLVMAVRDesc.a > > >>>>> lib/libLLVMAVRDisassembler.a lib/libLLVMAVRInfo.a > > >>>>> lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a > lib/libLLVMBPFDesc.a > > >>>>> lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a > > >>>>> lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a > > >>>>> lib/libLLVMHexagonDesc.a lib/libLLVMHexagonDisassembler.a > > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a > > >>>>> lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiDesc.a > > >>>>> lib/libLLVMLanaiDisassembler.a lib/libLLVMLanaiInfo.a > > >>>>> lib/libLLVMMipsCodeGen.a lib/libLLVMMipsAsmParser.a > > >>>>> lib/libLLVMMipsDesc.a lib/libLLVMMipsDisassembler.a > > >>>>> lib/libLLVMMipsInfo.a lib/libLLVMMSP430CodeGen.a > > >>>>> lib/libLLVMMSP430AsmParser.a lib/libLLVMMSP430Desc.a > > >>>>> lib/libLLVMMSP430Disassembler.a lib/libLLVMMSP430Info.a > > >>>>> lib/libLLVMNVPTXCodeGen.a lib/libLLVMNVPTXDesc.a > lib/libLLVMNVPTXInfo.a > > >>>>> lib/libLLVMPowerPCCodeGen.a lib/libLLVMPowerPCAsmParser.a > > >>>>> lib/libLLVMPowerPCDesc.a lib/libLLVMPowerPCDisassembler.a > > >>>>> lib/libLLVMPowerPCInfo.a lib/libLLVMRISCVCodeGen.a > > >>>>> lib/libLLVMRISCVAsmParser.a lib/libLLVMRISCVDesc.a > > >>>>> lib/libLLVMRISCVDisassembler.a lib/libLLVMRISCVInfo.a > > >>>>> lib/libLLVMRISCVUtils.a lib/libLLVMSparcCodeGen.a > > >>>>> lib/libLLVMSparcAsmParser.a lib/libLLVMSparcDesc.a > > >>>>> lib/libLLVMSparcDisassembler.a lib/libLLVMSparcInfo.a > > >>>>> lib/libLLVMSystemZCodeGen.a lib/libLLVMSystemZAsmParser.a > > >>>>> lib/libLLVMSystemZDesc.a lib/libLLVMSystemZDisassembler.a > > >>>>> lib/libLLVMSystemZInfo.a lib/libLLVMWebAssemblyCodeGen.a > > >>>>> lib/libLLVMWebAssemblyAsmParser.a > lib/libLLVMWebAssemblyDesc.a > > >>>>> lib/libLLVMWebAssemblyDisassembler.a > lib/libLLVMWebAssemblyInfo.a > > >>>>> lib/libLLVMX86CodeGen.a lib/libLLVMX86AsmParser.a > lib/libLLVMX86Desc.a > > >>>>> lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a > lib/libLLVMX86Utils.a > > >>>>> lib/libLLVMXCoreCodeGen.a lib/libLLVMXCoreDesc.a > > >>>>> lib/libLLVMXCoreDisassembler.a lib/libLLVMXCoreInfo.a > > >>>>> lib/libLLVMAnalysis.a lib/libLLVMCodeGen.a lib/libLLVMCore.a > > >>>>> lib/libLLVMipo.a lib/libLLVMAggressiveInstCombine.a > > >>>>> lib/libLLVMInstCombine.a lib/libLLVMInstrumentation.a > lib/libLLVMMC.a > > >>>>> lib/libLLVMMCParser.a lib/libLLVMObjCARCOpts.a > lib/libLLVMOption.a > > >>>>> lib/libLLVMScalarOpts.a lib/libLLVMSupport.a > lib/libLLVMTransformUtils.a > > >>>>> lib/libLLVMVectorize.a -lpthread lib/libclangBasic.a > > >>>>> lib/libclangCodeGen.a lib/libclangDriver.a > lib/libclangFrontend.a > > >>>>> lib/libclangFrontendTool.a lib/libclangSerialization.a > > >>>>> lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a > > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMMIRParser.a > lib/libLLVMAMDGPUDesc.a > > >>>>> lib/libLLVMAMDGPUInfo.a lib/libLLVMAMDGPUUtils.a > lib/libLLVMARMDesc.a > > >>>>> lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a > lib/libLLVMHexagonDesc.a > > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiDesc.a > lib/libLLVMLanaiInfo.a > > >>>>> lib/libLLVMSystemZDesc.a lib/libLLVMSystemZInfo.a > > >>>>> lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a > > >>>>> lib/libLLVMCFGuard.a lib/libLLVMGlobalISel.a > lib/libLLVMAsmPrinter.a > > >>>>> lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a > > >>>>> lib/libLLVMMCDisassembler.a lib/libclangCodeGen.a > lib/libLLVMCoverage.a > > >>>>> lib/libLLVMLTO.a lib/libLLVMObjCARCOpts.a > lib/libLLVMPasses.a > > >>>>> lib/libLLVMCodeGen.a lib/libLLVMTarget.a > lib/libLLVMCoroutines.a > > >>>>> lib/libLLVMipo.a lib/libLLVMInstrumentation.a > lib/libLLVMVectorize.a > > >>>>> lib/libLLVMBitWriter.a lib/libLLVMIRReader.a > lib/libLLVMAsmParser.a > > >>>>> lib/libLLVMLinker.a lib/libLLVMScalarOpts.a > > >>>>> lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a > > >>>>> lib/libclangRewriteFrontend.a lib/libclangARCMigrate.a > > >>>>> lib/libclangStaticAnalyzerFrontend.a > lib/libclangStaticAnalyzerCheckers.a > > >>>>> lib/libclangStaticAnalyzerCore.a lib/libclangCrossTU.a > > >>>>> lib/libclangIndex.a lib/libclangFrontend.a > lib/libclangDriver.a > > >>>>> lib/libLLVMOption.a lib/libclangParse.a > lib/libclangSerialization.a > > >>>>> lib/libclangSema.a lib/libclangAnalysis.a > lib/libclangASTMatchers.a > > >>>>> lib/libclangEdit.a lib/libclangFormat.a > lib/libclangToolingInclusions.a > > >>>>> lib/libclangToolingCore.a lib/libclangAST.a > lib/libLLVMFrontendOpenMP.a > > >>>>> lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a > > >>>>> lib/libLLVMProfileData.a lib/libLLVMObject.a > lib/libLLVMMCParser.a > > >>>>> lib/libLLVMBitReader.a lib/libLLVMTextAPI.a > lib/libclangRewrite.a > > >>>>> lib/libclangLex.a lib/libclangBasic.a lib/libLLVMCore.a > > >>>>> lib/libLLVMRemarks.a lib/libLLVMBitstreamReader.a > lib/libLLVMMC.a > > >>>>> lib/libLLVMBinaryFormat.a lib/libLLVMDebugInfoCodeView.a > > >>>>> lib/libLLVMDebugInfoMSF.a lib/libLLVMSupport.a -lz -lrt > -ldl -ltinfo > > >>>>> -lpthread -lm lib/libLLVMDemangle.a && : > > >>>>> ld.lld: error: failed to open bin/clang-11: Cannot allocate > memory > > >>>>> > > >>>>> _______________________________________________ > > >>>>> LLVM Developers mailing list > > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > >>>>> > > >>>>> _______________________________________________ > > >>>>> LLVM Developers mailing list > > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > >>>>> > > >>>>> > > > > >_______________________________________________ > > >LLVM Developers mailing list > > >llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > > >https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > > > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org > > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200402/376d8a4f/attachment-0001.html>
Itaru Kitayama via llvm-dev
2020-Apr-01 23:12 UTC
[llvm-dev] LLD issue on a massively parallel build machine
On another login node which is 256 (GB)/48 (nodes) JURECA at JSC, I never had an LLD issue without setting -j when executing ninja in the past few weeks. On Thu, Apr 2, 2020 at 7:17 AM Itaru Kitayama <itaru.kitayama at gmail.com> wrote:> Tom, > Then what ratio do you think it’s minimal? > > On Thu, Apr 2, 2020 at 6:11 Tom Stellard <tstellar at redhat.com> wrote: > >> On 04/01/2020 01:47 PM, Itaru Kitayama via llvm-dev wrote: >> > Thanks for the heads up the supercomputer >> > Is down for maintenance this week. >> > I’ll give it a try when it gets back. >> > >> >> This doesn't look to me like it's necessarily an lld issue. Trying >> to build all the sub-projects without limiting the number of ninja jobs >> is almost guaranteed to run out of memory on a machine like this with such >> a low memory:core ratio. >> >> -Tom >> >> > On Thu, Apr 2, 2020 at 1:49 Fangrui Song <maskray at google.com <mailto: >> maskray at google.com>> wrote: >> > >> > On 2020-03-29, Nemanja Ivanovic via llvm-dev wrote: >> > >Glad you got it working. >> > >My suggestion about LLVM_ENABLE_THREADS didn't work because you >> didn't >> > >apply it when building the build linker. >> > > >> > >When you don't have the ability to rebuild the build compiler, >> this doesn't >> > >apply. In those cases, I end up doing a dirty hack where I use a >> wrapper >> > >script with something like: >> > > >> > ><path-to-build-compiler> "$@" -Wl,--no-threads >> > > >> > >To invoke the build compiler. That way you still get full >> parallelism while >> > >compiling and linking, but you prevent LLD from using threads. >> > >> > >> > https://reviews.llvm.org/D76885 (committed yesterday) changed >> > --no-threads to --threads={1,2,...} (--no-threads is used rarely >> (GNU ld >> > does not support it) so I'd rather not keep the option for >> > compatibility). >> > >> > You may try -Wl,--threads=1 >> > >> > >On Sat., Mar. 28, 2020, 4:19 p.m. Itaru Kitayama, < >> itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> >> > >wrote: >> > > >> > >> Good news, I was able to use up to 37 cores for LLVM build with >> LLD. >> > >> The build speed, did not measure precisely though, is comparable >> to the >> > >> build with GNU ld case. >> > >> >> > >> Thank you all for your help! >> > >> >> > >> On Sun, Mar 29, 2020 at 5:04 AM Alex Brachet-Mialot < >> > >> alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>> >> wrote: >> > >> >> > >>> Yes unfortunately that would limit you to 4 cores. >> > >>> >> > >>> There’s no easy way to use lld with —no-threads with our build >> system at >> > >>> the moment unfortunately. >> > >>> >> > >>> I also had just been using ld.gold, but will switch now that >> numactl >> > >>> works for me. >> > >>> >> > >>> On Sat, Mar 28, 2020 at 3:58 PM Itaru Kitayama < >> itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> >> > >>> wrote: >> > >>> >> > >>>> That is slowing down the build visibly, for the speed I should >> stick >> > >>>> with ld at the moment. >> > >>>> >> > >>>> >> > >>>> On Sun, Mar 29, 2020 at 4:42 AM Alexandre Ganea < >> > >>>> alexandre.ganea at ubisoft.com <mailto: >> alexandre.ganea at ubisoft.com>> wrote: >> > >>>> >> > >>>>> Would `taskset -c 0-3 ninja check-all -j4` work? >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> *De :* Itaru Kitayama <itaru.kitayama at gmail.com <mailto: >> itaru.kitayama at gmail.com>> >> > >>>>> *Envoyé :* March 28, 2020 3:37 PM >> > >>>>> *À :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com >> <mailto:alexbrachetmialot at gmail.com>> >> > >>>>> *Cc :* Alexandre Ganea <alexandre.ganea at ubisoft.com <mailto: >> alexandre.ganea at ubisoft.com>>; LLVM Dev < >> > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>; >> Nemanja Ivanovic <nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com >> >> >> > >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel >> build >> > >>>>> machine >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> $ free -g >> > >>>>> total used free shared >> buff/cache >> > >>>>> available >> > >>>>> Mem: 376 149 20 1 >> 207 >> > >>>>> 225 >> > >>>>> Swap: 3 0 3 >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> Too small swap size for a 72-core login machine? >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> On Sun, Mar 29, 2020 at 4:28 AM Alex Brachet-Mialot < >> > >>>>> alexbrachetmialot at gmail.com <mailto: >> alexbrachetmialot at gmail.com>> wrote: >> > >>>>> >> > >>>>> Enable threads is for building llvm with or without thread >> support. To >> > >>>>> my knowledge it has no effect on the build system. >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> Like was discussed above you could try limiting the number of >> > >>>>> processes created (assuming your lld was after that commit) >> with numactl or >> > >>>>> taskset. >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> In your case though it looked like you were running out of >> memory. You >> > >>>>> could try increasing your swap space? >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> Alex >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> On Sat, Mar 28, 2020 at 3:15 PM Itaru Kitayama < >> > >>>>> itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>> >> wrote: >> > >>>>> >> > >>>>> Hi, >> > >>>>> >> > >>>>> My configuration is below: >> > >>>>> >> > >>>>> cmake -GNinja -DLLVM_ENABLE_LLD=ON -DLLVM_ENABLE_THREADS=OFF >> > >>>>> -DLLVM_ENABLE_LIBCXX=ON -DCMAKE_BUILD_TYPE=Release >> > >>>>> -DGCC_INSTALL_PREFIX=/home/usr4/c74014i/opt/gcc-7.5.0/ >> > >>>>> -DLIBOMPTARGET_ENABLE_DEBUG=ON >> > >>>>> -DCMAKE_INSTALL_PREFIX=$HOME/opt/clang/${today} >> -DCMAKE_C_COMPILER=clang >> > >>>>> -DCMAKE_C_FLAGS="" -DCMAKE_CXX_COMPILER=clang++ >> -DCMAKE_CXX_FLAGS="" >> > >>>>> >> -DLLVM_ENABLE_PROJECTS="clang;openmp;lld;libcxx;libcxxabi;compiler-rt;libunwind" >> > >>>>> -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_60 >> > >>>>> -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=60 >> -DLLVM_TARGETS_TO_BUILD="all" >> > >>>>> /tmp/llvm-project/llvm >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> but did not help my case: >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> [...] >> > >>>>> >> > >>>>> [4802/5201] Linking CXX executable bin/lld >> > >>>>> FAILED: bin/lld >> > >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++ >> -stdlib=libc++ >> > >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time >> > >>>>> -Werror=unguarded-availability-new -Wall -Wextra >> -Wno-unused-parameter >> > >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers >> -pedantic >> > >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default >> > >>>>> -Wno-noexcept-type -Wnon-virtual-dtor >> -Wdelete-non-virtual-dtor >> > >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections >> -fdata-sections >> > >>>>> -O3 -DNDEBUG -stdlib=libc++ -fuse-ld=lld >> -Wl,--color-diagnostics >> > >>>>> -Wl,-allow-shlib-undefined -Wl,--export-dynamic -Wl,-O3 >> > >>>>> tools/lld/tools/lld/CMakeFiles/lld.dir/lld.cpp.o -o bin/lld >> > >>>>> -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.a -lpthread >> > >>>>> lib/liblldCommon.a lib/liblldCOFF.a lib/liblldDriver.a >> lib/liblldELF.a >> > >>>>> lib/liblldMinGW.a lib/liblldWasm.a lib/liblldMachO.a >> > >>>>> lib/liblldReaderWriter.a lib/liblldYAML.a lib/liblldCore.a >> > >>>>> lib/liblldCOFF.a lib/libLLVMDebugInfoPDB.a >> lib/libLLVMLibDriver.a >> > >>>>> lib/libLLVMWindowsManifest.a /usr/lib64/libxml2.so >> lib/liblldCommon.a >> > >>>>> lib/libLLVMOption.a lib/libLLVMAArch64CodeGen.a >> > >>>>> lib/libLLVMAArch64AsmParser.a >> lib/libLLVMAArch64Disassembler.a >> > >>>>> lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a >> > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUCodeGen.a >> > >>>>> lib/libLLVMMIRParser.a lib/libLLVMAMDGPUAsmParser.a >> > >>>>> lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUDesc.a >> > >>>>> lib/libLLVMAMDGPUInfo.a lib/libLLVMAMDGPUUtils.a >> lib/libLLVMARMCodeGen.a >> > >>>>> lib/libLLVMARMAsmParser.a lib/libLLVMARMDisassembler.a >> > >>>>> lib/libLLVMARMDesc.a lib/libLLVMARMInfo.a >> lib/libLLVMARMUtils.a >> > >>>>> lib/libLLVMAVRCodeGen.a lib/libLLVMAVRAsmParser.a >> lib/libLLVMAVRDesc.a >> > >>>>> lib/libLLVMAVRDisassembler.a lib/libLLVMAVRInfo.a >> > >>>>> lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a >> lib/libLLVMBPFDesc.a >> > >>>>> lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a >> > >>>>> lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a >> > >>>>> lib/libLLVMHexagonDisassembler.a lib/libLLVMHexagonDesc.a >> > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a >> > >>>>> lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiDisassembler.a >> > >>>>> lib/libLLVMLanaiDesc.a lib/libLLVMLanaiInfo.a >> lib/libLLVMMipsCodeGen.a >> > >>>>> lib/libLLVMMipsAsmParser.a lib/libLLVMMipsDesc.a >> > >>>>> lib/libLLVMMipsDisassembler.a lib/libLLVMMipsInfo.a >> > >>>>> lib/libLLVMMSP430CodeGen.a lib/libLLVMMSP430AsmParser.a >> > >>>>> lib/libLLVMMSP430Desc.a lib/libLLVMMSP430Disassembler.a >> > >>>>> lib/libLLVMMSP430Info.a lib/libLLVMNVPTXCodeGen.a >> lib/libLLVMNVPTXDesc.a >> > >>>>> lib/libLLVMNVPTXInfo.a lib/libLLVMPowerPCCodeGen.a >> > >>>>> lib/libLLVMPowerPCAsmParser.a lib/libLLVMPowerPCDesc.a >> > >>>>> lib/libLLVMPowerPCDisassembler.a lib/libLLVMPowerPCInfo.a >> > >>>>> lib/libLLVMRISCVCodeGen.a lib/libLLVMRISCVAsmParser.a >> > >>>>> lib/libLLVMRISCVDesc.a lib/libLLVMRISCVDisassembler.a >> > >>>>> lib/libLLVMRISCVInfo.a lib/libLLVMRISCVUtils.a >> lib/libLLVMSparcCodeGen.a >> > >>>>> lib/libLLVMSparcAsmParser.a lib/libLLVMSparcDesc.a >> > >>>>> lib/libLLVMSparcDisassembler.a lib/libLLVMSparcInfo.a >> > >>>>> lib/libLLVMSystemZCodeGen.a lib/libLLVMSystemZAsmParser.a >> > >>>>> lib/libLLVMSystemZDisassembler.a lib/libLLVMSystemZDesc.a >> > >>>>> lib/libLLVMSystemZInfo.a lib/libLLVMWebAssemblyCodeGen.a >> > >>>>> lib/libLLVMWebAssemblyAsmParser.a >> lib/libLLVMWebAssemblyDisassembler.a >> > >>>>> lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a >> > >>>>> lib/libLLVMX86CodeGen.a lib/libLLVMCFGuard.a >> lib/libLLVMGlobalISel.a >> > >>>>> lib/libLLVMX86AsmParser.a lib/libLLVMX86Desc.a >> > >>>>> lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a >> lib/libLLVMX86Utils.a >> > >>>>> lib/libLLVMXCoreCodeGen.a lib/libLLVMAsmPrinter.a >> > >>>>> lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a >> > >>>>> lib/libLLVMXCoreDesc.a lib/libLLVMXCoreDisassembler.a >> > >>>>> lib/libLLVMMCDisassembler.a lib/libLLVMXCoreInfo.a >> lib/libLLVMLTO.a >> > >>>>> lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a >> lib/libLLVMCodeGen.a >> > >>>>> lib/libLLVMTarget.a lib/libLLVMCoroutines.a >> lib/libLLVMipo.a >> > >>>>> lib/libLLVMBitWriter.a lib/libLLVMScalarOpts.a >> lib/libLLVMVectorize.a >> > >>>>> lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a >> > >>>>> lib/libLLVMLinker.a lib/libLLVMFrontendOpenMP.a >> lib/libLLVMIRReader.a >> > >>>>> lib/libLLVMAsmParser.a lib/libLLVMInstrumentation.a >> > >>>>> lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a >> lib/libLLVMObject.a >> > >>>>> lib/libLLVMBitReader.a lib/libLLVMMCParser.a >> lib/libLLVMMC.a >> > >>>>> lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a >> > >>>>> lib/libLLVMTextAPI.a lib/libLLVMProfileData.a >> lib/libLLVMCore.a >> > >>>>> lib/libLLVMBinaryFormat.a lib/libLLVMRemarks.a >> > >>>>> lib/libLLVMBitstreamReader.a lib/libLLVMSupport.a -lz >> -lrt -ldl >> > >>>>> -ltinfo -lpthread -lm lib/libLLVMDemangle.a && cd >> > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && >> > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E >> > >>>>> create_symlink lld /tmp/build/202003290405/./bin/lld-link && >> cd >> > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && >> > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E >> > >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld.lld && cd >> > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && >> > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E >> > >>>>> create_symlink lld /tmp/build/202003290405/./bin/ld64.lld && >> cd >> > >>>>> /tmp/build/202003290405/tools/lld/tools/lld && >> > >>>>> /home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E >> > >>>>> create_symlink lld /tmp/build/202003290405/./bin/wasm-ld >> > >>>>> terminating with uncaught exception of type >> std::__1::system_error: >> > >>>>> thread constructor failed: Resource temporarily unavailable >> > >>>>> PLEASE submit a bug report to https://bugs.llvm.org/ and >> include the >> > >>>>> crash backtrace. >> > >>>>> #0 0x00000000007924f4 PrintStackTraceSignalHandler(void*) >> > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7924f4) >> > >>>>> #1 0x00000000007902ce llvm::sys::RunSignalHandlers() >> > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7902ce) >> > >>>>> #2 0x0000000000792bd5 SignalHandler(int) >> > >>>>> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x792bd5) >> > >>>>> #3 0x00007fddd54bb630 __restore_rt >> (/lib64/libpthread.so.0+0xf630) >> > >>>>> #4 0x00007fddd3b40377 raise >> > >>>>> >> /usr/src/debug/glibc-2.17-c758a686/signal/../nptl/sysdeps/unix/sysv/linux/raise.c:55:0 >> > >>>>> #5 0x00007fddd3b41a68 abort >> > >>>>> /usr/src/debug/glibc-2.17-c758a686/stdlib/abort.c:92:0 >> > >>>>> #6 0x00007fddd411ae8b >> > >>>>> >> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2be8b) >> > >>>>> #7 0x00007fddd40fdb9f demangling_terminate_handler() >> > >>>>> >> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0xeb9f) >> > >>>>> #8 0x00007fddd411a0a3 std::__terminate(void (*)()) >> > >>>>> >> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2b0a3) >> > >>>>> #9 0x00007fddd411ce06 >> > >>>>> >> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2de06) >> > >>>>> #10 0x00007fddd411cd9f >> > >>>>> >> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2dd9f) >> > >>>>> #11 0x00007fddd43bded4 std::__1::__throw_system_error(int, >> char const*) >> > >>>>> >> (/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++.so.1+0x90ed4) >> > >>>>> #12 0x0000000002a7a316 void* >> > >>>>> >> std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, >> > >>>>> std::__1::default_delete<std::__1::__thread_struct> >, >> > >>>>> llvm::parallel::detail::(anonymous >> > >>>>> >> namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::'lambda'()> >> > >>>>> >(void*) >> (/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x2a7a316) >> > >>>>> #13 0x00007fddd54b3ea5 start_thread >> > >>>>> /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307:0 >> > >>>>> #14 0x00007fddd3c088cd clone >> > >>>>> >> /usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113:0 >> > >>>>> clang-11: error: unable to execute command: Aborted >> > >>>>> clang-11: error: linker command failed due to signal (use -v >> to see >> > >>>>> invocation) >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> On Sun, Mar 29, 2020 at 1:44 AM Nemanja Ivanovic < >> > >>>>> nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com>> >> wrote: >> > >>>>> >> > >>>>> The discussion in https://reviews.llvm.org/D76240 >> > >>>>> >> > >>>>> may be of interest to the participants of this thread. >> Specifically, >> > >>>>> Fangrui's planned work on a command line option to limit >> threads for LLD. >> > >>>>> Until that becomes a reality, you might want to use the CMake >> variable >> > >>>>> -DLLVM_ENABLE_THREADS=OFF when building the LLD that you are >> using to >> > >>>>> prevent parallelism. >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> On Sat, Mar 28, 2020 at 12:32 PM Alexandre Ganea via llvm-dev >> < >> > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> >> wrote: >> > >>>>> >> > >>>>> Alex : >> > >>>>> >> > >>>>> Can you please try `numactl` or `taskset` after >> > >>>>> >> https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74 >> > >>>>> ? >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> There was a tiny bug in how *sched_getaffinity*() was used, >> see: >> > >>>>> https://reviews.llvm.org/D75153#1942336 >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> *De :* Alex Brachet-Mialot <alexbrachetmialot at gmail.com >> <mailto:alexbrachetmialot at gmail.com>> >> > >>>>> *Envoyé :* March 28, 2020 12:11 PM >> > >>>>> *À :* Itaru Kitayama <itaru.kitayama at gmail.com <mailto: >> itaru.kitayama at gmail.com>> >> > >>>>> *Cc :* LLVM Dev <llvm-dev at lists.llvm.org <mailto: >> llvm-dev at lists.llvm.org>>; Alexandre Ganea < >> > >>>>> alexandre.ganea at ubisoft.com <mailto: >> alexandre.ganea at ubisoft.com>>; Fāng-ruì Sòng <maskray at google.com <mailto: >> maskray at google.com>> >> > >>>>> *Objet :* Re: [llvm-dev] LLD issue on a massively parallel >> build >> > >>>>> machine >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> I also run into problems with lld on machines with lots of >> cores (208), >> > >>>>> but I run into rlimits for number of processes I can create, >> something >> > >>>>> outside of my control. >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> Unfortunately things like numactl or taskset do not limit the >> > >>>>> concurrency of lld currently. >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> On Sat, Mar 28, 2020 at 4:10 AM Itaru Kitayama via llvm-dev < >> > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> >> wrote: >> > >>>>> >> > >>>>> Hi, >> > >>>>> >> > >>>>> On a 1296-core Intel machine with 376 GB, setting >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> -DLLVM_PARALLEL_LINK_JOB=1 >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> does not help (switching back to ld scales) see: >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> [5085/5201] Linking CXX executable bin/clang-11 >> > >>>>> FAILED: bin/clang-11 >> > >>>>> : && /home/usr4/c74014i/opt/clang/current/bin/clang++ >> -stdlib=libc++ >> > >>>>> -fPIC -fvisibility-inlines-hidden -Werror=date-time >> > >>>>> -Werror=unguarded-availability-new -Wall -Wextra >> -Wno-unused-parameter >> > >>>>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers >> -pedantic >> > >>>>> -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default >> > >>>>> -Wno-noexcept-type -Wnon-virtual-dtor >> -Wdelete-non-virtual-dtor >> > >>>>> -Wstring-conversion -fdiagnostics-color -ffunction-sections >> -fdata-sections >> > >>>>> -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 >> -DNDEBUG >> > >>>>> -stdlib=libc++ -fuse-ld=lld -Wl,--color-diagnostics >> > >>>>> -Wl,-allow-shlib-undefined -Wl,--export-dynamic -Wl,-O3 >> > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o >> > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o >> > >>>>> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o >> > >>>>> >> tools/clang/tools/driver/CMakeFiles/clang.dir/cc1gen_reproducer_main.cpp.o >> > >>>>> -o bin/clang-11 -Wl,-rpath,"\$ORIGIN/../lib" >> lib/libLLVMAArch64CodeGen.a >> > >>>>> lib/libLLVMAArch64AsmParser.a lib/libLLVMAArch64Desc.a >> > >>>>> lib/libLLVMAArch64Disassembler.a lib/libLLVMAArch64Info.a >> > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUCodeGen.a >> > >>>>> lib/libLLVMAMDGPUAsmParser.a lib/libLLVMAMDGPUDesc.a >> > >>>>> lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUInfo.a >> > >>>>> lib/libLLVMAMDGPUUtils.a lib/libLLVMARMCodeGen.a >> > >>>>> lib/libLLVMARMAsmParser.a lib/libLLVMARMDesc.a >> > >>>>> lib/libLLVMARMDisassembler.a lib/libLLVMARMInfo.a >> lib/libLLVMARMUtils.a >> > >>>>> lib/libLLVMAVRCodeGen.a lib/libLLVMAVRAsmParser.a >> lib/libLLVMAVRDesc.a >> > >>>>> lib/libLLVMAVRDisassembler.a lib/libLLVMAVRInfo.a >> > >>>>> lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a >> lib/libLLVMBPFDesc.a >> > >>>>> lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a >> > >>>>> lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a >> > >>>>> lib/libLLVMHexagonDesc.a lib/libLLVMHexagonDisassembler.a >> > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a >> > >>>>> lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiDesc.a >> > >>>>> lib/libLLVMLanaiDisassembler.a lib/libLLVMLanaiInfo.a >> > >>>>> lib/libLLVMMipsCodeGen.a lib/libLLVMMipsAsmParser.a >> > >>>>> lib/libLLVMMipsDesc.a lib/libLLVMMipsDisassembler.a >> > >>>>> lib/libLLVMMipsInfo.a lib/libLLVMMSP430CodeGen.a >> > >>>>> lib/libLLVMMSP430AsmParser.a lib/libLLVMMSP430Desc.a >> > >>>>> lib/libLLVMMSP430Disassembler.a lib/libLLVMMSP430Info.a >> > >>>>> lib/libLLVMNVPTXCodeGen.a lib/libLLVMNVPTXDesc.a >> lib/libLLVMNVPTXInfo.a >> > >>>>> lib/libLLVMPowerPCCodeGen.a lib/libLLVMPowerPCAsmParser.a >> > >>>>> lib/libLLVMPowerPCDesc.a lib/libLLVMPowerPCDisassembler.a >> > >>>>> lib/libLLVMPowerPCInfo.a lib/libLLVMRISCVCodeGen.a >> > >>>>> lib/libLLVMRISCVAsmParser.a lib/libLLVMRISCVDesc.a >> > >>>>> lib/libLLVMRISCVDisassembler.a lib/libLLVMRISCVInfo.a >> > >>>>> lib/libLLVMRISCVUtils.a lib/libLLVMSparcCodeGen.a >> > >>>>> lib/libLLVMSparcAsmParser.a lib/libLLVMSparcDesc.a >> > >>>>> lib/libLLVMSparcDisassembler.a lib/libLLVMSparcInfo.a >> > >>>>> lib/libLLVMSystemZCodeGen.a lib/libLLVMSystemZAsmParser.a >> > >>>>> lib/libLLVMSystemZDesc.a lib/libLLVMSystemZDisassembler.a >> > >>>>> lib/libLLVMSystemZInfo.a lib/libLLVMWebAssemblyCodeGen.a >> > >>>>> lib/libLLVMWebAssemblyAsmParser.a >> lib/libLLVMWebAssemblyDesc.a >> > >>>>> lib/libLLVMWebAssemblyDisassembler.a >> lib/libLLVMWebAssemblyInfo.a >> > >>>>> lib/libLLVMX86CodeGen.a lib/libLLVMX86AsmParser.a >> lib/libLLVMX86Desc.a >> > >>>>> lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a >> lib/libLLVMX86Utils.a >> > >>>>> lib/libLLVMXCoreCodeGen.a lib/libLLVMXCoreDesc.a >> > >>>>> lib/libLLVMXCoreDisassembler.a lib/libLLVMXCoreInfo.a >> > >>>>> lib/libLLVMAnalysis.a lib/libLLVMCodeGen.a >> lib/libLLVMCore.a >> > >>>>> lib/libLLVMipo.a lib/libLLVMAggressiveInstCombine.a >> > >>>>> lib/libLLVMInstCombine.a lib/libLLVMInstrumentation.a >> lib/libLLVMMC.a >> > >>>>> lib/libLLVMMCParser.a lib/libLLVMObjCARCOpts.a >> lib/libLLVMOption.a >> > >>>>> lib/libLLVMScalarOpts.a lib/libLLVMSupport.a >> lib/libLLVMTransformUtils.a >> > >>>>> lib/libLLVMVectorize.a -lpthread lib/libclangBasic.a >> > >>>>> lib/libclangCodeGen.a lib/libclangDriver.a >> lib/libclangFrontend.a >> > >>>>> lib/libclangFrontendTool.a lib/libclangSerialization.a >> > >>>>> lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a >> > >>>>> lib/libLLVMAArch64Utils.a lib/libLLVMMIRParser.a >> lib/libLLVMAMDGPUDesc.a >> > >>>>> lib/libLLVMAMDGPUInfo.a lib/libLLVMAMDGPUUtils.a >> lib/libLLVMARMDesc.a >> > >>>>> lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a >> lib/libLLVMHexagonDesc.a >> > >>>>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiDesc.a >> lib/libLLVMLanaiInfo.a >> > >>>>> lib/libLLVMSystemZDesc.a lib/libLLVMSystemZInfo.a >> > >>>>> lib/libLLVMWebAssemblyDesc.a lib/libLLVMWebAssemblyInfo.a >> > >>>>> lib/libLLVMCFGuard.a lib/libLLVMGlobalISel.a >> lib/libLLVMAsmPrinter.a >> > >>>>> lib/libLLVMDebugInfoDWARF.a lib/libLLVMSelectionDAG.a >> > >>>>> lib/libLLVMMCDisassembler.a lib/libclangCodeGen.a >> lib/libLLVMCoverage.a >> > >>>>> lib/libLLVMLTO.a lib/libLLVMObjCARCOpts.a >> lib/libLLVMPasses.a >> > >>>>> lib/libLLVMCodeGen.a lib/libLLVMTarget.a >> lib/libLLVMCoroutines.a >> > >>>>> lib/libLLVMipo.a lib/libLLVMInstrumentation.a >> lib/libLLVMVectorize.a >> > >>>>> lib/libLLVMBitWriter.a lib/libLLVMIRReader.a >> lib/libLLVMAsmParser.a >> > >>>>> lib/libLLVMLinker.a lib/libLLVMScalarOpts.a >> > >>>>> lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a >> > >>>>> lib/libclangRewriteFrontend.a lib/libclangARCMigrate.a >> > >>>>> lib/libclangStaticAnalyzerFrontend.a >> lib/libclangStaticAnalyzerCheckers.a >> > >>>>> lib/libclangStaticAnalyzerCore.a lib/libclangCrossTU.a >> > >>>>> lib/libclangIndex.a lib/libclangFrontend.a >> lib/libclangDriver.a >> > >>>>> lib/libLLVMOption.a lib/libclangParse.a >> lib/libclangSerialization.a >> > >>>>> lib/libclangSema.a lib/libclangAnalysis.a >> lib/libclangASTMatchers.a >> > >>>>> lib/libclangEdit.a lib/libclangFormat.a >> lib/libclangToolingInclusions.a >> > >>>>> lib/libclangToolingCore.a lib/libclangAST.a >> lib/libLLVMFrontendOpenMP.a >> > >>>>> lib/libLLVMTransformUtils.a lib/libLLVMAnalysis.a >> > >>>>> lib/libLLVMProfileData.a lib/libLLVMObject.a >> lib/libLLVMMCParser.a >> > >>>>> lib/libLLVMBitReader.a lib/libLLVMTextAPI.a >> lib/libclangRewrite.a >> > >>>>> lib/libclangLex.a lib/libclangBasic.a lib/libLLVMCore.a >> > >>>>> lib/libLLVMRemarks.a lib/libLLVMBitstreamReader.a >> lib/libLLVMMC.a >> > >>>>> lib/libLLVMBinaryFormat.a lib/libLLVMDebugInfoCodeView.a >> > >>>>> lib/libLLVMDebugInfoMSF.a lib/libLLVMSupport.a -lz -lrt >> -ldl -ltinfo >> > >>>>> -lpthread -lm lib/libLLVMDemangle.a && : >> > >>>>> ld.lld: error: failed to open bin/clang-11: Cannot allocate >> memory >> > >>>>> >> > >>>>> _______________________________________________ >> > >>>>> LLVM Developers mailing list >> > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > >>>>> >> > >>>>> _______________________________________________ >> > >>>>> LLVM Developers mailing list >> > >>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> > >>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > >>>>> >> > >>>>> >> > >> > >_______________________________________________ >> > >LLVM Developers mailing list >> > >llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> > >https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > >> > >> > >> > _______________________________________________ >> > LLVM Developers mailing list >> > llvm-dev at lists.llvm.org >> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > >> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200402/ff10d762/attachment-0001.html>