[llvm-dev] LLD issue on a massively parallel build machine

Tom Stellard via llvm-dev
Wed, 01 Apr 2020 23:48:58 -0000

On 04/01/2020 04:12 PM, Itaru Kitayama wrote:
> On another login node which is 256 (GB)/48 (nodes) JURECA at JSC, I never
had an LLD issue without setting -j when executing ninja
> in the past few weeks.
>
> On Thu, Apr 2, 2020 at 7:17 AM Itaru Kitayama <itaru.kitayama at
gmail.com <mailto:itaru.kitayama at gmail.com>> wrote:
>
>     Tom,
>     Then what ratio do you think it’s minimal?
>

It really depends on your configuration, but I usually try to have at least 2 GB
of memory per core.  However, I usually do Release builds, so Debug builds might
need more.  If you aren't using LLVM_PARALLEL_LINK_JOBS it's pretty easy
to
run out of memory once ninja starts linking the tools and unittests.

-Tom

>     On Thu, Apr 2, 2020 at 6:11 Tom Stellard <tstellar at redhat.com
<mailto:tstellar at redhat.com>> wrote:
>
>         On 04/01/2020 01:47 PM, Itaru Kitayama via llvm-dev wrote:
>         > Thanks for the heads up the supercomputer
>         > Is down for maintenance this week.
>         > I’ll give it a try when it gets back.
>         >
>
>         This doesn't look to me like it's necessarily an lld issue.
Trying
>         to build all the sub-projects without limiting the number of ninja
jobs
>         is almost guaranteed to run out of memory on a machine like this
with such
>         a low memory:core ratio.
>
>         -Tom
>
>         > On Thu, Apr 2, 2020 at 1:49 Fangrui Song <maskray at
google.com <mailto:maskray at google.com> <mailto:maskray at google.com
<mailto:maskray at google.com>>> wrote:
>         >
>         >     On 2020-03-29, Nemanja Ivanovic via llvm-dev wrote:
>         >     >Glad you got it working.
>         >     >My suggestion about LLVM_ENABLE_THREADS didn't
work because you didn't
>         >     >apply it when building the build linker.
>         >     >
>         >     >When you don't have the ability to rebuild the
build compiler, this doesn't
>         >     >apply. In those cases, I end up doing a dirty hack
where I use a wrapper
>         >     >script with something like:
>         >     >
>         >     ><path-to-build-compiler> "$@"
-Wl,--no-threads
>         >     >
>         >     >To invoke the build compiler. That way you still get
full parallelism while
>         >     >compiling and linking, but you prevent LLD from using
threads.
>         >
>         >
>         >     https://reviews.llvm.org/D76885 (committed yesterday)
changed
>         >     --no-threads to --threads={1,2,...} (--no-threads is used
rarely (GNU ld
>         >     does not support it) so I'd rather not keep the option
for
>         >     compatibility).
>         >
>         >     You may try -Wl,--threads=1
>         >
>         >     >On Sat., Mar. 28, 2020, 4:19 p.m. Itaru Kitayama,
<itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>
<mailto:itaru.kitayama at gmail.com <mailto:itaru.kitayama at
gmail.com>>>
>         >     >wrote:
>         >     >
>         >     >> Good news, I was able to use up to 37 cores for
LLVM build with LLD.
>         >     >> The build speed, did not measure precisely
though, is comparable to the
>         >     >> build with GNU ld case.
>         >     >>
>         >     >> Thank you all for your help!
>         >     >>
>         >     >> On Sun, Mar 29, 2020 at 5:04 AM Alex
Brachet-Mialot <
>         >     >> alexbrachetmialot at gmail.com
<mailto:alexbrachetmialot at gmail.com> <mailto:alexbrachetmialot at
gmail.com <mailto:alexbrachetmialot at gmail.com>>> wrote:
>         >     >>
>         >     >>> Yes unfortunately that would limit you to 4
cores.
>         >     >>>
>         >     >>> There’s no easy way to use lld with
—no-threads with our build system at
>         >     >>> the moment unfortunately.
>         >     >>>
>         >     >>> I also had just been using ld.gold, but will
switch now that numactl
>         >     >>> works for me.
>         >     >>>
>         >     >>> On Sat, Mar 28, 2020 at 3:58 PM Itaru
Kitayama <itaru.kitayama at gmail.com <mailto:itaru.kitayama at
gmail.com> <mailto:itaru.kitayama at gmail.com <mailto:itaru.kitayama
at gmail.com>>>
>         >     >>> wrote:
>         >     >>>
>         >     >>>> That is slowing down the build visibly,
for the speed I should stick
>         >     >>>> with ld at the moment.
>         >     >>>>
>         >     >>>>
>         >     >>>> On Sun, Mar 29, 2020 at 4:42 AM Alexandre
Ganea <
>         >     >>>> alexandre.ganea at ubisoft.com
<mailto:alexandre.ganea at ubisoft.com> <mailto:alexandre.ganea at
ubisoft.com <mailto:alexandre.ganea at ubisoft.com>>> wrote:
>         >     >>>>
>         >     >>>>> Would `taskset -c 0-3 ninja check-all
-j4` work?
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> *De :* Itaru Kitayama
<itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>
<mailto:itaru.kitayama at gmail.com <mailto:itaru.kitayama at
gmail.com>>>
>         >     >>>>> *Envoyé :* March 28, 2020 3:37 PM
>         >     >>>>> *À :* Alex Brachet-Mialot
<alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>
<mailto:alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at
gmail.com>>>
>         >     >>>>> *Cc :* Alexandre Ganea
<alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at ubisoft.com>
<mailto:alexandre.ganea at ubisoft.com <mailto:alexandre.ganea at
ubisoft.com>>>; LLVM Dev <
>         >     >>>>> llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org>>>; Nemanja Ivanovic
<nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at gmail.com>
<mailto:nemanja.i.ibm at gmail.com <mailto:nemanja.i.ibm at
gmail.com>>>
>         >     >>>>> *Objet :* Re: [llvm-dev] LLD issue on
a massively parallel build
>         >     >>>>> machine
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> $ free -g
>         >     >>>>>               total        used
free      shared  buff/cache
>         >     >>>>> available
>         >     >>>>> Mem:            376         149
20           1         207
>         >     >>>>>     225
>         >     >>>>> Swap:             3           0
3
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> Too small swap size for a 72-core
login machine?
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> On Sun, Mar 29, 2020 at 4:28 AM Alex
Brachet-Mialot <
>         >     >>>>> alexbrachetmialot at gmail.com
<mailto:alexbrachetmialot at gmail.com> <mailto:alexbrachetmialot at
gmail.com <mailto:alexbrachetmialot at gmail.com>>> wrote:
>         >     >>>>>
>         >     >>>>> Enable threads is for building llvm
with or without thread support. To
>         >     >>>>> my knowledge it has no effect on the
build system.
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> Like was discussed above you could
try limiting  the number of
>         >     >>>>> processes created (assuming your lld
was after that commit) with numactl or
>         >     >>>>> taskset.
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> In your case though it looked like
you were running out of memory. You
>         >     >>>>> could try increasing your swap space?
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> Alex
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> On Sat, Mar 28, 2020 at 3:15 PM Itaru
Kitayama <
>         >     >>>>> itaru.kitayama at gmail.com
<mailto:itaru.kitayama at gmail.com> <mailto:itaru.kitayama at
gmail.com <mailto:itaru.kitayama at gmail.com>>> wrote:
>         >     >>>>>
>         >     >>>>> Hi,
>         >     >>>>>
>         >     >>>>> My configuration is below:
>         >     >>>>>
>         >     >>>>> cmake -GNinja -DLLVM_ENABLE_LLD=ON
-DLLVM_ENABLE_THREADS=OFF
>         >     >>>>> -DLLVM_ENABLE_LIBCXX=ON
-DCMAKE_BUILD_TYPE=Release
>         >     >>>>>
-DGCC_INSTALL_PREFIX=/home/usr4/c74014i/opt/gcc-7.5.0/
>         >     >>>>> -DLIBOMPTARGET_ENABLE_DEBUG=ON
>         >     >>>>>
-DCMAKE_INSTALL_PREFIX=$HOME/opt/clang/${today} -DCMAKE_C_COMPILER=clang
>         >     >>>>> -DCMAKE_C_FLAGS=""
-DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_FLAGS=""
>         >     >>>>>
-DLLVM_ENABLE_PROJECTS="clang;openmp;lld;libcxx;libcxxabi;compiler-rt;libunwind"
>         >     >>>>>
-DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_60
>         >     >>>>>
-DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=60
-DLLVM_TARGETS_TO_BUILD="all"
>         >     >>>>> /tmp/llvm-project/llvm
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> but did not help my case:
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> [...]
>         >     >>>>>
>         >     >>>>> [4802/5201] Linking CXX executable
bin/lld
>         >     >>>>> FAILED: bin/lld
>         >     >>>>> : &&
/home/usr4/c74014i/opt/clang/current/bin/clang++  -stdlib=libc++
>         >     >>>>> -fPIC -fvisibility-inlines-hidden
-Werror=date-time
>         >     >>>>> -Werror=unguarded-availability-new
-Wall -Wextra -Wno-unused-parameter
>         >     >>>>> -Wwrite-strings -Wcast-qual
-Wmissing-field-initializers -pedantic
>         >     >>>>> -Wno-long-long -Wimplicit-fallthrough
-Wcovered-switch-default
>         >     >>>>> -Wno-noexcept-type -Wnon-virtual-dtor
-Wdelete-non-virtual-dtor
>         >     >>>>> -Wstring-conversion
-fdiagnostics-color -ffunction-sections -fdata-sections
>         >     >>>>> -O3 -DNDEBUG  -stdlib=libc++
-fuse-ld=lld -Wl,--color-diagnostics
>         >     >>>>> -Wl,-allow-shlib-undefined
-Wl,--export-dynamic  -Wl,-O3
>         >     >>>>>
tools/lld/tools/lld/CMakeFiles/lld.dir/lld.cpp.o  -o bin/lld
>         >     >>>>>
-Wl,-rpath,"\$ORIGIN/../lib"  lib/libLLVMSupport.a  -lpthread
>         >     >>>>>  lib/liblldCommon.a  lib/liblldCOFF.a
lib/liblldDriver.a  lib/liblldELF.a
>         >     >>>>>  lib/liblldMinGW.a  lib/liblldWasm.a
lib/liblldMachO.a
>         >     >>>>>  lib/liblldReaderWriter.a
lib/liblldYAML.a  lib/liblldCore.a
>         >     >>>>>  lib/liblldCOFF.a
lib/libLLVMDebugInfoPDB.a  lib/libLLVMLibDriver.a
>         >     >>>>>  lib/libLLVMWindowsManifest.a
/usr/lib64/libxml2.so  lib/liblldCommon.a
>         >     >>>>>  lib/libLLVMOption.a
lib/libLLVMAArch64CodeGen.a
>         >     >>>>>  lib/libLLVMAArch64AsmParser.a
lib/libLLVMAArch64Disassembler.a
>         >     >>>>>  lib/libLLVMAArch64Desc.a
lib/libLLVMAArch64Info.a
>         >     >>>>>  lib/libLLVMAArch64Utils.a
lib/libLLVMAMDGPUCodeGen.a
>         >     >>>>>  lib/libLLVMMIRParser.a
lib/libLLVMAMDGPUAsmParser.a
>         >     >>>>>  lib/libLLVMAMDGPUDisassembler.a
lib/libLLVMAMDGPUDesc.a
>         >     >>>>>  lib/libLLVMAMDGPUInfo.a
lib/libLLVMAMDGPUUtils.a  lib/libLLVMARMCodeGen.a
>         >     >>>>>  lib/libLLVMARMAsmParser.a
lib/libLLVMARMDisassembler.a
>         >     >>>>>  lib/libLLVMARMDesc.a
lib/libLLVMARMInfo.a  lib/libLLVMARMUtils.a
>         >     >>>>>  lib/libLLVMAVRCodeGen.a
lib/libLLVMAVRAsmParser.a  lib/libLLVMAVRDesc.a
>         >     >>>>>  lib/libLLVMAVRDisassembler.a
lib/libLLVMAVRInfo.a
>         >     >>>>>  lib/libLLVMBPFCodeGen.a
lib/libLLVMBPFAsmParser.a  lib/libLLVMBPFDesc.a
>         >     >>>>>  lib/libLLVMBPFDisassembler.a
lib/libLLVMBPFInfo.a
>         >     >>>>>  lib/libLLVMHexagonCodeGen.a
lib/libLLVMHexagonAsmParser.a
>         >     >>>>>  lib/libLLVMHexagonDisassembler.a
lib/libLLVMHexagonDesc.a
>         >     >>>>>  lib/libLLVMHexagonInfo.a
lib/libLLVMLanaiCodeGen.a
>         >     >>>>>  lib/libLLVMLanaiAsmParser.a
lib/libLLVMLanaiDisassembler.a
>         >     >>>>>  lib/libLLVMLanaiDesc.a
lib/libLLVMLanaiInfo.a  lib/libLLVMMipsCodeGen.a
>         >     >>>>>  lib/libLLVMMipsAsmParser.a
lib/libLLVMMipsDesc.a
>         >     >>>>>  lib/libLLVMMipsDisassembler.a
lib/libLLVMMipsInfo.a
>         >     >>>>>  lib/libLLVMMSP430CodeGen.a
lib/libLLVMMSP430AsmParser.a
>         >     >>>>>  lib/libLLVMMSP430Desc.a
lib/libLLVMMSP430Disassembler.a
>         >     >>>>>  lib/libLLVMMSP430Info.a
lib/libLLVMNVPTXCodeGen.a  lib/libLLVMNVPTXDesc.a
>         >     >>>>>  lib/libLLVMNVPTXInfo.a
lib/libLLVMPowerPCCodeGen.a
>         >     >>>>>  lib/libLLVMPowerPCAsmParser.a
lib/libLLVMPowerPCDesc.a
>         >     >>>>>  lib/libLLVMPowerPCDisassembler.a
lib/libLLVMPowerPCInfo.a
>         >     >>>>>  lib/libLLVMRISCVCodeGen.a
lib/libLLVMRISCVAsmParser.a
>         >     >>>>>  lib/libLLVMRISCVDesc.a
lib/libLLVMRISCVDisassembler.a
>         >     >>>>>  lib/libLLVMRISCVInfo.a
lib/libLLVMRISCVUtils.a  lib/libLLVMSparcCodeGen.a
>         >     >>>>>  lib/libLLVMSparcAsmParser.a
lib/libLLVMSparcDesc.a
>         >     >>>>>  lib/libLLVMSparcDisassembler.a
lib/libLLVMSparcInfo.a
>         >     >>>>>  lib/libLLVMSystemZCodeGen.a
lib/libLLVMSystemZAsmParser.a
>         >     >>>>>  lib/libLLVMSystemZDisassembler.a
lib/libLLVMSystemZDesc.a
>         >     >>>>>  lib/libLLVMSystemZInfo.a
lib/libLLVMWebAssemblyCodeGen.a
>         >     >>>>>  lib/libLLVMWebAssemblyAsmParser.a
lib/libLLVMWebAssemblyDisassembler.a
>         >     >>>>>  lib/libLLVMWebAssemblyDesc.a
lib/libLLVMWebAssemblyInfo.a
>         >     >>>>>  lib/libLLVMX86CodeGen.a
lib/libLLVMCFGuard.a  lib/libLLVMGlobalISel.a
>         >     >>>>>  lib/libLLVMX86AsmParser.a
lib/libLLVMX86Desc.a
>         >     >>>>>  lib/libLLVMX86Disassembler.a
lib/libLLVMX86Info.a  lib/libLLVMX86Utils.a
>         >     >>>>>  lib/libLLVMXCoreCodeGen.a
lib/libLLVMAsmPrinter.a
>         >     >>>>>  lib/libLLVMDebugInfoDWARF.a
lib/libLLVMSelectionDAG.a
>         >     >>>>>  lib/libLLVMXCoreDesc.a
lib/libLLVMXCoreDisassembler.a
>         >     >>>>>  lib/libLLVMMCDisassembler.a
lib/libLLVMXCoreInfo.a  lib/libLLVMLTO.a
>         >     >>>>>  lib/libLLVMObjCARCOpts.a
lib/libLLVMPasses.a  lib/libLLVMCodeGen.a
>         >     >>>>>  lib/libLLVMTarget.a
lib/libLLVMCoroutines.a  lib/libLLVMipo.a
>         >     >>>>>  lib/libLLVMBitWriter.a
lib/libLLVMScalarOpts.a  lib/libLLVMVectorize.a
>         >     >>>>>  lib/libLLVMAggressiveInstCombine.a
lib/libLLVMInstCombine.a
>         >     >>>>>  lib/libLLVMLinker.a
lib/libLLVMFrontendOpenMP.a  lib/libLLVMIRReader.a
>         >     >>>>>  lib/libLLVMAsmParser.a
lib/libLLVMInstrumentation.a
>         >     >>>>>  lib/libLLVMTransformUtils.a
lib/libLLVMAnalysis.a  lib/libLLVMObject.a
>         >     >>>>>  lib/libLLVMBitReader.a
lib/libLLVMMCParser.a  lib/libLLVMMC.a
>         >     >>>>>  lib/libLLVMDebugInfoCodeView.a
lib/libLLVMDebugInfoMSF.a
>         >     >>>>>  lib/libLLVMTextAPI.a
lib/libLLVMProfileData.a  lib/libLLVMCore.a
>         >     >>>>>  lib/libLLVMBinaryFormat.a
lib/libLLVMRemarks.a
>         >     >>>>>  lib/libLLVMBitstreamReader.a
lib/libLLVMSupport.a  -lz  -lrt  -ldl
>         >     >>>>>  -ltinfo  -lpthread  -lm
lib/libLLVMDemangle.a && cd
>         >     >>>>>
/tmp/build/202003290405/tools/lld/tools/lld &&
>         >     >>>>>
/home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
>         >     >>>>> create_symlink lld
/tmp/build/202003290405/./bin/lld-link && cd
>         >     >>>>>
/tmp/build/202003290405/tools/lld/tools/lld &&
>         >     >>>>>
/home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
>         >     >>>>> create_symlink lld
/tmp/build/202003290405/./bin/ld.lld && cd
>         >     >>>>>
/tmp/build/202003290405/tools/lld/tools/lld &&
>         >     >>>>>
/home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
>         >     >>>>> create_symlink lld
/tmp/build/202003290405/./bin/ld64.lld && cd
>         >     >>>>>
/tmp/build/202003290405/tools/lld/tools/lld &&
>         >     >>>>>
/home/usr4/c74014i/opt/cmake-3.16.3-Linux-x86_64/bin/cmake -E
>         >     >>>>> create_symlink lld
/tmp/build/202003290405/./bin/wasm-ld
>         >     >>>>> terminating with uncaught exception
of type std::__1::system_error:
>         >     >>>>> thread constructor failed: Resource
temporarily unavailable
>         >     >>>>> PLEASE submit a bug report to
https://bugs.llvm.org/ and include the
>         >     >>>>> crash backtrace.
>         >     >>>>>  #0 0x00000000007924f4
PrintStackTraceSignalHandler(void*)
>         >     >>>>>
(/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7924f4)
>         >     >>>>>  #1 0x00000000007902ce
llvm::sys::RunSignalHandlers()
>         >     >>>>>
(/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x7902ce)
>         >     >>>>>  #2 0x0000000000792bd5
SignalHandler(int)
>         >     >>>>>
(/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x792bd5)
>         >     >>>>>  #3 0x00007fddd54bb630 __restore_rt
(/lib64/libpthread.so.0+0xf630)
>         >     >>>>>  #4 0x00007fddd3b40377 raise
>         >     >>>>>
/usr/src/debug/glibc-2.17-c758a686/signal/../nptl/sysdeps/unix/sysv/linux/raise.c:55:0
>         >     >>>>>  #5 0x00007fddd3b41a68 abort
>         >     >>>>>
/usr/src/debug/glibc-2.17-c758a686/stdlib/abort.c:92:0
>         >     >>>>>  #6 0x00007fddd411ae8b
>         >     >>>>>
(/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2be8b)
>         >     >>>>>  #7 0x00007fddd40fdb9f
demangling_terminate_handler()
>         >     >>>>>
(/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0xeb9f)
>         >     >>>>>  #8 0x00007fddd411a0a3
std::__terminate(void (*)())
>         >     >>>>>
(/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2b0a3)
>         >     >>>>>  #9 0x00007fddd411ce06
>         >     >>>>>
(/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2de06)
>         >     >>>>> #10 0x00007fddd411cd9f
>         >     >>>>>
(/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++abi.so.1+0x2dd9f)
>         >     >>>>> #11 0x00007fddd43bded4
std::__1::__throw_system_error(int, char const*)
>         >     >>>>>
(/home/usr4/c74014i/opt/clang/202003290200/bin/../lib/libc++.so.1+0x90ed4)
>         >     >>>>> #12 0x0000000002a7a316 void*
>         >     >>>>>
std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct,
>         >     >>>>>
std::__1::default_delete<std::__1::__thread_struct> >,
>         >     >>>>> llvm::parallel::detail::(anonymous
>         >     >>>>>
namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::'lambda'()>
>         >     >>>>> >(void*)
(/home/usr4/c74014i/opt/clang/current/bin/ld.lld+0x2a7a316)
>         >     >>>>> #13 0x00007fddd54b3ea5 start_thread
>         >     >>>>>
/usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307:0
>         >     >>>>> #14 0x00007fddd3c088cd clone
>         >     >>>>>
/usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:113:0
>         >     >>>>> clang-11: error: unable to execute
command: Aborted
>         >     >>>>> clang-11: error: linker command
failed due to signal (use -v to see
>         >     >>>>> invocation)
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> On Sun, Mar 29, 2020 at 1:44 AM
Nemanja Ivanovic <
>         >     >>>>> nemanja.i.ibm at gmail.com
<mailto:nemanja.i.ibm at gmail.com> <mailto:nemanja.i.ibm at gmail.com
<mailto:nemanja.i.ibm at gmail.com>>> wrote:
>         >     >>>>>
>         >     >>>>> The discussion in
https://reviews.llvm.org/D76240
>         >     >>>>>
>         >     >>>>> may be of interest to the
participants of this thread. Specifically,
>         >     >>>>> Fangrui's planned work on a
command line option to limit threads for LLD.
>         >     >>>>> Until that becomes a reality, you
might want to use the CMake variable
>         >     >>>>> -DLLVM_ENABLE_THREADS=OFF when
building the LLD that you are using to
>         >     >>>>> prevent parallelism.
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> On Sat, Mar 28, 2020 at 12:32 PM
Alexandre Ganea via llvm-dev <
>         >     >>>>> llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org>>> wrote:
>         >     >>>>>
>         >     >>>>> Alex :
>         >     >>>>>
>         >     >>>>> Can you please try `numactl` or
`taskset` after
>         >     >>>>>
https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74
>         >     >>>>> ?
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> There was a tiny bug in how
*sched_getaffinity*() was used, see:
>         >     >>>>>
https://reviews.llvm.org/D75153#1942336
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> *De :* Alex Brachet-Mialot
<alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at gmail.com>
<mailto:alexbrachetmialot at gmail.com <mailto:alexbrachetmialot at
gmail.com>>>
>         >     >>>>> *Envoyé :* March 28, 2020 12:11 PM
>         >     >>>>> *À :* Itaru Kitayama
<itaru.kitayama at gmail.com <mailto:itaru.kitayama at gmail.com>
<mailto:itaru.kitayama at gmail.com <mailto:itaru.kitayama at
gmail.com>>>
>         >     >>>>> *Cc :* LLVM Dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>>; Alexandre
Ganea <
>         >     >>>>> alexandre.ganea at ubisoft.com
<mailto:alexandre.ganea at ubisoft.com> <mailto:alexandre.ganea at
ubisoft.com <mailto:alexandre.ganea at ubisoft.com>>>; Fāng-ruì Sòng
<maskray at google.com <mailto:maskray at google.com>
<mailto:maskray at google.com <mailto:maskray at google.com>>>
>         >     >>>>> *Objet :* Re: [llvm-dev] LLD issue on
a massively parallel build
>         >     >>>>> machine
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> I also run into problems with lld on
machines with lots of cores (208),
>         >     >>>>> but I run into rlimits for number of
processes I can create, something
>         >     >>>>> outside of my control.
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> Unfortunately things like numactl or
taskset do not limit the
>         >     >>>>> concurrency of lld currently.
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> On Sat, Mar 28, 2020 at 4:10 AM Itaru
Kitayama via llvm-dev <
>         >     >>>>> llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org>>> wrote:
>         >     >>>>>
>         >     >>>>> Hi,
>         >     >>>>>
>         >     >>>>> On a 1296-core Intel machine with 376
GB, setting
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> -DLLVM_PARALLEL_LINK_JOB=1
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> does not help (switching back to ld
scales) see:
>         >     >>>>>
>         >     >>>>>
>         >     >>>>>
>         >     >>>>> [5085/5201] Linking CXX executable
bin/clang-11
>         >     >>>>> FAILED: bin/clang-11
>         >     >>>>> : &&
/home/usr4/c74014i/opt/clang/current/bin/clang++  -stdlib=libc++
>         >     >>>>> -fPIC -fvisibility-inlines-hidden
-Werror=date-time
>         >     >>>>> -Werror=unguarded-availability-new
-Wall -Wextra -Wno-unused-parameter
>         >     >>>>> -Wwrite-strings -Wcast-qual
-Wmissing-field-initializers -pedantic
>         >     >>>>> -Wno-long-long -Wimplicit-fallthrough
-Wcovered-switch-default
>         >     >>>>> -Wno-noexcept-type -Wnon-virtual-dtor
-Wdelete-non-virtual-dtor
>         >     >>>>> -Wstring-conversion
-fdiagnostics-color -ffunction-sections -fdata-sections
>         >     >>>>> -fno-common -Woverloaded-virtual
-Wno-nested-anon-types -O3 -DNDEBUG
>         >     >>>>>  -stdlib=libc++ -fuse-ld=lld
-Wl,--color-diagnostics
>         >     >>>>> -Wl,-allow-shlib-undefined
-Wl,--export-dynamic  -Wl,-O3
>         >     >>>>>
tools/clang/tools/driver/CMakeFiles/clang.dir/driver.cpp.o
>         >     >>>>>
tools/clang/tools/driver/CMakeFiles/clang.dir/cc1_main.cpp.o
>         >     >>>>>
tools/clang/tools/driver/CMakeFiles/clang.dir/cc1as_main.cpp.o
>         >     >>>>>
tools/clang/tools/driver/CMakeFiles/clang.dir/cc1gen_reproducer_main.cpp.o
>         >     >>>>>  -o bin/clang-11
-Wl,-rpath,"\$ORIGIN/../lib"  lib/libLLVMAArch64CodeGen.a
>         >     >>>>>  lib/libLLVMAArch64AsmParser.a
lib/libLLVMAArch64Desc.a
>         >     >>>>>  lib/libLLVMAArch64Disassembler.a
lib/libLLVMAArch64Info.a
>         >     >>>>>  lib/libLLVMAArch64Utils.a
lib/libLLVMAMDGPUCodeGen.a
>         >     >>>>>  lib/libLLVMAMDGPUAsmParser.a
lib/libLLVMAMDGPUDesc.a
>         >     >>>>>  lib/libLLVMAMDGPUDisassembler.a
lib/libLLVMAMDGPUInfo.a
>         >     >>>>>  lib/libLLVMAMDGPUUtils.a
lib/libLLVMARMCodeGen.a
>         >     >>>>>  lib/libLLVMARMAsmParser.a
lib/libLLVMARMDesc.a
>         >     >>>>>  lib/libLLVMARMDisassembler.a
lib/libLLVMARMInfo.a  lib/libLLVMARMUtils.a
>         >     >>>>>  lib/libLLVMAVRCodeGen.a
lib/libLLVMAVRAsmParser.a  lib/libLLVMAVRDesc.a
>         >     >>>>>  lib/libLLVMAVRDisassembler.a
lib/libLLVMAVRInfo.a
>         >     >>>>>  lib/libLLVMBPFCodeGen.a
lib/libLLVMBPFAsmParser.a  lib/libLLVMBPFDesc.a
>         >     >>>>>  lib/libLLVMBPFDisassembler.a
lib/libLLVMBPFInfo.a
>         >     >>>>>  lib/libLLVMHexagonCodeGen.a
lib/libLLVMHexagonAsmParser.a
>         >     >>>>>  lib/libLLVMHexagonDesc.a
lib/libLLVMHexagonDisassembler.a
>         >     >>>>>  lib/libLLVMHexagonInfo.a
lib/libLLVMLanaiCodeGen.a
>         >     >>>>>  lib/libLLVMLanaiAsmParser.a
lib/libLLVMLanaiDesc.a
>         >     >>>>>  lib/libLLVMLanaiDisassembler.a
lib/libLLVMLanaiInfo.a
>         >     >>>>>  lib/libLLVMMipsCodeGen.a
lib/libLLVMMipsAsmParser.a
>         >     >>>>>  lib/libLLVMMipsDesc.a
lib/libLLVMMipsDisassembler.a
>         >     >>>>>  lib/libLLVMMipsInfo.a
lib/libLLVMMSP430CodeGen.a
>         >     >>>>>  lib/libLLVMMSP430AsmParser.a
lib/libLLVMMSP430Desc.a
>         >     >>>>>  lib/libLLVMMSP430Disassembler.a
lib/libLLVMMSP430Info.a
>         >     >>>>>  lib/libLLVMNVPTXCodeGen.a
lib/libLLVMNVPTXDesc.a  lib/libLLVMNVPTXInfo.a
>         >     >>>>>  lib/libLLVMPowerPCCodeGen.a
lib/libLLVMPowerPCAsmParser.a
>         >     >>>>>  lib/libLLVMPowerPCDesc.a
lib/libLLVMPowerPCDisassembler.a
>         >     >>>>>  lib/libLLVMPowerPCInfo.a
lib/libLLVMRISCVCodeGen.a
>         >     >>>>>  lib/libLLVMRISCVAsmParser.a
lib/libLLVMRISCVDesc.a
>         >     >>>>>  lib/libLLVMRISCVDisassembler.a
lib/libLLVMRISCVInfo.a
>         >     >>>>>  lib/libLLVMRISCVUtils.a
lib/libLLVMSparcCodeGen.a
>         >     >>>>>  lib/libLLVMSparcAsmParser.a
lib/libLLVMSparcDesc.a
>         >     >>>>>  lib/libLLVMSparcDisassembler.a
lib/libLLVMSparcInfo.a
>         >     >>>>>  lib/libLLVMSystemZCodeGen.a
lib/libLLVMSystemZAsmParser.a
>         >     >>>>>  lib/libLLVMSystemZDesc.a
lib/libLLVMSystemZDisassembler.a
>         >     >>>>>  lib/libLLVMSystemZInfo.a
lib/libLLVMWebAssemblyCodeGen.a
>         >     >>>>>  lib/libLLVMWebAssemblyAsmParser.a
lib/libLLVMWebAssemblyDesc.a
>         >     >>>>>  lib/libLLVMWebAssemblyDisassembler.a
lib/libLLVMWebAssemblyInfo.a
>         >     >>>>>  lib/libLLVMX86CodeGen.a
lib/libLLVMX86AsmParser.a  lib/libLLVMX86Desc.a
>         >     >>>>>  lib/libLLVMX86Disassembler.a
lib/libLLVMX86Info.a  lib/libLLVMX86Utils.a
>         >     >>>>>  lib/libLLVMXCoreCodeGen.a
lib/libLLVMXCoreDesc.a
>         >     >>>>>  lib/libLLVMXCoreDisassembler.a
lib/libLLVMXCoreInfo.a
>         >     >>>>>  lib/libLLVMAnalysis.a
lib/libLLVMCodeGen.a  lib/libLLVMCore.a
>         >     >>>>>  lib/libLLVMipo.a
lib/libLLVMAggressiveInstCombine.a
>         >     >>>>>  lib/libLLVMInstCombine.a
lib/libLLVMInstrumentation.a  lib/libLLVMMC.a
>         >     >>>>>  lib/libLLVMMCParser.a
lib/libLLVMObjCARCOpts.a  lib/libLLVMOption.a
>         >     >>>>>  lib/libLLVMScalarOpts.a
lib/libLLVMSupport.a  lib/libLLVMTransformUtils.a
>         >     >>>>>  lib/libLLVMVectorize.a  -lpthread
lib/libclangBasic.a
>         >     >>>>>  lib/libclangCodeGen.a
lib/libclangDriver.a  lib/libclangFrontend.a
>         >     >>>>>  lib/libclangFrontendTool.a
lib/libclangSerialization.a
>         >     >>>>>  lib/libLLVMAArch64Desc.a
lib/libLLVMAArch64Info.a
>         >     >>>>>  lib/libLLVMAArch64Utils.a
lib/libLLVMMIRParser.a  lib/libLLVMAMDGPUDesc.a
>         >     >>>>>  lib/libLLVMAMDGPUInfo.a
lib/libLLVMAMDGPUUtils.a  lib/libLLVMARMDesc.a
>         >     >>>>>  lib/libLLVMARMInfo.a
lib/libLLVMARMUtils.a  lib/libLLVMHexagonDesc.a
>         >     >>>>>  lib/libLLVMHexagonInfo.a
lib/libLLVMLanaiDesc.a  lib/libLLVMLanaiInfo.a
>         >     >>>>>  lib/libLLVMSystemZDesc.a
lib/libLLVMSystemZInfo.a
>         >     >>>>>  lib/libLLVMWebAssemblyDesc.a
lib/libLLVMWebAssemblyInfo.a
>         >     >>>>>  lib/libLLVMCFGuard.a
lib/libLLVMGlobalISel.a  lib/libLLVMAsmPrinter.a
>         >     >>>>>  lib/libLLVMDebugInfoDWARF.a
lib/libLLVMSelectionDAG.a
>         >     >>>>>  lib/libLLVMMCDisassembler.a
lib/libclangCodeGen.a  lib/libLLVMCoverage.a
>         >     >>>>>  lib/libLLVMLTO.a
lib/libLLVMObjCARCOpts.a  lib/libLLVMPasses.a
>         >     >>>>>  lib/libLLVMCodeGen.a
lib/libLLVMTarget.a  lib/libLLVMCoroutines.a
>         >     >>>>>  lib/libLLVMipo.a
lib/libLLVMInstrumentation.a  lib/libLLVMVectorize.a
>         >     >>>>>  lib/libLLVMBitWriter.a
lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a
>         >     >>>>>  lib/libLLVMLinker.a
lib/libLLVMScalarOpts.a
>         >     >>>>>  lib/libLLVMAggressiveInstCombine.a
lib/libLLVMInstCombine.a
>         >     >>>>>  lib/libclangRewriteFrontend.a
lib/libclangARCMigrate.a
>         >     >>>>>  lib/libclangStaticAnalyzerFrontend.a
lib/libclangStaticAnalyzerCheckers.a
>         >     >>>>>  lib/libclangStaticAnalyzerCore.a
lib/libclangCrossTU.a
>         >     >>>>>  lib/libclangIndex.a
lib/libclangFrontend.a  lib/libclangDriver.a
>         >     >>>>>  lib/libLLVMOption.a
lib/libclangParse.a  lib/libclangSerialization.a
>         >     >>>>>  lib/libclangSema.a
lib/libclangAnalysis.a  lib/libclangASTMatchers.a
>         >     >>>>>  lib/libclangEdit.a
lib/libclangFormat.a  lib/libclangToolingInclusions.a
>         >     >>>>>  lib/libclangToolingCore.a
lib/libclangAST.a  lib/libLLVMFrontendOpenMP.a
>         >     >>>>>  lib/libLLVMTransformUtils.a
lib/libLLVMAnalysis.a
>         >     >>>>>  lib/libLLVMProfileData.a
lib/libLLVMObject.a  lib/libLLVMMCParser.a
>         >     >>>>>  lib/libLLVMBitReader.a
lib/libLLVMTextAPI.a  lib/libclangRewrite.a
>         >     >>>>>  lib/libclangLex.a
lib/libclangBasic.a  lib/libLLVMCore.a
>         >     >>>>>  lib/libLLVMRemarks.a
lib/libLLVMBitstreamReader.a  lib/libLLVMMC.a
>         >     >>>>>  lib/libLLVMBinaryFormat.a
lib/libLLVMDebugInfoCodeView.a
>         >     >>>>>  lib/libLLVMDebugInfoMSF.a
lib/libLLVMSupport.a  -lz  -lrt  -ldl  -ltinfo
>         >     >>>>>  -lpthread  -lm
lib/libLLVMDemangle.a && :
>         >     >>>>> ld.lld: error: failed to open
bin/clang-11: Cannot allocate memory
>         >     >>>>>
>         >     >>>>>
_______________________________________________
>         >     >>>>> LLVM Developers mailing list
>         >     >>>>> llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org>>
>         >     >>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>         >     >>>>>
>         >     >>>>>
_______________________________________________
>         >     >>>>> LLVM Developers mailing list
>         >     >>>>> llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org
<mailto:llvm-dev at lists.llvm.org>>
>         >     >>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>         >     >>>>>
>         >     >>>>>
>         >
>         >     >_______________________________________________
>         >     >LLVM Developers mailing list
>         >     >llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>>
>         >
>https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>         >
>         >
>         >
>         > _______________________________________________
>         > LLVM Developers mailing list
>         > llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>         > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>         >
>