llvm dev - Sep 2016 - (Thin)LTO llvm build

On Fri, Sep 16, 2016 at 5:00 PM, Mehdi Amini via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
> On Sep 16, 2016, at 4:59 PM, Mehdi Amini via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>
> On Sep 16, 2016, at 4:46 PM, Carsten Mattner <carstenmattner at
gmail.com>
> wrote:
>
> On Sat, Sep 17, 2016 at 12:48 AM, Teresa Johnson <tejohnson at
google.com>
> wrote:
>
>
> On Fri, Sep 16, 2016 at 2:54 PM, Carsten Mattner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>
> On Fri, Sep 16, 2016 at 11:28 PM, Mehdi Amini <mehdi.amini at
apple.com>
> wrote:
>
> You probably missed -DLLVM_BINUTILS_INCDIR.
>
> See: http://llvm.org/docs/GoldPlugin.html
>
>
> plugin-api.h is in /usr/include, so I'd expect it to be found, but I
> can explicitly set BINUTILS_INCDIR and re-bootstrap with gcc 6.2.1.
>
> I have ld.gold, but I'm not sure if /usr/bin/ld uses it, though I'd
expect
> it to since it's been in for a couple releases now.
>
> $ ld -v
> GNU ld (GNU Binutils) 2.27
> $ ld.bfd -v
> GNU ld (GNU Binutils) 2.27
> $ ld.gold -v
> GNU gold (GNU Binutils 2.27) 1.12
>
>
>
> Looks like your default ld is GNU ld.bfd not ld.gold. You can either
> change your
> /usr/bin/ld (which probably is a link to /usr/bin/ld.bfd) to point instead
> to
> /usr/bin/ld.gold, or if you prefer, set your PATH before the stage1
> compile to a
> location that has ld linked to ld.gold.
>
>
> I can look into and check why Arch Linux has it configured like that.
>
> In the meantime, Mehdi's suggestion to explicitly pass BINUTILS_INCDIR
> restored the previous configure behavior, and the new llvm build has
> lib/LLVMgold.so. Thanks to both of you for pointing out the missing cmake
> flag.
>
> I've checked the configure step and it didn't fail as it did
before, but
> before
> I try to build in ThinLTO mode: since the configure step checks for the
> gold
> plugin, is it safe to assume that I don't have to change the default
system
> ld to gold and ThinLTO will work, or is that a build requirement for
> bootstrapping llvm in ThinLTO mode?
>
>
Yeah, perhaps this is working somehow anyway.

>
> Try to build llvm-tblgen, you’ll know quite quickly :)
>
>
> Also, you should limit the number of parallel link jobs: cmake
> -DLLVM_PARALLEL_LINK_JOBS=1
> And use ninja if you don’t already:  cmake -GNinja
>
Yes and to add on - the ThinLTO backend by default will kick off
 std::thread::hardware_concurrency # of threads, which I'm finding is too
much for machines with hyperthreading. If that ends up being an issue I can
give you a workaround (I've been struggling to find a way that works on
various OS and arches to compute the max number of physical cores to fix
this in the source).

Teresa

> —
> Mehdi
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>

-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |  408-460-2413
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160916/35d87c90/attachment.html>

On Sat, Sep 17, 2016 at 2:07 AM, Teresa Johnson via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Yes and to add on - the ThinLTO backend by default will
> kick off  std::thread::hardware_concurrency # of threads, which I'm
finding is
Is it just me or does that sound not very intuitive or at least a
little unexpected?
It's good that it uses the resources eagerly, but in terms of build systems
this
is certainly surprising if there's no control of that parameter via
make/ninja/xcode.
> too much for machines with hyperthreading. If that ends up being an issue I
can
> give you a workaround (I've been struggling to find a way that works on
various
> OS and arches to compute the max number of physical cores to fix this in
the source).
I've been using ninja -jN so far. I suppose when building with ThinLTO I
should
run ninja -j1. Would that

What's the workaround?

> On Sep 16, 2016, at 6:13 PM, Carsten Mattner <carstenmattner at
gmail.com> wrote:
> 
> On Sat, Sep 17, 2016 at 2:07 AM, Teresa Johnson via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> 
>> Yes and to add on - the ThinLTO backend by default will
>> kick off  std::thread::hardware_concurrency # of threads, which I'm
finding is
> 
> Is it just me or does that sound not very intuitive or at least a
> little unexpected?
> It's good that it uses the resources eagerly, but in terms of build
systems this
> is certainly surprising if there's no control of that parameter via
> make/ninja/xcode.
You can control the parallelism used by the linker, but the option is linker
dependent
(On MacOS:   -Wl,-mllvm,-threads=1)
> 
>> too much for machines with hyperthreading. If that ends up being an
issue I can
>> give you a workaround (I've been struggling to find a way that
works on various
>> OS and arches to compute the max number of physical cores to fix this
in the source).
> 
> I've been using ninja -jN so far. I suppose when building with ThinLTO
I should
> run ninja -j1. Would that
> 
> What's the workaround?
Seems like you missed my previous email: : cmake -DLLVM_PARALLEL_LINK_JOBS=1
Also, ninja is parallel by default, so no need to pass -j.

This way you get nice parallelism during the compile phase, and ninja will issue
only one link job at a time.

— 
Mehdi