llvm dev - May 2017 - Should we split llvm Support and ADT?

On Tue, May 30, 2017 at 12:52 PM Bob Haarman via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I would like to better understand how you came to conclude that the
> tablegen re-runs based on changes in Support are what's causing your
build
> to be slow and what part specifically is taking all that time. I can do a
> clean release + assertions build of LLVM, Clang, compiler-rt and lld in
> about 5 minutes, plus 40 seconds to run cmake, on what I think is similar
> hardware to what Zach is using. If I swap Ptr += ret and Size -= Ret in
> raw_fd_ostream::write_impl so that Support changes, I can do an incremental
> build in about 10 seconds, including the regeneration of various .inc
> files. How do we get from there to 10 minutes for an incremental build?
>
Which build system are you using?

On my build config (16 core/32 thread, 128GB RAM) it takes 47 user seconds
to rebuild the clang binary after making the change you mentioned - with
ninja and lld, host compiler is a recent release build of clang. Ninja
reports building 377 things.

Comparatively, if I change something in lib/CodeGen/AsmPrinter, it builds 4
things and takes 19 user seconds. (most of that's probably the link step)

So it seems like rebuilding tablegen does rebuild quite a few things - but
I'm not sure exactly which parts are avoidable and which parts aren't.

>
> ---- On Fri, 26 May 2017 17:59:43 -0700 *Zachary Turner via llvm-dev
> <llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>>*
wrote ----
>
> It's that TableGen depends on Support, so if you change one file in
> support, support gets recompiled into a new static archive, which triggers
> a rerun of tablegen on all the tablegen inputs, which is extremely slow.
>
> On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov>
wrote:
>
>
>
> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:
>
> Changing a header file somewhere and having to spend 10 minutes waiting
> for a build leads to a lot of wasted developer time.
>
> The real culprit here is tablegen.  Can we split support and ADT into two
> - the parts that tablegen depends on and the parts that it doesn't?
>
>
> What's the actual problem here? Is it that TableGen regenerates
different
> files and so we then need to rebuild all dependencies of those files? Maybe
> we should use a diff-and-update approach (I thought, however, that we
> already did that).
>
>  -Hal
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/dc19d4f0/attachment.html>

> On May 30, 2017, at 1:16 PM, David Blaikie via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> 
> 
> On Tue, May 30, 2017 at 12:52 PM Bob Haarman via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> I would like to better understand how you came to conclude that the
tablegen re-runs based on changes in Support are what's causing your build
to be slow and what part specifically is taking all that time. I can do a clean
release + assertions build of LLVM, Clang, compiler-rt and lld in about 5
minutes, plus 40 seconds to run cmake, on what I think is similar hardware to
what Zach is using. If I swap Ptr += ret and Size -= Ret in
raw_fd_ostream::write_impl so that Support changes, I can do an incremental
build in about 10 seconds, including the regeneration of various .inc files. How
do we get from there to 10 minutes for an incremental build?
> 
> Which build system are you using?
> 
> On my build config (16 core/32 thread, 128GB RAM) it takes 47 user seconds
to rebuild the clang binary after making the change you mentioned - with ninja
and lld, host compiler is a recent release build of clang. Ninja reports
building 377 things.
At least for me ninja starts with a high estimated number of files needing a
rebuild. But the number of targets immediately jumps down by several hundreds as
soon as ninja realizes that the tablegen output didn't actually change.

- Matthias

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/cf11bfe7/attachment.html>

On Tue, May 30, 2017 at 1:21 PM Matthias Braun <mbraun at apple.com>
wrote:
> On May 30, 2017, at 1:16 PM, David Blaikie via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>
>
> On Tue, May 30, 2017 at 12:52 PM Bob Haarman via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> I would like to better understand how you came to conclude that the
>> tablegen re-runs based on changes in Support are what's causing
your build
>> to be slow and what part specifically is taking all that time. I can do
a
>> clean release + assertions build of LLVM, Clang, compiler-rt and lld in
>> about 5 minutes, plus 40 seconds to run cmake, on what I think is
similar
>> hardware to what Zach is using. If I swap Ptr += ret and Size -= Ret in
>> raw_fd_ostream::write_impl so that Support changes, I can do an
incremental
>> build in about 10 seconds, including the regeneration of various .inc
>> files. How do we get from there to 10 minutes for an incremental build?
>>
>
> Which build system are you using?
>
> On my build config (16 core/32 thread, 128GB RAM) it takes 47 user seconds
> to rebuild the clang binary after making the change you mentioned - with
> ninja and lld, host compiler is a recent release build of clang. Ninja
> reports building 377 things.
>
>
> At least for me ninja starts with a high estimated number of files needing
> a rebuild. But the number of targets immediately jumps down by several
> hundreds as soon as ninja realizes that the tablegen output didn't
actually
> change.
>
Fair - I think the estimate did start in the thousand+ and 377 was the
final number. So I guess it's pruning to some degree - not sure exactly
what/how.

>
> - Matthias
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/175dd68a/attachment.html>