llvm dev - May 2017 - Should we split llvm Support and ADT?

It's that TableGen depends on Support, so if you change one file in
support, support gets recompiled into a new static archive, which triggers
a rerun of tablegen on all the tablegen inputs, which is extremely slow.

On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov> wrote:
>
>
> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:
>
> Changing a header file somewhere and having to spend 10 minutes waiting
> for a build leads to a lot of wasted developer time.
>
> The real culprit here is tablegen.  Can we split support and ADT into two
> - the parts that tablegen depends on and the parts that it doesn't?
>
>
> What's the actual problem here? Is it that TableGen regenerates
different
> files and so we then need to rebuild all dependencies of those files? Maybe
> we should use a diff-and-update approach (I thought, however, that we
> already did that).
>
>  -Hal
>
>
> From what I can gather, Tablegen currently depends on these headers and
> all of their transitive dependencies.
>
> #include "llvm/Support/Casting.h"
> #include "llvm/Support/CommandLine.h"
> #include "llvm/Support/Compiler.h"
> #include "llvm/Support/DataTypes.h"
> #include "llvm/Support/Debug.h"
> #include "llvm/Support/Error.h"
> #include "llvm/Support/ErrorHandling.h"
> #include "llvm/Support/Format.h"
> #include "llvm/Support/FormattedStream.h"
> #include "llvm/Support/LEB128.h"
> #include "llvm/Support/LowLevelTypeImpl.h"
> #include "llvm/Support/ManagedStatic.h"
> #include "llvm/Support/MathExtras.h"
> #include "llvm/Support/MemoryBuffer.h"
> #include "llvm/Support/PrettyStackTrace.h"
> #include "llvm/Support/Regex.h"
> #include "llvm/Support/SMLoc.h"
> #include "llvm/Support/ScopedPrinter.h"
> #include "llvm/Support/Signals.h"
> #include "llvm/Support/SourceMgr.h"
> #include "llvm/Support/raw_ostream.h"
>
> #include "llvm/ADT/APInt.h"
> #include "llvm/ADT/ArrayRef.h"
> #include "llvm/ADT/BitVector.h"
> #include "llvm/ADT/CachedHashString.h"
> #include "llvm/ADT/DenseSet.h"
> #include "llvm/ADT/IndexedMap.h"
> #include "llvm/ADT/IntEqClasses.h"
> #include "llvm/ADT/MapVector.h"
> #include "llvm/ADT/Optional.h"
> #include "llvm/ADT/PointerUnion.h"
> #include "llvm/ADT/STLExtras.h"
> #include "llvm/ADT/SetVector.h"
> #include "llvm/ADT/SmallPtrSet.h"
> #include "llvm/ADT/SmallSet.h"
> #include "llvm/ADT/SmallVector.h"
> #include "llvm/ADT/SparseBitVector.h"
> #include "llvm/ADT/Statistic.h"
> #include "llvm/ADT/StringExtras.h"
> #include "llvm/ADT/StringMap.h"
> #include "llvm/ADT/StringRef.h"
> #include "llvm/ADT/StringSet.h"
> #include "llvm/ADT/StringSwitch.h"
> #include "llvm/ADT/TinyPtrVector.h"
> #include "llvm/ADT/Twine.h"
>
>
> Is this something worth putting effort into?  If so, I volunteer.
>
>
>
> _______________________________________________
> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> --
> Hal Finkel
> Lead, Compiler Technology and Programming Languages
> Leadership Computing Facility
> Argonne National Laboratory
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170527/7388c6ac/attachment.html>

On 05/26/2017 07:59 PM, Zachary Turner wrote:
> It's that TableGen depends on Support, so if you change one file in 
> support, support gets recompiled into a new static archive, which 
> triggers a rerun of tablegen on all the tablegen inputs, which is 
> extremely slow.
I thought that we had a "build an optimized tablegen even in debug
mode"
setting to work around this problem. Would that avoid this problem?

  -Hal
>
> On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov 
> <mailto:hfinkel at anl.gov>> wrote:
>
>
>
>     On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:
>>     Changing a header file somewhere and having to spend 10 minutes
>>     waiting for a build leads to a lot of wasted developer time.
>>
>>     The real culprit here is tablegen.  Can we split support and ADT
>>     into two - the parts that tablegen depends on and the parts that
>>     it doesn't?
>
>     What's the actual problem here? Is it that TableGen regenerates
>     different files and so we then need to rebuild all dependencies of
>     those files? Maybe we should use a diff-and-update approach (I
>     thought, however, that we already did that).
>
>      -Hal
>
>>
>>     From what I can gather, Tablegen currently depends on these
>>     headers and all of their transitive dependencies.
>>
>>     #include "llvm/Support/Casting.h"
>>     #include "llvm/Support/CommandLine.h"
>>     #include "llvm/Support/Compiler.h"
>>     #include "llvm/Support/DataTypes.h"
>>     #include "llvm/Support/Debug.h"
>>     #include "llvm/Support/Error.h"
>>     #include "llvm/Support/ErrorHandling.h"
>>     #include "llvm/Support/Format.h"
>>     #include "llvm/Support/FormattedStream.h"
>>     #include "llvm/Support/LEB128.h"
>>     #include "llvm/Support/LowLevelTypeImpl.h"
>>     #include "llvm/Support/ManagedStatic.h"
>>     #include "llvm/Support/MathExtras.h"
>>     #include "llvm/Support/MemoryBuffer.h"
>>     #include "llvm/Support/PrettyStackTrace.h"
>>     #include "llvm/Support/Regex.h"
>>     #include "llvm/Support/SMLoc.h"
>>     #include "llvm/Support/ScopedPrinter.h"
>>     #include "llvm/Support/Signals.h"
>>     #include "llvm/Support/SourceMgr.h"
>>     #include "llvm/Support/raw_ostream.h"
>>
>>     #include "llvm/ADT/APInt.h"
>>     #include "llvm/ADT/ArrayRef.h"
>>     #include "llvm/ADT/BitVector.h"
>>     #include "llvm/ADT/CachedHashString.h"
>>     #include "llvm/ADT/DenseSet.h"
>>     #include "llvm/ADT/IndexedMap.h"
>>     #include "llvm/ADT/IntEqClasses.h"
>>     #include "llvm/ADT/MapVector.h"
>>     #include "llvm/ADT/Optional.h"
>>     #include "llvm/ADT/PointerUnion.h"
>>     #include "llvm/ADT/STLExtras.h"
>>     #include "llvm/ADT/SetVector.h"
>>     #include "llvm/ADT/SmallPtrSet.h"
>>     #include "llvm/ADT/SmallSet.h"
>>     #include "llvm/ADT/SmallVector.h"
>>     #include "llvm/ADT/SparseBitVector.h"
>>     #include "llvm/ADT/Statistic.h"
>>     #include "llvm/ADT/StringExtras.h"
>>     #include "llvm/ADT/StringMap.h"
>>     #include "llvm/ADT/StringRef.h"
>>     #include "llvm/ADT/StringSet.h"
>>     #include "llvm/ADT/StringSwitch.h"
>>     #include "llvm/ADT/TinyPtrVector.h"
>>     #include "llvm/ADT/Twine.h"
>>
>>
>>     Is this something worth putting effort into?  If so, I volunteer.
>>
>>
>>
>>     _______________________________________________
>>     LLVM Developers mailing list
>>     llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>     -- 
>     Hal Finkel
>     Lead, Compiler Technology and Programming Languages
>     Leadership Computing Facility
>     Argonne National Laboratory
>
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170526/61d74471/attachment.html>

It would be better, because a debug tablegen is slower than an optimized
tablegen, but it's still slow and it doesn't address the problem that
tablegen runs *at all* when it doesn't really need to. I think if tablegen
wasn't running all the time we could incremental builds down from 15
minutes (and that's on my really powerful machine) to under 5, which seemed
like a big productivity boost averaged out over all users x time

We've got a lot of random stuff in Support, not all really related and not
all relevant to every project. It seems like we should be able to do better.
On Fri, May 26, 2017 at 6:48 PM Hal Finkel <hfinkel at anl.gov> wrote:
>
> On 05/26/2017 07:59 PM, Zachary Turner wrote:
>
> It's that TableGen depends on Support, so if you change one file in
> support, support gets recompiled into a new static archive, which triggers
> a rerun of tablegen on all the tablegen inputs, which is extremely slow.
>
>
> I thought that we had a "build an optimized tablegen even in debug
mode"
> setting to work around this problem. Would that avoid this problem?
>
>
>  -Hal
>
>
>
> On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov>
wrote:
>
>>
>>
>> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:
>>
>> Changing a header file somewhere and having to spend 10 minutes waiting
>> for a build leads to a lot of wasted developer time.
>>
>> The real culprit here is tablegen.  Can we split support and ADT into
two
>> - the parts that tablegen depends on and the parts that it doesn't?
>>
>>
>> What's the actual problem here? Is it that TableGen regenerates
different
>> files and so we then need to rebuild all dependencies of those files?
Maybe
>> we should use a diff-and-update approach (I thought, however, that we
>> already did that).
>>
>>  -Hal
>>
>>
>> From what I can gather, Tablegen currently depends on these headers and
>> all of their transitive dependencies.
>>
>> #include "llvm/Support/Casting.h"
>> #include "llvm/Support/CommandLine.h"
>> #include "llvm/Support/Compiler.h"
>> #include "llvm/Support/DataTypes.h"
>> #include "llvm/Support/Debug.h"
>> #include "llvm/Support/Error.h"
>> #include "llvm/Support/ErrorHandling.h"
>> #include "llvm/Support/Format.h"
>> #include "llvm/Support/FormattedStream.h"
>> #include "llvm/Support/LEB128.h"
>> #include "llvm/Support/LowLevelTypeImpl.h"
>> #include "llvm/Support/ManagedStatic.h"
>> #include "llvm/Support/MathExtras.h"
>> #include "llvm/Support/MemoryBuffer.h"
>> #include "llvm/Support/PrettyStackTrace.h"
>> #include "llvm/Support/Regex.h"
>> #include "llvm/Support/SMLoc.h"
>> #include "llvm/Support/ScopedPrinter.h"
>> #include "llvm/Support/Signals.h"
>> #include "llvm/Support/SourceMgr.h"
>> #include "llvm/Support/raw_ostream.h"
>>
>> #include "llvm/ADT/APInt.h"
>> #include "llvm/ADT/ArrayRef.h"
>> #include "llvm/ADT/BitVector.h"
>> #include "llvm/ADT/CachedHashString.h"
>> #include "llvm/ADT/DenseSet.h"
>> #include "llvm/ADT/IndexedMap.h"
>> #include "llvm/ADT/IntEqClasses.h"
>> #include "llvm/ADT/MapVector.h"
>> #include "llvm/ADT/Optional.h"
>> #include "llvm/ADT/PointerUnion.h"
>> #include "llvm/ADT/STLExtras.h"
>> #include "llvm/ADT/SetVector.h"
>> #include "llvm/ADT/SmallPtrSet.h"
>> #include "llvm/ADT/SmallSet.h"
>> #include "llvm/ADT/SmallVector.h"
>> #include "llvm/ADT/SparseBitVector.h"
>> #include "llvm/ADT/Statistic.h"
>> #include "llvm/ADT/StringExtras.h"
>> #include "llvm/ADT/StringMap.h"
>> #include "llvm/ADT/StringRef.h"
>> #include "llvm/ADT/StringSet.h"
>> #include "llvm/ADT/StringSwitch.h"
>> #include "llvm/ADT/TinyPtrVector.h"
>> #include "llvm/ADT/Twine.h"
>>
>>
>> Is this something worth putting effort into?  If so, I volunteer.
>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>> --
>> Hal Finkel
>> Lead, Compiler Technology and Programming Languages
>> Leadership Computing Facility
>> Argonne National Laboratory
>>
>>
> --
> Hal Finkel
> Lead, Compiler Technology and Programming Languages
> Leadership Computing Facility
> Argonne National Laboratory
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170527/c880c6d6/attachment.html>

On 5/26/2017 7:59 PM, Zachary Turner via llvm-dev wrote:
> It's that TableGen depends on Support, so if you change one file in
> support, support gets recompiled into a new static archive, which
> triggers a rerun of tablegen on all the tablegen inputs, which is
> extremely slow.
What exactly is extremely slow? In my experience TableGen itself takes a 
negligible amount of time compared to the rest of the build. This is 
particularly true in cases when something in Support or ADT is modified, 
as this usually triggers recompilation of large parts of LLVM.

-Krzysztof

2017-05-26 17:47 GMT-07:00 Zachary Turner via llvm-dev <
llvm-dev at lists.llvm.org>:
> Changing a header file somewhere and having to spend 10 minutes waiting
> for a build leads to a lot of wasted developer time.
>
> The real culprit here is tablegen.  Can we split support and ADT into two
> - the parts that tablegen depends on and the parts that it doesn't?
>
Splitting ADT just based on tablegen usage seems dubious to me. If we need
to go this route, I'd replace as many uses of ADT data structure with STL
ones to begin with to reduce the surface.


2017-05-28 8:25 GMT-07:00 Krzysztof Parzyszek via llvm-dev <
llvm-dev at lists.llvm.org>:
> On 5/26/2017 7:59 PM, Zachary Turner via llvm-dev wrote:
>
>> It's that TableGen depends on Support, so if you change one file in
>> support, support gets recompiled into a new static archive, which
>> triggers a rerun of tablegen on all the tablegen inputs, which is
>> extremely slow.
>>
>
> What exactly is extremely slow? In my experience TableGen itself takes a
> negligible amount of time compared to the rest of the build. This is
> particularly true in cases when something in Support or ADT is modified, as
> this usually triggers recompilation of large parts of LLVM.
>
Tablegen built in debug is really slow though, I remember an out-of-tree
backend where running llvm-tblgen was taking up to 5 min per file!
The CMake option LLVM_OPTIMIZED_TABLEGEN helps a lot with this. Otherwise
LLVM_TABLEGEN is even more efficient, but it's a double-edged sword.

But we could also use the diff-and-copy approach not on the tablegen output
but on the llvm-tblgen binary itself, that way we wouldn't re-run it when
it does not change itself (I'm not sure why CMake does not use this
strategy by default for any file including .o and .a?).

-- 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170528/d7ecf563/attachment.html>

I would like to better understand how you came to conclude that the tablegen
re-runs based on changes in Support are what's causing your build to be slow
and what part specifically is taking all that time. I can do a clean release +
assertions build of LLVM, Clang, compiler-rt and lld in about 5 minutes, plus 40
seconds to run cmake, on what I think is similar hardware to what Zach is using.
If I swap Ptr += ret and Size -= Ret in raw_fd_ostream::write_impl so that
Support changes, I can do an incremental build in about 10 seconds, including
the regeneration of various .inc files. How do we get from there to 10 minutes
for an incremental build?

---- On Fri, 26 May 2017 17:59:43 -0700 Zachary Turner via llvm-dev
<llvm-dev at lists.llvm.org> wrote ----



It's that TableGen depends on Support, so if you change one file in support,
support gets recompiled into a new static archive, which triggers a rerun of
tablegen on all the tablegen inputs, which is extremely slow.



On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov>
wrote:





On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:

Changing a header file somewhere and having to spend 10 minutes waiting for a
build leads to a lot of wasted developer time.



The real culprit here is tablegen.  Can we split support and ADT into two - the
parts that tablegen depends on and the parts that it doesn't?





What's the actual problem here? Is it that TableGen regenerates different
files and so we then need to rebuild all dependencies of those files? Maybe we
should use a diff-and-update approach (I thought, however, that we already did
that).

 

  -Hal

 








-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/4ad03d96/attachment.html>

2017-05-30 12:52 GMT-07:00 Bob Haarman via llvm-dev <llvm-dev at
lists.llvm.org
>:
> I would like to better understand how you came to conclude that the
> tablegen re-runs based on changes in Support are what's causing your
build
> to be slow and what part specifically is taking all that time. I can do a
> clean release + assertions build of LLVM, Clang, compiler-rt and lld in
> about 5 minutes, plus 40 seconds to run cmake, on what I think is similar
> hardware to what Zach is using. If I swap Ptr += ret and Size -= Ret in
> raw_fd_ostream::write_impl so that Support changes, I can do an incremental
> build in about 10 seconds, including the regeneration of various .inc
> files. How do we get from there to 10 minutes for an incremental build?
>
Have you tried a Debug build with all the backends configured in? I think
part of the issue is that Tablegen suffers from being compiled in debug
mode (hence the OPTIMIZED_TABLE_GEN cmake option).
Also not everyone has a "google-like" workstation ;)

-- 
Mehdi



>
> ---- On Fri, 26 May 2017 17:59:43 -0700 *Zachary Turner via llvm-dev
> <llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>>*
wrote ----
>
> It's that TableGen depends on Support, so if you change one file in
> support, support gets recompiled into a new static archive, which triggers
> a rerun of tablegen on all the tablegen inputs, which is extremely slow.
>
> On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov>
wrote:
>
>
>
> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:
>
> Changing a header file somewhere and having to spend 10 minutes waiting
> for a build leads to a lot of wasted developer time.
>
> The real culprit here is tablegen.  Can we split support and ADT into two
> - the parts that tablegen depends on and the parts that it doesn't?
>
>
> What's the actual problem here? Is it that TableGen regenerates
different
> files and so we then need to rebuild all dependencies of those files? Maybe
> we should use a diff-and-update approach (I thought, however, that we
> already did that).
>
>  -Hal
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/8e01acf9/attachment.html>

On Tue, May 30, 2017 at 12:52 PM Bob Haarman via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I would like to better understand how you came to conclude that the
> tablegen re-runs based on changes in Support are what's causing your
build
> to be slow and what part specifically is taking all that time. I can do a
> clean release + assertions build of LLVM, Clang, compiler-rt and lld in
> about 5 minutes, plus 40 seconds to run cmake, on what I think is similar
> hardware to what Zach is using. If I swap Ptr += ret and Size -= Ret in
> raw_fd_ostream::write_impl so that Support changes, I can do an incremental
> build in about 10 seconds, including the regeneration of various .inc
> files. How do we get from there to 10 minutes for an incremental build?
>
Which build system are you using?

On my build config (16 core/32 thread, 128GB RAM) it takes 47 user seconds
to rebuild the clang binary after making the change you mentioned - with
ninja and lld, host compiler is a recent release build of clang. Ninja
reports building 377 things.

Comparatively, if I change something in lib/CodeGen/AsmPrinter, it builds 4
things and takes 19 user seconds. (most of that's probably the link step)

So it seems like rebuilding tablegen does rebuild quite a few things - but
I'm not sure exactly which parts are avoidable and which parts aren't.

>
> ---- On Fri, 26 May 2017 17:59:43 -0700 *Zachary Turner via llvm-dev
> <llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>>*
wrote ----
>
> It's that TableGen depends on Support, so if you change one file in
> support, support gets recompiled into a new static archive, which triggers
> a rerun of tablegen on all the tablegen inputs, which is extremely slow.
>
> On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov>
wrote:
>
>
>
> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:
>
> Changing a header file somewhere and having to spend 10 minutes waiting
> for a build leads to a lot of wasted developer time.
>
> The real culprit here is tablegen.  Can we split support and ADT into two
> - the parts that tablegen depends on and the parts that it doesn't?
>
>
> What's the actual problem here? Is it that TableGen regenerates
different
> files and so we then need to rebuild all dependencies of those files? Maybe
> we should use a diff-and-update approach (I thought, however, that we
> already did that).
>
>  -Hal
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/dc19d4f0/attachment.html>

On Tue, May 30, 2017 at 12:52 PM Bob Haarman <llvm at inglorion.net>
wrote:
> I would like to better understand how you came to conclude that the
> tablegen re-runs based on changes in Support are what's causing your
build
> to be slow and what part specifically is taking all that time. I can do a
> clean release + assertions build of LLVM, Clang, compiler-rt and lld in
> about 5 minutes, plus 40 seconds to run cmake, on what I think is similar
> hardware to what Zach is using. If I swap Ptr += ret and Size -= Ret in
> raw_fd_ostream::write_impl so that Support changes, I can do an incremental
> build in about 10 seconds, including the regeneration of various .inc
> files. How do we get from there to 10 minutes for an incremental build?
>
For the sake of comparison, I made the same change and it took  1:58.39.  A
slightly different but more intrusive change to Format.h in
format_object_base::print() took 5:6.54.  A clean build on the same machine
takes about 12 minutes.

I tried a few hacks to the CMake to say "don't run tablegen, no matter
what, and don't make anything depend on tablegen's output", but I
couldn't
come up with the right magic.  If anyone knows, I can test a side-by-side
comparison without tablegen.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/bd3ccfb6/attachment.html>