llvm dev - Jul 2017 - CMake dependencies and building LLVM

> On 7 Jul 2017, at 17:12, David Blaikie via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> I'm /guessing/ this is probably pretty normal - but likely the best
solution would be to get down to granularity of 1 commit in each of those
ranges, to see if it's really justified that the change causes so much
recompilation (ie: do all the things that got compiled really depend on the
modified file)
> 
> Some of this might be tablegen recompiling - but I /think/ that ninja has
some support/seems to do the right thing when the resulting tablegen binary
isn't different from previously (or maybe when it doesn't generate
different output - this would be better) & doesn't cause everything to
rebuild (you'll notice Ninja sometimes starts with a very high estimate
(1000+) of targets to rebuild, but the estimate is adjusted down to only a
couple of hundred in many cases probably due to this.
I can confirm that tablegen-erated files do the right thing and that it's
based on the output of tablegen. Our cmake scripts do it by having llvm-tblgen
write to a temporary file and then use 'cmake -E copy_if_different' to
update the real one when necessary. Once ninja has run the copy_if_different
command it re-checks the dependencies and culls the ones it doesn't need.
> On Fri, Jul 7, 2017 at 6:47 AM Nemanja Ivanovic via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> Hi everyone,
> I have felt for quite some time that during my regular LLVM workflow, CMake
recompiles a lot of files after I update to the latest ToT (which I do more than
once per week). Of course, this wasn't really based on any real data, just
felt like every time I update (even if it's a day or two later), we end up
recompiling everything. For a while I assumed that patches touch a few headers
that are used all over the place or something like that, but this was happening
with what seems like unexpected frequency.
> 
> So what I've done is do a few update/rebuild cycles for revisions that
are 200 apart, then 100, then 50, then 25. And it seems that quite commonly, we
recompile more than 50% of the source with just a few landed revisions. I've
included the numbers below (and I do realize that this is a fairly small
sample).
> 
> If this seems reasonable to most of the community, so be it - I work on a
large server that chews through this stuff pretty quickly so I can certainly
live with it. On the other hand, if this seems way higher than what you expect,
it could very well be that something is wrong in my setup and I'd appreciate
some pointers about how to go about investigating this. Finally, if others are
seeing this and we as a community do feel that something can/should be done
about this (although I don't know what that might be), perhaps we can start
that discussion.
> 
> Setup: CMake with -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -G
Ninja
> 
> The numbers here are the total numbers of targets ninja reports as having
rebuilt. Ninja is invoked only `-j <N>` for parameters (i.e. not check-all
or any specific targets).
> 
> RevFrm, RevTo,  NumNinjaObjsRebuilt
> 300000, 300200, 2957
> 300200, 300400, 2692
> 300400, 300600, 3001
> 300600, 300800, 2877
> 300800, 301000, 3003
> 301000, 301200, 2750
> 301200, 301400, 2579
> 301400, 301600, 2709
> 301600, 301700, 2325
> 301700, 301800, 2357
> 301800, 301900, 2667
> 301900, 302000, 2466
> 302000, 302100, 2499
> 302100, 302200, 2063
> 302200, 302250, 2884
> 302250, 302300, 2007
> 302300, 302350, 2209
> 302350, 302400, 737
> 302400, 302450, 2398
> 302450, 302500, 2856
> 302500, 302550, 2090
> 302550, 302600, 2866
> 302600, 302625, 1786
> 302625, 302650, 1966
> 302650, 302675, 1583
> 302675, 302700, 307
> 302700, 302725, 2246
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170707/9039f750/attachment-0001.html>

Yes, I have seen TableGen creating tempoaries and doing the right thing.
Keep in mind that the stated numbers are final numbers (i.e. When ninja is
done). I'll try with some single changesets to try to narrow down what it
is that causes a lot of recompilation.

On Fri, Jul 7, 2017 at 7:42 PM Daniel Sanders <daniel_l_sanders at
apple.com>
wrote:
> On 7 Jul 2017, at 17:12, David Blaikie via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> I'm /guessing/ this is probably pretty normal - but likely the best
> solution would be to get down to granularity of 1 commit in each of those
> ranges, to see if it's really justified that the change causes so much
> recompilation (ie: do all the things that got compiled really depend on the
> modified file)
>
> Some of this might be tablegen recompiling - but I /think/ that ninja has
> some support/seems to do the right thing when the resulting tablegen binary
> isn't different from previously (or maybe when it doesn't generate
> different output - this would be better) & doesn't cause everything
to
> rebuild (you'll notice Ninja sometimes starts with a very high estimate
> (1000+) of targets to rebuild, but the estimate is adjusted down to only a
> couple of hundred in many cases probably due to this.
>
>
> I can confirm that tablegen-erated files do the right thing and that
it's
> based on the output of tablegen. Our cmake scripts do it by having
> llvm-tblgen write to a temporary file and then use 'cmake -E
> copy_if_different' to update the real one when necessary. Once ninja
has
> run the copy_if_different command it re-checks the dependencies and culls
> the ones it doesn't need.
>
> On Fri, Jul 7, 2017 at 6:47 AM Nemanja Ivanovic via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi everyone,
>> I have felt for quite some time that during my regular LLVM workflow,
>> CMake recompiles a lot of files after I update to the latest ToT (which
I
>> do more than once per week). Of course, this wasn't really based on
any
>> real data, just felt like every time I update (even if it's a day
or two
>> later), we end up recompiling everything. For a while I assumed that
>> patches touch a few headers that are used all over the place or
something
>> like that, but this was happening with what seems like unexpected
frequency.
>>
>> So what I've done is do a few update/rebuild cycles for revisions
that
>> are 200 apart, then 100, then 50, then 25. And it seems that quite
>> commonly, we recompile more than 50% of the source with just a few
landed
>> revisions. I've included the numbers below (and I do realize that
this is a
>> fairly small sample).
>>
>> If this seems reasonable to most of the community, so be it - I work on
a
>> large server that chews through this stuff pretty quickly so I can
>> certainly live with it. On the other hand, if this seems way higher
than
>> what you expect, it could very well be that something is wrong in my
setup
>> and I'd appreciate some pointers about how to go about
investigating this.
>> Finally, if others are seeing this and we as a community do feel that
>> something can/should be done about this (although I don't know what
that
>> might be), perhaps we can start that discussion.
>>
>> Setup: CMake with -DCMAKE_BUILD_TYPE=Release
-DLLVM_ENABLE_ASSERTIONS=ON
>> -G Ninja
>>
>> The numbers here are the total numbers of targets ninja reports as
having
>> rebuilt. Ninja is invoked only `-j <N>` for parameters (i.e. not
check-all
>> or any specific targets).
>>
>> RevFrm, RevTo,  NumNinjaObjsRebuilt
>> 300000, 300200, 2957
>> 300200, 300400, 2692
>> 300400, 300600, 3001
>> 300600, 300800, 2877
>> 300800, 301000, 3003
>> 301000, 301200, 2750
>> 301200, 301400, 2579
>> 301400, 301600, 2709
>> 301600, 301700, 2325
>> 301700, 301800, 2357
>> 301800, 301900, 2667
>> 301900, 302000, 2466
>> 302000, 302100, 2499
>> 302100, 302200, 2063
>> 302200, 302250, 2884
>> 302250, 302300, 2007
>> 302300, 302350, 2209
>> 302350, 302400, 737
>> 302400, 302450, 2398
>> 302450, 302500, 2856
>> 302500, 302550, 2090
>> 302550, 302600, 2866
>> 302600, 302625, 1786
>> 302625, 302650, 1966
>> 302650, 302675, 1583
>> 302675, 302700, 307
>> 302700, 302725, 2246
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170707/e892463d/attachment.html>

For what it's worth:

- I'm usually seeing similar behavior even in shorter timeframes like
updating after half a day
- From all I could tell the buildsystem was doing the right thing. There's
just a lot of headers in Support and ADT that are included pretty much
everywhere and people change them (legitimately).

Sure you may find the occasional instance where doing some class XXX; forward
declaration instead of #include "XXX.h" would improve dependencies.
But overall LLVM code is already amazingly clean in this regard so I don't
see a lot of room for improvement there either.

- If you haven't seen it look at Tillmanns excellent blogpost about all the
ways to optimize an LLVM build:
https://blogs.s-osg.org/an-introduction-to-accelerating-your-build-with-clang/
<https://blogs.s-osg.org/an-introduction-to-accelerating-your-build-with-clang/>
- The biggest savings for me personally are disabling clang builds and
restricting myself to the targets I am developing for
- Either way it's never fast and I end up drinking way too much coffee
because it's so easy to go down the hall to grab another one while the build
is running...

- Matthias
> On Jul 7, 2017, at 11:57 AM, Nemanja Ivanovic via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Yes, I have seen TableGen creating tempoaries and doing the right thing.
Keep in mind that the stated numbers are final numbers (i.e. When ninja is
done). I'll try with some single changesets to try to narrow down what it is
that causes a lot of recompilation.
> 
> On Fri, Jul 7, 2017 at 7:42 PM Daniel Sanders <daniel_l_sanders at
apple.com <mailto:daniel_l_sanders at apple.com>> wrote:
>> On 7 Jul 2017, at 17:12, David Blaikie via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> I'm /guessing/ this is probably pretty normal - but likely the best
solution would be to get down to granularity of 1 commit in each of those
ranges, to see if it's really justified that the change causes so much
recompilation (ie: do all the things that got compiled really depend on the
modified file)
>> 
>> Some of this might be tablegen recompiling - but I /think/ that ninja
has some support/seems to do the right thing when the resulting tablegen binary
isn't different from previously (or maybe when it doesn't generate
different output - this would be better) & doesn't cause everything to
rebuild (you'll notice Ninja sometimes starts with a very high estimate
(1000+) of targets to rebuild, but the estimate is adjusted down to only a
couple of hundred in many cases probably due to this.
> 
> I can confirm that tablegen-erated files do the right thing and that
it's based on the output of tablegen. Our cmake scripts do it by having
llvm-tblgen write to a temporary file and then use 'cmake -E
copy_if_different' to update the real one when necessary. Once ninja has run
the copy_if_different command it re-checks the dependencies and culls the ones
it doesn't need.
> 
>> On Fri, Jul 7, 2017 at 6:47 AM Nemanja Ivanovic via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>> Hi everyone,
>> I have felt for quite some time that during my regular LLVM workflow,
CMake recompiles a lot of files after I update to the latest ToT (which I do
more than once per week). Of course, this wasn't really based on any real
data, just felt like every time I update (even if it's a day or two later),
we end up recompiling everything. For a while I assumed that patches touch a few
headers that are used all over the place or something like that, but this was
happening with what seems like unexpected frequency.
>> 
>> So what I've done is do a few update/rebuild cycles for revisions
that are 200 apart, then 100, then 50, then 25. And it seems that quite
commonly, we recompile more than 50% of the source with just a few landed
revisions. I've included the numbers below (and I do realize that this is a
fairly small sample).
>> 
>> If this seems reasonable to most of the community, so be it - I work on
a large server that chews through this stuff pretty quickly so I can certainly
live with it. On the other hand, if this seems way higher than what you expect,
it could very well be that something is wrong in my setup and I'd appreciate
some pointers about how to go about investigating this. Finally, if others are
seeing this and we as a community do feel that something can/should be done
about this (although I don't know what that might be), perhaps we can start
that discussion.
>> 
>> Setup: CMake with -DCMAKE_BUILD_TYPE=Release
-DLLVM_ENABLE_ASSERTIONS=ON -G Ninja
>> 
>> The numbers here are the total numbers of targets ninja reports as
having rebuilt. Ninja is invoked only `-j <N>` for parameters (i.e. not
check-all or any specific targets).
>> 
>> RevFrm, RevTo,  NumNinjaObjsRebuilt
>> 300000, 300200, 2957
>> 300200, 300400, 2692
>> 300400, 300600, 3001
>> 300600, 300800, 2877
>> 300800, 301000, 3003
>> 301000, 301200, 2750
>> 301200, 301400, 2579
>> 301400, 301600, 2709
>> 301600, 301700, 2325
>> 301700, 301800, 2357
>> 301800, 301900, 2667
>> 301900, 302000, 2466
>> 302000, 302100, 2499
>> 302100, 302200, 2063
>> 302200, 302250, 2884
>> 302250, 302300, 2007
>> 302300, 302350, 2209
>> 302350, 302400, 737
>> 302400, 302450, 2398
>> 302450, 302500, 2856
>> 302500, 302550, 2090
>> 302550, 302600, 2866
>> 302600, 302625, 1786
>> 302625, 302650, 1966
>> 302650, 302675, 1583
>> 302675, 302700, 307
>> 302700, 302725, 2246
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170707/41cc8c85/attachment.html>