[llvm-dev] Should we split llvm Support and ADT?

David Blaikie via llvm-dev
Sat, 27 May 2017 18:02:31 -0000

On Fri, May 26, 2017 at 8:06 PM Zachary Turner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> It would be better, because a debug tablegen is slower than an optimized
> tablegen, but it's still slow and it doesn't address the problem
that
> tablegen runs *at all* when it doesn't really need to. I think if
tablegen
> wasn't running all the time we could incremental builds down from 15
> minutes (and that's on my really powerful machine) to under 5, which
seemed
> like a big productivity boost averaged out over all users x time
>

Is that 10 minutes of running tablegen, though? I think this touches on
Hal's comment:
"Maybe we should use a diff-and-update approach (I thought, however, that
we already did that)."

In that the 10 minutes is presumably from timestamp-based invalidation
chaining down from running tablegen, to then rebuilding everything that
depends on tablegen's output.

That could be addressed by either the build system or tablegen writing
output to a temporary file, comparing the temp to the current file, and not
modifying the current file if they're the same - thus not invalidating
everything down the chain that depends on the tablegen output (not sure if
that's actually enough to convince a build system not to rebuild other
things - but worth a shot). This is a bit of a hackaround what the build
system itself could/should be doing, but in the absence of a content based
invalidation scheme - addressing this one particularly critical point in
LLVM's build seems like it could be worth such a workaround. (this would
speed up the build for even the times when tablegen's real dependencies
change - add a new function to ArrayRef, or STLExtras, etc... and tablegen
runs, but it doesn't cascade to everything else).

(plus, opt build of tablegen, so that conclusion can be reached more
quickly)

As Hal also mentioned, I'd be OK with this if there's a good logical
division that can be taken more than "whatever is/isn't used in
tablegen".


>
> We've got a lot of random stuff in Support, not all really related and
not
> all relevant to every project. It seems like we should be able to do
better.
> On Fri, May 26, 2017 at 6:48 PM Hal Finkel <hfinkel at anl.gov>
wrote:
>
>>
>> On 05/26/2017 07:59 PM, Zachary Turner wrote:
>>
>> It's that TableGen depends on Support, so if you change one file in
>> support, support gets recompiled into a new static archive, which
triggers
>> a rerun of tablegen on all the tablegen inputs, which is extremely
slow.
>>
>>
>> I thought that we had a "build an optimized tablegen even in debug
mode"
>> setting to work around this problem. Would that avoid this problem?
>>
>>
>>  -Hal
>>
>>
>>
>> On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov>
wrote:
>>
>>>
>>>
>>> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:
>>>
>>> Changing a header file somewhere and having to spend 10 minutes
waiting
>>> for a build leads to a lot of wasted developer time.
>>>
>>> The real culprit here is tablegen.  Can we split support and ADT
into
>>> two - the parts that tablegen depends on and the parts that it
doesn't?
>>>
>>>
>>> What's the actual problem here? Is it that TableGen regenerates
>>> different files and so we then need to rebuild all dependencies of
those
>>> files? Maybe we should use a diff-and-update approach (I thought,
however,
>>> that we already did that).
>>>
>>>  -Hal
>>>
>>>
>>> From what I can gather, Tablegen currently depends on these headers
and
>>> all of their transitive dependencies.
>>>
>>> #include "llvm/Support/Casting.h"
>>> #include "llvm/Support/CommandLine.h"
>>> #include "llvm/Support/Compiler.h"
>>> #include "llvm/Support/DataTypes.h"
>>> #include "llvm/Support/Debug.h"
>>> #include "llvm/Support/Error.h"
>>> #include "llvm/Support/ErrorHandling.h"
>>> #include "llvm/Support/Format.h"
>>> #include "llvm/Support/FormattedStream.h"
>>> #include "llvm/Support/LEB128.h"
>>> #include "llvm/Support/LowLevelTypeImpl.h"
>>> #include "llvm/Support/ManagedStatic.h"
>>> #include "llvm/Support/MathExtras.h"
>>> #include "llvm/Support/MemoryBuffer.h"
>>> #include "llvm/Support/PrettyStackTrace.h"
>>> #include "llvm/Support/Regex.h"
>>> #include "llvm/Support/SMLoc.h"
>>> #include "llvm/Support/ScopedPrinter.h"
>>> #include "llvm/Support/Signals.h"
>>> #include "llvm/Support/SourceMgr.h"
>>> #include "llvm/Support/raw_ostream.h"
>>>
>>> #include "llvm/ADT/APInt.h"
>>> #include "llvm/ADT/ArrayRef.h"
>>> #include "llvm/ADT/BitVector.h"
>>> #include "llvm/ADT/CachedHashString.h"
>>> #include "llvm/ADT/DenseSet.h"
>>> #include "llvm/ADT/IndexedMap.h"
>>> #include "llvm/ADT/IntEqClasses.h"
>>> #include "llvm/ADT/MapVector.h"
>>> #include "llvm/ADT/Optional.h"
>>> #include "llvm/ADT/PointerUnion.h"
>>> #include "llvm/ADT/STLExtras.h"
>>> #include "llvm/ADT/SetVector.h"
>>> #include "llvm/ADT/SmallPtrSet.h"
>>> #include "llvm/ADT/SmallSet.h"
>>> #include "llvm/ADT/SmallVector.h"
>>> #include "llvm/ADT/SparseBitVector.h"
>>> #include "llvm/ADT/Statistic.h"
>>> #include "llvm/ADT/StringExtras.h"
>>> #include "llvm/ADT/StringMap.h"
>>> #include "llvm/ADT/StringRef.h"
>>> #include "llvm/ADT/StringSet.h"
>>> #include "llvm/ADT/StringSwitch.h"
>>> #include "llvm/ADT/TinyPtrVector.h"
>>> #include "llvm/ADT/Twine.h"
>>>
>>>
>>> Is this something worth putting effort into?  If so, I volunteer.
>>>
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>> --
>>> Hal Finkel
>>> Lead, Compiler Technology and Programming Languages
>>> Leadership Computing Facility
>>> Argonne National Laboratory
>>>
>>>
>> --
>> Hal Finkel
>> Lead, Compiler Technology and Programming Languages
>> Leadership Computing Facility
>> Argonne National Laboratory
>>
>> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170527/bf22cdf9/attachment-0001.html>