llvm dev - May 2017 - Should we split llvm Support and ADT?

In my experience the buildsystem works fine in combination with tablegen (at
least this aspect of it). The real problem here is that tablegen is just slow.
Some of the X86 tables take indeed

Last time I looked at it tablegen had still room to optimize the way it resolves
class hierarchies and the variables within which it did basically one at a time,
so it needed to traverse the hierarchies again for each variables.
> On May 27, 2017, at 12:54 PM, David Blaikie via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Maybe we do and build systems aren't respecting/noticing this? I'm
not sure.
> 
> On Sat, May 27, 2017 at 12:50 PM Craig Topper <craig.topper at gmail.com
<mailto:craig.topper at gmail.com>> wrote:
> I thought we already did write tablegen output to temporary files like
X86GenAsmWriter1.inc.tmp first and then diffed them with the real .inc file and
conditionally copied.
> 
> ~Craig
> 
> On Sat, May 27, 2017 at 11:02 AM, David Blaikie via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> 
> 
> On Fri, May 26, 2017 at 8:06 PM Zachary Turner via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> It would be better, because a debug tablegen is slower than an optimized
tablegen, but it's still slow and it doesn't address the problem that
tablegen runs *at all* when it doesn't really need to. I think if tablegen
wasn't running all the time we could incremental builds down from 15 minutes
(and that's on my really powerful machine) to under 5, which seemed like a
big productivity boost averaged out over all users x time
> 
> Is that 10 minutes of running tablegen, though? I think this touches on
Hal's comment:
> "Maybe we should use a diff-and-update approach (I thought, however,
that we already did that)."
> 
> In that the 10 minutes is presumably from timestamp-based invalidation
chaining down from running tablegen, to then rebuilding everything that depends
on tablegen's output.
> 
> That could be addressed by either the build system or tablegen writing
output to a temporary file, comparing the temp to the current file, and not
modifying the current file if they're the same - thus not invalidating
everything down the chain that depends on the tablegen output (not sure if
that's actually enough to convince a build system not to rebuild other
things - but worth a shot). This is a bit of a hackaround what the build system
itself could/should be doing, but in the absence of a content based invalidation
scheme - addressing this one particularly critical point in LLVM's build
seems like it could be worth such a workaround. (this would speed up the build
for even the times when tablegen's real dependencies change - add a new
function to ArrayRef, or STLExtras, etc... and tablegen runs, but it doesn't
cascade to everything else).
> 
> (plus, opt build of tablegen, so that conclusion can be reached more
quickly)
> 
> As Hal also mentioned, I'd be OK with this if there's a good
logical division that can be taken more than "whatever is/isn't used in
tablegen".
>  
> 
> We've got a lot of random stuff in Support, not all really related and
not all relevant to every project. It seems like we should be able to do better.
> On Fri, May 26, 2017 at 6:48 PM Hal Finkel <hfinkel at anl.gov
<mailto:hfinkel at anl.gov>> wrote:
> 
> On 05/26/2017 07:59 PM, Zachary Turner wrote:
>> It's that TableGen depends on Support, so if you change one file in
support, support gets recompiled into a new static archive, which triggers a
rerun of tablegen on all the tablegen inputs, which is extremely slow.
> 
> I thought that we had a "build an optimized tablegen even in debug
mode" setting to work around this problem. Would that avoid this problem?
> 
> 
>  -Hal
> 
> 
>> 
>> On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov
<mailto:hfinkel at anl.gov>> wrote:
>> 
>> 
>> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:
>>> Changing a header file somewhere and having to spend 10 minutes
waiting for a build leads to a lot of wasted developer time.
>>> 
>>> The real culprit here is tablegen.  Can we split support and ADT
into two - the parts that tablegen depends on and the parts that it doesn't?
>> 
>> What's the actual problem here? Is it that TableGen regenerates
different files and so we then need to rebuild all dependencies of those files?
Maybe we should use a diff-and-update approach (I thought, however, that we
already did that).
>> 
>>  -Hal
>> 
>>> 
>>> From what I can gather, Tablegen currently depends on these headers
and all of their transitive dependencies.
>>> 
>>> #include "llvm/Support/Casting.h"
>>> #include "llvm/Support/CommandLine.h"
>>> #include "llvm/Support/Compiler.h"
>>> #include "llvm/Support/DataTypes.h"
>>> #include "llvm/Support/Debug.h"
>>> #include "llvm/Support/Error.h"
>>> #include "llvm/Support/ErrorHandling.h"
>>> #include "llvm/Support/Format.h"
>>> #include "llvm/Support/FormattedStream.h"
>>> #include "llvm/Support/LEB128.h"
>>> #include "llvm/Support/LowLevelTypeImpl.h"
>>> #include "llvm/Support/ManagedStatic.h"
>>> #include "llvm/Support/MathExtras.h"
>>> #include "llvm/Support/MemoryBuffer.h"
>>> #include "llvm/Support/PrettyStackTrace.h"
>>> #include "llvm/Support/Regex.h"
>>> #include "llvm/Support/SMLoc.h"
>>> #include "llvm/Support/ScopedPrinter.h"
>>> #include "llvm/Support/Signals.h"
>>> #include "llvm/Support/SourceMgr.h"
>>> #include "llvm/Support/raw_ostream.h"
>>> 
>>> #include "llvm/ADT/APInt.h"
>>> #include "llvm/ADT/ArrayRef.h"
>>> #include "llvm/ADT/BitVector.h"
>>> #include "llvm/ADT/CachedHashString.h"
>>> #include "llvm/ADT/DenseSet.h"
>>> #include "llvm/ADT/IndexedMap.h"
>>> #include "llvm/ADT/IntEqClasses.h"
>>> #include "llvm/ADT/MapVector.h"
>>> #include "llvm/ADT/Optional.h"
>>> #include "llvm/ADT/PointerUnion.h"
>>> #include "llvm/ADT/STLExtras.h"
>>> #include "llvm/ADT/SetVector.h"
>>> #include "llvm/ADT/SmallPtrSet.h"
>>> #include "llvm/ADT/SmallSet.h"
>>> #include "llvm/ADT/SmallVector.h"
>>> #include "llvm/ADT/SparseBitVector.h"
>>> #include "llvm/ADT/Statistic.h"
>>> #include "llvm/ADT/StringExtras.h"
>>> #include "llvm/ADT/StringMap.h"
>>> #include "llvm/ADT/StringRef.h"
>>> #include "llvm/ADT/StringSet.h"
>>> #include "llvm/ADT/StringSwitch.h"
>>> #include "llvm/ADT/TinyPtrVector.h"
>>> #include "llvm/ADT/Twine.h"
>>> 
>>> 
>>> Is this something worth putting effort into?  If so, I volunteer.
>>> 
>>> 
>>> 
>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
>> 
>>  -- 
>> Hal Finkel
>> Lead, Compiler Technology and Programming Languages
>> Leadership Computing Facility
>> Argonne National Laboratory
> 
> -- 
> Hal Finkel
> Lead, Compiler Technology and Programming Languages
> Leadership Computing Facility
> Argonne National Laboratory
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/b8e6804e/attachment.html>

Let me try starting over without mentioning Tablegen :)

Is Support too big, and if so should we consider breaking it up?

So far it sounds like the general consensus is "yes", with some
quibbles
about precisely how to split it.

I'm not necessarily opposed to smaller libraries like Triple as you
suggest, but i think that only addresses what to with largely disparate set
of stuff which I'm calling "narrowly useful".  It doesn't
really address
the fact that even after doing it at every logical division point, there
will still be a set of stuff that doesn't belong with other stuff.

To address that, I'm proposing something like "Base" which is
similar in
spirit to Support as it exists today, but more restrictive, and must
satisfy the "would this be useful outside of LLVM" bar.




On Tue, May 30, 2017 at 11:05 AM Matthias Braun via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> In my experience the buildsystem works fine in combination with tablegen
> (at least this aspect of it). The real problem here is that tablegen is
> just slow. Some of the X86 tables take indeed
>
> Last time I looked at it tablegen had still room to optimize the way it
> resolves class hierarchies and the variables within which it did basically
> one at a time, so it needed to traverse the hierarchies again for each
> variables.
>
> On May 27, 2017, at 12:54 PM, David Blaikie via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Maybe we do and build systems aren't respecting/noticing this? I'm
not
> sure.
>
> On Sat, May 27, 2017 at 12:50 PM Craig Topper <craig.topper at
gmail.com>
> wrote:
>
>> I thought we already did write tablegen output to temporary files like
>> X86GenAsmWriter1.inc.tmp first and then diffed them with the real .inc
file
>> and conditionally copied.
>>
>> ~Craig
>>
>> On Sat, May 27, 2017 at 11:02 AM, David Blaikie via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>>
>>>
>>> On Fri, May 26, 2017 at 8:06 PM Zachary Turner via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> It would be better, because a debug tablegen is slower than an
>>>> optimized tablegen, but it's still slow and it doesn't
address the problem
>>>> that tablegen runs *at all* when it doesn't really need to.
I think if
>>>> tablegen wasn't running all the time we could incremental
builds down from
>>>> 15 minutes (and that's on my really powerful machine) to
under 5, which
>>>> seemed like a big productivity boost averaged out over all
users x time
>>>>
>>>
>>> Is that 10 minutes of running tablegen, though? I think this
touches on
>>> Hal's comment:
>>> "Maybe we should use a diff-and-update approach (I thought,
however,
>>> that we already did that)."
>>>
>>> In that the 10 minutes is presumably from timestamp-based
invalidation
>>> chaining down from running tablegen, to then rebuilding everything
that
>>> depends on tablegen's output.
>>>
>>> That could be addressed by either the build system or tablegen
writing
>>> output to a temporary file, comparing the temp to the current file,
and not
>>> modifying the current file if they're the same - thus not
invalidating
>>> everything down the chain that depends on the tablegen output (not
sure if
>>> that's actually enough to convince a build system not to
rebuild other
>>> things - but worth a shot). This is a bit of a hackaround what the
build
>>> system itself could/should be doing, but in the absence of a
content based
>>> invalidation scheme - addressing this one particularly critical
point in
>>> LLVM's build seems like it could be worth such a workaround.
(this would
>>> speed up the build for even the times when tablegen's real
dependencies
>>> change - add a new function to ArrayRef, or STLExtras, etc... and
tablegen
>>> runs, but it doesn't cascade to everything else).
>>>
>>> (plus, opt build of tablegen, so that conclusion can be reached
more
>>> quickly)
>>>
>>> As Hal also mentioned, I'd be OK with this if there's a
good logical
>>> division that can be taken more than "whatever is/isn't
used in tablegen".
>>>
>>>
>>>>
>>>> We've got a lot of random stuff in Support, not all really
related and
>>>> not all relevant to every project. It seems like we should be
able to do
>>>> better.
>>>> On Fri, May 26, 2017 at 6:48 PM Hal Finkel <hfinkel at
anl.gov> wrote:
>>>>
>>>>>
>>>>> On 05/26/2017 07:59 PM, Zachary Turner wrote:
>>>>>
>>>>> It's that TableGen depends on Support, so if you change
one file in
>>>>> support, support gets recompiled into a new static archive,
which triggers
>>>>> a rerun of tablegen on all the tablegen inputs, which is
extremely slow.
>>>>>
>>>>>
>>>>> I thought that we had a "build an optimized tablegen
even in debug
>>>>> mode" setting to work around this problem. Would that
avoid this problem?
>>>>>
>>>>>
>>>>>  -Hal
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at
anl.gov> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev
wrote:
>>>>>>
>>>>>> Changing a header file somewhere and having to spend 10
minutes
>>>>>> waiting for a build leads to a lot of wasted developer
time.
>>>>>>
>>>>>> The real culprit here is tablegen.  Can we split
support and ADT into
>>>>>> two - the parts that tablegen depends on and the parts
that it doesn't?
>>>>>>
>>>>>>
>>>>>> What's the actual problem here? Is it that TableGen
regenerates
>>>>>> different files and so we then need to rebuild all
dependencies of those
>>>>>> files? Maybe we should use a diff-and-update approach
(I thought, however,
>>>>>> that we already did that).
>>>>>>
>>>>>>  -Hal
>>>>>>
>>>>>>
>>>>>> From what I can gather, Tablegen currently depends on
these headers
>>>>>> and all of their transitive dependencies.
>>>>>>
>>>>>> #include "llvm/Support/Casting.h"
>>>>>> #include "llvm/Support/CommandLine.h"
>>>>>> #include "llvm/Support/Compiler.h"
>>>>>> #include "llvm/Support/DataTypes.h"
>>>>>> #include "llvm/Support/Debug.h"
>>>>>> #include "llvm/Support/Error.h"
>>>>>> #include "llvm/Support/ErrorHandling.h"
>>>>>> #include "llvm/Support/Format.h"
>>>>>> #include "llvm/Support/FormattedStream.h"
>>>>>> #include "llvm/Support/LEB128.h"
>>>>>> #include "llvm/Support/LowLevelTypeImpl.h"
>>>>>> #include "llvm/Support/ManagedStatic.h"
>>>>>> #include "llvm/Support/MathExtras.h"
>>>>>> #include "llvm/Support/MemoryBuffer.h"
>>>>>> #include "llvm/Support/PrettyStackTrace.h"
>>>>>> #include "llvm/Support/Regex.h"
>>>>>> #include "llvm/Support/SMLoc.h"
>>>>>> #include "llvm/Support/ScopedPrinter.h"
>>>>>> #include "llvm/Support/Signals.h"
>>>>>> #include "llvm/Support/SourceMgr.h"
>>>>>> #include "llvm/Support/raw_ostream.h"
>>>>>>
>>>>>> #include "llvm/ADT/APInt.h"
>>>>>> #include "llvm/ADT/ArrayRef.h"
>>>>>> #include "llvm/ADT/BitVector.h"
>>>>>> #include "llvm/ADT/CachedHashString.h"
>>>>>> #include "llvm/ADT/DenseSet.h"
>>>>>> #include "llvm/ADT/IndexedMap.h"
>>>>>> #include "llvm/ADT/IntEqClasses.h"
>>>>>> #include "llvm/ADT/MapVector.h"
>>>>>> #include "llvm/ADT/Optional.h"
>>>>>> #include "llvm/ADT/PointerUnion.h"
>>>>>> #include "llvm/ADT/STLExtras.h"
>>>>>> #include "llvm/ADT/SetVector.h"
>>>>>> #include "llvm/ADT/SmallPtrSet.h"
>>>>>> #include "llvm/ADT/SmallSet.h"
>>>>>> #include "llvm/ADT/SmallVector.h"
>>>>>> #include "llvm/ADT/SparseBitVector.h"
>>>>>> #include "llvm/ADT/Statistic.h"
>>>>>> #include "llvm/ADT/StringExtras.h"
>>>>>> #include "llvm/ADT/StringMap.h"
>>>>>> #include "llvm/ADT/StringRef.h"
>>>>>> #include "llvm/ADT/StringSet.h"
>>>>>> #include "llvm/ADT/StringSwitch.h"
>>>>>> #include "llvm/ADT/TinyPtrVector.h"
>>>>>> #include "llvm/ADT/Twine.h"
>>>>>>
>>>>>>
>>>>>> Is this something worth putting effort into?  If so, I
volunteer.
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Hal Finkel
>>>>>> Lead, Compiler Technology and Programming Languages
>>>>>> Leadership Computing Facility
>>>>>> Argonne National Laboratory
>>>>>>
>>>>>>
>>>>> --
>>>>> Hal Finkel
>>>>> Lead, Compiler Technology and Programming Languages
>>>>> Leadership Computing Facility
>>>>> Argonne National Laboratory
>>>>>
>>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/e5a10a3c/attachment.html>

On Tue, May 30, 2017 at 12:06 PM Zachary Turner <zturner at google.com>
wrote:
> Let me try starting over without mentioning Tablegen :)
>
> Is Support too big, and if so should we consider breaking it up?
>
> So far it sounds like the general consensus is "yes", with some
quibbles
> about precisely how to split it.
>
I'll dissent here - if there are observable/meaningful benefits in
real-world build/iteration time, that seems like it'd motivate the work -
without that, I don't think there's necessarily an optimal size for a
library or that breaking up Support is especially valuable/good.

- Dave

>
> I'm not necessarily opposed to smaller libraries like Triple as you
> suggest, but i think that only addresses what to with largely disparate set
> of stuff which I'm calling "narrowly useful".  It doesn't
really address
> the fact that even after doing it at every logical division point, there
> will still be a set of stuff that doesn't belong with other stuff.
>
> To address that, I'm proposing something like "Base" which is
similar in
> spirit to Support as it exists today, but more restrictive, and must
> satisfy the "would this be useful outside of LLVM" bar.
>
>
>
>
> On Tue, May 30, 2017 at 11:05 AM Matthias Braun via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> In my experience the buildsystem works fine in combination with
tablegen
>> (at least this aspect of it). The real problem here is that tablegen is
>> just slow. Some of the X86 tables take indeed
>>
>> Last time I looked at it tablegen had still room to optimize the way it
>> resolves class hierarchies and the variables within which it did
basically
>> one at a time, so it needed to traverse the hierarchies again for each
>> variables.
>>
>> On May 27, 2017, at 12:54 PM, David Blaikie via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> Maybe we do and build systems aren't respecting/noticing this?
I'm not
>> sure.
>>
>> On Sat, May 27, 2017 at 12:50 PM Craig Topper <craig.topper at
gmail.com>
>> wrote:
>>
>>> I thought we already did write tablegen output to temporary files
like
>>> X86GenAsmWriter1.inc.tmp first and then diffed them with the real
.inc file
>>> and conditionally copied.
>>>
>>> ~Craig
>>>
>>> On Sat, May 27, 2017 at 11:02 AM, David Blaikie via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>>
>>>>
>>>> On Fri, May 26, 2017 at 8:06 PM Zachary Turner via llvm-dev
<
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> It would be better, because a debug tablegen is slower than
an
>>>>> optimized tablegen, but it's still slow and it
doesn't address the problem
>>>>> that tablegen runs *at all* when it doesn't really need
to. I think if
>>>>> tablegen wasn't running all the time we could
incremental builds down from
>>>>> 15 minutes (and that's on my really powerful machine)
to under 5, which
>>>>> seemed like a big productivity boost averaged out over all
users x time
>>>>>
>>>>
>>>> Is that 10 minutes of running tablegen, though? I think this
touches on
>>>> Hal's comment:
>>>> "Maybe we should use a diff-and-update approach (I
thought, however,
>>>> that we already did that)."
>>>>
>>>> In that the 10 minutes is presumably from timestamp-based
invalidation
>>>> chaining down from running tablegen, to then rebuilding
everything that
>>>> depends on tablegen's output.
>>>>
>>>> That could be addressed by either the build system or tablegen
writing
>>>> output to a temporary file, comparing the temp to the current
file, and not
>>>> modifying the current file if they're the same - thus not
invalidating
>>>> everything down the chain that depends on the tablegen output
(not sure if
>>>> that's actually enough to convince a build system not to
rebuild other
>>>> things - but worth a shot). This is a bit of a hackaround what
the build
>>>> system itself could/should be doing, but in the absence of a
content based
>>>> invalidation scheme - addressing this one particularly critical
point in
>>>> LLVM's build seems like it could be worth such a
workaround. (this would
>>>> speed up the build for even the times when tablegen's real
dependencies
>>>> change - add a new function to ArrayRef, or STLExtras, etc...
and tablegen
>>>> runs, but it doesn't cascade to everything else).
>>>>
>>>> (plus, opt build of tablegen, so that conclusion can be reached
more
>>>> quickly)
>>>>
>>>> As Hal also mentioned, I'd be OK with this if there's a
good logical
>>>> division that can be taken more than "whatever
is/isn't used in tablegen".
>>>>
>>>>
>>>>>
>>>>> We've got a lot of random stuff in Support, not all
really related and
>>>>> not all relevant to every project. It seems like we should
be able to do
>>>>> better.
>>>>> On Fri, May 26, 2017 at 6:48 PM Hal Finkel <hfinkel at
anl.gov> wrote:
>>>>>
>>>>>>
>>>>>> On 05/26/2017 07:59 PM, Zachary Turner wrote:
>>>>>>
>>>>>> It's that TableGen depends on Support, so if you
change one file in
>>>>>> support, support gets recompiled into a new static
archive, which triggers
>>>>>> a rerun of tablegen on all the tablegen inputs, which
is extremely slow.
>>>>>>
>>>>>>
>>>>>> I thought that we had a "build an optimized
tablegen even in debug
>>>>>> mode" setting to work around this problem. Would
that avoid this problem?
>>>>>>
>>>>>>
>>>>>>  -Hal
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel
at anl.gov> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev
wrote:
>>>>>>>
>>>>>>> Changing a header file somewhere and having to
spend 10 minutes
>>>>>>> waiting for a build leads to a lot of wasted
developer time.
>>>>>>>
>>>>>>> The real culprit here is tablegen.  Can we split
support and ADT
>>>>>>> into two - the parts that tablegen depends on and
the parts that it doesn't?
>>>>>>>
>>>>>>>
>>>>>>> What's the actual problem here? Is it that
TableGen regenerates
>>>>>>> different files and so we then need to rebuild all
dependencies of those
>>>>>>> files? Maybe we should use a diff-and-update
approach (I thought, however,
>>>>>>> that we already did that).
>>>>>>>
>>>>>>>  -Hal
>>>>>>>
>>>>>>>
>>>>>>> From what I can gather, Tablegen currently depends
on these headers
>>>>>>> and all of their transitive dependencies.
>>>>>>>
>>>>>>> #include "llvm/Support/Casting.h"
>>>>>>> #include "llvm/Support/CommandLine.h"
>>>>>>> #include "llvm/Support/Compiler.h"
>>>>>>> #include "llvm/Support/DataTypes.h"
>>>>>>> #include "llvm/Support/Debug.h"
>>>>>>> #include "llvm/Support/Error.h"
>>>>>>> #include "llvm/Support/ErrorHandling.h"
>>>>>>> #include "llvm/Support/Format.h"
>>>>>>> #include "llvm/Support/FormattedStream.h"
>>>>>>> #include "llvm/Support/LEB128.h"
>>>>>>> #include
"llvm/Support/LowLevelTypeImpl.h"
>>>>>>> #include "llvm/Support/ManagedStatic.h"
>>>>>>> #include "llvm/Support/MathExtras.h"
>>>>>>> #include "llvm/Support/MemoryBuffer.h"
>>>>>>> #include
"llvm/Support/PrettyStackTrace.h"
>>>>>>> #include "llvm/Support/Regex.h"
>>>>>>> #include "llvm/Support/SMLoc.h"
>>>>>>> #include "llvm/Support/ScopedPrinter.h"
>>>>>>> #include "llvm/Support/Signals.h"
>>>>>>> #include "llvm/Support/SourceMgr.h"
>>>>>>> #include "llvm/Support/raw_ostream.h"
>>>>>>>
>>>>>>> #include "llvm/ADT/APInt.h"
>>>>>>> #include "llvm/ADT/ArrayRef.h"
>>>>>>> #include "llvm/ADT/BitVector.h"
>>>>>>> #include "llvm/ADT/CachedHashString.h"
>>>>>>> #include "llvm/ADT/DenseSet.h"
>>>>>>> #include "llvm/ADT/IndexedMap.h"
>>>>>>> #include "llvm/ADT/IntEqClasses.h"
>>>>>>> #include "llvm/ADT/MapVector.h"
>>>>>>> #include "llvm/ADT/Optional.h"
>>>>>>> #include "llvm/ADT/PointerUnion.h"
>>>>>>> #include "llvm/ADT/STLExtras.h"
>>>>>>> #include "llvm/ADT/SetVector.h"
>>>>>>> #include "llvm/ADT/SmallPtrSet.h"
>>>>>>> #include "llvm/ADT/SmallSet.h"
>>>>>>> #include "llvm/ADT/SmallVector.h"
>>>>>>> #include "llvm/ADT/SparseBitVector.h"
>>>>>>> #include "llvm/ADT/Statistic.h"
>>>>>>> #include "llvm/ADT/StringExtras.h"
>>>>>>> #include "llvm/ADT/StringMap.h"
>>>>>>> #include "llvm/ADT/StringRef.h"
>>>>>>> #include "llvm/ADT/StringSet.h"
>>>>>>> #include "llvm/ADT/StringSwitch.h"
>>>>>>> #include "llvm/ADT/TinyPtrVector.h"
>>>>>>> #include "llvm/ADT/Twine.h"
>>>>>>>
>>>>>>>
>>>>>>> Is this something worth putting effort into?  If
so, I volunteer.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Hal Finkel
>>>>>>> Lead, Compiler Technology and Programming Languages
>>>>>>> Leadership Computing Facility
>>>>>>> Argonne National Laboratory
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Hal Finkel
>>>>>> Lead, Compiler Technology and Programming Languages
>>>>>> Leadership Computing Facility
>>>>>> Argonne National Laboratory
>>>>>>
>>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/95ff8c3b/attachment.html>

> On May 30, 2017, at 12:05 PM, Zachary Turner via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Let me try starting over without mentioning Tablegen :)
Aw, but i had a list of issues, such as:
- tablegen doesn’t generate .d files
- CMake is set up with far reaching deps on .td files, so changing Options.td
rebuilds the backends, and adding an ARM intrinsic rebuilds X86 or vice versa
- All the tablegen backends are linked in to a single binary, so if you are
hacking on tablegen and change the disassemblers backend, you rebuild the
SelectionDAG code
> 
> Is Support too big, and if so should we consider breaking it up?
Yes please.  Even if it didn’t have a signifiant impact on build time, its
become something of a dumping ground for code which would otherwise cause
layering violations if put somewhere else.

The way I see it, there’s a bunch of functionality which effectively just wraps
the platform, (i.e., anything which abstracts away posix/unix vs Windows vs
macOS).  This is often stuff like file systems, streams, the host itself and
probably a few others.  Then there is obviously ADT.  There are utilities which
basically everyone uses like Error, Debug, Statistic and stack trace.  There are
utilities which only tools *should* use, like CommandLine, and utilities only
used by a few projects right now, but maybe more in future, like Threading.

Then there are things which should now live elsewhere because perhaps other
libraries exist now which didn’t before.  ELF.h and others should be in
libObjectFile.  If they can’t be there, then we should have a very good reason
why not.

BranchProbability shouldn’t be there at all but probably is due to layering, and
likely the same goes for TargetRegistry.  We should fix that.  Same goes for
ARM* which should all be in the ARM backend.  Dwarf should be in
libDebugInfoDWARF.  *DeltaAlgorithm aren’t even used outside of tests!

Anyway, sorry for the brain dump.  But yes, this has grown in to something which
can be cleaner.  Perhaps Support will still exist to wrap posix and friends,
hopefully with ADT separately, but we should also move a bunch of things to
better suited libraries if we can.

Cheers,
Pete
> 
> So far it sounds like the general consensus is "yes", with some
quibbles about precisely how to split it.
> 
> I'm not necessarily opposed to smaller libraries like Triple as you
suggest, but i think that only addresses what to with largely disparate set of
stuff which I'm calling "narrowly useful".  It doesn't really
address the fact that even after doing it at every logical division point, there
will still be a set of stuff that doesn't belong with other stuff.
> 
> To address that, I'm proposing something like "Base" which is
similar in spirit to Support as it exists today, but more restrictive, and must
satisfy the "would this be useful outside of LLVM" bar.
> 
> 
> 
> 
> On Tue, May 30, 2017 at 11:05 AM Matthias Braun via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> In my experience the buildsystem works fine in combination with tablegen
(at least this aspect of it). The real problem here is that tablegen is just
slow. Some of the X86 tables take indeed
> 
> Last time I looked at it tablegen had still room to optimize the way it
resolves class hierarchies and the variables within which it did basically one
at a time, so it needed to traverse the hierarchies again for each variables.
> 
>> On May 27, 2017, at 12:54 PM, David Blaikie via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> Maybe we do and build systems aren't respecting/noticing this?
I'm not sure.
>> 
>> On Sat, May 27, 2017 at 12:50 PM Craig Topper <craig.topper at
gmail.com <mailto:craig.topper at gmail.com>> wrote:
>> I thought we already did write tablegen output to temporary files like
X86GenAsmWriter1.inc.tmp first and then diffed them with the real .inc file and
conditionally copied.
>> 
>> ~Craig
>> 
>> On Sat, May 27, 2017 at 11:02 AM, David Blaikie via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>> 
>> 
>> On Fri, May 26, 2017 at 8:06 PM Zachary Turner via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>> It would be better, because a debug tablegen is slower than an
optimized tablegen, but it's still slow and it doesn't address the
problem that tablegen runs *at all* when it doesn't really need to. I think
if tablegen wasn't running all the time we could incremental builds down
from 15 minutes (and that's on my really powerful machine) to under 5, which
seemed like a big productivity boost averaged out over all users x time
>> 
>> Is that 10 minutes of running tablegen, though? I think this touches on
Hal's comment:
>> "Maybe we should use a diff-and-update approach (I thought,
however, that we already did that)."
>> 
>> In that the 10 minutes is presumably from timestamp-based invalidation
chaining down from running tablegen, to then rebuilding everything that depends
on tablegen's output.
>> 
>> That could be addressed by either the build system or tablegen writing
output to a temporary file, comparing the temp to the current file, and not
modifying the current file if they're the same - thus not invalidating
everything down the chain that depends on the tablegen output (not sure if
that's actually enough to convince a build system not to rebuild other
things - but worth a shot). This is a bit of a hackaround what the build system
itself could/should be doing, but in the absence of a content based invalidation
scheme - addressing this one particularly critical point in LLVM's build
seems like it could be worth such a workaround. (this would speed up the build
for even the times when tablegen's real dependencies change - add a new
function to ArrayRef, or STLExtras, etc... and tablegen runs, but it doesn't
cascade to everything else).
>> 
>> (plus, opt build of tablegen, so that conclusion can be reached more
quickly)
>> 
>> As Hal also mentioned, I'd be OK with this if there's a good
logical division that can be taken more than "whatever is/isn't used in
tablegen".
>>  
>> 
>> We've got a lot of random stuff in Support, not all really related
and not all relevant to every project. It seems like we should be able to do
better.
>> On Fri, May 26, 2017 at 6:48 PM Hal Finkel <hfinkel at anl.gov
<mailto:hfinkel at anl.gov>> wrote:
>> 
>> On 05/26/2017 07:59 PM, Zachary Turner wrote:
>>> It's that TableGen depends on Support, so if you change one
file in support, support gets recompiled into a new static archive, which
triggers a rerun of tablegen on all the tablegen inputs, which is extremely
slow.
>> 
>> I thought that we had a "build an optimized tablegen even in debug
mode" setting to work around this problem. Would that avoid this problem?
>> 
>> 
>>  -Hal
>> 
>> 
>>> 
>>> On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov
<mailto:hfinkel at anl.gov>> wrote:
>>> 
>>> 
>>> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:
>>>> Changing a header file somewhere and having to spend 10 minutes
waiting for a build leads to a lot of wasted developer time.
>>>> 
>>>> The real culprit here is tablegen.  Can we split support and
ADT into two - the parts that tablegen depends on and the parts that it
doesn't?
>>> 
>>> What's the actual problem here? Is it that TableGen regenerates
different files and so we then need to rebuild all dependencies of those files?
Maybe we should use a diff-and-update approach (I thought, however, that we
already did that).
>>> 
>>>  -Hal
>>> 
>>>> 
>>>> From what I can gather, Tablegen currently depends on these
headers and all of their transitive dependencies.
>>>> 
>>>> #include "llvm/Support/Casting.h"
>>>> #include "llvm/Support/CommandLine.h"
>>>> #include "llvm/Support/Compiler.h"
>>>> #include "llvm/Support/DataTypes.h"
>>>> #include "llvm/Support/Debug.h"
>>>> #include "llvm/Support/Error.h"
>>>> #include "llvm/Support/ErrorHandling.h"
>>>> #include "llvm/Support/Format.h"
>>>> #include "llvm/Support/FormattedStream.h"
>>>> #include "llvm/Support/LEB128.h"
>>>> #include "llvm/Support/LowLevelTypeImpl.h"
>>>> #include "llvm/Support/ManagedStatic.h"
>>>> #include "llvm/Support/MathExtras.h"
>>>> #include "llvm/Support/MemoryBuffer.h"
>>>> #include "llvm/Support/PrettyStackTrace.h"
>>>> #include "llvm/Support/Regex.h"
>>>> #include "llvm/Support/SMLoc.h"
>>>> #include "llvm/Support/ScopedPrinter.h"
>>>> #include "llvm/Support/Signals.h"
>>>> #include "llvm/Support/SourceMgr.h"
>>>> #include "llvm/Support/raw_ostream.h"
>>>> 
>>>> #include "llvm/ADT/APInt.h"
>>>> #include "llvm/ADT/ArrayRef.h"
>>>> #include "llvm/ADT/BitVector.h"
>>>> #include "llvm/ADT/CachedHashString.h"
>>>> #include "llvm/ADT/DenseSet.h"
>>>> #include "llvm/ADT/IndexedMap.h"
>>>> #include "llvm/ADT/IntEqClasses.h"
>>>> #include "llvm/ADT/MapVector.h"
>>>> #include "llvm/ADT/Optional.h"
>>>> #include "llvm/ADT/PointerUnion.h"
>>>> #include "llvm/ADT/STLExtras.h"
>>>> #include "llvm/ADT/SetVector.h"
>>>> #include "llvm/ADT/SmallPtrSet.h"
>>>> #include "llvm/ADT/SmallSet.h"
>>>> #include "llvm/ADT/SmallVector.h"
>>>> #include "llvm/ADT/SparseBitVector.h"
>>>> #include "llvm/ADT/Statistic.h"
>>>> #include "llvm/ADT/StringExtras.h"
>>>> #include "llvm/ADT/StringMap.h"
>>>> #include "llvm/ADT/StringRef.h"
>>>> #include "llvm/ADT/StringSet.h"
>>>> #include "llvm/ADT/StringSwitch.h"
>>>> #include "llvm/ADT/TinyPtrVector.h"
>>>> #include "llvm/ADT/Twine.h"
>>>> 
>>>> 
>>>> Is this something worth putting effort into?  If so, I
volunteer.
>>>> 
>>>> 
>>>> 
>>> 
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>> 
>>> 
>>>  -- 
>>> Hal Finkel
>>> Lead, Compiler Technology and Programming Languages
>>> Leadership Computing Facility
>>> Argonne National Laboratory
>> 
>> -- 
>> Hal Finkel
>> Lead, Compiler Technology and Programming Languages
>> Leadership Computing Facility
>> Argonne National Laboratory
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/3f930565/attachment-0001.html>