llvm dev - May 2017 - Should we split llvm Support and ADT?

Changing a header file somewhere and having to spend 10 minutes waiting for
a build leads to a lot of wasted developer time.

The real culprit here is tablegen.  Can we split support and ADT into two -
the parts that tablegen depends on and the parts that it doesn't?
>From what I can gather, Tablegen currently depends on these headers and all
of their transitive dependencies.

#include "llvm/Support/Casting.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Compiler.h"
#include "llvm/Support/DataTypes.h"
#include "llvm/Support/Debug.h"
#include "llvm/Support/Error.h"
#include "llvm/Support/ErrorHandling.h"
#include "llvm/Support/Format.h"
#include "llvm/Support/FormattedStream.h"
#include "llvm/Support/LEB128.h"
#include "llvm/Support/LowLevelTypeImpl.h"
#include "llvm/Support/ManagedStatic.h"
#include "llvm/Support/MathExtras.h"
#include "llvm/Support/MemoryBuffer.h"
#include "llvm/Support/PrettyStackTrace.h"
#include "llvm/Support/Regex.h"
#include "llvm/Support/SMLoc.h"
#include "llvm/Support/ScopedPrinter.h"
#include "llvm/Support/Signals.h"
#include "llvm/Support/SourceMgr.h"
#include "llvm/Support/raw_ostream.h"

#include "llvm/ADT/APInt.h"
#include "llvm/ADT/ArrayRef.h"
#include "llvm/ADT/BitVector.h"
#include "llvm/ADT/CachedHashString.h"
#include "llvm/ADT/DenseSet.h"
#include "llvm/ADT/IndexedMap.h"
#include "llvm/ADT/IntEqClasses.h"
#include "llvm/ADT/MapVector.h"
#include "llvm/ADT/Optional.h"
#include "llvm/ADT/PointerUnion.h"
#include "llvm/ADT/STLExtras.h"
#include "llvm/ADT/SetVector.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallSet.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/SparseBitVector.h"
#include "llvm/ADT/Statistic.h"
#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringMap.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/StringSet.h"
#include "llvm/ADT/StringSwitch.h"
#include "llvm/ADT/TinyPtrVector.h"
#include "llvm/ADT/Twine.h"


Is this something worth putting effort into?  If so, I volunteer.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170527/4f933182/attachment.html>

On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev
wrote:
> Changing a header file somewhere and having to spend 10 minutes 
> waiting for a build leads to a lot of wasted developer time.
>
> The real culprit here is tablegen.  Can we split support and ADT into 
> two - the parts that tablegen depends on and the parts that it doesn't?
What's the actual problem here? Is it that TableGen regenerates 
different files and so we then need to rebuild all dependencies of those 
files? Maybe we should use a diff-and-update approach (I thought, 
however, that we already did that).

  -Hal
>
> From what I can gather, Tablegen currently depends on these headers 
> and all of their transitive dependencies.
>
> #include "llvm/Support/Casting.h"
> #include "llvm/Support/CommandLine.h"
> #include "llvm/Support/Compiler.h"
> #include "llvm/Support/DataTypes.h"
> #include "llvm/Support/Debug.h"
> #include "llvm/Support/Error.h"
> #include "llvm/Support/ErrorHandling.h"
> #include "llvm/Support/Format.h"
> #include "llvm/Support/FormattedStream.h"
> #include "llvm/Support/LEB128.h"
> #include "llvm/Support/LowLevelTypeImpl.h"
> #include "llvm/Support/ManagedStatic.h"
> #include "llvm/Support/MathExtras.h"
> #include "llvm/Support/MemoryBuffer.h"
> #include "llvm/Support/PrettyStackTrace.h"
> #include "llvm/Support/Regex.h"
> #include "llvm/Support/SMLoc.h"
> #include "llvm/Support/ScopedPrinter.h"
> #include "llvm/Support/Signals.h"
> #include "llvm/Support/SourceMgr.h"
> #include "llvm/Support/raw_ostream.h"
>
> #include "llvm/ADT/APInt.h"
> #include "llvm/ADT/ArrayRef.h"
> #include "llvm/ADT/BitVector.h"
> #include "llvm/ADT/CachedHashString.h"
> #include "llvm/ADT/DenseSet.h"
> #include "llvm/ADT/IndexedMap.h"
> #include "llvm/ADT/IntEqClasses.h"
> #include "llvm/ADT/MapVector.h"
> #include "llvm/ADT/Optional.h"
> #include "llvm/ADT/PointerUnion.h"
> #include "llvm/ADT/STLExtras.h"
> #include "llvm/ADT/SetVector.h"
> #include "llvm/ADT/SmallPtrSet.h"
> #include "llvm/ADT/SmallSet.h"
> #include "llvm/ADT/SmallVector.h"
> #include "llvm/ADT/SparseBitVector.h"
> #include "llvm/ADT/Statistic.h"
> #include "llvm/ADT/StringExtras.h"
> #include "llvm/ADT/StringMap.h"
> #include "llvm/ADT/StringRef.h"
> #include "llvm/ADT/StringSet.h"
> #include "llvm/ADT/StringSwitch.h"
> #include "llvm/ADT/TinyPtrVector.h"
> #include "llvm/ADT/Twine.h"
>
>
> Is this something worth putting effort into?  If so, I volunteer.
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170526/9871e599/attachment.html>

It's that TableGen depends on Support, so if you change one file in
support, support gets recompiled into a new static archive, which triggers
a rerun of tablegen on all the tablegen inputs, which is extremely slow.

On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov> wrote:
>
>
> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:
>
> Changing a header file somewhere and having to spend 10 minutes waiting
> for a build leads to a lot of wasted developer time.
>
> The real culprit here is tablegen.  Can we split support and ADT into two
> - the parts that tablegen depends on and the parts that it doesn't?
>
>
> What's the actual problem here? Is it that TableGen regenerates
different
> files and so we then need to rebuild all dependencies of those files? Maybe
> we should use a diff-and-update approach (I thought, however, that we
> already did that).
>
>  -Hal
>
>
> From what I can gather, Tablegen currently depends on these headers and
> all of their transitive dependencies.
>
> #include "llvm/Support/Casting.h"
> #include "llvm/Support/CommandLine.h"
> #include "llvm/Support/Compiler.h"
> #include "llvm/Support/DataTypes.h"
> #include "llvm/Support/Debug.h"
> #include "llvm/Support/Error.h"
> #include "llvm/Support/ErrorHandling.h"
> #include "llvm/Support/Format.h"
> #include "llvm/Support/FormattedStream.h"
> #include "llvm/Support/LEB128.h"
> #include "llvm/Support/LowLevelTypeImpl.h"
> #include "llvm/Support/ManagedStatic.h"
> #include "llvm/Support/MathExtras.h"
> #include "llvm/Support/MemoryBuffer.h"
> #include "llvm/Support/PrettyStackTrace.h"
> #include "llvm/Support/Regex.h"
> #include "llvm/Support/SMLoc.h"
> #include "llvm/Support/ScopedPrinter.h"
> #include "llvm/Support/Signals.h"
> #include "llvm/Support/SourceMgr.h"
> #include "llvm/Support/raw_ostream.h"
>
> #include "llvm/ADT/APInt.h"
> #include "llvm/ADT/ArrayRef.h"
> #include "llvm/ADT/BitVector.h"
> #include "llvm/ADT/CachedHashString.h"
> #include "llvm/ADT/DenseSet.h"
> #include "llvm/ADT/IndexedMap.h"
> #include "llvm/ADT/IntEqClasses.h"
> #include "llvm/ADT/MapVector.h"
> #include "llvm/ADT/Optional.h"
> #include "llvm/ADT/PointerUnion.h"
> #include "llvm/ADT/STLExtras.h"
> #include "llvm/ADT/SetVector.h"
> #include "llvm/ADT/SmallPtrSet.h"
> #include "llvm/ADT/SmallSet.h"
> #include "llvm/ADT/SmallVector.h"
> #include "llvm/ADT/SparseBitVector.h"
> #include "llvm/ADT/Statistic.h"
> #include "llvm/ADT/StringExtras.h"
> #include "llvm/ADT/StringMap.h"
> #include "llvm/ADT/StringRef.h"
> #include "llvm/ADT/StringSet.h"
> #include "llvm/ADT/StringSwitch.h"
> #include "llvm/ADT/TinyPtrVector.h"
> #include "llvm/ADT/Twine.h"
>
>
> Is this something worth putting effort into?  If so, I volunteer.
>
>
>
> _______________________________________________
> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> --
> Hal Finkel
> Lead, Compiler Technology and Programming Languages
> Leadership Computing Facility
> Argonne National Laboratory
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170527/7388c6ac/attachment.html>

On Fri, May 26, 2017 at 5:47 PM, Zachary Turner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Changing a header file somewhere and having to spend 10 minutes waiting
> for a build leads to a lot of wasted developer time.
>
> The real culprit here is tablegen.  Can we split support and ADT into two
> - the parts that tablegen depends on and the parts that it doesn't?
>
> From what I can gather, Tablegen currently depends on these headers and
> all of their transitive dependencies.
>
> #include "llvm/Support/Casting.h"
> #include "llvm/Support/CommandLine.h"
> #include "llvm/Support/Compiler.h"
> #include "llvm/Support/DataTypes.h"
> #include "llvm/Support/Debug.h"
> #include "llvm/Support/Error.h"
> #include "llvm/Support/ErrorHandling.h"
> #include "llvm/Support/Format.h"
> #include "llvm/Support/FormattedStream.h"
> #include "llvm/Support/LEB128.h"
> #include "llvm/Support/LowLevelTypeImpl.h"
> #include "llvm/Support/ManagedStatic.h"
> #include "llvm/Support/MathExtras.h"
> #include "llvm/Support/MemoryBuffer.h"
> #include "llvm/Support/PrettyStackTrace.h"
> #include "llvm/Support/Regex.h"
> #include "llvm/Support/SMLoc.h"
> #include "llvm/Support/ScopedPrinter.h"
> #include "llvm/Support/Signals.h"
> #include "llvm/Support/SourceMgr.h"
> #include "llvm/Support/raw_ostream.h"
>
> #include "llvm/ADT/APInt.h"
> #include "llvm/ADT/ArrayRef.h"
> #include "llvm/ADT/BitVector.h"
> #include "llvm/ADT/CachedHashString.h"
> #include "llvm/ADT/DenseSet.h"
> #include "llvm/ADT/IndexedMap.h"
> #include "llvm/ADT/IntEqClasses.h"
> #include "llvm/ADT/MapVector.h"
> #include "llvm/ADT/Optional.h"
> #include "llvm/ADT/PointerUnion.h"
> #include "llvm/ADT/STLExtras.h"
> #include "llvm/ADT/SetVector.h"
> #include "llvm/ADT/SmallPtrSet.h"
> #include "llvm/ADT/SmallSet.h"
> #include "llvm/ADT/SmallVector.h"
> #include "llvm/ADT/SparseBitVector.h"
> #include "llvm/ADT/Statistic.h"
> #include "llvm/ADT/StringExtras.h"
> #include "llvm/ADT/StringMap.h"
> #include "llvm/ADT/StringRef.h"
> #include "llvm/ADT/StringSet.h"
> #include "llvm/ADT/StringSwitch.h"
> #include "llvm/ADT/TinyPtrVector.h"
> #include "llvm/ADT/Twine.h"
>
>
> Is this something worth putting effort into?  If so, I volunteer.
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
Just a note. Support used to be two separate libraries, support and system.
These were merged (7 years ago?) due to the need for and existence of cross
dependencies between them. I would be concerned with any split that brought
back these issues where adding an include changed which library something
needed to be in.

- Michael Spencer
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170531/38e4e25b/attachment.html>

Was this any different than the existing cross-dependencies that exist
between ADT and Support?

On Wed, May 31, 2017 at 5:04 PM Michael Spencer <bigcheesegs at gmail.com>
wrote:
> On Fri, May 26, 2017 at 5:47 PM, Zachary Turner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Changing a header file somewhere and having to spend 10 minutes waiting
>> for a build leads to a lot of wasted developer time.
>>
>> The real culprit here is tablegen.  Can we split support and ADT into
two
>> - the parts that tablegen depends on and the parts that it doesn't?
>>
>> From what I can gather, Tablegen currently depends on these headers and
>> all of their transitive dependencies.
>>
>> #include "llvm/Support/Casting.h"
>> #include "llvm/Support/CommandLine.h"
>> #include "llvm/Support/Compiler.h"
>> #include "llvm/Support/DataTypes.h"
>> #include "llvm/Support/Debug.h"
>> #include "llvm/Support/Error.h"
>> #include "llvm/Support/ErrorHandling.h"
>> #include "llvm/Support/Format.h"
>> #include "llvm/Support/FormattedStream.h"
>> #include "llvm/Support/LEB128.h"
>> #include "llvm/Support/LowLevelTypeImpl.h"
>> #include "llvm/Support/ManagedStatic.h"
>> #include "llvm/Support/MathExtras.h"
>> #include "llvm/Support/MemoryBuffer.h"
>> #include "llvm/Support/PrettyStackTrace.h"
>> #include "llvm/Support/Regex.h"
>> #include "llvm/Support/SMLoc.h"
>> #include "llvm/Support/ScopedPrinter.h"
>> #include "llvm/Support/Signals.h"
>> #include "llvm/Support/SourceMgr.h"
>> #include "llvm/Support/raw_ostream.h"
>>
>> #include "llvm/ADT/APInt.h"
>> #include "llvm/ADT/ArrayRef.h"
>> #include "llvm/ADT/BitVector.h"
>> #include "llvm/ADT/CachedHashString.h"
>> #include "llvm/ADT/DenseSet.h"
>> #include "llvm/ADT/IndexedMap.h"
>> #include "llvm/ADT/IntEqClasses.h"
>> #include "llvm/ADT/MapVector.h"
>> #include "llvm/ADT/Optional.h"
>> #include "llvm/ADT/PointerUnion.h"
>> #include "llvm/ADT/STLExtras.h"
>> #include "llvm/ADT/SetVector.h"
>> #include "llvm/ADT/SmallPtrSet.h"
>> #include "llvm/ADT/SmallSet.h"
>> #include "llvm/ADT/SmallVector.h"
>> #include "llvm/ADT/SparseBitVector.h"
>> #include "llvm/ADT/Statistic.h"
>> #include "llvm/ADT/StringExtras.h"
>> #include "llvm/ADT/StringMap.h"
>> #include "llvm/ADT/StringRef.h"
>> #include "llvm/ADT/StringSet.h"
>> #include "llvm/ADT/StringSwitch.h"
>> #include "llvm/ADT/TinyPtrVector.h"
>> #include "llvm/ADT/Twine.h"
>>
>>
>> Is this something worth putting effort into?  If so, I volunteer.
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
> Just a note. Support used to be two separate libraries, support and
> system. These were merged (7 years ago?) due to the need for and existence
> of cross dependencies between them. I would be concerned with any split
> that brought back these issues where adding an include changed which
> library something needed to be in.
>
> - Michael Spencer
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170601/93183f2b/attachment.html>

> On May 26, 2017, at 5:47 PM, Zachary Turner via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Changing a header file somewhere and having to spend 10 minutes waiting for
a build leads to a lot of wasted developer time.
> 
> The real culprit here is tablegen.  Can we split support and ADT into two -
the parts that tablegen depends on and the parts that it doesn’t?
In all the comments downthread, I think there is one thing that hasn't been
mentioned: doing a split like this makes tblgen evolution more difficult.  If
libsupport was split into “used by tblgen” and “not used by tblgen” sections,
and then a new tblgen feature needs to use other parts of libsupport, they’d
have to be moved into the “used by tblgen” directory.

Splitting libsupport as a whole out into its own llvm subproject has come up
many times though, and does make a lot of sense.

-Chris

Fair enough, i sort of regret mentioning that specific method of splitting
originally.

For the record, i think any splitting should make sense on its own merit
without considering tablegen, and hopefully the end result of "tablegen
eventually depends on less stuff" would happen naturally
On Sun, Jun 4, 2017 at 10:37 AM Chris Lattner <clattner at nondot.org>
wrote:
>
> > On May 26, 2017, at 5:47 PM, Zachary Turner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > Changing a header file somewhere and having to spend 10 minutes
waiting
> for a build leads to a lot of wasted developer time.
> >
> > The real culprit here is tablegen.  Can we split support and ADT into
> two - the parts that tablegen depends on and the parts that it doesn’t?
>
> In all the comments downthread, I think there is one thing that hasn't
> been mentioned: doing a split like this makes tblgen evolution more
> difficult.  If libsupport was split into “used by tblgen” and “not used by
> tblgen” sections, and then a new tblgen feature needs to use other parts of
> libsupport, they’d have to be moved into the “used by tblgen” directory.
>
> Splitting libsupport as a whole out into its own llvm subproject has come
> up many times though, and does make a lot of sense.
>
> -Chris
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170604/0f8333e7/attachment.html>