llvm dev - Jan 2018 - Layering Requirements in the LLVM Coding Style Guide

I would describe it from this angle: LLVM is layered just fine. Usually the
layering is enforced as we don't link all libraries to all targets and you
will notice missing symbols if you violate it. It just happens that you can
violate the layering with header-only implementations of features that are not
catched this way and sure enough we a handful of cases that violate the layering
this way as David nicely explained here.

I don't think there is a reason not to fix those layering violations. We
just need a plan on how to fix them.

- Matthias
> On Jan 16, 2018, at 10:15 AM, Robinson, Paul via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> I have found layering to be a particularly useful and beneficial model in
past large software projects.
>  
> Is LLVM's layering actually written down anywhere?  Last time I went
looking, there was nothing.  If there's no spec, there's no verifiable
conformance; you have to guess based on what other files do.
> --paulr
>   <>
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
David Blaikie via llvm-dev
> Sent: Tuesday, January 16, 2018 9:22 AM
> To: llvm-dev; Richard Smith; Chandler Carruth; Reid Kleckner
> Subject: [llvm-dev] Layering Requirements in the LLVM Coding Style Guide
>  
> Context: I've been looking at experimenting with using Modular Code
Generation (My talk at last year's LLVM dev meeting
https://www.youtube.com/watch?v=lYYxDXgbUZ0
<https://www.youtube.com/watch?v=lYYxDXgbUZ0> is about the best reference
at the moment) when building the LLVM project, as a good experiment for the
feature. This can/does enforce a stronger layering invariant than LLVM has
historically been enforced. So I'm curious to get buy-in and maybe document
this if it's something people like the idea of.
> 
> I'm starting this discussion here rather than in an actual code review
on llvm-commits since it seems like it could do with a bit of a wider
discussion, but once/if the general direction is agreed on, I'll send a
patch for review of specific wording for the LLVM Coding Standards.
> 
> 
> Currently the LLVM Coding Standards
<https://llvm.org/docs/CodingStandards.html> doesn't say much/anything
about layering. 'A Public Header File is a Module'
<https://llvm.org/docs/CodingStandards.html#a-public-header-file-is-a-module>
section talks about modules of functionality, mostly trying to describe why a
header file should be self contained - but uses anachronistic language about
modules that doesn't line up with the implicit or explicit modules concepts
in use today, I think.
> 
> I propose making this wording a bit more explicit, including:
> 
> 1) Headers should be standalone (include all their dependencies - this is
mentioned in the "is a Module" piece, by way of a technique to help
ensure this, but not explicit as a goal itself).
> 
> 2) Files intended to be included in a particular context (that aren't
safe/benign to include multiple times, in multiple .cpp files, etc) should use a
'.inc' or '.def' (.def specifically for those "define a
macro, include the header which will reference that macro" style setups we
have in a few places).
> 
> And the actual layering issue:
> 3) Each library should only include headers or otherwise reference entities
from libraries it depends on. Including in headers and inline functions. A
simple/explicit way to put this: every inline function should be able to be
moved into a .cpp file and the build (with a unix linker - one that cannot
handle circular library dependencies) should still succeed.
> 
> 
> This last point is the most interesting - and I hope one that people
generally find desirable, so it might not be immediately obvious why it may be
contentious or difficult:
> 
> LLVM violates this constraint by using inline functions in headers to avoid
certain layering constraints that might otherwise cause the build to fail. A
couple of major examples I've hit are:
> 
> TargetSelect.h 
<http://lists.llvm.org/pipermail/llvm-dev/2017-December/119494.html>and
similar: This one's especially tricky - the header is part of libSupport,
but each function in here depends on a different subset of targets (creating a
circular dependency) - to call the given function the programmer needs to choose
the right dependencies to link to or the program will not link.
> Clang Diagnostics <https://reviews.llvm.org/D41357> (work in
progress): The diagnostics for each component are in their own component
directories, but are then all included from libClangBasic, a library none of
those components depends on. (so this isn't so much an inlining case as
#include based circular dependency)
> 
> 
> Generally I'd like to get buy-in that stricter layering is desirable,
and that these few cases are at least sub-optimal, if accepted for now.
> 
> Happy to go into more details about any of this, examples, etc, but I
realize this is already a bit long.
> - Dave
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180116/53745cdb/attachment.html>

On Tue, Jan 16, 2018 at 10:43 AM Matthias Braun <mbraun at apple.com>
wrote:
> I would describe it from this angle: LLVM is layered just fine.
>
Yeah, in most cases/in general I agree.

> Usually the layering is enforced as we don't link all libraries to all
> targets and you will notice missing symbols if you violate it.
>
Actually even more than that - on unix, linkers don't resolve circular
dependencies (they start with a list of unresolved symbols and walk the
link line left to right - resolving any symbols they can iteratively while
looking at one library, then moving on to the next - never going back to an
earlier library to resolve a later dependency), this enforces quite a lot
of the layering constraints we usually think about. Google's build system
enforces things a bit more strictly, I think (even without modular codegen)
& that's generally been a good/accepted thing - though I don't have
any
precise examples to point to, unfortunately.

> It just happens that you can violate the layering with header-only
> implementations of features that are not catched this way and sure enough
> we a handful of cases that violate the layering this way as David nicely
> explained here.
>
> I don't think there is a reason not to fix those layering violations.
We
> just need a plan on how to fix them.
>
Yep - my goal is just to enshrine that understanding in documentation so
it's a bit more of an explicit/clear goal going forward.

I may be able to fix some of the existing violations - but mostly wanting
to sure up things for future changes.

- Dave

>
> - Matthias
>
> On Jan 16, 2018, at 10:15 AM, Robinson, Paul via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> I have found layering to be a particularly useful and beneficial model in
> past large software projects.
>
> Is LLVM's layering actually written down anywhere?  Last time I went
> looking, there was nothing.  If there's no spec, there's no
verifiable
> conformance; you have to guess based on what other files do.
> --paulr
>
> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org
> <llvm-dev-bounces at lists.llvm.org>] *On Behalf Of *David Blaikie
via
> llvm-dev
> *Sent:* Tuesday, January 16, 2018 9:22 AM
> *To:* llvm-dev; Richard Smith; Chandler Carruth; Reid Kleckner
> *Subject:* [llvm-dev] Layering Requirements in the LLVM Coding Style Guide
>
> Context: I've been looking at experimenting with using Modular Code
> Generation (My talk at last year's LLVM dev meeting
> https://www.youtube.com/watch?v=lYYxDXgbUZ0 is about the best reference
> at the moment) when building the LLVM project, as a good experiment for the
> feature. This can/does enforce a stronger layering invariant than LLVM has
> historically been enforced. So I'm curious to get buy-in and maybe
document
> this if it's something people like the idea of.
>
> I'm starting this discussion here rather than in an actual code review
on
> llvm-commits since it seems like it could do with a bit of a wider
> discussion, but once/if the general direction is agreed on, I'll send a
> patch for review of specific wording for the LLVM Coding Standards.
>
>
> Currently the LLVM Coding Standards
> <https://llvm.org/docs/CodingStandards.html> doesn't say
much/anything
> about layering. 'A Public Header File *is* a Module'
>
<https://llvm.org/docs/CodingStandards.html#a-public-header-file-is-a-module>
section
> talks about modules of functionality, mostly trying to describe why a
> header file should be self contained - but uses anachronistic language
> about modules that doesn't line up with the implicit or explicit
modules
> concepts in use today, I think.
>
> I propose making this wording a bit more explicit, including:
>
> 1) Headers should be standalone (include all their dependencies - this is
> mentioned in the "is a Module" piece, by way of a technique to
help ensure
> this, but not explicit as a goal itself).
>
> 2) Files intended to be included in a particular context (that aren't
> safe/benign to include multiple times, in multiple .cpp files, etc) should
> use a '.inc' or '.def' (.def specifically for those
"define a macro,
> include the header which will reference that macro" style setups we
have in
> a few places).
>
> And the actual layering issue:
> 3) Each library should only include headers or otherwise reference
> entities from libraries it depends on. Including in headers and inline
> functions. A simple/explicit way to put this: every inline function should
> be able to be moved into a .cpp file and the build (with a unix linker -
> one that cannot handle circular library dependencies) should still succeed.
>
>
> This last point is the most interesting - and I hope one that people
> generally find desirable, so it might not be immediately obvious why it may
> be contentious or difficult:
>
> LLVM violates this constraint by using inline functions in headers to
> avoid certain layering constraints that might otherwise cause the build to
> fail. A couple of major examples I've hit are:
>
> TargetSelect.h
>
<http://lists.llvm.org/pipermail/llvm-dev/2017-December/119494.html>and
> similar: This one's especially tricky - the header is part of
libSupport,
> but each function in here depends on a different subset of targets
> (creating a circular dependency) - to call the given function the
> programmer needs to choose the right dependencies to link to or the program
> will not link.
> Clang Diagnostics <https://reviews.llvm.org/D41357> (work in
progress):
> The diagnostics for each component are in their own component directories,
> but are then all included from libClangBasic, a library none of those
> components depends on. (so this isn't so much an inlining case as
#include
> based circular dependency)
>
>
> Generally I'd like to get buy-in that stricter layering is desirable,
and
> that these few cases are at least sub-optimal, if accepted for now.
>
> Happy to go into more details about any of this, examples, etc, but I
> realize this is already a bit long.
> - Dave
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180116/d0aab1d6/attachment.html>

On 16 Jan 2018, at 18:50, David Blaikie via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Actually even more than that - on unix, linkers don't resolve circular
dependencies
Unless your linker is LLD, in which case they do by default.

David