llvm dev - Dec 2020 - RFC: Contributing Bazel BUILD files in the "peripheral" support tier

On Fri, 4 Dec 2020 at 00:27, Geoffrey Martin-Noble <gcmn at google.com>
wrote:
> It seems like Tom and Renato still disagree about whether I should move
> this to a pitch. I would appreciate some consensus on that point at least
> :-D I do see the appeal of a living document for this sort of thing, so
> definitely see the appeal there, but also it seems like the pitch process
> is a heavier-weight and more unusual one, so I was hesitant. My inclination
> is to continue this as an RFC unless we are unable to reach consensus on
> the issue as outlined in the pitch process description. It does feel like
> this is really not quite as big a decision as you seem to be suggesting.
> It's also an easily reversible one since there are no build
dependencies
> and everything is contained.
>
Hi Geoffrey,

I don't think we're disagreeing as much as trying to find the best way
to
do it.

Tom is worried on a meta level, both as cementing the precedent (GN was a
trial, BAZEL makes it official) and as complicating the merge process. I
agree with him 100%.

It's already complicated to make sure backports on various projects
don't
break other projects (especially in the core LLVM), and by adding build
systems to the mix, we'd be adding a new dimension in the problem space.

I am more worried about following different paths for different build
systems (non-overlapping features) and encouraging people to build with an
"alternative" build system because CMake yet doesn't support
something that
they do.

I really don't want to get to a point where each system has a set of unique
features, in which case, we'll have three "official" build
systems. I know
this isn't what you're proposing, but it's a likely outcome once we
"support" (core or peripheral) more than one.

We have had that before with autoconf, as I mentioned.


Out-of-tree with a submodule is the current approach we have
with
> https://github.com/google/llvm-bazel. It's certainly doable, but
involves
> quite a bit of bookkeeping to track which version corresponds to a given
> version of LLVM such that someone can fetch the correct configuration
> (you'll note that the repository has about 7k tags at the moment). To
make
> things somewhat more complicated, the typical way to fetch something for
> use in Bazel is with an http_archive
>
<https://docs.bazel.build/versions/master/repo/http.html#http_archive>
which
> requires one to specify the archive digest to avoid refetching on each
> build. This doesn't work particularly well with tags that change which
> commit they point to. I'm not saying these issues aren't solvable,
but they
> add quite a bit of complexity.
>
Right, having the file in-tree means the history is unique and linear, and
there's no need to create a map between BAZEL and LLVM revisions. This was
a strong point of moving to the monorepo.

I'd certainly be open to discussing restrictions that would
avoid
> additional burden on release managers. I think that one makes contributing
> to the Bazel configuration more difficult because you cannot do it as part
> of a patch that requires a change, but if it's something that would
cause
> issues with the release then we can avoid it. My intuition is that this
> wouldn't actually come up often, however. For example, just looking at
the
> gn directory I see several commits in the last week that touch this and
> other files. Have you actually run into issues? Since this is unsupported
> the conflicts could also be resolved pretty much however you wanted (e.g.
> delete the conflict markers, delete the file), so they seem pretty trivial
> to deal with if they only happen occasionally. My preference would
> therefore be to see if this is actually a problem in practice before
> putting rules in place.
>
The main problem here is backports. If there is a change in the code that
"needs" a relative change in GN/BAZEL, we'll have to track both
code and
build system changes to make sure the patches apply correctly, and if not,
apply a series of corrections on both sides to make it work. This is
already fragile with just code, adding a new thread won't make it easier.

It may not currently be a problem with GN because there isn't a push-back
(that I know of) if GN breaks on a stable release.

Perhaps this could be one of the "peripheral" support contract points:
stable releases are not required to make them work.

So, if a code change needs a BAZEL change and no one from the BAZEL
community picks up the tab in time, it will be backported without it and
BAZEL will be broken on that stable release.

I am happy to put this in either location and agree it should be in
the
> same place as GN. If we were to decide that it should go `utils/` then I
> would also propose we move GN to there as well. I believe the GN files were
> contributed prior to the existence of the monorepo, so a top-level `utils/`
> wouldn't have been an option. I think living under the root `utils/`
> directory makes more sense because these are not configurations for only
> the LLVM subproject (we also build MLIR and Clang with perhaps more to
> come). I believe it was Mehdi's suggestion that this would help
mitigate
> some of the costs to having it in the monorepo because Tom mentioned commit
> list traffic as a concern. I don't think I agree that one directory up
is
> akin to a separate repo though :-D
>
I see. There is a current effort to move CMake to the root directory, so
that'd be on par with GN and BAZEL. I don't mind where, as long as
they're
all in the same pattern.

I can really only speak for Google projects. I have also noticed
several
> other Bazel build configurations in the wild, e.g. PlaidML
>
<https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD>
(Intel)
> or this bazel_llvm <https://github.com/ChrisCummins/bazel_llvm>
project
> that I found after someone contributed a doc fix. I believe in the last
> thread someone from Facebook mentioned that Bazel build files would also be
> relatively easily translatable to their internal Bazel-derived build
> system, Buck. Someone from Lyft also expressed interest in using a Bazel
> build configuration if it was in-tree. But I can't really speak to the
> motivations, road maps, etc. for any of these people, companies, or
> projects (if you're reading, please chime in ;-P).
>
The purpose of this question was to understand how many new projects will
want to move inside the LLVM umbrella (core, peripheral, incubator) that
can only be built with BAZEL.

If we accept GN/BAZEL as a supported build system, we should still require
new projects to build with CMake, in addition to their native ones, if
different.

This could still "leave out" some features in the native build system
(if
supported) that haven't been moved to CMake, and that's how you get
disjoint support levels I mentioned above.

I'm optimistically hoping that the likelihood of this happening and the
number of projects will be low enough that we won't have that problem in
practice.

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201204/809f9203/attachment-0001.html>

Hi Renato,

As a summary: At its core I think that this is a "lowering the barrier to
entry" for llvm rather than a support issue. No one is suggesting that this
is supported other than by its users inside an easily removed directory
similar to how a lot of specific support has been handled.

Your questions are seeming to move the bar far past what anyone is asking
for and so it seems that there's a disconnect occurring here. I'll
answer a
few of your concerns inline.

> Tom is worried on a meta level, both as cementing the precedent (GN was a
> trial, BAZEL makes it official) and as complicating the merge process. I
> agree with him 100%.
>
>
Makes what official? I think the "support" story - "there is
none" - is
pretty clear here. As far as merging it's pretty clear this is a "user
supported" rather than "community supported" process. No one,
including me,
wants to add another supported llvm build system. And let's be honest, most
people didn't even notice that gn was added until it was pointed out and it
hasn't caused any issues. There's no assumption of anything working
anywhere.

> It's already complicated to make sure backports on various projects
don't
> break other projects (especially in the core LLVM), and by adding build
> systems to the mix, we'd be adding a new dimension in the problem
space.
>
> I am more worried about following different paths for different build
> systems (non-overlapping features) and encouraging people to build with an
> "alternative" build system because CMake yet doesn't support
something that
> they do.
>
> I really don't want to get to a point where each system has a set of
> unique features, in which case, we'll have three "official"
build systems.
> I know this isn't what you're proposing, but it's a likely
outcome once we
> "support" (core or peripheral) more than one.
>
> We have had that before with autoconf, as I mentioned.
>
>
There's no notion of having official build systems here and as the previous
build system maintainer and owner of autoconf I'm pretty sure this
doesn't
reflect how the situation was. The dual aspect of cmake and autoconf was an
explicit acknowledgment that we wanted to support both in tree while we
moved from one to the other. In addition, we still have the vestiges of
some other build systems in tree and tools used exclusively for outside
projects for simplicity.

> I can really only speak for Google projects. I have also noticed several
>> other Bazel build configurations in the wild, e.g. PlaidML
>>
<https://github.com/plaidml/plaidml/blob/master/vendor/llvm/llvm.BUILD>
(Intel)
>> or this bazel_llvm <https://github.com/ChrisCummins/bazel_llvm>
project
>> that I found after someone contributed a doc fix. I believe in the last
>> thread someone from Facebook mentioned that Bazel build files would
also be
>> relatively easily translatable to their internal Bazel-derived build
>> system, Buck. Someone from Lyft also expressed interest in using a
Bazel
>> build configuration if it was in-tree. But I can't really speak to
the
>> motivations, road maps, etc. for any of these people, companies, or
>> projects (if you're reading, please chime in ;-P).
>>
>
> The purpose of this question was to understand how many new projects will
> want to move inside the LLVM umbrella (core, peripheral, incubator) that
> can only be built with BAZEL.
>
>
This is a different question and one I don't think we need to address,
however, if it helps...

As you say here:

> If we accept GN/BAZEL as a supported build system, we should still require
> new projects to build with CMake, in addition to their native ones, if
> different.
>
>
I would be against any project that does not build with CMake being added
to the llvm project. If they want to add support (or even support another
build system) to one of the optional, and unsupported, build systems then
that's up to them or the people that care about those build systems. Same
with editor integrations :)

> This could still "leave out" some features in the native build
system (if
> supported) that haven't been moved to CMake, and that's how you get
> disjoint support levels I mentioned above.
>
>
As I said in my intro - at its core this isn't a support issue. Adding in
the capability of something being able to build with different build
systems helps the llvm project both be open to new contributors and
integrations. We don't need to support them more than the people care to
work on it, but having the capability means that people whose primary goal
is integrating llvm into various projects can have those projects' build
systems live alongside our supported meta-build system. This is basically
"tooling" in a directory that makes integrating with projects that
depend
upon llvm easier and lowers the barrier for people that work on those
projects to more easily work with llvm.

I hope this helps and would be happy to continue the discussion with you
either here, offline, or via video conference.

Thanks!

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201204/16ab5ee8/attachment.html>