Renato Golin via llvm-dev
2020-Oct-29 19:49 UTC
[llvm-dev] Contributing Bazel BUILD files similar to gn
On Thu, 29 Oct 2020 at 19:16, David Blaikie <dblaikie at gmail.com> wrote:> I /believe/ the idea is that, like gn, there are folks maintaining these > build systems out of tree anyway - and having them in tree makes it easier > to coordinate that effort, with the express intent of not burdening the > general community with their upkeep (like gn currently - the idea is that > there's no burden on developers to update gn build files (& consequently > bazel build files)). >Perhaps the initial assumption about my concerns weren't well articulated. I get that those files would be "additional" and other developers won't need to care much about them. But what happens when people join the project with experience in Bazel and, instead of building pure LLVM with CMake, they start using Bazel for everything, just because they're used to it? Bazel is big enough (at least inside Google) that the probability of that happening is not trivial. What if they create sub-projects that can only build with Bazel? Do we refuse inclusion? But don't we have Bazel files already? One big example is Android. They used to build LLVM in a very different way, and the inclusion of run-time library files was completely different. So different it was not possible to merge some changes they had (128 bit maths IIRC) because of the amount of work required. My point is that adding another build system will not necessarily improve the chances of external people contributing to LLVM if they use those build systems. It may very well *reduce* those chances. Once we get to the point where Bazel support is "complete" enough, and enough other projects that use LLVM use Bazel (I assume many internal Google projects), the problem I describe above is bound to happen sooner or later. Personally, I'm happy to ignore Bazel and continue using CMake. But I just wanted to make clear that in the past, using a different build system did not increase the chances of contribution, so that's not a given in this case either. cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/152841f5/attachment.html>
Mehdi Amini via llvm-dev
2020-Oct-29 20:46 UTC
[llvm-dev] Contributing Bazel BUILD files similar to gn
On Thu, Oct 29, 2020 at 12:49 PM Renato Golin <rengolin at gmail.com> wrote:> On Thu, 29 Oct 2020 at 19:16, David Blaikie <dblaikie at gmail.com> wrote: > >> I /believe/ the idea is that, like gn, there are folks maintaining these >> build systems out of tree anyway - and having them in tree makes it easier >> to coordinate that effort, with the express intent of not burdening the >> general community with their upkeep (like gn currently - the idea is that >> there's no burden on developers to update gn build files (& consequently >> bazel build files)). >> > > Perhaps the initial assumption about my concerns weren't well articulated. > > I get that those files would be "additional" and other developers won't > need to care much about them. > > But what happens when people join the project with experience in Bazel > and, instead of building pure LLVM with CMake, they start using Bazel for > everything, just because they're used to it? >I would propose to have the files in a separate tree from llvm/, mlir/, clang/ ; labelling these clearly as unsupported (either in the path to these files or in the README, or both), and not provide any public documentation on llvm.org that would invite users to work with these. The readme would explain how to use them to include LLVM as a dependency to an existing Bazel project and document the intent as such.> Bazel is big enough (at least inside Google) that the probability of that > happening is not trivial. > > What if they create sub-projects that can only build with Bazel? Do we > refuse inclusion? But don't we have Bazel files already? >This is a fair concern: can we defend against this with a clear policy? Also: no public bot with Bazel or other build system than CMake should help right? One big example is Android. They used to build LLVM in a very different> way, and the inclusion of run-time library files was completely different. > So different it was not possible to merge some changes they had (128 bit > maths IIRC) because of the amount of work required. > > My point is that adding another build system will not necessarily improve > the chances of external people contributing to LLVM if they use those build > systems. It may very well *reduce* those chances. >My intuition was that by having the file upstream, we would instead encourage such users to track the HEAD of our main branch more closely and so provide them an easier path for upstream work. The fact that they can get upstream working with their build environment may provide an incentive to upstream along the way, even if they have to do the CMake integration first. Ultimately while this may facilitate people to go in one direction or another, I suspect they would just reinforce their natural tendency: people interested in working more upstream will have a better path, and people who have less of this tendency may also have an easier path of integration. Once we get to the point where Bazel support is "complete" enough, and> enough other projects that use LLVM use Bazel (I assume many internal > Google projects), the problem I describe above is bound to happen sooner or > later. > > Personally, I'm happy to ignore Bazel and continue using CMake. But I just > wanted to make clear that in the past, using a different build system did > not increase the chances of contribution, so that's not a given in this > case either. > > cheers, > --renato >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/446ad628/attachment.html>
Renato Golin via llvm-dev
2020-Oct-29 21:15 UTC
[llvm-dev] Contributing Bazel BUILD files similar to gn
On Thu, 29 Oct 2020 at 20:46, Mehdi Amini <aminim at google.com> wrote:> I would propose to have the files in a separate tree from llvm/, mlir/, > clang/ ; labelling these clearly as unsupported (either in the path to > these files or in the README, or both), and not provide any public > documentation on llvm.org that would invite users to work with these. The > readme would explain how to use them to include LLVM as a dependency to an > existing Bazel project and document the intent as such. >Sounds good, with the addendum below: This is a fair concern: can we defend against this with a clear policy?> Also: no public bot with Bazel or other build system than CMake should > help right? >Right, as you said later, no emails from broken bots, ie. people that use non-CMake build systems are expected to fix their own builds, even if the breakage came from a third-party change. Of course, we expect other contributors to help with what they can, but it's not their responsibility. Such is the cost of having a different build system. My intuition was that by having the file upstream, we would instead> encourage such users to track the HEAD of our main branch more closely and > so provide them an easier path for upstream work. The fact that they can > get upstream working with their build environment may provide an incentive > to upstream along the way, even if they have to do the CMake integration > first. >The benefit to keep the files in LLVM is clear to all Bazel users out there. I don't think that's a problem for the rest of LLVM. My point was that the reason why Arm's patch (128-bit) was "uncontributable" was because their build system was so different, it was impossible to keep both versions on their merged tree. This is a big problem to the Bazel users, not CMake users. If we keep the policy of "not my problem", I don't see a single problem to CMake users. But that's slightly unfriendly to people that came later to LLVM and "didn't know better" before creating a whole project in Bazel, etc. I'm strictly not thinking about myself here. --renato>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/396b731b/attachment.html>
Zachary Turner via llvm-dev
2020-Oct-29 23:11 UTC
[llvm-dev] Contributing Bazel BUILD files similar to gn
On Thu, Oct 29, 2020 at 12:49 PM Renato Golin via llvm-dev < llvm-dev at lists.llvm.org> wrote:> On Thu, 29 Oct 2020 at 19:16, David Blaikie <dblaikie at gmail.com> wrote: > >> I /believe/ the idea is that, like gn, there are folks maintaining these >> build systems out of tree anyway - and having them in tree makes it easier >> to coordinate that effort, with the express intent of not burdening the >> general community with their upkeep (like gn currently - the idea is that >> there's no burden on developers to update gn build files (& consequently >> bazel build files)). >> > > Perhaps the initial assumption about my concerns weren't well articulated. > > I get that those files would be "additional" and other developers won't > need to care much about them. > > But what happens when people join the project with experience in Bazel > and, instead of building pure LLVM with CMake, they start using Bazel for > everything, just because they're used to it? >Didn't the community already go through this exact discussion when gn was added? Let me ask a different question. If gn support was permitted, on what grounds should we refuse a different parallel build system? Either we should allow people to contribute build systems upstream that they wish to maintain, or we should keep every buidl system other than CMake out of the tree. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/672d2ab6/attachment.html>
Krzysztof Parzyszek via llvm-dev
2020-Oct-29 23:56 UTC
[llvm-dev] Contributing Bazel BUILD files similar to gn
On the grounds that it was a bad idea after all. Any commits going into the LLVM repository should not break any part of it, at least not without a consideration for a fix. There is an exception to it---experimental targets. They can be broken, but they are there with the explicit intent of becoming officially supported. Same thing applies to the cmake files. If they get broken, they need to be fixed, but the same doesn’t apply to the extraneous build systems. They can be broken and never fixed. There is no commitment from the community as a whole to keep them working. IMO, this isn’t right, and files like that should not be a part of the official repository. Whether GN or Bazel have superior features is irrelevant. Unless their configuration files are a part of a longer-term transition process, they don’t belong in the repo. -- Krzysztof Parzyszek kparzysz at quicinc.com<mailto:kparzysz at quicinc.com> AI tools development From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Zachary Turner via llvm-dev Sent: Thursday, October 29, 2020 6:11 PM To: Renato Golin <rengolin at gmail.com> Cc: Mehdi Amini <aminim at google.com>; LLVM Dev <llvm-dev at lists.llvm.org>; Stella Laurenzo <laurenzo at google.com>; Tres Popp <tpopp at google.com>; Geoffrey Martin-Noble <gcmn at google.com>; Thomas Joerg <tjoerg at google.com> Subject: [EXT] Re: [llvm-dev] Contributing Bazel BUILD files similar to gn On Thu, Oct 29, 2020 at 12:49 PM Renato Golin via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: On Thu, 29 Oct 2020 at 19:16, David Blaikie <dblaikie at gmail.com<mailto:dblaikie at gmail.com>> wrote: I /believe/ the idea is that, like gn, there are folks maintaining these build systems out of tree anyway - and having them in tree makes it easier to coordinate that effort, with the express intent of not burdening the general community with their upkeep (like gn currently - the idea is that there's no burden on developers to update gn build files (& consequently bazel build files)). Perhaps the initial assumption about my concerns weren't well articulated. I get that those files would be "additional" and other developers won't need to care much about them. But what happens when people join the project with experience in Bazel and, instead of building pure LLVM with CMake, they start using Bazel for everything, just because they're used to it? Didn't the community already go through this exact discussion when gn was added? Let me ask a different question. If gn support was permitted, on what grounds should we refuse a different parallel build system? Either we should allow people to contribute build systems upstream that they wish to maintain, or we should keep every buidl system other than CMake out of the tree. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/f1080e20/attachment.html>
Geoffrey Martin-Noble via llvm-dev
2020-Oct-29 23:58 UTC
[llvm-dev] Contributing Bazel BUILD files similar to gn
This seems to have fragmented into a few separate threads, so apologies if I'm missing someone's response. I don't think I'm going to be able to effectively reply inline. The intention here is not to propose Bazel be another community-maintained build system or that it replace CMake. In my initial message I deliberately didn't focus on the specific advantages to Bazel because I'm not really trying to convince anyone to use it. Personally I prefer it to CMake, but it's got some annoying parts I don't like as well (my project uses both). The goal here is that if you don't care about Bazel, you should not be impacted. But some projects (and people) *do* use Bazel and depend on LLVM and right now that means copying around different versions of these files. We've been working on at least consolidating these in https://github.com/google/llvm-bazel. I think Mehdi already summarized the reasons we think it would make more sense for these to be in-tree than in a separate repo: it's a more natural collaboration point. Tom raised some specific concerns about the grounds for adding or removing a build system and when we know we've got bit rot, pointing out that it can be harder to tell with a build system if it's not one you actually use! In this case, we've got a build bot <https://buildkite.com/llvm-project/bazel-rbe> (lowercase, I'm using BuildKite) that builds against head every 15 minutes or so (there are currently some delays caused by pulling in new changes from the monorepo). Does a functioning bot with a clearly visible (though not noisy!) status indicator seem like a reasonable requirement? If this bot remains broken for a long time and/or no one has been updating the build files, then that would be an indication of bit rot. Someone would send a message to the list proposing the build files be deleted and doing so should be relatively easy. I think some of the other general concerns about people using Bazel instead of CMake and assuming support or breaking the CMake build should be solvable by Bazel being in a side-directory (as proposed) with a clear readme that explains the level of support. I'll draft such a readme to include with the patch (I was waiting to see some responses to the RFC before doing so). A side point regarding extra commit traffic. I proposed putting these in the top-level utils/ directory. Would it also make sense to move gn there to similarly remove it from commit mailing list traffic? On Thu, Oct 29, 2020 at 4:11 PM Zachary Turner <zturner at roblox.com> wrote:> > > On Thu, Oct 29, 2020 at 12:49 PM Renato Golin via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> On Thu, 29 Oct 2020 at 19:16, David Blaikie <dblaikie at gmail.com> wrote: >> >>> I /believe/ the idea is that, like gn, there are folks maintaining these >>> build systems out of tree anyway - and having them in tree makes it easier >>> to coordinate that effort, with the express intent of not burdening the >>> general community with their upkeep (like gn currently - the idea is that >>> there's no burden on developers to update gn build files (& consequently >>> bazel build files)). >>> >> >> Perhaps the initial assumption about my concerns weren't well >> articulated. >> >> I get that those files would be "additional" and other developers won't >> need to care much about them. >> >> But what happens when people join the project with experience in Bazel >> and, instead of building pure LLVM with CMake, they start using Bazel for >> everything, just because they're used to it? >> > > Didn't the community already go through this exact discussion when gn was > added? Let me ask a different question. If gn support was permitted, on > what grounds should we refuse a different parallel build system? Either we > should allow people to contribute build systems upstream that they wish to > maintain, or we should keep every buidl system other than CMake out of the > tree. >I believe that Tom actually highlighted this particular slippery slope as a concern. Hopefully having some reasonable maintenance standards as described above can help us avoid it. But I do think we can use gn as an experiment into how that went. It's been a couple years. Has anyone experienced issues with their presence in the monorepo? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/a8ce0a9f/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3992 bytes Desc: S/MIME Cryptographic Signature URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201029/a8ce0a9f/attachment.bin>