On Jan 13, 2017, at 1:37 PM, Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org> wrote:> The main outcome of the BoF had the dev meeting was that we agree’d that moving to GitHub was the best choice forward for LLVM (IIRC only one person in the room expressed concerned about GitHub, but he said he had personal grief with them and nothing specific for LLVM). > > The unknown that remains is: will we use a mono-repo or a multi-repo. On this aspect: > > - We got consensus at the BoF that downstream users (i.e. non-contributors) are not impacted by this choice, and we’re not gonna optimize the repository structure for them. > - My reading of the survey is that the monorepo has a significant lead. > - My understanding of the dynamic of the discussions and question during the BoF is that monorepo has a significant lead, is likely to satisfy more people, and has a very small number of people concerned about it. On the other hand many people have strong feeling about the multirepo.FWIW, I have spoken to a large number of people about the mono vs multi-repo tradeoffs, and I’m personally convinced that mono repo is the way to go. For a few reasons: - Monorepo is the “natural” way to use git. Submodules are possible to use, but add significant complexity. - The download size of a mono-repo is manageable, and seems scalable for a project the size of LLVM (including reasonable growth over the next 10 years). - As Medhi says, according to surveys and discussions in forums like the LLVM Dev Meeting BoF, most people who care are in favor of mono-repo. - The people most impacted by mono-repo are those who want to build just compiler-rt. We want these people to be happy, but they are very few in number, and their benefit needs to be balanced against the benefit for the larger community that builds llvm (and typically clang or another front end). Overall, it seems clear that either approach could work, but mono seems to win out because it is more popular and more simple. It would require tweaks to LLVM’s cmake system though: instead of deciding to build a subproject based on whether it is checked out, it should instead be based on configuration time flags. -Chris> > Considering all the current tradeoffs, it is likely that we will move-on with a monorepo, even if there are no guarantee or decision made at this point. > > The path forward (already engaged) is to engage a prototype phase: we’re building a monorepo and trying to make it usable as much as possible, without making any change or building anything that would commit us to a monorepo (for instance we’re not gonna migrate any bots to it). > > The goal of this prototype is that developers can start using a monorepo to try it, and we can evaluate how it plays in practice, outside of theoretical considerations. If anyone finds concerns about a given workflow, we can study what can be improved to address it, or maybe we’ll hit a wall that would show that monorepo can’t address what we think it will. > > At some point, if the experiment is conclusive, we should be able to build a larger majority and hopefully reach a consensus that the proposed prototype can be considered viable for development and start planning the actual committing changes. > > The monorepo is not totally ready yet, but you can already experience it (I live on day-to-day for my development, and a few other people as well), instructions are in the doc: http://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo <http://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo> > > I don’t have any schedule to announce, hopefully we can make it all happen in 2017. > > Best, > > — > Mehdi > > >> On Jan 13, 2017, at 1:07 PM, Keane, Erich via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> Hi all- >> I was wondering if anyone knew what the status/schedule of the SVN to git/github transition was? I thought I saw that at the November meeting it was agreed upon, but I'm not sure I saw any progress since? >> >> Thanks, >> Erich >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170116/d02c1134/attachment.html>
"The people most impacted by mono-repo are those who want to build just compiler-rt" Perhaps more accurate to say "Those who want to contribute only to compiler-rt"? Those who only want to build compiler-rt can check it out from a slave repo that mirrors commits to compiler-rt in the mono-repo. Some infrastructure is needed for this, but it's easy and automated. On Tue, Jan 17, 2017 at 4:17 AM, Chris Lattner via llvm-dev < llvm-dev at lists.llvm.org> wrote:> On Jan 13, 2017, at 1:37 PM, Mehdi Amini via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > The main outcome of the BoF had the dev meeting was that we agree’d that > moving to GitHub was the best choice forward for LLVM (IIRC only one person > in the room expressed concerned about GitHub, but he said he had personal > grief with them and nothing specific for LLVM). > > The unknown that remains is: will we use a mono-repo or a multi-repo. On > this aspect: > > - We got consensus at the BoF that downstream users (i.e. > non-contributors) are not impacted by this choice, and we’re not gonna > optimize the repository structure for them. > - My reading of the survey is that the monorepo has a significant lead. > - My understanding of the dynamic of the discussions and question during > the BoF is that monorepo has a significant lead, is likely to satisfy more > people, and has a very small number of people concerned about it. On the > other hand many people have strong feeling about the multirepo. > > > FWIW, I have spoken to a large number of people about the mono vs > multi-repo tradeoffs, and I’m personally convinced that mono repo is the > way to go. For a few reasons: > > - Monorepo is the “natural” way to use git. Submodules are possible to > use, but add significant complexity. > - The download size of a mono-repo is manageable, and seems scalable for a > project the size of LLVM (including reasonable growth over the next 10 > years). > - As Medhi says, according to surveys and discussions in forums like the > LLVM Dev Meeting BoF, most people who care are in favor of mono-repo. > - The people most impacted by mono-repo are those who want to build just > compiler-rt. We want these people to be happy, but they are very few in > number, and their benefit needs to be balanced against the benefit for the > larger community that builds llvm (and typically clang or another front > end). > > Overall, it seems clear that either approach could work, but mono seems to > win out because it is more popular and more simple. It would require tweaks > to LLVM’s cmake system though: instead of deciding to build a subproject > based on whether it is checked out, it should instead be based on > configuration time flags. > > -Chris > > > > Considering all the current tradeoffs, it is likely that we will move-on > with a monorepo, even if there are no guarantee or decision made at this > point. > > The path forward (already engaged) is to engage a prototype phase: we’re > building a monorepo and trying to make it usable as much as possible, > without making any change or building anything that would commit us to a > monorepo (for instance we’re not gonna migrate any bots to it). > > The goal of this prototype is that developers can start using a monorepo > to try it, and we can evaluate how it plays in practice, outside of > theoretical considerations. If anyone finds concerns about a given > workflow, we can study what can be improved to address it, or maybe we’ll > hit a wall that would show that monorepo can’t address what we think it > will. > > At some point, if the experiment is conclusive, we should be able to build > a larger majority and hopefully reach a consensus that the proposed > prototype can be considered viable for development and start planning the > actual committing changes. > > The monorepo is not totally ready yet, but you can already experience it > (I live on day-to-day for my development, and a few other people as well), > instructions are in the doc: http://llvm.org/docs/GettingStarted.html#for- > developers-to-work-with-a-git-monorepo > > I don’t have any schedule to announce, hopefully we can make it all happen > in 2017. > > Best, > > — > Mehdi > > > On Jan 13, 2017, at 1:07 PM, Keane, Erich via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > Hi all- > I was wondering if anyone knew what the status/schedule of the SVN to > git/github transition was? I thought I saw that at the November meeting it > was agreed upon, but I'm not sure I saw any progress since? > > Thanks, > Erich > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170117/92f13367/attachment.html>
> > Some infrastructure is needed for this, but it's easy and automatedWell sounds like Submodules to me. Hasn't there been a discussion about it in one of the first Gitposts? 2017-01-17 12:31 GMT+01:00 Bruce Hoult via llvm-dev <llvm-dev at lists.llvm.org>:> "The people most impacted by mono-repo are those who want to build just > compiler-rt" > > Perhaps more accurate to say "Those who want to contribute only to > compiler-rt"? > > Those who only want to build compiler-rt can check it out from a slave > repo that mirrors commits to compiler-rt in the mono-repo. Some > infrastructure is needed for this, but it's easy and automated. > > On Tue, Jan 17, 2017 at 4:17 AM, Chris Lattner via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> On Jan 13, 2017, at 1:37 PM, Mehdi Amini via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >> The main outcome of the BoF had the dev meeting was that we agree’d that >> moving to GitHub was the best choice forward for LLVM (IIRC only one person >> in the room expressed concerned about GitHub, but he said he had personal >> grief with them and nothing specific for LLVM). >> >> The unknown that remains is: will we use a mono-repo or a multi-repo. On >> this aspect: >> >> - We got consensus at the BoF that downstream users (i.e. >> non-contributors) are not impacted by this choice, and we’re not gonna >> optimize the repository structure for them. >> - My reading of the survey is that the monorepo has a significant lead. >> - My understanding of the dynamic of the discussions and question during >> the BoF is that monorepo has a significant lead, is likely to satisfy more >> people, and has a very small number of people concerned about it. On the >> other hand many people have strong feeling about the multirepo. >> >> >> FWIW, I have spoken to a large number of people about the mono vs >> multi-repo tradeoffs, and I’m personally convinced that mono repo is the >> way to go. For a few reasons: >> >> - Monorepo is the “natural” way to use git. Submodules are possible to >> use, but add significant complexity. >> - The download size of a mono-repo is manageable, and seems scalable for >> a project the size of LLVM (including reasonable growth over the next 10 >> years). >> - As Medhi says, according to surveys and discussions in forums like the >> LLVM Dev Meeting BoF, most people who care are in favor of mono-repo. >> - The people most impacted by mono-repo are those who want to build just >> compiler-rt. We want these people to be happy, but they are very few in >> number, and their benefit needs to be balanced against the benefit for the >> larger community that builds llvm (and typically clang or another front >> end). >> >> Overall, it seems clear that either approach could work, but mono seems >> to win out because it is more popular and more simple. It would require >> tweaks to LLVM’s cmake system though: instead of deciding to build a >> subproject based on whether it is checked out, it should instead be based >> on configuration time flags. >> >> -Chris >> >> >> >> Considering all the current tradeoffs, it is likely that we will move-on >> with a monorepo, even if there are no guarantee or decision made at this >> point. >> >> The path forward (already engaged) is to engage a prototype phase: we’re >> building a monorepo and trying to make it usable as much as possible, >> without making any change or building anything that would commit us to a >> monorepo (for instance we’re not gonna migrate any bots to it). >> >> The goal of this prototype is that developers can start using a monorepo >> to try it, and we can evaluate how it plays in practice, outside of >> theoretical considerations. If anyone finds concerns about a given >> workflow, we can study what can be improved to address it, or maybe we’ll >> hit a wall that would show that monorepo can’t address what we think it >> will. >> >> At some point, if the experiment is conclusive, we should be able to >> build a larger majority and hopefully reach a consensus that the proposed >> prototype can be considered viable for development and start planning the >> actual committing changes. >> >> The monorepo is not totally ready yet, but you can already experience it >> (I live on day-to-day for my development, and a few other people as well), >> instructions are in the doc: http://llvm.org/docs/Gett >> ingStarted.html#for-developers-to-work-with-a-git-monorepo >> >> I don’t have any schedule to announce, hopefully we can make it all >> happen in 2017. >> >> Best, >> >> — >> Mehdi >> >> >> On Jan 13, 2017, at 1:07 PM, Keane, Erich via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >> Hi all- >> I was wondering if anyone knew what the status/schedule of the SVN to >> git/github transition was? I thought I saw that at the November meeting it >> was agreed upon, but I'm not sure I saw any progress since? >> >> Thanks, >> Erich >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170117/43ad06aa/attachment-0001.html>
On 17 Jan 2017, at 01:17, Chris Lattner via llvm-dev <llvm-dev at lists.llvm.org> wrote:> > - Monorepo is the “natural” way to use git. Submodules are possible to use, but add significant complexity.Having used submodules in a couple of projects, I’ve not found them to cause more difficulty than they avoided; however, they do have an issue specifically with GitHub, which is that tarballs don’t include submodules so packages are slightly harder to construct (they must point to two releases).> - The download size of a mono-repo is manageable, and seems scalable for a project the size of LLVM (including reasonable growth over the next 10 years).The download size of a mono-repo is fine for anyone who would be checking out LLVM today. compiler-rt and libc++ are both useful without any of the rest of LLVM and contributors to libc++ rarely check out anything more than libc++ (perhaps libc++abi) today.> - As Medhi says, according to surveys and discussions in forums like the LLVM Dev Meeting BoF, most people who care are in favor of mono-repo.From the online surveys, I think the split was roughly 50:50. I’d be very hesitant to regard anything at a BoF as representative of the wider community, as the set of people who have the time and funding to attend a conference is quite distinct from the wider community (particularly for the US DevMeeting, which is right in the middle of university term times). We’ve made this mistake in FreeBSD before.> - The people most impacted by mono-repo are those who want to build just compiler-rt. We want these people to be happy, but they are very few in number, and their benefit needs to be balanced against the benefit for the larger community that builds llvm (and typically clang or another front end).I believe that the big win for the monorepo is the ability to bisect usefully. It’s currently very difficult to bisect clang, because you can’t bisect clang and llvm independently (LLVM API changes frequently break clang) and they’re in different git repos (or non-enclosing svn subtrees) and so it needs some manual intervention. Having them in the same repo would ensure that they are in sync and make bisecting trivial. In contrast, there is not (and should not be) tight coupling between LLVM and libc++, libunwind, libc++abi, and compiler-rt. There *may* be ordering requirements (e.g. revision X of libc++ requires c++17 features of revision Y of clang for c++17 features to work), but it is incredibly valuable to bisect these independently to find whether a particular change is a new compiler bug, a new library bug, or an old library bug that is triggered by new compiler behaviour (or an old compiler bug that is triggered by new code). I would be in favour of a monorepo for everything that links against LLVM libraries and everything else being in separate repos.> Overall, it seems clear that either approach could work, but mono seems to win out because it is more popular and more simple. It would require tweaks to LLVM’s cmake system though: instead of deciding to build a subproject based on whether it is checked out, it should instead be based on configuration time flags.I believe that most of this works already - you can opt out of building components that are checked out. David
> On Jan 17, 2017, at 7:24 AM, David Chisnall <David.Chisnall at cl.cam.ac.uk> wrote: > > On 17 Jan 2017, at 01:17, Chris Lattner via llvm-dev <llvm-dev at lists.llvm.org> wrote: >> >> - Monorepo is the “natural” way to use git. Submodules are possible to use, but add significant complexity. > > Having used submodules in a couple of projects, I’ve not found them to cause more difficulty than they avoided; however, they do have an issue specifically with GitHub, which is that tarballs don’t include submodules so packages are slightly harder to construct (they must point to two releases).Another one is the future possibility of “pull request”, which is annoying to get across repository.> >> - The download size of a mono-repo is manageable, and seems scalable for a project the size of LLVM (including reasonable growth over the next 10 years). > > The download size of a mono-repo is fine for anyone who would be checking out LLVM today. compiler-rt and libc++ are both useful without any of the rest of LLVM and contributors to libc++ rarely check out anything more than libc++ (perhaps libc++abi) today. > >> - As Medhi says, according to surveys and discussions in forums like the LLVM Dev Meeting BoF, most people who care are in favor of mono-repo. > > From the online surveys, I think the split was roughly 50:50.I don’t know on what data you’re basis this on. I looked very closely and here are two questions that contradicts your view. Question: “If we could go back in time and restart the project with today's technologies, which repository scheme would be best for the LLVM project?” -> 55 to 36 in favor of the mono-repo Question: "Assuming mono-repo gets adopted, how do you plan to contribute?” -> Only 11% were saying before the BoF that they will continue to use split repository (through git-svn, as today), 86% will use the mono-repo.> I’d be very hesitant to regard anything at a BoF as representative of the wider community, as the set of people who have the time and funding to attend a conference is quite distinct from the wider community (particularly for the US DevMeeting, which is right in the middle of university term times). We’ve made this mistake in FreeBSD before.I believe it representative enough of the people contributing to LLVM that have an opinion on the question. It obviously can’t be *all* of them in the same room, but with over 400 attendees, the conference seems like a valid signal to me. Especially many of the people that have been very active on this issue were in the room.>> - The people most impacted by mono-repo are those who want to build just compiler-rt. We want these people to be happy, but they are very few in number, and their benefit needs to be balanced against the benefit for the larger community that builds llvm (and typically clang or another front end). > > I believe that the big win for the monorepo is the ability to bisect usefully. It’s currently very difficult to bisect clang, because you can’t bisect clang and llvm independently (LLVM API changes frequently break clang) and they’re in different git repos (or non-enclosing svn subtrees) and so it needs some manual intervention. Having them in the same repo would ensure that they are in sync and make bisecting trivial. > > In contrast, there is not (and should not be) tight coupling between LLVM and libc++, libunwind, libc++abi, and compiler-rt. There *may* be ordering requirements (e.g. revision X of libc++ requires c++17 features of revision Y of clang for c++17 features to work), but it is incredibly valuable to bisect these independently to find whether a particular change is a new compiler bug, a new library bug, or an old library bug that is triggered by new compiler behaviour (or an old compiler bug that is triggered by new code). > > I would be in favour of a monorepo for everything that links against LLVM libraries and everything else being in separate repos. > >> Overall, it seems clear that either approach could work, but mono seems to win out because it is more popular and more simple. It would require tweaks to LLVM’s cmake system though: instead of deciding to build a subproject based on whether it is checked out, it should instead be based on configuration time flags. > > I believe that most of this works already - you can opt out of building components that are checked out.Actually you have to opt-in instead of opt-out right now, and I encourage you to try it if you’re contributing to LLVM: http://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo <http://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo> — Mehdi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170117/fcf9fac0/attachment.html>
On 17 Jan 2017, at 01:17, Chris Lattner via llvm-dev <llvm-dev at lists.llvm.org> wrote:> - The download size of a mono-repo is manageable, and seems scalable for a project the size of LLVM (including reasonable growth over the next 10 years).Did you consider that GitHub has a 1GB size limit/recommendation? https://help.github.com/articles/what-is-my-disk-quota/ The full mono-repo is already approaching this limit: llvm-project 1.5G (.git 871M) (GitHub API 830,702K) On Tue, Jan 17, 2017 at 4:24 PM, David Chisnall via llvm-dev <llvm-dev at lists.llvm.org> wrote:> > I would be in favour of a monorepo for everything that links against LLVM libraries and everything else being in separate repos.I agree fully. I work on a project that use submodules for external dependencies. For llvm this means compiling and linking statically with dependencies for Linux and OSX platforms. So these: (compiler-rt, libcxx, libcxxabi, libunwind) 141M (.git 60M) I would hesitate to add a dependency that use 1.5G disk space, so if you go with the mono-repo approach I would probably use the supposed read only mirror repos for these modules. This in turn would mean that the hurdle for contributing back to these projects would increase and thus the the likelihood of contributions. Consolidating other projects with interdependencies into other mono-repos would also make sense. E.g. libcxx with dependencies. The most important thing is to make a decision. I have some small contributions just waiting for a sane contribution process (GitHub pull requests) ;) Erik