Hi all- I was wondering if anyone knew what the status/schedule of the SVN to git/github transition was? I thought I saw that at the November meeting it was agreed upon, but I'm not sure I saw any progress since? Thanks, Erich
Hi, The main outcome of the BoF had the dev meeting was that we agree’d that moving to GitHub was the best choice forward for LLVM (IIRC only one person in the room expressed concerned about GitHub, but he said he had personal grief with them and nothing specific for LLVM). The unknown that remains is: will we use a mono-repo or a multi-repo. On this aspect: - We got consensus at the BoF that downstream users (i.e. non-contributors) are not impacted by this choice, and we’re not gonna optimize the repository structure for them. - My reading of the survey is that the monorepo has a significant lead. - My understanding of the dynamic of the discussions and question during the BoF is that monorepo has a significant lead, is likely to satisfy more people, and has a very small number of people concerned about it. On the other hand many people have strong feeling about the multirepo. Considering all the current tradeoffs, it is likely that we will move-on with a monorepo, even if there are no guarantee or decision made at this point. The path forward (already engaged) is to engage a prototype phase: we’re building a monorepo and trying to make it usable as much as possible, without making any change or building anything that would commit us to a monorepo (for instance we’re not gonna migrate any bots to it). The goal of this prototype is that developers can start using a monorepo to try it, and we can evaluate how it plays in practice, outside of theoretical considerations. If anyone finds concerns about a given workflow, we can study what can be improved to address it, or maybe we’ll hit a wall that would show that monorepo can’t address what we think it will. At some point, if the experiment is conclusive, we should be able to build a larger majority and hopefully reach a consensus that the proposed prototype can be considered viable for development and start planning the actual committing changes. The monorepo is not totally ready yet, but you can already experience it (I live on day-to-day for my development, and a few other people as well), instructions are in the doc: http://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo <http://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo> I don’t have any schedule to announce, hopefully we can make it all happen in 2017. Best, — Mehdi> On Jan 13, 2017, at 1:07 PM, Keane, Erich via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Hi all- > I was wondering if anyone knew what the status/schedule of the SVN to git/github transition was? I thought I saw that at the November meeting it was agreed upon, but I'm not sure I saw any progress since? > > Thanks, > Erich > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170113/3ba66e6c/attachment.html>
I see, thank you very much for the update! I’m glad it is moving forward. -Erich From: mehdi.amini at apple.com [mailto:mehdi.amini at apple.com] Sent: Friday, January 13, 2017 1:38 PM To: Keane, Erich <erich.keane at intel.com> Cc: llvm-dev at lists.llvm.org Subject: Re: [llvm-dev] Git Transition status? Hi, The main outcome of the BoF had the dev meeting was that we agree’d that moving to GitHub was the best choice forward for LLVM (IIRC only one person in the room expressed concerned about GitHub, but he said he had personal grief with them and nothing specific for LLVM). The unknown that remains is: will we use a mono-repo or a multi-repo. On this aspect: - We got consensus at the BoF that downstream users (i.e. non-contributors) are not impacted by this choice, and we’re not gonna optimize the repository structure for them. - My reading of the survey is that the monorepo has a significant lead. - My understanding of the dynamic of the discussions and question during the BoF is that monorepo has a significant lead, is likely to satisfy more people, and has a very small number of people concerned about it. On the other hand many people have strong feeling about the multirepo. Considering all the current tradeoffs, it is likely that we will move-on with a monorepo, even if there are no guarantee or decision made at this point. The path forward (already engaged) is to engage a prototype phase: we’re building a monorepo and trying to make it usable as much as possible, without making any change or building anything that would commit us to a monorepo (for instance we’re not gonna migrate any bots to it). The goal of this prototype is that developers can start using a monorepo to try it, and we can evaluate how it plays in practice, outside of theoretical considerations. If anyone finds concerns about a given workflow, we can study what can be improved to address it, or maybe we’ll hit a wall that would show that monorepo can’t address what we think it will. At some point, if the experiment is conclusive, we should be able to build a larger majority and hopefully reach a consensus that the proposed prototype can be considered viable for development and start planning the actual committing changes. The monorepo is not totally ready yet, but you can already experience it (I live on day-to-day for my development, and a few other people as well), instructions are in the doc: http://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo I don’t have any schedule to announce, hopefully we can make it all happen in 2017. Best, — Mehdi On Jan 13, 2017, at 1:07 PM, Keane, Erich via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: Hi all- I was wondering if anyone knew what the status/schedule of the SVN to git/github transition was? I thought I saw that at the November meeting it was agreed upon, but I'm not sure I saw any progress since? Thanks, Erich _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170113/fb2f7942/attachment-0001.html>
I've been working on some scripting for re-converting the svn repository into git. The existing git conversions are entirely sufficient for day-to-day development purposes, but not up to the standard (at least, *my* standard) for replacing svn as the authoritative source repository. For one example, clang.git doesn't actually go back to the first commit of clang, because weird stuff happened early in the svn history that threw off git-svn. Here's my work in progress, but there's still more work to be done: https://github.com/jyknight/llvm-git-migration <https://github.com/jyknight/llvm-monorepo> (conversion scripts) https://github.com/jyknight/llvm-monorepo (test repository) On Fri, Jan 13, 2017 at 4:07 PM, Keane, Erich via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Hi all- > I was wondering if anyone knew what the status/schedule of the SVN to > git/github transition was? I thought I saw that at the November meeting it > was agreed upon, but I'm not sure I saw any progress since? > > Thanks, > Erich > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170113/ca7ce70e/attachment.html>
So the Github-Importer screws up the old dark places of the SVN-History and the scripts try to handle this? 2017-01-13 22:57 GMT+01:00 James Y Knight via llvm-dev < llvm-dev at lists.llvm.org>:> I've been working on some scripting for re-converting the svn repository > into git. > > The existing git conversions are entirely sufficient for day-to-day > development purposes, but not up to the standard (at least, *my* standard) > for replacing svn as the authoritative source repository. For one example, > clang.git doesn't actually go back to the first commit of clang, because > weird stuff happened early in the svn history that threw off git-svn. > > Here's my work in progress, but there's still more work to be done: > https://github.com/jyknight/llvm-git-migration > <https://github.com/jyknight/llvm-monorepo> (conversion scripts) > https://github.com/jyknight/llvm-monorepo (test repository) > > > > On Fri, Jan 13, 2017 at 4:07 PM, Keane, Erich via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> Hi all- >> I was wondering if anyone knew what the status/schedule of the SVN to >> git/github transition was? I thought I saw that at the November meeting it >> was agreed upon, but I'm not sure I saw any progress since? >> >> Thanks, >> Erich >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170116/067a9ab3/attachment-0001.html>
On 13 January 2017 at 21:37, Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org> wrote:> - My reading of the survey is that the monorepo has a significant lead.That was not my understanding, but we shouldn't be arguing over small differences in percentages. We had enough of that on both sides of the Atlantic already in 2016.> - My understanding of the dynamic of the discussions and question during the > BoF is that monorepo has a significant lead, is likely to satisfy more > people, and has a very small number of people concerned about it. On the > other hand many people have strong feeling about the multirepo.My understanding is that people were happy with the mono-repo as long as it had the right balance between what's in the mono-repo and what's left out. That break down hasn't reached a consensus yet.> Considering all the current tradeoffs, it is likely that we will move-on > with a monorepo, even if there are no guarantee or decision made at this > point.I wouldn't start betting on the likelihood of anything at this stage. Right now we need to understand what's the most logical and the least impacting split for a mono-repo.>From what I gathered at the BoF, most people were happy with the core(revision sync) repos in the mono and everything else out. What that means is that they will be physically separated, which is a very different scenario than what we have today and those issues will have to be sorted out before any decision. Also, not everyone agreed on the definition of "core repo".> The path forward (already engaged) is to engage a prototype phase: we’re > building a monorepo and trying to make it usable as much as possible, > without making any change or building anything that would commit us to a > monorepo (for instance we’re not gonna migrate any bots to it).Is this anywhere we can use? Did I miss the announcement of such a project?> I don’t have any schedule to announce, hopefully we can make it all happen > in 2017.There will be no schedules to announce without consensus and collaboration. I haven't seen anything public since the BoF, and I'm a little surprised that things are happening and not being shared in the mailing list. This was a task effort that multiple people have put together around the mailing list. I don't think this should take any offline form on the most crucial moment, which is deciding how the repo will look like and how are we all going to interact with it. Downstream users said "they will all eventually pay the costs" with whatever decision upstream takes. They didn't say it was going to be cheap (most of them said it was not), nor did they say that they'll abide to whatever a closed group decision had formed. This is an upstream process and needs to happen upstream, which means the mailing list. Not socials, not IRC. This needs record, and the mailing list is the only channel that has that feature and reaches all developers. cheers, --renato
On Jan 13, 2017, at 1:37 PM, Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org> wrote:> The main outcome of the BoF had the dev meeting was that we agree’d that moving to GitHub was the best choice forward for LLVM (IIRC only one person in the room expressed concerned about GitHub, but he said he had personal grief with them and nothing specific for LLVM). > > The unknown that remains is: will we use a mono-repo or a multi-repo. On this aspect: > > - We got consensus at the BoF that downstream users (i.e. non-contributors) are not impacted by this choice, and we’re not gonna optimize the repository structure for them. > - My reading of the survey is that the monorepo has a significant lead. > - My understanding of the dynamic of the discussions and question during the BoF is that monorepo has a significant lead, is likely to satisfy more people, and has a very small number of people concerned about it. On the other hand many people have strong feeling about the multirepo.FWIW, I have spoken to a large number of people about the mono vs multi-repo tradeoffs, and I’m personally convinced that mono repo is the way to go. For a few reasons: - Monorepo is the “natural” way to use git. Submodules are possible to use, but add significant complexity. - The download size of a mono-repo is manageable, and seems scalable for a project the size of LLVM (including reasonable growth over the next 10 years). - As Medhi says, according to surveys and discussions in forums like the LLVM Dev Meeting BoF, most people who care are in favor of mono-repo. - The people most impacted by mono-repo are those who want to build just compiler-rt. We want these people to be happy, but they are very few in number, and their benefit needs to be balanced against the benefit for the larger community that builds llvm (and typically clang or another front end). Overall, it seems clear that either approach could work, but mono seems to win out because it is more popular and more simple. It would require tweaks to LLVM’s cmake system though: instead of deciding to build a subproject based on whether it is checked out, it should instead be based on configuration time flags. -Chris> > Considering all the current tradeoffs, it is likely that we will move-on with a monorepo, even if there are no guarantee or decision made at this point. > > The path forward (already engaged) is to engage a prototype phase: we’re building a monorepo and trying to make it usable as much as possible, without making any change or building anything that would commit us to a monorepo (for instance we’re not gonna migrate any bots to it). > > The goal of this prototype is that developers can start using a monorepo to try it, and we can evaluate how it plays in practice, outside of theoretical considerations. If anyone finds concerns about a given workflow, we can study what can be improved to address it, or maybe we’ll hit a wall that would show that monorepo can’t address what we think it will. > > At some point, if the experiment is conclusive, we should be able to build a larger majority and hopefully reach a consensus that the proposed prototype can be considered viable for development and start planning the actual committing changes. > > The monorepo is not totally ready yet, but you can already experience it (I live on day-to-day for my development, and a few other people as well), instructions are in the doc: http://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo <http://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo> > > I don’t have any schedule to announce, hopefully we can make it all happen in 2017. > > Best, > > — > Mehdi > > >> On Jan 13, 2017, at 1:07 PM, Keane, Erich via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> Hi all- >> I was wondering if anyone knew what the status/schedule of the SVN to git/github transition was? I thought I saw that at the November meeting it was agreed upon, but I'm not sure I saw any progress since? >> >> Thanks, >> Erich >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170116/d02c1134/attachment.html>
Sorry I'm dredging up an old discussion. FWIW, for converting our (extremely long and complex history) compiler to git, I found svn-all-fast-export to be very useful, even essential. Its rule structure was flexible enough to handle a lot of corner cases and weird history. I also developed several thousands of lines of conversion scripts based around svn-all-fast-export and git_fast_filter (as well as some fixes and enhancements to the latter). Most of that involved taking completely independent SVN repositories and weaving them into one git repository. I don't think that's as complex for the LLVM ecosystem since everything lives under one SVN repository. Some of the directory remapping issues that have been raised would be handled beautifully by svn-all-fast-export. -David James Y Knight via llvm-dev <llvm-dev at lists.llvm.org> writes:> I've been working on some scripting for re-converting the svn repository into git. > > The existing git conversions are entirely sufficient for day-to-day development purposes, but not up to the standard (at least, *my* standard) for replacing svn as the authoritative source > repository. For one example, clang.git doesn't actually go back to the first commit of clang, because weird stuff happened early in the svn history that threw off git-svn. > > Here's my work in progress, but there's still more work to be done: > https://github.com/jyknight/llvm-git-migration (conversion scripts) > https://github.com/jyknight/llvm-monorepo (test repository) > > On Fri, Jan 13, 2017 at 4:07 PM, Keane, Erich via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Hi all- > I was wondering if anyone knew what the status/schedule of the SVN to git/github transition was? I thought I saw that at the November meeting it was agreed upon, but I'm not sure I saw > any progress since? > > Thanks, > Erich > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev