Folks, There has been some discussion on IRC about SVN hosting and the perils of doing it ourselves. The consensus on the current discussion was that moving to a Git-only solution would have some disvantages, but many advantages. Furthermore, not hosting our own repos would save us a lot of headaches, admin costs and timed out connections. TL;DR: GitHub + git submodules [1] could replace all the functionality we have currently with SVN. (also GitLab, BitBucketc, etc). Here are some of the arguments made on IRC... 1. Due to SVN, we can't re-write history. If we use some GitHub properties [2], we could have the same effect. 2. Due to SVN, we have a mandatory time sequence, so commits go first in LLVM, then Clang (for example), and buildbots don't get lost. If we use submodules [1], we can have a similar relationship, but in a more explicit way, and the problem could be solved elegantly. 3. Some people still can only use SVN. For that, GitHub has an SVN interface [3] to the repositories. 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, etc. Not only this incurs in additional admin cost, but it also gets outdated, locally modified, and it needs to be backed up, etc. GitHub gives all that for us for free. 5. We can still use Bugzilla (and lock GitHub's own bug system), but we can also use GitHub's system to manage releases (it's actually quite good for that). 6. GitHub has automated testing of merge requests, meaning we can have pre-commit tests enabled on a set of fast bots, triggered by GitHub's own validation hooks. Even though that wouldn't cover everything, having a few pre-commit bots would considerably reduce the need to revert patches [citation needed]. 7. With git submodules, we'd probably want to follow the same style we have today (llvm-projects/<prj>) instead of modelling how they look in tree (llvm/tools/clang still as a symlink). 8. Once we're solo Git, we can shop around *much* more easily. By using SVN, we're basically forced to host, or choose Source Forge. Using just Git, we can choose GitLab, BitBucket and many others, if GitHub is not appealing enough. Essentially, it doesn't matter where you are, the tools are good, there and largely replaceable [citation needed]. What do people think? Any issue not covered that we should? How would that disrupt downstream users? Would it be a temporary disruption, but with long lasting benefits? Or will it just break everything for you? cheers, --renato [1] https://git-scm.com/book/en/v2/Git-Tools-Submodules [2] https://help.github.com/articles/defining-the-mergeability-of-pull-requests/ [3] https://help.github.com/articles/support-for-subversion-clients/
peanut gallery comments from a git hater (We use git for all source revision control) On Wed, Jun 1, 2016 at 3:31 AM, Renato Golin via cfe-dev <cfe-dev at lists.llvm.org> wrote:> Folks, > > There has been some discussion on IRC about SVN hosting and the perils > of doing it ourselves. The consensus on the current discussion was > that moving to a Git-only solution would have some disvantages, but > many advantages. Furthermore, not hosting our own repos would save us > a lot of headaches, admin costs and timed out connections. > > TL;DR: GitHub + git submodules [1] could replace all the functionality> we have currently with SVN. > > (also GitLab, BitBucketc, etc). > > Here are some of the arguments made on IRC... > > 1. Due to SVN, we can't re-write history. If we use some GitHub > properties [2], we could have the same effect.Are you referring to linear commit history being maintained going forward or what specifically?> > 2. Due to SVN, we have a mandatory time sequence, so commits go first > in LLVM, then Clang (for example), and buildbots don't get lost. If we > use submodules [1], we can have a similar relationship, but in a more > explicit way, and the problem could be solved elegantly.Sub modules don't work exactly like svn. afaik they are glued to a specific commit? So you get questions like this http://stackoverflow.com/questions/1030169/easy-way-pull-latest-of-all-submodules So if this path is chosen, please make sure the minimum git version supported is clarified.> > 3. Some people still can only use SVN. For that, GitHub has an SVN > interface [3] to the repositories. > > 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, > etc. Not only this incurs in additional admin cost, but it also gets > outdated, locally modified, and it needs to be backed up, etc. GitHub > gives all that for us for free. > > 5. We can still use Bugzilla (and lock GitHub's own bug system), but > we can also use GitHub's system to manage releases (it's actually > quite good for that).Bugzilla sucks, but github's issue tracker sucks worse (imnsho). I'm not a stakeholder and have no vote, but if it ain't broke don't fix it.. or try jira.. (they give free licenses to open source and it rocks)> > 6. GitHub has automated testing of merge requests, meaning we can have > pre-commit tests enabled on a set of fast bots, triggered by GitHub's > own validation hooks. Even though that wouldn't cover everything, > having a few pre-commit bots would considerably reduce the need to > revert patches [citation needed]. > > 7. With git submodules, we'd probably want to follow the same style we > have today (llvm-projects/<prj>) instead of modelling how they look in > tree (llvm/tools/clang still as a symlink). > > 8. Once we're solo Git, we can shop around *much* more easily. By > using SVN, we're basically forced to host, or choose Source Forge. > Using just Git, we can choose GitLab, BitBucket and many others, if > GitHub is not appealing enough. Essentially, it doesn't matter where > you are, the tools are good, there and largely replaceable [citation > needed]. > > What do people think? Any issue not covered that we should? How would > that disrupt downstream users? Would it be a temporary disruption, but > with long lasting benefits? Or will it just break everything for you?There's already git mirrors of all this stuff on github - *if* you're going to go down this path, I think the 1st step is updating docs to ask people to use those as a preferred solution. Based on that feedback you can wean people off svn instead of hard cut. This really falls into 2 categories - read-only and write.. For read-only the migration should be straight forward, but write gets a bit more tricky as you'd need to likely rely on git-svn (which I suspect some people are already using) Even though git 1.7 or something deals with submodules better - I don't personally like them at all. I'd rather have a convenience script or something which pulls and clones the sources. Not everyone needs all the sources and then there's the question of all the subprojects and build.. etc Not changing the workflow here would be most sane.. So clone llvm ; cd tools / ; clone ... ---------- good luck with this.. I foresee near religious opinions on the horizon..
On 31 May 2016 at 12:31, Renato Golin via cfe-dev <cfe-dev at lists.llvm.org> wrote:> What do people think? Any issue not covered that we should?I'm in favour of the move. Git-svn just about works most of the time, but I find it makes committing to release branches particularly painful. It also randomly corrupts its database occasionally, just for the giggles I assume. Tim.
On Tue, May 31, 2016 at 12:51 PM, Tim Northover via llvm-dev < llvm-dev at lists.llvm.org> wrote:> On 31 May 2016 at 12:31, Renato Golin via cfe-dev > <cfe-dev at lists.llvm.org> wrote: > > What do people think? Any issue not covered that we should? > > I'm in favour of the move. Git-svn just about works most of the time, > but I find it makes committing to release branches particularly > painful. It also randomly corrupts its database occasionally, just for > the giggles I assume. >This hit me over the weekend, it was quite annoying. +1 from me.> > Tim. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/33d0f096/attachment.html>
> On May 31, 2016, at 12:31 PM, Renato Golin via llvm-dev <llvm-dev at lists.llvm.org> wrote: > There has been some discussion on IRC about SVN hosting and the perils > of doing it ourselves. The consensus on the current discussion was > that moving to a Git-only solution would have some disvantages, but > many advantages. Furthermore, not hosting our own repos would save us > a lot of headaches, admin costs and timed out connections.Personally, I’m hugely in favor of moving llvm’s source hosting to github at some point, despite the fact that I continue to dislike git as a tool and consider monotonicly increasing version numbers to be hugely beneficial. The killer feature to me is the community aspects of github, allowing people to get involved in the project more easily and make “drive by” contributions through the pull request model. Github also has a very scriptable interface, allowing integration of external bug trackers etc into the workflow (which is good, because its bugtracker is anemic).> 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, > etc. Not only this incurs in additional admin cost, but it also gets > outdated, locally modified, and it needs to be backed up, etc. GitHub > gives all that for us for free.Yes, it would be great to get out of this business.> 5. We can still use Bugzilla (and lock GitHub's own bug system), but > we can also use GitHub's system to manage releases (it's actually > quite good for that).If we made this change, I think we should only change one thing at a time: change source hosting, but not phabricator or the bug tracker. We could then discuss moving off phabricator to the github PR model, etc.> 6. GitHub has automated testing of merge requests, meaning we can have > pre-commit tests enabled on a set of fast bots, triggered by GitHub's > own validation hooks.This works pretty well. The major problem is with tests that are flakey. -Chris
svn sucks, and git with github rocks. It is much easier for new people to do pull and merge than to screw with svn. 2016-05-31 21:31 GMT+02:00 Renato Golin via cfe-dev <cfe-dev at lists.llvm.org> :> Folks, > > There has been some discussion on IRC about SVN hosting and the perils > of doing it ourselves. The consensus on the current discussion was > that moving to a Git-only solution would have some disvantages, but > many advantages. Furthermore, not hosting our own repos would save us > a lot of headaches, admin costs and timed out connections. > > TL;DR: GitHub + git submodules [1] could replace all the functionality > we have currently with SVN. > > (also GitLab, BitBucketc, etc). > > Here are some of the arguments made on IRC... > > 1. Due to SVN, we can't re-write history. If we use some GitHub > properties [2], we could have the same effect. > > 2. Due to SVN, we have a mandatory time sequence, so commits go first > in LLVM, then Clang (for example), and buildbots don't get lost. If we > use submodules [1], we can have a similar relationship, but in a more > explicit way, and the problem could be solved elegantly. > > 3. Some people still can only use SVN. For that, GitHub has an SVN > interface [3] to the repositories. > > 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, > etc. Not only this incurs in additional admin cost, but it also gets > outdated, locally modified, and it needs to be backed up, etc. GitHub > gives all that for us for free. > > 5. We can still use Bugzilla (and lock GitHub's own bug system), but > we can also use GitHub's system to manage releases (it's actually > quite good for that). > > 6. GitHub has automated testing of merge requests, meaning we can have > pre-commit tests enabled on a set of fast bots, triggered by GitHub's > own validation hooks. Even though that wouldn't cover everything, > having a few pre-commit bots would considerably reduce the need to > revert patches [citation needed]. > > 7. With git submodules, we'd probably want to follow the same style we > have today (llvm-projects/<prj>) instead of modelling how they look in > tree (llvm/tools/clang still as a symlink). > > 8. Once we're solo Git, we can shop around *much* more easily. By > using SVN, we're basically forced to host, or choose Source Forge. > Using just Git, we can choose GitLab, BitBucket and many others, if > GitHub is not appealing enough. Essentially, it doesn't matter where > you are, the tools are good, there and largely replaceable [citation > needed]. > > What do people think? Any issue not covered that we should? How would > that disrupt downstream users? Would it be a temporary disruption, but > with long lasting benefits? Or will it just break everything for you? > > cheers, > --renato > > > [1] https://git-scm.com/book/en/v2/Git-Tools-Submodules > [2] > https://help.github.com/articles/defining-the-mergeability-of-pull-requests/ > [3] https://help.github.com/articles/support-for-subversion-clients/ > _______________________________________________ > cfe-dev mailing list > cfe-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/26c04ea9/attachment.html>
+1 for git Best regards, Alexey Bataev> 31 мая 2016 г., в 22:31, Renato Golin via cfe-dev <cfe-dev at lists.llvm.org> написал(а): > > Folks, > > There has been some discussion on IRC about SVN hosting and the perils > of doing it ourselves. The consensus on the current discussion was > that moving to a Git-only solution would have some disvantages, but > many advantages. Furthermore, not hosting our own repos would save us > a lot of headaches, admin costs and timed out connections. > > TL;DR: GitHub + git submodules [1] could replace all the functionality > we have currently with SVN. > > (also GitLab, BitBucketc, etc). > > Here are some of the arguments made on IRC... > > 1. Due to SVN, we can't re-write history. If we use some GitHub > properties [2], we could have the same effect. > > 2. Due to SVN, we have a mandatory time sequence, so commits go first > in LLVM, then Clang (for example), and buildbots don't get lost. If we > use submodules [1], we can have a similar relationship, but in a more > explicit way, and the problem could be solved elegantly. > > 3. Some people still can only use SVN. For that, GitHub has an SVN > interface [3] to the repositories. > > 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, > etc. Not only this incurs in additional admin cost, but it also gets > outdated, locally modified, and it needs to be backed up, etc. GitHub > gives all that for us for free. > > 5. We can still use Bugzilla (and lock GitHub's own bug system), but > we can also use GitHub's system to manage releases (it's actually > quite good for that). > > 6. GitHub has automated testing of merge requests, meaning we can have > pre-commit tests enabled on a set of fast bots, triggered by GitHub's > own validation hooks. Even though that wouldn't cover everything, > having a few pre-commit bots would considerably reduce the need to > revert patches [citation needed]. > > 7. With git submodules, we'd probably want to follow the same style we > have today (llvm-projects/<prj>) instead of modelling how they look in > tree (llvm/tools/clang still as a symlink). > > 8. Once we're solo Git, we can shop around *much* more easily. By > using SVN, we're basically forced to host, or choose Source Forge. > Using just Git, we can choose GitLab, BitBucket and many others, if > GitHub is not appealing enough. Essentially, it doesn't matter where > you are, the tools are good, there and largely replaceable [citation > needed]. > > What do people think? Any issue not covered that we should? How would > that disrupt downstream users? Would it be a temporary disruption, but > with long lasting benefits? Or will it just break everything for you? > > cheers, > --renato > > > [1] https://git-scm.com/book/en/v2/Git-Tools-Submodules > [2] https://help.github.com/articles/defining-the-mergeability-of-pull-requests/ > [3] https://help.github.com/articles/support-for-subversion-clients/ >
I'm in favor of both going to git as the source of truth, and then switching the hosting to github. Echoing everyone else, this unlocks a lot of good stuff that I won't repeat, and most of it can be handled independently from the VCS move. The major blocker I see for the move is figuring out how we want to coordinate versions between the related LLVM projects. I hear *terrible* things about submodules, so I'd prefer a different sync mechanism, even if it is a bad reimplementation of repo, gclient, submodules, and all the other multi-repo sync tools. On Tue, May 31, 2016 at 12:31 PM, Renato Golin via cfe-dev < cfe-dev at lists.llvm.org> wrote:> Folks, > > There has been some discussion on IRC about SVN hosting and the perils > of doing it ourselves. The consensus on the current discussion was > that moving to a Git-only solution would have some disvantages, but > many advantages. Furthermore, not hosting our own repos would save us > a lot of headaches, admin costs and timed out connections. > > TL;DR: GitHub + git submodules [1] could replace all the functionality > we have currently with SVN. > > (also GitLab, BitBucketc, etc). > > Here are some of the arguments made on IRC... > > 1. Due to SVN, we can't re-write history. If we use some GitHub > properties [2], we could have the same effect. > > 2. Due to SVN, we have a mandatory time sequence, so commits go first > in LLVM, then Clang (for example), and buildbots don't get lost. If we > use submodules [1], we can have a similar relationship, but in a more > explicit way, and the problem could be solved elegantly. > > 3. Some people still can only use SVN. For that, GitHub has an SVN > interface [3] to the repositories. > > 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, > etc. Not only this incurs in additional admin cost, but it also gets > outdated, locally modified, and it needs to be backed up, etc. GitHub > gives all that for us for free. > > 5. We can still use Bugzilla (and lock GitHub's own bug system), but > we can also use GitHub's system to manage releases (it's actually > quite good for that). > > 6. GitHub has automated testing of merge requests, meaning we can have > pre-commit tests enabled on a set of fast bots, triggered by GitHub's > own validation hooks. Even though that wouldn't cover everything, > having a few pre-commit bots would considerably reduce the need to > revert patches [citation needed]. > > 7. With git submodules, we'd probably want to follow the same style we > have today (llvm-projects/<prj>) instead of modelling how they look in > tree (llvm/tools/clang still as a symlink). > > 8. Once we're solo Git, we can shop around *much* more easily. By > using SVN, we're basically forced to host, or choose Source Forge. > Using just Git, we can choose GitLab, BitBucket and many others, if > GitHub is not appealing enough. Essentially, it doesn't matter where > you are, the tools are good, there and largely replaceable [citation > needed]. > > What do people think? Any issue not covered that we should? How would > that disrupt downstream users? Would it be a temporary disruption, but > with long lasting benefits? Or will it just break everything for you? > > cheers, > --renato > > > [1] https://git-scm.com/book/en/v2/Git-Tools-Submodules > [2] > https://help.github.com/articles/defining-the-mergeability-of-pull-requests/ > [3] https://help.github.com/articles/support-for-subversion-clients/ > _______________________________________________ > cfe-dev mailing list > cfe-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/a9a8a1ab/attachment.html>
On Tue, May 31, 2016 at 3:31 PM, Renato Golin via cfe-dev <cfe-dev at lists.llvm.org> wrote:> Folks, > > There has been some discussion on IRC about SVN hosting and the perils > of doing it ourselves. The consensus on the current discussion was > that moving to a Git-only solution would have some disvantages, but > many advantages. Furthermore, not hosting our own repos would save us > a lot of headaches, admin costs and timed out connections.Not everyone thinks git is a step forward. Please do not force people to use a "git-only" solution.> TL;DR: GitHub + git submodules [1] could replace all the functionality > we have currently with SVN. > > (also GitLab, BitBucketc, etc). > > Here are some of the arguments made on IRC... > > 1. Due to SVN, we can't re-write history. If we use some GitHub > properties [2], we could have the same effect. > > 2. Due to SVN, we have a mandatory time sequence, so commits go first > in LLVM, then Clang (for example), and buildbots don't get lost. If we > use submodules [1], we can have a similar relationship, but in a more > explicit way, and the problem could be solved elegantly.I actually consider the monotonically increasing revisions to be a feature, but not sufficient to warrant a decision one way or the other.> 3. Some people still can only use SVN. For that, GitHub has an SVN > interface [3] to the repositories.Are we sure that github's svn integration works with common tools on Windows, like TortoiseSVN?> 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, > etc. Not only this incurs in additional admin cost, but it also gets > outdated, locally modified, and it needs to be backed up, etc. GitHub > gives all that for us for free. > > 5. We can still use Bugzilla (and lock GitHub's own bug system), but > we can also use GitHub's system to manage releases (it's actually > quite good for that). > > 6. GitHub has automated testing of merge requests, meaning we can have > pre-commit tests enabled on a set of fast bots, triggered by GitHub's > own validation hooks. Even though that wouldn't cover everything, > having a few pre-commit bots would considerably reduce the need to > revert patches [citation needed]. > > 7. With git submodules, we'd probably want to follow the same style we > have today (llvm-projects/<prj>) instead of modelling how they look in > tree (llvm/tools/clang still as a symlink). > > 8. Once we're solo Git, we can shop around *much* more easily. By > using SVN, we're basically forced to host, or choose Source Forge. > Using just Git, we can choose GitLab, BitBucket and many others, if > GitHub is not appealing enough. Essentially, it doesn't matter where > you are, the tools are good, there and largely replaceable [citation > needed]. > > What do people think? Any issue not covered that we should? How would > that disrupt downstream users? Would it be a temporary disruption, but > with long lasting benefits? Or will it just break everything for you?I'm not opposed to moving to GitHub, provided its svn interface proves to meet our needs. I am opposed to switching to a git-only solution, but I'm unclear whether that's currently on the table or not. ~Aaron
Likewise, I'd definitely be in favor of doing so. It would be great to have the entire LLDB development community on GitHub instead of the current story. Kate Stone k8stone at apple.com <mailto:k8stone at apple.com> Xcode Low Level Tools> On May 31, 2016, at 1:16 PM, Chris Lattner via lldb-dev <lldb-dev at lists.llvm.org> wrote: > >> On May 31, 2016, at 12:31 PM, Renato Golin via llvm-dev <llvm-dev at lists.llvm.org> wrote: >> There has been some discussion on IRC about SVN hosting and the perils >> of doing it ourselves. The consensus on the current discussion was >> that moving to a Git-only solution would have some disvantages, but >> many advantages. Furthermore, not hosting our own repos would save us >> a lot of headaches, admin costs and timed out connections. > > Personally, I’m hugely in favor of moving llvm’s source hosting to github at some point, despite the fact that I continue to dislike git as a tool and consider monotonicly increasing version numbers to be hugely beneficial. > > The killer feature to me is the community aspects of github, allowing people to get involved in the project more easily and make “drive by” contributions through the pull request model. Github also has a very scriptable interface, allowing integration of external bug trackers etc into the workflow (which is good, because its bugtracker is anemic). > >> 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, >> etc. Not only this incurs in additional admin cost, but it also gets >> outdated, locally modified, and it needs to be backed up, etc. GitHub >> gives all that for us for free. > > Yes, it would be great to get out of this business. > >> 5. We can still use Bugzilla (and lock GitHub's own bug system), but >> we can also use GitHub's system to manage releases (it's actually >> quite good for that). > > If we made this change, I think we should only change one thing at a time: change source hosting, but not phabricator or the bug tracker. We could then discuss moving off phabricator to the github PR model, etc. > >> 6. GitHub has automated testing of merge requests, meaning we can have >> pre-commit tests enabled on a set of fast bots, triggered by GitHub's >> own validation hooks. > > This works pretty well. The major problem is with tests that are flakey. > > -Chris > _______________________________________________ > lldb-dev mailing list > lldb-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/38882484/attachment.html>
On 31 May 2016 at 21:24, Aaron Ballman <aaron at aaronballman.com> wrote:> Are we sure that github's svn integration works with common tools on > Windows, like TortoiseSVN?That's a good question. Can you try them out and report back? cheers, --renato
> On May 31, 2016, at 1:16 PM, Chris Lattner via llvm-dev <llvm-dev at lists.llvm.org> wrote: > >> On May 31, 2016, at 12:31 PM, Renato Golin via llvm-dev <llvm-dev at lists.llvm.org> wrote: >> There has been some discussion on IRC about SVN hosting and the perils >> of doing it ourselves. The consensus on the current discussion was >> that moving to a Git-only solution would have some disvantages, but >> many advantages. Furthermore, not hosting our own repos would save us >> a lot of headaches, admin costs and timed out connections. > > Personally, I’m hugely in favor of moving llvm’s source hosting to github at some point, despite the fact that I continue to dislike git as a tool and consider monotonicly increasing version numbers to be hugely beneficial.Getting a monotonically increasing revision number seems doable in git with some server-side scripting using git notes or named tags (yet to be seen is how to achieve it *reliably* with github hosting). However the challenge is more about sharing this number across repositories (i.e. having clang and llvm in sync). I can imagine some tooling for that, but with a github hosting it may still be fragile. Ideally, I'd prefer the cross-repository to be handled with an extra layer, in a way similar as described in: https://gerrit-review.googlesource.com/Documentation/user-submodules.htm (somehow conceptually similar to Android manifests XML files). It would be easy to have tooling/scripts for llvm that would easily say "checkout llvm+clang+compiler-rt+libcxx+clang-extra here", or "update all llvm subproject under this root", or "checkout this specific revision for all these" (with a monotonic number for the revision). (+1 to all the rest of what you wrote) -- Mehdi> The killer feature to me is the community aspects of github, allowing people to get involved in the project more easily and make “drive by” contributions through the pull request model. Github also has a very scriptable interface, allowing integration of external bug trackers etc into the workflow (which is good, because its bugtracker is anemic). > >> 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, >> etc. Not only this incurs in additional admin cost, but it also gets >> outdated, locally modified, and it needs to be backed up, etc. GitHub >> gives all that for us for free. > > Yes, it would be great to get out of this business. > >> 5. We can still use Bugzilla (and lock GitHub's own bug system), but >> we can also use GitHub's system to manage releases (it's actually >> quite good for that). > > If we made this change, I think we should only change one thing at a time: change source hosting, but not phabricator or the bug tracker. We could then discuss moving off phabricator to the github PR model, etc. > >> 6. GitHub has automated testing of merge requests, meaning we can have >> pre-commit tests enabled on a set of fast bots, triggered by GitHub's >> own validation hooks. > > This works pretty well. The major problem is with tests that are flakey. > > -Chris > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
On Tue, May 31, 2016 at 04:24:08PM -0400, Aaron Ballman via cfe-dev wrote:> On Tue, May 31, 2016 at 3:31 PM, Renato Golin via cfe-dev > <cfe-dev at lists.llvm.org> wrote: > > Folks, > > > > There has been some discussion on IRC about SVN hosting and the perils > > of doing it ourselves. The consensus on the current discussion was > > that moving to a Git-only solution would have some disvantages, but > > many advantages. Furthermore, not hosting our own repos would save us > > a lot of headaches, admin costs and timed out connections. > > Not everyone thinks git is a step forward. Please do not force people > to use a "git-only" solution.Amen.> > 2. Due to SVN, we have a mandatory time sequence, so commits go first > > in LLVM, then Clang (for example), and buildbots don't get lost. If we > > use submodules [1], we can have a similar relationship, but in a more > > explicit way, and the problem could be solved elegantly. > > I actually consider the monotonically increasing revisions to be a > feature, but not sufficient to warrant a decision one way or the > other.Has the situation with git-submodules and bisect improved at all or is bisecting clang+llvm going to be manual mess? Joerg
Mehmet Erol Sanliturk via llvm-dev
2016-May-31 20:56 UTC
[llvm-dev] [cfe-dev] GitHub anyone?
On Tue, May 31, 2016 at 1:23 PM, Reid Kleckner via llvm-dev < llvm-dev at lists.llvm.org> wrote:> I'm in favor of both going to git as the source of truth, and then > switching the hosting to github. > > Echoing everyone else, this unlocks a lot of good stuff that I won't > repeat, and most of it can be handled independently from the VCS move. > > The major blocker I see for the move is figuring out how we want to > coordinate versions between the related LLVM projects. I hear *terrible* > things about submodules, so I'd prefer a different sync mechanism, even if > it is a bad reimplementation of repo, gclient, submodules, and all the > other multi-repo sync tools. > >In previous months , I have studied many thousands of Github repositories , and cloned many of them locally to compile and run . The following difficulties may exist during LLVM works : If a directory contains large number ( approximately more than thousand ) of files , only a part of these files are displayed and others are not allowed to view . You may check this from OS repositories . During cloning , if there is a reference to another repository , clone --recursive are giving errors about contained @ sign . In that case it is necessary to enter into sub-directories and manually clone that referenced sub-module ( or a script should do this ) . Instead of --recursive , the other statements may be used , but all of them have advantages/disadvantages . When "Download" is selected for repository , sub-modules are not downloaded into respective sub-directories . It is necessary to visit such directories manually one by one and download , expand , and adjust these directory contents . When a file is viewed in Github and returned back , Github is switching to the top of the directory , not aligning the page at the current cursor position . When there are large number of files in a directory , it is causing difficulty to go down to the current cursor position again and continue from there . A limited kind and size of files are shown to the user . There are many kinds that it is possible to only viewing the content is to save repository locally . To my experience , any single file in a repository directory is not permitted to download to view it in that case . I consider revision numbers as only a disastrous design : A very long number conveying nothing other than inconvenience . Therefore it will not be possible to specify "revert to _a_simple_number_" elegantly . I do not know what will be shown to say "revert to _a_cryptic_number_" . Previously it was possible to search Github for repositories on supplied keywords . Now , they have disabled that feature . Now , it seems only Internet searches may find a repository ( to my experience , only very few of them are found ) , or it may be listed in their category lists to find only ones selected by Github . The most affecting points are the above ones for me as a "visitor user" of Github repositories . I do not have any experience as "developer user" of Github . Mehmet Erol Sanliturk> On Tue, May 31, 2016 at 12:31 PM, Renato Golin via cfe-dev < > cfe-dev at lists.llvm.org> wrote: > >> Folks, >> >> There has been some discussion on IRC about SVN hosting and the perils >> of doing it ourselves. The consensus on the current discussion was >> that moving to a Git-only solution would have some disvantages, but >> many advantages. Furthermore, not hosting our own repos would save us >> a lot of headaches, admin costs and timed out connections. >> >> TL;DR: GitHub + git submodules [1] could replace all the functionality >> we have currently with SVN. >> >> (also GitLab, BitBucketc, etc). >> >> Here are some of the arguments made on IRC... >> >> 1. Due to SVN, we can't re-write history. If we use some GitHub >> properties [2], we could have the same effect. >> >> 2. Due to SVN, we have a mandatory time sequence, so commits go first >> in LLVM, then Clang (for example), and buildbots don't get lost. If we >> use submodules [1], we can have a similar relationship, but in a more >> explicit way, and the problem could be solved elegantly. >> >> 3. Some people still can only use SVN. For that, GitHub has an SVN >> interface [3] to the repositories. >> >> 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, >> etc. Not only this incurs in additional admin cost, but it also gets >> outdated, locally modified, and it needs to be backed up, etc. GitHub >> gives all that for us for free. >> >> 5. We can still use Bugzilla (and lock GitHub's own bug system), but >> we can also use GitHub's system to manage releases (it's actually >> quite good for that). >> >> 6. GitHub has automated testing of merge requests, meaning we can have >> pre-commit tests enabled on a set of fast bots, triggered by GitHub's >> own validation hooks. Even though that wouldn't cover everything, >> having a few pre-commit bots would considerably reduce the need to >> revert patches [citation needed]. >> >> 7. With git submodules, we'd probably want to follow the same style we >> have today (llvm-projects/<prj>) instead of modelling how they look in >> tree (llvm/tools/clang still as a symlink). >> >> 8. Once we're solo Git, we can shop around *much* more easily. By >> using SVN, we're basically forced to host, or choose Source Forge. >> Using just Git, we can choose GitLab, BitBucket and many others, if >> GitHub is not appealing enough. Essentially, it doesn't matter where >> you are, the tools are good, there and largely replaceable [citation >> needed]. >> >> What do people think? Any issue not covered that we should? How would >> that disrupt downstream users? Would it be a temporary disruption, but >> with long lasting benefits? Or will it just break everything for you? >> >> cheers, >> --renato >> >> >> [1] https://git-scm.com/book/en/v2/Git-Tools-Submodules >> [2] >> https://help.github.com/articles/defining-the-mergeability-of-pull-requests/ >> [3] https://help.github.com/articles/support-for-subversion-clients/ >> _______________________________________________ >> cfe-dev mailing list >> cfe-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev >> > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/054ffd2e/attachment.html>
Chris Lattner via llvm-dev wrote:> Personally, I’m hugely in favor of moving llvm’s source hosting to github at > some point, despite the fact that I continue to dislike git as a tool and > consider monotonicly increasing version numbers to be hugely beneficial.For whatever it's worth, our projects define a `buildnum` git alias: alias.buildnum=!sh -c "git rev-list --all | wc -l" So from the shell: $ git buildnum 17475 This number increases monotonically per commit. Our build scripts make this number available in various #define forms. (We use a little extra scripting logic to also determine whether there are currently any unmerged or uncommitted changes, and add an annotation to the program version in that case, e.g. "9.3.17475 [unmerged]") It's all stupidly simple, but seems to work well enough for us. Regards, Bill
On Tue, May 31, 2016 at 12:51 PM, Tim Northover via llvm-dev < llvm-dev at lists.llvm.org> wrote:> On 31 May 2016 at 12:31, Renato Golin via cfe-dev > <cfe-dev at lists.llvm.org> wrote: > > What do people think? Any issue not covered that we should? > > I'm in favour of the move. Git-svn just about works most of the time, > but I find it makes committing to release branches particularly > painful. It also randomly corrupts its database occasionally, just for > the giggles I assume. >I get hit by that every so often :-(. As others have mentioned, the monotonically incrementing ids are extremely useful, particularly when bisecting across clang/llvm. I think that Medhi's suggestion may be a viable solution. As long as a mechanism for bisecting across the repositories is worked out, definitely a +1 from me.> Tim. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-- Saleem Abdulrasool compnerd (at) compnerd (dot) org -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/02d14cfa/attachment.html>
> On May 31, 2016, at 1:16 PM, Chris Lattner via cfe-dev <cfe-dev at lists.llvm.org> wrote: > >> On May 31, 2016, at 12:31 PM, Renato Golin via llvm-dev <llvm-dev at lists.llvm.org> wrote: >> There has been some discussion on IRC about SVN hosting and the perils >> of doing it ourselves. The consensus on the current discussion was >> that moving to a Git-only solution would have some disvantages, but >> many advantages. Furthermore, not hosting our own repos would save us >> a lot of headaches, admin costs and timed out connections. > > Personally, I’m hugely in favor of moving llvm’s source hosting to github at some point, despite the fact that I continue to dislike git as a tool and consider monotonicly increasing version numbers to be hugely beneficial. > > The killer feature to me is the community aspects of github, allowing people to get involved in the project more easily and make “drive by” contributions through the pull request model. Github also has a very scriptable interface, allowing integration of external bug trackers etc into the workflow (which is good, because its bugtracker is anemic).Full agreed.> >> 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, >> etc. Not only this incurs in additional admin cost, but it also gets >> outdated, locally modified, and it needs to be backed up, etc. GitHub >> gives all that for us for free. > > Yes, it would be great to get out of this business.Yep.>> 6. GitHub has automated testing of merge requests, meaning we can have >> pre-commit tests enabled on a set of fast bots, triggered by GitHub's >> own validation hooks. > > This works pretty well. The major problem is with tests that are flakey.Performance can also be an issue; it takes a bunch of fast bots to keep up with developers testing their pull requests, especially when what you’re testing is a very large C++ code base. That said, “test and merge on success” workflows are *wonderful* for keeping the buildbots happy. - Doug
TL;DR :-) Git-submodules works fine for bisecting for read-only use. I have the repo to do that. https://github.com/llvm-project/llvm-project-submodule With a simple hooks/post-checkout, It should help effective bisecting. https://github.com/chapuni/llvm-project-scripts/blob/master/hooks/post-checkout It has refs/notes/commits, aka git-notes. That said, I am afraid that submodules would lead us to the hell for committers. More discussions would be required to manage multiple git repos. FYI, I have been using the unified repo, https://github.com/llvm-project/llvm-project , for years. It requires a wrapper script to invoke git-svn commit-diff. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160601/c8589e16/attachment.html>
Dear Renato, Do you have a set of volunteers lined up to do such a migration? Getting people willing to do the migration will obviously be key, and that was the one thing I didn't see in the original email. Regarding the issue of git sub-modules and keeping Clang/LLVM in sync, perhaps we should just put Clang and LLVM into a single git repository and add a CMake option to disable compilation of Clang (the same could be done for other LLVM sub-projects for which bisection and other nifty features require a single revision number to refer to code across projects). Keeping these projects in separate repositories is just more work, and I don't see what we're getting out of that extra effort. IIRC, Clang was initially separate from LLVM because it significantly added to the download time and compilation time. I don't think download time is really a problem anymore, and compilation time can be fixed with a CMake option. For what it's worth, John Criswell On 5/31/16 2:31 PM, Renato Golin via cfe-dev wrote:> Folks, > > There has been some discussion on IRC about SVN hosting and the perils > of doing it ourselves. The consensus on the current discussion was > that moving to a Git-only solution would have some disvantages, but > many advantages. Furthermore, not hosting our own repos would save us > a lot of headaches, admin costs and timed out connections. > > TL;DR: GitHub + git submodules [1] could replace all the functionality > we have currently with SVN. > > (also GitLab, BitBucketc, etc). > > Here are some of the arguments made on IRC... > > 1. Due to SVN, we can't re-write history. If we use some GitHub > properties [2], we could have the same effect. > > 2. Due to SVN, we have a mandatory time sequence, so commits go first > in LLVM, then Clang (for example), and buildbots don't get lost. If we > use submodules [1], we can have a similar relationship, but in a more > explicit way, and the problem could be solved elegantly. > > 3. Some people still can only use SVN. For that, GitHub has an SVN > interface [3] to the repositories. > > 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, > etc. Not only this incurs in additional admin cost, but it also gets > outdated, locally modified, and it needs to be backed up, etc. GitHub > gives all that for us for free. > > 5. We can still use Bugzilla (and lock GitHub's own bug system), but > we can also use GitHub's system to manage releases (it's actually > quite good for that). > > 6. GitHub has automated testing of merge requests, meaning we can have > pre-commit tests enabled on a set of fast bots, triggered by GitHub's > own validation hooks. Even though that wouldn't cover everything, > having a few pre-commit bots would considerably reduce the need to > revert patches [citation needed]. > > 7. With git submodules, we'd probably want to follow the same style we > have today (llvm-projects/<prj>) instead of modelling how they look in > tree (llvm/tools/clang still as a symlink). > > 8. Once we're solo Git, we can shop around *much* more easily. By > using SVN, we're basically forced to host, or choose Source Forge. > Using just Git, we can choose GitLab, BitBucket and many others, if > GitHub is not appealing enough. Essentially, it doesn't matter where > you are, the tools are good, there and largely replaceable [citation > needed]. > > What do people think? Any issue not covered that we should? How would > that disrupt downstream users? Would it be a temporary disruption, but > with long lasting benefits? Or will it just break everything for you? > > cheers, > --renato > > > [1]https://git-scm.com/book/en/v2/Git-Tools-Submodules > [2]https://help.github.com/articles/defining-the-mergeability-of-pull-requests/ > [3]https://help.github.com/articles/support-for-subversion-clients/ > _______________________________________________ > cfe-dev mailing list > cfe-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev-- John Criswell Assistant Professor Department of Computer Science, University of Rochester http://www.cs.rochester.edu/u/criswell
On 1 Jun 2016, at 17:02, John Criswell via llvm-dev <llvm-dev at lists.llvm.org> wrote:> > Regarding the issue of git sub-modules and keeping Clang/LLVM in sync, perhaps we should just put Clang and LLVM into a single git repository and add a CMake option to disable compilation of Clang (the same could be done for other LLVM sub-projects for which bisection and other nifty features require a single revision number to refer to code across projects). Keeping these projects in separate repositories is just more work, and I don't see what we're getting out of that extra effort.The thing that we’d gain by keeping them together is not breaking all of the people who have forks of the clang git repo. GitHub tracks 563 so far. David
> Even though git 1.7 or something deals with submodules better - I > don't personally like them at all. I'd rather have a convenience > script or something which pulls and clones the sources. Not everyone > needs all the sources and then there's the question of all the > subprojects and build.. etc Not changing the workflow here would be > most sane..Git submodule don't force you to clone all the submodules. Until you run ``` git submodule init git submodule update ``` the directories for each submodule will stay empty. Thus it isn't necessary to pull down all the sources when using git submodules. This would need support from the build system though. I'm not sure what the build system would do right now if it found an empty ``tools/clang`` directory in the source tree.> So clone llvm ; cd tools / ; clone ... > ---------- > good luck with this.. I foresee near religious opinions on the horizon..As do I.
+1 from me On Tue, May 31, 2016 at 12:51 PM, Tim Northover via llvm-dev < llvm-dev at lists.llvm.org> wrote:> On 31 May 2016 at 12:31, Renato Golin via cfe-dev > <cfe-dev at lists.llvm.org> wrote: > > What do people think? Any issue not covered that we should? > > I'm in favour of the move. Git-svn just about works most of the time, > but I find it makes committing to release branches particularly > painful. It also randomly corrupts its database occasionally, just for > the giggles I assume. > > Tim. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160601/4a00fac1/attachment.html>
On 1 June 2016 at 17:02, John Criswell <jtcriswel at gmail.com> wrote:> Do you have a set of volunteers lined up to do such a migration? Getting > people willing to do the migration will obviously be key, and that was the > one thing I didn't see in the original email.Hi John, Well, first we need to know if people are in favour, then if the migration won't bring any serious problem, and then we can think of a migration plan. :) So far, it seems people are mostly in favour, with a few that reported being locked into SVN. I had anticipated that, and have proposed GitHub's SVN integration, which allows read-write access, so it should be mostly ok. We need more tests on that side to be sure, though. The biggest problem we're facing right now is how to sync the repos. The existing llvm-repos format with all projects as sub-modules seem to be a good candidate, but I still haven't seen a consensus on how we'd do that. About the migration plan, most people seem to agree a step-by-step process is necessary. So, first we move to git-only, possibly with sub-modules, then we move to GitHub/Lab/BB, then we move Phab to GitHub merge requests, etc. Regarding volunteers, I haven't yet asked around yet, but I'm sure we have enough interested people to help. If everything else fails, I'm more than happy to do all of that myself. Though, I'd have to learn a lot more about sub-modules than I know today, which is effectively nothing. :) cheers, --renato
(As a downstream user and very infrequent committer) +1 for a move to GitHub. I'm indifferent about git, but pull requests significantly reduce the effort of contributing small patches. I've found I'm much more likely to submit a patch to a project if it is on GitHub. On Tue, May 31, 2016 at 1:31 PM, Renato Golin via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Folks, > > There has been some discussion on IRC about SVN hosting and the perils > of doing it ourselves. The consensus on the current discussion was > that moving to a Git-only solution would have some disvantages, but > many advantages. Furthermore, not hosting our own repos would save us > a lot of headaches, admin costs and timed out connections. > > TL;DR: GitHub + git submodules [1] could replace all the functionality > we have currently with SVN. > > (also GitLab, BitBucketc, etc). > > Here are some of the arguments made on IRC... > > 1. Due to SVN, we can't re-write history. If we use some GitHub > properties [2], we could have the same effect. > > 2. Due to SVN, we have a mandatory time sequence, so commits go first > in LLVM, then Clang (for example), and buildbots don't get lost. If we > use submodules [1], we can have a similar relationship, but in a more > explicit way, and the problem could be solved elegantly. > > 3. Some people still can only use SVN. For that, GitHub has an SVN > interface [3] to the repositories. > > 4. We currently host our own SVN/Git, ViewVC and Klaus, Phabricator, > etc. Not only this incurs in additional admin cost, but it also gets > outdated, locally modified, and it needs to be backed up, etc. GitHub > gives all that for us for free. > > 5. We can still use Bugzilla (and lock GitHub's own bug system), but > we can also use GitHub's system to manage releases (it's actually > quite good for that). > > 6. GitHub has automated testing of merge requests, meaning we can have > pre-commit tests enabled on a set of fast bots, triggered by GitHub's > own validation hooks. Even though that wouldn't cover everything, > having a few pre-commit bots would considerably reduce the need to > revert patches [citation needed]. > > 7. With git submodules, we'd probably want to follow the same style we > have today (llvm-projects/<prj>) instead of modelling how they look in > tree (llvm/tools/clang still as a symlink). > > 8. Once we're solo Git, we can shop around *much* more easily. By > using SVN, we're basically forced to host, or choose Source Forge. > Using just Git, we can choose GitLab, BitBucket and many others, if > GitHub is not appealing enough. Essentially, it doesn't matter where > you are, the tools are good, there and largely replaceable [citation > needed]. > > What do people think? Any issue not covered that we should? How would > that disrupt downstream users? Would it be a temporary disruption, but > with long lasting benefits? Or will it just break everything for you? > > cheers, > --renato > > > [1] https://git-scm.com/book/en/v2/Git-Tools-Submodules > [2] > https://help.github.com/articles/defining-the-mergeability-of-pull-requests/ > [3] https://help.github.com/articles/support-for-subversion-clients/ > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160601/85fd6a5a/attachment.html>
A little summary...
After a lot of discussion, I think we converged to a few issues that
we need to solved before we finally decide to move.
Firstly, the responses were overwhelmingly positive (I counted 20 of
the ~25 people strongly supporting and another 2~3 weakly supporting).
This is a good indication that the move could be very beneficial to
the community as a whole, including downstream infrastructure, not
just the reduction in upstream infrastructure admin costs.
But that doesn't mean we have cleared up all the issues...
The benefits I gathered from the thread:
* Infrastructure admin (not just server costs) is too expensive.
We're not sysadmins and maintaining all the tools is a full time job.
Volunteering works for odd problems, not for production services.
Furthermore, most of the infrastructure we need is covered by
GitHub/Lab/BB for free, on a scale that we would not have, even with a
full time sysadmin. Gratis.
* Having one official repository instead of two is beneficial to most
developers. A lot of people (most people replying on this thread), use
Git in addition to SVN. Git also seems to be used more on validation
infrastructure than SVN (no example was put forward on this thread, at
least), due to the simplicity of controlling the repository and the
tools available. Reports of how teams decided to script Git to have
linear behaviour instead of falling back to SVN are enlightening.
* Git developer tooling is a growing trend, while SVN tooling is
dying. This is not just about GUIs, but repository management (GitHub,
GitLab, BitBucket, etc versus SourceForge), bisects, branches,
remotes, hooks, workdir, submodules and all the new development seem
to be done on Git nowadays, not SVN. Windows may be an odd one related
to GUIs, but Visual Studio has Git integration and I hear it's similar
to the other MSVC VCSs. GitHub's desktop interface seems pretty cool,
too.
* Web repositories make it *a lot* easier to create add-hoc pull
requests by non-developers, which could boost the number of
contributions and future contributors, as well as external projects
using LLVM components.
* GitHub's SVN RW interface has been reported to work well for
simpler projects, but we need a more thorough examination before
declare it good enough for our purposes.
* All reports on the thread pointed that downstream infrastructure is
already using Git, so that's one less problem to worry about if we do
move.
The issues that were raised:
* Co-dependent patches already break buildbots, but the sequential ID
helps us identify and ignore. They will continue to break, even if we
use git sub-modules, so that doesn't change much, but it will be
harder to spot the issue. Server side hooks may help, as well as
sub-modules.
* Windows tooling may be an issue. There's a separate thread handling
that part, so I won't cover it here. But I have to say it wasn't by a
long shot a resonant problem. It may also have some problem with
symlinks and in-tree checkouts (when interacting with llvm-projects
and sub-modules).
* Sub-modules may help with a lot of the current relationship we have
inside the SVN repo, but it also has some problems. Namely they:
- require a modern version of git (1.7/1.9), but that's 2013 onward.
- may need additional server side scripting, but we can keep that
in another repo to control it.
- won't replace SVN's monotonic IDs, but do we *really* need them?
Sub-modules have a bad fame, I gather, but people in the thread
reported success on using it to build validation and release
infrastructure as well as doing bisects, checking out code, etc. We
probably need some documentation on how to do these things, as well as
some scripts to help people work out the dependencies (or use them).
* GitHub/Lab/BB are not perfect. They have some interface issues, but
nothing more serious than we already have on our current
infrastructure. We'll probably have to keep Bugzilla (as GitHub's own
is really poor), but we can replace all our repos (SVN, Git),
visualisation tools (ViewVC, Klaus) and Phabricator.
Of all those issues, Windows tooling is a minor problem that shouldn't
impact decision that much and sub-modules need a lot of ironing out to
be considered good enough. My *personal* take away is that sub-modules
(or an alternative server side solution) is the only strong technical
issue we need to solve before we decide.
How does a move look like?
If we decide to move, the proposed schedule is something like this:
STEP #1 : Pre Move
0. Update docs to mention the move, so people are aware the it's going on.
1. Register an official GitHub project with the LLVM foundation.
2. Setup another (read-only) mirror of llvm.org/git at this GitHub project
3. Make sure we have a la llvm-project-submodules setup in the
official account. (Optional or necessary for the buildbots?)
4. Make sure bisecting with llvm-project-submodules is a good experience
5. Make sure no one has any blocker
STEP #2 : Git Move
6. Update the buildbots to pick up updates and commits from the
official git repository
7. Update Phabricator to pick up commits from the official git repository
8. Tell people living downstream to pick up commits from the official
git repository
9. Give things time to settle. We could play some games like disabling
the svn repository for a few hours on purpose so that people can test
that their infrastructure has really become independent of the svn
repository.
... Until this point nothing has changed for developers, it will just
boil down to a lot of work for buildbot and other infrastructure
owners ...
STEP #3: Write Access Move
10. Collect peoples GitHub account information, give them push access.
Ideally while still locking the GitHub repository somehow...
11. Switch SVN repository to read-only and allow pushes to the GitHub
repository.
12. Mirror Git to SVN.
STEP #4 : Post Move
13. Archive the SVN repository, if GitHub's SVN is good enough.
14. Review and update *all* LLVM documentation.
15. Review website links pointing to viewvc/klaus/phab etc. to point
to GitHub instead.
This is an adapted version of Matthias' and Mehdi's proposal, and
it's
not a final version in any way, but these are the basic things we need
to worry about.
Steps from here...
Aaron has started the Windows tooling thread, and if you have any
comments, please follow from there. I suggest sub-modules supporters
to start another thread to iron that out separately.
Once those issues are resolved, we shall start another thread to
finally take a decision to move or not.
Thanks everyone!
cheers,
--renato
Renato,> Firstly, the responses were overwhelmingly positive (I counted 20 of > the ~25 people strongly supporting and another 2~3 weakly supporting). > This is a good indication that the move could be very beneficial to > the community as a whole, including downstream infrastructure, not > just the reduction in upstream infrastructure admin costs.I'm not quite sure that this might be a good indication. It might be possible that we're just seeing the tip of the iceberg. Maybe we should have some sort of survey so we could collect more opinions? -- With best regards, Anton Korobeynikov Department of Statistical Modelling, Saint Petersburg State University
I personally find this email thread very hard to follow and read (this isn’t anyones fault.. its just a lot of replies). I am sure others do as well. I think it would be good to have a form/survey of some sort that can get feedback from users such as: who they are, how they use LLVM/contributions/etc, if they are pro-github move, how it impacts them, etc. People could then submit their feedback in an organized way and we could get a better idea of how the community feels on the topic. I am happy to try to set something like this up. -Tanya> On Jun 2, 2016, at 8:48 AM, Renato Golin via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > A little summary... > > After a lot of discussion, I think we converged to a few issues that > we need to solved before we finally decide to move. > > Firstly, the responses were overwhelmingly positive (I counted 20 of > the ~25 people strongly supporting and another 2~3 weakly supporting). > This is a good indication that the move could be very beneficial to > the community as a whole, including downstream infrastructure, not > just the reduction in upstream infrastructure admin costs. > > But that doesn't mean we have cleared up all the issues... > > > The benefits I gathered from the thread: > > * Infrastructure admin (not just server costs) is too expensive. > We're not sysadmins and maintaining all the tools is a full time job. > Volunteering works for odd problems, not for production services. > Furthermore, most of the infrastructure we need is covered by > GitHub/Lab/BB for free, on a scale that we would not have, even with a > full time sysadmin. Gratis. > > * Having one official repository instead of two is beneficial to most > developers. A lot of people (most people replying on this thread), use > Git in addition to SVN. Git also seems to be used more on validation > infrastructure than SVN (no example was put forward on this thread, at > least), due to the simplicity of controlling the repository and the > tools available. Reports of how teams decided to script Git to have > linear behaviour instead of falling back to SVN are enlightening. > > * Git developer tooling is a growing trend, while SVN tooling is > dying. This is not just about GUIs, but repository management (GitHub, > GitLab, BitBucket, etc versus SourceForge), bisects, branches, > remotes, hooks, workdir, submodules and all the new development seem > to be done on Git nowadays, not SVN. Windows may be an odd one related > to GUIs, but Visual Studio has Git integration and I hear it's similar > to the other MSVC VCSs. GitHub's desktop interface seems pretty cool, > too. > > * Web repositories make it *a lot* easier to create add-hoc pull > requests by non-developers, which could boost the number of > contributions and future contributors, as well as external projects > using LLVM components. > > * GitHub's SVN RW interface has been reported to work well for > simpler projects, but we need a more thorough examination before > declare it good enough for our purposes. > > * All reports on the thread pointed that downstream infrastructure is > already using Git, so that's one less problem to worry about if we do > move. > > > The issues that were raised: > > * Co-dependent patches already break buildbots, but the sequential ID > helps us identify and ignore. They will continue to break, even if we > use git sub-modules, so that doesn't change much, but it will be > harder to spot the issue. Server side hooks may help, as well as > sub-modules. > > * Windows tooling may be an issue. There's a separate thread handling > that part, so I won't cover it here. But I have to say it wasn't by a > long shot a resonant problem. It may also have some problem with > symlinks and in-tree checkouts (when interacting with llvm-projects > and sub-modules). > > * Sub-modules may help with a lot of the current relationship we have > inside the SVN repo, but it also has some problems. Namely they: > - require a modern version of git (1.7/1.9), but that's 2013 onward. > - may need additional server side scripting, but we can keep that > in another repo to control it. > - won't replace SVN's monotonic IDs, but do we *really* need them? > Sub-modules have a bad fame, I gather, but people in the thread > reported success on using it to build validation and release > infrastructure as well as doing bisects, checking out code, etc. We > probably need some documentation on how to do these things, as well as > some scripts to help people work out the dependencies (or use them). > > * GitHub/Lab/BB are not perfect. They have some interface issues, but > nothing more serious than we already have on our current > infrastructure. We'll probably have to keep Bugzilla (as GitHub's own > is really poor), but we can replace all our repos (SVN, Git), > visualisation tools (ViewVC, Klaus) and Phabricator. > > Of all those issues, Windows tooling is a minor problem that shouldn't > impact decision that much and sub-modules need a lot of ironing out to > be considered good enough. My *personal* take away is that sub-modules > (or an alternative server side solution) is the only strong technical > issue we need to solve before we decide. > > > How does a move look like? > > If we decide to move, the proposed schedule is something like this: > > STEP #1 : Pre Move > > 0. Update docs to mention the move, so people are aware the it's going on. > 1. Register an official GitHub project with the LLVM foundation. > 2. Setup another (read-only) mirror of llvm.org/git at this GitHub project > 3. Make sure we have a la llvm-project-submodules setup in the > official account. (Optional or necessary for the buildbots?) > 4. Make sure bisecting with llvm-project-submodules is a good experience > 5. Make sure no one has any blocker > > STEP #2 : Git Move > > 6. Update the buildbots to pick up updates and commits from the > official git repository > 7. Update Phabricator to pick up commits from the official git repository > 8. Tell people living downstream to pick up commits from the official > git repository > 9. Give things time to settle. We could play some games like disabling > the svn repository for a few hours on purpose so that people can test > that their infrastructure has really become independent of the svn > repository. > > ... Until this point nothing has changed for developers, it will just > boil down to a lot of work for buildbot and other infrastructure > owners ... > > STEP #3: Write Access Move > > 10. Collect peoples GitHub account information, give them push access. > Ideally while still locking the GitHub repository somehow... > 11. Switch SVN repository to read-only and allow pushes to the GitHub > repository. > 12. Mirror Git to SVN. > > STEP #4 : Post Move > > 13. Archive the SVN repository, if GitHub's SVN is good enough. > 14. Review and update *all* LLVM documentation. > 15. Review website links pointing to viewvc/klaus/phab etc. to point > to GitHub instead. > > This is an adapted version of Matthias' and Mehdi's proposal, and it's > not a final version in any way, but these are the basic things we need > to worry about. > > > Steps from here... > > Aaron has started the Windows tooling thread, and if you have any > comments, please follow from there. I suggest sub-modules supporters > to start another thread to iron that out separately. > > Once those issues are resolved, we shall start another thread to > finally take a decision to move or not. > > Thanks everyone! > > cheers, > --renato > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
On Thu, Jun 02, 2016 at 04:48:36PM +0100, Renato Golin via llvm-dev wrote:> * Git developer tooling is a growing trend, while SVN tooling is > dying. This is not just about GUIs, but repository management (GitHub, > GitLab, BitBucket, etc versus SourceForge), bisects, branches, > remotes, hooks, workdir, submodules and all the new development seem > to be done on Git nowadays, not SVN. Windows may be an odd one related > to GUIs, but Visual Studio has Git integration and I hear it's similar > to the other MSVC VCSs. GitHub's desktop interface seems pretty cool, > too.I find it somewhat dishonest that the alternatives are Subversion and Git. I don't see any reason why Mercurial is excluded from the list. Given the existing Python requirement for a lot of the LLVM tooling, that part is a non-issue. It has much better Windows support.> * Web repositories make it *a lot* easier to create add-hoc pull > requests by non-developers, which could boost the number of > contributions and future contributors, as well as external projects > using LLVM components.I hear this assertion, but I don't buy it. As I said on IRC, if I follow the typical iteration process for patches from first (or even second and third) time contributors, few such patches are directly acceptable. That leaves the GitHub interface as inferior alternative to Phabricator. It seems energy would be better spend on improving the latter, e.g. by allowing registered users to trigger an integration test etc.> * Co-dependent patches already break buildbots, but the sequential ID > helps us identify and ignore. They will continue to break, even if we > use git sub-modules, so that doesn't change much, but it will be > harder to spot the issue. Server side hooks may help, as well as > sub-modules.They also make bisecting across repository boundaries much easier. I haven't heard any evidence that submodules properly deal with this. Given that one of the primary interaction with the VCS for me is bisection, anything making it more fragile or more difficult to work is a huge no. Joerg
Renato Golin via cfe-dev <cfe-dev at lists.llvm.org> writes:> * Sub-modules may help with a lot of the current relationship we have > inside the SVN repo, but it also has some problems. Namely they: > - require a modern version of git (1.7/1.9), but that's 2013 onward. > - may need additional server side scripting, but we can keep that > in another repo to control it. > - won't replace SVN's monotonic IDs, but do we *really* need them? > Sub-modules have a bad fame, I gather, but people in the thread > reported success on using it to build validation and release > infrastructure as well as doing bisects, checking out code, etc. We > probably need some documentation on how to do these things, as well as > some scripts to help people work out the dependencies (or use them).I'd like to generate and add to your notes a list of submodule alternatives so we can explore options. Here's a start. Not all of these are equally good IMHO. - Creating one big repository - Simply require multiple clones + tooling for bisect - git-subtree (with to-be-contributed enhancements) - Google repo -David
On Fri, Jun 3, 2016 at 1:18 AM, Renato Golin via llvm-dev < llvm-dev at lists.llvm.org> wrote:> The issues that were raised: > > * Co-dependent patches already break buildbots, but the sequential ID > helps us identify and ignore. They will continue to break, even if we > use git sub-modules, so that doesn't change much, but it will be > harder to spot the issue. Server side hooks may help, as well as > sub-modules. >So as I understand it, there will be a master repository with it's own commit history. Each commit in this repo should be automatically generated as a consistent snapshot of all sub-projects after they have passed build-bot smoke tests. No matter how this master repo is maintained, I'm fairly certain we can make sure that this doesn't become an issue in practice.> > - won't replace SVN's monotonic IDs, but do we *really* need them? > Sub-modules have a bad fame, I gather, but people in the thread > reported success on using it to build validation and release > infrastructure as well as doing bisects, checking out code, etc. We > probably need some documentation on how to do these things, as well as > some scripts to help people work out the dependencies (or use them). > > This master repo could replicate svn's ID's. I mean if the commits arecreated by a bot / script, each commit message could automatically include `git rev-list --count`+1. Note however that each subproject might have more than one commit in their history between the commits in the master repo. For example; write a new feature as a series of patches against both llvm & clang open a pull request somebody pushes some "approval" button (in Phabricator?) both patch series are rebased onto llvm & clang's TOT a single commit is created in the master repo, capturing how the llvm & clang submodules jumped from the commit before my patches, to the last commit in each series. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160603/60a35cca/attachment.html>
On 2 June 2016 at 08:48, Renato Golin via lldb-dev <lldb-dev at lists.llvm.org> wrote:> Of all those issues, Windows tooling is a minor problem that shouldn't > impact decision that much and sub-modules need a lot of ironing out to > be considered good enough. My *personal* take away is that sub-modules > (or an alternative server side solution) is the only strong technical > issue we need to solve before we decide.I think Takumi's repository and script covers the real uses (bisection, principally) as well as we need and probably better than SVN. I suppose we might extend it to put an auto-incrementing revision number in the commit message too, but that's trivial. I think the responses in the thread have been heavily in favour (including after AAron's mention in LLVM weekly), so we should either get a more rigorous survey going as Tanya suggested (if we think it's useful) or get started on the actual move. I'm not really convinced that a survey would reach enough of a wider audience to affect our actions here, though I think the results would be very interesting anyway in the longer term (particularly on preferred workflows).> How does a move look like? > > If we decide to move, the proposed schedule is something like this: > > STEP #1 : Pre Move > > [...] > 1. Register an official GitHub project with the LLVM foundation.Chandler already did that: https://github.com/llvm. So we're one step in already! Tim.