Now that we have switched to git and I had to leave behind my wonderfully simple svn workflow I have noticed something curious when committing. My typical workflow once my patch is approved on Phabricator has always been: - Update my source tree to latest - Apply the approved patch - Rebuild - Retest - If all is well, commit Having to update again after rebuilding/retesting was extremely rare since SVN only prevented the commit if I am modifying the same file(s) that have been modified upstream since my update. So I tried replicating that workflow with git, but apparently that isn't really an option. Apparently git won't let me push if there have been any commits upstream at all between my last pull and my push. This means that I can never push from a fully tested state since it is almost impossible to find a window of 20-30 minutes without any commits (which is how long a rebuild/retest takes for me). One might argue that this is no worse than what I had with SVN since I would commit on top of changes that already happened upstream. But this is not always the case. Namely, if an upstream commit modifies the same file, causing a semantic conflict, I would not end up committing with the old svn workflow whereas I would end up committing with the new git workflow. I am not sure if this is something that can be solved (nor if it is something that needs to be solved) but I thought I would just bring it up. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191028/d2cf4254/attachment.html>
I've noticed the same. The net effect for me is that I add a "git pull --rebase && git diff" immediately before the commit and push. It makes me a tad nervous for exactly the reason you mention, but I don't know of a better option. Anyone else? Philip On 10/28/19 4:18 PM, Nemanja Ivanovic via llvm-dev wrote:> Now that we have switched to git and I had to leave behind my > wonderfully simple svn workflow I have noticed something curious when > committing. > > My typical workflow once my patch is approved on Phabricator has > always been: > - Update my source tree to latest > - Apply the approved patch > - Rebuild > - Retest > - If all is well, commit > > Having to update again after rebuilding/retesting was extremely rare > since SVN only prevented the commit if I am modifying the same file(s) > that have been modified upstream since my update. > > So I tried replicating that workflow with git, but apparently that > isn't really an option. Apparently git won't let me push if there have > been any commits upstream at all between my last pull and my push. > This means that I can never push from a fully tested state since it is > almost impossible to find a window of 20-30 minutes without any > commits (which is how long a rebuild/retest takes for me). > > One might argue that this is no worse than what I had with SVN since I > would commit on top of changes that already happened upstream. But > this is not always the case. Namely, if an upstream commit modifies > the same file, causing a semantic conflict, I would not end up > committing with the old svn workflow whereas I would end up committing > with the new git workflow. > > I am not sure if this is something that can be solved (nor if it is > something that needs to be solved) but I thought I would just bring it up. > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191028/ca38d619/attachment.html>
No, this is exactly right, due to the nature of git you must be fully up-to-date with the entire branch before pushing. And because the volume of patches in the project is so high, you would need to keep mid-Pacific working hours to get decent intervals of time between commits. I brought this up before the move, and nobody seemed to think it would be a big deal. Really it means you need to trust the bots more, and understand that it’s no scandal if you break something. Revert, catch up, try again. Unless/until we get “real” pull requests as the project workflow, it’s necessarily how we have to work. --paulr From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Philip Reames via llvm-dev Sent: Monday, October 28, 2019 7:24 PM To: Nemanja Ivanovic <nemanja.i.ibm at gmail.com>; llvm-dev <llvm-dev at lists.llvm.org> Subject: Re: [llvm-dev] Committing with git I've noticed the same. The net effect for me is that I add a "git pull --rebase && git diff" immediately before the commit and push. It makes me a tad nervous for exactly the reason you mention, but I don't know of a better option. Anyone else? Philip On 10/28/19 4:18 PM, Nemanja Ivanovic via llvm-dev wrote: Now that we have switched to git and I had to leave behind my wonderfully simple svn workflow I have noticed something curious when committing. My typical workflow once my patch is approved on Phabricator has always been: - Update my source tree to latest - Apply the approved patch - Rebuild - Retest - If all is well, commit Having to update again after rebuilding/retesting was extremely rare since SVN only prevented the commit if I am modifying the same file(s) that have been modified upstream since my update. So I tried replicating that workflow with git, but apparently that isn't really an option. Apparently git won't let me push if there have been any commits upstream at all between my last pull and my push. This means that I can never push from a fully tested state since it is almost impossible to find a window of 20-30 minutes without any commits (which is how long a rebuild/retest takes for me). One might argue that this is no worse than what I had with SVN since I would commit on top of changes that already happened upstream. But this is not always the case. Namely, if an upstream commit modifies the same file, causing a semantic conflict, I would not end up committing with the old svn workflow whereas I would end up committing with the new git workflow. I am not sure if this is something that can be solved (nor if it is something that needs to be solved) but I thought I would just bring it up. _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191028/7a3e72f1/attachment-0001.html>
I imagine there's probably some way to test whether the rebase did anything like what subversion would've flagged as a conflict. But can't say I know how/much about that sort of thing. (maybe there's flags to rebase that make it more cautious/verbose/interactive) On Mon, Oct 28, 2019 at 4:24 PM Philip Reames via llvm-dev < llvm-dev at lists.llvm.org> wrote:> I've noticed the same. The net effect for me is that I add a "git pull > --rebase && git diff" immediately before the commit and push. It makes me > a tad nervous for exactly the reason you mention, but I don't know of a > better option. Anyone else? > > Philip > On 10/28/19 4:18 PM, Nemanja Ivanovic via llvm-dev wrote: > > Now that we have switched to git and I had to leave behind my wonderfully > simple svn workflow I have noticed something curious when committing. > > My typical workflow once my patch is approved on Phabricator has always > been: > - Update my source tree to latest > - Apply the approved patch > - Rebuild > - Retest > - If all is well, commit > > Having to update again after rebuilding/retesting was extremely rare since > SVN only prevented the commit if I am modifying the same file(s) that have > been modified upstream since my update. > > So I tried replicating that workflow with git, but apparently that isn't > really an option. Apparently git won't let me push if there have been any > commits upstream at all between my last pull and my push. This means that I > can never push from a fully tested state since it is almost impossible to > find a window of 20-30 minutes without any commits (which is how long a > rebuild/retest takes for me). > > One might argue that this is no worse than what I had with SVN since I > would commit on top of changes that already happened upstream. But this is > not always the case. Namely, if an upstream commit modifies the same file, > causing a semantic conflict, I would not end up committing with the old svn > workflow whereas I would end up committing with the new git workflow. > > I am not sure if this is something that can be solved (nor if it is > something that needs to be solved) but I thought I would just bring it up. > > _______________________________________________ > LLVM Developers mailing listllvm-dev at lists.llvm.orghttps://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191028/582ac567/attachment.html>
Yep, that's the case. I would say that we're no worse off than we were in terms of testing, as you note. Previously svn would let us racily commit to separate files without testing the new result. Now we're forced to observe the race. At the dev meeting I heard Doug Gregor say something like, "what kind of dirty animals are you, you just push directly to master!?" Based on that, I think other communities may set up workflows where they push branches to places, and some automation rebases and updates master asynchronously, optionally conditioned on some (light) testing or approval. Maybe we'll want something like that one day, but for now, we just have to pull, rebase, push, and hope. On Mon, Oct 28, 2019 at 4:19 PM Nemanja Ivanovic via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Now that we have switched to git and I had to leave behind my wonderfully > simple svn workflow I have noticed something curious when committing. > > My typical workflow once my patch is approved on Phabricator has always > been: > - Update my source tree to latest > - Apply the approved patch > - Rebuild > - Retest > - If all is well, commit > > Having to update again after rebuilding/retesting was extremely rare since > SVN only prevented the commit if I am modifying the same file(s) that have > been modified upstream since my update. > > So I tried replicating that workflow with git, but apparently that isn't > really an option. Apparently git won't let me push if there have been any > commits upstream at all between my last pull and my push. This means that I > can never push from a fully tested state since it is almost impossible to > find a window of 20-30 minutes without any commits (which is how long a > rebuild/retest takes for me). > > One might argue that this is no worse than what I had with SVN since I > would commit on top of changes that already happened upstream. But this is > not always the case. Namely, if an upstream commit modifies the same file, > causing a semantic conflict, I would not end up committing with the old svn > workflow whereas I would end up committing with the new git workflow. > > I am not sure if this is something that can be solved (nor if it is > something that needs to be solved) but I thought I would just bring it up. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191028/5c930064/attachment.html>
> At the dev meeting I heard Doug Gregor say something like, "what kind of dirty animals are you, you just push directly to master!?" Based on that, I think other communities may set up workflows where they push branches to places, and some automation rebases and updates master asynchronously, optionally conditioned on some (light) testing or approval.Someone has already mentioned rust-lang/rust GitHub project in other thread related to GitHub migration: from their contribution guide (https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#pull-requests):> After someone has reviewed your pull request, they will leave an annotation on the pull request with an r+. It will look something like this: > @bors r+ > This tells @bors, our lovable integration bot, that your pull request has been approved. The PR then enters the merge queue, where @bors will run all the tests on every platform we support. If it all works out, @bors will merge your code into master and close the pull request. > Depending on the scale of the change, you may see a slightly different form of r+: > @bors r+ rollup > The additional rollup tells @bors that this change is eligible for to be "rolled up". Changes that are rolled up are tested and merged at the same time, to speed the process up. Typically only small changes that are expected not to conflict with one another are rolled up.Just an idea of possible automation which will contribute to stability of master on merges From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Reid Kleckner via llvm-dev Sent: Tuesday, October 29, 2019 3:21 AM To: Nemanja Ivanovic <nemanja.i.ibm at gmail.com> Cc: llvm-dev <llvm-dev at lists.llvm.org> Subject: Re: [llvm-dev] Committing with git Yep, that's the case. I would say that we're no worse off than we were in terms of testing, as you note. Previously svn would let us racily commit to separate files without testing the new result. Now we're forced to observe the race. At the dev meeting I heard Doug Gregor say something like, "what kind of dirty animals are you, you just push directly to master!?" Based on that, I think other communities may set up workflows where they push branches to places, and some automation rebases and updates master asynchronously, optionally conditioned on some (light) testing or approval. Maybe we'll want something like that one day, but for now, we just have to pull, rebase, push, and hope. On Mon, Oct 28, 2019 at 4:19 PM Nemanja Ivanovic via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: Now that we have switched to git and I had to leave behind my wonderfully simple svn workflow I have noticed something curious when committing. My typical workflow once my patch is approved on Phabricator has always been: - Update my source tree to latest - Apply the approved patch - Rebuild - Retest - If all is well, commit Having to update again after rebuilding/retesting was extremely rare since SVN only prevented the commit if I am modifying the same file(s) that have been modified upstream since my update. So I tried replicating that workflow with git, but apparently that isn't really an option. Apparently git won't let me push if there have been any commits upstream at all between my last pull and my push. This means that I can never push from a fully tested state since it is almost impossible to find a window of 20-30 minutes without any commits (which is how long a rebuild/retest takes for me). One might argue that this is no worse than what I had with SVN since I would commit on top of changes that already happened upstream. But this is not always the case. Namely, if an upstream commit modifies the same file, causing a semantic conflict, I would not end up committing with the old svn workflow whereas I would end up committing with the new git workflow. I am not sure if this is something that can be solved (nor if it is something that needs to be solved) but I thought I would just bring it up. _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------------------------------------------------------------- Joint Stock Company Intel A/O Registered legal address: Krylatsky Hills Business Park, 17 Krylatskaya Str., Bldg 4, Moscow 121614, Russian Federation This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191029/306d1d18/attachment.html>
On 2019-10-29 01:21, Reid Kleckner via llvm-dev wrote:> At the dev meeting I heard Doug Gregor say something like, "what kind of > dirty animals are you, you just push directly to master!?" Based on > that, I think other communities may set up workflows where they push > branches to places, and some automation rebases and updates master > asynchronously, optionally conditioned on some (light) testing or approval.I think in most open source projects the workflow is to create a pull request. Then the CI infrastructure will build and test that PR which is hopefully up to date with the master branch. When that passes the tests and review it will be merged into master. Then the CI infrastructure builds and tests it again. In the mean time some changes might have been made to master. In my experience it's rare that a PR that passes breaks something when it gets merged. But most projects are not as large as LLVM. So yes, one definitely needs to rely on the CI infrastructure. In my experience it's rare that a contributor will run the tests on another platforms than the contributor's main development platform. The CI infrastructure for the D programming language works a bit different (a quite large project but not as large as LLVM). The CI infrastructure will pull down the PR and the latest master and merge that and then run all the builds and tests. If a commit has been made to master it will invalidate the test results of all existing PRs. If a PR is too old it won't be tested again (until someone makes a new commit to that PR). This is to avoid old inactive PRs to delay new active ones. But nothing will automatically rebase existing PRs. With the monorepo there will be more traffic/commits but it will also be less likely that a new commit will break something because it might be to completely different project. For example, if you work on some changes to the compiler (LLVM or Clang) and someone else makes a commit to libc++. It's less likely that that commit will break my changes. After a while one gets a feeling on what's likely and less likely to break something. -- /Jacob Carlborg