llvm dev - Nov 2015 - [cfe-dev] "Living Downstream Without Drowning" BOF @ Dev Meeting

On 19 Oct 2015, at 19:05, Bruce Hoult via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> I find the git imerge script extremely useful for this kind of situation.
> 
> https://github.com/mhagger/git-imerge
> 
> Logically, it does something similar to rebasing your local branch onto
EVERY commit in the upstream branch, in turn, until it finds conflicts. There is
cleverness to make this efficient, let you stop and restart the merge, share
intermediate state with others, build&test intermediate results
(automatically if you want). At the end of the process you can choose to keep
the intermediate commits that are the same as the result of a "git
merge" or a "git rebase" or a new feature the author calls
"rebase with history" that keeps the original branch too and makes
each rebased commit have the corresponding commit in the original branch as a
parent.
Thank you very much for the recommendation!  The down side of git-imerge is that
it is O(NM), where N is the number of your commits since the last merge and M is
the number of upstream commits.  I just finished a merge of about 6 months with
4000 upstream commits and a bit over 100 local ones - it took about a week of
CPU time.  The huge up side is that it took a lot less human time than previous
merges.

The only irritation is that LLVM has a strong policy of reverting stuff that’s
broken.  The Tuples work was committed and reverted three times in quick
succession.  Each time introduced merge conflicts, though by the third time I’d
had to fix them, I was getting pretty good at it.  I don’t wish to change the
policy with regard to reverts (having a working and stable head is very
valuable), but it would be good if we could have a stronger policy that stuff
that is reverted goes into Phabricator and is not recommitted until the person
who initially reverted it has signed off (or, at least, there’s a strong
consensus that it’s the right thing).

The main lesson for me is that more frequent merges will be less painful. 
That’s a big change, because previously I’d spend a day or two fixing merge
conflicts no matter how frequently or infrequently I merged, so the incentive
was to merge every 3-6 months.

David

On 5 November 2015 at 09:18, David Chisnall via cfe-dev
<cfe-dev at lists.llvm.org> wrote:
> but it would be good if we could have a stronger policy that stuff that is
reverted goes into Phabricator and is not recommitted until the person who
initially reverted it has signed off (or, at least, there’s a strong consensus
that it’s the right thing).
I'm not sure we have that as a policy, but I assume this is the
consensus. Though, sometimes, it happens. I'd say, if that was the
only big problem you had in 4000 commits, means even with the revert
policy, the tree is pretty stable. :)

--renato

On 5 Nov 2015, at 11:43, Renato Golin <renato.golin at linaro.org>
wrote:
> 
> On 5 November 2015 at 09:18, David Chisnall via cfe-dev
> <cfe-dev at lists.llvm.org> wrote:
>> but it would be good if we could have a stronger policy that stuff that
is reverted goes into Phabricator and is not recommitted until the person who
initially reverted it has signed off (or, at least, there’s a strong consensus
that it’s the right thing).
> 
> I'm not sure we have that as a policy, but I assume this is the
> consensus. Though, sometimes, it happens. I'd say, if that was the
> only big problem you had in 4000 commits, means even with the revert
> policy, the tree is pretty stable. :)
It really was surprisingly painless, especially given that the MIPS back end
merged a load of my patches with tweaks and bug fixes in between the two merges.
Finding the cause of each conflict and throwing away my local version when there
was an improved version upstream was made very easy.

We really should document git-imerge somewhere public for the many people that
end up maintaining their own downstream forks of LLVM.

David

On 5 November 2015 at 09:18, David Chisnall via cfe-dev
<cfe-dev at lists.llvm.org> wrote:
> On 19 Oct 2015, at 19:05, Bruce Hoult via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>>
>> I find the git imerge script extremely useful for this kind of
situation.
>>
>> https://github.com/mhagger/git-imerge
>>
>> Logically, it does something similar to rebasing your local branch onto
EVERY commit in the upstream branch, in turn, until it finds conflicts. There is
cleverness to make this efficient, let you stop and restart the merge, share
intermediate state with others, build&test intermediate results
(automatically if you want). At the end of the process you can choose to keep
the intermediate commits that are the same as the result of a "git
merge" or a "git rebase" or a new feature the author calls
"rebase with history" that keeps the original branch too and makes
each rebased commit have the corresponding commit in the original branch as a
parent.
>
> Thank you very much for the recommendation!  The down side of git-imerge is
that it is O(NM), where N is the number of your commits since the last merge and
M is the number of upstream commits.  I just finished a merge of about 6 months
with 4000 upstream commits and a bit over 100 local ones - it took about a week
of CPU time.  The huge up side is that it took a lot less human time than
previous merges.
>
> The only irritation is that LLVM has a strong policy of reverting stuff
that’s broken.  The Tuples work was committed and reverted three times in quick
succession.  Each time introduced merge conflicts, though by the third time I’d
had to fix them, I was getting pretty good at it.  I don’t wish to change the
policy with regard to reverts (having a working and stable head is very
valuable), but it would be good if we could have a stronger policy that stuff
that is reverted goes into Phabricator and is not recommitted until the person
who initially reverted it has signed off (or, at least, there’s a strong
consensus that it’s the right thing).
It seems many causes of the commit->revert->commit->revert->...
cycle
are unexpected buildbot failures. The ability to submit changesets
from phabricator to test by the buildbots (which as I understand it,
isn't currently available) would reduce the frequency of this
ping-ponging. That said, it happens fairly rarely and the current
approach seems to work just fine, other than the minor annoyance David
notes.

Alex

On Thu, Nov 5, 2015 at 12:18 PM, David Chisnall <David.Chisnall at
cl.cam.ac.uk
> wrote:
> Thank you very much for the recommendation!  The down side of git-imerge
> is that it is O(NM), where N is the number of your commits since the last
> merge and M is the number of upstream commits.

Often it should be a lot less than O(NM), due to the bisection strategy,
but if there are a lot of merge conflicts then it can be more.

>   I just finished a merge of about 6 months with 4000 upstream commits and
> a bit over 100 local ones - it took about a week of CPU time.  The huge up
> side is that it took a lot less human time than previous merges

I presume you did a build and ran tests at each node? Simple git merges
won't take long at all.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151105/0d9068d7/attachment-0001.html>

On 5 Nov 2015, at 12:20, Bruce Hoult <bruce at hoult.org>
wrote:
> 
> On Thu, Nov 5, 2015 at 12:18 PM, David Chisnall <David.Chisnall at
cl.cam.ac.uk> wrote:
>> Thank you very much for the recommendation!  The down side of
git-imerge is that it is O(NM), where N is the number of your commits since the
last merge and M is the number of upstream commits.
> 
> Often it should be a lot less than O(NM), due to the bisection strategy,
but if there are a lot of merge conflicts then it can be more.
The ‘autofilling’ was what took most time, and it autofills the merge of every
pair of (upstream, local) commits.
>>   I just finished a merge of about 6 months with 4000 upstream commits
and a bit over 100 local ones - it took about a week of CPU time.  The huge up
side is that it took a lot less human time than previous merges
> 
> I presume you did a build and ran tests at each node? Simple git merges
won't take long at all.
Nope.  Unfortunately, you can’t git-imerge clang and LLVM simultaneously, and
the lack of API stability in LLVM meant that there was no way of building my
tree until I’d finished merging both.  I’m not sure if this is something that
can be fixed by using git submodules for the tools in the svn exports in some
automatic way.

David

> -----Original Message-----
> From: cfe-dev [mailto:cfe-dev-bounces at lists.llvm.org] On Behalf Of Alex
> Bradbury via cfe-dev
> Sent: 05 November 2015 12:14
> To: David Chisnall
> Cc: llvm-dev at lists.llvm.org; Bruce Hoult; cfe-dev at lists.llvm.org
> Subject: Re: [cfe-dev] [llvm-dev] "Living Downstream Without
Drowning"
> BOF @ Dev Meeting
> 
> On 5 November 2015 at 09:18, David Chisnall via cfe-dev
> <cfe-dev at lists.llvm.org> wrote:
> > On 19 Oct 2015, at 19:05, Bruce Hoult via llvm-dev <llvm-dev at
lists.llvm.org>
> wrote:
> >>
> >> I find the git imerge script extremely useful for this kind of
situation.
> >>
> >> https://github.com/mhagger/git-imerge
> >>
> >> Logically, it does something similar to rebasing your local branch
onto
> EVERY commit in the upstream branch, in turn, until it finds conflicts.
There is
> cleverness to make this efficient, let you stop and restart the merge,
share
> intermediate state with others, build&test intermediate results
> (automatically if you want). At the end of the process you can choose to
> keep the intermediate commits that are the same as the result of a
"git
> merge" or a "git rebase" or a new feature the author calls
"rebase with
> history" that keeps the original branch too and makes each rebased
commit
> have the corresponding commit in the original branch as a parent.
> >
> > Thank you very much for the recommendation!  The down side of git-
> imerge is that it is O(NM), where N is the number of your commits since the
> last merge and M is the number of upstream commits.  I just finished a
> merge of about 6 months with 4000 upstream commits and a bit over 100
> local ones - it took about a week of CPU time.  The huge up side is that it
took
> a lot less human time than previous merges.
> >
> > The only irritation is that LLVM has a strong policy of reverting
stuff that’s
> broken.  The Tuples work was committed and reverted three times in quick
> succession.  Each time introduced merge conflicts, though by the third time
Sorry for the noise.
> I’d had to fix them, I was getting pretty good at it.  I don’t wish to
change the
> policy with regard to reverts (having a working and stable head is very
> valuable), but it would be good if we could have a stronger policy that
stuff
> that is reverted goes into Phabricator and is not recommitted until the
> person who initially reverted it has signed off (or, at least, there’s a
strong
> consensus that it’s the right thing).
That's the policy the tuples work has followed. IIRC, the first couple
reverts were responses to lldb and BPF buildbot failures. The lldb failure was
fixed by someone else by the time I'd committed the revert so I immediately
re-committed. After that, the BPF failure appeared which turned out to be an
easy fix so that was another cycle. The last revert was in response to
objections from Eric. At this point, the work remains out of tree since
we're still resolving those comments.

The MCTargetMachine work that replaces the Tuples work is likely to be noisier
in some ways. It's not the simple mechanical change it used to be any more
so apologies in advance for the merge pain it may cause. The good news is that
the increments are generally smaller.
> It seems many causes of the commit->revert->commit->revert->...
cycle
> are unexpected buildbot failures. The ability to submit changesets
> from phabricator to test by the buildbots (which as I understand it,
> isn't currently available) would reduce the frequency of this
> ping-ponging. That said, it happens fairly rarely and the current
> approach seems to work just fine, other than the minor annoyance David
> notes.
> 
> Alex
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

> On Nov 5, 2015, at 1:18 AM, David Chisnall via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> On 19 Oct 2015, at 19:05, Bruce Hoult via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>> 
>> I find the git imerge script extremely useful for this kind of
situation.
>> 
>> https://github.com/mhagger/git-imerge
>> 
>> Logically, it does something similar to rebasing your local branch onto
EVERY commit in the upstream branch, in turn, until it finds conflicts. There is
cleverness to make this efficient, let you stop and restart the merge, share
intermediate state with others, build&test intermediate results
(automatically if you want). At the end of the process you can choose to keep
the intermediate commits that are the same as the result of a "git
merge" or a "git rebase" or a new feature the author calls
"rebase with history" that keeps the original branch too and makes
each rebased commit have the corresponding commit in the original branch as a
parent.
> 
> Thank you very much for the recommendation!  The down side of git-imerge is
that it is O(NM), where N is the number of your commits since the last merge and
M is the number of upstream commits.  I just finished a merge of about 6 months
with 4000 upstream commits and a bit over 100 local ones - it took about a week
of CPU time.  The huge up side is that it took a lot less human time than
previous merges.
> 
> The only irritation is that LLVM has a strong policy of reverting stuff
that’s broken.  The Tuples work was committed and reverted three times in quick
succession.  Each time introduced merge conflicts, though by the third time I’d
had to fix them, I was getting pretty good at it.  I don’t wish to change the
policy with regard to reverts (having a working and stable head is very
valuable), but it would be good if we could have a stronger policy that stuff
that is reverted goes into Phabricator and is not recommitted until the person
who initially reverted it has signed off (or, at least, there’s a strong
consensus that it’s the right thing).
If you are using git: I had good experiences with "git config
rerere.enabled true" which builds a database on how conflicts are resolved
and will look those up for you when you encounter the same conflict a 2nd time.

- Matthias

According to the author of git imerge, rerere interacts badly with it, so
he temporarily turns it off.

Half the point of git imerge is that you don't *have * *to* solve the same
conflicts repeatedly. And you don't. Unless that commit has been backed out
and reapplied multiple times. Which sould be very much the exception.

The other half of the point of git imerge is that you solve the minimum
possible conflict, that of exactly one trunk commit against one branch
commit. This is in general vastly easier than trying to decipher how your
branch commit conflicts with the entire history of changes to master.

On Thu, Nov 5, 2015 at 9:04 PM, Matthias Braun <mbraun at apple.com>
wrote:
>
> > On Nov 5, 2015, at 1:18 AM, David Chisnall via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > On 19 Oct 2015, at 19:05, Bruce Hoult via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >>
> >> I find the git imerge script extremely useful for this kind of
> situation.
> >>
> >> https://github.com/mhagger/git-imerge
> >>
> >> Logically, it does something similar to rebasing your local branch
onto
> EVERY commit in the upstream branch, in turn, until it finds conflicts.
> There is cleverness to make this efficient, let you stop and restart the
> merge, share intermediate state with others, build&test intermediate
> results (automatically if you want). At the end of the process you can
> choose to keep the intermediate commits that are the same as the result of
> a "git merge" or a "git rebase" or a new feature the
author calls "rebase
> with history" that keeps the original branch too and makes each
rebased
> commit have the corresponding commit in the original branch as a parent.
> >
> > Thank you very much for the recommendation!  The down side of
git-imerge
> is that it is O(NM), where N is the number of your commits since the last
> merge and M is the number of upstream commits.  I just finished a merge of
> about 6 months with 4000 upstream commits and a bit over 100 local ones -
> it took about a week of CPU time.  The huge up side is that it took a lot
> less human time than previous merges.
> >
> > The only irritation is that LLVM has a strong policy of reverting
stuff
> that’s broken.  The Tuples work was committed and reverted three times in
> quick succession.  Each time introduced merge conflicts, though by the
> third time I’d had to fix them, I was getting pretty good at it.  I don’t
> wish to change the policy with regard to reverts (having a working and
> stable head is very valuable), but it would be good if we could have a
> stronger policy that stuff that is reverted goes into Phabricator and is
> not recommitted until the person who initially reverted it has signed off
> (or, at least, there’s a strong consensus that it’s the right thing).
> If you are using git: I had good experiences with "git config
> rerere.enabled true" which builds a database on how conflicts are
resolved
> and will look those up for you when you encounter the same conflict a 2nd
> time.
>
> - Matthias
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151105/66bfd6ae/attachment.html>