llvm dev - May 2021 - git strategy for handling the llvm-project repo together with ours

Geoffrey, Mehdi,

Excellent observations, however I think it's worth remembering
the stated use-case.

Geoffrey Martin-Noble wrote:
> To avoid the issue of constantly having to rewrite history,
Note that the OP said:
> from time to time get the latest changes from the official
> llvm-project repo so we keep up to date over time.
I think "from time to time" is far from "constantly."  If
they
were going to do continual updates, I wouldn't suggest rebasing 
at all; but when updates are rare, I think it's an extremely
viable choice.  Also they said,
> In the future, if support for our target architecture is mature,
> and the hardware is publicly available,
This (especially the not-public part) implies to me that the cadre 
of developers is small, and imposing a rare (2x/year?) requirement 
to do a force-pull or just re-clone is not a harsh burden.

Mehdi AMINI wrote:
> Another aspect is that rebasing a long-lived branch leads to an
> history that does not make sense: you would likely just fix the
> APIs uses for the top of the branch after rebasing which will
> lead to most of the history that can't be build
I find rebasing is effectively a commit-by-commit merge-to-HEAD.
Normally when I've done this, conflicts are quite likely for API 
changes, which would have to be fixed up in the middle of the 
rebase before you could do the --continue; not a fix-at-the-end 
situation.

It's true that for a new target, most of the work would be in
target-specific files and git wouldn't notice any conflicts.
If I were doing this semi-annual rebase, I'd probably want to do
an incremental build after each commit just to catch that kind
of thing.  Tedious but not super expensive compute-wise, for a
new target, and very scriptable.

Rebasing instead of merging would also *improve* bisection, if
you pay attention to keeping the rebased commits buildable.
I promise you, having done it, bisecting the current problem to
a 6-months-of-upstream-changes merge commit *really* isn't helpful.
Eliminating those headaches was a significant benefit of our 
conversion to continual integration.

Basically, for bisection to work in a reasonable way, you have
to have either a linear history of small merges like we get now
with our continual integration, or you want a linear history
that is pure upstream with local changes at the very tip.

Anyway, best of luck to the original poster, and back to doing
real work!

--paulr

On Mon, May 10, 2021 at 2:58 PM <paul.robinson at sony.com> wrote:
> Geoffrey, Mehdi,
>
> Excellent observations, however I think it's worth remembering
> the stated use-case.
>
> Geoffrey Martin-Noble wrote:
> > To avoid the issue of constantly having to rewrite history,
>
> Note that the OP said:
>
> > from time to time get the latest changes from the official
> > llvm-project repo so we keep up to date over time.
>
> I think "from time to time" is far from "constantly." 
If they
> were going to do continual updates, I wouldn't suggest rebasing
> at all; but when updates are rare, I think it's an extremely
> viable choice.  Also they said,
>
> > In the future, if support for our target architecture is mature,
> > and the hardware is publicly available,
>
> This (especially the not-public part) implies to me that the cadre
> of developers is small, and imposing a rare (2x/year?) requirement
> to do a force-pull or just re-clone is not a harsh burden.
>
> Mehdi AMINI wrote:
> > Another aspect is that rebasing a long-lived branch leads to an
> > history that does not make sense: you would likely just fix the
> > APIs uses for the top of the branch after rebasing which will
> > lead to most of the history that can't be build
>
> I find rebasing is effectively a commit-by-commit merge-to-HEAD.
> Normally when I've done this, conflicts are quite likely for API
> changes, which would have to be fixed up in the middle of the
> rebase before you could do the --continue; not a fix-at-the-end
> situation.
>
> It's true that for a new target, most of the work would be in
> target-specific files and git wouldn't notice any conflicts.
> If I were doing this semi-annual rebase, I'd probably want to do
> an incremental build after each commit just to catch that kind
> of thing.  Tedious but not super expensive compute-wise, for a
> new target, and very scriptable.
>
> Rebasing instead of merging would also *improve* bisection, if
> you pay attention to keeping the rebased commits buildable.
>
Yes, that the option I mentioned as "not scalable": if you are
actively
developing, you'll have lot of code churn and making sure all the
intermediate state of your branch are buildable on top of the most recent
LLVM can be a lot of "unnecessary" work (I'm using quote because
it seems
subjective here).
Also you may have O(100s) or O(1000s) commits to build and fix individually
instead of making sure the end state is good. Take this over multiple years
of developments potentially...

> I promise you, having done it, bisecting the current problem to
> a 6-months-of-upstream-changes merge commit *really* isn't helpful.
>
Yes: the answer to this isn't obviously rebasing to me, I rather merge
multiple times a day (continuously if possible).
Ideally you're only limited by the length of your testing suite: if you
have a bot that continuously does "attempt-merge, test, and
push-if-passing" you get very good bisection for the minimum amount of work
(you only have to fix something when an upstream commit breaks you, and the
fix is scoped to the actual commit that broke you). This may be what you're
mentionning below with "small merges"?


> Eliminating those headaches was a significant benefit of our
> conversion to continual integration.
>
> Basically, for bisection to work in a reasonable way, you have
> to have either a linear history of small merges like we get now
> with our continual integration, or you want a linear history
> that is pure upstream with local changes at the very tip.
>
> Anyway, best of luck to the original poster, and back to doing
> real work!
>
> --paulr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210510/e5f64e22/attachment.html>