llvm dev - Nov 2018 - RFC: Dealing with out of tree changes and the LLVM git monorepo

Well shoot, you beat me to it.  :)

I've been working on a similar tool but it's not ready yet.  Looking
forward to trying yours!

                            -David

James Y Knight via llvm-dev <llvm-dev at lists.llvm.org> writes:
> I'm about to post exactly this tool -- I've been testing it on the
> CHERI forks of llvm/clang/lld (lots of history and merges and stuff
> there, makes a pretty nice test case!)
>
> On Mon, Nov 5, 2018 at 1:07 PM David Greene via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>
>     Mehdi AMINI <joker.eph at gmail.com> writes:
>     
>     > Yes, but that's the case for the zipper repo anyway: one merge
>     per
>     > commit. The point is that the second commit is just a trivial
>     merge,
>     > it wouldn't show up in a file `git log` for example.
>     > In the linear rewritten monorepo, adding the history taken from
>     the
>     > existing git mirror would lead to duplicated commits, as in
>     > *identical* commit / commit with the same diff but different git
>     > hashes. I'd expect git log to show us the two commits in the
git
>     log
>     > of a single file.
>     
>     Would it be valuable to have a tool to take branches from existing
>     git
>     mirrors and rewrite them in terms of the monorepo so there would
>     be no
>     duplicate commits and everything would appear to have been done
>     against
>     the monorepo?
>     
>     I know Justin is worried about hashes in old e-mails being
>     invalid, but
>     the tool could include the mapping from old hash to new hash in
>     the
>     commit message. Of course that would only be done for local
>     downstream
>     commits, as the monorepo commits were already rewritten without
>     including that information. Would it be helpful to have the
>     monorepo
>     commits contain that information?
>     
>     -David
>     _______________________________________________
>     LLVM Developers mailing list
>     llvm-dev at lists.llvm.org
>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

As promised, here's a tool for migrating downstream forks of the split
subproject repositories (e.g. https://github.com/llvm-mirror/**) into a
monorepo. It could be extended to handle conversion of a previous monorepo
into the current one, too, but it doesn't currently.

(BTW -- just a reminder that the "llvm-git-prototype" repositories are
NOT
FINAL yet and may still change. Until it is finalized, please don't make
repositories based on it, other than for demonstration/testing purposes.)

To use this tool, you must first pull together a git repository which
includes the upstream monorepo, the relevant upstream split repositories,
and any number of your private split repositories. Then, run the script to
rewrite all the specified branches (destructively, in-place!), as if you'd
made the commits on top of the monorepo.

Note -- this does not _interleave_ commits made to different subproject
forks/branches. E.g., If you start with a branch of the "llvm"
subproject
repo and a branch of the "clang" subproject repo, after running this,
you'll still have two branches. But they'll now both be based on a
single
upstream repository, and one will have your changes to the llvm/ subdir,
and one will have your changes to the "clang" subdir. (if you wish to
merge
them afterwards, you may)

A known issue is that the script will give up if given a commit-graph which
merges from multiple upstream svn branches, as that could cause conflicts
in subprojects other than the one you've forked and resolved in the merge
commit within. I'm not sure if anyone will run into that, but if they do,
some heuristics can likely be added to handle it (e.g., throw out the old
release branch's changes, just keep the new).

The tool is available here:
https://github.com/jyknight/llvm-git-migration/blob/master/migrate-downstream-fork.py

I've tested it on the CHERI project, and it took about a minute to run
'migrate-downstream-fork' itself. (Pulling all the various source
repositories took longer, of course.)

This result is here, for now (again, only for demonstration/testing!):
https://github.com/jyknight/CHERI-monorepo-prototype

To that, I've only uploaded a single branch, "cheri", made from
git merge
on 3 of cheri's subproject repositories (llvm, clang, lld) after the
migration. The other subprojects that CHERI forked have not been pulled up
to a consistent revision in a long time, so can't be merged into a
consistent target branch. I didn't look into any of the other branches in
their repositories.

But, all branches of all of them did get migrated by the tool, and could be
uploaded (either as-is, to their own branches, or after making appropriate
merges between the subproject branches, whichever is preferable.)

Here's what I ran:

#!/bin/sh

set -exu


mkdir cheri-monorepo

cd cheri-monorepo

git init


git remote add monorepo https://github.com/llvm-git-prototype/llvm.git


for x in llvm lld clang lldb libunwind libcxx; do

  git remote add split/$x https://github.com/llvm-mirror/$x

  git remote add cheri/$x https://github.com/CTSRD-CHERI/$x

done


git fetch --all


../llvm-git-migration/migrate-downstream-fork.py refs/remotes/cheri
refs/tags


git checkout -b cheri $(git merge-base refs/remotes/monorepo/master
refs/remotes/cheri/llvm/master)

git merge refs/remotes/cheri/{llvm,clang,lld}/master


On Mon, Nov 5, 2018 at 1:53 PM David Greene <dag at cray.com> wrote:
> Well shoot, you beat me to it.  :)
>
> I've been working on a similar tool but it's not ready yet. 
Looking
> forward to trying yours!
>
>                             -David
>
> James Y Knight via llvm-dev <llvm-dev at lists.llvm.org> writes:
>
> > I'm about to post exactly this tool -- I've been testing it on
the
> > CHERI forks of llvm/clang/lld (lots of history and merges and stuff
> > there, makes a pretty nice test case!)
> >
> > On Mon, Nov 5, 2018 at 1:07 PM David Greene via llvm-dev
> > <llvm-dev at lists.llvm.org> wrote:
> >
> >     Mehdi AMINI <joker.eph at gmail.com> writes:
> >
> >     > Yes, but that's the case for the zipper repo anyway: one
merge
> >     per
> >     > commit. The point is that the second commit is just a trivial
> >     merge,
> >     > it wouldn't show up in a file `git log` for example.
> >     > In the linear rewritten monorepo, adding the history taken
from
> >     the
> >     > existing git mirror would lead to duplicated commits, as in
> >     > *identical* commit / commit with the same diff but different
git
> >     > hashes. I'd expect git log to show us the two commits in
the git
> >     log
> >     > of a single file.
> >
> >     Would it be valuable to have a tool to take branches from existing
> >     git
> >     mirrors and rewrite them in terms of the monorepo so there would
> >     be no
> >     duplicate commits and everything would appear to have been done
> >     against
> >     the monorepo?
> >
> >     I know Justin is worried about hashes in old e-mails being
> >     invalid, but
> >     the tool could include the mapping from old hash to new hash in
> >     the
> >     commit message. Of course that would only be done for local
> >     downstream
> >     commits, as the monorepo commits were already rewritten without
> >     including that information. Would it be helpful to have the
> >     monorepo
> >     commits contain that information?
> >
> >     -David
> >     _______________________________________________
> >     LLVM Developers mailing list
> >     llvm-dev at lists.llvm.org
> >     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181106/ccf97221/attachment.html>

This works great, thanks!

I opened a pull request for a couple of fixes I neede to get
--revmap-out to work.

                           -David

James Y Knight <jyknight at google.com> writes:
> As promised, here's a tool for migrating downstream forks of the split
> subproject repositories (e.g. https://github.com/llvm-mirror/**) into
> a monorepo. It could be extended to handle conversion of a previous
> monorepo into the current one, too, but it doesn't currently.
>
> (BTW -- just a reminder that the "llvm-git-prototype"
repositories are
> NOT FINAL yet and may still change. Until it is finalized, please don't
> make repositories based on it, other than for demonstration/testing
> purposes.)
>
> To use this tool, you must first pull together a git repository which
> includes the upstream monorepo, the relevant upstream split
> repositories, and any number of your private split repositories. Then,
> run the script to rewrite all the specified branches (destructively,
> in-place!), as if you'd made the commits on top of the monorepo.
>
> Note -- this does not _interleave_ commits made to different
> subproject forks/branches. E.g., If you start with a branch of the
> "llvm" subproject repo and a branch of the "clang"
subproject repo,
> after running this, you'll still have two branches. But they'll now
> both be based on a single upstream repository, and one will have your
> changes to the llvm/ subdir, and one will have your changes to the
> "clang" subdir. (if you wish to merge them afterwards, you may)
>
> A known issue is that the script will give up if given a commit-graph
> which merges from multiple upstream svn branches, as that could cause
> conflicts in subprojects other than the one you've forked and resolved
> in the merge commit within. I'm not sure if anyone will run into that,
> but if they do, some heuristics can likely be added to handle it
> (e.g., throw out the old release branch's changes, just keep the new).
>
> The tool is available here:
>
https://github.com/jyknight/llvm-git-migration/blob/master/migrate-downstream-
> fork.py
>
> I've tested it on the CHERI project, and it took about a minute to run
> 'migrate-downstream-fork' itself. (Pulling all the various source
> repositories took longer, of course.)
>
> This result is here, for now (again, only for demonstration/testing!):
> https://github.com/jyknight/CHERI-monorepo-prototype
>
> To that, I've only uploaded a single branch, "cheri", made
from git
> merge on 3 of cheri's subproject repositories (llvm, clang, lld) after
> the migration. The other subprojects that CHERI forked have not been
> pulled up to a consistent revision in a long time, so can't be merged
> into a consistent target branch. I didn't look into any of the other
> branches in their repositories.
>
> But, all branches of all of them did get migrated by the tool, and
> could be uploaded (either as-is, to their own branches, or after
> making appropriate merges between the subproject branches, whichever
> is preferable.)
>
> Here's what I ran:
>
> #!/bin/sh
>
> set -exu
>
> mkdir cheri-monorepo
>
> cd cheri-monorepo
>
> git init
>
> git remote add monorepo https://github.com/llvm-git-prototype/llvm.git
>
> for x in llvm lld clang lldb libunwind libcxx; do
>
> git remote add split/$x https://github.com/llvm-mirror/$x
>
> git remote add cheri/$x https://github.com/CTSRD-CHERI/$x
>
> done
>
> git fetch --all
>
> ../llvm-git-migration/migrate-downstream-fork.py refs/remotes/cheri
> refs/tags
>
> git checkout -b cheri $(git merge-base refs/remotes/monorepo/master
> refs/remotes/cheri/llvm/master)
>
> git merge refs/remotes/cheri/{llvm,clang,lld}/master
>
> On Mon, Nov 5, 2018 at 1:53 PM David Greene <dag at cray.com> wrote:
>
>     Well shoot, you beat me to it. :)
>     
>     I've been working on a similar tool but it's not ready yet.
>     Looking
>     forward to trying yours!
>     
>     -David
>     
>     James Y Knight via llvm-dev <llvm-dev at lists.llvm.org> writes:
>     
>     > I'm about to post exactly this tool -- I've been testing
it on
>     the
>     > CHERI forks of llvm/clang/lld (lots of history and merges and
>     stuff
>     > there, makes a pretty nice test case!)
>     >
>     > On Mon, Nov 5, 2018 at 1:07 PM David Greene via llvm-dev
>     > <llvm-dev at lists.llvm.org> wrote:
>     >
>     > Mehdi AMINI <joker.eph at gmail.com> writes:
>     > 
>     > > Yes, but that's the case for the zipper repo anyway: one
merge
>     > per
>     > > commit. The point is that the second commit is just a trivial
>     > merge,
>     > > it wouldn't show up in a file `git log` for example.
>     > > In the linear rewritten monorepo, adding the history taken
>     from
>     > the
>     > > existing git mirror would lead to duplicated commits, as in
>     > > *identical* commit / commit with the same diff but different
>     git
>     > > hashes. I'd expect git log to show us the two commits in
the
>     git
>     > log
>     > > of a single file.
>     > 
>     > Would it be valuable to have a tool to take branches from
>     existing
>     > git
>     > mirrors and rewrite them in terms of the monorepo so there would
>     > be no
>     > duplicate commits and everything would appear to have been done
>     > against
>     > the monorepo?
>     > 
>     > I know Justin is worried about hashes in old e-mails being
>     > invalid, but
>     > the tool could include the mapping from old hash to new hash in
>     > the
>     > commit message. Of course that would only be done for local
>     > downstream
>     > commits, as the monorepo commits were already rewritten without
>     > including that information. Would it be helpful to have the
>     > monorepo
>     > commits contain that information?
>     > 
>     > -David
>     > _______________________________________________
>     > LLVM Developers mailing list
>     > llvm-dev at lists.llvm.org
>     > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>     >
>     > _______________________________________________
>     > LLVM Developers mailing list
>     > llvm-dev at lists.llvm.org
>     > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev