llvm dev - May 2021 - git strategy for handling the llvm-project repo together with ours

Dear all,

we are integrating support in Clang + LLVM for a target architecture (an 
accelerator device) we are developing. Currently, we have a git 
repository, which contains LLVM 9 together with the added/changed files 
from us.

Now, we want to have a sustainable strategy for handling the 
llvm-project repository together with our changes. To summarize, we want 
to migrate to LLVM 12, start with 12.0.0, add our target architecture 
support based on this version, continue development of our target, and 
from time to time get the latest changes from the official llvm-project 
repo so we keep up to date over time.

Hence, we somehow have to "merge" the llvm-project and our repository,
and keep history of both. As far as I know, there are several ways we 
could do this, e.g. using submodules or subtree merge.

In the future, if support for our target architecture is mature, and the 
hardware is publicly available, for us it would be interesting to have 
our target support in the official llvm-project repository, in which 
case our additions potentially would have to go into the official repo.

My question is: Is there an intended way of how we should handle this 
with git?

Any hints would be greatly appreciated.

Best regards,

Kai Plociennik

-- 
Dr. Kai Plociennik
Fraunhofer-Institut für Techno- und Wirtschaftsmathematik ITWM
Competence Center High Performance Computing
Fraunhofer-Platz 1
67663 Kaiserslautern
Tel: +49 (0)631 31600 4081
mail: kai.plociennik at itwm.fraunhofer.de
www.itwm.fraunhofer.de

> Dear all,
> 
> we are integrating support in Clang + LLVM for a target architecture (an
> accelerator device) we are developing. Currently, we have a git
> repository, which contains LLVM 9 together with the added/changed files
> from us.
> 
> Now, we want to have a sustainable strategy for handling the
> llvm-project repository together with our changes. To summarize, we want
> to migrate to LLVM 12, start with 12.0.0, add our target architecture
> support based on this version, continue development of our target, and
> from time to time get the latest changes from the official llvm-project
> repo so we keep up to date over time.
> 
> Hence, we somehow have to "merge" the llvm-project and our
repository,
> and keep history of both. As far as I know, there are several ways we
> could do this, e.g. using submodules or subtree merge.
I haven't personally had to deal with a new target, so someone who
has done it might have different suggestions.  I've cc'd Min who
seems to be the one driving the recent addition of the M68K target,
which is the most recent new target added to LLVM.

I do have extensive experience with managing downstream changes and 
handling merges, so I have suggestions based on that experience.

In your situation, I think the simplest way to manage your changes
is to start with a clone of upstream LLVM, which has its 'main'
branch.  Then create a 'target' branch off of 'main' and do all
your
work on 'target'.  When you decide to update to a new revision of
LLVM, you use 'git pull' to update 'main', and then *rebase*
your
'target' branch.  This tactic keeps all your changes at HEAD, with
history intact, which vastly simplifies figuring out where bugs 
have come from.  It would also be straightforward to move from a
rare pull/rebase to doing this more frequently, which will reduce
the "merge pain" of each pull/rebase.

I do *not* recommend what we (Sony) did, which is to keep our
changes intermixed with upstream changes.  That decision was made
too long ago to do anything about it with practical cost.  We have
an automated merge system to keep ourselves continually updated to
upstream HEAD, which is an ongoing maintenance cost but much much
better than trying to do the same thing once every six months or so.

More about how we operate can be found here:
https://llvm.org/devmtg/2015-10/#tutorial4
> In the future, if support for our target architecture is mature, and the
> hardware is publicly available, for us it would be interesting to have
> our target support in the official llvm-project repository, in which
> case our additions potentially would have to go into the official repo.
This is another reason to try to keep your downstream changes at HEAD.
It will be much easier to post your new target's patches if you are
already based on top of 'main'.

Best Regards,
--paulr