llvm dev - Mar 2018 - Local svn strategy for future LLVM release updates

We are starting a local LLVM backend project (using svn) and are looking for a
mechanism to periodically update to later LLVM releases.
Certainly, this seems like it would be a relatively common use case and I
expected a google search would reveal several options.  However, I didn’t really
find anything.  Perhaps I’m not looking for the right keywords, if so please
help me out and point me to any relevant docs.
Could others share their experiences with maintaining local LLVM projects?

Without any further guidance, the path we will follow comprises of the following
flow:

Create a local repo with trunk/llvm_60
Copy trunk/llvm_60 to trunk/myproj60

Make project changes on trunk/myproj60

When LLVM 7.0 is released, create trunk/llvm_70 and copy this to trunk/myproj70

Diff trunk/llvm_60 with trunk/myproj60, identify and fix any conflicts, and
apply resulting patch to trunk/myproj70

Continue project development on trunk/myproj70


I’m pretty sure this will work, what I don’t like about it is that every 6
months we are rebasing our tree. Assuming a successful project, over the years
we will have many rebased trees and finding a particular change/check-in will
require knowledge of which base a change was made.  I don’t know of any good
alternatives.

thanks
Doug


For completeness, I’ve included specific svn commands for starting with a LLVM
6.0 release and later upgrading to LLVM 7.0 below.

svnadmin create repos
svn mkdir file://`pwd`/repos/llvm<file:///%60pwd%60/repos/llvm>
svn mkdir
file://`pwd`/repos/llvm/trunk<file:///%60pwd%60/repos/llvm/trunk>
svn mkdir
file://`pwd`/repos/llvm/branches<file:///%60pwd%60/repos/llvm/branches>

svn export  http://llvm.org/svn/llvm-project/llvm/branches/release_60llvm_60
svn import llvm_60
file://`pwd`/repos/llvm/trunk/llvm_60<file:///%60pwd%60/repos/llvm/trunk/llvm_60>


svn copy trunk/llvm_60 
file://`pwd`/repos/llvm/trunk/myproj60<file:///%60pwd%60/repos/llvm/trunk/myproj60>

//Make project changes to trunk/myproj60.

//When next llvm release is available (say llvm 7.0)


svn diff
file://`pwd`/repos/llvm/trunk/llvm_60<file:///%60pwd%60/repos/llvm/trunk/llvm_60>
file://`pwd`/repos/llvm/trunk/myproj<file:///%60pwd%60/repos/llvm/trunk/myproj>
> /tmp/diffs60.patch
// Identify and fixup any conflicts, repeat above until no conflicts

// Repeat same steps as above to make a trunk/llvm_70 and trunk/myproj70
svn co
file://`pwd`/repos/llvm/trunk/llvm_70<file:///%60pwd%60/repos/llvm/trunk/llvm_70>
wc70

cd wc70
svn patch /tmp/diffs60.patch
svn commit

// Start using trunk/myproj70 as project base

Sent from my iPhone

Hi Doug,
This is a common problem but little gets published about it, at
least little that I found helpful when trying to deal with it.
You can look here for links to "Living Downstream Without Drowning"
http://llvm.org/devmtg/2015-10/
and notes from a follow-up BoF "Surviving Downstream"
http://llvm.org/devmtg/2016-03/BoF-Minutes/SurvivingDowntream.pdf

I have some additional remarks inline below.
> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
Doug
> Vanesko (dvanesko) via llvm-dev
> Sent: Saturday, March 17, 2018 8:13 AM
> To: llvm-dev at lists.llvm.org
> Subject: [llvm-dev] Local svn strategy for future LLVM release updates
> 
> We are starting a local LLVM backend project (using svn) and are looking
> for a mechanism to periodically update to later LLVM releases.
> Certainly, this seems like it would be a relatively common use case and I
> expected a google search would reveal several options.  However, I didn’t
> really find anything.  Perhaps I’m not looking for the right keywords, if
> so please help me out and point me to any relevant docs.
> Could others share their experiences with maintaining local LLVM projects?
> 
> Without any further guidance, the path we will follow comprises of the
> following flow:
> 
> Create a local repo with trunk/llvm_60
> Copy trunk/llvm_60 to trunk/myproj60
> 
> Make project changes on trunk/myproj60
> 
> When LLVM 7.0 is released, create trunk/llvm_70 and copy this to
> trunk/myproj70
> 
> Diff trunk/llvm_60 with trunk/myproj60, identify and fix any conflicts,
> and apply resulting patch to trunk/myproj70
> 
> Continue project development on trunk/myproj70
> 
> 
> I’m pretty sure this will work, what I don’t like about it is that every 6
> months we are rebasing our tree. Assuming a successful project, over the
> years we will have many rebased trees and finding a particular
> change/check-in will require knowledge of which base a change was made.  I
> don’t know of any good alternatives.
> 
> thanks
> Doug
You've correctly identified the problem with this approach, which is that
you'll lose all your individual patch comments and history.  This can be
tolerable if you have a small number of changes, and I've heard some
people talk about how they actually maintain their work as a series of
patch files that they just keep reapplying.  But it sounds like you have
a much bigger project than that.

Another tactic would be to use 'svn merge' to move all your stuff to the
new base.  This will preserve patch history to some extent but you will
have the same piles of conflicts to resolve.  The first time I did this
it took me 3 months to resolve all the differences; admittedly I was new
to LLVM at the time but it's still not what you'd call efficient. :-)

I would, nevertheless, suggest that you try it at least once and see how
it goes.  If you're adding a new target, LLVM tries to make that not too
onerous, and if you don't have lots of patches scattered all through the
codebase it might not be too painful to take the merge hit twice a year.

If it is too painful, then probably your best bet is to go with a more
continuous-integration plan, which is what we have evolved to at Sony
and the tactic we mainly describe in the Downstream talk.  This does
require some investment in automation, but if you have the resources it
is definitely the easiest on the team as a whole.

Let us know what you think, and how things are going.  You might also
find the git hooks to be a more tractable way to handle merges, but
that's for your organization to decide.

Good luck!
--paulr
> 
> 
> For completeness, I’ve included specific svn commands for starting with a
> LLVM 6.0 release and later upgrading to LLVM 7.0 below.
> 
> svnadmin create repos
> svn mkdir file://`pwd`/repos/llvm<file:///%60pwd%60/repos/llvm>
> svn mkdir
> file://`pwd`/repos/llvm/trunk<file:///%60pwd%60/repos/llvm/trunk>
> svn mkdir
>
file://`pwd`/repos/llvm/branches<file:///%60pwd%60/repos/llvm/branches>
> 
> svn export  http://llvm.org/svn/llvm-
> project/llvm/branches/release_60llvm_60
> svn import llvm_60
>
file://`pwd`/repos/llvm/trunk/llvm_60<file:///%60pwd%60/repos/llvm/trunk/l
> lvm_60>
> 
> 
> svn copy trunk/llvm_60
>
file://`pwd`/repos/llvm/trunk/myproj60<file:///%60pwd%60/repos/llvm/trunk/
> myproj60>
> 
> //Make project changes to trunk/myproj60.
> 
> //When next llvm release is available (say llvm 7.0)
> 
> 
> svn diff
>
file://`pwd`/repos/llvm/trunk/llvm_60<file:///%60pwd%60/repos/llvm/trunk/l
> lvm_60>
>
file://`pwd`/repos/llvm/trunk/myproj<file:///%60pwd%60/repos/llvm/trunk/my
> proj> > /tmp/diffs60.patch
> // Identify and fixup any conflicts, repeat above until no conflicts
> 
> // Repeat same steps as above to make a trunk/llvm_70 and trunk/myproj70
> svn co
>
file://`pwd`/repos/llvm/trunk/llvm_70<file:///%60pwd%60/repos/llvm/trunk/l
> lvm_70> wc70
> 
> cd wc70
> svn patch /tmp/diffs60.patch
> svn commit
> 
> // Start using trunk/myproj70 as project base
> 
> Sent from my iPhone
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev