llvm dev - Oct 2013 - [LLVMdev] Downstream distributions as first class citizens in the LLVM repository

Hi all,
I mentioned this idea yesterday on IRC already and would like to discuss
in the greater context of the mailing list. NetBSD is about to import
LLVM and Clang into its repository; FreeBSD already has done that a
while ago. This creates some interesting maintainance questions. FreeBSD
has followed the LLVM/Clang releases and backported various fixes
locally. NetBSD will after the 3.4 release likely end up doing the same.
In the past, this process has created some fragmentation for GCC as
various changed tended to accumulate over time. One part was always the
somewhat tidious process of getting those changes upstream, the other
problem was the difficulty of keeping track of who exactly had what
state.

Luckily with LLVM we are in much better position when it comes to
getting changes integrated, so that's not an issue. There is still the
problem of keeping track of who has which additional (bug fixing)
patches and release management in general. For this purpose I would like
to be able to create "vendor" branches in the main repository to
reflect
exactly what it is used by the corresponding downstream repository.
This would increase the visibility of changes by any of the vendors
involved, so that others can pick up the same changes. The impact on
mailing list traffic should be low as changes are relatively rare
compared to the rest of the development speed. Code access should follow
similar practises as release management, e.g. every vendor branch has a
code owner responsible for it.

Joerg

I think Diego implemented something like this for gcc at google.


On Friday, October 18, 2013, Joerg Sonnenberger wrote:
> Hi all,
> I mentioned this idea yesterday on IRC already and would like to discuss
> in the greater context of the mailing list. NetBSD is about to import
> LLVM and Clang into its repository; FreeBSD already has done that a
> while ago. This creates some interesting maintainance questions. FreeBSD
> has followed the LLVM/Clang releases and backported various fixes
> locally. NetBSD will after the 3.4 release likely end up doing the same.
> In the past, this process has created some fragmentation for GCC as
> various changed tended to accumulate over time. One part was always the
> somewhat tidious process of getting those changes upstream, the other
> problem was the difficulty of keeping track of who exactly had what
> state.
>
> Luckily with LLVM we are in much better position when it comes to
> getting changes integrated, so that's not an issue. There is still the
> problem of keeping track of who has which additional (bug fixing)
> patches and release management in general. For this purpose I would like
> to be able to create "vendor" branches in the main repository to
reflect
> exactly what it is used by the corresponding downstream repository.
> This would increase the visibility of changes by any of the vendors
> involved, so that others can pick up the same changes. The impact on
> mailing list traffic should be low as changes are relatively rare
> compared to the rest of the development speed. Code access should follow
> similar practises as release management, e.g. every vendor branch has a
> code owner responsible for it.
>
> Joerg
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu <javascript:;>         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131018/6d52aeca/attachment.html>

On Fri, Oct 18, 2013 at 07:09:47PM +0200, Joerg Sonnenberger
wrote:
> Hi all,
> I mentioned this idea yesterday on IRC already and would like to discuss
> in the greater context of the mailing list. NetBSD is about to import
> LLVM and Clang into its repository; FreeBSD already has done that a
> while ago. This creates some interesting maintainance questions. FreeBSD
> has followed the LLVM/Clang releases and backported various fixes
> locally. NetBSD will after the 3.4 release likely end up doing the same.
> In the past, this process has created some fragmentation for GCC as
> various changed tended to accumulate over time. One part was always the
> somewhat tidious process of getting those changes upstream, the other
> problem was the difficulty of keeping track of who exactly had what
> state.
> 
> Luckily with LLVM we are in much better position when it comes to
> getting changes integrated, so that's not an issue. There is still the
> problem of keeping track of who has which additional (bug fixing)
> patches and release management in general. For this purpose I would like
> to be able to create "vendor" branches in the main repository to
reflect
> exactly what it is used by the corresponding downstream repository.
> This would increase the visibility of changes by any of the vendors
> involved, so that others can pick up the same changes. The impact on
> mailing list traffic should be low as changes are relatively rare
> compared to the rest of the development speed. Code access should follow
> similar practises as release management, e.g. every vendor branch has a
> code owner responsible for it.
I would really like to have something like this.  However, I think there
should be just be one 'vendor' branch.  There are already way too many
forks of LLVM both public and private and having a common branch for
fixes would be very beneficial and save everyone a lot of work.  Plus,
I think it would give users with private forks of LLVM an incentive to
contribute changes back to the project.

-Tom

On Fri, Oct 18, 2013 at 06:47:55PM -0400, Tom Stellard
wrote:
> On Fri, Oct 18, 2013 at 07:09:47PM +0200, Joerg Sonnenberger wrote:
> > Hi all,
> > I mentioned this idea yesterday on IRC already and would like to
discuss
> > in the greater context of the mailing list. NetBSD is about to import
> > LLVM and Clang into its repository; FreeBSD already has done that a
> > while ago. This creates some interesting maintainance questions.
FreeBSD
> > has followed the LLVM/Clang releases and backported various fixes
> > locally. NetBSD will after the 3.4 release likely end up doing the
same.
> > In the past, this process has created some fragmentation for GCC as
> > various changed tended to accumulate over time. One part was always
the
> > somewhat tidious process of getting those changes upstream, the other
> > problem was the difficulty of keeping track of who exactly had what
> > state.
> > 
> > Luckily with LLVM we are in much better position when it comes to
> > getting changes integrated, so that's not an issue. There is still
the
> > problem of keeping track of who has which additional (bug fixing)
> > patches and release management in general. For this purpose I would
like
> > to be able to create "vendor" branches in the main
repository to reflect
> > exactly what it is used by the corresponding downstream repository.
> > This would increase the visibility of changes by any of the vendors
> > involved, so that others can pick up the same changes. The impact on
> > mailing list traffic should be low as changes are relatively rare
> > compared to the rest of the development speed. Code access should
follow
> > similar practises as release management, e.g. every vendor branch has
a
> > code owner responsible for it.
> 
> I would really like to have something like this.  However, I think there
> should be just be one 'vendor' branch.  There are already way too
many
> forks of LLVM both public and private and having a common branch for
> fixes would be very beneficial and save everyone a lot of work.  Plus,
> I think it would give users with private forks of LLVM an incentive to
> contribute changes back to the project.
Having a single branch doesn't work as soon as maintaince for releases
comes into the game. Consider FreeBSD 10 shipping with Clang 3.3 and
FreeBSD 11 with Clang 3.4...

Joerg

On Fri, Oct 18, 2013 at 1:09 PM, Joerg Sonnenberger
<joerg at britannica.bec.de> wrote:
> This would increase the visibility of changes by any of the vendors
> involved, so that others can pick up the same changes. The impact on
> mailing list traffic should be low as changes are relatively rare
> compared to the rest of the development speed. Code access should follow
> similar practises as release management, e.g. every vendor branch has a
> code owner responsible for it.
This is more or less the model we follow in GCC. Vendor (and, really,
anybody) can create their own branches.
These branches are usually created off of the main release branch for
a particular version.  They are completely under the control of the
third party who created them and they usually contain a few additional
patches over the standard release.

The only requirements for third parties is that they should never
allow a bug in their branch to be filed as a GCC bug unless the same
bug can be reproduced in either trunk or the FSF release branch.

This model has worked well for many years.

At Google, we have a slightly more elaborate branching scheme because
of the amount of work we put in the compiler:

- There is one branch off of trunk (called 'integration') which we use
as a base for internal development. This branch contains a few patches
that are needed to integrate into our build system. These are usually
small changes which either make no sense to anyone else but us, or we
have not yet been able to send upstream. Ideally, this branch should
not exist.

- The branch where we do most of our development is called 'main'. It
contains the bulk of all the changes we make to the compiler for peak
performance in our applications. This branch is a buffer for major
development, which allows us to keep our internal development/release
schedule that is independent from upstream GCC. We are constantly
taking patches out of this branch and proposing them for trunk.

- Release branches.  These branches are created from our 'main' branch
and the current FSF release branch. We use them for our internal
releases. It also acts as a continuous release branch which follows
all the minor releases from the FSF. We are merging changes in the
upstream release branches to get bug fixes and minor feature changes.

All these branches are publicly available from the FSF repository.
I've been in environments where branches are kept behind closed doors.
 They are nothing but a big headache to maintain.


Diego.