llvm dev - Mar 2011 - [LLVMdev] How to make release branch available in git (topic changed)

Hi David
> I think the trouble with branches is the lockdown of the root repository
> directory.
Surely not (at the server)
> git svn init --stdlayout
https://<user>@llvm.org/svn/llvm-project/llvm \
>
 --ignore-paths="^.*(Apple|PowerPC.*|SVA|eh-experimental|ggreif|non-call-eh|parallel|release_.*|vector_llvm|wendling|May2007|checker|cremebrulee|start|RELEASE_1.*|RELEASE_2[0-7])"
Several problems here:
1. Bunch of additional branches / tags are created due to multiple
branch points. I don't recall for llvm, but for clang we'll end with
two tags per each release. Something like:
$ git branch -r
  trunk
  tags/RELEASE_26
  tags/RELEASE_26 at 84939
  tags/RELEASE_27
  tags/RELEASE_27 at 102415
  tags/RELEASE_28
  tags/RELEASE_28 at 115869

The problem will be much worse with new release branch scheme,
basically we'll need to add each branch by hand, etc...
2. We really don't want to push arbitrary branches to git repository.
It's really easy to add branch by an accident, so it will be much
better not to ignore stuff, but except - add by some pattern.
Unfortunately, git-svn does not allow this yet.

So, right now I'm experimenting with various ways of doing stuff, but
the results looks not pretty good.
If anything would give me a working all-in-one cmdline / .git/config
entry - I'd really appreciate this :)

-- 
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University

Anton Korobeynikov <anton at korobeynikov.info> writes:
> Hi David
>
>> I think the trouble with branches is the lockdown of the root
repository
>> directory.
> Surely not (at the server)
Yes, but ordinary users can't try to experiment and find a set of
options that work as long as the server is locked down.  So we have to
go through server admins, which seems inefficient.  But, I'm ok with
that if we're making progress.
>> git svn init --stdlayout
https://<user>@llvm.org/svn/llvm-project/llvm \
>>
 --ignore-paths="^.*(Apple|PowerPC.*|SVA|eh-experimental|ggreif|non-call-eh|parallel|release_.*|vector_llvm|wendling|May2007|checker|cremebrulee|start|RELEASE_1.*|RELEASE_2[0-7])"
> Several problems here:
> 1. Bunch of additional branches / tags are created due to multiple
> branch points. I don't recall for llvm, but for clang we'll end
with
> two tags per each release. Something like:
> $ git branch -r
>   trunk
>   tags/RELEASE_26
>   tags/RELEASE_26 at 84939
>   tags/RELEASE_27
>   tags/RELEASE_27 at 102415
>   tags/RELEASE_28
>   tags/RELEASE_28 at 115869
Yep.  But is this really a problem?  All of these tags and branches must
be useful, or why create them?  If they aren't useful, add them to the
--ignore-paths list.
> The problem will be much worse with new release branch scheme,
> basically we'll need to add each branch by hand, etc...
Why?  Ultimately the top level of llvm looks like the standard
subversion layout:

http://llvm.org/svn/llvm-project/llvm/
  # branches/
  # tags/
  # trunk/
> 2. We really don't want to push arbitrary branches to git repository.
> It's really easy to add branch by an accident, so it will be much
You mean add a brach through the git-svn mirror via "git svn branch?"
How can someone do that without having permission on the server?  This
seems like an svn permissions issue to me.

Or are you worried about some svn user adding a branch?  Again, this
seems like a server permissions issue.  The LLVM policy is "no
branches"
so why don't we enforce it and only allow the release manager to create
them?
> better not to ignore stuff, but except - add by some pattern.
> Unfortunately, git-svn does not allow this yet.
That certainly would be a useful feature.  It shouldn't be hard to
implement as it's just a perl script.  Getting it accepted upstream and
pushed out to clients is the bigger problem.
> So, right now I'm experimenting with various ways of doing stuff, but
> the results looks not pretty good.
What's not good about them?

So far my experience is that git svn does a really good job following
svn branches and making them available via git.  See for example this
posting:

http://www.jukie.net/bart/blog/svn-branches-in-git

More experimentation is necessary, though.  I'm still pretty new at
this.
> If anything would give me a working all-in-one cmdline / .git/config
> entry - I'd really appreciate this :)
I just blew mine away to try something else.  :) But it did work well
before.  I'll send the new one to you when it's ready.

I'm playing around with some git-svn stuff here with our repositories.
If I find anything interesting I'll let you know.

I believe the key to making this work is to have a separate "svn
commit"
branch in the user's git clone so that people who do dcommit always do
it from there.  That way git svn rebase won't screw up their
"normal"
git branches.  I've tried a bit of this and it seems if one isolates the
git svn stuff to its own branch, things work pretty smoothly.

Graphically:

                       svn
                        | git svn init/fetch
                        V
                    git-svn clone
                        | git clone
                        V
                   user's clone
                    /       \
                   /         \
                master     commit

The user does a "git pull" into their local master, works on it,
creates
local branches, etc. just as a normal git user would.  Once something is
ready to go upstream, the user does the following:

git commit -a (commit to local git master)
git checkout commit
git merge master
git svn rebase
git svn dcommit

Then the next git clone to master will pick up the change history and
not conflict with the change as it exists in the user's local master.

I did this with the existing LLVM git mirror on my most recent commit to
verify that there were no issues.  It was a simple change to a component
that doesn't change much, but I'm going to get some more experience with
this in the coming days and weeks.

I set things up a suggested by Tobias Grosser:

http://permalink.gmane.org/gmane.comp.compilers.clang.devel/12843

It works well for trunk.  I think it should work equally well if/when
branches are added to the git mirror.  Since no one should be committing
to branches except the release manager, everything will always go
upstream through the local "commit" branch.  This keeps the svn
metadata
sane.
>From my point of view, the main reason to add the branches to the git
mirror is to *greatly* ease the burden for third parties when they
upgrade to a new release AND when they send patches upstream.  The git
branch/merge/conflict resolution process is just killer for this.  I
imagine most third parties don't work off of trunk.  We certainly don't.

                                -Dave

On 03/07/2011 08:30 PM, Anton Korobeynikov wrote:
> Hi David
>
>> I think the trouble with branches is the lockdown of the root
repository
>> directory.
> Surely not (at the server)
>
>> git svn init --stdlayout
https://<user>@llvm.org/svn/llvm-project/llvm \
>>  
--ignore-paths="^.*(Apple|PowerPC.*|SVA|eh-experimental|ggreif|non-call-eh|parallel|release_.*|vector_llvm|wendling|May2007|checker|cremebrulee|start|RELEASE_1.*|RELEASE_2[0-7])"
> Several problems here:
> 1. Bunch of additional branches / tags are created due to multiple
> branch points. I don't recall for llvm, but for clang we'll end
with
> two tags per each release. Something like:
> $ git branch -r
>    trunk
>    tags/RELEASE_26
>    tags/RELEASE_26 at 84939
>    tags/RELEASE_27
>    tags/RELEASE_27 at 102415
>    tags/RELEASE_28
>    tags/RELEASE_28 at 115869
>
> The problem will be much worse with new release branch scheme,
> basically we'll need to add each branch by hand, etc...
> 2. We really don't want to push arbitrary branches to git repository.
> It's really easy to add branch by an accident, so it will be much
> better not to ignore stuff, but except - add by some pattern.
> Unfortunately, git-svn does not allow this yet.
Why not? As far as I understand --ignore-paths takes a perl regular 
expression. So we could just provide a regular expression that matches 
on all paths except the ones we want to keep.

The following expression e.g.

/^.*(?<!trunk|RELEASE_2.).$/m

uses lookbehind to matches on:

   tags/SVA
   tags/eh-experimental
   tags/ggreif
   tags/non-call-eh
   tags/RELEASE_28 at 115869
   branches/Apple
   branches/PowerPC-A
   branches/PowerPC-B

but not on

   trunk
   tags/RELEASE_28
   tags/RELEASE_29
   tags/RELEASE_27

It probably needs some adjustments to get it right, but I believe this 
is basically a problem of finding the right regex.
> So, right now I'm experimenting with various ways of doing stuff, but
> the results looks not pretty good.
> If anything would give me a working all-in-one cmdline / .git/config
> entry - I'd really appreciate this :)
>
I would love to. However, as David pointed out, this is difficult with 
the blocked svn access.

Cheers
Tobi

Hi Tobias,
> The following expression e.g.
>
> /^.*(?<!trunk|RELEASE_2.).$/m
>
> uses lookbehind to matches on:
Thanks. Clever trick, but...

Variable length lookbehind not implemented in regex
m/^.*(?<!trunk|RELEASE_2.).$/ at /usr/lib/git-core/git-svn line 4078.

:(

-- 
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University

> I would love to. However, as David pointed out, this is difficult with the
> blocked svn access.
I believe that --no-minimize-url might help:

--no-minimize-url
When tracking multiple directories (using --stdlayout, --branches, or
--tags options), git svn will attempt to connect to the root (or
highest allowed level) of the Subversion repository. This default
allows better tracking of history if entire projects are moved within
a repository, but may cause issues on repositories where read access
restrictions are in place. Passing --no-minimize-url will allow git
svn to accept URLs as-is without attempting to connect to a higher
level directory. This option is off by default when only one
URL/branch is tracked (it would do little good).

-- 
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University