llvm dev - Jul 2016 - [RFC] One or many git repositories?

On 21 Jul 2016, at 16:11, James Y Knight <jyknight at google.com>
wrote:
> 
> I don't think this is a really hard problem though -- I can think of a
few ways to help existing users that probably will work (although I'd want
to try them first, to ensure it actually does work, of course). The two I'm
thinking of are just doing "git diff" followed by "git apply
--directory=llvm" if you just want to save a patch. Or, some "git
filter-branch" invocation to rename all the files in your existing repo,
followed by "git rebase" (or "git merge"), if you have some
more history you want to maintain.
Our clones of LLVM and clang have a reasonable amount of history (a couple of
hundred commits, I believe), including multiple branches, that we’d want to
preserve.  Both branches have merged from upstream multiple times.  It’s one of
the smaller friendly forks that I know about.  I’ve not used git filter-branch
before, but I’d be very impressed if there is some simple invocation that can
can move from this model.

I was in favour of the GitHub migration primarily because a lot of downstream
LLVM users already have a workflow based around GitHub that works well and the
proposal was to make this closer to the official workflow.  I’m very nervous
about a last-minute change to require everyone downstream to restructure their
workflows.

In particular, the fact that we have a third more public GitHub forks of LLVM
than of clang, and eight times as many as of lldb implies to me that forcing
everyone downstream to pull in all subprojects would not be particularly well
received.

David

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3719 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160721/900b3418/attachment.bin>

> Our clones of LLVM and clang have a reasonable amount of history (a couple
of hundred commits, I believe), including multiple branches, that we’d want to
preserve.  Both branches have merged from upstream multiple times.  It’s one of
the smaller friendly forks that I know about.  I’ve not used git filter-branch
before, but I’d be very impressed if there is some simple invocation that can
can move from this model.
James and I owe you something here.  I think this can be handled in a
straightforward manner, but I am not 100% sure how at the moment.  I
agree this is very important.

Our demo would be much more compelling if we can use an existing
branch.  Does anyone know of one we can play with?
> In particular, the fact that we have a third more public GitHub forks of
LLVM than of clang, and eight times as many as of lldb implies to me that
forcing everyone downstream to pull in all subprojects would not be particularly
well received.
I have a hard time understanding this particular argument.  Per the
original e-mail, with three shell commands, you can hide whichever
llvm subprojects you want.  After doing that, the only overhead of the
subprojects is extra space in your .git directory, which would still
be much smaller than an llvm+clang objdir.

Is there something specific that you think will not be well-received?
Or maybe it's better to speak personally -- is there something
specific that will bother you personally about having to clone (but
not check out) everything?

On Thu, Jul 21, 2016 at 8:22 AM, David Chisnall via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> On 21 Jul 2016, at 16:11, James Y Knight <jyknight at google.com>
wrote:
>>
>> I don't think this is a really hard problem though -- I can think
of a few ways to help existing users that probably will work (although I'd
want to try them first, to ensure it actually does work, of course). The two
I'm thinking of are just doing "git diff" followed by "git
apply --directory=llvm" if you just want to save a patch. Or, some
"git filter-branch" invocation to rename all the files in your
existing repo, followed by "git rebase" (or "git merge"), if
you have some more history you want to maintain.
>
> Our clones of LLVM and clang have a reasonable amount of history (a couple
of hundred commits, I believe), including multiple branches, that we’d want to
preserve.  Both branches have merged from upstream multiple times.  It’s one of
the smaller friendly forks that I know about.  I’ve not used git filter-branch
before, but I’d be very impressed if there is some simple invocation that can
can move from this model.
>
> I was in favour of the GitHub migration primarily because a lot of
downstream LLVM users already have a workflow based around GitHub that works
well and the proposal was to make this closer to the official workflow.  I’m
very nervous about a last-minute change to require everyone downstream to
restructure their workflows.
>
> In particular, the fact that we have a third more public GitHub forks of
LLVM than of clang, and eight times as many as of lldb implies to me that
forcing everyone downstream to pull in all subprojects would not be particularly
well received.
>
> David
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

One other point about maintaining branches:

With the single repository approach, maintaining a long-running branch
that touches multiple subprojects (e.g. llvm and clang) becomes *far*
simpler.

With the umbrella repo, you have to do the submodules trickery I
described in the original e-mail.  It is complicated, and takes a lot
of typing (or requires you to develop custom scripts).  But with the
single repo, this cross-cutting branch is just a branch.

In fact even if your branch isn't cross-cutting, if it's not a branch
of LLVM proper, I'm curious how you'd do things like bisect the
branch, or even just check out and build an old version.  You check
out an old version of the (say) clang branch, and then presumably you
try to figure out the corresponding version in the LLVM repo that you
need to check out.  I guess you could find the upstream parent of your
branch, get the SVN revision number from the commit message, then go
to the LLVM branch and find a commit which has an SVN number that's
nearby?

This would all become as simple as "git checkout" under the monolithic
model.

On Thu, Jul 21, 2016 at 9:16 AM, Justin Lebar <jlebar at google.com>
wrote:
>> Our clones of LLVM and clang have a reasonable amount of history (a
couple of hundred commits, I believe), including multiple branches, that we’d
want to preserve.  Both branches have merged from upstream multiple times.  It’s
one of the smaller friendly forks that I know about.  I’ve not used git
filter-branch before, but I’d be very impressed if there is some simple
invocation that can can move from this model.
>
> James and I owe you something here.  I think this can be handled in a
> straightforward manner, but I am not 100% sure how at the moment.  I
> agree this is very important.
>
> Our demo would be much more compelling if we can use an existing
> branch.  Does anyone know of one we can play with?
>
>> In particular, the fact that we have a third more public GitHub forks
of LLVM than of clang, and eight times as many as of lldb implies to me that
forcing everyone downstream to pull in all subprojects would not be particularly
well received.
>
> I have a hard time understanding this particular argument.  Per the
> original e-mail, with three shell commands, you can hide whichever
> llvm subprojects you want.  After doing that, the only overhead of the
> subprojects is extra space in your .git directory, which would still
> be much smaller than an llvm+clang objdir.
>
> Is there something specific that you think will not be well-received?
> Or maybe it's better to speak personally -- is there something
> specific that will bother you personally about having to clone (but
> not check out) everything?
>
> On Thu, Jul 21, 2016 at 8:22 AM, David Chisnall via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> On 21 Jul 2016, at 16:11, James Y Knight <jyknight at google.com>
wrote:
>>>
>>> I don't think this is a really hard problem though -- I can
think of a few ways to help existing users that probably will work (although
I'd want to try them first, to ensure it actually does work, of course). The
two I'm thinking of are just doing "git diff" followed by
"git apply --directory=llvm" if you just want to save a patch. Or,
some "git filter-branch" invocation to rename all the files in your
existing repo, followed by "git rebase" (or "git merge"), if
you have some more history you want to maintain.
>>
>> Our clones of LLVM and clang have a reasonable amount of history (a
couple of hundred commits, I believe), including multiple branches, that we’d
want to preserve.  Both branches have merged from upstream multiple times.  It’s
one of the smaller friendly forks that I know about.  I’ve not used git
filter-branch before, but I’d be very impressed if there is some simple
invocation that can can move from this model.
>>
>> I was in favour of the GitHub migration primarily because a lot of
downstream LLVM users already have a workflow based around GitHub that works
well and the proposal was to make this closer to the official workflow.  I’m
very nervous about a last-minute change to require everyone downstream to
restructure their workflows.
>>
>> In particular, the fact that we have a third more public GitHub forks
of LLVM than of clang, and eight times as many as of lldb implies to me that
forcing everyone downstream to pull in all subprojects would not be particularly
well received.
>>
>> David
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>

On 21 Jul 2016, at 17:16, Justin Lebar <jlebar at google.com>
wrote:
> 
>> Our clones of LLVM and clang have a reasonable amount of history (a
couple of hundred commits, I believe), including multiple branches, that we’d
want to preserve.  Both branches have merged from upstream multiple times.  It’s
one of the smaller friendly forks that I know about.  I’ve not used git
filter-branch before, but I’d be very impressed if there is some simple
invocation that can can move from this model.
> 
> James and I owe you something here.  I think this can be handled in a
> straightforward manner, but I am not 100% sure how at the moment.  I
> agree this is very important.
> 
> Our demo would be much more compelling if we can use an existing
> branch.  Does anyone know of one we can play with?
Ours are public:

https://github.com/ctsrd-cheri/llvm/
https://github.com/ctsrd-cheri/clang/

If you can provide a convincing demo of how we migrate from there (and how we
migrate local clones of that) to a new model then I’d happily withdraw all
objections.

I still don’t really like the single repo model, as I’d rather see LLVM fulfil
its promise of being a set of modular reusable libraries with less tight
coupling, allowing all of the subprojects to evolve at different rates, but I
won’t object on purely aesthetic grounds.

David

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3719 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160721/c178fe20/attachment.bin>