llvm dev - Jul 2016 - [RFC] One or many git repositories?

On Wed, Jul 20, 2016 at 6:18 PM, Renato Golin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Why every one take my comments as my own personal motives?
>
> I'm just the "consensus seeker". None of these ideas are
mine, I'm
> just echoing what was said in 320 emails, plus what was said in the
> past few years when people discussed about using pure Git.
>
> People in the IRC were saying I had ulterior motives, that I was
> pushing people to use GitHub or sub-modules, or whatever. This is
> *really* not cool.
>
> Every single thread so far has died down and I wrote a summary, and no
> one said anything. Then I created another thread, and wrote another
> summary. Once no one was disagreeing, I wrote the text.
>
> Now every one wants to disagree again. Seriously?

I really really sympathize.

I, too, had read the previous N-hundred messages as having mostly
opposition to a single repository solution, and, despite feeling it would
be better, had decided it would not be worth my time to push for it. I can
live with multi-repo, and I certainly want to move to git *more* than I
want to move to single-repo. And so, here we are -- you've developed a
complete proposal for the multi-repo solution, with a high degree of
consensus around it.

And, then, suddenly, in the last day or so, a bunch of support seems to
have shown up for the one-repo solution. Way more than *I'd* ever expected
for sure. Even from people I had thought were super-opposed to it turned
out not to be!

I'm also really sad to hear that people have been impugning your motives,
because you've done a tremendous amount of work to bring this to a
conclusion, and it *really* ought to be clear to everyone that you've been
doing an admirable job of driving towards consensus here, and basically
nothing more.

IMO, the only reason we can even have this conversation about a single-repo
reasonably now is because of your work in writing up clearly the scheme for
a multi-repo solution. So I hope you don't feel discouraged by this turn of
events! I personally put the entire credit of getting to this point on your
hard work.

But, anyways, +1 on a single-repo solution from me.

Before we can agree to merge to a single-repo, there's one further question
that must be resolved:

Should the layout in the merged repository be:
1) Like the "llvm-project" git repository is now:

<root>/llvm/
<root>/clang/
<root>/compiler-rt
...

2) Like the "ideal merged checkout" is now:
llvm/
llvm/tools/clang
llvm/projects/compiler-rt
...


I don't much care which of those is chosen. I have a slight preference for
#1, for ease of doing things like grep/log/etc on llvm by itself, excluding
all the other projects. But either way seems probably fine, and an
improvement over multiple repositories.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160720/8eb05cdd/attachment.html>

On Wed, Jul 20, 2016 at 7:08 PM James Y Knight via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Before we can agree to merge to a single-repo, there's one further
> question that must be resolved:
>
I disagree that we have to solve this first. Personally, I think that we
should ifgure out whether we want to have separate repos despite them being
version-locked first. And if we do, then we should discuss what the layout
should look like.

>
> Should the layout in the merged repository be:
> 1) Like the "llvm-project" git repository is now:
>
> <root>/llvm/
> <root>/clang/
> <root>/compiler-rt
> ...
>
> 2) Like the "ideal merged checkout" is now:
> llvm/
> llvm/tools/clang
> llvm/projects/compiler-rt
> ...
>
>
> I don't much care which of those is chosen. I have a slight preference
for
> #1, for ease of doing things like grep/log/etc on llvm by itself, excluding
> all the other projects. But either way seems probably fine, and an
> improvement over multiple repositories.
>
FWIW, I strongly prefer #2, but I think the high order bit is the
repository question.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160721/a558cacf/attachment.html>

On 21 July 2016 at 03:08, James Y Knight <jyknight at google.com>
wrote:
> And, then, suddenly, in the last day or so, a bunch of support seems to
have
> shown up for the one-repo solution. Way more than I'd ever expected for
> sure. Even from people I had thought were super-opposed to it turned out
not
> to be!
Same here.

> I'm also really sad to hear that people have been impugning your
motives,
> because you've done a tremendous amount of work to bring this to a
> conclusion, and it really ought to be clear to everyone that you've
been
> doing an admirable job of driving towards consensus here, and basically
> nothing more.
Thank you. Appreciated.

> IMO, the only reason we can even have this conversation about a single-repo
> reasonably now is because of your work in writing up clearly the scheme for
> a multi-repo solution. So I hope you don't feel discouraged by this
turn of
> events! I personally put the entire credit of getting to this point on your
> hard work.
Haven't though of it that way. I feel better already. :)

> I don't much care which of those is chosen. I have a slight preference
for
> #1, for ease of doing things like grep/log/etc on llvm by itself, excluding
> all the other projects. But either way seems probably fine, and an
> improvement over multiple repositories.
I don't have a strong preference, but #1 proponents weakly convinced
me with two arguments:

1. it is easier to mix-and-match repositories as you like

I'd still symlink as I do today, but I can see why this would be
interesting for off-tree users.

2. it "makes more sense" to let Clang *use* LLVM instead of LLVM
*host* Clang

this seems more preference than anything, but people that know CMake
more than I do said it would be "easier" and I trust them. I have no
technical arguments pro or against.

Though, I'd be fine with anything really.

cheers,
--renato

On 21 Jul 2016, at 07:12, Renato Golin via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
>> I don't much care which of those is chosen. I have a slight
preference for
>> #1, for ease of doing things like grep/log/etc on llvm by itself,
excluding
>> all the other projects. But either way seems probably fine, and an
>> improvement over multiple repositories.
> 
> I don't have a strong preference, but #1 proponents weakly convinced
> me with two arguments:
> 
> 1. it is easier to mix-and-match repositories as you like
> 
> I'd still symlink as I do today, but I can see why this would be
> interesting for off-tree users.
> 
> 2. it "makes more sense" to let Clang *use* LLVM instead of LLVM
*host* Clang
> 
> this seems more preference than anything, but people that know CMake
> more than I do said it would be "easier" and I trust them. I have
no
> technical arguments pro or against.
> 
> Though, I'd be fine with anything really.
First of all, thank you very much for driving this Renato.  It’s a horrible task
to do and I’m very grateful that you’ve taken this on.

I would, however, like to add one argument against a single repo model.  If you
look at the current LLVM GitHub repo, GitHub is tracking 806 forks.  It is
tracking 595 forks for clang.  Not everyone using git for downstream development
has a fork on GitHub.  In particular, GitHub does not allow private forks of
public repos, so anyone who has a non-public git fork of LLVM will have done a
git clone and a git push to their own private repo (on or off GitHub).  I know
of about a dozen such private repos and (for some bizarre reason) most companies
don’t tell me about the secret things that they’re doing with LLVM so there are
undoubtedly a lot more that I don’t know about.

Conservatively, I would estimate that we have at least a thousand downstream
forks of the current LLVM git repository.  Moving to a single repo model with
break all of them.  It is completely unacceptable to break so many downstream
consumers unless we are able to provide them with some coherent migration plan,
but I have not seen anyone in the single-repo camp suggest anything.

David

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3719 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160721/fccc4f19/attachment.bin>

TL;DR, excuse me.

Before we can agree to merge to a single-repo, there's one further
question
> that must be resolved:
>
> Should the layout in the merged repository be:
> 1) Like the "llvm-project" git repository is now:
>
> <root>/llvm/
> <root>/clang/
> <root>/compiler-rt
> ...
>
FYI, the layout can be synthesized with tree objects in each *.git with
git-plumbing commands.

> 2) Like the "ideal merged checkout" is now:
> llvm/
> llvm/tools/clang
> llvm/projects/compiler-rt
> ...
>
I suppose the layout is just standing on "the historical reason".
At least, clang is position-independent. It'd work if clang is not on
$(BUILD_ROOT)/tools/clang, but on $(BUILD_ROOT)/projects/clang.

Since the llvm-project unified tree repo has been unofficial, I didn't
propose any llvm-project-specific things.
I have just been tweaking that makes subprojects not assumed
position-dependent.

I think we could take any option, svn, set of single repos, or unified tree.
ATM, I don't agree just to migrate to the single git repo. :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160721/a2c81379/attachment.html>

I wanted to present some of the particular reasons why I'm pretty strongly
opposed to a purely flat layout of projects the way the current github
'llvm-project' repository looks, as that hasn't happened on the list
yet.
I'm replying to myself as I don't see a much better place to hang that
conversation.

On Wed, Jul 20, 2016 at 7:38 PM Chandler Carruth <chandlerc at google.com>
wrote:
> On Wed, Jul 20, 2016 at 7:08 PM James Y Knight via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Should the layout in the merged repository be:
>> 1) Like the "llvm-project" git repository is now:
>>
>> <root>/llvm/
>> <root>/clang/
>> <root>/compiler-rt
>> ...
>>
>> 2) Like the "ideal merged checkout" is now:
>> llvm/
>> llvm/tools/clang
>> llvm/projects/compiler-rt
>> ...
>>
>>
>> I don't much care which of those is chosen. I have a slight
preference
>> for #1, for ease of doing things like grep/log/etc on llvm by itself,
>> excluding all the other projects. But either way seems probably fine,
and
>> an improvement over multiple repositories.
>>
>
> FWIW, I strongly prefer #2, but I think the high order bit is the
> repository question.
>
So, a reasonable question might be, why do I prefer #2?
I have a lot of not terribly connected reasons.

First, I want to consider what happens if we go with #1. Today, LLVM
subprojects have been formed essentially any time it was conducive to do
so. This worked around the subversion sparse checkout challenges (arguably
also solved by newer subversion features, but that's neither here nor
there) and didn't cause any problem because we could lay out the tree any
way that made sense and we always had a global revision number. A classic
example: clang-tools-extra. At the time it was added, it was perceived as
very useful to segregate. These days, I'm not sure the risk is interesting
any more, and the cost is probably higher than the benefit. But it probably
doesn't make sense to have a "cfe" directory and
"clang-tools-extra"
directory as peers. If we're moving to a monolithic repo, the
clang-tools-extra stuff should almost *certainly* move under the 'tools'
directory in the clang repo, where ever that ends up.

So, if we go with #1 above and just use the existing subversion repos as
the top level directories, how would we rationally make a decision in the
future about "should X new directory be a top-level directory, or a just
fit it into the existing hierarchy?". I don't think we will ever have a
good and principled response. We will constantly have oddball warts where
things happen to be top-level because at one point we wanted the ability to
not check out those Subversion repos, and now that has been enshrined.


I'm not actually arguing that #2 is a *good* layout. But I think it is a
(slightly) less arbitrary layout than #1. And by breaking this weird mold
of "all Subversion projects are top level", I think we'll be in a
better
place to make reasonably and considered decisions about re-structuring the
layout long-term to reflect a useful and rational layout based on some set
of reasonable technical principles.

It also has the advantage of being the layout which, if people's existing
scripts and systems are set up around the defaults in the CMake build, will
be the simplest to migrate to. I certainly know that all of my habits and
patterns are geared around this layout and it will be dramatically easier
for me to migrate to a single repo if it preserves this layout.


Long term, I want to see us use a layout that reasonably connotes the
logical and practical structure of the code and project as a whole. I also
long-term want to see the layout effectively address the pragmatic needs of
tools and systems developers rely on such as "git log". On the whole,
I
think #2 is (slightly) closer to that than #1 so I strongly prefer it, but
it clearly isn't perfect here. I just think we can incrementally fix and
improve the layout over time. I don't think we're stuck in a single
layout
forever.

Hope this helps motivate why I would very much prefer to retain the default
layout suggested in our docs and build system for now, and phrase any
re-organization as follow-on changes once we had a single repo that made
such changes straightforward and easily history-preserving.

-Chandler
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160722/8b17ecdc/attachment.html>