llvm dev - Jul 2016 - [RFC] One or many git repositories?

On Wed, Jul 20, 2016 at 5:56 PM, Renato Golin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 21 July 2016 at 01:39, Justin Lebar <jlebar at google.com> wrote:
> > This is true if you s/checkout/clone/.  With a single repo, you must
> > clone (download) everything (*), but after you've done so you can
use
> > sparse checkouts to check out (create a working copy of) only llvm and
> > clang.  So you should only notice the fact that there exist things
> > other than llvm and clang when you first clone (download) llvm.
>
> So, we use that to a certain extent.
>
> Linaro's GCC validation uses the full checkout, then do a shallow
> checkout that only has the updates.
>
> Our LLVM scripts, OTOH, clone all repos and use worktree for *all*
> branches, and we only branch on the repos that we choose, for each
> "working dir".
>
> Our scripts probably would need certain modifications... but it should be
> fine.
>
> But I'm not, by far, the most problematic user.
>
> The real problem, and why people accepted sub-modules, is that a lot
> of downstream people only use one or another projects. Mostly LLVM or
> Clang or libc++.
>
> Checking out all of it is bad,

Define bad?
Time?
Disk space?
Bandwidth?

I mean, we already assume you have a lot of each anyway?

but having them officially interlinked,
> it seems, is worse.

Why?
Below it sounds like you want to do this as a way of enforcing projects to
stay independent of each other.

I would posit that this is not the best way to do this?
> IIUC, the problem is that the projects are now
> built independently on their projects, but more and more CMake changes
> are creeping in, making it harder and harder to separate their
> projects from the rest of LLVM. This means they'll now depend on a
> much larger body of sources that will need to be compiled together,
> and will probably mean they'll abandon LLVM in favour of something
> lighter.
>
> I honestly don't know how big is that problem, I don't have it
myself,
> but I "can imagine" compiling LLVM and Clang without need would
be
> pretty bad.
>
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160720/76a7333a/attachment.html>

On 21 July 2016 at 02:06, Daniel Berlin <dberlin at dberlin.org>
wrote:
>> Checking out all of it is bad,
>
> Define bad?
> Time?
> Disk space?
> Bandwidth?
>
> I mean, we already assume you have a lot of each anyway?
This is not about me, it's about people that use LLVM projects elsewhere.

>> but having them officially interlinked, it seems, is worse.
>
> Why?
> Below it sounds like you want to do this as a way of enforcing projects to
> stay independent of each other.
Why every one take my comments as my own personal motives?

I'm just the "consensus seeker". None of these ideas are mine,
I'm
just echoing what was said in 320 emails, plus what was said in the
past few years when people discussed about using pure Git.

People in the IRC were saying I had ulterior motives, that I was
pushing people to use GitHub or sub-modules, or whatever. This is
*really* not cool.

Every single thread so far has died down and I wrote a summary, and no
one said anything. Then I created another thread, and wrote another
summary. Once no one was disagreeing, I wrote the text.

Now every one wants to disagree again. Seriously?

I *personally* don't care if we use GitHub, or GitLab, Git or
mercurial. I don't care if we have sub-modules or a monolithic
repository, but I'm not the only user.

LLVM has, so far, taken the modular approach that other projects can
embed our projects on their products. Downstream commercial products
do that, other OSS projects do that, and that's pretty cool.

GCC has had a *huge* flying monster in the last decade because they
weren't modular enough and that has been the big difference of LLVM,
and why it gained traction on impossible partners, like Emacs.

If we're saying we want to close everything down and make a compiler
like GCC, that will make my life **MUCH** easier. So there is
absolutely *no* point in me pushing the other way.

But I'm not the only user... And I'd rather not be selfish.

If the consensus has changed from last week, or if no one has actually
read the emails and threads and want to do it all over again, please
be my guest.

cheers,
--renato

On Wed, Jul 20, 2016 at 6:18 PM, Renato Golin <renato.golin at linaro.org>
wrote:
> On 21 July 2016 at 02:06, Daniel Berlin <dberlin at dberlin.org>
wrote:
> >> Checking out all of it is bad,
> >
> > Define bad?
> > Time?
> > Disk space?
> > Bandwidth?
> >
> > I mean, we already assume you have a lot of each anyway?
>
> This is not about me, it's about people that use LLVM projects
elsewhere.
>
>
>
> >> but having them officially interlinked, it seems, is worse.
> >
> > Why?
> > Below it sounds like you want to do this as a way of enforcing
projects
> to
> > stay independent of each other.
>
> Why every one take my comments as my own personal motives
I don't, but i can see how others might
See below.

> I'm just the "consensus seeker". None of these ideas are
mine, I'm
> just echoing what was said in 320 emails, plus what was said in the
> past few years when people discussed about using pure Git.
>
So, if you want to raise the concerns of others, you really need to be a
bit more detailed about who and what.
Otherwise it honestly just comes off as "vague objection".

Even a minimum of "if you look at what X said about Y in the thread",
or
something, would go a long way here.

Otherwise you are basically saying "hey, i think i heard, in the past 300
emails, X".  That's not really something that one can respond to
reasonably.


> People in the IRC were saying I had ulterior motives, that I was
> pushing people to use GitHub or sub-modules, or whatever. This is
> *really* not cool.
>
That is definitely not cool. I don't think you do. I converted GCC from CVS
to SVN, so i know how this feels, believe me :)

>
> Every single thread so far has died down and I wrote a summary, and no
> one said anything. Then I created another thread, and wrote another
> summary. Once no one was disagreeing, I wrote the text.
>
> Now every one wants to disagree again. Seriously?
>

FWIW: I actually think the LLVM community ratholes on a lot of things, *way
too much*. Not sure we are quite at that point yet on this.
>
> I *personally* don't care if we use GitHub, or GitLab, Git or
> mercurial. I don't care if we have sub-modules or a monolithic
> repository, but I'm not the only user.
>
> LLVM has, so far, taken the modular approach that other projects can
> embed our projects on their products. Downstream commercial products
> do that, other OSS projects do that, and that's pretty cool.
>
> GCC has had a *huge* flying monster in the last decade because they
> weren't modular enough and that has been the big difference of LLVM,
> and why it gained traction on impossible partners, like Emacs.
>
Errr, i'm not sure this is really the reason, but let's ignore that :)


>
> If we're saying we want to close everything down and make a compiler
> like GCC, that will make my life **MUCH** easier.

I don't think anyone has said that. I simply pointed out having a
monolithic repo or not should be 100% orthogonal to that.

> So there is
> absolutely *no* point in me pushing the other way.
>
> But I'm not the only user... And I'd rather not be selfish.
>
> If the consensus has changed from last week, or if no one has actually
> read the emails and threads and want to do it all over again, please
> be my guest.
>
I think you may need to move a *little* slower, FWIW. On one hand you are
saying "there are 300+ emails", but you expect consensus in a week?
That seems .. a bit much :)
What if someone was on vacation last week?

I read literally every email in that thread. I guess i don't see all the
concerns being raised you do? I see like one or two emails that could be
taken as concerns.
So like I said, if you are going to seek consensus and drive it by voicing
the concerns of others, that's great. I applaud it. But when doing it, you
may want to make clear that is what you are doing, and who said what, so
that the right people can be cc'd with the right responses, etc

Otherwise i'm not sure it's as helpful as one might think. (i could be
wrong of course)
>
> cheers,
> --renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160720/b8bb49d0/attachment.html>

On Wed, Jul 20, 2016 at 6:18 PM, Renato Golin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Why every one take my comments as my own personal motives?
>
> I'm just the "consensus seeker". None of these ideas are
mine, I'm
> just echoing what was said in 320 emails, plus what was said in the
> past few years when people discussed about using pure Git.
>
> People in the IRC were saying I had ulterior motives, that I was
> pushing people to use GitHub or sub-modules, or whatever. This is
> *really* not cool.
>
> Every single thread so far has died down and I wrote a summary, and no
> one said anything. Then I created another thread, and wrote another
> summary. Once no one was disagreeing, I wrote the text.
>
> Now every one wants to disagree again. Seriously?

I really really sympathize.

I, too, had read the previous N-hundred messages as having mostly
opposition to a single repository solution, and, despite feeling it would
be better, had decided it would not be worth my time to push for it. I can
live with multi-repo, and I certainly want to move to git *more* than I
want to move to single-repo. And so, here we are -- you've developed a
complete proposal for the multi-repo solution, with a high degree of
consensus around it.

And, then, suddenly, in the last day or so, a bunch of support seems to
have shown up for the one-repo solution. Way more than *I'd* ever expected
for sure. Even from people I had thought were super-opposed to it turned
out not to be!

I'm also really sad to hear that people have been impugning your motives,
because you've done a tremendous amount of work to bring this to a
conclusion, and it *really* ought to be clear to everyone that you've been
doing an admirable job of driving towards consensus here, and basically
nothing more.

IMO, the only reason we can even have this conversation about a single-repo
reasonably now is because of your work in writing up clearly the scheme for
a multi-repo solution. So I hope you don't feel discouraged by this turn of
events! I personally put the entire credit of getting to this point on your
hard work.

But, anyways, +1 on a single-repo solution from me.

Before we can agree to merge to a single-repo, there's one further question
that must be resolved:

Should the layout in the merged repository be:
1) Like the "llvm-project" git repository is now:

<root>/llvm/
<root>/clang/
<root>/compiler-rt
...

2) Like the "ideal merged checkout" is now:
llvm/
llvm/tools/clang
llvm/projects/compiler-rt
...


I don't much care which of those is chosen. I have a slight preference for
#1, for ease of doing things like grep/log/etc on llvm by itself, excluding
all the other projects. But either way seems probably fine, and an
improvement over multiple repositories.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160720/8eb05cdd/attachment.html>