llvm dev - Jul 2016 - [RFC] One or many git repositories?

> Today I *can* checkout only LLVM and Clang. On a single Git repo I
can't.
This is true if you s/checkout/clone/.  With a single repo, you must
clone (download) everything (*), but after you've done so you can use
sparse checkouts to check out (create a working copy of) only llvm and
clang.  So you should only notice the fact that there exist things
other than llvm and clang when you first clone (download) llvm.

Either way switching to git is going to be a change from the status
quo.  Personally I'm more interested in finding the best overall
solution than the solution which is "most similar" to the current
setup under some metric.

(*) Technically, if you do a shallow clone, you have to download a
single revision of everything.  That's the 90mb number from my
original post.

On Wed, Jul 20, 2016 at 5:25 PM, Renato Golin via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> On 21 July 2016 at 01:23, Sanjoy Das via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> Right now we *are* in a monorepo, with sequential revision numbers
>> across llvm and clang, so I'd say trying to move to separate repos
is
>> actually the "change" here.  :)
>
> Not true. SVN can be checked out by directory, Git needs to be cloned
> on the root.
>
> Today I *can* checkout only LLVM and Clang. On a single Git repo I
can't.
>
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On 21 July 2016 at 01:39, Justin Lebar <jlebar at google.com>
wrote:
> This is true if you s/checkout/clone/.  With a single repo, you must
> clone (download) everything (*), but after you've done so you can use
> sparse checkouts to check out (create a working copy of) only llvm and
> clang.  So you should only notice the fact that there exist things
> other than llvm and clang when you first clone (download) llvm.
So, we use that to a certain extent.

Linaro's GCC validation uses the full checkout, then do a shallow
checkout that only has the updates.

Our LLVM scripts, OTOH, clone all repos and use worktree for *all*
branches, and we only branch on the repos that we choose, for each
"working dir".

Our scripts probably would need certain modifications... but it should be fine.

But I'm not, by far, the most problematic user.

The real problem, and why people accepted sub-modules, is that a lot
of downstream people only use one or another projects. Mostly LLVM or
Clang or libc++.

Checking out all of it is bad, but having them officially interlinked,
it seems, is worse. IIUC, the problem is that the projects are now
built independently on their projects, but more and more CMake changes
are creeping in, making it harder and harder to separate their
projects from the rest of LLVM. This means they'll now depend on a
much larger body of sources that will need to be compiled together,
and will probably mean they'll abandon LLVM in favour of something
lighter.

I honestly don't know how big is that problem, I don't have it myself,
but I "can imagine" compiling LLVM and Clang without need would be
pretty bad.

cheers,
--renato

> On Jul 20, 2016, at 5:56 PM, Renato Golin via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> On 21 July 2016 at 01:39, Justin Lebar <jlebar at google.com> wrote:
>> This is true if you s/checkout/clone/.  With a single repo, you must
>> clone (download) everything (*), but after you've done so you can
use
>> sparse checkouts to check out (create a working copy of) only llvm and
>> clang.  So you should only notice the fact that there exist things
>> other than llvm and clang when you first clone (download) llvm.
> 
> So, we use that to a certain extent.
> 
> Linaro's GCC validation uses the full checkout, then do a shallow
> checkout that only has the updates.
> 
> Our LLVM scripts, OTOH, clone all repos and use worktree for *all*
> branches, and we only branch on the repos that we choose, for each
> "working dir".
> 
> Our scripts probably would need certain modifications... but it should be
fine.
> 
> But I'm not, by far, the most problematic user.
> 
> The real problem, and why people accepted sub-modules, is that a lot
> of downstream people only use one or another projects. Mostly LLVM or
> Clang or libc++.
> 
> Checking out all of it is bad, but having them officially interlinked,
> it seems, is worse. IIUC, the problem is that the projects are now
> built independently on their projects, but more and more CMake changes
> are creeping in, making it harder and harder to separate their
> projects from the rest of LLVM. This means they'll now depend on a
> much larger body of sources that will need to be compiled together,
> and will probably mean they'll abandon LLVM in favour of something
> lighter.
> 
> I honestly don't know how big is that problem, I don't have it
myself,
> but I "can imagine" compiling LLVM and Clang without need would
be
> pretty bad.

You seem to imply that all the projects in the single repo would be built by
default, while it is not part of the proposal.
Actually I’d expect an opt-in mechanism, so that: `mkdir build-llvm &&
cd build-llvm && cmake ../llvm` only builds LLVM.

— 
Mehdi

On Wed, Jul 20, 2016 at 5:56 PM, Renato Golin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 21 July 2016 at 01:39, Justin Lebar <jlebar at google.com> wrote:
> > This is true if you s/checkout/clone/.  With a single repo, you must
> > clone (download) everything (*), but after you've done so you can
use
> > sparse checkouts to check out (create a working copy of) only llvm and
> > clang.  So you should only notice the fact that there exist things
> > other than llvm and clang when you first clone (download) llvm.
>
> So, we use that to a certain extent.
>
> Linaro's GCC validation uses the full checkout, then do a shallow
> checkout that only has the updates.
>
> Our LLVM scripts, OTOH, clone all repos and use worktree for *all*
> branches, and we only branch on the repos that we choose, for each
> "working dir".
>
> Our scripts probably would need certain modifications... but it should be
> fine.
>
> But I'm not, by far, the most problematic user.
>
> The real problem, and why people accepted sub-modules, is that a lot
> of downstream people only use one or another projects. Mostly LLVM or
> Clang or libc++.
>
> Checking out all of it is bad,

Define bad?
Time?
Disk space?
Bandwidth?

I mean, we already assume you have a lot of each anyway?

but having them officially interlinked,
> it seems, is worse.

Why?
Below it sounds like you want to do this as a way of enforcing projects to
stay independent of each other.

I would posit that this is not the best way to do this?
> IIUC, the problem is that the projects are now
> built independently on their projects, but more and more CMake changes
> are creeping in, making it harder and harder to separate their
> projects from the rest of LLVM. This means they'll now depend on a
> much larger body of sources that will need to be compiled together,
> and will probably mean they'll abandon LLVM in favour of something
> lighter.
>
> I honestly don't know how big is that problem, I don't have it
myself,
> but I "can imagine" compiling LLVM and Clang without need would
be
> pretty bad.
>
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160720/76a7333a/attachment.html>