llvm dev - Jul 2016 - [RFC] One or many git repositories?

David,

While I understand the rationale here, I vote for all or nothing. A middle
ground seems to me to hold none of the advantages and all of the
disadvantages!

James
On Wed, 27 Jul 2016 at 19:30, David Chisnall via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 27 Jul 2016, at 19:03, Bruce Hoult via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > What do you believe is this increased burden?
> >
> > The entire commit history of all llvm projects in a mono-repository is
a
> 449 MB .git directory. It can be downloaded in about two minutes on a
> typical domestic internet connection (50 Mbps).
> >
> > If you download only a snapshot of the current HEAD commit then the
.git
> repository is 88 MB and takes under a minute. Any other individual commit
> should be similar.
> >
> > This doesn't seem like a big burden to me.
> >
> > The checked out llvm source directory -- which you say is all that
many
> people want -- is 202 MB. That's without even building it.
> >
> > Why is this burden unacceptable? It seems rather small to me.
> >
> > For comparison, using svn to checkout llvm using ...
> >
> > svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
> >
> > ... took me 1 minute 28 seconds and gives a 222 MB .svn directory, 428
> MB total (so 206 MB for the source files checked out).
>
> The burden, as he pointed out, is for people who contribute to
> llvm-umbrella projects that are not llvm.
>
> I think that this is a very good reason for not including libc++ in the
> monorepo.  Currently, it takes me about 15 seconds to do an checkout of
> libc++ and the result is about 100MB.  A clone from the git mirror takes
> just under 6 seconds, downloads 17.15MB, and produces a 72MB directory.
> The build directory is about 7MB.
>
> There is no dependency from libc++ to anything else in LLVM.  In FreeBSD,
> we use libc++ with both clang and gcc and people hacking on libc++ do not
> typically need to build a new toolchain to do so.  Pulling in all of LLVM,
> clang, lldb, lld, and so on just to hack on libc++ seems a bit excessive.
>
> libunwind is similar.  It exposes a public interface that’s ABI compatible
> with two other libraries and is tiny.  We currently have people working on
> it in the process of replacing the libunwind in FreeBSD with the LLVM
> version.  Some of them also hack on other bits of LLVM, but getting the
> entire LLVM repo just to hack on a library that’s 1.6MB of source (1.5
> seconds to do an svn checkout currently) seems a bit excessive.
>
> I don’t think it’s a problem for the larger projects (clang, lldb, lld) to
> be in a combined repo, because the size of any of them is fairly small in
> comparison to the size of LLVM and tiny in comparison to the build
> directory.
>
> David
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160727/2f6b7194/attachment.html>

Hi James,

I don’t agree that it holds none of the advantages.  LLVM components are,
broadly speaking, in two categories:

- Those that are tightly coupled to others
- Those that are not

Pulling the LLVM Support library out of the LLVM repository, for example, would
not make sense because (even though it can be built stand-alone and doesn’t
depend on the rest of LLVM) it is tightly coupled to other things and someone
who changes it may need to atomically update the other LLVM libraries and
possibly projects such as clang to conform to the new APIs.

Similarly, lldb, lld, and clang are all dependent on unstable APIs in LLVM (and
lldb on unstable APIs in clang).  We therefore need to either retain the ability
to update these consistently.  This gives us three choices:

1) Put LLVM, lldb, lld and clang all into the same repository.

2) Make LLVM a submodule of clang and lld, and clang a submodule of lld so that
any API-breaking change doesn’t get propagated into the projects that depend on
them until the consumers of the APIs have also been updated

3) Provide stronger API stability guarantees and deprecation mechanisms so that
(for example) LLVM can change an API, then clang can adopt it.

Of the three, option 1 is the closest to what we have now and probably the
simplest (and also has other advantages with respect to bisecting and so on).

This does not apply to libc++.  We support building the entire LLVM suite with
other C++ standard library implementations (at least libstdc++, and I think also
with Visual Studio’s implementation), so there is no dependency of anything on
libc++.  Similarly, we support building libc++ with other compilers (in FreeBSD,
we currently build it with gcc 6.1 for RISC-V, for example, where the LLVM
toolchain is not quite useable).

The same applies to libunwind, to an even greater degree (where libc++
implements a standard API, libunwind implements a standard ABI).

David
> On 27 Jul 2016, at 19:32, James Molloy <james at jamesmolloy.co.uk>
wrote:
> 
> David,
> 
> While I understand the rationale here, I vote for all or nothing. A middle
ground seems to me to hold none of the advantages and all of the disadvantages!
> 
> James
> On Wed, 27 Jul 2016 at 19:30, David Chisnall via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> On 27 Jul 2016, at 19:03, Bruce Hoult via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> >
> > What do you believe is this increased burden?
> >
> > The entire commit history of all llvm projects in a mono-repository is
a 449 MB .git directory. It can be downloaded in about two minutes on a typical
domestic internet connection (50 Mbps).
> >
> > If you download only a snapshot of the current HEAD commit then the
.git repository is 88 MB and takes under a minute. Any other individual commit
should be similar.
> >
> > This doesn't seem like a big burden to me.
> >
> > The checked out llvm source directory -- which you say is all that
many people want -- is 202 MB. That's without even building it.
> >
> > Why is this burden unacceptable? It seems rather small to me.
> >
> > For comparison, using svn to checkout llvm using ...
> >
> > svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
> >
> > ... took me 1 minute 28 seconds and gives a 222 MB .svn directory, 428
MB total (so 206 MB for the source files checked out).
> 
> The burden, as he pointed out, is for people who contribute to
llvm-umbrella projects that are not llvm.
> 
> I think that this is a very good reason for not including libc++ in the
monorepo.  Currently, it takes me about 15 seconds to do an checkout of libc++
and the result is about 100MB.  A clone from the git mirror takes just under 6
seconds, downloads 17.15MB, and produces a 72MB directory.  The build directory
is about 7MB.
> 
> There is no dependency from libc++ to anything else in LLVM.  In FreeBSD,
we use libc++ with both clang and gcc and people hacking on libc++ do not
typically need to build a new toolchain to do so.  Pulling in all of LLVM,
clang, lldb, lld, and so on just to hack on libc++ seems a bit excessive.
> 
> libunwind is similar.  It exposes a public interface that’s ABI compatible
with two other libraries and is tiny.  We currently have people working on it in
the process of replacing the libunwind in FreeBSD with the LLVM version.  Some
of them also hack on other bits of LLVM, but getting the entire LLVM repo just
to hack on a library that’s 1.6MB of source (1.5 seconds to do an svn checkout
currently) seems a bit excessive.
> 
> I don’t think it’s a problem for the larger projects (clang, lldb, lld) to
be in a combined repo, because the size of any of them is fairly small in
comparison to the size of LLVM and tiny in comparison to the build directory.
> 
> David
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3719 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160728/5bfbab85/attachment.bin>

On Thu, Jul 28, 2016 at 3:36 PM, David Chisnall via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Hi James,
>
> I don’t agree that it holds none of the advantages.  LLVM components are,
broadly speaking, in two categories:
>
> - Those that are tightly coupled to others
> - Those that are not
>
> Pulling the LLVM Support library out of the LLVM repository, for example,
would not make sense because (even though it can be built stand-alone and
doesn’t depend on the rest of LLVM) it is tightly coupled to other things and
someone who changes it may need to atomically update the other LLVM libraries
and possibly projects such as clang to conform to the new APIs.
>
> Similarly, lldb, lld, and clang are all dependent on unstable APIs in LLVM
(and lldb on unstable APIs in clang).  We therefore need to either retain the
ability to update these consistently.  This gives us three choices:
>
> 1) Put LLVM, lldb, lld and clang all into the same repository.
>
> 2) Make LLVM a submodule of clang and lld, and clang a submodule of lld so
that any API-breaking change doesn’t get propagated into the projects that
depend on them until the consumers of the APIs have also been updated
I don't agree that you need submodules or bundling in order to achieve
this. Lots of projects for decades have worked just fine without doing
either of these. Solaris kernel and libc are pretty tightly dependent
on each other, but existed in separate repo.

Part of how they overcame this was frequent tags which acted as sync points.

binutils and glibc also have dependencies and compatibly requirements,
but they shouldn't be bundled, should they?

I'm strongly opposed to bundling and monolithic design, but it's
likely idiot proof.
--------------------------
One thing to keep in mind - it's easier to bundle than it is to
unbundle. The migration could start with 500 git projects all without
submodules and as people hit real world problems, things can be
incrementally fixed. It seems like everyone is trying to boil the
ocean in advance.

It's somewhat funny and ironic that when it comes to API design and
some compiler internal planning people play kick the can, but when it
comes to something like git vs svn or other things dealing with infra
everyone is an expert and willing to post and post and post to the
same thread..

On 28 Jul 2016 8:36 a.m., "David Chisnall via llvm-dev" <
llvm-dev at lists.llvm.org> wrote:
> This does not apply to libc++.  We support building the entire LLVM suite
with other C++ standard library implementations (at least libstdc++, and I
think also with Visual Studio’s implementation), so there is no dependency
of anything on libc++.  Similarly, we support building libc++ with other
compilers (in FreeBSD, we currently build it with gcc 6.1 for RISC-V, for
example, where the LLVM toolchain is not quite useable).
>
> The same applies to libunwind, to an even greater degree (where libc++
implements a standard API, libunwind implements a standard ABI).

I think the dependencies of lib* in LLVM are more conceptual than version
lock, but they're still there.

I agree with you in all other points, mind you, but RT needs an unwind
library as much as it needs clang. Without them, RT "can" (and indeed
does)
work, but we're not providing a complete solution.

I won't *push* to bundle libunwind, libcxxabi (and ultimately libcxx) on
those merits alone, but my opinion is that we should. I can't see much use
in RT without them. That's why we're still defaulting to libgcc on
Linux.

My tuppence.

Cheers,
Renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160728/e0222547/attachment.html>