llvm dev - Jul 2016 - [RFC] One or many git repositories?

On Thu, Jul 28, 2016 at 4:47 AM, Chris Bieneman via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Beyond all that I want to point out that the git multi-repository story is
> basically the same thing we have today with SVN except for the absence of a
> monotonically increasing number that corresponds across repositories. While
> admittedly you do get a linear history with using the mono-repository, that
> isn’t the only way to solve the problem, and I don’t really think that the
> benefit (not needing to write some tooling) justifies the increased burden
> applied to contributors that don’t use the full LLVM family of projects.
>
What do you believe is this increased burden?

The entire commit history of all llvm projects in a mono-repository is a
449 MB .git directory. It can be downloaded in about two minutes on a
typical domestic internet connection (50 Mbps).

If you download only a snapshot of the current HEAD commit then the .git
repository is 88 MB and takes under a minute. Any other individual commit
should be similar.

This doesn't seem like a big burden to me.

The checked out llvm source directory -- which you say is all that many
people want -- is 202 MB. That's without even building it.

Why is this burden unacceptable? It seems rather small to me.

For comparison, using svn to checkout llvm using ...

svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm

... took me 1 minute 28 seconds and gives a 222 MB .svn directory, 428 MB
total (so 206 MB for the source files checked out).
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160728/97493dae/attachment.html>

On 27 Jul 2016, at 19:03, Bruce Hoult via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> What do you believe is this increased burden?
> 
> The entire commit history of all llvm projects in a mono-repository is a
449 MB .git directory. It can be downloaded in about two minutes on a typical
domestic internet connection (50 Mbps).
> 
> If you download only a snapshot of the current HEAD commit then the .git
repository is 88 MB and takes under a minute. Any other individual commit should
be similar.
> 
> This doesn't seem like a big burden to me.
> 
> The checked out llvm source directory -- which you say is all that many
people want -- is 202 MB. That's without even building it.
> 
> Why is this burden unacceptable? It seems rather small to me.
> 
> For comparison, using svn to checkout llvm using ...
> 
> svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
>  
> ... took me 1 minute 28 seconds and gives a 222 MB .svn directory, 428 MB
total (so 206 MB for the source files checked out).
The burden, as he pointed out, is for people who contribute to llvm-umbrella
projects that are not llvm.

I think that this is a very good reason for not including libc++ in the
monorepo.  Currently, it takes me about 15 seconds to do an checkout of libc++
and the result is about 100MB.  A clone from the git mirror takes just under 6
seconds, downloads 17.15MB, and produces a 72MB directory.  The build directory
is about 7MB.

There is no dependency from libc++ to anything else in LLVM.  In FreeBSD, we use
libc++ with both clang and gcc and people hacking on libc++ do not typically
need to build a new toolchain to do so.  Pulling in all of LLVM, clang, lldb,
lld, and so on just to hack on libc++ seems a bit excessive.

libunwind is similar.  It exposes a public interface that’s ABI compatible with
two other libraries and is tiny.  We currently have people working on it in the
process of replacing the libunwind in FreeBSD with the LLVM version.  Some of
them also hack on other bits of LLVM, but getting the entire LLVM repo just to
hack on a library that’s 1.6MB of source (1.5 seconds to do an svn checkout
currently) seems a bit excessive.

I don’t think it’s a problem for the larger projects (clang, lldb, lld) to be in
a combined repo, because the size of any of them is fairly small in comparison
to the size of LLVM and tiny in comparison to the build directory.

David

David,

While I understand the rationale here, I vote for all or nothing. A middle
ground seems to me to hold none of the advantages and all of the
disadvantages!

James
On Wed, 27 Jul 2016 at 19:30, David Chisnall via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 27 Jul 2016, at 19:03, Bruce Hoult via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > What do you believe is this increased burden?
> >
> > The entire commit history of all llvm projects in a mono-repository is
a
> 449 MB .git directory. It can be downloaded in about two minutes on a
> typical domestic internet connection (50 Mbps).
> >
> > If you download only a snapshot of the current HEAD commit then the
.git
> repository is 88 MB and takes under a minute. Any other individual commit
> should be similar.
> >
> > This doesn't seem like a big burden to me.
> >
> > The checked out llvm source directory -- which you say is all that
many
> people want -- is 202 MB. That's without even building it.
> >
> > Why is this burden unacceptable? It seems rather small to me.
> >
> > For comparison, using svn to checkout llvm using ...
> >
> > svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
> >
> > ... took me 1 minute 28 seconds and gives a 222 MB .svn directory, 428
> MB total (so 206 MB for the source files checked out).
>
> The burden, as he pointed out, is for people who contribute to
> llvm-umbrella projects that are not llvm.
>
> I think that this is a very good reason for not including libc++ in the
> monorepo.  Currently, it takes me about 15 seconds to do an checkout of
> libc++ and the result is about 100MB.  A clone from the git mirror takes
> just under 6 seconds, downloads 17.15MB, and produces a 72MB directory.
> The build directory is about 7MB.
>
> There is no dependency from libc++ to anything else in LLVM.  In FreeBSD,
> we use libc++ with both clang and gcc and people hacking on libc++ do not
> typically need to build a new toolchain to do so.  Pulling in all of LLVM,
> clang, lldb, lld, and so on just to hack on libc++ seems a bit excessive.
>
> libunwind is similar.  It exposes a public interface that’s ABI compatible
> with two other libraries and is tiny.  We currently have people working on
> it in the process of replacing the libunwind in FreeBSD with the LLVM
> version.  Some of them also hack on other bits of LLVM, but getting the
> entire LLVM repo just to hack on a library that’s 1.6MB of source (1.5
> seconds to do an svn checkout currently) seems a bit excessive.
>
> I don’t think it’s a problem for the larger projects (clang, lldb, lld) to
> be in a combined repo, because the size of any of them is fairly small in
> comparison to the size of LLVM and tiny in comparison to the build
> directory.
>
> David
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160727/2f6b7194/attachment.html>

Agree.

Those kinds of projects that don't depend on llvm, but llvm uses and other
projects use too (wthout using llvm) should be treated as different
projects in different repos. It's effectively no different to something
like zlib, other than that it originated from llvm people because they
needed it.


On Thu, Jul 28, 2016 at 6:29 AM, David Chisnall <David.Chisnall at
cl.cam.ac.uk
> wrote:
> On 27 Jul 2016, at 19:03, Bruce Hoult via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > What do you believe is this increased burden?
> >
> > The entire commit history of all llvm projects in a mono-repository is
a
> 449 MB .git directory. It can be downloaded in about two minutes on a
> typical domestic internet connection (50 Mbps).
> >
> > If you download only a snapshot of the current HEAD commit then the
.git
> repository is 88 MB and takes under a minute. Any other individual commit
> should be similar.
> >
> > This doesn't seem like a big burden to me.
> >
> > The checked out llvm source directory -- which you say is all that
many
> people want -- is 202 MB. That's without even building it.
> >
> > Why is this burden unacceptable? It seems rather small to me.
> >
> > For comparison, using svn to checkout llvm using ...
> >
> > svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
> >
> > ... took me 1 minute 28 seconds and gives a 222 MB .svn directory, 428
> MB total (so 206 MB for the source files checked out).
>
> The burden, as he pointed out, is for people who contribute to
> llvm-umbrella projects that are not llvm.
>
> I think that this is a very good reason for not including libc++ in the
> monorepo.  Currently, it takes me about 15 seconds to do an checkout of
> libc++ and the result is about 100MB.  A clone from the git mirror takes
> just under 6 seconds, downloads 17.15MB, and produces a 72MB directory.
> The build directory is about 7MB.
>
> There is no dependency from libc++ to anything else in LLVM.  In FreeBSD,
> we use libc++ with both clang and gcc and people hacking on libc++ do not
> typically need to build a new toolchain to do so.  Pulling in all of LLVM,
> clang, lldb, lld, and so on just to hack on libc++ seems a bit excessive.
>
> libunwind is similar.  It exposes a public interface that’s ABI compatible
> with two other libraries and is tiny.  We currently have people working on
> it in the process of replacing the libunwind in FreeBSD with the LLVM
> version.  Some of them also hack on other bits of LLVM, but getting the
> entire LLVM repo just to hack on a library that’s 1.6MB of source (1.5
> seconds to do an svn checkout currently) seems a bit excessive.
>
> I don’t think it’s a problem for the larger projects (clang, lldb, lld) to
> be in a combined repo, because the size of any of them is fairly small in
> comparison to the size of LLVM and tiny in comparison to the build
> directory.
>
> David
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160728/b2b5213f/attachment.html>