llvm dev - Jul 2016 - [RFC] One or many git repositories?

One other point about maintaining branches:

With the single repository approach, maintaining a long-running branch
that touches multiple subprojects (e.g. llvm and clang) becomes *far*
simpler.

With the umbrella repo, you have to do the submodules trickery I
described in the original e-mail.  It is complicated, and takes a lot
of typing (or requires you to develop custom scripts).  But with the
single repo, this cross-cutting branch is just a branch.

In fact even if your branch isn't cross-cutting, if it's not a branch
of LLVM proper, I'm curious how you'd do things like bisect the
branch, or even just check out and build an old version.  You check
out an old version of the (say) clang branch, and then presumably you
try to figure out the corresponding version in the LLVM repo that you
need to check out.  I guess you could find the upstream parent of your
branch, get the SVN revision number from the commit message, then go
to the LLVM branch and find a commit which has an SVN number that's
nearby?

This would all become as simple as "git checkout" under the monolithic
model.

On Thu, Jul 21, 2016 at 9:16 AM, Justin Lebar <jlebar at google.com>
wrote:
>> Our clones of LLVM and clang have a reasonable amount of history (a
couple of hundred commits, I believe), including multiple branches, that we’d
want to preserve.  Both branches have merged from upstream multiple times.  It’s
one of the smaller friendly forks that I know about.  I’ve not used git
filter-branch before, but I’d be very impressed if there is some simple
invocation that can can move from this model.
>
> James and I owe you something here.  I think this can be handled in a
> straightforward manner, but I am not 100% sure how at the moment.  I
> agree this is very important.
>
> Our demo would be much more compelling if we can use an existing
> branch.  Does anyone know of one we can play with?
>
>> In particular, the fact that we have a third more public GitHub forks
of LLVM than of clang, and eight times as many as of lldb implies to me that
forcing everyone downstream to pull in all subprojects would not be particularly
well received.
>
> I have a hard time understanding this particular argument.  Per the
> original e-mail, with three shell commands, you can hide whichever
> llvm subprojects you want.  After doing that, the only overhead of the
> subprojects is extra space in your .git directory, which would still
> be much smaller than an llvm+clang objdir.
>
> Is there something specific that you think will not be well-received?
> Or maybe it's better to speak personally -- is there something
> specific that will bother you personally about having to clone (but
> not check out) everything?
>
> On Thu, Jul 21, 2016 at 8:22 AM, David Chisnall via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> On 21 Jul 2016, at 16:11, James Y Knight <jyknight at google.com>
wrote:
>>>
>>> I don't think this is a really hard problem though -- I can
think of a few ways to help existing users that probably will work (although
I'd want to try them first, to ensure it actually does work, of course). The
two I'm thinking of are just doing "git diff" followed by
"git apply --directory=llvm" if you just want to save a patch. Or,
some "git filter-branch" invocation to rename all the files in your
existing repo, followed by "git rebase" (or "git merge"), if
you have some more history you want to maintain.
>>
>> Our clones of LLVM and clang have a reasonable amount of history (a
couple of hundred commits, I believe), including multiple branches, that we’d
want to preserve.  Both branches have merged from upstream multiple times.  It’s
one of the smaller friendly forks that I know about.  I’ve not used git
filter-branch before, but I’d be very impressed if there is some simple
invocation that can can move from this model.
>>
>> I was in favour of the GitHub migration primarily because a lot of
downstream LLVM users already have a workflow based around GitHub that works
well and the proposal was to make this closer to the official workflow.  I’m
very nervous about a last-minute change to require everyone downstream to
restructure their workflows.
>>
>> In particular, the fact that we have a third more public GitHub forks
of LLVM than of clang, and eight times as many as of lldb implies to me that
forcing everyone downstream to pull in all subprojects would not be particularly
well received.
>>
>> David
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>

Question:

Which projects do we put under this monolithic repository?

SVN has about 42 projects, some of them dead, some of them in life support.

So far, being "an upstream repository" meant being inside the LLVM SVN
server. We'll change that to "being inside the monolithic LLVM
repository". But this can become huge, and not all projects "ink"
back
to LLVM.

An alternative would be to just have some core projects in the
monolithic and everything else as separate, but then what's core?

As a back-of-the-envelope, I suggest: llvm, clang, clang-tools-extra,
compiler-rt, libc++, libc++abi, libunwind, test-suite.

I'm thinking LLD and LLDB could remain out, but I don't think it would
be too weird for them to be in...

Anything else? Less?

cheers,
--renato

Monolithic is trying to solve the wrong problem - it's that simple.
Any discussion or attempt to coddle those who think it's necessary is
a waste of time. #dictator

As part of any potential migration, everyone involved must start to
accept certain changes, (large or small) to the workflow. The big
challenge here isn't technical, it's mindset. It's convincing any
group of people who object that it won't be as painful to them as they
think. (I hope this is a true statement)

#if - there's a group of people are : dogmatic, stubborn and
unreasonable - others outside that group should decide how to deal
with them: ignore, coddle, placate or other.

I don't think there's a perfect technical solution to make everyone
happy - I think focusing on the social engineering will be an equal or
greater importance. (herding cats)

With the survey - I guess you could include some level of objection
like - strongly against and over my dead body type reactions are
probably the most to be cautious about. Anyone surveyed who fall in
the middle or slightly left/right can be seen as "flexible". If it
turns out that they survey shows only 1-5 people with extreme views
and 100 people with moderate or flexible views - those are hard
numbers. From there decisions can be made and long unending threads
like this can die - so we can all get back to reading more important
things.



On Fri, Jul 22, 2016 at 12:49 AM, Renato Golin via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Question:
>
> Which projects do we put under this monolithic repository?
>
> SVN has about 42 projects, some of them dead, some of them in life support.
>
> So far, being "an upstream repository" meant being inside the
LLVM SVN
> server. We'll change that to "being inside the monolithic LLVM
> repository". But this can become huge, and not all projects
"ink" back
> to LLVM.
>
> An alternative would be to just have some core projects in the
> monolithic and everything else as separate, but then what's core?
>
> As a back-of-the-envelope, I suggest: llvm, clang, clang-tools-extra,
> compiler-rt, libc++, libc++abi, libunwind, test-suite.
>
> I'm thinking LLD and LLDB could remain out, but I don't think it
would
> be too weird for them to be in...
>
> Anything else? Less?
>
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On 21 Jul 2016, at 17:49, Renato Golin <renato.golin at linaro.org>
wrote:
> 
> As a back-of-the-envelope, I suggest: llvm, clang, clang-tools-extra,
> compiler-rt, libc++, libc++abi, libunwind, test-suite.
The minimum that makes sense is llvm, though that defeats the point of a
combined repo.

I don’t think that libc++ / libc++abi make sense there for several reasons:

- You very rarely need to update them in lockstep with anything else

- LLVM/Clang is useful and frequently built without libc++

- libc++ is useful and frequently built without any of the rest of LLVM

The same applies to libunwind.  If you’re building an entire toolchain then you
might want to use it, but most projects don’t benefit from it and it implements
a well-defined standard ABI and so doesn’t need to be updated in lockstep with
anything else.

clang-tools-extra is explicitly a bunch of stuff that doesn’t belong in the main
clang repo because it’s not of interest to most people doing clang work, so it’s
hard to see why it would be of interest to everyone doing LLVM work. 
Additionally, I believe that they’re mostly things that are built on top of APIs
in clang that are supposed to be moderately stable, so shouldn’t need atomically
updating with respect to clang very often.

Compiler-rt probably makes sense if clang is there, as it includes a lot of the
run-time support for clang.

David

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3719 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160721/ac9a4787/attachment.bin>

> Which projects do we put under this monolithic repository?
The proposal at the moment is to include

  llvm, clang, clang-tools-extra, lld, polly, lldb, llgo, compiler-rt,
openmp, and parallel-libs.

This is the set {llvm} plus the transitive closure of "projects that
are version-locked to a project in the set", where the closure is
taken over the set of all active LLVM subprojects.

Projects that don't depend on a specific version of llvm or some other
subproject -- test-suite and libc++ -- are not included.  Everything
else is, because the whole idea is to have one repository that
captures the implicit versioning dependencies between (say) lldb and
llvm.

As soon as we have one version-locked subproject that's not in the
monolithic repo, we now we have to maintain an umbrella repo that
tells you which version of llvm corresponds to which version of the
version-locked-but-not-in-monolithic repo.

The cost of including additional projects in the monolithic repository
is very low, since you can ignore them using sparse checkouts.

On Thu, Jul 21, 2016 at 9:49 AM, Renato Golin <renato.golin at linaro.org>
wrote:
> Question:
>
> Which projects do we put under this monolithic repository?
>
> SVN has about 42 projects, some of them dead, some of them in life support.
>
> So far, being "an upstream repository" meant being inside the
LLVM SVN
> server. We'll change that to "being inside the monolithic LLVM
> repository". But this can become huge, and not all projects
"ink" back
> to LLVM.
>
> An alternative would be to just have some core projects in the
> monolithic and everything else as separate, but then what's core?
>
> As a back-of-the-envelope, I suggest: llvm, clang, clang-tools-extra,
> compiler-rt, libc++, libc++abi, libunwind, test-suite.
>
> I'm thinking LLD and LLDB could remain out, but I don't think it
would
> be too weird for them to be in...
>
> Anything else? Less?
>
> cheers,
> --renato