llvm dev - Jul 2016 - [RFC] One or many git repositories?

On 22 July 2016 at 11:16, Daniel Sanders <Daniel.Sanders at imgtec.com>
wrote:
> There's another reason I've been staying quiet too which is that
past
> experience with VCS migrations has taught me that people tend to over-value
> some things and that discussion tends not to convince people in advance of
> direct experience. I think some of these topics will end up being moot once
> we've moved to git and gotten used to it. For example, I've seen
talk of
> wanting to preserve linear history which is understandable since it's
quite
> nice to have. However, I suspect we'll drop that after a month or so as
> people find 'git push' doesn't work very well on a high traffic
repo and
> start looking for alternatives. At that point I think we'll end up
switching
> to pull requests and accepting non-linear history. Similarly, I think the
> desire for incremental revision numbers will gradually fade as people get
> used to git.
This is valid on a monolithic model, and that is one of the reasons I prefer it.

Today, I personally prefer the Git model (merges, pull requests, fuzzy
history), but I haven't always done so. The more I learnt how to use
Git, the more I realised how valuable the "confusing model" is for
distributed development.

Trying to force Git into an SVN model for the long term feels like
creating a niche that will be hard to work with (no hard evidence,
pinch of salt and all that).

I don't maintain a downstream fork, so I can't speak for that niche.
But forks in GitHub (all, not just LLVM's) seem to be fine merging
their patches over the original repository.

What this feels to me is that we were too complacent with the old
model and were slowly creeping Git support in an SVN world, and now we
realised how unusual is our "requirements".

Maybe you're right. Maybe moving to yet another model that satisfies
those requirements would be a step back, because we'd be setting in
stone a rule that was accommodated, not designed.

Maybe we should propose a third model: Use Git like Git. Pull requests and all.

As a quick recap of the things could go wrong, here's a
back-of-the-envelope idea of what could go wrong...

Changes that are the same as in linear monolithic core with external projects:

* the repositories themselves will have to adapt
* the build system (CMake and all)
* how the non-core repositories interact (relates to build system, bisect)
* all public forks (GitHub and others)
* all downstream forks (Many current LLVM active development affected)

New problems will be created:

* public and downstream forks that *rely* on linear history
* validation (buildbots will have to be re-written, or we'd have to
move to Jenkins, pull-request testing, etc)
* bisection (all our current tools will have to understand Git)
* library dependencies will be hard to bisect, because they won't be
in the same repository with the same history. This happens today in
GNU-land with binutils, glibc, etc.

All in all, not *that* different from the linear monolithic proposal,
and in my opinion, a future facing design, not a past driven
conformance.

cheers,
--renato

> -----Original Message-----
> From: Renato Golin [mailto:renato.golin at linaro.org]
> Sent: 22 July 2016 11:40
> To: Daniel Sanders
> Cc: Pete Cooper; LLVM Developers
> Subject: Re: [llvm-dev] [RFC] One or many git repositories?
> 
> On 22 July 2016 at 11:16, Daniel Sanders <Daniel.Sanders at
imgtec.com>
> wrote:
> > There's another reason I've been staying quiet too which is
that past
> > experience with VCS migrations has taught me that people tend to over-
> value
> > some things and that discussion tends not to convince people in
advance of
> > direct experience. I think some of these topics will end up being moot
once
> > we've moved to git and gotten used to it. For example, I've
seen talk of
> > wanting to preserve linear history which is understandable since
it's quite
> > nice to have. However, I suspect we'll drop that after a month or
so as
> > people find 'git push' doesn't work very well on a high
traffic repo and
> > start looking for alternatives. At that point I think we'll end up
switching
> > to pull requests and accepting non-linear history. Similarly, I think
the
> > desire for incremental revision numbers will gradually fade as people
get
> > used to git.
> 
> This is valid on a monolithic model, and that is one of the reasons I
prefer it.
>
> Today, I personally prefer the Git model (merges, pull requests, fuzzy
> history), but I haven't always done so. The more I learnt how to use
> Git, the more I realised how valuable the "confusing model" is
for
> distributed development.
Same here, accepting the git model was a gradual thing for me. One particular
milestone was realizing that I didn't really need any logic in the commit
id's
because I was only copy/pasting them or using things like 'my-branch^^'.
> Trying to force Git into an SVN model for the long term feels like
> creating a niche that will be hard to work with (no hard evidence,
> pinch of salt and all that).
> 
> I don't maintain a downstream fork, so I can't speak for that
niche.
> But forks in GitHub (all, not just LLVM's) seem to be fine merging
> their patches over the original repository.
> 
> What this feels to me is that we were too complacent with the old
> model and were slowly creeping Git support in an SVN world, and now we
> realised how unusual is our "requirements".
> 
> Maybe you're right. Maybe moving to yet another model that satisfies
> those requirements would be a step back, because we'd be setting in
> stone a rule that was accommodated, not designed.
I don't see it as a step backwards but rather as a way of making people
comfortable with the switch. I think opinions may gradually shift towards
a more conventional git model after the switch but that doesn't necessarily
detract from the value of a more svn-ish model if having one helps people
switch.
> Maybe we should propose a third model: Use Git like Git. Pull requests and
> all.
> 
> As a quick recap of the things could go wrong, here's a
> back-of-the-envelope idea of what could go wrong...
> 
> Changes that are the same as in linear monolithic core with external
projects:
> 
> * the repositories themselves will have to adapt
> * the build system (CMake and all)
> * how the non-core repositories interact (relates to build system, bisect)
> * all public forks (GitHub and others)
> * all downstream forks (Many current LLVM active development affected)
> 
> New problems will be created:
> 
> * public and downstream forks that *rely* on linear history
Do you have an example in mind? I'd expect them to rely on each
'master' being
an improvement on 'master^'. I wouldn't expect them to be interested
in how
'master^' became 'master'.
> * validation (buildbots will have to be re-written, or we'd have to
> move to Jenkins, pull-request testing, etc)
Assuming the goal is to preserve what we have rather than improve it, buildbot
will be fine without any changes (beyond switching the source steps from svn to
git of course) whichever model we pick. It would just check out the latest
'master'
on each build like it currently does for trunk. 
> * bisection (all our current tools will have to understand Git)
> * library dependencies will be hard to bisect, because they won't be
> in the same repository with the same history. This happens today in
> GNU-land with binutils, glibc, etc.
> 
> All in all, not *that* different from the linear monolithic proposal,
> and in my opinion, a future facing design, not a past driven
> conformance.
> 
> cheers,
> --renato

On 22 July 2016 at 13:40, Daniel Sanders <Daniel.Sanders at imgtec.com>
wrote:
> I don't see it as a step backwards but rather as a way of making people
> comfortable with the switch. I think opinions may gradually shift towards
> a more conventional git model after the switch but that doesn't
necessarily
> detract from the value of a more svn-ish model if having one helps people
> switch.
The original idea was to change one thing at a time. SVN to Git, keep
everything else the same.

But that has proven harder than we imagined. So, maybe the best way
forward is not to do one step at a time, but to understand where we
are and what we need and take the "right" (tm) step forwards. Even if
it requires multiple steps, we can combine them into larger, fewer
steps.

>> * public and downstream forks that *rely* on linear history
>
> Do you have an example in mind? I'd expect them to rely on each
'master' being
> an improvement on 'master^'. I wouldn't expect them to be
interested in how
> 'master^' became 'master'.
Paul Robinson was outlining some of the issues he had with git
history. I don't know their setup, so I'll let him describe the issues
(or he may have done so already in some thread, but I haven't read it
all).

> Assuming the goal is to preserve what we have rather than improve it,
buildbot
> will be fine without any changes (beyond switching the source steps from
svn to
> git of course) whichever model we pick. It would just check out the latest
'master'
> on each build like it currently does for trunk.
I meant Zorg and the like. Buildbot itself can handle Git, but we may
have assumptions that the repos are linked and linear in the builders.

But we have been discussing pre-commit testing for a while and it's
clear that Buildbots, in the way they're setup now, are not the
answer.

For the sake of the argument, here is the list of things we found:
 * buildbots can have pre-commit testing via patch submission, but
controlling security and load is not trivial if we want people to
actually use it
 * buildbots tracking non-master branches have the load problems if we
allow people to create branches, but not the security problems
 * having a mirror so that bots track that mirror would solve the
security and load problems, but remove the ability for other people to
use it.

In essence, buildbots are single purpose and hard to configure (much
of it needs master restart).

OTOH, Jenkins can have configurable build scripts, with parameters and
customisations, that allow for us to pick pull requests and build
them, as they come.

It also scales independently, per architecture, from the number of
configurations, if you can use something like containers. So, in the
long term, it's cheaper and more robust to maintain.

However, it's a big change and will require another massive change in
how we do things, and the repository is already big enough.

cheers,
--renato

On Fri, Jul 22, 2016 at 3:40 PM, Daniel Sanders via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Do you have an example in mind? I'd expect them to rely on each
'master'
> being
> an improvement on 'master^'. I wouldn't expect them to be
interested in how
> 'master^' became 'master'.
>
I would love it if each master commit was an improvement on the previous
commit, or at last was virtually guaranteed to be not broken. It's most
annoying that the existing LLVM history has a lot of examples of commits
being reversed by a later commit.

The ease in git of branching -- and more importantly rebasing the branch on
a later state of master -- means that you can run buildbots for all the
different platforms on each pull request BEFORE merging it to master.

If buildbots are not fast enough to test every change (let alone
repeatedly) then you can keep a pristine "master" head and a
"proposed
master" head that might have several pull requests added onto it
sequentially. Then have the buildbots test the "proposed master" and
if it
passes then fast-forward advance the "master" head to the current
"proposed
master" head. Then merge the next batch of pull requests onto
"proposed
master", rinse and repeat.

If a "proposed master" fails and it has more than one pull request in
it,
then you can bisect it to find the bad pull request, throw it out, and try
again without it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160722/f828d23e/attachment.html>

A couple of points that I haven't seen raised yet (I'm mid-vacation so
this is pretty-much a fly-by; sorry if I missed these earlier in the thread).

I haven't thought about this as deeply as the rest of you.  Maybe these are
easily refuted?

1. If there's a move toward a monolithic repository, it's important to
remember this is the LLVM project, not the Clang project.  A nested layout that
optimizes for Clang developers at the expense of everyone else's workflow
would be a disservice to the greater community, even if it's temporary
"while we figure out what really makes sense" kind of state.  For that
reason, I'm against a monolithic layout that has "clang" living at
"tools/clang"... or, really, having any sub-project live inside of the
"llvm" directory.

2. Those working on projects *outside* the monolithic repo will get the
downsides of both: a monolithic repo that they are only using parts of, and
multiple repos that are somehow version-locked.

3. For many (most?) developers, changing to a monolithic git repo is a *bigger*
workflow change than switching to separate git repos.  Many people (and at least
some downstream infrastructure) use the git mirrors exclusively, aside from
git-svn for committing.

#1 and #2 don't negatively impact Clang developers really -- and we have the
loudest voices -- but we should be intentional about any changes here.

I mention #3 specifically to address Richard's claim that a monolithic,
nested, git repo is a smaller change than separate git repos.  On the contrary,
with separate git repos, I just need to update a couple of remotes and I'm
finished.
- If "minimize incremental change" is important, we should start with
separate git repos (since only SVN users need to change their workflow).
- If "minimize number of changes" is important, we should figure out a
close approximation of the end goal and move directly there.

Two specific replies below.
> On 2016-Jul-25, at 12:54, Justin Lebar via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi, all.
> 
> I feel like we've strayed pretty far from the question originally
> posed in this thread.
> 
> One of the pieces of feedback I got before I started this thread was
> that many people felt that, the last time the question of multiple
> repos vs. monorepo was discussed, it was interspersed with other
> topics, making it difficult for some people to weigh in appropriately
> (or even to be aware that the discussion was occurring).  I'm afraid
> that the discussion of github workflows we're having here may cause
> the same problem.
> 
> Maybe we can move the discussion about github workflows into a
> different thread?  Again, I don't mean to stop it, just move it.
> 
> To re-focus this thread on its original topic: It sounds to me like,
> broadly speaking, we have consensus on using a single repository.
I'm not convinced.  I'd be interested in hearing via the survey which
path (separate repos vs. monolithic) causes the most workflow disruption.
> But
> there are still some outstanding related questions.  Among these are:
> 
> 1) Should the repository have "unified history"?  (Meaning,
should I
> be able to check out a single git revision from before the migration
> and have it contain all of the llvm subprojects?)
> 
> 2) Should the monorepo have a "nested" repository layout (e.g.
clang
> goes in /tools/clang) or a "flat" layout (clang goes in /clang)?
> 
> 3) Assuming we want unified history, should the new canonical
> repository's hashes be based on
> https://github.com/llvm-project/llvm-project, or should it start
> afresh?
> 
> FWIW my answers to these are:
> 
> 1) Yes to unified history.  The main advantage of non-unified history
> is that it's easier for people to import old branches -- it's a
matter
> of "git merge" instead of running the git filter-branch script I
> wrote.  But this is a relatively small (~20 minute) one-time cost to
> some of us, whereas our repository history is born by all of us
> forever.  Moreover unified history also helps people with long-running
> branches, as it lets them check out old versions of their branch and
> get a compatible version of all of the other llvm subprojects.
> 
> 2) Yes to nested layout.  I find Chandler and Richard Smith's
> arguments compelling.
I disagree with having "clang" nested inside "llvm".
> 3) No to basing the new canonical repo on
> https://github.com/llvm-project/llvm-project.  That repo's history is
> missing svn revision numbers, and there are enough emails floating
> around that reference svn revision numbers that I think we need them
> in our canonical repo.  Also llvm-project/llvm-project has a flat
> structure, and if we end up going with a nested layout, it would be
> better to have that layout starting with the first commit.
> 
> -Justin
> 
> On Mon, Jul 25, 2016 at 8:10 AM, Bruce Hoult via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> git-imerge can run an arbitrary script to decide whether a commit is
good or
>> bad. Lack of textual merge conflicts is only the most basic test -- you
can
>> check that it compiles, run tests .. whatever you want and have time to
>> execute.
>> 
>> On Tue, Jul 26, 2016 at 2:12 AM, Robinson, Paul via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>>> 
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: Renato Golin [mailto:renato.golin at linaro.org]
>>>> Sent: Monday, July 25, 2016 7:11 AM
>>>> To: Daniel Sanders
>>>> Cc: Robinson, Paul; llvm-dev at lists.llvm.org
>>>> Subject: Re: [llvm-dev] [RFC] One or many git repositories?
>>>> 
>>>> On 25 July 2016 at 14:55, Daniel Sanders <Daniel.Sanders at
imgtec.com>
>>>> wrote:
>>>>> I know of a way but it's not very nice. The gist of it
is to checkout
>>>> the
>>>>> downstream branch just before the bad merge and then merge
the first
>>>>> 100 commits from upstream. If the result is good then merge
the next
>>>>> 100, but if it's bad then 'git reset --hard'
and merge 10 instead.
>>>> You'll
>>>>> eventually find the commit that made it bad. Essentially,
the idea is
>>>>> to
>>>>> make a throwaway branch that merges more frequently. I do
something
>>>>> similar to rebase my work to master since gradually
rebasing often
>>>>> causes all the conflicts to go away.
>>>> 
>>>> This is essentially what git-imerge does, you only need to
define
>>>> "good merge" in the form of a script or CI job.
>>>> 
>>>> cheers,
>>>> -renato
>>> 
>>> Except I understood git-imerge to be looking for physical
conflicts,
>>> not "when did this test start failing."  If it does the
latter also,
>>> that would be awesome.
>>> --paulr
>>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> 
>> 
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

> 2. Those working on projects *outside* the monolithic repo will get the
downsides of both: a monolithic repo that they are only using parts of, and
multiple repos that are somehow version-locked.
We've addressed the first downside -- that you have to download a
bigger repo -- extensively above.  The gist is, it's minimal.  The
additional disk space is very small compared to e.g. the size of an
llvm objdir.  If you don't want to look at the files from the projects
you don't hack on, you can hide them using git sparse checkouts.  And
obviously we're not going to make everybody build every project --
we'll change the build system so it doesn't use "directories
present
in tree" to determine which projects to build by default.

With respect to the second part, I agree that the monolithic
repository wouldn't be a big help for people who are consuming llvm
externally, and further that the extra bits consumed by our new
repository would have a nonzero cost.  But the downsides are so small
I think it would be a serious mistake to weigh this heavily.
> I'd be interested in hearing via the survey which path (separate repos
vs. monolithic) causes the most workflow disruption.
As phrased this is begging the question.

The question is, what choice is best?  One dimension of "best"
certainly is "minimizes workflow disruption."  But that's not the
only
one, nor even (necessarily) the most important one.  Certainly we
don't have to send out a survey to conclude that the non-monolithic
repository would change workflows the least.  :)

If we really are going to use a survey as an important signal in the
decision here, I would want us to be thoughtful about how we design
it, so that we have a chance of getting informed -- rather than gut --
opinions.  At the very least I'd want every option to be accompanied
by pros/cons assembled by people who advocate for the position.

But personally I'd much prefer if we could just engage here as much as
possible.  We have changed a number of peoples' minds, and that's just
not possible with a survey.

On Mon, Jul 25, 2016 at 3:04 PM, Duncan P. N. Exon Smith
<dexonsmith at apple.com> wrote:
> A couple of points that I haven't seen raised yet (I'm mid-vacation
so this is pretty-much a fly-by; sorry if I missed these earlier in the thread).
>
> I haven't thought about this as deeply as the rest of you.  Maybe these
are easily refuted?
>
> 1. If there's a move toward a monolithic repository, it's important
to remember this is the LLVM project, not the Clang project.  A nested layout
that optimizes for Clang developers at the expense of everyone else's
workflow would be a disservice to the greater community, even if it's
temporary "while we figure out what really makes sense" kind of state.
For that reason, I'm against a monolithic layout that has "clang"
living at "tools/clang"... or, really, having any sub-project live
inside of the "llvm" directory.
>
> 2. Those working on projects *outside* the monolithic repo will get the
downsides of both: a monolithic repo that they are only using parts of, and
multiple repos that are somehow version-locked.
>
> 3. For many (most?) developers, changing to a monolithic git repo is a
*bigger* workflow change than switching to separate git repos.  Many people (and
at least some downstream infrastructure) use the git mirrors exclusively, aside
from git-svn for committing.
>
> #1 and #2 don't negatively impact Clang developers really -- and we
have the loudest voices -- but we should be intentional about any changes here.
>
> I mention #3 specifically to address Richard's claim that a monolithic,
nested, git repo is a smaller change than separate git repos.  On the contrary,
with separate git repos, I just need to update a couple of remotes and I'm
finished.
> - If "minimize incremental change" is important, we should start
with separate git repos (since only SVN users need to change their workflow).
> - If "minimize number of changes" is important, we should figure
out a close approximation of the end goal and move directly there.
>
> Two specific replies below.
>
>> On 2016-Jul-25, at 12:54, Justin Lebar via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>>
>> Hi, all.
>>
>> I feel like we've strayed pretty far from the question originally
>> posed in this thread.
>>
>> One of the pieces of feedback I got before I started this thread was
>> that many people felt that, the last time the question of multiple
>> repos vs. monorepo was discussed, it was interspersed with other
>> topics, making it difficult for some people to weigh in appropriately
>> (or even to be aware that the discussion was occurring).  I'm
afraid
>> that the discussion of github workflows we're having here may cause
>> the same problem.
>>
>> Maybe we can move the discussion about github workflows into a
>> different thread?  Again, I don't mean to stop it, just move it.
>>
>> To re-focus this thread on its original topic: It sounds to me like,
>> broadly speaking, we have consensus on using a single repository.
>
> I'm not convinced.  I'd be interested in hearing via the survey
which path (separate repos vs. monolithic) causes the most workflow disruption.
>
>> But
>> there are still some outstanding related questions.  Among these are:
>>
>> 1) Should the repository have "unified history"?  (Meaning,
should I
>> be able to check out a single git revision from before the migration
>> and have it contain all of the llvm subprojects?)
>>
>> 2) Should the monorepo have a "nested" repository layout
(e.g. clang
>> goes in /tools/clang) or a "flat" layout (clang goes in
/clang)?
>>
>> 3) Assuming we want unified history, should the new canonical
>> repository's hashes be based on
>> https://github.com/llvm-project/llvm-project, or should it start
>> afresh?
>>
>> FWIW my answers to these are:
>>
>> 1) Yes to unified history.  The main advantage of non-unified history
>> is that it's easier for people to import old branches -- it's a
matter
>> of "git merge" instead of running the git filter-branch
script I
>> wrote.  But this is a relatively small (~20 minute) one-time cost to
>> some of us, whereas our repository history is born by all of us
>> forever.  Moreover unified history also helps people with long-running
>> branches, as it lets them check out old versions of their branch and
>> get a compatible version of all of the other llvm subprojects.
>>
>> 2) Yes to nested layout.  I find Chandler and Richard Smith's
>> arguments compelling.
>
> I disagree with having "clang" nested inside "llvm".
>
>> 3) No to basing the new canonical repo on
>> https://github.com/llvm-project/llvm-project.  That repo's history
is
>> missing svn revision numbers, and there are enough emails floating
>> around that reference svn revision numbers that I think we need them
>> in our canonical repo.  Also llvm-project/llvm-project has a flat
>> structure, and if we end up going with a nested layout, it would be
>> better to have that layout starting with the first commit.
>>
>> -Justin
>>
>> On Mon, Jul 25, 2016 at 8:10 AM, Bruce Hoult via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>>> git-imerge can run an arbitrary script to decide whether a commit
is good or
>>> bad. Lack of textual merge conflicts is only the most basic test --
you can
>>> check that it compiles, run tests .. whatever you want and have
time to
>>> execute.
>>>
>>> On Tue, Jul 26, 2016 at 2:12 AM, Robinson, Paul via llvm-dev
>>> <llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Renato Golin [mailto:renato.golin at linaro.org]
>>>>> Sent: Monday, July 25, 2016 7:11 AM
>>>>> To: Daniel Sanders
>>>>> Cc: Robinson, Paul; llvm-dev at lists.llvm.org
>>>>> Subject: Re: [llvm-dev] [RFC] One or many git repositories?
>>>>>
>>>>> On 25 July 2016 at 14:55, Daniel Sanders <Daniel.Sanders
at imgtec.com>
>>>>> wrote:
>>>>>> I know of a way but it's not very nice. The gist of
it is to checkout
>>>>> the
>>>>>> downstream branch just before the bad merge and then
merge the first
>>>>>> 100 commits from upstream. If the result is good then
merge the next
>>>>>> 100, but if it's bad then 'git reset
--hard' and merge 10 instead.
>>>>> You'll
>>>>>> eventually find the commit that made it bad.
Essentially, the idea is
>>>>>> to
>>>>>> make a throwaway branch that merges more frequently. I
do something
>>>>>> similar to rebase my work to master since gradually
rebasing often
>>>>>> causes all the conflicts to go away.
>>>>>
>>>>> This is essentially what git-imerge does, you only need to
define
>>>>> "good merge" in the form of a script or CI job.
>>>>>
>>>>> cheers,
>>>>> -renato
>>>>
>>>> Except I understood git-imerge to be looking for physical
conflicts,
>>>> not "when did this test start failing."  If it does
the latter also,
>>>> that would be awesome.
>>>> --paulr
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

> On 2016-Jul-25, at 20:31, Justin Lebar <jlebar at google.com> wrote:
> 
>> I'd be interested in hearing via the survey which path (separate
repos vs. monolithic) causes the most workflow disruption.
> 
> As phrased this is begging the question.
> 
> The question is, what choice is best?  One dimension of "best"
> certainly is "minimizes workflow disruption."  But that's not
the only
> one, nor even (necessarily) the most important one.  Certainly we
> don't have to send out a survey to conclude that the non-monolithic
> repository would change workflows the least.  :)
There are two different questions.  First, what's the end goal?  Second, how
should we get there?  Workflow disruption should be a key consideration for the
latter.

There seemed to be an assumption in Richard's post (perhaps, a reading he
didn't intend) that a nested, monolithic git repo was a small, incremental
step; a stepping stone toward a flatter structure.  I disagree.

I don't feel strongly about the directory structure; I'll leave it to
those who do to argue that point.

But can we separate out the idea of a monolithic repository from the
particulars of its directory structure?  It sounds from your latest
e-mail that what you're actually opposed to is the nested directory
structure in a monolithic repo, not the monorepo itself.  In which
case, we have more in common than not.

On Mon, Jul 25, 2016 at 8:49 PM, Duncan P. N. Exon Smith
<dexonsmith at apple.com> wrote:
>
>> On 2016-Jul-25, at 20:31, Justin Lebar <jlebar at google.com>
wrote:
>>
>>> I'd be interested in hearing via the survey which path
(separate repos vs. monolithic) causes the most workflow disruption.
>>
>> As phrased this is begging the question.
>>
>> The question is, what choice is best?  One dimension of
"best"
>> certainly is "minimizes workflow disruption."  But that's
not the only
>> one, nor even (necessarily) the most important one.  Certainly we
>> don't have to send out a survey to conclude that the non-monolithic
>> repository would change workflows the least.  :)
>
> There are two different questions.  First, what's the end goal? 
Second, how should we get there?  Workflow disruption should be a key
consideration for the latter.
>
> There seemed to be an assumption in Richard's post (perhaps, a reading
he didn't intend) that a nested, monolithic git repo was a small,
incremental step; a stepping stone toward a flatter structure.  I disagree.

Hi Duncan,
>  […]
> 2. Those working on projects *outside* the monolithic repo will get the
downsides of both: a monolithic repo that they are only using parts of, and
multiple repos that are somehow version-locked.
> 
> 3. For many (most?) developers, changing to a monolithic git repo is a
*bigger* workflow change than switching to separate git repos. Many people (and
at least some downstream infrastructure) use the git mirrors exclusively, aside
from git-svn for committing.
I believe the idea is to continue to maintain the read-only independent git
repos for each project. The only change is instead of sourcing those commits
from the official upstream(independent) svn repos, they will be sourced from the
official upstream monorepo.

Thus downstream developers can continue to use the read-only view of the
independent projects if that is easier for them; but people hacking on
llvm/clang itself get the benefits of easier checkout, patching, bisection,
atomic commits between projects, etc that come from using a monorepo as the
official repository.

Simon