llvm dev - Jul 2016 - [RFC] One or many git repositories?

> On 26 Jul 2016, at 10:15, Renato Golin <renato.golin at linaro.org>
wrote:
> 
> On 26 July 2016 at 10:09, Simon Taylor via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> Thus downstream developers can continue to use the read-only view of
the independent projects if that is easier for them; but people hacking on
llvm/clang itself get the benefits of easier checkout, patching, bisection,
atomic commits between projects, etc that come from using a monorepo as the
official repository.
> 
> Would this read-only repositories remain with the synchronous version
> stream? I think this was one of the points against pure-git without
> sub-modules and without monolithic repository.
Which “synchronous version stream” are you referring to?

My understanding is that currently the official repos are in SVN but are
separate for each project.

That situation could of course be recreated exactly with git.

The downside is that some of the projects have cross-dependencies (clang rev x
will only work with llvm rev y) and I don’t believe these cross-repository
dependencies are currently stored anywhere.

If my understanding is mistaken, then apologies, I don’t do any day-to-day work
with LLVM.

git submodules would let you add an umbrella repository that would ensure the
submodules were at mutually compatible versions.

An alternative is to use a monorepo as the ultimate source of truth and the
“official upstream”, which would ensure all projects are mutually compatible,
and make cross-project patches / bisection etc easier.

With a monorepo upstream it would still be possible to maintain read-only views
of parts of the repository (ie individual projects) by projecting commits from
the monorepo.

Say a patch is committed to the monorepo that touches libc++, clang, and llvm.
Those 3 individual read-only repos would then get be updated with the changes in
the commit that affects their files. The commit message would be the same as
from the monorepo, but would have a line added referencing the monorepo commit
(in the same way the git repos currently list the svn rev in their commit
messages). It would also be possible to maintain a read-only umbrella repo that
references the individual ones as submodules; that would also receive a commit
updating the versions of the individual git repos.

These read-only projections of the monorepo would be entirely deterministic -
every commit in the monorepo would generate a matching commit in any project
that it touches (and a commit in the umbrella submodule-based repo too if
desired). The read-only views could be regenerated from scratch from the
monorepo [and optionally a starting state, so you could keep the existing hashes
in the current individual git repo views].

Whether or not to go with a monorepo really depends for me on how intertwined
the modules are, and how often cross-repo commits happen. That’s not something I
know personally, so I won’t make any recommendations either way.


Simon

On 26 July 2016 at 12:08, Simon Taylor <simontaylor1 at ntlworld.com>
wrote:
> My understanding is that currently the official repos are in SVN but are
separate for each project.
> That situation could of course be recreated exactly with git.
> The downside is that some of the projects have cross-dependencies (clang
rev x will only work with llvm rev y) and I don’t believe these cross-repository
dependencies are currently stored anywhere.
> If my understanding is mistaken, then apologies, I don’t do any day-to-day
work with LLVM.
You are. :)

The SVN repos are version-locked, so three commits to llvm, clang and
llvm again would be:

llvm: 123
clang: 124
llvm: 125

Git can't do that, at the moment, without being on the same repo, or
using sub-modules.

> With a monorepo upstream it would still be possible to maintain read-only
views of parts of the repository (ie individual projects) by projecting commits
from the monorepo.
Neither sub-modules nor monolithic would allow a read-only llvm-only
git repository to be synchronised with clang without some external
meta data (ie. going back to the original repo).

Even in GitHub's own SVN view, the "revision" number on each RO
repo
would be internally sequential, but not externally consistent.

cheers,
--renato

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Renato Golin via llvm-dev
> Sent: 26 July 2016 12:22
> To: Simon Taylor
> Cc: llvm-dev at lists.llvm.org; Richard Smith
> Subject: Re: [llvm-dev] [RFC] One or many git repositories?
> 
> On 26 July 2016 at 12:08, Simon Taylor <simontaylor1 at ntlworld.com>
wrote:
> > My understanding is that currently the official repos are in SVN but
are
> separate for each project.
> > That situation could of course be recreated exactly with git.
> > The downside is that some of the projects have cross-dependencies
(clang
> rev x will only work with llvm rev y) and I don’t believe these
cross-repository
> dependencies are currently stored anywhere.
> > If my understanding is mistaken, then apologies, I don’t do any
day-to-day
> work with LLVM.
> 
> You are. :)
> 
> The SVN repos are version-locked, so three commits to llvm, clang and
> llvm again would be:
> 
> llvm: 123
> clang: 124
> llvm: 125
> 
> Git can't do that, at the moment, without being on the same repo, or
> using sub-modules.
I think there's some confusion caused by fuzzy terminology here. If I
understand Simon correctly, he's saying that we have multiple subversion
repositories.  In your reply, you also talk about multiple subversion
repositories. However, there's only one subversion repository in our setup
(called 'llvm-project'). The projects are 'version-locked'
because they are
all housed in the same repository. Our concept of 'project' is just a
convention
within our svn repository in the same way that branches and tags are.
> > With a monorepo upstream it would still be possible to maintain
read-only
> views of parts of the repository (ie individual projects) by projecting
commits
> from the monorepo.
> 
> Neither sub-modules nor monolithic would allow a read-only llvm-only
> git repository to be synchronised with clang without some external
> meta data (ie. going back to the original repo).
> 
> Even in GitHub's own SVN view, the "revision" number on each
RO repo
> would be internally sequential, but not externally consistent.
> 
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev