llvm dev - Jul 2016 - [RFC] One or many git repositories?

On 22 July 2016 at 13:48, Piotr Padlewski via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>> The build system can help, you just need to have two (sparse) checkout:
one for LLVM/clang and the other for clang-tidy, and configure the build with
the LLVM/clang checkout adding the clang-tidy as external.
> Can you describe it more?
Something like this

$ git clone git at github.com:llvm/llvm.git
$ git clone git at github.com:llvm/llvm.git clang-tools-extras
$ <make the clang-tools sparse if you want, doesn't seem strictly
necessary though>
$ mkdir build && cd build
$ cmake ../llvm
-DLLVM_EXTERNAL_CLANG_TOOLS_EXTRA_SOURCE_DIR=../clang-tools-extras/tools/clang/tools/clang-tools-extras
$ make
> I don't get the approach, but it seems we are trying to make it easier
to use llvm, but in the same time we are making it harder.
As Justin said, this isn't an issue for the majority of developers and
it's a solvable problem for you.
> BTW Does anyone knows why cmake is reloading each time I update llvm/clang
repo? I hope that both approaches would solve this problem, because it
doesn't seem like a something that should happen.
It reconfigures every time a CMakelists.txt file used in the build is
changed, which is unavoidable as far as I'm aware.

Cheers.

Tim.

2016-07-22 14:04 GMT-07:00 Tim Northover <t.p.northover at gmail.com>:
> On 22 July 2016 at 13:48, Piotr Padlewski via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >> The build system can help, you just need to have two (sparse)
checkout:
> one for LLVM/clang and the other for clang-tidy, and configure the build
> with the LLVM/clang checkout adding the clang-tidy as external.
>
> > Can you describe it more?
>
> Something like this
>
> $ git clone git at github.com:llvm/llvm.git
> $ git clone git at github.com:llvm/llvm.git clang-tools-extras
> $ <make the clang-tools sparse if you want, doesn't seem strictly
> necessary though>
> $ mkdir build && cd build
> $ cmake ../llvm
>
>
-DLLVM_EXTERNAL_CLANG_TOOLS_EXTRA_SOURCE_DIR=../clang-tools-extras/tools/clang/tools/clang-tools-extras
> $ make
>
Oh ok, it actually seems much easier than I thought! Thanks.

> > I don't get the approach, but it seems we are trying to make it
easier
> to use llvm, but in the same time we are making it harder.
As Justin said, this isn't an issue for the majority of developers
and
> it's a solvable problem for you.
>
> > BTW Does anyone knows why cmake is reloading each time I update
> llvm/clang repo? I hope that both approaches would solve this problem,
> because it doesn't seem like a something that should happen.
>
> It reconfigures every time a CMakelists.txt file used in the build is
> changed, which is unavoidable as far as I'm aware.
>
>
That might be the case.

> Cheers.
>
> Tim.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160722/8211edbb/attachment.html>

On Sat, Jul 23, 2016 at 9:04 AM, Tim Northover via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 22 July 2016 at 13:48, Piotr Padlewski via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >> The build system can help, you just need to have two (sparse)
checkout:
> one for LLVM/clang and the other for clang-tidy, and configure the build
> with the LLVM/clang checkout adding the clang-tidy as external.
>
> > Can you describe it more?
>
> Something like this
>
> $ git clone git at github.com:llvm/llvm.git
> $ git clone git at github.com:llvm/llvm.git clang-tools-extras
>
That 2nd clone is better as either:

$ git clone ./llvm clang-tools-extras

or

$ cd llvm
$ git worktree add ../clang-tools-extras

Both result in (initially) sharing one copy of the repository.

With the first, the llvm directory is the origin for the clang-tools-extras
directory, and after committing in clang-tools-extras you need to push to
origin (the llvm dir), and then to github. Or add github as another remote
for the clang-tools-extras checkout and push directly to it.

With the second version, commits in both checkouts go to the exact same
repository. In this case you are forced to have different branches checked
out in each directory -- the worktree add command automatically creates a
new branch called clang-tools-extras and the 2nd checkout is put on it. (or
you can explicitly give the new branch a different name)


> $ <make the clang-tools sparse if you want, doesn't seem strictly
> necessary though>
> $ mkdir build && cd build
> $ cmake ../llvm
>
>
-DLLVM_EXTERNAL_CLANG_TOOLS_EXTRA_SOURCE_DIR=../clang-tools-extras/tools/clang/tools/clang-tools-extras
> $ make
>
> > I don't get the approach, but it seems we are trying to make it
easier
> to use llvm, but in the same time we are making it harder.
>
> As Justin said, this isn't an issue for the majority of developers and
> it's a solvable problem for you.
>
> > BTW Does anyone knows why cmake is reloading each time I update
> llvm/clang repo? I hope that both approaches would solve this problem,
> because it doesn't seem like a something that should happen.
>
> It reconfigures every time a CMakelists.txt file used in the build is
> changed, which is unavoidable as far as I'm aware.
>
> Cheers.
>
> Tim.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160723/40bef029/attachment-0001.html>

On Sat, Jul 23, 2016 at 9:04 AM, Tim Northover via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 22 July 2016 at 13:48, Piotr Padlewski via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >> The build system can help, you just need to have two (sparse)
checkout:
> one for LLVM/clang and the other for clang-tidy, and configure the build
> with the LLVM/clang checkout adding the clang-tidy as external.
>
> > Can you describe it more?
>
> Something like this
>
> $ git clone git at github.com:llvm/llvm.git
> $ git clone git at github.com:llvm/llvm.git clang-tools-extras
>
That 2nd clone is better as either:

$ git clone ./llvm clang-tools-extras

or

$ cd llvm
$ git worktree add ../clang-tools-extras

Both result in (initially) sharing one copy of the repository.

With the first, the llvm directory is the origin for the clang-tools-extras
directory, and after committing in clang-tools-extras you need to push to
origin (the llvm dir), and then to github. Or add github as another remote
for the clang-tools-extras checkout and push directly to it.

With the second version, commits in both checkouts go to the exact same
repository. In this case you are forced to have different branches checked
out in each directory -- the worktree add command automatically creates a
new branch called clang-tools-extras and the 2nd checkout is on it.

> $ <make the clang-tools sparse if you want, doesn't seem strictly
> necessary though>
> $ mkdir build && cd build
> $ cmake ../llvm
>
>
-DLLVM_EXTERNAL_CLANG_TOOLS_EXTRA_SOURCE_DIR=../clang-tools-extras/tools/clang/tools/clang-tools-extras
> $ make
>
> > I don't get the approach, but it seems we are trying to make it
easier
> to use llvm, but in the same time we are making it harder.
>
> As Justin said, this isn't an issue for the majority of developers and
> it's a solvable problem for you.
>
> > BTW Does anyone knows why cmake is reloading each time I update
> llvm/clang repo? I hope that both approaches would solve this problem,
> because it doesn't seem like a something that should happen.
>
> It reconfigures every time a CMakelists.txt file used in the build is
> changed, which is unavoidable as far as I'm aware.
>
> Cheers.
>
> Tim.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160723/b1ce37e7/attachment.html>

>> Our clones of LLVM and clang have a reasonable amount of history (a
couple of hundred commits, I believe), including multiple branches, that we’d
want to preserve.  Both branches have merged from upstream multiple times.  It’s
one of the smaller friendly forks that I know about.  I’ve not used git
filter-branch before, but I’d be very impressed if there is some simple
invocation that can can move from this model.
>
> James and I owe you something here.
OK, I managed to port the master branch of the CHERI clang fork to the
existing monorepo.

The ported CHERI branch and the script James and I cooked up are at

https://github.com/jlebar/llvm-project/tree/cheri-clang-master
https://github.com/jlebar/llvm-port-commits.

At the moment, the script isn't particularly user-friendly, nor is it
at all general -- it assumes you're porting specifically a clang
branch into a branch with the format of the existing monorepo [1].  It
also assumes certain branch names.  All this can be easily fixed;
please consider it just a proof of concept.

To answer the question of how others with in-progress work would move
to the new repo, they could either

a) Run the script themselves (it takes 20 minutes to port 300 commits
on my machine, although this could be sped up ~10x, see the README),
or

b) Make a new clone from the translated monorepo upstream and then
  - exporting each commit using git format-patch
  - editing the paths in each patch (sed is sufficient), and finally
  - applying the patches to the new clone.

I can write up more explicit instructions for (b) when the time comes,
if desired.

-Justin

[1] https://github.com/llvm-project/llvm-project

On Thu, Jul 21, 2016 at 9:16 AM, Justin Lebar <jlebar at google.com>
wrote:
>> Our clones of LLVM and clang have a reasonable amount of history (a
couple of hundred commits, I believe), including multiple branches, that we’d
want to preserve.  Both branches have merged from upstream multiple times.  It’s
one of the smaller friendly forks that I know about.  I’ve not used git
filter-branch before, but I’d be very impressed if there is some simple
invocation that can can move from this model.
>
> James and I owe you something here.  I think this can be handled in a
> straightforward manner, but I am not 100% sure how at the moment.  I
> agree this is very important.
>
> Our demo would be much more compelling if we can use an existing
> branch.  Does anyone know of one we can play with?
>
>> In particular, the fact that we have a third more public GitHub forks
of LLVM than of clang, and eight times as many as of lldb implies to me that
forcing everyone downstream to pull in all subprojects would not be particularly
well received.
>
> I have a hard time understanding this particular argument.  Per the
> original e-mail, with three shell commands, you can hide whichever
> llvm subprojects you want.  After doing that, the only overhead of the
> subprojects is extra space in your .git directory, which would still
> be much smaller than an llvm+clang objdir.
>
> Is there something specific that you think will not be well-received?
> Or maybe it's better to speak personally -- is there something
> specific that will bother you personally about having to clone (but
> not check out) everything?
>
> On Thu, Jul 21, 2016 at 8:22 AM, David Chisnall via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> On 21 Jul 2016, at 16:11, James Y Knight <jyknight at google.com>
wrote:
>>>
>>> I don't think this is a really hard problem though -- I can
think of a few ways to help existing users that probably will work (although
I'd want to try them first, to ensure it actually does work, of course). The
two I'm thinking of are just doing "git diff" followed by
"git apply --directory=llvm" if you just want to save a patch. Or,
some "git filter-branch" invocation to rename all the files in your
existing repo, followed by "git rebase" (or "git merge"), if
you have some more history you want to maintain.
>>
>> Our clones of LLVM and clang have a reasonable amount of history (a
couple of hundred commits, I believe), including multiple branches, that we’d
want to preserve.  Both branches have merged from upstream multiple times.  It’s
one of the smaller friendly forks that I know about.  I’ve not used git
filter-branch before, but I’d be very impressed if there is some simple
invocation that can can move from this model.
>>
>> I was in favour of the GitHub migration primarily because a lot of
downstream LLVM users already have a workflow based around GitHub that works
well and the proposal was to make this closer to the official workflow.  I’m
very nervous about a last-minute change to require everyone downstream to
restructure their workflows.
>>
>> In particular, the fact that we have a third more public GitHub forks
of LLVM than of clang, and eight times as many as of lldb implies to me that
forcing everyone downstream to pull in all subprojects would not be particularly
well received.
>>
>> David
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>