llvm dev - Jul 2016 - [RFC] One or many git repositories?

> On Jul 22, 2016, at 7:28 PM, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> 
>> On Jul 22, 2016, at 12:31 AM, David Chisnall <David.Chisnall at
cl.cam.ac.uk> wrote:
>> 
>> On 22 Jul 2016, at 07:14, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>>> 
>>> Just tried to set it up there:
https://github.com/joker-eph/llvm-unified
>>> (git log —follow is working fine with this setup).
>>> 
>>> While it preserves the history fine (I.e. the hashes are identical
to the current git), it has a drawback: there isn’t anymore a common ancestor
for the parents of the merge (this may or may not be an issue in practice, not
sure yet, but it is uncommon for git).
>> 
>> Perhaps I’m missing something, but on GitHub I can’t see any of the
history.  Assuming that this is just a GitHub UI issue,
> 
> Probably, this should be populated:
https://github.com/joker-eph/llvm-unified/commits/master
> History for one file isn’t great on github because it does not follow move:
https://github.com/joker-eph/llvm-unified/commits/master/llvm/CMakeLists.txt
> 
> On the command line, `git log —follow llvm/CMakeLists.txt` works fine.
> 
>> can you explain the steps for a downstream user who has clones of the
llvm and clang repos that were forked at some point in the past, have been
periodically merged, and contain a load of commits on top of upstream would
migrate to using this?
> 
> Taking cheri as an example, it seems you have a repo for clang and one for
llvm (https://github.com/CTSRD-CHERI/clang and
https://github.com/CTSRD-CHERI/llvm)
> 
> You can create the unified repo for cheri quite easily, it is roughly doing
this:
> 
> 1) checkout llvm, move everything to a llvm subdir, commit
> 2) checkout clang, move everything to a clang subdir, commit
> 3) merge clang/master into llvm/master.
> 4) done.
> 
> From now you should be able to merge the unified repo (the first merge may
require extra git-fu)
> 
> I just tried the attached script to unify cheri
I forgot to mention that I pushed the result there:
https://github.com/joker-eph/llvm-unified/tree/cheri-unified
(History: https://github.com/joker-eph/llvm-unified/commits/cheri-unified )

— 
Mehdi




> , it takes a few minutes to fetch from github, and a few seconds to merge.
To be used as `./unify-cheri.sh cheri-unified` (it’ll create a directory
cheri-unified, fetch and merge).
> 
> <unify-cheri.sh>
> 
> 
> The only advantages of this approach over Justin’s script is to preserve
the history of the existing official git repositories and yours, including not
changing the hashes (t may or may not matter that much).
> 
> 
> — 
> Mehdi
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On 23 Jul 2016, at 03:39, Mehdi Amini <mehdi.amini at apple.com>
wrote:
> 
> I forgot to mention that I pushed the result there:
https://github.com/joker-eph/llvm-unified/tree/cheri-unified
> (History: https://github.com/joker-eph/llvm-unified/commits/cheri-unified )
Thank you very much for doing this!  I’m now convinced that there’s a clean way
to move from existing git clones to the unified repository.

I think that we do need some decent documentation for how people with clones of
the existing github.com/llvm-mirror repos produce a unified repository that they
can then merge from upstream, but it looks pretty nice.

It will require a bit of effort for us to update various scripts, CI jobs, and
so on to make use of this, but it looks as if the end result will be a lot
easier to work with and worth the transition pain.

On a fast machine with a decent Internet connection, a fresh clone of this repo
takes about one minute, 40 seconds, transfers 458.56MB and produces a directory
that uses 876MB of disk space.  These numbers all look very reasonable.

David

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3719 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160723/6c255e05/attachment.bin>

On 23 July 2016 at 12:38, David Chisnall via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> I think that we do need some decent documentation for how people with
clones of the existing github.com/llvm-mirror repos produce a unified repository
that they can then merge from upstream, but it looks pretty nice.
Indeed, it does, Thanks Mehdi and Justin for going the extra mile to
show how it's done.

> It will require a bit of effort for us to update various scripts, CI jobs,
and so on to make use of this, but it looks as if the end result will be a lot
easier to work with and worth the transition pain.
The point I made earlier is that we'll have *some* pain regardless of
what we change. If we change more, we'll have more pain, true, but if
we change to a place where we want to be, the change will only happen
once.

I think that's a worthy goal.

cheers,
--renato

On a decent machine, on ethernet from an laptop sharing its WIFI, in a flat
in Moscow ... two minutes and four seconds.

The actual .git directory is 523 MB. My total checkout is bigger than yours
(1.1 GB), presumably because my disk has bigger clusters.

Using 'git clone --depth 1 https://github.com/joker-eph/llvm-unified'
(i.e.
without history ... but you can build head from it), it took 26 seconds to
download 79.62 MB and the whole directory is 604 MB.

llvm is 202 MB, clang 117 MB, and compiler-rt 35 MB. Not including the
other stuff would only save about 170 MB in the checkout (also savable with
a sparse checkout)  and about 30 MB in the .git directory.

Also, running ...

git repack -a -d -f --depth=250 --window=250

.. on the original 523 MB .git checkout reduced it to 449 MB. This took
5m44s on an i7 6700K and needs to be done only once, before the initial
push to github. Every user will then benefit from 74 MB of saved disk space
and network bandwidth on the existing history, forever.


On Sat, Jul 23, 2016 at 11:38 PM, David Chisnall via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 23 Jul 2016, at 03:39, Mehdi Amini <mehdi.amini at apple.com>
wrote:
> >
> > I forgot to mention that I pushed the result there:
> https://github.com/joker-eph/llvm-unified/tree/cheri-unified
> > (History:
> https://github.com/joker-eph/llvm-unified/commits/cheri-unified )
>
> Thank you very much for doing this!  I’m now convinced that there’s a
> clean way to move from existing git clones to the unified repository.
>
> I think that we do need some decent documentation for how people with
> clones of the existing github.com/llvm-mirror repos produce a unified
> repository that they can then merge from upstream, but it looks pretty
nice.
>
> It will require a bit of effort for us to update various scripts, CI jobs,
> and so on to make use of this, but it looks as if the end result will be a
> lot easier to work with and worth the transition pain.
>
> On a fast machine with a decent Internet connection, a fresh clone of this
> repo takes about one minute, 40 seconds, transfers 458.56MB and produces a
> directory that uses 876MB of disk space.  These numbers all look very
> reasonable.
>
> David
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160724/760faf28/attachment.html>