llvm dev - Jul 2016 - [RFC] One or many git repositories?

> On Jul 21, 2016, at 2:29 PM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
> 
> 
>> On Jul 21, 2016, at 11:03 AM, Sanjoy Das via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>> 
>> FWIW, like David Chisnall, we (Azul) have a problem with rewriting
>> history.  
>> Our LLVM fork has O(100) changes diverging from upstream
>> (though our branching structure is simple), and keeping all of that
>> history is important.
> 
> Nobody downstream has to adopt the new structure, I believe it is possible
to extract only the “llvm” commits from the new repo and rebase them on top of
the existing llvm repo.
> This can be done on the fly by you CI, but it is also a deterministic
process, i.e. you can restart from scratch anytime (assuming you have the
original llvm.git repo and the new one).
> 
>> 
>> What do people think of having one (or a set of) merge commit(s)
>> merging in the non-llvm projects that will be part of the new
>> monorepo?  That's the only technique I can think of that will
preserve
>> history for downstream users by construction.
> 
> I have no idea what you mean here?
I think I understand what you mean:

1) checkout the existing clang repo
2) move everything in a subdirectory “clang”
3) commit the move
4) merge this into the new “llvm-project”.
5) repeat for every single project

That should preserve the hashes and avoid user to have to “extract” the
subproject to merge into their own branch.
Annoyingly, it breaks git log path/to/file though.

— 
Mehdi



>> 
>> 
>> 
>> On Thu, Jul 21, 2016 at 10:44 AM, Jonathan Roelofs via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>>> 
>>> 
>>> On 7/21/16 11:03 AM, C Bergström via llvm-dev wrote:
>>>> 
>>>> Monolithic is trying to solve the wrong problem - it's that
simple.
>>>> Any discussion or attempt to coddle those who think it's
necessary is
>>>> a waste of time. #dictator
>>> 
>>> 
>>> Christopher,
>>> 
>>> AFAICT, you haven't explained *why* it is the wrong problem.
Mind
>>> elaborating on that?
>>> 
>>> 
>>> Jon
>>> 
>>> p.s: edicts, appeals to authority, and ad hominems are not useful
for
>>> discussion. Doing that, and following up with "#dictator"
further solidifies
>>> that you know your own argument is b-s.... please stop.
>>> 
>>> --
>>> Jon Roelofs
>>> jonathan at codesourcery.com
>>> CodeSourcery / Mentor Embedded
>>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> 
>> 
>> 
>> -- 
>> Sanjoy Das
>> http://playingwithpointers.com
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org>
writes:
>>> What do people think of having one (or a set of) merge commit(s)
>>> merging in the non-llvm projects that will be part of the new
>>> monorepo?  That's the only technique I can think of that will
preserve
>>> history for downstream users by construction.
>> 
>> I have no idea what you mean here?
>
> I think I understand what you mean:
>
> 1) checkout the existing clang repo
> 2) move everything in a subdirectory “clang”
> 3) commit the move
> 4) merge this into the new “llvm-project”.
> 5) repeat for every single project
>
> That should preserve the hashes and avoid user to have to “extract”
> the subproject to merge into their own branch.
> Annoyingly, it breaks git log path/to/file though.
Use `git log --follow path/to/file`. It's better ;)

> On Jul 21, 2016, at 2:56 PM, Justin Bogner <mail at justinbogner.com>
wrote:
> 
> Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org> writes:
>>>> What do people think of having one (or a set of) merge
commit(s)
>>>> merging in the non-llvm projects that will be part of the new
>>>> monorepo?  That's the only technique I can think of that
will preserve
>>>> history for downstream users by construction.
>>> 
>>> I have no idea what you mean here?
>> 
>> I think I understand what you mean:
>> 
>> 1) checkout the existing clang repo
>> 2) move everything in a subdirectory “clang”
>> 3) commit the move
>> 4) merge this into the new “llvm-project”.
>> 5) repeat for every single project
>> 
>> That should preserve the hashes and avoid user to have to “extract”
>> the subproject to merge into their own branch.
>> Annoyingly, it breaks git log path/to/file though.
> 
> Use `git log --follow path/to/file`. It's better ;)
I know, it works most of the time for log, but how do blame it at a revision
older than the move?

— 
Mehdi

> On Jul 21, 2016, at 2:32 PM, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
>> 
>> On Jul 21, 2016, at 2:29 PM, Mehdi Amini <mehdi.amini at
apple.com> wrote:
>> 
>> 
>>> On Jul 21, 2016, at 11:03 AM, Sanjoy Das via llvm-dev <llvm-dev
at lists.llvm.org> wrote:
>>> 
>>> FWIW, like David Chisnall, we (Azul) have a problem with rewriting
>>> history.  
>>> Our LLVM fork has O(100) changes diverging from upstream
>>> (though our branching structure is simple), and keeping all of that
>>> history is important.
>> 
>> Nobody downstream has to adopt the new structure, I believe it is
possible to extract only the “llvm” commits from the new repo and rebase them on
top of the existing llvm repo.
>> This can be done on the fly by you CI, but it is also a deterministic
process, i.e. you can restart from scratch anytime (assuming you have the
original llvm.git repo and the new one).
To clarify, I believe we* can maintain this read-only view of the individual
repo like (https://github.com/llvm-mirror/llvm
<https://github.com/llvm-mirror/llvm> and siblings) with their current
history (hashes…), and adding on top the new commits from the new repo:

*: we could be the LLVM foundation or some volunteer or each downstream
user/org.

— 
Mehdi


>> 
>>> 
>>> What do people think of having one (or a set of) merge commit(s)
>>> merging in the non-llvm projects that will be part of the new
>>> monorepo?  That's the only technique I can think of that will
preserve
>>> history for downstream users by construction.
>> 
>> I have no idea what you mean here?
> 
> I think I understand what you mean:
> 
> 1) checkout the existing clang repo
> 2) move everything in a subdirectory “clang”
> 3) commit the move
> 4) merge this into the new “llvm-project”.
> 5) repeat for every single project
> 
> That should preserve the hashes and avoid user to have to “extract” the
subproject to merge into their own branch.
> Annoyingly, it breaks git log path/to/file though.
> 
> — 
> Mehdi
> 
> 
> 
> 
>>> 
>>> 
>>> 
>>> On Thu, Jul 21, 2016 at 10:44 AM, Jonathan Roelofs via llvm-dev
>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>>> 
>>>> 
>>>> On 7/21/16 11:03 AM, C Bergström via llvm-dev wrote:
>>>>> 
>>>>> Monolithic is trying to solve the wrong problem - it's
that simple.
>>>>> Any discussion or attempt to coddle those who think
it's necessary is
>>>>> a waste of time. #dictator
>>>> 
>>>> 
>>>> Christopher,
>>>> 
>>>> AFAICT, you haven't explained *why* it is the wrong
problem. Mind
>>>> elaborating on that?
>>>> 
>>>> 
>>>> Jon
>>>> 
>>>> p.s: edicts, appeals to authority, and ad hominems are not
useful for
>>>> discussion. Doing that, and following up with
"#dictator" further solidifies
>>>> that you know your own argument is b-s.... please stop.
>>>> 
>>>> --
>>>> Jon Roelofs
>>>> jonathan at codesourcery.com
>>>> CodeSourcery / Mentor Embedded
>>>> 
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> 
>>> 
>>> 
>>> -- 
>>> Sanjoy Das
>>> http://playingwithpointers.com
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160721/5d64d33b/attachment.html>

> On Jul 21, 2016, at 2:32 PM, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
>> 
>> On Jul 21, 2016, at 2:29 PM, Mehdi Amini <mehdi.amini at
apple.com> wrote:
>> 
>> 
>>> On Jul 21, 2016, at 11:03 AM, Sanjoy Das via llvm-dev <llvm-dev
at lists.llvm.org> wrote:
>>> 
>>> FWIW, like David Chisnall, we (Azul) have a problem with rewriting
>>> history.  
>>> Our LLVM fork has O(100) changes diverging from upstream
>>> (though our branching structure is simple), and keeping all of that
>>> history is important.
>> 
>> Nobody downstream has to adopt the new structure, I believe it is
possible to extract only the “llvm” commits from the new repo and rebase them on
top of the existing llvm repo.
>> This can be done on the fly by you CI, but it is also a deterministic
process, i.e. you can restart from scratch anytime (assuming you have the
original llvm.git repo and the new one).
>> 
>>> 
>>> What do people think of having one (or a set of) merge commit(s)
>>> merging in the non-llvm projects that will be part of the new
>>> monorepo?  That's the only technique I can think of that will
preserve
>>> history for downstream users by construction.
>> 
>> I have no idea what you mean here?
> 
> I think I understand what you mean:
> 
> 1) checkout the existing clang repo
> 2) move everything in a subdirectory “clang”
> 3) commit the move
> 4) merge this into the new “llvm-project”.
> 5) repeat for every single project
> 
> That should preserve the hashes and avoid user to have to “extract” the
subproject to merge into their own branch.
> Annoyingly, it breaks git log path/to/file though.
Just tried to set it up there: https://github.com/joker-eph/llvm-unified
(git log —follow is working fine with this setup).

While it preserves the history fine (I.e. the hashes are identical to the
current git), it has a drawback: there isn’t anymore a common ancestor for the
parents of the merge (this may or may not be an issue in practice, not sure yet,
but it is uncommon for git).

Now I need to write the script to regenerate the independent “read-only” repos.

— 
Mehdi

> 
> 
> 
> 
>>> 
>>> 
>>> 
>>> On Thu, Jul 21, 2016 at 10:44 AM, Jonathan Roelofs via llvm-dev
>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>>> 
>>>> 
>>>> On 7/21/16 11:03 AM, C Bergström via llvm-dev wrote:
>>>>> 
>>>>> Monolithic is trying to solve the wrong problem - it's
that simple.
>>>>> Any discussion or attempt to coddle those who think
it's necessary is
>>>>> a waste of time. #dictator
>>>> 
>>>> 
>>>> Christopher,
>>>> 
>>>> AFAICT, you haven't explained *why* it is the wrong
problem. Mind
>>>> elaborating on that?
>>>> 
>>>> 
>>>> Jon
>>>> 
>>>> p.s: edicts, appeals to authority, and ad hominems are not
useful for
>>>> discussion. Doing that, and following up with
"#dictator" further solidifies
>>>> that you know your own argument is b-s.... please stop.
>>>> 
>>>> --
>>>> Jon Roelofs
>>>> jonathan at codesourcery.com
>>>> CodeSourcery / Mentor Embedded
>>>> 
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> 
>>> 
>>> 
>>> -- 
>>> Sanjoy Das
>>> http://playingwithpointers.com
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160721/81ab58bb/attachment.html>

On 22 Jul 2016, at 07:14, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Just tried to set it up there: https://github.com/joker-eph/llvm-unified
> (git log —follow is working fine with this setup).
> 
> While it preserves the history fine (I.e. the hashes are identical to the
current git), it has a drawback: there isn’t anymore a common ancestor for the
parents of the merge (this may or may not be an issue in practice, not sure yet,
but it is uncommon for git).
Perhaps I’m missing something, but on GitHub I can’t see any of the history. 
Assuming that this is just a GitHub UI issue, can you explain the steps for a
downstream user who has clones of the llvm and clang repos that were forked at
some point in the past, have been periodically merged, and contain a load of
commits on top of upstream would migrate to using this?

David

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3719 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160722/a19c4213/attachment.bin>