llvm dev - Jun 2020 - RFC: Adding a staging branch (temporarily) to facilitate upstreaming

To facilitate collaboration on an upstreaming effort (see "More
context" below), we'd like to push a branch (with history) called
"staging/apple" to github.com/llvm/llvm-project to serve as an
official contribution to the LLVM project. This enables motivated parties to
work with us to craft incremental patches for review on Phabricator. This branch
would live during the effort and then be deleted after. It would not be merged.

Does this seem fine? If you have a strong objection, please share your concern.

For reference, I ran some experiments:
A `--bare` clone (just the Git database) I have of github.com/llvm/llvm-project
was around ~1GB. Fetching this branch from github.com/apple/llvm-project
increased it to ~1.2GB. Running `git gc --aggressive` brought it down to ~850MB.
The worktree of the "master" branch is ~1GB. Adding the Git database
gives ~2GB, ~2.2GB, and ~1.9GB.
The diff of the proposed staging/apple branch is 3.1MB at `-U0`, 4.1MB at `-U3`,
and 32MB at `-U999999` (Phabricator settings).


More context

We're making a major push over the next few months to upstream changes that
have accumulated over time in the branch called "apple/master" at
github.com/apple/llvm-project. It has always been a non-goal for us to have
changes, but over the years we've accumulated a non-trivial diff vs.
github.com/llvm/llvm-project. This includes (but is not limited to)
tweaks/features related to tuple hashing, modules hashing, source attributes,
API notes, pointer authentication, indexing-while-building, and local
refactoring.

Our goal is to eliminate this difference. Besides paying off some debt, this
upstreaming effort unblocks the Swift compiler (github.com/apple/swift) from
building directly against an upstream checkout of LLVM. That's why non-Apple
contributors are motivated to help craft incremental patches.


Alternatives considered

As an alternative, we could post a GitHub pull request and close it without
merging. From our perspective this would serve the same purpose. However, pull
requests are contentious in LLVM.

Another alternative is to post a bulk Phabricator review and then
"abandon" it. However, this has the disadvantage of not contributing
the history (~30k commits).
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200629/15521df7/attachment.html>

Hey Duncan,

On Mon, Jun 29, 2020 at 8:28 PM Duncan Exon Smith via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> To facilitate collaboration on an upstreaming effort (see "More
context"
> below), we'd like to *push a branch* (with history) called "
> *staging/apple*" to github.com/llvm/llvm-project to serve as an
official
> contribution to the LLVM project. This enables motivated parties to work
> with us to craft incremental patches for review on Phabricator. This branch
> would live during the effort and then be deleted after. It would not be
> merged.
>
> *Does this seem fine?* If you have a strong objection, please share your
> concern.
>
> For reference, I ran some experiments:
>
>    - A `--bare` clone (just the Git database) I have of
>    github.com/llvm/llvm-project was around ~1GB. Fetching this branch
>    from github.com/apple/llvm-project increased it to ~1.2GB. Running
>    `git gc --aggressive` brought it down to ~850MB.
>    - The worktree of the "master" branch is ~1GB. Adding the Git
database
>    gives ~2GB, ~2.2GB, and ~1.9GB.
>    - The diff of the proposed staging/apple branch is 3.1MB at `-U0`,
>    4.1MB at `-U3`, and 32MB at `-U999999` (Phabricator settings).
>
>
>
> *More context*
>
> We're making a major push over the next few months to upstream
> changes that have accumulated over time in the branch called
"apple/master"
> at github.com/apple/llvm-project. It has always been a non-goal for us to
> have changes, but over the years we've accumulated a non-trivial diff
vs.
> github.com/llvm/llvm-project. This includes (but is not limited
> to) tweaks/features related to tuple hashing, modules hashing, source
> attributes, API notes, pointer authentication, indexing-while-building, and
> local refactoring.
>
> Our goal is to eliminate this difference. Besides paying off some debt,
> this upstreaming effort unblocks the Swift compiler (
> github.com/apple/swift) from building directly against an upstream
> checkout of LLVM. That's why non-Apple contributors are motivated to
help
> craft incremental patches.
>
>
> *Alternatives considered*
>
> As an alternative, we could post a GitHub pull request and close it
> without merging. From our perspective this would serve the same purpose.
> However, pull requests are contentious in LLVM.
>
> Another alternative is to post a bulk Phabricator review and then
> "abandon" it. However, this has the disadvantage of not
contributing the
> history (~30k commits).
>
Seeing the alternatives, if a closed pull-request would fit your use-case,
then I'm not sure why you need the branch to actually live in the monorepo
instead of in any fork (github.com/apple/llvm-project or another)? It seems
publicly accessible the same way?
Your initial description mentions "collaboration on an upstreaming
effort",
maybe you can elaborate a bit on what this collaboration would look like
and how these patches would end up in master?

Thanks,

-- 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200629/3c36701a/attachment.html>

On Mon, Jun 29, 2020 at 9:43 PM Mehdi AMINI via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hey Duncan,
>
> On Mon, Jun 29, 2020 at 8:28 PM Duncan Exon Smith via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> To facilitate collaboration on an upstreaming effort (see "More
context"
>> below), we'd like to *push a branch* (with history) called "
>> *staging/apple*" to github.com/llvm/llvm-project to serve as an
official
>> contribution to the LLVM project. This enables motivated parties to
work
>> with us to craft incremental patches for review on Phabricator. This
branch
>> would live during the effort and then be deleted after. It would not be
>> merged.
>>
>> *Does this seem fine?* If you have a strong objection, please share
your
>> concern.
>>
>> For reference, I ran some experiments:
>>
>>    - A `--bare` clone (just the Git database) I have of
>>    github.com/llvm/llvm-project was around ~1GB. Fetching this branch
>>    from github.com/apple/llvm-project increased it to ~1.2GB. Running
>>    `git gc --aggressive` brought it down to ~850MB.
>>    - The worktree of the "master" branch is ~1GB. Adding the
Git
>>    database gives ~2GB, ~2.2GB, and ~1.9GB.
>>    - The diff of the proposed staging/apple branch is 3.1MB at `-U0`,
>>    4.1MB at `-U3`, and 32MB at `-U999999` (Phabricator settings).
>>
>>
>>
>> *More context*
>>
>> We're making a major push over the next few months to upstream
>> changes that have accumulated over time in the branch called
"apple/master"
>> at github.com/apple/llvm-project. It has always been a non-goal for us
>> to have changes, but over the years we've accumulated a non-trivial
diff
>> vs. github.com/llvm/llvm-project. This includes (but is not limited
>> to) tweaks/features related to tuple hashing, modules hashing, source
>> attributes, API notes, pointer authentication, indexing-while-building,
and
>> local refactoring.
>>
>> Our goal is to eliminate this difference. Besides paying off some debt,
>> this upstreaming effort unblocks the Swift compiler (
>> github.com/apple/swift) from building directly against an upstream
>> checkout of LLVM. That's why non-Apple contributors are motivated
to help
>> craft incremental patches.
>>
>>
>> *Alternatives considered*
>>
>> As an alternative, we could post a GitHub pull request and close it
>> without merging. From our perspective this would serve the same
purpose.
>> However, pull requests are contentious in LLVM.
>>
>> Another alternative is to post a bulk Phabricator review and then
>> "abandon" it. However, this has the disadvantage of not
contributing the
>> history (~30k commits).
>>
>
> Seeing the alternatives, if a closed pull-request would fit your use-case,
> then I'm not sure why you need the branch to actually live in the
monorepo
> instead of in any fork (github.com/apple/llvm-project or another)? It
> seems publicly accessible the same way?
>
As I understand it, a key need is to explicitly contribute this to the LLVM
project to make it unambiguous that it has been contributed and is
completely available for folks not at Apple to iterate on the code and turn
it into code-reviewable chunks.

So whatever happens needs to be quite explicit in its nature as a
contribution. IMO, a branch of the repository definitely qualifies.

IMO, a pull request isn't as clear given that they haven't been used for
contributions before. This is not a time to be innovative IMO. A branch as
a staging location has been used many times over the history of the project
though and seems nicely unambiguous in that regard.

> Your initial description mentions "collaboration on an upstreaming
> effort", maybe you can elaborate a bit on what this collaboration
> would look like and how these patches would end up in master?
>
> Thanks,
>
> --
> Mehdi
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200629/9602de38/attachment.html>