llvm dev - Jul 2016 - [RFC] One or many git repositories?

> On Jul 29, 2016, at 2:19 AM, David Chisnall <david.chisnall at
cl.cam.ac.uk> wrote:
> 
> On 29 Jul 2016, at 05:11, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>> 
>> What I meant by “different problem" is that “downstream users” for
instance don’t need to commit, that makes their problem/workflow quite different
from an upstream developer (for instance it is fairly easy to maintain a
read-only view of the existing individual git repo currently on llvm.org).
> 
> I’m not convinced by this distinction.  A lot of downstream developers need
to patch LLVM and we benefit when they upstream their changes.
I made a difference between downstream users and developers. I.e. someone that
just need to get and build compiler-rt vs someone that want to *commit* to LLVM.
Note that even by getting a single repo you can still send a patch to the
mailing list and someone can commit it for you (including correct author
attribution contrary to SVN).
> We should not make it harder for them to do this.  To give a couple of
example downstream projects, both FreeBSD and Swift have patches on LLVM / Clang
in their versions that they gradually filter upstream.  Both projects have LLVM
committers among their members.  If the workflow that we recommend for them
makes upstreaming easy then they benefit (maintaining a fork is effort) and LLVM
benefits (having people provide bug fixes makes our code better).
> 
> The workflow that we want to recommend to these people is:
> 
> - Fork the repo that you’re interested in from the LLVM GitHub organisation
> - Make your changes
> - Send pull requests for anything that you think is of interest to upstream

Note that the workflow you describe above still requires to export their patch
and import it in this clone before pushing.
(Note also that we accept patches on the mailing list, so one does not even need
to clone the official repo).
> This makes the barrier to entry for sending code back upstream *much* lower
than it currently is,
I don’t understand this statement. As of today you can send a diff to the
mailing list, I don’t see how lower the bar can be.

> to the benefit of all.  If the alternative is:
> 
> - Fork a read-only repo that you’re interested in from the LLVM GitHub
organisation
> - Make your changes
Why? If you know you want to *push* commits upstream, fork the only useful repo
for that in the first place.
> - Fork a different repo from the LLVM GitHub organisation
> - Run a script to filter some of your changes into that one
I don’t know why you think there is a need for a script, or why it is different
from today.
Let say I’m working on a fork of the compiler-rt read-only repo and I want to
upstream a patch at some point:

Today:

- cd /path/to/compiler_rt-forked
- git format-patch …
- cd /path/to/compiler_rt-upstream
- git am  /path/to/compiler_rt-forked/0001-My-awesome-changes.patch
- git svn dcommit
- done

Tomorrow with a monorepo:

- cd /path/to/compiler_rt-forked
- git format-patch …
- cd /path/to/unifiedrepo-upstream
- git am  /path/to/compiler_rt-forked/0001-My-awesome-changes.patch
—directory=compiler-rt
- git push
- done

Alternatively, if I’m upstream a patch once a year, I don’t really need to push
it myself.

- cd /path/to/compiler_rt-forked
- git format-patch …
- email the patch.

> - Send a pull request from that
Note that I think we deferred any change to the workflow for future discussions
(pull-request are not part of our workflow today).
> - Deal with merging between the two yourself
I don’t know what you mean by dealing with the merging, I don’t expect any
difficulties, you need to elaborate.
> 
> I strongly suspect that we’ll get a lot fewer useful contributions from
downstream.  Or downstream people will just work on the monorepo and eat the
cost.
> 
> If someone is working on a downstream LLVM project and becoming familiar
with our codebase, then we want them to be subtly nudging their workflow so that
they eventually become LLVM contributors without noticing!
Sure. The distinction between “downstream users” and “developers” was made in
response to “there exists many user that just download and build a subproject”.
These are not people that are *developing* on a downstream fork.

— 
Mehdi

On 29 Jul 2016, at 18:01, Mehdi Amini <mehdi.amini at apple.com>
wrote:
> 
> I don’t understand this statement. As of today you can send a diff to the
mailing list, I don’t see how lower the bar can be.
Really?  You don’t see that hitting the web UI buttons that construct a pull
request are lower than finding a mailing list to post to, subscribing, preparing
a patch, emailing it, filtering out the firehose of unrelated things?  I
migrated another open source project to GitHub last year.  After several years
of accepting patches via a mailing list and receiving no patches as a result, I
immediately started getting patches sent as pull requests.  I simply cannot
overstate how much easier this mechanism is for downstream users to push fixes
back, as long as the repo that they clone is the one that accepts pull requests.

David

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
Mehdi
> Amini via llvm-dev
> Sent: Friday, July 29, 2016 10:02 AM
> To: David Chisnall
> Cc: llvm-dev at lists.llvm.org; Bruce Hoult
> Subject: Re: [llvm-dev] [RFC] One or many git repositories?
> 
> 
> > On Jul 29, 2016, at 2:19 AM, David Chisnall
> <david.chisnall at cl.cam.ac.uk> wrote:
> >
> > On 29 Jul 2016, at 05:11, Mehdi Amini via llvm-dev <llvm-
> dev at lists.llvm.org> wrote:
> >>
> >> What I meant by “different problem" is that “downstream
users” for
> instance don’t need to commit, that makes their problem/workflow quite
> different from an upstream developer (for instance it is fairly easy to
> maintain a read-only view of the existing individual git repo currently on
> llvm.org).
> >
> > I’m not convinced by this distinction.  A lot of downstream developers
> need to patch LLVM and we benefit when they upstream their changes.
> 
> I made a difference between downstream users and developers. I.e. someone
> that just need to get and build compiler-rt vs someone that want to
> *commit* to LLVM. Note that even by getting a single repo you can still
> send a patch to the mailing list and someone can commit it for you
> (including correct author attribution contrary to SVN).
> 
> > We should not make it harder for them to do this.  To give a couple of
> example downstream projects, both FreeBSD and Swift have patches on LLVM /
> Clang in their versions that they gradually filter upstream.  Both
> projects have LLVM committers among their members.  If the workflow that
> we recommend for them makes upstreaming easy then they benefit
> (maintaining a fork is effort) and LLVM benefits (having people provide
> bug fixes makes our code better).
> >
> > The workflow that we want to recommend to these people is:
> >
> > - Fork the repo that you’re interested in from the LLVM GitHub
> organisation
> > - Make your changes
> > - Send pull requests for anything that you think is of interest to
> upstream
> 
> 
> Note that the workflow you describe above still requires to export their
> patch and import it in this clone before pushing.
> (Note also that we accept patches on the mailing list, so one does not
> even need to clone the official repo).
> 
> > This makes the barrier to entry for sending code back upstream *much*
> lower than it currently is,
> 
> I don’t understand this statement. As of today you can send a diff to the
> mailing list, I don’t see how lower the bar can be.
> 
> 
> > to the benefit of all.  If the alternative is:
> >
> > - Fork a read-only repo that you’re interested in from the LLVM GitHub
> organisation
> > - Make your changes
> 
> Why? If you know you want to *push* commits upstream, fork the only useful
> repo for that in the first place.
> 
> > - Fork a different repo from the LLVM GitHub organisation
> > - Run a script to filter some of your changes into that one
> 
> I don’t know why you think there is a need for a script, or why it is
> different from today.
> Let say I’m working on a fork of the compiler-rt read-only repo and I want
> to upstream a patch at some point:
> 
> Today:
> 
> - cd /path/to/compiler_rt-forked
> - git format-patch …
> - cd /path/to/compiler_rt-upstream
> - git am  /path/to/compiler_rt-forked/0001-My-awesome-changes.patch
> - git svn dcommit
> - done
> 
> Tomorrow with a monorepo:
> 
> - cd /path/to/compiler_rt-forked
> - git format-patch …
> - cd /path/to/unifiedrepo-upstream
> - git am  /path/to/compiler_rt-forked/0001-My-awesome-changes.patch —
> directory=compiler-rt
> - git push
> - done
> 
> Alternatively, if I’m upstream a patch once a year, I don’t really need to
> push it myself.
> 
> - cd /path/to/compiler_rt-forked
> - git format-patch …
> - email the patch.
> 
> 
> > - Send a pull request from that
> 
> Note that I think we deferred any change to the workflow for future
> discussions (pull-request are not part of our workflow today).
> 
> > - Deal with merging between the two yourself
> 
> I don’t know what you mean by dealing with the merging, I don’t expect any
> difficulties, you need to elaborate.
Almost no upstream patch survives contact with the reviewers unscathed
(and quite often it doesn't look exactly like your local patch anyway,
for reasons ranging from expediency to local coding standards).
Therefore the patch that you merge back in to your local repo from 
upstream differs from the one you already have locally.  If you are *very*
lucky, the differences can be handled with accept-theirs; in my experience
this is not normal, and in many cases you need to manually revert your
original local patch before merging the patch as received from upstream.

That's what "dealing with merging" means.  I invite you to review
"Living Downstream Without Drowning" if you still don't
understand;
it describes a number of tactics we've developed over the years to
try to simplify the dealing-with-merging problem.

I had two entire dev-meeting sessions full of people who seemed to be
having this problem.  Maybe you don't; if so, you're lucky, but it
doesn't mean the problem fails to exist.
--paulr
> 
> >
> > I strongly suspect that we’ll get a lot fewer useful contributions
from
> downstream.  Or downstream people will just work on the monorepo and eat
> the cost.
> >
> > If someone is working on a downstream LLVM project and becoming
familiar
> with our codebase, then we want them to be subtly nudging their workflow
> so that they eventually become LLVM contributors without noticing!
> 
> Sure. The distinction between “downstream users” and “developers” was made
> in response to “there exists many user that just download and build a
> subproject”. These are not people that are *developing* on a downstream
> fork.
> 
> —
> Mehdi
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

> On Jul 29, 2016, at 11:07 AM, Robinson, Paul <paul.robinson at
sony.com> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org
<mailto:llvm-dev-bounces at lists.llvm.org>] On Behalf Of Mehdi
>> Amini via llvm-dev
>> Sent: Friday, July 29, 2016 10:02 AM
>> To: David Chisnall
>> Cc: llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>; Bruce Hoult
>> Subject: Re: [llvm-dev] [RFC] One or many git repositories?
>> 
>> 
>>> On Jul 29, 2016, at 2:19 AM, David Chisnall
>> <david.chisnall at cl.cam.ac.uk> wrote:
>>> 
>>> On 29 Jul 2016, at 05:11, Mehdi Amini via llvm-dev <llvm-
>> dev at lists.llvm.org> wrote:
>>>> 
>>>> What I meant by “different problem" is that “downstream
users” for
>> instance don’t need to commit, that makes their problem/workflow quite
>> different from an upstream developer (for instance it is fairly easy to
>> maintain a read-only view of the existing individual git repo currently
on
>> llvm.org).
>>> 
>>> I’m not convinced by this distinction.  A lot of downstream
developers
>> need to patch LLVM and we benefit when they upstream their changes.
>> 
>> I made a difference between downstream users and developers. I.e.
someone
>> that just need to get and build compiler-rt vs someone that want to
>> *commit* to LLVM. Note that even by getting a single repo you can still
>> send a patch to the mailing list and someone can commit it for you
>> (including correct author attribution contrary to SVN).
>> 
>>> We should not make it harder for them to do this.  To give a couple
of
>> example downstream projects, both FreeBSD and Swift have patches on
LLVM /
>> Clang in their versions that they gradually filter upstream.  Both
>> projects have LLVM committers among their members.  If the workflow
that
>> we recommend for them makes upstreaming easy then they benefit
>> (maintaining a fork is effort) and LLVM benefits (having people provide
>> bug fixes makes our code better).
>>> 
>>> The workflow that we want to recommend to these people is:
>>> 
>>> - Fork the repo that you’re interested in from the LLVM GitHub
>> organisation
>>> - Make your changes
>>> - Send pull requests for anything that you think is of interest to
>> upstream
>> 
>> 
>> Note that the workflow you describe above still requires to export
their
>> patch and import it in this clone before pushing.
>> (Note also that we accept patches on the mailing list, so one does not
>> even need to clone the official repo).
>> 
>>> This makes the barrier to entry for sending code back upstream
*much*
>> lower than it currently is,
>> 
>> I don’t understand this statement. As of today you can send a diff to
the
>> mailing list, I don’t see how lower the bar can be.
>> 
>> 
>>> to the benefit of all.  If the alternative is:
>>> 
>>> - Fork a read-only repo that you’re interested in from the LLVM
GitHub
>> organisation
>>> - Make your changes
>> 
>> Why? If you know you want to *push* commits upstream, fork the only
useful
>> repo for that in the first place.
>> 
>>> - Fork a different repo from the LLVM GitHub organisation
>>> - Run a script to filter some of your changes into that one
>> 
>> I don’t know why you think there is a need for a script, or why it is
>> different from today.
>> Let say I’m working on a fork of the compiler-rt read-only repo and I
want
>> to upstream a patch at some point:
>> 
>> Today:
>> 
>> - cd /path/to/compiler_rt-forked
>> - git format-patch …
>> - cd /path/to/compiler_rt-upstream
>> - git am  /path/to/compiler_rt-forked/0001-My-awesome-changes.patch
>> - git svn dcommit
>> - done
>> 
>> Tomorrow with a monorepo:
>> 
>> - cd /path/to/compiler_rt-forked
>> - git format-patch …
>> - cd /path/to/unifiedrepo-upstream
>> - git am  /path/to/compiler_rt-forked/0001-My-awesome-changes.patch —
>> directory=compiler-rt
>> - git push
>> - done
>> 
>> Alternatively, if I’m upstream a patch once a year, I don’t really need
to
>> push it myself.
>> 
>> - cd /path/to/compiler_rt-forked
>> - git format-patch …
>> - email the patch.
>> 
>> 
>>> - Send a pull request from that
>> 
>> Note that I think we deferred any change to the workflow for future
>> discussions (pull-request are not part of our workflow today).
>> 
>>> - Deal with merging between the two yourself
>> 
>> I don’t know what you mean by dealing with the merging, I don’t expect
any
>> difficulties, you need to elaborate.
> 
> Almost no upstream patch survives contact with the reviewers unscathed
> (and quite often it doesn't look exactly like your local patch anyway,
> for reasons ranging from expediency to local coding standards).
> Therefore the patch that you merge back in to your local repo from 
> upstream differs from the one you already have locally.  If you are *very*
> lucky, the differences can be handled with accept-theirs; in my experience
> this is not normal, and in many cases you need to manually revert your
> original local patch before merging the patch as received from upstream.
> That's what "dealing with merging" means.  I invite you to
review
> "Living Downstream Without Drowning" if you still don't
understand;
Don’t worry, I’m living downstream, and perfectly aware of the situation.
(And I was in the room at the dev meeting).

> it describes a number of tactics we've developed over the years to
> try to simplify the dealing-with-merging problem.
> 
> I had two entire dev-meeting sessions full of people who seemed to be
> having this problem.  Maybe you don't; if so, you're lucky, but it
> doesn't mean the problem fails to exist.
The “problem” exists, it exists today, and it will tomorrow whatever solution we
go with (monorepo or split ones).

What I don’t see you addressing here is why this should be more of a problem in
the monorepo case (as it was implied in the email I was answering to).

— 
Mehdi

> --paulr
> 
>> 
>>> 
>>> I strongly suspect that we’ll get a lot fewer useful contributions
from
>> downstream.  Or downstream people will just work on the monorepo and
eat
>> the cost.
>>> 
>>> If someone is working on a downstream LLVM project and becoming
familiar
>> with our codebase, then we want them to be subtly nudging their
workflow
>> so that they eventually become LLVM contributors without noticing!
>> 
>> Sure. The distinction between “downstream users” and “developers” was
made
>> in response to “there exists many user that just download and build a
>> subproject”. These are not people that are *developing* on a downstream
>> fork.
>> 
>> —
>> Mehdi
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160729/5cb08a66/attachment.html>

> On Jul 29, 2016, at 10:48 AM, David Chisnall <David.Chisnall at
cl.cam.ac.uk> wrote:
> 
> On 29 Jul 2016, at 18:01, Mehdi Amini <mehdi.amini at apple.com>
wrote:
>> 
>> I don’t understand this statement. As of today you can send a diff to
the mailing list, I don’t see how lower the bar can be.
> 
> Really?  You don’t see that hitting the web UI buttons that construct a
pull request are lower than finding a mailing list to post to, subscribing,
preparing a patch, emailing it, filtering out the firehose of unrelated things? 
I migrated another open source project to GitHub last year.  After several years
of accepting patches via a mailing list and receiving no patches as a result, I
immediately started getting patches sent as pull requests.  I simply cannot
overstate how much easier this mechanism is for downstream users to push fixes
back, as long as the repo that they clone is the one that accepts pull requests.
Sure, I used Github in the past, and even set-up Jenkins bots that were testing
pull-request before merge. However changing the upstream LLVM workflow has been
deemed for future discussions, and I believe this is part of it. I.e. even if we
move to github, using pull-request is not a given.

— 
Mehdi