llvm dev - Nov 2016 - RFC: Improving the experience of first-time contributors

Hi all,

Some discussions the night before prompted me to do a short lightning
talk on 'improving the experience of first-time contributors' at the
LLVM Cauldron back in September. I intended to write it up as an RFC,
but have only just got round to doing so. I've tried to make this
email self-contained, but you may still want to look at the slides or
recording of the lightning talk.

Slides:
https://speakerdeck.com/asb/can-we-improve-the-experience-of-first-time-llvm-contributors
Video: https://www.youtube.com/watch?v=32fmbEI9WrM

# The problem
As a first-time contributor, it's easy to be put off during the patch
submission process. e.g. the patch may sit unreviewed for weeks.
Typically people new to the project haven't yet reviewed 'review
currency', and so their patches are less likely to be picked out by
others scanning through llvm-commits. Obviously nobody is doing
anything wrong here, it's just likely to be off-putting to newcomers.
Even if feedback is negative, it's valuable to know someone has at
least looked at your code.

# How do other projects solve this?
I've always been impressed at how Rust has been able to attract a
steady stream of new contributors. They have a simple but seemingly
effective tool for helping people land their first patch: the highfive
bot. Upon a user making their first pull request, it posts a welcome
message and assigns someone to review it.

# A proposal
I propose that we form a group of people who specifically volunteer to
help review first-time patches. A simple phabricator bot could assign
someone randomly much like Rust's highfive.

One potential pitfall I'd like to highlight is that many patches
really require the sign-off of the code owner. It could be equally
frustrating for newcomers if people helpfully tell them to tweak their
variable names, fix their formatting, and make other cosmetic changes
only to find that time was wasted when the code owner step in a week
or two later to say a completely different approach needs to be used.

# Next steps

Does anybody have any views here? My 'raise your hand' poll at the
LLVM Cauldron did seem to think many felt things could be improved.
I'll be at the LLVM Dev Meeting this week and would love to discuss
further. Perhaps the BoF on raising the next generation of LLVM
developers would be an appropriate venue?
http://lists.llvm.org/pipermail/llvm-dev/2016-October/106658.html

Best,

Alex

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
Alex
> Bradbury via llvm-dev
> Sent: Tuesday, November 01, 2016 6:17 AM
> To: llvm-dev
> Subject: [llvm-dev] RFC: Improving the experience of first-time
> contributors
> 
> Hi all,
> 
> Some discussions the night before prompted me to do a short lightning
> talk on 'improving the experience of first-time contributors' at
the
> LLVM Cauldron back in September. I intended to write it up as an RFC,
> but have only just got round to doing so. I've tried to make this
> email self-contained, but you may still want to look at the slides or
> recording of the lightning talk.
> 
> Slides: https://speakerdeck.com/asb/can-we-improve-the-experience-of-
> first-time-llvm-contributors
> Video: https://www.youtube.com/watch?v=32fmbEI9WrM
> 
> # The problem
> As a first-time contributor, it's easy to be put off during the patch
> submission process. e.g. the patch may sit unreviewed for weeks.
> Typically people new to the project haven't yet reviewed 'review
> currency', and so their patches are less likely to be picked out by
> others scanning through llvm-commits. Obviously nobody is doing
> anything wrong here, it's just likely to be off-putting to newcomers.
> Even if feedback is negative, it's valuable to know someone has at
> least looked at your code.
It's not just first-time contributors who can have this problem...
...but it can be especially off-putting to first-timers.
> 
> # How do other projects solve this?
> I've always been impressed at how Rust has been able to attract a
> steady stream of new contributors. They have a simple but seemingly
> effective tool for helping people land their first patch: the highfive
> bot. Upon a user making their first pull request, it posts a welcome
> message and assigns someone to review it.
Sweet.
> 
> # A proposal
> I propose that we form a group of people who specifically volunteer to
> help review first-time patches. A simple phabricator bot could assign
> someone randomly much like Rust's highfive.
That's an awesome idea for people who do actually post a patch to
phabricator.  I'll just point out that phabricator isn't officially
required, and is in fact an extra barrier to entry, so we should
have some tactic for handling emailed patches if we can manage it.

I know people will react with "what barrier?" but it's YADA
(Yet Another Damn Account) and I know *I* hate doing that for
something I view as a one-off.
> 
> One potential pitfall I'd like to highlight is that many patches
> really require the sign-off of the code owner. It could be equally
> frustrating for newcomers if people helpfully tell them to tweak their
> variable names, fix their formatting, and make other cosmetic changes
> only to find that time was wasted when the code owner step in a week
> or two later to say a completely different approach needs to be used.
People who sign up to be first-responders :-) should probably have
some sense of when patches are of that nature, and say so in their
initial response.  If nothing else, cosmetic feedback will get new
contributors in tune with our style guide and make subsequent rounds 
feel more substantive/less picky.
> 
> # Next steps
> 
> Does anybody have any views here? My 'raise your hand' poll at the
> LLVM Cauldron did seem to think many felt things could be improved.
Agreed.  Partly it's the raw size of the project; when I was on a
two-person team it was never a problem to get reviews done, but
with a dozen or more and looking for someone to volunteer, there's
a significant "somebody else will get to it" factor.  And then nobody
does.  With volunteer first-responders it's something they've signed
up to do, however, and that makes a big difference.
--paulr
> I'll be at the LLVM Dev Meeting this week and would love to discuss
> further. Perhaps the BoF on raising the next generation of LLVM
> developers would be an appropriate venue?
> http://lists.llvm.org/pipermail/llvm-dev/2016-October/106658.html
> 
> Best,
> 
> Alex
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

> On Nov 1, 2016, at 6:17 AM, Alex Bradbury via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi all,
> 
> Some discussions the night before prompted me to do a short lightning
> talk on 'improving the experience of first-time contributors' at
the
> LLVM Cauldron back in September. I intended to write it up as an RFC,
> but have only just got round to doing so. I've tried to make this
> email self-contained, but you may still want to look at the slides or
> recording of the lightning talk.
> 
> Slides:
https://speakerdeck.com/asb/can-we-improve-the-experience-of-first-time-llvm-contributors
> Video: https://www.youtube.com/watch?v=32fmbEI9WrM
<https://www.youtube.com/watch?v=32fmbEI9WrM>
Great initiative! Thanks.

Have you seen this past discussion:
http://lists.llvm.org/pipermail/llvm-dev/2016-March/097027.html
I think the main idea has been to set-up a regular IRC session dedicated to “new
contributors”.
> 
> # The problem
> As a first-time contributor, it's easy to be put off during the patch
> submission process. e.g. the patch may sit unreviewed for weeks.
> Typically people new to the project haven't yet reviewed 'review
> currency', and so their patches are less likely to be picked out by
> others scanning through llvm-commits. Obviously nobody is doing
> anything wrong here, it's just likely to be off-putting to newcomers.
> Even if feedback is negative, it's valuable to know someone has at
> least looked at your code.
> 
> # How do other projects solve this?
> I've always been impressed at how Rust has been able to attract a
> steady stream of new contributors. They have a simple but seemingly
> effective tool for helping people land their first patch: the highfive
> bot. Upon a user making their first pull request, it posts a welcome
> message and assigns someone to review it.
> 
> # A proposal
> I propose that we form a group of people who specifically volunteer to
> help review first-time patches. A simple phabricator bot could assign
> someone randomly much like Rust's highfive.
> 
> One potential pitfall I'd like to highlight is that many patches
> really require the sign-off of the code owner.
I don’t believe it is exact, unless you used “code owner” in a broader sense
that what we mean usually with CODE_OWNERS.txt ; as you can see here:
http://llvm.org/docs/DeveloperPolicy.html#code-owners "The sole
responsibility of a code owner is to ensure that a commit to their area of the
code is appropriately reviewed, either by themself or by someone else."
You may need a sign-off from a “core contributor” well aware of the particular
area you’re touching. And even that depends on the level of the changes: a small
bug fix is less involved than something that requires large change or involves
some design.

> It could be equally
> frustrating for newcomers if people helpfully tell them to tweak their
> variable names, fix their formatting, and make other cosmetic changes
> only to find that time was wasted when the code owner step in a week
> or two later to say a completely different approach needs to be used.
I hope the “group of volunteers” will be well aware of LLVM and be able to spot
when they need to pull-in someone else for a particular review (e.g. an
important change in the new pass manager is likely to require Chandler to
sign-off).

Best,

— 
Mehdi


> 
> # Next steps
> 
> Does anybody have any views here? My 'raise your hand' poll at the
> LLVM Cauldron did seem to think many felt things could be improved.
> I'll be at the LLVM Dev Meeting this week and would love to discuss
> further. Perhaps the BoF on raising the next generation of LLVM
> developers would be an appropriate venue?
> http://lists.llvm.org/pipermail/llvm-dev/2016-October/106658.html
> 
> Best,
> 
> Alex
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161101/62717283/attachment.html>

On Tue, Nov 1, 2016 at 2:17 PM Alex Bradbury via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi all,
>
> Some discussions the night before prompted me to do a short lightning
> talk on 'improving the experience of first-time contributors' at
the
> LLVM Cauldron back in September. I intended to write it up as an RFC,
> but have only just got round to doing so. I've tried to make this
> email self-contained, but you may still want to look at the slides or
> recording of the lightning talk.
>
> Slides:
>
https://speakerdeck.com/asb/can-we-improve-the-experience-of-first-time-llvm-contributors
> Video: https://www.youtube.com/watch?v=32fmbEI9WrM
>
> # The problem
> As a first-time contributor, it's easy to be put off during the patch
> submission process. e.g. the patch may sit unreviewed for weeks.
> Typically people new to the project haven't yet reviewed 'review
> currency', and so their patches are less likely to be picked out by
> others scanning through llvm-commits. Obviously nobody is doing
> anything wrong here, it's just likely to be off-putting to newcomers.
> Even if feedback is negative, it's valuable to know someone has at
> least looked at your code.
>
> # How do other projects solve this?
> I've always been impressed at how Rust has been able to attract a
> steady stream of new contributors. They have a simple but seemingly
> effective tool for helping people land their first patch: the highfive
> bot. Upon a user making their first pull request, it posts a welcome
> message and assigns someone to review it.
>
On GitHub there is a project called "mention bot" [1] by Facebook that
assigns reviewers by analysing the history of the files touched by the
patch. Recently I was welcomed by a message: "thanks for your PR! By
analyzing the history of the files in this pull request, we identified @A,
@B and @C to be potential reviewers."

[1] https://github.com/facebook/mention-bot

>
> # A proposal
> I propose that we form a group of people who specifically volunteer to
> help review first-time patches. A simple phabricator bot could assign
> someone randomly much like Rust's highfive.
>
> One potential pitfall I'd like to highlight is that many patches
> really require the sign-off of the code owner. It could be equally
> frustrating for newcomers if people helpfully tell them to tweak their
> variable names, fix their formatting, and make other cosmetic changes
> only to find that time was wasted when the code owner step in a week
> or two later to say a completely different approach needs to be used.
>
> # Next steps
>
> Does anybody have any views here? My 'raise your hand' poll at the
> LLVM Cauldron did seem to think many felt things could be improved.
> I'll be at the LLVM Dev Meeting this week and would love to discuss
> further. Perhaps the BoF on raising the next generation of LLVM
> developers would be an appropriate venue?
> http://lists.llvm.org/pipermail/llvm-dev/2016-October/106658.html
>
> Best,
>
> Alex
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161101/ff080f0a/attachment.html>

On 1 November 2016 at 16:42, Mehdi Amini <mehdi.amini at apple.com>
wrote:
> Have you seen this past discussion:
> http://lists.llvm.org/pipermail/llvm-dev/2016-March/097027.html
> I think the main idea has been to set-up a regular IRC session dedicated to
> “new contributors”.
Thanks. I don't think I had looked at that discussion in detail.
> I don’t believe it is exact, unless you used “code owner” in a broader
sense
> that what we mean usually with CODE_OWNERS.txt ; as you can see here:
> http://llvm.org/docs/DeveloperPolicy.html#code-owners "The sole
> responsibility of a code owner is to ensure that a commit to their area of
> the code is appropriately reviewed, either by themself or by someone
else."
> You may need a sign-off from a “core contributor” well aware of the
> particular area you’re touching. And even that depends on the level of the
> changes: a small bug fix is less involved than something that requires
large
> change or involves some design.
Yes, I simply meant in a broader sense - just like the example you
give below of getting sign-off from Chandler for the pass manager.

> I hope the “group of volunteers” will be well aware of LLVM and be able to
> spot when they need to pull-in someone else for a particular review (e.g.
an
> important change in the new pass manager is likely to require Chandler to
> sign-off).
I agree, it is an easily solved problem. Just something I thought I'd
highlight. Another reason for mentioning it is that if the fundamental
issue is that code owners who are needed to provide sign-off are
fundamentally too busy to review, then having this 'first responder'
team won't help in those cases.

Best,

Alex

----- Original Message -----
> From: "Paweł Bylica via llvm-dev" <llvm-dev at
lists.llvm.org>
> To: "Alex Bradbury" <asb at asbradbury.org>,
"llvm-dev"
> <llvm-dev at lists.llvm.org>
> Sent: Tuesday, November 1, 2016 12:02:12 PM
> Subject: Re: [llvm-dev] RFC: Improving the experience of first-time
> contributors
> On Tue, Nov 1, 2016 at 2:17 PM Alex Bradbury via llvm-dev <
> llvm-dev at lists.llvm.org > wrote:
> > Hi all,
> 
> > Some discussions the night before prompted me to do a short
> > lightning
> 
> > talk on 'improving the experience of first-time contributors'
at
> > the
> 
> > LLVM Cauldron back in September. I intended to write it up as an
> > RFC,
> 
> > but have only just got round to doing so. I've tried to make this
> 
> > email self-contained, but you may still want to look at the slides
> > or
> 
> > recording of the lightning talk.
> 
> > Slides:
> >
https://speakerdeck.com/asb/can-we-improve-the-experience-of-first-time-llvm-contributors
> 
> > Video: https://www.youtube.com/watch?v=32fmbEI9WrM
> 
> > # The problem
> 
> > As a first-time contributor, it's easy to be put off during the
> > patch
> 
> > submission process. e.g. the patch may sit unreviewed for weeks.
> 
> > Typically people new to the project haven't yet reviewed
'review
> 
> > currency', and so their patches are less likely to be picked out
by
> 
> > others scanning through llvm-commits. Obviously nobody is doing
> 
> > anything wrong here, it's just likely to be off-putting to
> > newcomers.
> 
> > Even if feedback is negative, it's valuable to know someone has at
> 
> > least looked at your code.
> 
> > # How do other projects solve this?
> 
> > I've always been impressed at how Rust has been able to attract a
> 
> > steady stream of new contributors. They have a simple but seemingly
> 
> > effective tool for helping people land their first patch: the
> > highfive
> 
> > bot. Upon a user making their first pull request, it posts a
> > welcome
> 
> > message and assigns someone to review it.
> 
> On GitHub there is a project called "mention bot" [1] by Facebook
> that assigns reviewers by analysing the history of the files touched
> by the patch. Recently I was welcomed by a message: "thanks for your
> PR! By analyzing the history of the files in this pull request, we
> identified @A, @B and @C to be potential reviewers."
> [1] https://github.com/facebook/mention-bot
That sounds really nice! 

-Hal 
> > # A proposal
> 
> > I propose that we form a group of people who specifically volunteer
> > to
> 
> > help review first-time patches. A simple phabricator bot could
> > assign
> 
> > someone randomly much like Rust's highfive.
> 
> > One potential pitfall I'd like to highlight is that many patches
> 
> > really require the sign-off of the code owner. It could be equally
> 
> > frustrating for newcomers if people helpfully tell them to tweak
> > their
> 
> > variable names, fix their formatting, and make other cosmetic
> > changes
> 
> > only to find that time was wasted when the code owner step in a
> > week
> 
> > or two later to say a completely different approach needs to be
> > used.
> 
> > # Next steps
> 
> > Does anybody have any views here? My 'raise your hand' poll at
the
> 
> > LLVM Cauldron did seem to think many felt things could be improved.
> 
> > I'll be at the LLVM Dev Meeting this week and would love to
discuss
> 
> > further. Perhaps the BoF on raising the next generation of LLVM
> 
> > developers would be an appropriate venue?
> 
> > http://lists.llvm.org/pipermail/llvm-dev/2016-October/106658.html
> 
> > Best,
> 
> > Alex
> 
> > _______________________________________________
> 
> > LLVM Developers mailing list
> 
> > llvm-dev at lists.llvm.org
> 
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-- 

Hal Finkel 
Lead, Compiler Technology and Programming Languages 
Leadership Computing Facility 
Argonne National Laboratory 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161101/d5392b3e/attachment.html>

On 1 November 2016 at 13:17, Alex Bradbury <asb at asbradbury.org>
wrote:
> Hi all,
>
> Some discussions the night before prompted me to do a short lightning
> talk on 'improving the experience of first-time contributors' at
the
> LLVM Cauldron back in September. I intended to write it up as an RFC,
> but have only just got round to doing so. I've tried to make this
> email self-contained, but you may still want to look at the slides or
> recording of the lightning talk.
>
> Slides:
https://speakerdeck.com/asb/can-we-improve-the-experience-of-first-time-llvm-contributors
> Video: https://www.youtube.com/watch?v=32fmbEI9WrM
+Chandler, who has apparently been thinking along similar lines.

One point I'd like to reiterate is that this proposal is not the same
as just making it easier to select a reviewer. It also doesn't try to
solve the review problem in general, simply to pick one area where
slow reviews are likely to be most painful (first time contributors
being put off) and to try to improve it through a concerted group
effort. I like the 'first responders' name from Paul.

Best,

Alex

Hi,
The process of finding out something to work on that will be welcome
upstream is also somewhat tricky.
I would appreciate an effort towards streamlining that.
Manasij

On Fri, Nov 4, 2016 at 4:10 AM, Alex Bradbury via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 1 November 2016 at 13:17, Alex Bradbury <asb at asbradbury.org>
wrote:
> > Hi all,
> >
> > Some discussions the night before prompted me to do a short lightning
> > talk on 'improving the experience of first-time contributors'
at the
> > LLVM Cauldron back in September. I intended to write it up as an RFC,
> > but have only just got round to doing so. I've tried to make this
> > email self-contained, but you may still want to look at the slides or
> > recording of the lightning talk.
> >
> > Slides: https://speakerdeck.com/asb/can-we-improve-the-experience-
> of-first-time-llvm-contributors
> > Video: https://www.youtube.com/watch?v=32fmbEI9WrM
>
> +Chandler, who has apparently been thinking along similar lines.
>
> One point I'd like to reiterate is that this proposal is not the same
> as just making it easier to select a reviewer. It also doesn't try to
> solve the review problem in general, simply to pick one area where
> slow reviews are likely to be most painful (first time contributors
> being put off) and to try to improve it through a concerted group
> effort. I like the 'first responders' name from Paul.
>
> Best,
>
> Alex
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161104/1d6c2f9b/attachment.html>

On 11/3/2016 5:40 PM, Alex Bradbury via llvm-dev wrote:
>
> One point I'd like to reiterate is that this proposal is not the same
> as just making it easier to select a reviewer. It also doesn't try to
> solve the review problem in general, simply to pick one area where
> slow reviews are likely to be most painful (first time contributors
> being put off) and to try to improve it through a concerted group
> effort. I like the 'first responders' name from Paul.
I think that addressing slow reviews would be the one thing that would 
have the most impact. A timely response of any kind is better than no 
response at all.


-Krzysztof