llvm dev - Feb 2017 - LLVM GSOC Projects Criteria Consultation (before 2/28)

Hello all,

GSOC is around the corner, and the LLVM projects plans to participate again this
year. For those who don’t know about GSOC, students are proposing a project that
they will work on for 3 months. Amongst other, one goal for LLVM is to mentor
students to become good developers and also contributors to the LLVM project (or
user/advocate of LLVM for building other cool projects).

A key part of the process is about how do we select the projects. The way we’ve
done it last year, is that the volunteer mentors shared an email thread and
ultimately each one individually ranked the projects. We kept the average
grading for each project to ranked them overall.

 In order to make the process more transparent to student applicants, we want to
formalize and announce the criteria for ranking and selection project.

The purpose of this email is to gather community feedback about the criterion
that mentors should apply when ranking projects, for example:

- Should we favor a student which has past contributions to LLVM compared to a
newcomer? Is it more important or as equally important as the quality of the
proposal?
- How should we rank (if any) “research or unbounded projects” vs “project with
an immediate application”? Should we strive to keep a balance?
- What about “projects that touch LLVM/Clang/…” vs “projects that are building
something on top of LLVM”? (For example this project was first proposed to be
selected by LLVM before being rejected and finally picked by the Julia project
directly:
https://summerofcode.withgoogle.com/archive/2016/projects/6197080536121344/ )?
- Should we ask that the work is done upstream all along? In the past there have
been project developed on GitHub outside the community that have never been
merged. The LLVM developer policy has a full section insisting on incremental
development ( http://llvm.org/docs/DeveloperPolicy.html#incremental-development
).

Hopefully we should be able to provide a set of guidelines to student that would
help them fill the best application, and avoid unfortunate surprise at the end
of the process.

We should have this discussion before receiving the projects proposals, which
opens on 2/28.

Best,

-- 
Mehdi

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170216/7079d1b5/attachment.html>

On 02/17/2017 12:16 AM, Mehdi Amini via lldb-dev wrote:
> Hello all,
>
> GSOC is around the corner, and the LLVM projects plans to participate 
> again this year. For those who don’t know about GSOC, students are 
> proposing a project that they will work on for 3 months. Amongst 
> other, one goal for LLVM is to mentor students to become good 
> developers and also contributors to the LLVM project (or user/advocate 
> of LLVM for building other cool projects).
>
> A key part of the process is about how do we select the projects. The 
> way we’ve done it last year, is that the volunteer mentors shared an 
> email thread and ultimately each one individually ranked the projects. 
> We kept the average grading for each project to ranked them overall.
>
> In order to make the process more transparent to student applicants, 
> we want to formalize and announce the criteria for ranking and 
> selection project.
>
> The purpose of this email is to gather community feedback about the 
> criterion that mentors should apply when ranking projects, for example:
>
> - Should we favor a student which has past contributions to LLVM 
> compared to a newcomer? Is it more important or as equally important 
> as the quality of the proposal?
I think that the quality of the proposal is the most important thing, 
however, we should be realistic about the amount of time it will take to 
bring a newcomer up to speed on our coding conventions, quality 
standards, review process, etc. and factor that into the decision-making 
process.
> - How should we rank (if any) “research or unbounded projects” vs 
> “project with an immediate application”? Should we strive to keep a 
> balance?
Given the short time period involved, I favor projects of more immediate 
utility. This does not mean that there can't be interesting open 
questions, but in our judgment, these should be answerable in a matter 
of weeks (i.e. choosing between clear alternatives, understood tuning 
parameters, and the like).
> - What about “projects that touch LLVM/Clang/…” vs “projects that are 
> building something on top of LLVM”? (For example this project was 
> first proposed to be selected by LLVM before being rejected and 
> finally picked by the Julia project directly: 
> https://summerofcode.withgoogle.com/archive/2016/projects/6197080536121344/
> )?
I think we need to base this on the overall benefit to the community. If 
working on a project built on top of LLVM will bring additional users, 
end up making LLVM more robust, etc. then there might be significant 
anticipated value. That having been said, in practice, justifying this 
value will probably be more difficult than for a project contributing to 
LLVM, Clang, etc.
> - Should we ask that the work is done upstream all along? In the past 
> there have been project developed on GitHub outside the community that 
> have never been merged. The LLVM developer policy has a full section 
> insisting on incremental development ( 
> http://llvm.org/docs/DeveloperPolicy.html#incremental-development ).
I think we should insist on incremental upstream development.

Thanks again,
Hal
>
> Hopefully we should be able to provide a set of guidelines to student 
> that would help them fill the best application, and avoid unfortunate 
> surprise at the end of the process.
>
> We should have this discussion before receiving the projects 
> proposals, which opens on 2/28.
>
> Best,
>
> -- 
> Mehdi
>
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170217/e5cd3ee4/attachment.html>

On Thu, Feb 16, 2017 at 10:16 PM, Mehdi Amini via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Hello all,
>
> GSOC is around the corner, and the LLVM projects plans to participate again
> this year. For those who don’t know about GSOC, students are proposing a
> project that they will work on for 3 months. Amongst other, one goal for
> LLVM is to mentor students to become good developers and also contributors
> to the LLVM project (or user/advocate of LLVM for building other cool
> projects).
>
> A key part of the process is about how do we select the projects. The way
> we’ve done it last year, is that the volunteer mentors shared an email
> thread and ultimately each one individually ranked the projects. We kept
the
> average grading for each project to ranked them overall.
>
> In order to make the process more transparent to student applicants, we
want
> to formalize and announce the criteria for ranking and selection project.
>
> The purpose of this email is to gather community feedback about the
> criterion that mentors should apply when ranking projects, for example:
>
> - Should we favor a student which has past contributions to LLVM compared
to
> a newcomer? Is it more important or as equally important as the quality of
> the proposal?
I really think that past contributions are *very* important and *a big plus*.
When I was involved in FreeBSD we ended up with people submitting
high-quality proposals, got accepted and then we realized at day 1 of
coding they didn't even set up a VM or knew how to build things. In
other words, I'm under the impression that students need to be well
aware of the development workflow and being able to show up some
involvement in the community.
> - How should we rank (if any) “research or unbounded projects” vs “project
> with an immediate application”? Should we strive to keep a balance?
Again, my opinion, but we should give high(er) priority to projects
which can produce something reasonable in 3 months.
If it's a research project, it's fine, but I think there should be a
bound (e.g. tune the thresholds for unrolling).
> - What about “projects that touch LLVM/Clang/…” vs “projects that are
> building something on top of LLVM”? (For example this project was first
> proposed to be selected by LLVM before being rejected and finally picked by
> the Julia project directly:
> https://summerofcode.withgoogle.com/archive/2016/projects/6197080536121344/
> )?
> - Should we ask that the work is done upstream all along? In the past there
> have been project developed on GitHub outside the community that have never
> been merged. The LLVM developer policy has a full section insisting on
> incremental development (
> http://llvm.org/docs/DeveloperPolicy.html#incremental-development ).
>
Yes, I think incremental patches are the way to go. If there are
bugfixes, etc.. they should be merged immediately (after code review).
If it's a larger chunk of work it should be in a state where others
can take over if the student stops being interested after the summer
(or we want to remove the thing altogether, e.g. a pass).
> Hopefully we should be able to provide a set of guidelines to student that
> would help them fill the best application, and avoid unfortunate surprise
at
> the end of the process.
>
> We should have this discussion before receiving the projects proposals,
> which opens on 2/28.
>
-- 
Davide

"There are no solved problems; there are only problems that are more
or less solved" -- Henri Poincare

On 2/17/17 1:16 AM, Mehdi Amini via lldb-dev wrote:
> Hello all,
>
> GSOC is around the corner, and the LLVM projects plans to participate 
> again this year. For those who don’t know about GSOC, students are 
> proposing a project that they will work on for 3 months. Amongst 
> other, one goal for LLVM is to mentor students to become good 
> developers and also contributors to the LLVM project (or user/advocate 
> of LLVM for building other cool projects).
>
> A key part of the process is about how do we select the projects. The 
> way we’ve done it last year, is that the volunteer mentors shared an 
> email thread and ultimately each one individually ranked the projects. 
> We kept the average grading for each project to ranked them overall.
>
> In order to make the process more transparent to student applicants, 
> we want to formalize and announce the criteria for ranking and 
> selection project.
>
> The purpose of this email is to gather community feedback about the 
> criterion that mentors should apply when ranking projects, for example:
>
> - Should we favor a student which has past contributions to LLVM 
> compared to a newcomer? Is it more important or as equally important 
> as the quality of the proposal?
Historically, we have opted for students with previous LLVM experience 
because they are more likely to make progress and complete their 
project.  For most projects, this makes sense. However, it does have the 
downside of attracting students that are already "on board" with LLVM;
it doesn't introduce new developers to the LLVM community.

We therefore might want to devise projects that would fit beginner 
developers.  For example, a proposal that aims to fix a set of bugs 
might work.
> - How should we rank (if any) “research or unbounded projects” vs 
> “project with an immediate application”? Should we strive to keep a 
> balance?
We should strive to keep a balance.  The important factor is getting 
students to either improve the existing LLVM software or to build cool 
tools using LLVM.  We want to grow the LLVM community, and both the 
software development and research communities are important.

I also believe that research projects and open-ended projects can 
provide long-term gains even if they don't provide immediate short-term 
benefits.  Remember that LLVM itself was once a research project.

If we keep a balance, then we will have a nice mix of short-term, low 
risk projects and projects that contribute to longer-term, higher risk 
goals.


> - What about “projects that touch LLVM/Clang/…” vs “projects that are 
> building something on top of LLVM”? (For example this project was 
> first proposed to be selected by LLVM before being rejected and 
> finally picked by the Julia project directly: 
> https://summerofcode.withgoogle.com/archive/2016/projects/6197080536121344/
> )?
We should encourage proposals that build something *with* 
LLVM/Clang/etc. as well as projects that improve the core compiler 
infrastructure.  Part of LLVM's appeal is that one can build *a lot* of 
tools with it.  We want to encourage students to use LLVM to build 
interesting things.

In some cases, we will have overlap with other communities (the Julia 
and FreeBSD communities come to mind).  I think that's fine.

> - Should we ask that the work is done upstream all along? In the past 
> there have been project developed on GitHub outside the community that 
> have never been merged. The LLVM developer policy has a full section 
> insisting on incremental development ( 
> http://llvm.org/docs/DeveloperPolicy.html#incremental-development ).
It depends on the project.  Projects that enhance the core 
infrastructure should probably follow the developer policy and 
contribute upstream.  Projects that are "trying out" a new idea or are
research-oriented should probably not worry about working upstream; why 
upstream if you don't even know if what you're trying to do is going to 
work?
>
> Hopefully we should be able to provide a set of guidelines to student 
> that would help them fill the best application, and avoid unfortunate 
> surprise at the end of the process.
I think what actually makes a strong proposal is whether the student 
presents a good idea, whether the proposal convinces the reader that the 
student will be able to perform the work, and whether there is a mentor 
interested in the project.  If you have these three things, then you 
have a project that is more likely to achieve its goals and grow the 
LLVM community.

Regards,

John Criswell

-- 
John Criswell
Assistant Professor
Department of Computer Science, University of Rochester
http://www.cs.rochester.edu/u/criswell

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170217/dd1c0bb8/attachment.html>

> On Feb 16, 2017, at 10:16 PM, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hello all,
> 
> GSOC is around the corner, and the LLVM projects plans to participate again
this year. For those who don’t know about GSOC, students are proposing a project
that they will work on for 3 months. Amongst other, one goal for LLVM is to
mentor students to become good developers and also contributors to the LLVM
project (or user/advocate of LLVM for building other cool projects).
> 
> A key part of the process is about how do we select the projects. The way
we’ve done it last year, is that the volunteer mentors shared an email thread
and ultimately each one individually ranked the projects. We kept the average
grading for each project to ranked them overall.
> 
>  In order to make the process more transparent to student applicants, we
want to formalize and announce the criteria for ranking and selection project.
> 
> The purpose of this email is to gather community feedback about the
criterion that mentors should apply when ranking projects, for example:
> 
> - Should we favor a student which has past contributions to LLVM compared
to a newcomer? Is it more important or as equally important as the quality of
the proposal?
While students with previous LLVM dev experience will be more productive, GSoC
is a great way of attracting newcomers to the project! So if a proposal is
scoped to be completed in time even by someone not familiar with LLVM, we should
still accept it.
> - How should we rank (if any) “research or unbounded projects” vs “project
with an immediate application”? Should we strive to keep a balance?
My preference would be to give priority to projects that are on a trajectory to
have practical benefit.
> - What about “projects that touch LLVM/Clang/…” vs “projects that are
building something on top of LLVM”? (For example this project was first proposed
to be selected by LLVM before being rejected and finally picked by the Julia
project directly:
https://summerofcode.withgoogle.com/archive/2016/projects/6197080536121344/
<https://summerofcode.withgoogle.com/archive/2016/projects/6197080536121344/>
)?
> - Should we ask that the work is done upstream all along? In the past there
have been project developed on GitHub outside the community that have never been
merged. The LLVM developer policy has a full section insisting on incremental
development ( http://llvm.org/docs/DeveloperPolicy.html#incremental-development
<http://llvm.org/docs/DeveloperPolicy.html#incremental-development> ).
An invaluable part of participating in GSoC is for students to learn about how
open source projects work. Following the developer policy is one of the main
aspects of development in these codebases. In addition, if a project is being
developed on a branch it is likely not to get merged and the impact of the
student, the mentor, as well as the GSoC budget will be minimal.
> Hopefully we should be able to provide a set of guidelines to student that
would help them fill the best application, and avoid unfortunate surprise at the
end of the process.
> 
> We should have this discussion before receiving the projects proposals,
which opens on 2/28.
> 
> Best,
> 
> -- 
> Mehdi
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170217/cb0e2a8b/attachment-0001.html>

Agreed. I think it's worth thinking about what GSoC is actually about,
which according to the website: "Google Summer of Code is a global
program focused on introducing students to open source software
development."

Strongly favouring students with prior experience in community
involvement with LLVM is somewhat missing the point, and I think we
should be mindful of the differences in opportunities that some
students have had.

Amara

On 18 February 2017 at 02:10, Anna Zaks via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> On Feb 16, 2017, at 10:16 PM, Mehdi Amini via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>
> Hello all,
>
> GSOC is around the corner, and the LLVM projects plans to participate again
> this year. For those who don’t know about GSOC, students are proposing a
> project that they will work on for 3 months. Amongst other, one goal for
> LLVM is to mentor students to become good developers and also contributors
> to the LLVM project (or user/advocate of LLVM for building other cool
> projects).
>
> A key part of the process is about how do we select the projects. The way
> we’ve done it last year, is that the volunteer mentors shared an email
> thread and ultimately each one individually ranked the projects. We kept
the
> average grading for each project to ranked them overall.
>
> In order to make the process more transparent to student applicants, we
want
> to formalize and announce the criteria for ranking and selection project.
>
> The purpose of this email is to gather community feedback about the
> criterion that mentors should apply when ranking projects, for example:
>
> - Should we favor a student which has past contributions to LLVM compared
to
> a newcomer? Is it more important or as equally important as the quality of
> the proposal?
>
>
> While students with previous LLVM dev experience will be more productive,
> GSoC is a great way of attracting newcomers to the project! So if a
proposal
> is scoped to be completed in time even by someone not familiar with LLVM,
we
> should still accept it.
>
> - How should we rank (if any) “research or unbounded projects” vs “project
> with an immediate application”? Should we strive to keep a balance?
>
>
> My preference would be to give priority to projects that are on a
trajectory
> to have practical benefit.
>
> - What about “projects that touch LLVM/Clang/…” vs “projects that are
> building something on top of LLVM”? (For example this project was first
> proposed to be selected by LLVM before being rejected and finally picked by
> the Julia project directly:
> https://summerofcode.withgoogle.com/archive/2016/projects/6197080536121344/
> )?
>
> - Should we ask that the work is done upstream all along? In the past there
> have been project developed on GitHub outside the community that have never
> been merged. The LLVM developer policy has a full section insisting on
> incremental development (
> http://llvm.org/docs/DeveloperPolicy.html#incremental-development ).
>
>
> An invaluable part of participating in GSoC is for students to learn about
> how open source projects work. Following the developer policy is one of the
> main aspects of development in these codebases. In addition, if a project
is
> being developed on a branch it is likely not to get merged and the impact
of
> the student, the mentor, as well as the GSoC budget will be minimal.
>
> Hopefully we should be able to provide a set of guidelines to student that
> would help them fill the best application, and avoid unfortunate surprise
at
> the end of the process.
>
> We should have this discussion before receiving the projects proposals,
> which opens on 2/28.
>
> Best,
>
> --
> Mehdi
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

On 18 Feb 2017, at 02:10, Anna Zaks via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> While students with previous LLVM dev experience will be more productive,
GSoC is a great way of attracting newcomers to the project! So if a proposal is
scoped to be completed in time even by someone not familiar with LLVM, we should
still accept it.
I’d also add, from having mentored GSoC projects for several open source
projects, that there are two ways of considering success for a GSoC project:

1. The student completes the work and it’s incorporated into trunk.
2. The student becomes engaged with the project and continues to contribute
after the end of the GSoC.

The former treats the GSoC as an expensive way of getting a particular feature
added (yes, Google pays the student, but generally the mentor time comes from
somewhere and that’s not free, it’s time that would otherwise be spent
contributing to the project in another way).  The second treats the GSoC as a
powerful recruitment tool.  I’ve had a couple of students that succeeded in the
first way, but didn’t really benefit the project, because they left at the end
of the GSoC.  I’ve also had a few that managed some proof-of-concept code that
ended up showing that the entire GSoC project idea was the wrong approach, but
then remained active contributors for years and ended up contributing far more
(good) code than I’d have written if I’d spent the time that I spent mentoring
them on hacking.

I would strongly encourage treating the GSoC as a way of engaging students.  I’d
also hope that the Foundation’s Women in Compilers and Tools project will help
encourage this kind of participation in a way that meshes well with the GSoC.

David