llvm dev - May 2016 - [RFC] Helping release management

> On May 4, 2016, at 4:41 PM, Jim Grosbach <grosbach at apple.com>
wrote:
> 
>> 
>> On May 2, 2016, at 4:03 PM, Quentin Colombet via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> Hi Hans,
>> 
>> Since you are actively doing this kind of things, your feedbacks is
particularly valuable.
>> Thanks!
>> 
>>> On May 2, 2016, at 3:45 PM, Hans Wennborg <hans at chromium.org
<mailto:hans at chromium.org>> wrote:
>>> 
>>> On Mon, May 2, 2016 at 1:35 PM, Quentin Colombet via llvm-dev
>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>>> I am sending this proposal to get feedbacks on how we could
make the tagging of bug fixes and regressions more obvious. The idea is to
provide easily accessible information to help deciding what to cherry-pick in a
release branch.
>>>> 
>>>> * Context *
>>>> 
>>>> People shipping compilers based on LLVM may not completely
align with the official releases of LLVM. Thus, the stabilization of each custom
release may happen at different period of time. Because of that, release
managers have to come up with their own strategy to decide which commits should
be cherry-picked during the stabilization of their release branch.
>>> 
>>> (Unrelated to your proposal, I'm curious how common it is to
base
>>> releases of LLVM-based tools off the upstream release branches vs.
>>> other revisions.)
>>> 
>>>> For the official LLVM releases, people (committers, code
owners, etc.) notice LLVM release managers that a given commit is worth pulling
into the release. I would like to put in place something more systematic and
that plays nicely with scripting and such that would extend this mechanism.
>>>> 
>>>> * Proposal *
>>>> 
>>>> 1. Use [Fix] for commit related to bug fixes.
>>> 
>>> I think we're mostly pretty good at referencing PR's in
commit
>>> messages already. That's also easy to grep for, so maybe
that's good
>>> enough?
>> 
>> When a PR is available, that is certainly good enough.
>> I do not want to make filling a PR mandatory for each bug we fix
though. Having a PR is great, but I can see why we may not want to create one
each time we fix something.
> 
> Why not?
I should have phrase it differently :).
I do not want to make filing a PR *manually* mandatory for each bug we fix.

Typically, what I had in mind was things like typos/thinko, that are bugs, that
we notice a few minutes after we made the “main” commit. I do not want we have
to file a PR that is going to repeat what we are going to say in the commit
message.

Ultimately, yes, we should have a PR for that, but it should not be in the way
of our productivity, which leads me to your second comment
:).
> 
>> 
>> If we do want to go into that direction though, I believe we would need
to provide more support to make that easier. (E.g. filing via command line,
getting a number back and feeding this number to a commit.)
> 
> Making our bug tracking system and related tools better seems independently
good.
Agreed. Having better tools could be a way to have a PR for every fixes we do,
without actually having to file one. Like I know I am fixing a bug I introduced
in my last commit, please awesome_command_line_tool™ file a PR alongside with
the commit message.
> 
>> 
>>> 
>>>> 2. Add a description of the problem in the commit message to
help answer the following questions:
>>>> - What is fixed?
>>>> - Which targets are impacted?
>>>> - What is required to trigger the bug? (I.e., how often the end
users may encounter it.)
>>>> - When was the bug introduced?
>>> 
>>> This sounds like the kind of information that should be in a great
>>> commit message anyways.
>>> 
>>> But I'm also thinking that maybe we could be better at using
our bug
>>> tracker? Whether a bug is a feature request, something that was
always
>>> broken, or a regression (and from what version), sounds like a
perfect
>>> fit for a bug tracker. Someone doing a release could then query the
>>> Bugzilla to see e.g. what regression bugs were fixed in a certain
time
>>> span.
>> 
>> Sounds great to me.
>> 
>> Would that kind of workflow ease your job for tracking what should be
pulled into the release branch after you’ve branched?
>> What would be the best workflow for such task for you?
>> 
>>> 
>>>> #1 At the very least, I would like that each bug fix has a tag
on the first line of the commit (i.e., what ends up in the subject line of the
related email.) Something like [Fix] would do.
>>>> Thanks to that tag, it would be possible to easily filter bug
fixes in email and other cherry-picking helper tools, I believe.
>>> 
>>> If we really do want to make a guideline about this, I propose we
>>> standardize on suffixing the first line of the commit with (PRnnn).
>> 
>> That would work for me.
>> 
>> Anywhere in the commit message would work as well (like Mehdi and Hal
said), though I tend to prefer the first line (but Mehdi does not like it :)).
>> 
>> Cheers,
>> -Quentin
>> 
>>> 
>>> Cheers,
>>> Hans
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160504/14db3233/attachment.html>

> On May 4, 2016, at 5:07 PM, Quentin Colombet <qcolombet at apple.com>
wrote:
> 
>> 
>> On May 4, 2016, at 4:41 PM, Jim Grosbach <grosbach at apple.com
<mailto:grosbach at apple.com>> wrote:
>> 
>>> 
>>> On May 2, 2016, at 4:03 PM, Quentin Colombet via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>>> 
>>> Hi Hans,
>>> 
>>> Since you are actively doing this kind of things, your feedbacks is
particularly valuable.
>>> Thanks!
>>> 
>>>> On May 2, 2016, at 3:45 PM, Hans Wennborg <hans at
chromium.org <mailto:hans at chromium.org>> wrote:
>>>> 
>>>> On Mon, May 2, 2016 at 1:35 PM, Quentin Colombet via llvm-dev
>>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>>>> I am sending this proposal to get feedbacks on how we could
make the tagging of bug fixes and regressions more obvious. The idea is to
provide easily accessible information to help deciding what to cherry-pick in a
release branch.
>>>>> 
>>>>> * Context *
>>>>> 
>>>>> People shipping compilers based on LLVM may not completely
align with the official releases of LLVM. Thus, the stabilization of each custom
release may happen at different period of time. Because of that, release
managers have to come up with their own strategy to decide which commits should
be cherry-picked during the stabilization of their release branch.
>>>> 
>>>> (Unrelated to your proposal, I'm curious how common it is
to base
>>>> releases of LLVM-based tools off the upstream release branches
vs.
>>>> other revisions.)
>>>> 
>>>>> For the official LLVM releases, people (committers, code
owners, etc.) notice LLVM release managers that a given commit is worth pulling
into the release. I would like to put in place something more systematic and
that plays nicely with scripting and such that would extend this mechanism.
>>>>> 
>>>>> * Proposal *
>>>>> 
>>>>> 1. Use [Fix] for commit related to bug fixes.
>>>> 
>>>> I think we're mostly pretty good at referencing PR's in
commit
>>>> messages already. That's also easy to grep for, so maybe
that's good
>>>> enough?
>>> 
>>> When a PR is available, that is certainly good enough.
>>> I do not want to make filling a PR mandatory for each bug we fix
though. Having a PR is great, but I can see why we may not want to create one
each time we fix something.
>> 
>> Why not?
> 
> I should have phrase it differently :).
> I do not want to make filing a PR *manually* mandatory for each bug we fix.
> 
> Typically, what I had in mind was things like typos/thinko, that are bugs,
that we notice a few minutes after we made the “main” commit. I do not want we
have to file a PR that is going to repeat what we are going to say in the commit
message.
> 
> Ultimately, yes, we should have a PR for that, but it should not be in the
way of our productivity, which leads me to your second comment :).
>> 
>>> 
>>> If we do want to go into that direction though, I believe we would
need to provide more support to make that easier. (E.g. filing via command line,
getting a number back and feeding this number to a commit.)
>> 
>> Making our bug tracking system and related tools better seems
independently good.
> 
> Agreed. Having better tools could be a way to have a PR for every fixes we
do, without actually having to file one. Like I know I am fixing a bug I
introduced in my last commit, please awesome_command_line_tool™ file a PR
alongside with the commit message.
Aha, yes. We are in essential agreement. Our tools should be tailored so the
path of least resistance is the path we actually want people to follow. If
that’s to have a PR for all the fixes, then our tools need to make that even
easier than not having a PR. Likewise, we’ll need to make sure the bug database
can cope with the non-trivial increase in traffic such a thing would cause.
We’ll need to make sure the usability of the database is not degraded, but
rather is improved since it will be a more central part of the normal workflow.
> 
>> 
>>> 
>>>> 
>>>>> 2. Add a description of the problem in the commit message
to help answer the following questions:
>>>>> - What is fixed?
>>>>> - Which targets are impacted?
>>>>> - What is required to trigger the bug? (I.e., how often the
end users may encounter it.)
>>>>> - When was the bug introduced?
>>>> 
>>>> This sounds like the kind of information that should be in a
great
>>>> commit message anyways.
>>>> 
>>>> But I'm also thinking that maybe we could be better at
using our bug
>>>> tracker? Whether a bug is a feature request, something that was
always
>>>> broken, or a regression (and from what version), sounds like a
perfect
>>>> fit for a bug tracker. Someone doing a release could then query
the
>>>> Bugzilla to see e.g. what regression bugs were fixed in a
certain time
>>>> span.
>>> 
>>> Sounds great to me.
>>> 
>>> Would that kind of workflow ease your job for tracking what should
be pulled into the release branch after you’ve branched?
>>> What would be the best workflow for such task for you?
>>> 
>>>> 
>>>>> #1 At the very least, I would like that each bug fix has a
tag on the first line of the commit (i.e., what ends up in the subject line of
the related email.) Something like [Fix] would do.
>>>>> Thanks to that tag, it would be possible to easily filter
bug fixes in email and other cherry-picking helper tools, I believe.
>>>> 
>>>> If we really do want to make a guideline about this, I propose
we
>>>> standardize on suffixing the first line of the commit with
(PRnnn).
>>> 
>>> That would work for me.
>>> 
>>> Anywhere in the commit message would work as well (like Mehdi and
Hal said), though I tend to prefer the first line (but Mehdi does not like it
:)).
>>> 
>>> Cheers,
>>> -Quentin
>>> 
>>>> 
>>>> Cheers,
>>>> Hans
>>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160504/4417c3a7/attachment-0001.html>

On Wed, May 4, 2016 at 5:07 PM, Quentin Colombet via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
> Typically, what I had in mind was things like typos/thinko, that are bugs,
> that we notice a few minutes after we made the “main” commit. I do not want
> we have to file a PR that is going to repeat what we are going to say in
> the commit message.
>
Yeah, I agree, we shouldn't have to file PRs for that kind of stuff. Quick
fixes for the build or tests on other platforms obviously fall into this
category.

Do release managers have problems keeping track of these kinds of changes
in practice, though? You can always cut the branch from some quiescent
period on the weekend of night before the cut.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160505/cc346ea7/attachment.html>

Filing a PR for *every* fix is overkill.  But filing a PR to express
"please merge this to the branch" (if there isn't already a PR)
seems reasonable.
--paulr

From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Reid
Kleckner via llvm-dev
Sent: Thursday, May 05, 2016 10:09 AM
To: Quentin Colombet
Cc: llvm-dev
Subject: Re: [llvm-dev] [RFC] Helping release management

On Wed, May 4, 2016 at 5:07 PM, Quentin Colombet via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
Typically, what I had in mind was things like typos/thinko, that are bugs, that
we notice a few minutes after we made the “main” commit. I do not want we have
to file a PR that is going to repeat what we are going to say in the commit
message.

Yeah, I agree, we shouldn't have to file PRs for that kind of stuff. Quick
fixes for the build or tests on other platforms obviously fall into this
category.

Do release managers have problems keeping track of these kinds of changes in
practice, though? You can always cut the branch from some quiescent period on
the weekend of night before the cut.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160505/c480d3be/attachment-0001.html>

> On May 5, 2016, at 10:08 AM, Reid Kleckner <rnk at google.com> wrote:
> 
> On Wed, May 4, 2016 at 5:07 PM, Quentin Colombet via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> Typically, what I had in mind was things like typos/thinko, that are bugs,
that we notice a few minutes after we made the “main” commit. I do not want we
have to file a PR that is going to repeat what we are going to say in the commit
message.
> 
> Yeah, I agree, we shouldn't have to file PRs for that kind of stuff.
Quick fixes for the build or tests on other platforms obviously fall into this
category.
> 
> Do release managers have problems keeping track of these kinds of changes
in practice, though? You can always cut the branch from some quiescent period on
the weekend of night before the cut.
Well, I think the main problem as a release manager is to not miss the follow-up
commit.

Say, we branch, then two weeks later a PR comes, it gets fixed and everything is
documented into the PR.
The release managers see that and pull that fix into the release.

Then, two days, later we realize we missed a case and do a follow-up commit.
There is a high chance we won’t update the PR and that the release managers will
miss it, unless someone/something pings them.

Q.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160510/f10289ae/attachment.html>

> On May 5, 2016, at 10:08 AM, Reid Kleckner via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> On Wed, May 4, 2016 at 5:07 PM, Quentin Colombet via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> Typically, what I had in mind was things like typos/thinko, that are bugs,
that we notice a few minutes after we made the “main” commit. I do not want we
have to file a PR that is going to repeat what we are going to say in the commit
message.
> 
> Yeah, I agree, we shouldn't have to file PRs for that kind of stuff.
Quick fixes for the build or tests on other platforms obviously fall into this
category.
> 
> Do release managers have problems keeping track of these kinds of changes
in practice, though? You can always cut the branch from some quiescent period on
the weekend of night before the cut.
I’ve been doing quite a bit of that lately. The problem is not when you cut the
branch, but how you maintain it. For multiple months after you branched, you’re
willing to take fixes for miscompiles. So you have to got through every commit
and gauge the risk/benefit, see if it applies, if it passes the tests…

Anything that helps tracking that a particular commit need followup commits to
be fully functional would be a win. I think just mentioning the original commit
in the followups is a minimum (then you can have tooling to relate the commits),
but I don’t see a way to enforce this.

Ideally every commit message would be very explicit about what the commit is:
 - a compiler crash fix (with a description of the crash trigger)
 - a miscompile fix
 - an optimization to the compiler code
 - an optimization to the generated code
 - a refactoring
It’s not always easy to distinguish between the first four, and you really want
to understand it before cherry-picking in a long living branch.

Oh, and PRs are nice, but when the commit message is just “Fix PR1234”, it’s
still quite some work to figure out what to do with the commit :-)

Fred
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160510/f6965811/attachment.html>