llvm dev - May 2016 - [RFC] Helping release management

Hi,

I am sending this proposal to get feedbacks on how we could make the tagging of
bug fixes and regressions more obvious. The idea is to provide easily accessible
information to help deciding what to cherry-pick in a release branch.

* Context *

People shipping compilers based on LLVM may not completely align with the
official releases of LLVM. Thus, the stabilization of each custom release may
happen at different period of time. Because of that, release managers have to
come up with their own strategy to decide which commits should be cherry-picked
during the stabilization of their release branch.

For the official LLVM releases, people (committers, code owners, etc.) notice
LLVM release managers that a given commit is worth pulling into the release. I
would like to put in place something more systematic and that plays nicely with
scripting and such that would extend this mechanism.

* Proposal *

1. Use [Fix] for commit related to bug fixes.
2. Add a description of the problem in the commit message to help answer the
following questions:
- What is fixed?
- Which targets are impacted?
- What is required to trigger the bug? (I.e., how often the end users may
encounter it.)
- When was the bug introduced?

#1 At the very least, I would like that each bug fix has a tag on the first line
of the commit (i.e., what ends up in the subject line of the related email.)
Something like [Fix] would do.
Thanks to that tag, it would be possible to easily filter bug fixes in email and
other cherry-picking helper tools, I believe.

#2 Although it may be difficult to come up with that information, I believe it
should be provided as the best of the committer knowledge. Indeed, this kind of
information is useful to help release managers to ascertain how relevant is a
change for their release and thus help them to decide whether to cherry-pick
this change or not.

What do people think?

Thanks,
-Quentin

>-----Original Message-----
>From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
>Quentin Colombet via llvm-dev
>Sent: Monday, May 02, 2016 1:35 PM
>To: llvm-dev <llvm-dev at lists.llvm.org>
>Subject: [llvm-dev] [RFC] Helping release management
>
>Hi,
>
>I am sending this proposal to get feedbacks on how we could make the
>tagging of bug fixes and regressions more obvious. The idea is to provide
>easily accessible information to help deciding what to cherry-pick in a
release
>branch.
>
>* Context *
>
>People shipping compilers based on LLVM may not completely align with the
>official releases of LLVM. Thus, the stabilization of each custom release
may
>happen at different period of time. Because of that, release managers have
>to come up with their own strategy to decide which commits should be
>cherry-picked during the stabilization of their release branch.
>
>For the official LLVM releases, people (committers, code owners, etc.)
notice
>LLVM release managers that a given commit is worth pulling into the release.
>I would like to put in place something more systematic and that plays nicely
>with scripting and such that would extend this mechanism.
>
This seems like a great idea to me.
>* Proposal *
>
>1. Use [Fix] for commit related to bug fixes.
a specific form would be best: [Fix] PR12345, or [FIX] Bug introduced in r56789.
>2. Add a description of the problem in the commit message to help answer
>the following questions:
>- What is fixed?
>- Which targets are impacted?
>- What is required to trigger the bug? (I.e., how often the end users may
>encounter it.)
>- When was the bug introduced?
This is super important for things that regress, and then get fixed before a PR
is even filed.  Even for a PR if the problem was introduced in a known
revision, having that in the commit message helps to understand whether some
other revision is really safe to pull into a release.
>
>#1 At the very least, I would like that each bug fix has a tag on the first
line of
>the commit (i.e., what ends up in the subject line of the related email.)
>Something like [Fix] would do.
>Thanks to that tag, it would be possible to easily filter bug fixes in email
and
>other cherry-picking helper tools, I believe.
>
>#2 Although it may be difficult to come up with that information, I believe
it
>should be provided as the best of the committer knowledge. Indeed, this
>kind of information is useful to help release managers to ascertain how
>relevant is a change for their release and thus help them to decide whether
to
>cherry-pick this change or not.
>
>What do people think?
I like it.
>
>Thanks,
>-Quentin
>_______________________________________________
>LLVM Developers mailing list
>llvm-dev at lists.llvm.org
>http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

> On May 2, 2016, at 2:05 PM, Smith, Kevin B via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
>> Quentin Colombet via llvm-dev
>> Sent: Monday, May 02, 2016 1:35 PM
>> To: llvm-dev <llvm-dev at lists.llvm.org>
>> Subject: [llvm-dev] [RFC] Helping release management
>> 
>> Hi,
>> 
>> I am sending this proposal to get feedbacks on how we could make the
>> tagging of bug fixes and regressions more obvious. The idea is to
provide
>> easily accessible information to help deciding what to cherry-pick in a
release
>> branch.
>> 
>> * Context *
>> 
>> People shipping compilers based on LLVM may not completely align with
the
>> official releases of LLVM. Thus, the stabilization of each custom
release may
>> happen at different period of time. Because of that, release managers
have
>> to come up with their own strategy to decide which commits should be
>> cherry-picked during the stabilization of their release branch.
>> 
>> For the official LLVM releases, people (committers, code owners, etc.)
notice
>> LLVM release managers that a given commit is worth pulling into the
release.
>> I would like to put in place something more systematic and that plays
nicely
>> with scripting and such that would extend this mechanism.
>> 
> 
> This seems like a great idea to me.
> 
>> * Proposal *
>> 
>> 1. Use [Fix] for commit related to bug fixes.
> 
> a specific form would be best: [Fix] PR12345, or [FIX] Bug introduced in
r56789.
The first line of the commit is interpreted as a "title" by "git
based" tools.
I rather have a self-contained first line and avoid *any* external reference
there.
(The commit message can be as long as you want).

-- 
Mehdi



> 
>> 2. Add a description of the problem in the commit message to help
answer
>> the following questions:
>> - What is fixed?
>> - Which targets are impacted?
>> - What is required to trigger the bug? (I.e., how often the end users
may
>> encounter it.)
>> - When was the bug introduced?
> 
> This is super important for things that regress, and then get fixed before
a PR is even filed.  Even for a PR if the problem was introduced in a known
> revision, having that in the commit message helps to understand whether
some other revision is really safe to pull into a release.
> 
>> 
>> #1 At the very least, I would like that each bug fix has a tag on the
first line of
>> the commit (i.e., what ends up in the subject line of the related
email.)
>> Something like [Fix] would do.
>> Thanks to that tag, it would be possible to easily filter bug fixes in
email and
>> other cherry-picking helper tools, I believe.
>> 
>> #2 Although it may be difficult to come up with that information, I
believe it
>> should be provided as the best of the committer knowledge. Indeed, this
>> kind of information is useful to help release managers to ascertain how
>> relevant is a change for their release and thus help them to decide
whether to
>> cherry-pick this change or not.
>> 
>> What do people think?
> 
> I like it.
> 
>> 
>> Thanks,
>> -Quentin
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On Mon, May 02, 2016 at 01:35:27PM -0700, Quentin Colombet via llvm-dev
wrote:
> 1. Use [Fix] for commit related to bug fixes.
I'm not really such a big fan of this format, adds too much noise.
> 2. Add a description of the problem in the commit message to help answer
the following questions:
> - What is fixed?
> - Which targets are impacted?
> - What is required to trigger the bug? (I.e., how often the end users may
encounter it.)
> - When was the bug introduced?
...but this is how a commit message should look like anyway. Worst
examples in the past are references so some internal bug report systems
without actual description for the rest of the world.

Joerg

Hi Joerg,
> On May 2, 2016, at 2:33 PM, Joerg Sonnenberger via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> On Mon, May 02, 2016 at 01:35:27PM -0700, Quentin Colombet via llvm-dev
wrote:
>> 1. Use [Fix] for commit related to bug fixes.
> 
> I'm not really such a big fan of this format, adds too much noise.
What alternatives do you have in mind? 
> 
>> 2. Add a description of the problem in the commit message to help
answer the following questions:
>> - What is fixed?
>> - Which targets are impacted?
>> - What is required to trigger the bug? (I.e., how often the end users
may encounter it.)
>> - When was the bug introduced?
> 
> ...but this is how a commit message should look like anyway.
Sure, but we lack a standardized way to highlight the commits that are fixes
IMHO.
It is a pain to go through all the commit logs without a bit of guidance, even
when the commit is well documented.

Also, if this is how it is supposed to look, maybe we could write it down as a
recommendation somewhere :).
> Worst
> examples in the past are references so some internal bug report systems
> without actual description for the rest of the world.
Agreed.

Cheers,
-Quentin
> 
> Joerg
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On Mon, May 2, 2016 at 1:35 PM, Quentin Colombet via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> I am sending this proposal to get feedbacks on how we could make the
tagging of bug fixes and regressions more obvious. The idea is to provide easily
accessible information to help deciding what to cherry-pick in a release branch.
>
> * Context *
>
> People shipping compilers based on LLVM may not completely align with the
official releases of LLVM. Thus, the stabilization of each custom release may
happen at different period of time. Because of that, release managers have to
come up with their own strategy to decide which commits should be cherry-picked
during the stabilization of their release branch.
(Unrelated to your proposal, I'm curious how common it is to base
releases of LLVM-based tools off the upstream release branches vs.
other revisions.)
> For the official LLVM releases, people (committers, code owners, etc.)
notice LLVM release managers that a given commit is worth pulling into the
release. I would like to put in place something more systematic and that plays
nicely with scripting and such that would extend this mechanism.
>
> * Proposal *
>
> 1. Use [Fix] for commit related to bug fixes.
I think we're mostly pretty good at referencing PR's in commit
messages already. That's also easy to grep for, so maybe that's good
enough?
> 2. Add a description of the problem in the commit message to help answer
the following questions:
> - What is fixed?
> - Which targets are impacted?
> - What is required to trigger the bug? (I.e., how often the end users may
encounter it.)
> - When was the bug introduced?
This sounds like the kind of information that should be in a great
commit message anyways.

But I'm also thinking that maybe we could be better at using our bug
tracker? Whether a bug is a feature request, something that was always
broken, or a regression (and from what version), sounds like a perfect
fit for a bug tracker. Someone doing a release could then query the
Bugzilla to see e.g. what regression bugs were fixed in a certain time
span.
> #1 At the very least, I would like that each bug fix has a tag on the first
line of the commit (i.e., what ends up in the subject line of the related
email.) Something like [Fix] would do.
> Thanks to that tag, it would be possible to easily filter bug fixes in
email and other cherry-picking helper tools, I believe.
If we really do want to make a guideline about this, I propose we
standardize on suffixing the first line of the commit with (PRnnn).

Cheers,
Hans

Hi Hans,

Since you are actively doing this kind of things, your feedbacks is particularly
valuable.
Thanks!
> On May 2, 2016, at 3:45 PM, Hans Wennborg <hans at chromium.org>
wrote:
> 
> On Mon, May 2, 2016 at 1:35 PM, Quentin Colombet via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> I am sending this proposal to get feedbacks on how we could make the
tagging of bug fixes and regressions more obvious. The idea is to provide easily
accessible information to help deciding what to cherry-pick in a release branch.
>> 
>> * Context *
>> 
>> People shipping compilers based on LLVM may not completely align with
the official releases of LLVM. Thus, the stabilization of each custom release
may happen at different period of time. Because of that, release managers have
to come up with their own strategy to decide which commits should be
cherry-picked during the stabilization of their release branch.
> 
> (Unrelated to your proposal, I'm curious how common it is to base
> releases of LLVM-based tools off the upstream release branches vs.
> other revisions.)
> 
>> For the official LLVM releases, people (committers, code owners, etc.)
notice LLVM release managers that a given commit is worth pulling into the
release. I would like to put in place something more systematic and that plays
nicely with scripting and such that would extend this mechanism.
>> 
>> * Proposal *
>> 
>> 1. Use [Fix] for commit related to bug fixes.
> 
> I think we're mostly pretty good at referencing PR's in commit
> messages already. That's also easy to grep for, so maybe that's
good
> enough?
When a PR is available, that is certainly good enough.
I do not want to make filling a PR mandatory for each bug we fix though. Having
a PR is great, but I can see why we may not want to create one each time we fix
something.

If we do want to go into that direction though, I believe we would need to
provide more support to make that easier. (E.g. filing via command line, getting
a number back and feeding this number to a commit.)
> 
>> 2. Add a description of the problem in the commit message to help
answer the following questions:
>> - What is fixed?
>> - Which targets are impacted?
>> - What is required to trigger the bug? (I.e., how often the end users
may encounter it.)
>> - When was the bug introduced?
> 
> This sounds like the kind of information that should be in a great
> commit message anyways.
> 
> But I'm also thinking that maybe we could be better at using our bug
> tracker? Whether a bug is a feature request, something that was always
> broken, or a regression (and from what version), sounds like a perfect
> fit for a bug tracker. Someone doing a release could then query the
> Bugzilla to see e.g. what regression bugs were fixed in a certain time
> span.
Sounds great to me.

Would that kind of workflow ease your job for tracking what should be pulled
into the release branch after you’ve branched?
What would be the best workflow for such task for you?
> 
>> #1 At the very least, I would like that each bug fix has a tag on the
first line of the commit (i.e., what ends up in the subject line of the related
email.) Something like [Fix] would do.
>> Thanks to that tag, it would be possible to easily filter bug fixes in
email and other cherry-picking helper tools, I believe.
> 
> If we really do want to make a guideline about this, I propose we
> standardize on suffixing the first line of the commit with (PRnnn).
That would work for me.

Anywhere in the commit message would work as well (like Mehdi and Hal said),
though I tend to prefer the first line (but Mehdi does not like it :)).

Cheers,
-Quentin
> 
> Cheers,
> Hans

+1 to everything Hans said. We should standardize how we reference bugs
from commits, and then get better about tagging the issues in ways that are
useful to release managers.

I like the idea of a custom phabricator field, so we can make the
references clickable from reviews.

On Mon, May 2, 2016 at 3:45 PM, Hans Wennborg via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Mon, May 2, 2016 at 1:35 PM, Quentin Colombet via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > I am sending this proposal to get feedbacks on how we could make the
> tagging of bug fixes and regressions more obvious. The idea is to provide
> easily accessible information to help deciding what to cherry-pick in a
> release branch.
> >
> > * Context *
> >
> > People shipping compilers based on LLVM may not completely align with
> the official releases of LLVM. Thus, the stabilization of each custom
> release may happen at different period of time. Because of that, release
> managers have to come up with their own strategy to decide which commits
> should be cherry-picked during the stabilization of their release branch.
>
> (Unrelated to your proposal, I'm curious how common it is to base
> releases of LLVM-based tools off the upstream release branches vs.
> other revisions.)
>
> > For the official LLVM releases, people (committers, code owners, etc.)
> notice LLVM release managers that a given commit is worth pulling into the
> release. I would like to put in place something more systematic and that
> plays nicely with scripting and such that would extend this mechanism.
> >
> > * Proposal *
> >
> > 1. Use [Fix] for commit related to bug fixes.
>
> I think we're mostly pretty good at referencing PR's in commit
> messages already. That's also easy to grep for, so maybe that's
good
> enough?
>
> > 2. Add a description of the problem in the commit message to help
answer
> the following questions:
> > - What is fixed?
> > - Which targets are impacted?
> > - What is required to trigger the bug? (I.e., how often the end users
> may encounter it.)
> > - When was the bug introduced?
>
> This sounds like the kind of information that should be in a great
> commit message anyways.
>
> But I'm also thinking that maybe we could be better at using our bug
> tracker? Whether a bug is a feature request, something that was always
> broken, or a regression (and from what version), sounds like a perfect
> fit for a bug tracker. Someone doing a release could then query the
> Bugzilla to see e.g. what regression bugs were fixed in a certain time
> span.
>
> > #1 At the very least, I would like that each bug fix has a tag on the
> first line of the commit (i.e., what ends up in the subject line of the
> related email.) Something like [Fix] would do.
> > Thanks to that tag, it would be possible to easily filter bug fixes in
> email and other cherry-picking helper tools, I believe.
>
> If we really do want to make a guideline about this, I propose we
> standardize on suffixing the first line of the commit with (PRnnn).
>
> Cheers,
> Hans
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160502/e95b18a7/attachment-0001.html>

On May 2, 2016, at 5:45 PM, Hans Wennborg via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>> People shipping compilers based on LLVM may not completely align with
the official releases of LLVM. Thus, the stabilization of each custom release
may happen at different period of time. Because of that, release managers have
to come up with their own strategy to decide which commits should be
cherry-picked during the stabilization of their release branch.
> 
> (Unrelated to your proposal, I'm curious how common it is to base
> releases of LLVM-based tools off the upstream release branches vs.
> other revisions.)

We do both.   We strongly prefer to base our releases on llvm.org releases, but
y’all frequently don’t comply with our timetables.   :)


Jim Rowan
jmr at codeaurora.org
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the
Linux Foundation



-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160502/1cede19d/attachment.html>

On Mon, May 02, 2016 at 01:35:27PM -0700, Quentin Colombet via llvm-dev
wrote:
> Hi,
> 
> I am sending this proposal to get feedbacks on how we could make the
tagging of bug fixes and regressions more obvious. The idea is to provide easily
accessible information to help deciding what to cherry-pick in a release branch.
> 
> * Context *
> 
> People shipping compilers based on LLVM may not completely align with the
official releases of LLVM. Thus, the stabilization of each custom release may
happen at different period of time. Because of that, release managers have to
come up with their own strategy to decide which commits should be cherry-picked
during the stabilization of their release branch.
> 
> For the official LLVM releases, people (committers, code owners, etc.)
notice LLVM release managers that a given commit is worth pulling into the
release. I would like to put in place something more systematic and that plays
nicely with scripting and such that would extend this mechanism.
> 
> * Proposal *
> 
> 1. Use [Fix] for commit related to bug fixes.
> 2. Add a description of the problem in the commit message to help answer
the following questions:
> - What is fixed?
> - Which targets are impacted?
> - What is required to trigger the bug? (I.e., how often the end users may
encounter it.)
> - When was the bug introduced?
> 
> #1 At the very least, I would like that each bug fix has a tag on the first
line of the commit (i.e., what ends up in the subject line of the related
email.) Something like [Fix] would do.
> Thanks to that tag, it would be possible to easily filter bug fixes in
email and other cherry-picking helper tools, I believe.
> 
> #2 Although it may be difficult to come up with that information, I believe
it should be provided as the best of the committer knowledge. Indeed, this kind
of information is useful to help release managers to ascertain how relevant is a
change for their release and thus help them to decide whether to cherry-pick
this change or not.
> 
> What do people think?
> 
Having a centralized place to track bug-fixes and stable patches would
be great.  I think having tags on commits would help, but it won't be
able to cover everything, especially for dot releases where bug fixes
are sometimes identified some time after they have been committed to trunk.

I experimented using phabricator's audit tool for the purpose of tracking
stable patches for the 3.7.1 release.  Advantages of using audit are:

- The audit page has a simple url with the svn revision right in the
  url. e.g. http://reviews.llvm.org/rL12345.
- The audits would appear on the code owner's phabricator page, so they
  would know which patches they needed to approve.
- There was a record of approvals and discussion that was easily
  accessible to everyone.

The problem was, though, that audit is really designed for post-commit
review.  It had states that were analogous to differential's open and
accepted states, but there was no state like closed to indicate that the
patch had been merged to the release branch.  This made it impossible
to do queries to see which patches had been merged, which made it not
very useful for managing the release.

I think phabricator could probably be the best tool for managing
releases, because it is integrated with source control and already has
information about all the commits, but it does not currently have the
right tool/features to make this work.

Hans mentioned in another email about using Bugzilla, and I think this
is a good idea.  If we required users/developers to file a bug for each
patch they wanted merged, then it would be really easy to track.

-Tom

> Thanks,
> -Quentin
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On 4 May 2016 at 19:35, Tom Stellard via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> I think phabricator could probably be the best tool for managing
> releases, because it is integrated with source control and already has
> information about all the commits, but it does not currently have the
> right tool/features to make this work.
I personally don't like Phabricator that much. It's not really
integrated with the code. For instance, it has no idea which project
the commit is on, it doesn't have the context (we need to diff -U999),
etc. Compared to GitHub, Phab is very primitive.

I also don't like it that we have to use special tools to interact
with it. Both GitHub and GitLab work with plain git and a very nice
web interface. I find Phab's interface very confusing.

> Hans mentioned in another email about using Bugzilla, and I think this
> is a good idea.  If we required users/developers to file a bug for each
> patch they wanted merged, then it would be really easy to track.
Bugzilla is as good as any other. Not suited to the task, but equally
not-so-bad at providing the functionality we need. Either Phab or
Bugzilla would have to be customized to have release management in
them, and I don't think we want that.

I know we can't use GitHub or GitLab because of the SVN repo, but it
would be good to have a tool that actually does release management and
code review in a clear and simple way.

cheers,
--renato