llvm dev - Mar 2017 - 4.0.1 Release Schedule + Need feedback for improving stable releases

Hi,

I would like to start a discussion about improvements to the
stable release process, but first, here is a proposed schedule
for the 4.0.1 release.

May 1, 2017 -rc1
May 22, 2017 Deadline for submitting merge request
May 29, 2017 Deadline for merging changes.
June 3, 2017 -rc2
June 10, 2017 Final Release

This is slightly different from previous stable releases in that
we are doing an early -rc1 release.  This matches what we do for
major releases, and I think it will help us get more feedback from
testers sooner.

There is now a script in utils/release, called merge-request.sh, which
you can use to submit merge requests to bugzilla.  It's recommend
that you use this script in order to give us consistent bugzilla entries.
However, you can still file bugzilla requests manually if for some
reason you are unable to use the script.

If you file bugs manually, make sure you mark it as a blocker for
the release metabug: 32061 and add a link the commit on phabricator
in the url field: e.g http://reviews.llvm.org/rL12345

Description for the bug should be in the form of:

Merge r123456 into the 4.0 branch : <80 character commit message summary>


* Stable Release Improvements *

I would like to start an informal discussion about stable release
improvements.  My goals for this discussion are to find ways to
get more community members/organizations using the official stable
releases, and also to help simplify the release process for users,
testers, and release managers.

* Ideas:

1. Longer support for stable release. We currently have 2 major and 2
stable releases per year.  Each major release is effectively supported
for only 3 months, because after the x.y.1 release , we move on to the
next major version.  Our current release schedule is approximately:

March: x.y.0
June x.y.1
Sept: (x+1).y.0 
Dec: (x+1).y.1

One idea is to move to 6 releases per year and have an additional
x.y.2 release after (x+1).y.0, but before (x+1).y.0.  This would
mean that stable releases are supported for ~ 8 months. So something
like:

Jan:   (x-1).y.1 
March: x.y.0
May:   (x-1).y.2
July:  x.y.1
Sept:  (x+1).y.0
Nov:   x.y.2 

What do people think about this or other alternative release schedules?
Would this kind of extended support make stable releases more
attractive to users/organizations that use LLVM.

My main concern with adding an extra release is placing an extra
burden on release testers.  For release testers is doing two
more releases per year practical?



2. Automated release testing.  This is connected to the previous
idea.  Can we automate the release testing?  It would be great if
we had some release tester bots that would test the release
candidates and upload the binaries.  It seems like this should
be doable within the current buildbot infrastructure.  Is the
only thing that keeps us from doing this, lack of dedicated test
hardware?


3. Automated merge requests.  I would like to come up with a way
that we can tag commit messages so that a post-commit hook would
be able to automatically create merge requests in bugzilla.
There are a few ways I can think of to tag the messages, but I'm
just going to propose that adding the following line:

STABLE: 4.0

To commit messages would make it a candidate for an automated
merge request.  Any thoughts?


4. Other Ideas.  Does anyone else have other ideas for improving
stable releases?


Thanks,
Tom

On Mon, Mar 27, 2017 at 5:58 PM, Tom Stellard via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Hi,
>
> I would like to start a discussion about improvements to the
> stable release process, but first, here is a proposed schedule
> for the 4.0.1 release.
>
> May 1, 2017 -rc1
> May 22, 2017 Deadline for submitting merge request
> May 29, 2017 Deadline for merging changes.
> June 3, 2017 -rc2
> June 10, 2017 Final Release
>
> This is slightly different from previous stable releases in that
> we are doing an early -rc1 release.  This matches what we do for
> major releases, and I think it will help us get more feedback from
> testers sooner.
Sounds good to me.
> There is now a script in utils/release, called merge-request.sh, which
> you can use to submit merge requests to bugzilla.  It's recommend
> that you use this script in order to give us consistent bugzilla entries.
> However, you can still file bugzilla requests manually if for some
> reason you are unable to use the script.
>
> If you file bugs manually, make sure you mark it as a blocker for
> the release metabug: 32061 and add a link the commit on phabricator
> in the url field: e.g http://reviews.llvm.org/rL12345
>
> Description for the bug should be in the form of:
>
> Merge r123456 into the 4.0 branch : <80 character commit message
summary>
>
>
> * Stable Release Improvements *
>
> I would like to start an informal discussion about stable release
> improvements.  My goals for this discussion are to find ways to
> get more community members/organizations using the official stable
> releases, and also to help simplify the release process for users,
> testers, and release managers.
>
> * Ideas:
>
> 1. Longer support for stable release. We currently have 2 major and 2
> stable releases per year.  Each major release is effectively supported
> for only 3 months, because after the x.y.1 release , we move on to the
> next major version.  Our current release schedule is approximately:
>
> March: x.y.0
> June x.y.1
> Sept: (x+1).y.0
> Dec: (x+1).y.1
>
> One idea is to move to 6 releases per year and have an additional
> x.y.2 release after (x+1).y.0, but before (x+1).y.0.  This would
> mean that stable releases are supported for ~ 8 months. So something
> like:
>
> Jan:   (x-1).y.1
> March: x.y.0
> May:   (x-1).y.2
> July:  x.y.1
> Sept:  (x+1).y.0
> Nov:   x.y.2
>
> What do people think about this or other alternative release schedules?
> Would this kind of extended support make stable releases more
> attractive to users/organizations that use LLVM.
>
> My main concern with adding an extra release is placing an extra
> burden on release testers.  For release testers is doing two
> more releases per year practical?
I'd be fine with it. I don't find the release testing itself to be
particularly burdensome. The hard part of releases is chasing down and
fixing bugs, but for .2 releases I expect the process to be pretty
smooth, basically building and testing  a build with a few extra
patches on top of .1.
> 2. Automated release testing.  This is connected to the previous
> idea.  Can we automate the release testing?  It would be great if
> we had some release tester bots that would test the release
> candidates and upload the binaries.  It seems like this should
> be doable within the current buildbot infrastructure.  Is the
> only thing that keeps us from doing this, lack of dedicated test
> hardware?
I think it's more a problem of finding someone to configure and
maintain the bots.
> 3. Automated merge requests.  I would like to come up with a way
> that we can tag commit messages so that a post-commit hook would
> be able to automatically create merge requests in bugzilla.
> There are a few ways I can think of to tag the messages, but I'm
> just going to propose that adding the following line:
>
> STABLE: 4.0
>
> To commit messages would make it a candidate for an automated
> merge request.  Any thoughts?
I'm not sure this is worth it. Aren't many of the "we should merge
this to stable" decisions made after the commit lands?

A commit hook I'd really like is something that updates the bugzilla
when a commit message mentions a PR. That's a different story though
:-)

Cheers,
Hans

Tom - In terms of release testing - What about using OpenSUSE build
service? It could be possible to setup a toolchain and try to build a
ton of packages. Most of this could be easily automated and you may
significantly extend the amount of code coverage you get. I don't
think build service is able to run make test, but you'd possibly flush
out build and link issues. (As well as seeing how compatible packages
are with clang)

https://build.opensuse.org/



On Wed, Mar 29, 2017 at 6:14 PM, Hans Wennborg via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> On Mon, Mar 27, 2017 at 5:58 PM, Tom Stellard via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> Hi,
>>
>> I would like to start a discussion about improvements to the
>> stable release process, but first, here is a proposed schedule
>> for the 4.0.1 release.
>>
>> May 1, 2017 -rc1
>> May 22, 2017 Deadline for submitting merge request
>> May 29, 2017 Deadline for merging changes.
>> June 3, 2017 -rc2
>> June 10, 2017 Final Release
>>
>> This is slightly different from previous stable releases in that
>> we are doing an early -rc1 release.  This matches what we do for
>> major releases, and I think it will help us get more feedback from
>> testers sooner.
>
> Sounds good to me.
>
>> There is now a script in utils/release, called merge-request.sh, which
>> you can use to submit merge requests to bugzilla.  It's recommend
>> that you use this script in order to give us consistent bugzilla
entries.
>> However, you can still file bugzilla requests manually if for some
>> reason you are unable to use the script.
>>
>> If you file bugs manually, make sure you mark it as a blocker for
>> the release metabug: 32061 and add a link the commit on phabricator
>> in the url field: e.g http://reviews.llvm.org/rL12345
>>
>> Description for the bug should be in the form of:
>>
>> Merge r123456 into the 4.0 branch : <80 character commit message
summary>
>>
>>
>> * Stable Release Improvements *
>>
>> I would like to start an informal discussion about stable release
>> improvements.  My goals for this discussion are to find ways to
>> get more community members/organizations using the official stable
>> releases, and also to help simplify the release process for users,
>> testers, and release managers.
>>
>> * Ideas:
>>
>> 1. Longer support for stable release. We currently have 2 major and 2
>> stable releases per year.  Each major release is effectively supported
>> for only 3 months, because after the x.y.1 release , we move on to the
>> next major version.  Our current release schedule is approximately:
>>
>> March: x.y.0
>> June x.y.1
>> Sept: (x+1).y.0
>> Dec: (x+1).y.1
>>
>> One idea is to move to 6 releases per year and have an additional
>> x.y.2 release after (x+1).y.0, but before (x+1).y.0.  This would
>> mean that stable releases are supported for ~ 8 months. So something
>> like:
>>
>> Jan:   (x-1).y.1
>> March: x.y.0
>> May:   (x-1).y.2
>> July:  x.y.1
>> Sept:  (x+1).y.0
>> Nov:   x.y.2
>>
>> What do people think about this or other alternative release schedules?
>> Would this kind of extended support make stable releases more
>> attractive to users/organizations that use LLVM.
>>
>> My main concern with adding an extra release is placing an extra
>> burden on release testers.  For release testers is doing two
>> more releases per year practical?
>
> I'd be fine with it. I don't find the release testing itself to be
> particularly burdensome. The hard part of releases is chasing down and
> fixing bugs, but for .2 releases I expect the process to be pretty
> smooth, basically building and testing  a build with a few extra
> patches on top of .1.
>
>> 2. Automated release testing.  This is connected to the previous
>> idea.  Can we automate the release testing?  It would be great if
>> we had some release tester bots that would test the release
>> candidates and upload the binaries.  It seems like this should
>> be doable within the current buildbot infrastructure.  Is the
>> only thing that keeps us from doing this, lack of dedicated test
>> hardware?
>
> I think it's more a problem of finding someone to configure and
> maintain the bots.
>
>> 3. Automated merge requests.  I would like to come up with a way
>> that we can tag commit messages so that a post-commit hook would
>> be able to automatically create merge requests in bugzilla.
>> There are a few ways I can think of to tag the messages, but I'm
>> just going to propose that adding the following line:
>>
>> STABLE: 4.0
>>
>> To commit messages would make it a candidate for an automated
>> merge request.  Any thoughts?
>
> I'm not sure this is worth it. Aren't many of the "we should
merge
> this to stable" decisions made after the commit lands?
>
> A commit hook I'd really like is something that updates the bugzilla
> when a commit message mentions a PR. That's a different story though
> :-)
>
> Cheers,
> Hans
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On 03/29/2017 06:14 AM, Hans Wennborg wrote:
> On Mon, Mar 27, 2017 at 5:58 PM, Tom Stellard via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> Hi,
>>
>> I would like to start a discussion about improvements to the
>> stable release process, but first, here is a proposed schedule
>> for the 4.0.1 release.
>>
>> May 1, 2017 -rc1
>> May 22, 2017 Deadline for submitting merge request
>> May 29, 2017 Deadline for merging changes.
>> June 3, 2017 -rc2
>> June 10, 2017 Final Release
>>
>> This is slightly different from previous stable releases in that
>> we are doing an early -rc1 release.  This matches what we do for
>> major releases, and I think it will help us get more feedback from
>> testers sooner.
> 
> Sounds good to me.
> 
>> There is now a script in utils/release, called merge-request.sh, which
>> you can use to submit merge requests to bugzilla.  It's recommend
>> that you use this script in order to give us consistent bugzilla
entries.
>> However, you can still file bugzilla requests manually if for some
>> reason you are unable to use the script.
>>
>> If you file bugs manually, make sure you mark it as a blocker for
>> the release metabug: 32061 and add a link the commit on phabricator
>> in the url field: e.g http://reviews.llvm.org/rL12345
>>
>> Description for the bug should be in the form of:
>>
>> Merge r123456 into the 4.0 branch : <80 character commit message
summary>
>>
>>
>> * Stable Release Improvements *
>>
>> I would like to start an informal discussion about stable release
>> improvements.  My goals for this discussion are to find ways to
>> get more community members/organizations using the official stable
>> releases, and also to help simplify the release process for users,
>> testers, and release managers.
>>
>> * Ideas:
>>
>> 1. Longer support for stable release. We currently have 2 major and 2
>> stable releases per year.  Each major release is effectively supported
>> for only 3 months, because after the x.y.1 release , we move on to the
>> next major version.  Our current release schedule is approximately:
>>
>> March: x.y.0
>> June x.y.1
>> Sept: (x+1).y.0
>> Dec: (x+1).y.1
>>
>> One idea is to move to 6 releases per year and have an additional
>> x.y.2 release after (x+1).y.0, but before (x+1).y.0.  This would
>> mean that stable releases are supported for ~ 8 months. So something
>> like:
>>
>> Jan:   (x-1).y.1
>> March: x.y.0
>> May:   (x-1).y.2
>> July:  x.y.1
>> Sept:  (x+1).y.0
>> Nov:   x.y.2
>>
>> What do people think about this or other alternative release schedules?
>> Would this kind of extended support make stable releases more
>> attractive to users/organizations that use LLVM.
>>
>> My main concern with adding an extra release is placing an extra
>> burden on release testers.  For release testers is doing two
>> more releases per year practical?
> 
> I'd be fine with it. I don't find the release testing itself to be
> particularly burdensome. The hard part of releases is chasing down and
> fixing bugs, but for .2 releases I expect the process to be pretty
> smooth, basically building and testing  a build with a few extra
> patches on top of .1.
> 
>> 2. Automated release testing.  This is connected to the previous
>> idea.  Can we automate the release testing?  It would be great if
>> we had some release tester bots that would test the release
>> candidates and upload the binaries.  It seems like this should
>> be doable within the current buildbot infrastructure.  Is the
>> only thing that keeps us from doing this, lack of dedicated test
>> hardware?
> 
> I think it's more a problem of finding someone to configure and
> maintain the bots.
> 
>> 3. Automated merge requests.  I would like to come up with a way
>> that we can tag commit messages so that a post-commit hook would
>> be able to automatically create merge requests in bugzilla.
>> There are a few ways I can think of to tag the messages, but I'm
>> just going to propose that adding the following line:
>>
>> STABLE: 4.0
>>
>> To commit messages would make it a candidate for an automated
>> merge request.  Any thoughts?
> 
> I'm not sure this is worth it. Aren't many of the "we should
merge
> this to stable" decisions made after the commit lands?
> 
Yes, good point.
> A commit hook I'd really like is something that updates the bugzilla
> when a commit message mentions a PR. That's a different story though
> :-)
I like this idea better than what I had in mind, this way instead
of parsing commit messages, we would just need to look at closed bugs
to figure out what to merge.
> 
> Cheers,
> Hans
>

> On Mar 27, 2017, at 8:58 AM, Tom Stellard via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi,
> 
> I would like to start a discussion about improvements to the
> stable release process, but first, here is a proposed schedule
> for the 4.0.1 release.
> 
> May 1, 2017 -rc1
> May 22, 2017 Deadline for submitting merge request
> May 29, 2017 Deadline for merging changes.
> June 3, 2017 -rc2
> June 10, 2017 Final Release
> 
> This is slightly different from previous stable releases in that
> we are doing an early -rc1 release.  This matches what we do for
> major releases, and I think it will help us get more feedback from
> testers sooner.
> 
> There is now a script in utils/release, called merge-request.sh, which
> you can use to submit merge requests to bugzilla.  It's recommend
> that you use this script in order to give us consistent bugzilla entries.
> However, you can still file bugzilla requests manually if for some
> reason you are unable to use the script.
I like the fact that there is a script!
It’d be really nice to have an entry on the website describing how to use this.
I always struggle to find the right email that tells me how to handle
cherry-picks.

> If you file bugs manually, make sure you mark it as a blocker for
> the release metabug: 32061 and add a link the commit on phabricator
> in the url field: e.g http://reviews.llvm.org/rL12345
> 
> Description for the bug should be in the form of:
> 
> Merge r123456 into the 4.0 branch : <80 character commit message
summary>
> 
> 
> * Stable Release Improvements *
> 
> I would like to start an informal discussion about stable release
> improvements.  My goals for this discussion are to find ways to
> get more community members/organizations using the official stable
> releases, and also to help simplify the release process for users,
> testers, and release managers.
> 
> * Ideas:
> 
> 1. Longer support for stable release. We currently have 2 major and 2
> stable releases per year.  Each major release is effectively supported
> for only 3 months, because after the x.y.1 release , we move on to the
> next major version.  Our current release schedule is approximately:
> 
> March: x.y.0
> June x.y.1
> Sept: (x+1).y.0 
> Dec: (x+1).y.1
> 
> One idea is to move to 6 releases per year and have an additional
> x.y.2 release after (x+1).y.0, but before (x+1).y.0.  This would
> mean that stable releases are supported for ~ 8 months. So something
> like:
> 
> Jan:   (x-1).y.1 
> March: x.y.0
> May:   (x-1).y.2
> July:  x.y.1
> Sept:  (x+1).y.0
> Nov:   x.y.2 
> 
> What do people think about this or other alternative release schedules?
> Would this kind of extended support make stable releases more
> attractive to users/organizations that use LLVM.
> 
> My main concern with adding an extra release is placing an extra
> burden on release testers.  For release testers is doing two
> more releases per year practical?
> 
> 
> 
> 2. Automated release testing.  This is connected to the previous
> idea.  Can we automate the release testing?  It would be great if
> we had some release tester bots that would test the release
> candidates and upload the binaries.  It seems like this should
> be doable within the current buildbot infrastructure.  Is the
> only thing that keeps us from doing this, lack of dedicated test
> hardware?

+1 for the automation! This is key to me.

I’m not sure we need *dedicated* hardware for this: the release testing does not
consume a lot of cycles continuously, does it? I expect we need some hardware
for a limited period of time when preparing a release.

What’s missing in my opinion is rather the software infrastructure: an easy way
to run a script and validate the compiler more extensively than the LLVM
test-suite.
Maybe we should lead an effort towards an “extended test-suite”: having the
ability to clone a bunch of selected open-source project and build & run
their unitests. Even if it is something that takes a full day to run, it seems
possible to do this on each RCs.
> 3. Automated merge requests.  I would like to come up with a way
> that we can tag commit messages so that a post-commit hook would
> be able to automatically create merge requests in bugzilla.
> There are a few ways I can think of to tag the messages, but I'm
> just going to propose that adding the following line:
> 
> STABLE: 4.0
> 
> To commit messages would make it a candidate for an automated
> merge request.  Any thoughts?
I’m not a big fan of hijacking commit messages for this.
What does this by? Just saving running the script?
I rather have this as part of the scripts that will be available after we
migrate to git.
For instance you can already commit from git with `git llvm push` (see
http://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo
), we can imagine extending this in the future to be able to send merge request
easily from the command line.

— 
Mehdi

Hi Tom,

First, kudos for the new merge scripts.

On 29 March 2017 at 11:14, Hans Wennborg via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> (...) I don't find the release testing itself to be
> particularly burdensome. The hard part of releases is chasing down and
> fixing bugs, but for .2 releases I expect the process to be pretty
> smooth, basically building and testing  a build with a few extra
> patches on top of .1.
I agree with Hans. This is why dot releases cannot have any new
feature in it, no matter how small.

> I think it's more a problem of finding someone to configure and
> maintain the bots.
Yup.

Finding the available hardware that is upstream, outside firewalls and
ready for deployment as buildbots, either all year round or on/off
during releases, is going to be hard.

If/when we move to Jenkins, this could be easier, but right now, it's
pretty hard.

> I'm not sure this is worth it. Aren't many of the "we should
merge
> this to stable" decisions made after the commit lands?
Again, agreed. The decision process should be a conscious one, and
with review from the release manager, code owner, etc.

Simple scripting won't go all the way to providing that auditing path
and could make release branches unstable.

Release testing isn't complicated enough to automate that part, at
least not yet.

If/when we have more tests beyond the test-suite, we'll have to
investigate each individual test on their own.

For now, involving distros and large projects directly is the easiest
way to test the release.

cheers,
--renato