llvm dev - Mar 2017 - [Release-testers] 4.0.1 Release Schedule + Need feedback for improving stable releases

Tom - In terms of release testing - What about using OpenSUSE build
service? It could be possible to setup a toolchain and try to build a
ton of packages. Most of this could be easily automated and you may
significantly extend the amount of code coverage you get. I don't
think build service is able to run make test, but you'd possibly flush
out build and link issues. (As well as seeing how compatible packages
are with clang)

https://build.opensuse.org/



On Wed, Mar 29, 2017 at 6:14 PM, Hans Wennborg via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> On Mon, Mar 27, 2017 at 5:58 PM, Tom Stellard via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> Hi,
>>
>> I would like to start a discussion about improvements to the
>> stable release process, but first, here is a proposed schedule
>> for the 4.0.1 release.
>>
>> May 1, 2017 -rc1
>> May 22, 2017 Deadline for submitting merge request
>> May 29, 2017 Deadline for merging changes.
>> June 3, 2017 -rc2
>> June 10, 2017 Final Release
>>
>> This is slightly different from previous stable releases in that
>> we are doing an early -rc1 release.  This matches what we do for
>> major releases, and I think it will help us get more feedback from
>> testers sooner.
>
> Sounds good to me.
>
>> There is now a script in utils/release, called merge-request.sh, which
>> you can use to submit merge requests to bugzilla.  It's recommend
>> that you use this script in order to give us consistent bugzilla
entries.
>> However, you can still file bugzilla requests manually if for some
>> reason you are unable to use the script.
>>
>> If you file bugs manually, make sure you mark it as a blocker for
>> the release metabug: 32061 and add a link the commit on phabricator
>> in the url field: e.g http://reviews.llvm.org/rL12345
>>
>> Description for the bug should be in the form of:
>>
>> Merge r123456 into the 4.0 branch : <80 character commit message
summary>
>>
>>
>> * Stable Release Improvements *
>>
>> I would like to start an informal discussion about stable release
>> improvements.  My goals for this discussion are to find ways to
>> get more community members/organizations using the official stable
>> releases, and also to help simplify the release process for users,
>> testers, and release managers.
>>
>> * Ideas:
>>
>> 1. Longer support for stable release. We currently have 2 major and 2
>> stable releases per year.  Each major release is effectively supported
>> for only 3 months, because after the x.y.1 release , we move on to the
>> next major version.  Our current release schedule is approximately:
>>
>> March: x.y.0
>> June x.y.1
>> Sept: (x+1).y.0
>> Dec: (x+1).y.1
>>
>> One idea is to move to 6 releases per year and have an additional
>> x.y.2 release after (x+1).y.0, but before (x+1).y.0.  This would
>> mean that stable releases are supported for ~ 8 months. So something
>> like:
>>
>> Jan:   (x-1).y.1
>> March: x.y.0
>> May:   (x-1).y.2
>> July:  x.y.1
>> Sept:  (x+1).y.0
>> Nov:   x.y.2
>>
>> What do people think about this or other alternative release schedules?
>> Would this kind of extended support make stable releases more
>> attractive to users/organizations that use LLVM.
>>
>> My main concern with adding an extra release is placing an extra
>> burden on release testers.  For release testers is doing two
>> more releases per year practical?
>
> I'd be fine with it. I don't find the release testing itself to be
> particularly burdensome. The hard part of releases is chasing down and
> fixing bugs, but for .2 releases I expect the process to be pretty
> smooth, basically building and testing  a build with a few extra
> patches on top of .1.
>
>> 2. Automated release testing.  This is connected to the previous
>> idea.  Can we automate the release testing?  It would be great if
>> we had some release tester bots that would test the release
>> candidates and upload the binaries.  It seems like this should
>> be doable within the current buildbot infrastructure.  Is the
>> only thing that keeps us from doing this, lack of dedicated test
>> hardware?
>
> I think it's more a problem of finding someone to configure and
> maintain the bots.
>
>> 3. Automated merge requests.  I would like to come up with a way
>> that we can tag commit messages so that a post-commit hook would
>> be able to automatically create merge requests in bugzilla.
>> There are a few ways I can think of to tag the messages, but I'm
>> just going to propose that adding the following line:
>>
>> STABLE: 4.0
>>
>> To commit messages would make it a candidate for an automated
>> merge request.  Any thoughts?
>
> I'm not sure this is worth it. Aren't many of the "we should
merge
> this to stable" decisions made after the commit lands?
>
> A commit hook I'd really like is something that updates the bugzilla
> when a commit message mentions a PR. That's a different story though
> :-)
>
> Cheers,
> Hans
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On Wed, Mar 29, 2017 at 12:40 PM, C Bergström via Release-testers
<release-testers at lists.llvm.org> wrote:
> Tom - In terms of release testing - What about using OpenSUSE build
> service? It could be possible to setup a toolchain and try to build a
> ton of packages. Most of this could be easily automated and you may
> significantly extend the amount of code coverage you get. I don't
> think build service is able to run make test, but you'd possibly flush
> out build and link issues. (As well as seeing how compatible packages
> are with clang)
OBS can run tests on a lot of platforms, see the official llvm package
for example: https://build.opensuse.org/package/show/devel:tools:compiler/llvm4
and clang-trunk package which is kind-of-daily snapshots:
https://build.opensuse.org/project/show/home%3Anamtrac%3Aclang

If we want to use OBS, I am ready to help with integration and such.

Regards,
ismail

tbh I think swapping out clang for gcc and building against the most
bleeding edge wouldn't be a bad thing. I'm not sure if clang can build
the latest linux kernel or what patchset would be required as well as
grub2 and glibc, but besides those I'm not aware of any particularly
nasty blockers.

opensuse+clang (without gcc at all) is in my mind an ideal goal. I
don't know what weird things gentoo has corrupted clang with, but
someone with gentoo and clang may also be able to help flush out a
whole world level of testing.

Just some drive-by thoughts since additional testing is coming up as a topic..

On Wed, Mar 29, 2017 at 7:29 PM, Ismail Donmez <ismail at i10z.com>
wrote:
> On Wed, Mar 29, 2017 at 12:40 PM, C Bergström via Release-testers
> <release-testers at lists.llvm.org> wrote:
>> Tom - In terms of release testing - What about using OpenSUSE build
>> service? It could be possible to setup a toolchain and try to build a
>> ton of packages. Most of this could be easily automated and you may
>> significantly extend the amount of code coverage you get. I don't
>> think build service is able to run make test, but you'd possibly
flush
>> out build and link issues. (As well as seeing how compatible packages
>> are with clang)
>
> OBS can run tests on a lot of platforms, see the official llvm package
> for example:
https://build.opensuse.org/package/show/devel:tools:compiler/llvm4
> and clang-trunk package which is kind-of-daily snapshots:
> https://build.opensuse.org/project/show/home%3Anamtrac%3Aclang
>
> If we want to use OBS, I am ready to help with integration and such.
>
> Regards,
> ismail