llvm dev - May 2020 - RFC: Release qualification criteria

Hi,

I'm splitting this discussion off of my RFC for release process
changes.

We currently have no official release qualification criteria.  In
other words, we don't have any blocking tests that need to pass in
order to make a new release.

We do time-based releases, which is not always compatible with having
quality-based criteria for tagging a final release.  So, I think another
way to look at this issue is to talk about what kinds of CI testing we
have for trunk and if there are any additional kinds of
testing (e.g. compile-time performance) that we want to prioritize.

So, for the purposes of this discussion, I see 2 main questions:

1. Should we define some set of blocking tests that need to pass before a
release
   can happen?

2. What gaps do we currently have in our CI testing?

Hi Tom,

You hit exactly the questions that are also going through my mind. To add
to the two questions:
- What is the testing strategy for LLVM (ie. who tests for which bugs
when)?
- What do we expect from the people contributing patches?
- Do we want to also enforce some of these checks/tests on EVERY patch to
master or a release branch (ie. enforced pre-merge testing)?

And the meta question:
What's the process to create such an agreement?

Best,
Christian


On Thu, May 21, 2020 at 8:59 PM Tom Stellard via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi,
>
> I'm splitting this discussion off of my RFC for release process
> changes.
>
> We currently have no official release qualification criteria.  In
> other words, we don't have any blocking tests that need to pass in
> order to make a new release.
>
> We do time-based releases, which is not always compatible with having
> quality-based criteria for tagging a final release.  So, I think another
> way to look at this issue is to talk about what kinds of CI testing we
> have for trunk and if there are any additional kinds of
> testing (e.g. compile-time performance) that we want to prioritize.
>
> So, for the purposes of this discussion, I see 2 main questions:
>
> 1. Should we define some set of blocking tests that need to pass before a
> release
>    can happen?
>
> 2. What gaps do we currently have in our CI testing?
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200525/82de26bd/attachment-0001.html>

On Thu, May 21, 2020 at 8:59 PM Tom Stellard via Openmp-dev
<openmp-dev at lists.llvm.org> wrote:
>
> Hi,
>
> I'm splitting this discussion off of my RFC for release process
> changes.
>
> We currently have no official release qualification criteria.  In
> other words, we don't have any blocking tests that need to pass in
> order to make a new release.
>
> We do time-based releases, which is not always compatible with having
> quality-based criteria for tagging a final release.  So, I think another
> way to look at this issue is to talk about what kinds of CI testing we
> have for trunk and if there are any additional kinds of
> testing (e.g. compile-time performance) that we want to prioritize.
>
> So, for the purposes of this discussion, I see 2 main questions:
>
> 1. Should we define some set of blocking tests that need to pass before a
release
>    can happen?
I suppose we could have a baseline about clang bootstrap + lit tests
(what test-release.sh does essentially) passes on major platforms.

But the really interesting question for me is really what kind of bugs
we're considering as release blocking. It's the trade-off between
shipping on (or not too much behind schedule) and delaying to
investigate more issues that's tricky..
>
> 2. What gaps do we currently have in our CI testing?
The latest release made it clear we don't track compile time very
well, or at least not well enough to catch the regressions in that
release early enough.

Also I think there's no 32-bit Windows buildbot anymore :/

On 05/25/2020 06:10 AM, Hans Wennborg wrote:
> On Thu, May 21, 2020 at 8:59 PM Tom Stellard via Openmp-dev
> <openmp-dev at lists.llvm.org> wrote:
>>
>> Hi,
>>
>> I'm splitting this discussion off of my RFC for release process
>> changes.
>>
>> We currently have no official release qualification criteria.  In
>> other words, we don't have any blocking tests that need to pass in
>> order to make a new release.
>>
>> We do time-based releases, which is not always compatible with having
>> quality-based criteria for tagging a final release.  So, I think
another
>> way to look at this issue is to talk about what kinds of CI testing we
>> have for trunk and if there are any additional kinds of
>> testing (e.g. compile-time performance) that we want to prioritize.
>>
>> So, for the purposes of this discussion, I see 2 main questions:
>>
>> 1. Should we define some set of blocking tests that need to pass before
a release
>>    can happen?
> 
> I suppose we could have a baseline about clang bootstrap + lit tests
> (what test-release.sh does essentially) passes on major platforms.
> 
I think for testing like this that can be easily automated we're almost
better off just adding these as CI tests from trunk rather than having
them gate releases.

-Tom
> But the really interesting question for me is really what kind of bugs
> we're considering as release blocking. It's the trade-off between
> shipping on (or not too much behind schedule) and delaying to
> investigate more issues that's tricky..
> 
>>
>> 2. What gaps do we currently have in our CI testing?
> 
> The latest release made it clear we don't track compile time very
> well, or at least not well enough to catch the regressions in that
> release early enough.
> 
> Also I think there's no 32-bit Windows buildbot anymore :/
>