llvm dev - Feb 2015 - [LLVMdev] [cfe-dev] [3.6 Release] RC3 has been tagged

On Wed, Feb 18, 2015 at 11:53 AM, Renato Golin <renato.golin at
linaro.org> wrote:
> On 18 February 2015 at 16:39, Jack Howarth
> <howarth.mailing.lists at gmail.com> wrote:
>> This still all begs the question of what exact metrics exist for the
>> Q/A of llvm releases? IMHO, the bad PR from shoving out compiler
>> releases with severe performance regressions in the generated code far
>> outweighs a brief delay to triage these issues as much as possible.
>
> Hi Jack,
>
> It depends on how you interpret LLVM releases.
>
> I still see them as points in time where we actually did some deeper
> investigations as a community and made sure the compiler is still
> sane. If it's not, future points in time shall fix it, for example, on
> dot releases or next releases, depending on how severe the regression
> is.
>
> A while ago we kind of agreed on a release Go/NoGo table that was
> somewhat like this:
>
> Before release:
> * Any conformance regressions
> * Bad conformance breakage (new)
>
> Next dot release:
> * Any conformance breakage
> * Bad performance regression
>
> Next release:
> * Any performance regression
>
> Since most people use LLVM on top-of-tree, the problem of having a
> release that is not as good as the previous is somewhat washed away by
> the fact that we now have dot-releases, and can fix the horrible cases
> before a new full release comes in 6 months time.
>
> Also, we don't have a limit for dot releases, and last year we had a
> dot release almost right after the major release due to similar
> problems. So, what we release now won't necessarily go to
> distributions, if we continue the process on the next dot-release,
> 3.6.1.
>
> I think we all agree that this is a "bad performance regression",
but
> most of us think this is not bad enough to delay the release further,
> but probably bad enough to start the next dot release right after the
> end of this one.
>
> Does that really sound like such a bad idea?
>
> cheers,
> --renato
Renato,
       My concern is that, without strict enforcement of the triaging
serious P1-type bugs, the major  llvm.org releases will devolve into a
continual exchange of one set of major regressions for another set.
Strict enforcement of back porting the required fixes from later trunk
for point releases seem to require as just much effort and even more
due diligence than just defining certain types of bugs as release
blockers and dealing with them upfront. Otherwise, folks will decide
that the major releases are unsafe and eventually avoid them entirely.
               Jack

On Wed, Feb 18, 2015 at 12:04 PM, Jack Howarth
<howarth.mailing.lists at gmail.com> wrote:
> On Wed, Feb 18, 2015 at 11:53 AM, Renato Golin <renato.golin at
linaro.org> wrote:
>> On 18 February 2015 at 16:39, Jack Howarth
>> <howarth.mailing.lists at gmail.com> wrote:
>>> This still all begs the question of what exact metrics exist for
the
>>> Q/A of llvm releases? IMHO, the bad PR from shoving out compiler
>>> releases with severe performance regressions in the generated code
far
>>> outweighs a brief delay to triage these issues as much as possible.
>>
>> Hi Jack,
>>
>> It depends on how you interpret LLVM releases.
>>
>> I still see them as points in time where we actually did some deeper
>> investigations as a community and made sure the compiler is still
>> sane. If it's not, future points in time shall fix it, for example,
on
>> dot releases or next releases, depending on how severe the regression
>> is.
>>
>> A while ago we kind of agreed on a release Go/NoGo table that was
>> somewhat like this:
>>
>> Before release:
>> * Any conformance regressions
>> * Bad conformance breakage (new)
>>
>> Next dot release:
>> * Any conformance breakage
>> * Bad performance regression
>>
>> Next release:
>> * Any performance regression
>>
>> Since most people use LLVM on top-of-tree, the problem of having a
>> release that is not as good as the previous is somewhat washed away by
>> the fact that we now have dot-releases, and can fix the horrible cases
>> before a new full release comes in 6 months time.
>>
>> Also, we don't have a limit for dot releases, and last year we had
a
>> dot release almost right after the major release due to similar
>> problems. So, what we release now won't necessarily go to
>> distributions, if we continue the process on the next dot-release,
>> 3.6.1.
>>
>> I think we all agree that this is a "bad performance
regression", but
>> most of us think this is not bad enough to delay the release further,
>> but probably bad enough to start the next dot release right after the
>> end of this one.
>>
>> Does that really sound like such a bad idea?
>>
>> cheers,
>> --renato
>
> Renato,
>        My concern is that, without strict enforcement of the triaging
> serious P1-type bugs, the major  llvm.org releases will devolve into a
> continual exchange of one set of major regressions for another set.
> Strict enforcement of back porting the required fixes from later trunk
> for point releases seem to require as just much effort and even more
> due diligence than just defining certain types of bugs as release
> blockers and dealing with them upfront. Otherwise, folks will decide
> that the major releases are unsafe and eventually avoid them entirely.
>                Jack
Renato,
      I also find it very telling that Apple avoided syncing their
clang to the 3.6svn release for the Xcode 6.2 public betas after doing
so habitually for past releases. Might that not say something about
the state of 3.6?
         Jack

On 18 February 2015 at 17:04, Jack Howarth
<howarth.mailing.lists at gmail.com> wrote:
> Renato,
>        My concern is that, without strict enforcement of the triaging
> serious P1-type bugs, the major  llvm.org releases will devolve into a
> continual exchange of one set of major regressions for another set.
This is a very pessimistic view of our community. We do care about
bugs and we do tend to fix it pretty quickly, even without the beating
drum of a release cycle.

> Otherwise, folks will decide that the major releases are unsafe and
eventually avoid them entirely.
As I said, releases are not meant to be a product, but a point in
time. If you want do build a product out of a release, I suggest you
to always wait for the dot-releases, as they have the fixes for
problems that were found when spinning the release, while keeping
ABI/API compatibility.

You are expecting our releases to be something it's not, and it's not
a surprise if that doesn't abide by the level of quality you aspire.
Most, if not all users don't rely that much on releases and tend to
pick a commit in between that suit their needs. This is not just
Apple, as you mentioned, but almost every one do that. Furthermore,
new versions of GCC *also* have major regressions in some benchmarks
and they also release and fix on the next dot-release.

cheers,
--renato

On Wed, Feb 18, 2015 at 12:04:47PM -0500, Jack Howarth
wrote:
>        My concern is that, without strict enforcement of the triaging
> serious P1-type bugs, the major  llvm.org releases will devolve into a
> continual exchange of one set of major regressions for another set.
A performance regression is not a P1 type bug. Especially if the change
has existed for a while and noone noticed.

Joerg

On Wed, Feb 18, 2015 at 1:04 PM, Renato Golin <renato.golin at linaro.org>
wrote:
> On 18 February 2015 at 17:04, Jack Howarth
> <howarth.mailing.lists at gmail.com> wrote:
>> Renato,
>>        My concern is that, without strict enforcement of the triaging
>> serious P1-type bugs, the major  llvm.org releases will devolve into a
>> continual exchange of one set of major regressions for another set.
>
> This is a very pessimistic view of our community. We do care about
> bugs and we do tend to fix it pretty quickly, even without the beating
> drum of a release cycle.
>
>
>> Otherwise, folks will decide that the major releases are unsafe and
eventually avoid them entirely.
>
> As I said, releases are not meant to be a product, but a point in
> time. If you want do build a product out of a release, I suggest you
> to always wait for the dot-releases, as they have the fixes for
> problems that were found when spinning the release, while keeping
> ABI/API compatibility.
>
> You are expecting our releases to be something it's not, and it's
not
> a surprise if that doesn't abide by the level of quality you aspire.
> Most, if not all users don't rely that much on releases and tend to
> pick a commit in between that suit their needs. This is not just
> Apple, as you mentioned, but almost every one do that. Furthermore,
> new versions of GCC *also* have major regressions in some benchmarks
> and they also release and fix on the next dot-release.
Define users. If compiler developers, your argument holds. For other
users who just want a stable compiler, the argument does not. As for
FSF gcc, I disagree. If an egregious  regression in an important
benchmarks is discovered, in general they would escalate it to a P1
and most certainly fix it before release.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65028
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65006

Especially when the problem is understood and a fix is at hand.
>
> cheers,
> --renato

On Wed, Feb 18, 2015 at 1:39 PM, Joerg Sonnenberger
<joerg at britannica.bec.de> wrote:
> On Wed, Feb 18, 2015 at 12:04:47PM -0500, Jack Howarth wrote:
>>        My concern is that, without strict enforcement of the triaging
>> serious P1-type bugs, the major  llvm.org releases will devolve into a
>> continual exchange of one set of major regressions for another set.
>
> A performance regression is not a P1 type bug. Especially if the change
> has existed for a while and noone noticed.
This issue has been publicly reported since 3.6-RC1...

http://www.phoronix.com/scan.php?page=article&item=llvm-clang-3.5-3.6-rc1&num=2
>
> Joerg
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev