llvm dev - Feb 2015 - [LLVMdev] [cfe-dev] [3.6 Release] RC3 has been tagged

On Wed, Feb 18, 2015 at 11:22 AM, Jack Howarth
<howarth.mailing.lists at gmail.com> wrote:
> On Wed, Feb 18, 2015 at 10:55 AM, Jack Howarth
> <howarth.mailing.lists at gmail.com> wrote:
>> I finally got around to testing this on a Bloomfield processor (Early
>> 2009 MacPro 2x2.66 GHz dual-quad core) and the regressions from
>> http://llvm.org/bugs/show_bug.cgi?id=22589 are even more severe. For
>> 10 runs of scimark2_1c built with "-O3 -march=native"...
>>
>> llvm 3.5.1   1204.16+/-2.66 Mflops
>> 3.6 branch   866.49+/-1.26 Mflops
>>
>
> The proposed patch mitigates the damage on Bloomfield...
>
> patched 3.6svn   1073.69+/-1.97
>
> so we are only regressing the benchmark 24% rather than 39% there.
>          Jack
>
This still all begs the question of what exact metrics exist for the
Q/A of llvm releases? IMHO, the bad PR from shoving out compiler
releases with severe performance regressions in the generated code far
outweighs a brief delay to triage these issues as much as possible.
         Jack
>
>> Do you seriously want to ship with a 39% performance regression in a
>> major benchmark?
>>               Jack
>>
>> On Wed, Feb 18, 2015 at 10:33 AM, Renato Golin <renato.golin at
linaro.org> wrote:
>>> On 18 February 2015 at 14:37, Hal Finkel <hfinkel at anl.gov>
wrote:
>>>> I think that, unfortunately, for the others, there's not
sufficient time to investigate before the release.
>>>
>>> This looks like a serious case for 3.6.1, not RC5.
>>>
>>> cheers,
>>> --renato
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On 18 February 2015 at 16:39, Jack Howarth
<howarth.mailing.lists at gmail.com> wrote:
> This still all begs the question of what exact metrics exist for the
> Q/A of llvm releases? IMHO, the bad PR from shoving out compiler
> releases with severe performance regressions in the generated code far
> outweighs a brief delay to triage these issues as much as possible.
Hi Jack,

It depends on how you interpret LLVM releases.

I still see them as points in time where we actually did some deeper
investigations as a community and made sure the compiler is still
sane. If it's not, future points in time shall fix it, for example, on
dot releases or next releases, depending on how severe the regression
is.

A while ago we kind of agreed on a release Go/NoGo table that was
somewhat like this:

Before release:
* Any conformance regressions
* Bad conformance breakage (new)

Next dot release:
* Any conformance breakage
* Bad performance regression

Next release:
* Any performance regression

Since most people use LLVM on top-of-tree, the problem of having a
release that is not as good as the previous is somewhat washed away by
the fact that we now have dot-releases, and can fix the horrible cases
before a new full release comes in 6 months time.

Also, we don't have a limit for dot releases, and last year we had a
dot release almost right after the major release due to similar
problems. So, what we release now won't necessarily go to
distributions, if we continue the process on the next dot-release,
3.6.1.

I think we all agree that this is a "bad performance regression", but
most of us think this is not bad enough to delay the release further,
but probably bad enough to start the next dot release right after the
end of this one.

Does that really sound like such a bad idea?

cheers,
--renato

On Wed, Feb 18, 2015 at 11:53 AM, Renato Golin <renato.golin at
linaro.org> wrote:
> On 18 February 2015 at 16:39, Jack Howarth
> <howarth.mailing.lists at gmail.com> wrote:
>> This still all begs the question of what exact metrics exist for the
>> Q/A of llvm releases? IMHO, the bad PR from shoving out compiler
>> releases with severe performance regressions in the generated code far
>> outweighs a brief delay to triage these issues as much as possible.
>
> Hi Jack,
>
> It depends on how you interpret LLVM releases.
>
> I still see them as points in time where we actually did some deeper
> investigations as a community and made sure the compiler is still
> sane. If it's not, future points in time shall fix it, for example, on
> dot releases or next releases, depending on how severe the regression
> is.
>
> A while ago we kind of agreed on a release Go/NoGo table that was
> somewhat like this:
>
> Before release:
> * Any conformance regressions
> * Bad conformance breakage (new)
>
> Next dot release:
> * Any conformance breakage
> * Bad performance regression
>
> Next release:
> * Any performance regression
>
> Since most people use LLVM on top-of-tree, the problem of having a
> release that is not as good as the previous is somewhat washed away by
> the fact that we now have dot-releases, and can fix the horrible cases
> before a new full release comes in 6 months time.
>
> Also, we don't have a limit for dot releases, and last year we had a
> dot release almost right after the major release due to similar
> problems. So, what we release now won't necessarily go to
> distributions, if we continue the process on the next dot-release,
> 3.6.1.
>
> I think we all agree that this is a "bad performance regression",
but
> most of us think this is not bad enough to delay the release further,
> but probably bad enough to start the next dot release right after the
> end of this one.
>
> Does that really sound like such a bad idea?
>
> cheers,
> --renato
Renato,
       My concern is that, without strict enforcement of the triaging
serious P1-type bugs, the major  llvm.org releases will devolve into a
continual exchange of one set of major regressions for another set.
Strict enforcement of back porting the required fixes from later trunk
for point releases seem to require as just much effort and even more
due diligence than just defining certain types of bugs as release
blockers and dealing with them upfront. Otherwise, folks will decide
that the major releases are unsafe and eventually avoid them entirely.
               Jack