On Wed, Feb 18, 2015 at 11:22 AM, Jack Howarth <howarth.mailing.lists at gmail.com> wrote:> On Wed, Feb 18, 2015 at 10:55 AM, Jack Howarth > <howarth.mailing.lists at gmail.com> wrote: >> I finally got around to testing this on a Bloomfield processor (Early >> 2009 MacPro 2x2.66 GHz dual-quad core) and the regressions from >> http://llvm.org/bugs/show_bug.cgi?id=22589 are even more severe. For >> 10 runs of scimark2_1c built with "-O3 -march=native"... >> >> llvm 3.5.1 1204.16+/-2.66 Mflops >> 3.6 branch 866.49+/-1.26 Mflops >> > > The proposed patch mitigates the damage on Bloomfield... > > patched 3.6svn 1073.69+/-1.97 > > so we are only regressing the benchmark 24% rather than 39% there. > Jack >This still all begs the question of what exact metrics exist for the Q/A of llvm releases? IMHO, the bad PR from shoving out compiler releases with severe performance regressions in the generated code far outweighs a brief delay to triage these issues as much as possible. Jack> >> Do you seriously want to ship with a 39% performance regression in a >> major benchmark? >> Jack >> >> On Wed, Feb 18, 2015 at 10:33 AM, Renato Golin <renato.golin at linaro.org> wrote: >>> On 18 February 2015 at 14:37, Hal Finkel <hfinkel at anl.gov> wrote: >>>> I think that, unfortunately, for the others, there's not sufficient time to investigate before the release. >>> >>> This looks like a serious case for 3.6.1, not RC5. >>> >>> cheers, >>> --renato >>> _______________________________________________ >>> LLVM Developers mailing list >>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On 18 February 2015 at 16:39, Jack Howarth <howarth.mailing.lists at gmail.com> wrote:> This still all begs the question of what exact metrics exist for the > Q/A of llvm releases? IMHO, the bad PR from shoving out compiler > releases with severe performance regressions in the generated code far > outweighs a brief delay to triage these issues as much as possible.Hi Jack, It depends on how you interpret LLVM releases. I still see them as points in time where we actually did some deeper investigations as a community and made sure the compiler is still sane. If it's not, future points in time shall fix it, for example, on dot releases or next releases, depending on how severe the regression is. A while ago we kind of agreed on a release Go/NoGo table that was somewhat like this: Before release: * Any conformance regressions * Bad conformance breakage (new) Next dot release: * Any conformance breakage * Bad performance regression Next release: * Any performance regression Since most people use LLVM on top-of-tree, the problem of having a release that is not as good as the previous is somewhat washed away by the fact that we now have dot-releases, and can fix the horrible cases before a new full release comes in 6 months time. Also, we don't have a limit for dot releases, and last year we had a dot release almost right after the major release due to similar problems. So, what we release now won't necessarily go to distributions, if we continue the process on the next dot-release, 3.6.1. I think we all agree that this is a "bad performance regression", but most of us think this is not bad enough to delay the release further, but probably bad enough to start the next dot release right after the end of this one. Does that really sound like such a bad idea? cheers, --renato
On Wed, Feb 18, 2015 at 11:53 AM, Renato Golin <renato.golin at linaro.org> wrote:> On 18 February 2015 at 16:39, Jack Howarth > <howarth.mailing.lists at gmail.com> wrote: >> This still all begs the question of what exact metrics exist for the >> Q/A of llvm releases? IMHO, the bad PR from shoving out compiler >> releases with severe performance regressions in the generated code far >> outweighs a brief delay to triage these issues as much as possible. > > Hi Jack, > > It depends on how you interpret LLVM releases. > > I still see them as points in time where we actually did some deeper > investigations as a community and made sure the compiler is still > sane. If it's not, future points in time shall fix it, for example, on > dot releases or next releases, depending on how severe the regression > is. > > A while ago we kind of agreed on a release Go/NoGo table that was > somewhat like this: > > Before release: > * Any conformance regressions > * Bad conformance breakage (new) > > Next dot release: > * Any conformance breakage > * Bad performance regression > > Next release: > * Any performance regression > > Since most people use LLVM on top-of-tree, the problem of having a > release that is not as good as the previous is somewhat washed away by > the fact that we now have dot-releases, and can fix the horrible cases > before a new full release comes in 6 months time. > > Also, we don't have a limit for dot releases, and last year we had a > dot release almost right after the major release due to similar > problems. So, what we release now won't necessarily go to > distributions, if we continue the process on the next dot-release, > 3.6.1. > > I think we all agree that this is a "bad performance regression", but > most of us think this is not bad enough to delay the release further, > but probably bad enough to start the next dot release right after the > end of this one. > > Does that really sound like such a bad idea? > > cheers, > --renatoRenato, My concern is that, without strict enforcement of the triaging serious P1-type bugs, the major llvm.org releases will devolve into a continual exchange of one set of major regressions for another set. Strict enforcement of back porting the required fixes from later trunk for point releases seem to require as just much effort and even more due diligence than just defining certain types of bugs as release blockers and dealing with them upfront. Otherwise, folks will decide that the major releases are unsafe and eventually avoid them entirely. Jack