On Wed, Feb 18, 2015 at 11:53 AM, Renato Golin <renato.golin at linaro.org> wrote:> On 18 February 2015 at 16:39, Jack Howarth > <howarth.mailing.lists at gmail.com> wrote: >> This still all begs the question of what exact metrics exist for the >> Q/A of llvm releases? IMHO, the bad PR from shoving out compiler >> releases with severe performance regressions in the generated code far >> outweighs a brief delay to triage these issues as much as possible. > > Hi Jack, > > It depends on how you interpret LLVM releases. > > I still see them as points in time where we actually did some deeper > investigations as a community and made sure the compiler is still > sane. If it's not, future points in time shall fix it, for example, on > dot releases or next releases, depending on how severe the regression > is. > > A while ago we kind of agreed on a release Go/NoGo table that was > somewhat like this: > > Before release: > * Any conformance regressions > * Bad conformance breakage (new) > > Next dot release: > * Any conformance breakage > * Bad performance regression > > Next release: > * Any performance regression > > Since most people use LLVM on top-of-tree, the problem of having a > release that is not as good as the previous is somewhat washed away by > the fact that we now have dot-releases, and can fix the horrible cases > before a new full release comes in 6 months time. > > Also, we don't have a limit for dot releases, and last year we had a > dot release almost right after the major release due to similar > problems. So, what we release now won't necessarily go to > distributions, if we continue the process on the next dot-release, > 3.6.1. > > I think we all agree that this is a "bad performance regression", but > most of us think this is not bad enough to delay the release further, > but probably bad enough to start the next dot release right after the > end of this one. > > Does that really sound like such a bad idea? > > cheers, > --renatoRenato, My concern is that, without strict enforcement of the triaging serious P1-type bugs, the major llvm.org releases will devolve into a continual exchange of one set of major regressions for another set. Strict enforcement of back porting the required fixes from later trunk for point releases seem to require as just much effort and even more due diligence than just defining certain types of bugs as release blockers and dealing with them upfront. Otherwise, folks will decide that the major releases are unsafe and eventually avoid them entirely. Jack
On Wed, Feb 18, 2015 at 12:04 PM, Jack Howarth <howarth.mailing.lists at gmail.com> wrote:> On Wed, Feb 18, 2015 at 11:53 AM, Renato Golin <renato.golin at linaro.org> wrote: >> On 18 February 2015 at 16:39, Jack Howarth >> <howarth.mailing.lists at gmail.com> wrote: >>> This still all begs the question of what exact metrics exist for the >>> Q/A of llvm releases? IMHO, the bad PR from shoving out compiler >>> releases with severe performance regressions in the generated code far >>> outweighs a brief delay to triage these issues as much as possible. >> >> Hi Jack, >> >> It depends on how you interpret LLVM releases. >> >> I still see them as points in time where we actually did some deeper >> investigations as a community and made sure the compiler is still >> sane. If it's not, future points in time shall fix it, for example, on >> dot releases or next releases, depending on how severe the regression >> is. >> >> A while ago we kind of agreed on a release Go/NoGo table that was >> somewhat like this: >> >> Before release: >> * Any conformance regressions >> * Bad conformance breakage (new) >> >> Next dot release: >> * Any conformance breakage >> * Bad performance regression >> >> Next release: >> * Any performance regression >> >> Since most people use LLVM on top-of-tree, the problem of having a >> release that is not as good as the previous is somewhat washed away by >> the fact that we now have dot-releases, and can fix the horrible cases >> before a new full release comes in 6 months time. >> >> Also, we don't have a limit for dot releases, and last year we had a >> dot release almost right after the major release due to similar >> problems. So, what we release now won't necessarily go to >> distributions, if we continue the process on the next dot-release, >> 3.6.1. >> >> I think we all agree that this is a "bad performance regression", but >> most of us think this is not bad enough to delay the release further, >> but probably bad enough to start the next dot release right after the >> end of this one. >> >> Does that really sound like such a bad idea? >> >> cheers, >> --renato > > Renato, > My concern is that, without strict enforcement of the triaging > serious P1-type bugs, the major llvm.org releases will devolve into a > continual exchange of one set of major regressions for another set. > Strict enforcement of back porting the required fixes from later trunk > for point releases seem to require as just much effort and even more > due diligence than just defining certain types of bugs as release > blockers and dealing with them upfront. Otherwise, folks will decide > that the major releases are unsafe and eventually avoid them entirely. > JackRenato, I also find it very telling that Apple avoided syncing their clang to the 3.6svn release for the Xcode 6.2 public betas after doing so habitually for past releases. Might that not say something about the state of 3.6? Jack
On 18 February 2015 at 17:04, Jack Howarth <howarth.mailing.lists at gmail.com> wrote:> Renato, > My concern is that, without strict enforcement of the triaging > serious P1-type bugs, the major llvm.org releases will devolve into a > continual exchange of one set of major regressions for another set.This is a very pessimistic view of our community. We do care about bugs and we do tend to fix it pretty quickly, even without the beating drum of a release cycle.> Otherwise, folks will decide that the major releases are unsafe and eventually avoid them entirely.As I said, releases are not meant to be a product, but a point in time. If you want do build a product out of a release, I suggest you to always wait for the dot-releases, as they have the fixes for problems that were found when spinning the release, while keeping ABI/API compatibility. You are expecting our releases to be something it's not, and it's not a surprise if that doesn't abide by the level of quality you aspire. Most, if not all users don't rely that much on releases and tend to pick a commit in between that suit their needs. This is not just Apple, as you mentioned, but almost every one do that. Furthermore, new versions of GCC *also* have major regressions in some benchmarks and they also release and fix on the next dot-release. cheers, --renato
Joerg Sonnenberger
2015-Feb-18 18:39 UTC
[LLVMdev] [cfe-dev] [3.6 Release] RC3 has been tagged
On Wed, Feb 18, 2015 at 12:04:47PM -0500, Jack Howarth wrote:> My concern is that, without strict enforcement of the triaging > serious P1-type bugs, the major llvm.org releases will devolve into a > continual exchange of one set of major regressions for another set.A performance regression is not a P1 type bug. Especially if the change has existed for a while and noone noticed. Joerg
On Wed, Feb 18, 2015 at 1:04 PM, Renato Golin <renato.golin at linaro.org> wrote:> On 18 February 2015 at 17:04, Jack Howarth > <howarth.mailing.lists at gmail.com> wrote: >> Renato, >> My concern is that, without strict enforcement of the triaging >> serious P1-type bugs, the major llvm.org releases will devolve into a >> continual exchange of one set of major regressions for another set. > > This is a very pessimistic view of our community. We do care about > bugs and we do tend to fix it pretty quickly, even without the beating > drum of a release cycle. > > >> Otherwise, folks will decide that the major releases are unsafe and eventually avoid them entirely. > > As I said, releases are not meant to be a product, but a point in > time. If you want do build a product out of a release, I suggest you > to always wait for the dot-releases, as they have the fixes for > problems that were found when spinning the release, while keeping > ABI/API compatibility. > > You are expecting our releases to be something it's not, and it's not > a surprise if that doesn't abide by the level of quality you aspire. > Most, if not all users don't rely that much on releases and tend to > pick a commit in between that suit their needs. This is not just > Apple, as you mentioned, but almost every one do that. Furthermore, > new versions of GCC *also* have major regressions in some benchmarks > and they also release and fix on the next dot-release.Define users. If compiler developers, your argument holds. For other users who just want a stable compiler, the argument does not. As for FSF gcc, I disagree. If an egregious regression in an important benchmarks is discovered, in general they would escalate it to a P1 and most certainly fix it before release. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65028 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65006 Especially when the problem is understood and a fix is at hand.> > cheers, > --renato
On Wed, Feb 18, 2015 at 1:39 PM, Joerg Sonnenberger <joerg at britannica.bec.de> wrote:> On Wed, Feb 18, 2015 at 12:04:47PM -0500, Jack Howarth wrote: >> My concern is that, without strict enforcement of the triaging >> serious P1-type bugs, the major llvm.org releases will devolve into a >> continual exchange of one set of major regressions for another set. > > A performance regression is not a P1 type bug. Especially if the change > has existed for a while and noone noticed.This issue has been publicly reported since 3.6-RC1... http://www.phoronix.com/scan.php?page=article&item=llvm-clang-3.5-3.6-rc1&num=2> > Joerg > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev