On Wed, Feb 18, 2015 at 1:39 PM, Joerg Sonnenberger <joerg at britannica.bec.de> wrote:> On Wed, Feb 18, 2015 at 12:04:47PM -0500, Jack Howarth wrote: >> My concern is that, without strict enforcement of the triaging >> serious P1-type bugs, the major llvm.org releases will devolve into a >> continual exchange of one set of major regressions for another set. > > A performance regression is not a P1 type bug. Especially if the change > has existed for a while and noone noticed.This issue has been publicly reported since 3.6-RC1... http://www.phoronix.com/scan.php?page=article&item=llvm-clang-3.5-3.6-rc1&num=2> > Joerg > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On 18 February 2015 at 18:54, Jack Howarth <howarth.mailing.lists at gmail.com> wrote:> http://www.phoronix.com/scan.php?page=article&item=llvm-clang-3.5-3.6-rc1&num=2First, I have to say, I was saddened by that post, since the new release regress in a lot of the benchmarks, but not enough to stop the world and fix all of those regressions. I have seen other similar regressions in my internal benchmarks, but again, not enough to stop everything. My focus now is to make it work well. I'll only focus on performance when the whole toolchain is working as I expect, or when I have more people to look at performance in parallel. I assume other people feel in the same way. Second, how many benchmarks are there, and which of them classify as "important"? Should we stop for 20% regressions on any benchmark on the Internet? SciMark is a well known benchmark, I give you that, but to be honest, it means nothing to me. Maybe it will, one day, but not today, and probably not for another year or two. It seems that others in the community feel in the same way, or we would have seen a lot of people jumping up and down until the regression got fixed. Hal and others seem to be on top of the issue, and I assume they're the ones most interested in getting that fixed. If they're happy with it going to 3.6.1, an no one else is trying to get that fixed, I'm happy with that outcome. As Hans said, we'll probably have an RC4. If you're so worried with this specific regression, can you try to fix it? It seems two commits are responsible in equal measure, so it shouldn't be too hard to see what they do, produce the code with and without each one of them, and see why things go slower. I'm not an expert on x86, and I didn't even run that benchmark on ARM/AArch64, so I can't possibly help. cheers, --renato
Joerg Sonnenberger
2015-Feb-18 20:08 UTC
[LLVMdev] [cfe-dev] [3.6 Release] RC3 has been tagged
On Wed, Feb 18, 2015 at 01:54:31PM -0500, Jack Howarth wrote:> On Wed, Feb 18, 2015 at 1:39 PM, Joerg Sonnenberger > <joerg at britannica.bec.de> wrote: > > On Wed, Feb 18, 2015 at 12:04:47PM -0500, Jack Howarth wrote: > >> My concern is that, without strict enforcement of the triaging > >> serious P1-type bugs, the major llvm.org releases will devolve into a > >> continual exchange of one set of major regressions for another set. > > > > A performance regression is not a P1 type bug. Especially if the change > > has existed for a while and noone noticed. > > This issue has been publicly reported since 3.6-RC1... > > http://www.phoronix.com/scan.php?page=article&item=llvm-clang-3.5-3.6-rc1&num=2A post in some obscure forum does not count as bug report. Even then: the change in trunk that triggered it has been in the tree for while. Joerg
On 18 February 2015 at 20:08, Joerg Sonnenberger <joerg at britannica.bec.de> wrote:> A post in some obscure forum does not count as bug report.Phronix is not *that* obscure, but yes, it's NOT a bug report.> Even then: the change in trunk that triggered it has been in the tree > for while.I believe that was your original point for "been there for a while". cheers, --renato
----- Original Message -----> From: "Renato Golin" <renato.golin at linaro.org> > To: "Jack Howarth" <howarth.mailing.lists at gmail.com> > Cc: "cfe-dev" <cfe-dev at cs.uiuc.edu>, "llvmdev" <llvmdev at cs.uiuc.edu> > Sent: Wednesday, February 18, 2015 1:55:38 PM > Subject: Re: [LLVMdev] [cfe-dev] [3.6 Release] RC3 has been tagged > > On 18 February 2015 at 18:54, Jack Howarth > <howarth.mailing.lists at gmail.com> wrote: > > http://www.phoronix.com/scan.php?page=article&item=llvm-clang-3.5-3.6-rc1&num=2 > > First, I have to say, I was saddened by that post, since the new > release regress in a lot of the benchmarks, but not enough to stop > the > world and fix all of those regressions. > > I have seen other similar regressions in my internal benchmarks, but > again, not enough to stop everything. My focus now is to make it work > well. I'll only focus on performance when the whole toolchain is > working as I expect, or when I have more people to look at > performance > in parallel. I assume other people feel in the same way. > > Second, how many benchmarks are there, and which of them classify as > "important"? Should we stop for 20% regressions on any benchmark on > the Internet? SciMark is a well known benchmark, I give you that, but > to be honest, it means nothing to me. Maybe it will, one day, but not > today, and probably not for another year or two. > > It seems that others in the community feel in the same way, or we > would have seen a lot of people jumping up and down until the > regression got fixed. Hal and others seem to be on top of the issue,There are a number of us who watch performance numbers regularly, certainly more now then there used to be, but this is still an area where we need improvement. Also, regarding SciMark, this benchmark is already in our test suite (in MultiSource/Benchmarks/SciMark2-C), but this only gives us the composite over all of the SciMark tests. We should probably split this out so that we get timings for the individual tests (or relevant groups) in LNT. This is what I did for the TSVC benchmark, for example. Jack, would you be interested in working on this? -Hal> and I assume they're the ones most interested in getting that fixed. > If they're happy with it going to 3.6.1, an no one else is trying to > get that fixed, I'm happy with that outcome. > > As Hans said, we'll probably have an RC4. If you're so worried with > this specific regression, can you try to fix it? It seems two commits > are responsible in equal measure, so it shouldn't be too hard to see > what they do, produce the code with and without each one of them, and > see why things go slower. I'm not an expert on x86, and I didn't even > run that benchmark on ARM/AArch64, so I can't possibly help. > > cheers, > --renato > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory