Hi Jack, On Sun, Feb 15, 2015 at 6:05 AM, Jack Howarth <howarth.mailing.lists at gmail.com> wrote:> What are the acceptable performance regressions in the generated > code for a llvm release? We seem to be badly regressed in some > benchmarks (which I first noticed from the review of 3.6-rc1 at > http://www.phoronix.com/scan.php?page=article&item=llvm-clang-3.5-3.6-rc1). > This same issue has also been reported in > http://llvm.org/bugs/show_bug.cgi?id=22058. In the case of the 22% > performance degradation in SciMark2's Sparse matmult benchmark, I have > identified both commits that contribute equally to this regression in > http://llvm.org/bugs/show_bug.cgi?id=22589...Thank you very much for trying out the release candidate. I asked a few of the other developers, and the consensus was that while unfortunate, we won't block the release on a perf regression like this, at least not at this stage in the release process. Having said that, we will be doing an rc4, and Hal said that he or Sanjoy might be able to squeeze in a patch for the issue you pointed out before that. Otherwise, it will have to wait to 3.6.1 or 3.7. Thanks, Hans
In the interest of educating others, can someone explain what the regression is? Is it understood? I'm new to this area and would love to understand how we identified what is causing this regression? An optimization gone rogue? Some optimization missing, etc. On Tue, Feb 17, 2015 at 11:12 AM, Hans Wennborg <hans at chromium.org> wrote:> Hi Jack, > > On Sun, Feb 15, 2015 at 6:05 AM, Jack Howarth > <howarth.mailing.lists at gmail.com> wrote: > > What are the acceptable performance regressions in the generated > > code for a llvm release? We seem to be badly regressed in some > > benchmarks (which I first noticed from the review of 3.6-rc1 at > > > http://www.phoronix.com/scan.php?page=article&item=llvm-clang-3.5-3.6-rc1 > ). > > This same issue has also been reported in > > http://llvm.org/bugs/show_bug.cgi?id=22058. In the case of the 22% > > performance degradation in SciMark2's Sparse matmult benchmark, I have > > identified both commits that contribute equally to this regression in > > http://llvm.org/bugs/show_bug.cgi?id=22589... > > Thank you very much for trying out the release candidate. > > I asked a few of the other developers, and the consensus was that > while unfortunate, we won't block the release on a perf regression > like this, at least not at this stage in the release process. > > Having said that, we will be doing an rc4, and Hal said that he or > Sanjoy might be able to squeeze in a patch for the issue you pointed > out before that. Otherwise, it will have to wait to 3.6.1 or 3.7. > > Thanks, > Hans > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150217/e398e98d/attachment.html>
On Tue, Feb 17, 2015 at 2:12 PM, Hans Wennborg <hans at chromium.org> wrote:> Hi Jack, > > On Sun, Feb 15, 2015 at 6:05 AM, Jack Howarth > <howarth.mailing.lists at gmail.com> wrote: >> What are the acceptable performance regressions in the generated >> code for a llvm release? We seem to be badly regressed in some >> benchmarks (which I first noticed from the review of 3.6-rc1 at >> http://www.phoronix.com/scan.php?page=article&item=llvm-clang-3.5-3.6-rc1). >> This same issue has also been reported in >> http://llvm.org/bugs/show_bug.cgi?id=22058. In the case of the 22% >> performance degradation in SciMark2's Sparse matmult benchmark, I have >> identified both commits that contribute equally to this regression in >> http://llvm.org/bugs/show_bug.cgi?id=22589... > > Thank you very much for trying out the release candidate. > > I asked a few of the other developers, and the consensus was that > while unfortunate, we won't block the release on a perf regression > like this, at least not at this stage in the release process. > > Having said that, we will be doing an rc4, and Hal said that he or > Sanjoy might be able to squeeze in a patch for the issue you pointed > out before that. Otherwise, it will have to wait to 3.6.1 or 3.7. > > Thanks, > HansHans, We probably ought to open a meta-bug for all of these new performance regressions introduced in 3.6. The 20% performance regression observed in the "8 queens puzzle" solver example (http://llvm.org/bugs/show_bug.cgi?id=22058) doesn't seem to be introduced by either the r217953 or r222451 commits. I'll try to do a regression hunt later this week to pin down the offending commit for that regression. Likewise, the reported regression in the -m32 performance of the sudoku solver example (reported in http://llvm.org/bugs/show_bug.cgi?id=22589) doesn't match up to those two commits either (but I suspect a register pressure issue since it only shows up at 32-bit). Jack ps Do we have a complete set of benchmark results for current 3.6svn vs 3.5.1 posted anywhere? Hopefully some these are showing up the SPEC benchmarks and should be added to the meta-bug by someone who has access to the sources for those benchmarks.
On Tue, Feb 17, 2015 at 2:44 PM, Hayden Livingston <halivingston at gmail.com> wrote:> In the interest of educating others, can someone explain what the regression > is? Is it understood? > > I'm new to this area and would love to understand how we identified what is > causing this regression? An optimization gone rogue? Some optimization > missing, etc. >Hayden, See... http://llvm.org/bugs/show_bug.cgi?id=22589#c5 http://llvm.org/bugs/show_bug.cgi?id=22589#c7 http://llvm.org/bugs/show_bug.cgi?id=22589#c9 http://llvm.org/bugs/show_bug.cgi?id=22589#c26 http://llvm.org/bugs/show_bug.cgi?id=22589#c27 http://llvm.org/bugs/show_bug.cgi?id=22589#c33 http://llvm.org/bugs/show_bug.cgi?id=22589#c36 which are the key postings on this bug. I would describe it as an optimization no longer taken. As for identifying cause of regression, the process is a brutal walk back through prior svn trunk until the offending commit which cause the loss of optimization is pin-pointed. Jack> On Tue, Feb 17, 2015 at 11:12 AM, Hans Wennborg <hans at chromium.org> wrote: >> >> Hi Jack, >> >> On Sun, Feb 15, 2015 at 6:05 AM, Jack Howarth >> <howarth.mailing.lists at gmail.com> wrote: >> > What are the acceptable performance regressions in the generated >> > code for a llvm release? We seem to be badly regressed in some >> > benchmarks (which I first noticed from the review of 3.6-rc1 at >> > >> > http://www.phoronix.com/scan.php?page=article&item=llvm-clang-3.5-3.6-rc1). >> > This same issue has also been reported in >> > http://llvm.org/bugs/show_bug.cgi?id=22058. In the case of the 22% >> > performance degradation in SciMark2's Sparse matmult benchmark, I have >> > identified both commits that contribute equally to this regression in >> > http://llvm.org/bugs/show_bug.cgi?id=22589... >> >> Thank you very much for trying out the release candidate. >> >> I asked a few of the other developers, and the consensus was that >> while unfortunate, we won't block the release on a perf regression >> like this, at least not at this stage in the release process. >> >> Having said that, we will be doing an rc4, and Hal said that he or >> Sanjoy might be able to squeeze in a patch for the issue you pointed >> out before that. Otherwise, it will have to wait to 3.6.1 or 3.7. >> >> Thanks, >> Hans >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > >
On Tue, Feb 17, 2015 at 11:47 AM, Jack Howarth <howarth.mailing.lists at gmail.com> wrote:> We probably ought to open a meta-bug for all of these new > performance regressions introduced in 3.6.Feel free to open one. The important thing is whether anyone is investigating the bugs, though.> The 20% performance > regression observed in the "8 queens puzzle" solver example > (http://llvm.org/bugs/show_bug.cgi?id=22058) doesn't seem to be > introduced by either the r217953 or r222451 commits. I'll try to do a > regression hunt later this week to pin down the offending commit for > that regression. Likewise, the reported regression in the -m32 > performance of the sudoku solver example (reported in > http://llvm.org/bugs/show_bug.cgi?id=22589) doesn't match up to those > two commits either (but I suspect a register pressure issue since it > only shows up at 32-bit). > Jack > ps Do we have a complete set of benchmark results for current 3.6svn > vs 3.5.1 posted anywhere?Not that I know. There are test results going into the LNT tool at http://llvm.org/perf, but I'm not really familiar with that and also don't think it makes it easy to do comparisons between releases. - Hans
Joerg Sonnenberger
2015-Feb-18 13:42 UTC
[LLVMdev] [cfe-dev] [3.6 Release] RC3 has been tagged
On Tue, Feb 17, 2015 at 11:12:20AM -0800, Hans Wennborg wrote:> I asked a few of the other developers, and the consensus was that > while unfortunate, we won't block the release on a perf regression > like this, at least not at this stage in the release process.Wasn't the result of the discussion much stronger -- the performance regression seems to be more a result of hypersensitivity to specific code generation choices and less a fundamental issue? Joerg
----- Original Message -----> From: "Joerg Sonnenberger" <joerg at britannica.bec.de> > To: cfe-dev at cs.uiuc.edu, "llvmdev" <llvmdev at cs.uiuc.edu> > Sent: Wednesday, February 18, 2015 7:42:11 AM > Subject: Re: [LLVMdev] [cfe-dev] [3.6 Release] RC3 has been tagged > > On Tue, Feb 17, 2015 at 11:12:20AM -0800, Hans Wennborg wrote: > > I asked a few of the other developers, and the consensus was that > > while unfortunate, we won't block the release on a perf regression > > like this, at least not at this stage in the release process. > > Wasn't the result of the discussion much stronger -- the performance > regression seems to be more a result of hypersensitivity to specific > code generation choices and less a fundamental issue?There are several commits involved, and I don't believe we understand whether the issues are all hypersensitivity, or some other more systematic interactions. There is one issue, which involved a correctness fix for runtime unrolling that was done in a sub-optimal way (causing it to introduce an unnecessary comparison) that I think is well understood, and for which we have a fix (http://reviews.llvm.org/D7715). We should pick up this in the release. I think that, unfortunately, for the others, there's not sufficient time to investigate before the release. -Hal> > Joerg > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >