Folks, First win of the benchmark buildbot! http://llvm.org/perf/db_default/v4/nts/graph?plot.0=49.128.2&highlight_run=31861 It seems mandel-2 had a huge regression a few commits ago, and based on a quick look, it may have to do with the inst combine changes. I haven't investigated yet, but this is the first time I spot regressions on test-suite, so I'd like to first congratulate the people that worked on making that a reality! Now, I'd like to have been warned of that automatically, by email, based on some simple heuristics... But that's a topic for the Performance/Benchmarking BoF in a few weeks... :) cheers, --renato
----- Original Message -----> From: "Renato Golin" <renato.golin at linaro.org> > To: "LLVM Dev" <llvmdev at cs.uiuc.edu> > Sent: Thursday, October 16, 2014 2:47:11 AM > Subject: [LLVMdev] Performance regression on ARM > > Folks, > > First win of the benchmark buildbot! > > http://llvm.org/perf/db_default/v4/nts/graph?plot.0=49.128.2&highlight_run=31861 > > It seems mandel-2 had a huge regression a few commits ago, and based > on a quick look, it may have to do with the inst combine changes. I > haven't investigated yet, but this is the first time I spot > regressions on test-suite, so I'd like to first congratulate the > people that worked on making that a reality!Interesting. Looks like the problem is in (219545, 219569]. Based on the commit logs, it does seem like the best candidates are: r219566, r219567, r219568 - InstCombine bug fixes re: no-wrap flags r219550, r219562 - Fixes to trip-count computations in the vectorizer/unroller and given the magnitude of the change, I think that the trip-count changes are more likely. Of course, all of these things are bug fixes :( -- So how do we follow-up on this? -Hal> > Now, I'd like to have been warned of that automatically, by email, > based on some simple heuristics... But that's a topic for the > Performance/Benchmarking BoF in a few weeks... :) > > cheers, > --renato > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory
On 16 October 2014 09:34, Hal Finkel <hfinkel at anl.gov> wrote:> Interesting. Looks like the problem is in (219545, 219569].Yes.> and given the magnitude of the change, I think that the trip-count changes are more likely.Good point.> Of course, all of these things are bug fixes :( -- So how do we follow-up on this?Correctness before performance. Always. I'll create a bug on this pointing to the delta for someone to investigate. It doesn't have to be me, or the committer, the idea is that this is not high priority. At least, not for now. Once we have a way to track this more consistently, I think a good approach is to be pragmatic. So, something along the lines of working with the implementer, trying to understand the reason of the regression. It could be a bad implementation or just a target-specific reason for the regression. Depending on the importance of the regression and of the patch, I'd consider turning it off for ARM (for example, PR18996) and later investigating why. I'd only consider reverting the patch in extreme circumstances, for example, if we're close to a release AND the regression is big AND the patch is a new feature, etc. I believe that's what you were concerned about. :) cheers, --renato
On Thu, Oct 16, 2014 at 12:47 AM, Renato Golin <renato.golin at linaro.org> wrote:> Folks, > > First win of the benchmark buildbot! >Yay! Out of curiosity, how long has it/you been monitoring for regressions? -- Sean Silva> > > http://llvm.org/perf/db_default/v4/nts/graph?plot.0=49.128.2&highlight_run=31861 > > It seems mandel-2 had a huge regression a few commits ago, and based > on a quick look, it may have to do with the inst combine changes. I > haven't investigated yet, but this is the first time I spot > regressions on test-suite, so I'd like to first congratulate the > people that worked on making that a reality! > > Now, I'd like to have been warned of that automatically, by email, > based on some simple heuristics... But that's a topic for the > Performance/Benchmarking BoF in a few weeks... :) > > cheers, > --renato > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20141016/13b24c12/attachment.html>
On 17 October 2014 06:19, Sean Silva <chisophugis at gmail.com> wrote:> Out of curiosity, how long has it/you been monitoring for regressions?I'm not. :) I was looking for a few examples for my talk at LPC and found this by accident. That's why I wanted to have some automated warnings on a low-noise regression score. cheers, --renato