similar to: [LLVMdev] New -O3 Performance tester - Use hardware to get reliable numbers

Hi, I am currently investigating how to ensure that LNT only shows relevant performance regressions for the -O3 performance tests I am running. One question that came up here is why the default aggregate function for LNT is 'min' instead of 'mean'. This looks a little surprising from the statistical point, but also from looking at my test results picking 'min' seems

Dear all, Following the Benchmarking BOF from 2013 US dev meeting, I’d like to propose some improvements to the LNT performance tracking software. The most significant issue with current implementation is that the report is filled with extremely noisy values. Hence it is hard to notice performance improvements or regressions. After investigation of LNT and the LLVM test suite, I propose

Right - you usually won't see a normal distribution in the noise of test results. You'll see results clustered around the lower bound with a long tail of slower and slower results. Depending on how many samples you do it might be appropriate to take the mean of the best 3, for example - but the general approach of taking the fastest N does have some basis in any case. Not necessarily the

On 06/28/2013 01:19 PM, Renato Golin wrote: > On 28 June 2013 19:45, Chris Matthews <chris.matthews at apple.com> > wrote: > >> Given this tradeoff I think we want to tend towards false positives >> (over false negatives) strictly as a matter of compiler quality. >> > > False hits are not binary, but (at least) two-dimensional. You can't > say it's

tl;dr in low data situations we don’t look at past information, and that increases the false positive regression rate. We should look at the possibly incorrect recent past runs to fix that. Motivation: LNT’s current regression detection system has false positive rate that is too high to make it useful. With test suites as large as the llvm “test-suite” a single report will show hundreds of

On Thu, Jan 16, 2014 at 5:32 PM, Tobias Grosser <tobias at grosser.es> wrote: > On 01/17/2014 02:17 AM, David Blaikie wrote: > >> Right - you usually won't see a normal distribution in the noise of test >> results. You'll see results clustered around the lower bound with a long >> tail of slower and slower results. Depending on how many samples you do it

On 20 May 2014 17:55, Tobias Grosser <tobias at grosser.es> wrote: > On 20/05/2014 18:20, Yi Kong wrote: >> >> On 20 May 2014 16:40, Tobias Grosser <tobias at grosser.es> wrote: >>> >>> On 20/05/2014 16:01, Yi Kong wrote: >>>> >>>> >>>> I've set up a public LNT server to show the result of perf stat. There

On Tue, Jan 7, 2014 at 8:48 PM, Sean Silva <chisophugis at gmail.com> wrote: > sean:~/pg/llvm/llvm % git log --oneline --since='1 month ago' | wc -l > 706 > sean:~/pg/llvm/llvm % git log --oneline --since='1 month ago' ./test | wc -l > 317 Wouldn't this also catch commits to code generation that added tests as well? Diego.

On 06/30/2013 02:14 AM, Anton Korobeynikov wrote: > Hi Tobi, > > First of all, all this is http://llvm.org/bugs/show_bug.cgi?id=1367 :) > >> The statistical test ministat is performing seems simple and pretty >> standard. Is there any reason we could not do something similar? Or are we >> doing it already and it just does not work as expected? > The main problem

The debug info statistics bot is triggered by this job: http://green.lab.llvm.org/green/job/clang-stage2-Rthinlto/ which unfortunately hasn't been green in a very long time (>1mo). Alex/Azhar, do you know what's blocking that job? -- adrian > On Nov 20, 2019, at 9:46 AM, David Blaikie <dblaikie at gmail.com> wrote: > > +usual debug info folks (but I think in this case

Hi llvm-dev@ LNT produces statistics and graphs (such as [0]) of debuginfo metrics, such as number of source variables with locations. It looks like these haven't run [1] since the move from svn to git -- are there any plans to get these running again? I find it highly useful to identify what commits have affected variable locations and how significant an affect. [0]

Hi Tobi, First of all, all this is http://llvm.org/bugs/show_bug.cgi?id=1367 :) > The statistical test ministat is performing seems simple and pretty > standard. Is there any reason we could not do something similar? Or are we > doing it already and it just does not work as expected? The main problem with such sort of tests is that we cannot trust them, unless: 1. The data has the

Hi, The new LNT server instance http://lnt.llvm.org seems to fail in many cases. Any entrance to a 'Run page' (e.g. http://lnt.llvm.org/db_default/v4/nts/62475) and lately also many perf bots result submissions (e.g. http://lab.llvm.org:8014/builders/clang-native-arm-lnt-perf/builds/2262/steps/test-suite/logs/stdio ) fails with: "500 Internal Server Error". Any ideas? Thanks,

On Wed, Jan 8, 2014 at 4:07 PM, Sean Silva <chisophugis at gmail.com> wrote: > > > > On Wed, Jan 8, 2014 at 7:58 AM, Diego Novillo <dnovillo at google.com> wrote: >> >> On Tue, Jan 7, 2014 at 8:48 PM, Sean Silva <chisophugis at gmail.com> wrote: >> >> > sean:~/pg/llvm/llvm % git log --oneline --since='1 month ago' | wc -l >>

The run page problem were triggered by one of my commits (sorry) and should be mitigated now, see the thread at http://lists.llvm.org/pipermail/llvm-dev/2017-July/115971.html <http://lists.llvm.org/pipermail/llvm-dev/2017-July/115971.html> I don't know about the submission problems, could they just an occasional network problem or are they a common phenomenon? Chris did some

Hi Tobias, > I trust your knowledge about statistics, but am wondering why ministat (and > it's t-test) is promoted as a statistical sane tool for benchmarking > results. I do not know... Ask author of ministat? > Is the use of the t-test for benchmark results a bad idea in > general? No, in general. But one should be aware about the assumptions of the underlying theory.

> On Sep 1, 2016, at 8:14 AM, Renato Golin via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > On 1 September 2016 at 07:45, Dean Michael Berris via llvm-dev > <llvm-dev at lists.llvm.org> wrote: >> I've lately been wondering where benchmarks for LLVM-generated binaries are hosted, and whether they're tracked over time. > > Hi Dean, > > Do you

Hi Eric, Yeah, I know about Externals and SPEC specifically. But as far as I understand, you have to have kind of description of the tests in test-suite even if you don’t provide the source codes - that’s what I would like to avoid. I.e. you have to have CMakeLists.txt and other files in place all the time, open to everyone. Now, imagine I have a small testsuite, which probably is not very

Is O3-vectorize redundant now that the loop vectorizer is enabled by default? On 2013-01-28, at 12:25 PM, David Blaikie wrote: > On Mon, Jan 28, 2013 at 8:57 AM, Redmond, Paul <paul.redmond at intel.com> wrote: >> Is there a reason why existing buildbots are not generating LNT results? > > Those running LNT should be/are: > >

On 28 June 2013 19:45, Chris Matthews <chris.matthews at apple.com> wrote: > Given this tradeoff I think we want to tend towards false positives (over > false negatives) strictly as a matter of compiler quality. > False hits are not binary, but (at least) two-dimensional. You can't say it's better to have any amount of false positives than any amount of false negatives