search for: num_comparison_run

...lts picking 'min' seems to be an inferior choice. For all test runs I have looked at, picking mean largely reduces the run-over-run changes reported due to noise. See this run e.g: If we use the median, we just get just one change reported: http://llvm.org/perf/db_default/v4/nts/20661?num_comparison_runs=10&test_filter=&test_min_value_filter=&aggregation_fn=median&compare_to=20659&submit=Update If you use min, we get eight reports one claiming over 100% performance reduction for a case that is really just pure noise. I am planning to look into using better statistical methods...

...l test runs I have looked at, picking mean largely reduces the > run-over-run changes reported due to noise. > > > > See this run e.g: > > > > If we use the median, we just get just one change reported: > > > > > http://llvm.org/perf/db_default/v4/nts/20661?num_comparison_runs=10&test_filter=&test_min_value_filter=&aggregation_fn=median&compare_to=20659&submit=Update > > > > If you use min, we get eight reports one claiming over 100% performance > > reduction for a case that is really just pure noise. I am planning to > look into...

...sions/changes easily. The results that have been reported so far seem to provide sufficient information to catch performance changes. Specifically, when setting the aggregation function to median, most runs are shown to not impact performance: e.g: http://llvm.org/perf/db_default/v4/nts/19939?num_comparison_runs=10&test_filter=&test_min_value_filter=&aggregation_fn=median&compare_to=19934&submit=Update We still have a couple of runs that report performance differences, but where looking at the performance graph of the changed test cases makes it very clear that those are false posit...

Dear all, Following the Benchmarking BOF from 2013 US dev meeting, I’d like to propose some improvements to the LNT performance tracking software. The most significant issue with current implementation is that the report is filled with extremely noisy values. Hence it is hard to notice performance improvements or regressions. After investigation of LNT and the LLVM test suite, I propose