search for: aggregation_fn

...t runs I have looked at, picking mean largely reduces the run-over-run changes reported due to noise. See this run e.g: If we use the median, we just get just one change reported: http://llvm.org/perf/db_default/v4/nts/20661?num_comparison_runs=10&test_filter=&test_min_value_filter=&aggregation_fn=median&compare_to=20659&submit=Update If you use min, we get eight reports one claiming over 100% performance reduction for a case that is really just pure noise. I am planning to look into using better statistical methods. However, as a start, could we switch the default to 'mean...

...-over-run changes reported due to noise. > > > > See this run e.g: > > > > If we use the median, we just get just one change reported: > > > > > http://llvm.org/perf/db_default/v4/nts/20661?num_comparison_runs=10&test_filter=&test_min_value_filter=&aggregation_fn=median&compare_to=20659&submit=Update > > > > If you use min, we get eight reports one claiming over 100% performance > > reduction for a case that is really just pure noise. I am planning to > look into using better statistical methods. However, as a start, could we &g...

...to provide sufficient information to catch performance changes. Specifically, when setting the aggregation function to median, most runs are shown to not impact performance: e.g: http://llvm.org/perf/db_default/v4/nts/19939?num_comparison_runs=10&test_filter=&test_min_value_filter=&aggregation_fn=median&compare_to=19934&submit=Update We still have a couple of runs that report performance differences, but where looking at the performance graph of the changed test cases makes it very clear that those are false positives due to test case noise. Here comes the point of this mail. I...

Dear all, Following the Benchmarking BOF from 2013 US dev meeting, I’d like to propose some improvements to the LNT performance tracking software. The most significant issue with current implementation is that the report is filled with extremely noisy values. Hence it is hard to notice performance improvements or regressions. After investigation of LNT and the LLVM test suite, I propose

...run e.g: >>>> >>>> If we use the median, we just get just one change reported: >>>> >>>> >>>> http://llvm.org/perf/db_default/v4/nts/20661?num_ >>> comparison_runs=10&test_filter=&test_min_value_filter= >>> &aggregation_fn=median&compare_to=20659&submit=Update >>> >>>> >>>> If you use min, we get eight reports one claiming over 100% performance >>>> reduction for a case that is really just pure noise. I am planning to >>>> >>> look into using b...