[LLVMdev] [LNT] Question about results reliability in LNT infrustructure

Bob Wilson
Thu, 27 Jun 2013 16:41:36 -0000

On Jun 27, 2013, at 9:27 AM, Renato Golin <renato.golin at linaro.org>
wrote:

> On 27 June 2013 17:05, Tobias Grosser <tobias at grosser.es> wrote:
> We are looking for a good way/value to show the reliability of individual
results in the UI. Do you have some experience, what a good measure of the
reliability of test results is?
>
> Hi Tobi,
>
> I had a look at this a while ago, but never got around to actually work on
it. My idea was to never use point-changes as indication of
progress/regressions, unless there was a significant change (2/3 sigma). What we
should do is to compare the current moving-average with the past moving averages
(of K runs) with both last-avg and the (N-K)th moving-average (to make sure
previous values included in the current moving average are not toning it
down/up), and keep the biggest difference as the final result.
>
> We should also compare the current mov-avg with M non-overlapping mov-avgs
before, and calculate if we're monotonically increasing, decreasing or if
there is a difference of 2/3 sigma between the current mov-avg (N) and the
(N-M)th mov-avg. That would give us an idea on the trends of each test.

Chris Matthews has recently been working on implementing something similar to
that.  Chris, can you share some details?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130627/44c243dc/attachment.html>