llvm dev - Jan 2014 - [LLVMdev] Why is the default LNT aggregation function min instead of mean

Is it the case that you converge on the min faster than the mean?  

Right now there is no way to set a per-tester aggregation function.

I had spent a little time trying to detect regressions using k-means clustering.
It looked promising.  That was outside LNT though.


On Jan 16, 2014, at 11:28 PM, Tobias Grosser <tobias at grosser.es> wrote:
> On 01/17/2014 03:09 AM, David Blaikie wrote:
>> On Thu, Jan 16, 2014 at 5:32 PM, Tobias Grosser <tobias at
grosser.es> wrote:
>> 
>>> On 01/17/2014 02:17 AM, David Blaikie wrote:
>>> 
>>>> Right - you usually won't see a normal distribution in the
noise of test
>>>> results. You'll see results clustered around the lower
bound with a long
>>>> tail of slower and slower results. Depending on how many
samples you do it
>>>> might be appropriate to take the mean of the best 3, for
example - but the
>>>> general approach of taking the fastest N does have some basis
in any case.
>>>> 
>>>> Not necessarily the right answer, the only right answer, etc.
>>>> 
>>> 
>>> Interesting. In fact I had the very same thoughts at the beginning.
>>> 
>>> However, when looking at my test results the common pattern looks
like
>>> this example:
>>> 
>>> http://llvm.org/perf/db_default/v4/nts/graph?show_all_
>>>
points=yes&moving_window_size=10&plot.0=34.95.3&submit=Update
>>> 
>>> The run-time of a test case is very consistently one of several
fixed
>>> values. The distribution of the different times is very consistent
and
>>> seems to form, in fact, something like a normal distribution (more
in the
>>> center, less at the border).
>>> 
>>> The explanation I have here is that the machine is by itself in
fact not
>>> very noisy. Instead, changes of the execution context (e.g. due to
>>> allocation of memory at a different location) influences the
performance.
>>> If we, by luck, have a run where all 'choices' have been
optimal we get
>>> minimal performance. However, in case of several independent
factors, it is
>>> more likely that we get a non-optimal configuration that yields a
value in
>>> the middle. Consequently, the minimal seems to be a non-optimal
choice here.
>>> 
>>> I understand that there may be some 'real' noise values,
but as the median
>>> does not seem to be affected very much by 'extremal'
values, I have the
>>> feeling it should be reasonable robust to such noise.
>>> 
>>> Have you seen examples where the median value gives a wrong
impression
>>> regarding performance?
>>> 
>> 
>> I have - and I've also seen the kind of results you're seeing
too. One of
>> the issues here is the quantization of results due to very short tests
and
>> not very granular timing. This is perhaps the only reason the results
even
>> /have/ a median (with finer grained timing and longer tests I expect
you'd
>> see fewer results with exactly the same time - yes, you might be in a
>> situation where the exact runtimes repeat due to very short tests being
>> wholely scheduled in one way or another - but in that case you'll
get wide,
>> solid swings depending on that scheduling behavior which is also
unhelpful)
>> in your results.
>> 
>> It's one of the reasons I gave up on trying to do timing on Linux -
I
>> couldn't get a machine quiet enough to look real. Though in the
long run I
>> still did tend to get results for many tests that were clustered around
a
>> minima with outliers going upwards...
> 
> Interesting. How many tests did you run in general?
> 
>> I'm perhaps rambling a bit here, and I'm by no means an
authority on this
>> subject (I tried and failed - gave up & worked on other things
instead) but
>> I think so long as the data is that noisy and quantized like that,
I'm not
>> sure how useful it'll be & not sure if it's the best data
to be trying to
>> figure out data processing on. Maybe I'm wrong, perhaps this is as
good as
>> that data can get and we do need an answer to how to handle it.
> 
> As a first step, my current goal is to ensure that we do not report
performance changes for test cases that have slightly noisy run-time behavior.
Similar to Chandler, I have doubts we can use such changes to measure
performance changes reliably.
> 
> I am also no authority in general here. In fact, I just look at my
performance testers and try to take conclusions from this very limited picture.
For the latest run using the 'median' again reduces the 'noise'
> nicely. I get 13 noise results with the 'min' and only a single one
with the median.
> 
> http://llvm.org/perf/db_default/v4/nts/20686
> 
> It seems at least for my tester, this really helps to get the noise down.
Is there a way I can switch the default displaying and reporting for my tester?
Or, as you have kind of seen mixed results in both configurations, might we even
be able to switch the default for now?
> 
> Cheers,
> Tobias
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140116/312899b8/attachment.html>

On 01/17/2014 08:58 AM, Chris Matthews wrote:
> Is it the case that you converge on the min faster than the mean?
Sorry, I do not fully understand what you mean here. What exactly would 
I need to do to check this? Should I just pick a couple of test/run 
pairs and see after how many samples the min/mean does not change any more?

What conclusion can I take from this?
> Right now there is no way to set a per-tester aggregation function.
>
> I had spent a little time trying to detect regressions using k-means
clustering.  It looked promising.  That was outside LNT though.
Interesting idea and that would most likely be helpful in configurations 
where we actually get run-times clustered in different groups. I was 
initially assuming this, but after Chandlers comments I have the feeling 
we actually only have a single cluster where the statements are just 
grouped due to the limited resolution of the analysis.

Cheers
Tobias

Note that it's very possible to get the those kind of effects from
other sources of computational load on the machine, see the fib35
graphs on

http://www.serpentine.com/blog/2009/09/29/criterion-a-new-benchmarking-library-for-haskell/


On Sat, Jan 18, 2014 at 12:57 AM, Tobias Grosser <tobias at grosser.es>
wrote:
> On 01/17/2014 08:58 AM, Chris Matthews wrote:
>>
>> Is it the case that you converge on the min faster than the mean?
>
>
> Sorry, I do not fully understand what you mean here. What exactly would I
> need to do to check this? Should I just pick a couple of test/run pairs and
> see after how many samples the min/mean does not change any more?
>
> What conclusion can I take from this?
>
>
>> Right now there is no way to set a per-tester aggregation function.
>>
>> I had spent a little time trying to detect regressions using k-means
>> clustering.  It looked promising.  That was outside LNT though.
>
>
> Interesting idea and that would most likely be helpful in configurations
> where we actually get run-times clustered in different groups. I was
> initially assuming this, but after Chandlers comments I have the feeling we
> actually only have a single cluster where the statements are just grouped
> due to the limited resolution of the analysis.
>
> Cheers
>
> Tobias
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


-- 
cheers, dave tweed__________________________
high-performance computing and machine vision expert: david.tweed at gmail.com
"while having code so boring anyone can maintain it, use Python." --
attempted insult seen on slashdot