[LLVMdev] [RFC] Benchmarking subset of the test suite

Tobias Grosser
2014-05-04T18:40:52Z

oOn 04/05/2014 14:39, Hal Finkel wrote:
> At the LLVM Developers' Meeting in November, I promised to work on
isolating a subset of the current test suite that is useful for benchmarking.
Having looked at this in more detail, most of the applications and benchmarks in
the test suite are useful for benchmarking, and so I think that a better way of
phrasing it is that we should construct a list of programs in the test suite
that are not useful for benchmarking.
>
> My proposed exclusion list is provided below. I constructed this exclusion
list primarily based on the following experiment: I ran the test suite 10 times
in three configurations: 1) On an IBM POWER7 (P7) with -O3 -mvsx, 2) On a P7 at
-O0 and 3) On an Intel Xeon E5430 with -O3 all using make -j6. I then used the
ministat utility (which performs a T test) to compare the timings of the two P7
configurations against each other and the Xeon configuration, requiring a
detectable difference at 99.5% confidence. I looked for tests that showed no
significant difference in all three comparisons. The running configuration here
is purposefully noisy, the idea is to eliminate those tests that are
significantly sensitive to startup time, file I/O time, memory bandwidth, etc.,
or just too short, and by running many tests in parallel
(non-deterministically), my hope is to eliminate those tests can cannot usefully
serve as benchmarks in a "normal" environment.
>
> I'll admit being somewhat surprised by so many of the Prolangs and
Shootout "benchmarks" seemingly not serving as useful benchmarks;
perhaps someone can look into improving the problem size, etc. of these.
>
> Without further ado, I propose that a test-suite configuration designed for
benchmarking exclude the following:

Hi Hal,

thanks for putting the effort! I think the systematic approach you have
taken is very sensible.

I went through your list and looked at a couple of interesting cases.
For the shootout benchmarks I looked at the results and the history my
LNT -O3 builder shows (long history, always 10 samples per run,
http://llvm.org/perf/db_default/v4/nts/25326)

Some observations from my side:

## Many benchmarks from your list have a runtime of zero seconds
reported in my tester

## For some of the benchmarks you propose, manually looking at the
   historic samples allows a human to spot certain trends:

 > MultiSource/Benchmarks/Prolangs-C/football/football

http://llvm.org/perf/db_default/v4/nts/graph?show_all_points=yes&moving_window_size=10&plot.237=34.237.3&submit=Update

 > MultiSource/Benchmarks/Prolangs-C/simulator/simulator

http://llvm.org/perf/db_default/v4/nts/graph?show_all_points=yes&moving_window_size=10&plot.314=34.314.3&submit=Update

## Some other benchmarks with zero seconds execution time are not
contained in your list. E.g.:

SingleSource/Benchmarks/Shootout/objinst
SingleSource/Benchmarks/Shootout-C++/objinst

## Some benchmarks on your list are really _no_ benchmarks:

Shoothout hello:

#include <stdio.h>

int main() {
     puts("hello world\n");
     return(0);
}

Shootout sumcol:

int main(int argc, char * * argv) {
     char line[MAXLINELEN];
     int sum = 0;
     char buff[4096];
     cin.rdbuf()->pubsetbuf(buff, 4096); // enable buffering

     while (cin.getline(line, MAXLINELEN)) {
         sum += atoi(line);
     }
     cout << sum << '\n';
}

To subsum, I believe this list might benefit from some improvements, but
it seems to be a really good start. If someone wants to do a more
extensive analysis, we can always analyze the historic data available in
my -O3 performance buildbot. It should give us a very good idea on how
noisy certain benchmarks are.

Cheers,
Tobias