llvm dev - Mar 2009 - [LLVMdev] [llvm-testresults] cfarm-x86-64 x86_64 nightly tester results

This nightly tester is now using an llvm-g++ that produces the new ODR linkage
types.  This means that many more functions are being considered by the
inter-procedural optimization passes (for example, "linkonce"
functions defined
in a header).  The result seems to be pretty huge swings (both good and bad) in
the C++ tests in the testsuite, see below.  Note that this tester is often under
heavy load due to other users, so take timing results with a pinch of salt (a
pinch is about +-10% in my experience with this tester).

Ciao,

Duncan.
> Significant changes in test results:
> GCCAS:
>  singlesource/Benchmarks/Adobe-C++/functionobjects: -102.68% (0.5960 =>
1.2080)
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: -44.04% (5.3323 =>
7.6804)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: -10.34%
(5.6883 => 6.2763)
>  singlesource/Benchmarks/Adobe-C++/simple_types_loop_invariant: -19.15%
(3.4882 => 4.1562)
>  singlesource/Benchmarks/Adobe-C++/stepanov_abstraction: -97.63% (1.0080
=> 1.9921)
>  singlesource/Benchmarks/Adobe-C++/stepanov_vector: -171.51% (0.8280 =>
2.2481)
>  singlesource/Benchmarks/Misc-C++/bigfib: -43.51% (0.5240 => 0.7520)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -110.33% (1.0080
=> 2.1201)
>  singlesource/Benchmarks/Shootout-C++/wordfreq: -43.20% (0.5000 =>
0.7160)
>  multisource/Applications/hbd/hbd: -31.91% (1.7801 => 2.3481)
>  multisource/Applications/kimwitu++/kc: -18.78% (32.6580 => 38.7903)
>  multisource/Applications/minisat/minisat: -72.05% (1.2160 => 2.0921)
>  multisource/Benchmarks/PAQ8p/paq8p: -144.14% (3.2082 => 7.8324)
>  multisource/Benchmarks/Trimaran/enc-3des/enc-3des: -9.06% (2.6041 =>
2.8401)
>  multisource/Benchmarks/tramp3d-v4/tramp3d-v4: -195.05% (22.2893 =>
65.7641)
> CBE:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: 10.56% (3.22 => 2.88)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 69.10%
(9.71 => 3.00)
>  singlesource/Benchmarks/CoyoteBench/fftbench: -382.67% (0.75 => 3.62)
>  singlesource/Benchmarks/Misc-C++/sphereflake: 18.66% (11.90 => 9.68)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -1811.11% (0.54 =>
10.32)
>  singlesource/Benchmarks/Misc-C++/stepanov_v1p2: 16.90% (25.33 => 21.05)
>  singlesource/Benchmarks/Misc/ffbench: -17.61% (3.18 => 3.74)
>  singlesource/Benchmarks/Shootout-C++/random: 5.51% (3.81 => 3.60)
>  singlesource/Benchmarks/Shootout/hash: -25.67% (9.70 => 12.19)
>  singlesource/Benchmarks/Shootout/methcall: 23.51% (7.74 => 5.92)
>  multisource/Applications/lambda-0.1.3/lambda: 18.80% (8.67 => 7.04)
>  multisource/Applications/minisat/minisat: 13.07% (24.25 => 21.08)
>  multisource/Benchmarks/ASCI_Purple/SMG2000/smg2000: -7.93% (10.84 =>
11.70)
> JIT:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: -7.34% (14.30 => 15.35)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 36.05%
(25.05 => 16.02)
>  singlesource/Benchmarks/CoyoteBench/fftbench: -381.65% (1.09 => 5.25)
>  singlesource/Benchmarks/CoyoteBench/lpbench: -25.22% (15.11 => 18.92)
>  singlesource/Benchmarks/Misc-C++/ray: 5.34% (7.87 => 7.45)
>  singlesource/Benchmarks/Misc-C++/sphereflake: 14.86% (6.46 => 5.50)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -1698.18% (0.55 =>
9.89)
>  singlesource/Benchmarks/Shootout-C++/objinst: 5.25% (7.81 => 7.40)
>  singlesource/Benchmarks/Shootout/methcall: -16.47% (8.56 => 9.97)
>  multisource/Applications/aha/aha: 59.87% (15.40 => 6.18)
>  multisource/Applications/hexxagon/hexxagon: 9.50% (15.27 => 13.82)
>  multisource/Benchmarks/PAQ8p/paq8p: 5.16% (173.51 => 164.55)
> LLC:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: 14.13% (3.75 => 3.22)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 69.51%
(6.92 => 2.11)
>  singlesource/Benchmarks/CoyoteBench/fftbench: -563.16% (0.57 => 3.78)
>  singlesource/Benchmarks/Misc-C++/ray: 6.45% (7.60 => 7.11)
>  singlesource/Benchmarks/Misc-C++/sphereflake: 12.32% (6.09 => 5.34)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -2438.89% (0.36 =>
9.14)
>  singlesource/Benchmarks/Misc/ffbench: -7.76% (3.61 => 3.89)
>  multisource/Applications/lambda-0.1.3/lambda: 18.24% (9.76 => 7.98)
> LLC-BETA:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: 14.89% (3.76 => 3.20)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 69.51%
(6.92 => 2.11)
>  singlesource/Benchmarks/CoyoteBench/fftbench: -555.26% (0.76 => 4.98)
>  singlesource/Benchmarks/CoyoteBench/lpbench: 20.54% (18.99 => 15.09)
>  singlesource/Benchmarks/Misc-C++/ray: 6.58% (7.60 => 7.10)
>  singlesource/Benchmarks/Misc-C++/sphereflake: 11.38% (6.15 => 5.45)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -2430.56% (0.36 =>
9.11)
>  singlesource/Benchmarks/Misc/ffbench: 6.78% (3.54 => 3.30)
>  singlesource/Benchmarks/Misc/flops-1: -7.75% (4.00 => 4.31)
>  singlesource/Benchmarks/Shootout/ary3: 23.85% (6.96 => 5.30)
> LLC compile:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: -19.52% (2.5201 =>
3.0121)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -317.65% (0.0680
=> 0.2840)
>  multisource/Benchmarks/PAQ8p/paq8p: -19.62% (1.8761 => 2.2441)
> LLC-BETA compile:
>  singlesource/Benchmarks/Adobe-C++/loop_unroll: -18.74% (2.4761 =>
2.9401)
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 13.42%
(1.7881 => 1.5481)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -406.67% (0.0600
=> 0.3040)
>  multisource/Benchmarks/PAQ8p/paq8p: -11.42% (1.9961 => 2.2241)
> Bytecode:
>  singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding: 13.35%
(205736 => 178276)
>  singlesource/Benchmarks/Adobe-C++/stepanov_abstraction: -24.39% (29780
=> 37044)
>  singlesource/Benchmarks/Adobe-C++/stepanov_vector: -50.45% (29196 =>
43924)
>  singlesource/Benchmarks/CoyoteBench/fftbench: -7.53% (13596 => 14620)
>  singlesource/Benchmarks/Misc-C++/bigfib: -26.93% (19936 => 25304)
>  singlesource/Benchmarks/Misc-C++/stepanov_container: -333.72% (10664 =>
46252)
>  singlesource/Benchmarks/Misc-C++/stepanov_v1p2: -13.34% (9056 => 10264)
>  singlesource/Benchmarks/Shootout-C++/ary: 5.35% (4560 => 4316)
>  singlesource/Benchmarks/Shootout-C++/ary3: 5.10% (4780 => 4536)
>  singlesource/Benchmarks/Shootout-C++/hash: -175.14% (3636 => 10004)
>  singlesource/Benchmarks/Shootout-C++/hash2: -217.73% (3812 => 12112)
>  singlesource/Benchmarks/Shootout-C++/lists: -47.33% (4420 => 6512)
>  singlesource/Benchmarks/Shootout-C++/lists1: -40.03% (5396 => 7556)
>  singlesource/Benchmarks/Shootout-C++/moments: -14.37% (7880 => 9012)
>  singlesource/Benchmarks/Shootout-C++/reversefile: 7.33% (10640 => 9860)
>  singlesource/Benchmarks/Shootout-C++/spellcheck: 5.53% (16196 => 15300)
>  singlesource/Benchmarks/Shootout-C++/wordfreq: -76.77% (10212 => 18052)
>  singlesource/UnitTests/2006-12-04-DynAllocAndRestore: 10.61% (528 =>
472)
>  multisource/Applications/minisat/minisat: 5.46% (48116 => 45488)
>  multisource/Benchmarks/PAQ8p/paq8p: -10.47% (196084 => 216608)
>  multisource/Benchmarks/Prolangs-C++/garage/garage: 5.45% (6748 => 6380)
>  multisource/Benchmarks/tramp3d-v4/tramp3d-v4: -234.99% (687532 =>
2303148)

On Mar 9, 2009, at 8:53 AM, Duncan Sands wrote:
> This nightly tester is now using an llvm-g++ that produces the new  
> ODR linkage
> types.  This means that many more functions are being considered by  
> the
> inter-procedural optimization passes (for example, "linkonce"  
> functions defined
> in a header).  The result seems to be pretty huge swings (both good  
> and bad) in
> the C++ tests in the testsuite, see below.  Note that this tester is  
> often under
> heavy load due to other users, so take timing results with a pinch  
> of salt (a
> pinch is about +-10% in my experience with this tester).
Awesome Duncan, thank you for working on this!  From the LLC tests, I  
see:
> LLC:
> singlesource/Benchmarks/Adobe-C++/loop_unroll: 14.13% (3.75 => 3.22)
> singlesource/Benchmarks/Adobe-C++/simple_types_constant_folding:  
> 69.51% (6.92 => 2.11)
> singlesource/Benchmarks/CoyoteBench/fftbench: -563.16% (0.57 => 3.78)
> singlesource/Benchmarks/Misc-C++/ray: 6.45% (7.60 => 7.11)
> singlesource/Benchmarks/Misc-C++/sphereflake: 12.32% (6.09 => 5.34)
> singlesource/Benchmarks/Misc-C++/stepanov_container: -2438.89% (0.36  
> => 9.14)
> singlesource/Benchmarks/Misc/ffbench: -7.76% (3.61 => 3.89)
> multisource/Applications/lambda-0.1.3/lambda: 18.24% (9.76 => 7.98)
Can you check to see if the stepanov_container/fftbench regressions  
are real?  If so, it would be very interesting to know what is "going  
wrong" on them.

-Chris

Hi Chris,
> Can you check to see if the stepanov_container/fftbench regressions  
> are real?  If so, it would be very interesting to know what is "going
> wrong" on them.
I think these may not be real.  This version of llvm-gcc was built with
checking enabled - does this turn on checking in libstdc++?  It seems that
a bunch of linkonce libstdc++ checking code is now being inlined (presumably
because it compiled to something smaller than before).  Previously it was
not being inlined so was discarded, and the version from the system libstdc++
was used instead.  My guess is that the system libstdc++ version is faster
because it doesn't do any checking.  To test this theory I've told this
nightly tester to build llvm-gcc with checking disabled.  If I'm correct
then stepanov_container/fftbench will speed up hugely in a few days time
when the tester starts using the new llvm-gcc.

Ciao,

Duncan.