Dimitry Andric
2012-Sep-04 20:49 UTC
[LLVMdev] Clang/llvm performance tests on FreeBSD 10.0-CURRENT
Hi all,
I recently performed a series of compiler performance tests on FreeBSD
10.0-CURRENT, particularly comparing gcc 4.2.1 and gcc 4.7.1 against
clang 3.1 and clang 3.2.
The attached text file[1] contains more information about the tests,
some semi-cooked performance data, and my conclusions. Any errors and
omissions are also my fault, so if you notice them, please let me know.
The executive summary: clang compiles mostly faster than gcc (sometimes
much faster), and uses significantly less memory.
Finally, please note these tests were purely about compilation speed,
not about the performance of the resulting executables. This still
needs to be tested.
-Dimitry
[1]: Also available at:
<http://www.andric.com/freebsd/perftest/perftest-2012-09-01a.txt>
-------------- next part --------------
COMPILER PERFORMANCE TESTS ON FREEBSD 10.0-CURRENT, SEPTEMBER 2012
=================================================================
INTRODUCTION
------------
The compilers tested were:
- gcc 4.2.1, the system compiler in FreeBSD, which is compiled by gcc 4.2.1.
- gcc 4.7.1, from the official gcc.gnu.org release, compiled via a three-stage
bootstrap, so the final compiler has been compiled by gcc 4.7.1.
- clang 3.1 (branches/release_31 156863), which is the default version of clang
in FreeBSD 10-CURRENT before r239462. The used executable was compiled by a
previous copy of itself.
- clang 3.2 (trunk 162107), which is the default version of clang in FreeBSD
10.0-CURRENT, after r239462. The used executable was compiled by a previous
copy of itself.
All tests were run on ref10-amd64.freebsd.org, which is a Dell 2950, 1.86GHz
Core2 Xeon, 2x4 Core, 16G RAM. It runs FreeBSD/amd64 10.0-CURRENT #0 r231914:
Sun Feb 19 17:24:37 UTC 2012.
Each build was repeated 6 times, after cleaning out the object directories, and
syncing. Each build was timed using the system time(1) command, using the -l
argument to obtain rusage information.
The programs tested by compilation were:
- A large C++ program: clang 3.2, as it occurs in the FreeBSD 10.0-CURRENT
source tree as of r239532.
- A medium-large C program: gcc 4.2.1, as it occurs in the FreeBSD 10.0-CURRENT
source tree as of r239532.
- A large C++ library: boost 1.50.0, the officially released version from
<http://www.boost.org/>.
Building a large C++ program (clang 3.2) single-threaded
=======================================================
Using clang 3.1:
----------------
N Min Max Median Avg
Stddev
real 6 2283.69 2288.46 2285.74 2285.505
1.6470064
user 6 2145.2 2147.2 2146.18 2146.0567
0.68266146
sys 6 128.3 132.08 130.65 130.54833
1.256653
maxrss 6 179264 179264 179264 179264
0
ixrss 6 21407 21436 21420 21419.833
9.6211572
idrss 6 3628 3632 3630 3629.8333
1.3291601
isrss 6 252 252 252 252
0
minflt 6 12485556 12485556 12485556 12485556
0
majflt 6 0 0 0 0
0
nswap 6 0 0 0 0
0
inblock 6 0 0 0 0
0
oublock 6 2058 2106 2103 2081.3333
25.216397
msgsnd 6 18 18 18 18
0
msgrcv 6 0 0 0 0
0
nsignals 6 1878 1878 1878 1878
0
nvcsw 6 16288 16357 16333 16320.667
29.615311
nivcsw 6 2071535 3998751 3057756 2966314
635381.66
Using clang 3.2:
----------------
N Min Max Median Avg
Stddev
real 6 2358.61 2362.84 2362.67 2361.22
1.7831321
user 6 2215.33 2221.13 2218.72 2218.57
2.0094278
sys 6 130.78 134.63 133.41 132.99833
1.4702301
maxrss 6 177796 177796 177796 177796
0
ixrss 6 21388 21413 21408 21400.833
11.052903
idrss 6 3702 3707 3706 3704.6667
2.2509257
isrss 6 253 253 253 253
0
minflt 6 12583827 12583827 12583827 12583827
0
majflt 6 0 0 0 0
0
nswap 6 0 0 0 0
0
inblock 6 0 0 0 0
0
oublock 6 2036 2074 2071 2054.8333
19.589963
msgsnd 6 18 26 26 23.333333
4.1311822
msgrcv 6 0 0 0 0
0
nsignals 6 1878 1878 1878 1878
0
nvcsw 6 16266 16391 16354 16327.667
53.909801
nivcsw 6 2118900 3891231 3534528 3168715.7
673236.29
Using gcc 4.2.1:
----------------
N Min Max Median Avg
Stddev
real 6 4238.49 4241.76 4240.78 4240.1867
1.2375244
user 6 3903.48 3908.6 3907.58 3906.5583
1.8932661
sys 6 358.38 361.43 359.94 359.94667
1.1494984
maxrss 6 568592 568592 568592 568592
0
ixrss 6 6348 6353 6350 6350.3333
1.6329932
idrss 6 3495 3498 3497 3496.5
1.0488088
isrss 6 146 146 146 146
0
minflt 6 47304156 47304184 47304175 47304172
10.545141
majflt 6 0 0 0 0
0
nswap 6 0 0 0 0
0
inblock 6 0 0 0 0
0
oublock 6 2620 2754 2732 2683.6667
68.730391
msgsnd 6 0 0 0 0
0
msgrcv 6 0 0 0 0
0
nsignals 6 1878 1878 1878 1878
0
nvcsw 6 67561 67763 67674 67648
75.198404
nivcsw 6 3994087 5821442 4846679 4810028.2
597366.52
Using gcc 4.7.1:
----------------
N Min Max Median Avg
Stddev
real 6 3818.41 3974.54 3820.49 3846.7417
62.715466
user 6 3506.86 3591.97 3509.96 3522.8283
33.896088
sys 6 333.58 364.34 338.93 340.70833
11.839839
maxrss 6 480724 480736 480724 480727.33
5.316639
ixrss 6 12173 12198 12194 12188.333
9.4375138
idrss 6 1520 1523 1522 1521.8333
1.1690452
isrss 6 134 134 134 134
0
minflt 6 38406568 38406673 38406592 38406599
38.768544
majflt 6 0 90 0 20.333333
36.45088
nswap 6 0 0 0 0
0
inblock 6 0 4775 0 1233.3333
2028.0327
oublock 6 2266 2301 2286 2284.3333
13.662601
msgsnd 6 30 31 30 30.166667
0.40824829
msgrcv 6 0 0 0 0
0
nsignals 6 1878 1878 1878 1878
0
nvcsw 6 59792 67936 60369 61859.833
3186.3204
nivcsw 6 2867702 4546665 4361653 3753550.8
769382.51
Summary:
--------
For building this specific large C++ program, gcc 4.2.1 is ~86% slower than
clang 3.1 in real time, ~82% slower in user time, and ~176% slower in system
time. The maximum resident set size during building is ~217% larger, and it
causes ~279% more page reclaims.
Though gcc 4.7.1 is faster than its older version, it is still ~68% slower than
clang 3.1 in real time, ~64% slower in user time, and ~161% slower in system
time. The maximum resident set size during building is ~220% larger, and it
causes ~208% more page reclaims.
Finally, clang 3.2 is ~3% slower than clang 3.1 in both real time and user time,
and ~2% slower in system time. The maximum resident set size and the number of
page reclaims during building are approximately equal.
Conclusion:
-----------
Clang 3.1 is clearly the fastest compiler for building this specific large C++
program, with clang 3.2 trailing closely behind. Both are significantly faster,
and use much less memory than either version of gcc.
Building a medium-large C program (gcc 4.2.1) single-threaded
============================================================
Using clang 3.1:
----------------
N Min Max Median Avg
Stddev
real 6 303.31 304.06 303.65 303.67167
0.24991332
user 6 275.42 277.09 275.99 276.11167
0.57766484
sys 6 24.92 26.15 25.6 25.656667
0.44643775
maxrss 6 177876 177876 177876 177876
0
ixrss 6 20529 20559 20544 20542.833
12.38413
idrss 6 3618 3623 3621 3620.3333
1.9663842
isrss 6 247 247 247 247
0
minflt 6 2214250 2214250 2214250 2214250
0
majflt 6 0 0 0 0
0
nswap 6 0 0 0 0
0
inblock 6 0 0 0 0
0
oublock 6 677 677 677 677
0
msgsnd 6 18 18 18 18
0
msgrcv 6 0 0 0 0
0
nsignals 6 883 883 883 883
0
nvcsw 6 5705 5837 5819 5793.6667
49.33018
nivcsw 6 205418 467152 449398 371699.67
114414.58
Using clang 3.2:
----------------
N Min Max Median Avg
Stddev
real 6 330.22 331.23 330.95 330.69833
0.43687145
user 6 301.29 302.59 302.3 302.05667
0.49649438
sys 6 26.12 27.19 27.06 26.875
0.39747956
maxrss 6 186260 186260 186260 186260
0
ixrss 6 20639 20674 20660 20656.833
14.469508
idrss 6 3699 3705 3703 3702.3333
2.1602469
isrss 6 316 319 318 317.33333
1.2110601
minflt 6 2290933 2290934 2290934 2290933.7
0.51614557
majflt 6 0 0 0 0
0
nswap 6 0 0 0 0
0
inblock 6 0 0 0 0
0
oublock 6 668 669 668 668.16667
0.40824829
msgsnd 6 18 18 18 18
0
msgrcv 6 0 0 0 0
0
nsignals 6 883 883 883 883
0
nvcsw 6 5783 5822 5801 5799
17.944358
nivcsw 6 111115 520961 396082 316725.33
164041.32
Using gcc 4.2.1:
----------------
N Min Max Median Avg
Stddev
real 6 422.68 425.44 423.23 423.47333
1.0273396
user 6 389.1 391.67 390.58 390.41333
0.82734918
sys 6 36.85 39.2 38.65 38.23
0.83840324
maxrss 6 392560 392560 392560 392560
0
ixrss 6 5529 5542 5534 5534.5
6.0580525
idrss 6 3915 3924 3919 3919
4.1472883
isrss 6 142 142 142 142
0
minflt 6 4055461 4055464 4055463 4055462.7
1.21063
majflt 6 0 4 0 0.66666667
1.6329932
nswap 6 0 0 0 0
0
inblock 6 0 730 0 121.66667
298.02125
oublock 6 659 693 662 667
12.884099
msgsnd 6 0 0 0 0
0
msgrcv 6 0 0 0 0
0
nsignals 6 883 883 883 883
0
nvcsw 6 15645 16454 15874 15888
298.36019
nivcsw 6 121293 661776 414611 363556.83
207101.28
Using gcc 4.7.1:
----------------
N Min Max Median Avg
Stddev
real 6 461.58 462.55 462.01 461.98333
0.40287302
user 6 425.22 426.36 425.92 425.835
0.43825791
sys 6 40.83 42.94 41.99 41.925
0.71034499
maxrss 6 445624 445624 445624 445624
0
ixrss 6 10781 10816 10801 10797.5
12.405644
idrss 6 2427 2433 2430 2430
2.1908902
isrss 6 178 178 178 178
0
minflt 6 3883735 3883740 3883739 3883738
2.3664319
majflt 6 0 0 0 0
0
nswap 6 0 0 0 0
0
inblock 6 0 14 0 2.3333333
5.7154761
oublock 6 677 681 679 679
1.4142136
msgsnd 6 20 20 20 20
0
msgrcv 6 0 0 0 0
0
nsignals 6 883 883 883 883
0
nvcsw 6 16411 16660 16532 16542.333
98.544744
nivcsw 6 284414 901533 384379 449845.33
241447.41
Summary:
--------
For building this specific medium C program, gcc 4.2.1 is ~40% slower than clang
3.1 in real time, ~41% slower in user time, and ~49% slower in system time. The
maximum resident set size during building is ~121% larger, and it causes ~83%
more page reclaims.
For C, gcc 4.7.1 is even slower than its older version; it is ~52% slower than
clang 3.1 in real time, ~54% slower in user time, and ~63% slower in system
time. The maximum resident set size during building is ~151% larger, and it
causes ~75% more page reclaims.
Finally, clang 3.2 is ~9% slower than clang 3.1 in both real time and user
time, and ~5% slower in system time. The maximum resident set size during
building is ~5% larger, and it causes ~4% more page reclaims.
Conclusion:
-----------
Clang 3.1 is clearly the fastest compiler for building this specific medium-
large C program, with clang 3.2 somewhat behind. Both are significantly faster,
and use much less memory than either version of gcc.
Building a large C++ library (boost 1.50.0) single-threaded
==========================================================
Using clang 3.1:
----------------
N Min Max Median Avg
Stddev
real 6 1056.69 1060.49 1059.09 1058.6783
1.5028695
user 6 975.49 978.88 978.53 977.55
1.4653464
sys 6 73.75 76.42 74.87 74.95
1.0609618
maxrss 6 212324 216712 213668 214260.67
1774.6309
ixrss 6 22472 22549 22525 22514.5
31.232995
idrss 6 3793 3806 3802 3800.1667
5.492419
isrss 6 276 277 277 276.5
0.54772256
minflt 6 9543701 9543702 9543701 9543701.3
0.51234754
majflt 6 0 0 0 0
0
nswap 6 0 0 0 0
0
inblock 6 0 0 0 0
0
oublock 6 1453 1461 1457 1455.8333
3.3714487
msgsnd 6 115 115 115 115
0
msgrcv 6 0 0 0 0
0
nsignals 6 0 0 0 0
0
nvcsw 6 7352 7834 7576 7567.1667
167.70023
nivcsw 6 27478 2350999 1699745 1337037.8
1040439
Using clang 3.2:
----------------
N Min Max Median Avg
Stddev
real 6 1075.33 1077.94 1076.39 1076.4267
0.93958856
user 6 995.14 997.61 995.43 995.88833
0.9489661
sys 6 72.34 74.67 74.23 73.843333
0.81563881
maxrss 6 208552 211148 210436 209936
921.08458
ixrss 6 22437 22484 22458 22459
19.768662
idrss 6 3869 3878 3873 3873.3333
3.8815804
isrss 6 275 275 275 275
0
minflt 6 9351477 9351478 9351478 9351477.5
0.54772256
majflt 6 0 0 0 0
0
nswap 6 0 0 0 0
0
inblock 6 0 0 0 0
0
oublock 6 1448 1454 1449 1449.6667
2.4221203
msgsnd 6 115 115 115 115
0
msgrcv 6 0 0 0 0
0
nsignals 6 0 0 0 0
0
nvcsw 6 10481 12934 11049 11105.333
936.9249
nivcsw 6 975292 2383586 1633650 1615797.3
605542.76
Using gcc 4.2.1:
----------------
N Min Max Median Avg
Stddev
real 6 1037.86 1047.78 1039.71 1040.21
3.8054592
user 6 938.74 944.49 941.52 941.55667
1.8382999
sys 6 86.37 92.84 89.89 89.57
2.1105639
maxrss 6 560256 560316 560272 560274
21.428952
ixrss 6 6435 6453 6441 6443
7.2663608
idrss 6 3563 3573 3566 3567.5
4.0373258
isrss 6 136 136 136 136
0
minflt 6 12360490 12360492 12360491 12360491
0.63245553
majflt 6 0 0 0 0
0
nswap 6 0 0 0 0
0
inblock 6 0 4283 0 713.83333
1748.5274
oublock 6 2648 2656 2655 2653.3333
2.9439203
msgsnd 6 44 51 51 49.833333
2.857738
msgrcv 6 0 0 0 0
0
nsignals 6 0 0 0 0
0
nvcsw 6 7897 12281 8004 8696
1757.1741
nivcsw 6 19915 1989580 897003 957452.5
764999.83
Using gcc 4.7.1:
----------------
N Min Max Median Avg
Stddev
real 6 1038.13 1041.29 1040.98 1039.92
1.3837774
user 6 937.73 943.59 941.35 941.14167
2.0323919
sys 6 89.19 95.1 91.61 91.588333
2.0745739
maxrss 6 361268 361268 361268 361268
0
ixrss 6 12431 12474 12469 12457.333
17.385818
idrss 6 1547 1552 1551 1549.8333
1.9407902
isrss 6 129 129 129 129
0
minflt 6 10455489 10455489 10455489 10455489
0
majflt 6 0 0 0 0
0
nswap 6 0 0 0 0
0
inblock 6 0 162 0 27
66.136223
oublock 6 2537 2540 2539 2538.5
1.0488088
msgsnd 6 113 113 113 113
0
msgrcv 6 0 0 0 0
0
nsignals 6 0 0 0 0
0
nvcsw 6 7778 7975 7880 7874.1667
78.036957
nivcsw 6 27055 2302383 2187401 1478808.3
946760.86
Summary:
--------
For building this specific large C++ library, clang 3.1 is ~2% slower than gcc
4.2.1 in real time, ~4% slower in user time, but ~20% faster in system time.
The maximum resident set size during building is ~162% smaller, and it causes
~30% less page reclaims.
As before, clang 3.2 is slower than its older version; it is ~3% slower than
gcc 4.2.1 in real time, ~6% slower in user time, but ~21% faster in system
time. The maximum resident set size is ~167% smaller, and it causes ~32% less
page reclaims.
Finally, gcc 4.7.1 is equally fast as gcc 4.2.1 in real time and user time, and
~2% slower in system time. The maximum resident set size is ~36% smaller, and
it causes ~15% less page reclaims.
Conclusion:
-----------
Both gcc 4.2.1 and 4.7.1 are the fastest compilers for building this specific
large C++ library, but both versions of clang are not far behind. Both versions
of gcc use quite a bit more memory than either version of clang.
===============================================================================Copyright
(c) 2012 Dimitry Andric <dimitry at andric.com>
Verbatim copying and redistribution of this entire text are permitted, provided
this notice is preserved.
================================================================================
Sean Silva
2012-Sep-05 00:54 UTC
[LLVMdev] [cfe-dev] Clang/llvm performance tests on FreeBSD 10.0-CURRENT
It would be cool to have an up to date version of <http://clang.llvm.org/performance.html>. The page is currently 3+ years old. --Sean Silva On Tue, Sep 4, 2012 at 4:49 PM, Dimitry Andric <dimitry at andric.com> wrote:> Hi all, > > I recently performed a series of compiler performance tests on FreeBSD > 10.0-CURRENT, particularly comparing gcc 4.2.1 and gcc 4.7.1 against > clang 3.1 and clang 3.2. > > The attached text file[1] contains more information about the tests, > some semi-cooked performance data, and my conclusions. Any errors and > omissions are also my fault, so if you notice them, please let me know. > > The executive summary: clang compiles mostly faster than gcc (sometimes > much faster), and uses significantly less memory. > > Finally, please note these tests were purely about compilation speed, > not about the performance of the resulting executables. This still > needs to be tested. > > -Dimitry > > [1]: Also available at: > <http://www.andric.com/freebsd/perftest/perftest-2012-09-01a.txt> > > > _______________________________________________ > cfe-dev mailing list > cfe-dev at cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev >
Duncan Sands
2012-Sep-05 08:06 UTC
[LLVMdev] Clang/llvm performance tests on FreeBSD 10.0-CURRENT
Hi Dimitry,> I recently performed a series of compiler performance tests on FreeBSD > 10.0-CURRENT, particularly comparing gcc 4.2.1 and gcc 4.7.1 against > clang 3.1 and clang 3.2. > > The attached text file[1] contains more information about the tests, > some semi-cooked performance data, and my conclusions. Any errors and > omissions are also my fault, so if you notice them, please let me know. > > The executive summary: clang compiles mostly faster than gcc (sometimes > much faster), and uses significantly less memory. > > Finally, please note these tests were purely about compilation speed, > not about the performance of the resulting executables. This still > needs to be tested.presumably you were compiling with optimization, but as you didn't mention what the compiler flags were it is hard to know. Ciao, Duncan.
Dimitry Andric
2012-Sep-05 09:35 UTC
[LLVMdev] Clang/llvm performance tests on FreeBSD 10.0-CURRENT
On 2012-09-05 10:06, Duncan Sands wrote: ...>> I recently performed a series of compiler performance tests on FreeBSD >> 10.0-CURRENT, particularly comparing gcc 4.2.1 and gcc 4.7.1 against >> clang 3.1 and clang 3.2....> presumably you were compiling with optimization, but as you didn't mention > what the compiler flags were it is hard to know.Yes, I have been reminded of this by posters on the FreeBSD mailing lists already. :) In any case, for all three builds, the default optimization options were used. Basically: 1) For building the FreeBSD in-tree version of clang 3.2: -O2 -pipe -fno-strict-aliasing These are just the default FreeBSD optimization flags for building clang, which are probably used by the majority of users out there. This is the case that I was interested in particularly. The -fno-strict-aliasing is not really my choice, but it was introduced in the past by Nathan Whitehorn, who apparently saw problems without it. It will hopefully disappear in the future. 2) For building the FreeBSD in-tree version of gcc 4.2.1: -O2 -pipe These are the default FreeBSD optimization flags. 3) For building Boost 1.50.0: -ftemplate-depth-128 -O3 -finline-functions These are the Boost defaults for gcc-compatible compilers, from tools/build/v2/tools/gcc.jam. I will add this information to the text file.
Matthieu Monrocq
2012-Sep-05 17:00 UTC
[LLVMdev] [cfe-dev] Clang/llvm performance tests on FreeBSD 10.0-CURRENT
On Tue, Sep 4, 2012 at 10:49 PM, Dimitry Andric <dimitry at andric.com> wrote:> Hi all, > > I recently performed a series of compiler performance tests on FreeBSD > 10.0-CURRENT, particularly comparing gcc 4.2.1 and gcc 4.7.1 against > clang 3.1 and clang 3.2. > > The attached text file[1] contains more information about the tests, > some semi-cooked performance data, and my conclusions. Any errors and > omissions are also my fault, so if you notice them, please let me know. > > The executive summary: clang compiles mostly faster than gcc (sometimes > much faster), and uses significantly less memory. > > Finally, please note these tests were purely about compilation speed, > not about the performance of the resulting executables. This still > needs to be tested. > > -Dimitry > > [1]: Also available at: > <http://www.andric.com/**freebsd/perftest/perftest-**2012-09-01a.txt<http://www.andric.com/freebsd/perftest/perftest-2012-09-01a.txt> > > > > >Thanks for the detailed report! I am somewhat concerned that Clang 3.2 seems to always be 3% slower than the 3.1 version, though I guess that there performance will be tweaked prior to release as usual. -- Matthieu -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20120905/97ae9ec1/attachment.html>
Maybe Matching Threads
- [LLVMdev] Clang/llvm performance tests on FreeBSD 10.0-CURRENT
- [LLVMdev] Building Boost library failed with Clang 2.9
- XMMS plugin build fix
- [LLVMdev] [RFC] New command line parsing/generating framework for clang and lld.
- Compiling static libFLAC.a still requires libogg.dylib