Hi All ! Recently in the mailing list there was the question about benchmarking LLVM. I was told that LLVM get's benchmarked in the nightly test. While this is true, I wanted to have a tool to compare LLVM against other compilers, so I wrote a little python program (attached) that filled my need. It is completely outside of the LLVM makefile framework, but this stems from the fact that I can use it to compare any GCC-like compiler, e.g. gcc-4.2 against Intel ICC. It also allows one to compare the generated code sizes, which can be quite nice. Here is a sample output of a boring run on hello programs: $ ./ccbench -v Checking compilers ... Compiler /usr/bin/g++-3.3 not found g++-4.1: g++-4.1 (GCC) 4.1.3 20070812 (prerelease) (Debian 4.1.2-15) g++-4.2: g++-4.2 (GCC) 4.2.1 (Debian 4.2.1-3) llvm-g++: llvm-g++ (GCC) 4.0.1 (Apple Computer, Inc. build 5449)(LLVM build ) llvm-g++-bc: llvm-g++ (GCC) 4.0.1 (Apple Computer, Inc. build 5449)(LLVM build ) icc: icc (ICC) 10.0 20070809 gcc-3.3: gcc-3.3 (GCC) 3.3.6 (Debian 1:3.3.6-15) gcc-4.1: gcc-4.1 (GCC) 4.1.3 20070812 (prerelease) (Debian 4.1.2-15) gcc-4.2: gcc-4.2 (GCC) 4.2.1 (Debian 4.2.1-3) llvm-gcc: llvm-gcc (GCC) 4.0.1 (Apple Computer, Inc. build 5449)(LLVM build ) llvm-gcc-bc: llvm-gcc (GCC) 4.0.1 (Apple Computer, Inc. build 5449)(LLVM build ) Benchmark runtime comp code data bss compiler options hello.cpp 0.01 1.20 1904 320 148 g++-4.1 -O0 hello.cpp 0.01 1.24 2365 328 148 g++-4.2 -O0 hello.cpp 0.01 1.36 353236 11103 23736 llvm-g++ -O0 hello.cpp 0.01 1.23 1499 300 148 llvm-g++-bc -O0 hello.cpp 0.01 1.27 2186 320 148 icc -O0 hello.c 0.01 0.29 892 284 4 icc -O0 hello.c 0.01 0.20 852 260 4 gcc-3.3 -O0 hello.c 0.01 0.24 844 268 4 gcc-4.1 -O0 hello.c 0.01 0.24 848 268 4 gcc-4.2 -O0 hello.c 0.01 0.20 852 256 4 llvm-gcc -O0 hello.c 0.01 0.25 860 268 4 llvm-gcc-bc -O0 hello.cpp 0.01 1.17 1765 316 148 g++-4.1 -O2 hello.cpp 0.01 1.20 1837 316 148 g++-4.2 -O2 hello.cpp 0.01 1.44 353204 11103 23736 llvm-g++ -O2 hello.cpp 0.01 1.36 1380 296 148 llvm-g++-bc -O2 hello.cpp 0.01 1.25 10210 996 1252 icc -O2 hello.c 0.01 0.29 8949 960 1092 icc -O2 hello.c 0.01 0.21 852 260 4 gcc-3.3 -O2 hello.c 0.01 0.23 860 268 4 gcc-4.1 -O2 hello.c 0.01 0.23 864 268 4 gcc-4.2 -O2 hello.c 0.01 0.20 836 256 4 llvm-gcc -O2 hello.c 0.01 0.25 860 268 4 llvm-gcc-bc -O2 Total benchmark time 24.31s Description: Benchmark: basename of the benchmark that has been compiled runtime: minimum time of execution for the benchmark progra comp: minimum compile time code: code size (.text segment of resulting ELF executable) data: data size (.data segment of resulting ELF executable) bss: zero-initialized data size (.bss segment ...) compiler: short-name for the compiler. llvm-g??-bc means that the program was compiled with --emit-llvm and linked with llvm-ld options: optimization options used for this run You can specify the optimization options to be used on the command line, e.g. $ ./ccbench -- "-O2" "-O2 -march=pentium4" Other command line options are available, see --help. The list of benchmarks to be compiled and run has to be specified inside the source code, look at the end of the program. NOTE: the "llvm-g++" produces VERY large programs for any program that uses libstdc++, because it was compiled with --disable-shared. Here is an example where you can see the effect of the "-Os" compiler switch to various C++ programs: $ ./ccbench -- "-O0" "-O2" "-Os" | sort bigfib.cpp 0.16 5.28 18763 432 572 g++-4.2 -O2 bigfib.cpp 0.17 2.42 371805 12589 23736 llvm-g++ -Os bigfib.cpp 0.17 3.75 26810 1116 1380 icc -Os bigfib.cpp 0.17 3.85 28902 1116 1380 icc -O2 bigfib.cpp 0.17 4.48 18902 444 572 g++-4.1 -O2 bigfib.cpp 0.18 2.30 7403 376 156 llvm-g++-bc -Os bigfib.cpp 0.18 2.74 11838 380 420 llvm-g++-bc -O2 bigfib.cpp 0.18 2.82 374391 12589 23736 llvm-g++ -O2 bigfib.cpp 0.22 2.21 8090 388 156 llvm-g++-bc -O0 bigfib.cpp 0.29 2.99 14101 424 312 g++-4.1 -Os bigfib.cpp 0.29 3.00 12419 424 312 g++-4.2 -Os bigfib.cpp 0.59 1.96 20897 448 312 g++-4.2 -O0 bigfib.cpp 0.71 2.05 24029 440 312 g++-4.1 -O0 bigfib.cpp 0.72 2.05 24029 440 312 g++-4.1 -O0 bigfib.cpp 0.87 2.71 385897 12589 23736 llvm-g++ -O0 bigfib.cpp 0.91 2.03 42886 440 312 icc -O0 mandel-text.cpp 0.06 0.38 1381 296 4 g++-4.1 -Os mandel-text.cpp 0.06 0.39 1385 296 4 g++-4.2 -Os mandel-text.cpp 0.06 0.39 1483 296 8 g++-4.1 -O2 mandel-text.cpp 0.06 0.41 1491 296 8 g++-4.2 -O2 mandel-text.cpp 0.06 0.46 9505 972 1092 icc -Os mandel-text.cpp 0.06 0.51 9621 972 1092 icc -O2 mandel-text.cpp 0.11 0.26 1346 264 4 llvm-g++ -O0 mandel-text.cpp 0.11 0.36 1533 296 4 g++-4.1 -O0 mandel-text.cpp 0.12 0.35 1533 296 4 g++-4.1 -O0 mandel-text.cpp 0.12 0.36 1537 296 4 g++-4.2 -O0 mandel-text.cpp 0.14 0.47 1219 276 4 llvm-g++-bc -O0 mandel-text.cpp 0.14 0.48 1309 276 8 llvm-g++-bc -O2 mandel-text.cpp 0.14 0.49 1309 276 8 llvm-g++-bc -Os mandel-text.cpp 0.23 0.26 1260 264 8 llvm-g++ -O2 mandel-text.cpp 0.24 0.26 1260 264 8 llvm-g++ -Os mandel-text.cpp 0.27 0.47 1636 296 4 icc -O0 oopack_v1p8.cpp 0.62 1.07 14580 1288 118500 icc -O2 oopack_v1p8.cpp 0.66 0.88 5398 352 116256 g++-4.1 -O2 oopack_v1p8.cpp 0.66 0.94 5630 352 116256 g++-4.2 -O2 oopack_v1p8.cpp 0.76 0.97 13909 1292 118500 icc -Os oopack_v1p8.cpp 2.05 0.61 51011 1207 132940 llvm-g++ -Os oopack_v1p8.cpp 2.07 0.60 50874 1207 132940 llvm-g++ -O2 oopack_v1p8.cpp 2.28 0.85 5320 356 116256 g++-4.1 -Os oopack_v1p8.cpp 2.28 0.93 5352 356 116176 g++-4.2 -Os oopack_v1p8.cpp 2.54 0.82 4636 340 116196 llvm-g++-bc -O2 oopack_v1p8.cpp 2.63 0.84 4668 340 116196 llvm-g++-bc -Os oopack_v1p8.cpp 2.65 0.75 5337 312 116276 llvm-g++-bc -O0 oopack_v1p8.cpp 4.53 0.59 53582 1207 132972 llvm-g++ -O0 oopack_v1p8.cpp 4.92 0.59 7195 352 116256 g++-4.2 -O0 oopack_v1p8.cpp 4.97 0.58 7215 352 116256 g++-4.1 -O0 oopack_v1p8.cpp 5.45 0.70 8856 640 117440 icc -O0 stepanov_v1p2.cpp 1.01 1.09 64931 1600 33408 icc -O2 stepanov_v1p2.cpp 1.02 1.00 64471 1600 33408 icc -Os stepanov_v1p2.cpp 1.02 1.17 6519 340 32352 g++-4.2 -O2 stepanov_v1p2.cpp 1.07 1.18 6555 340 32352 g++-4.1 -O2 stepanov_v1p2.cpp 3.06 0.68 6082 308 32420 llvm-g++ -Os stepanov_v1p2.cpp 3.38 0.69 5839 308 32420 llvm-g++ -O2 stepanov_v1p2.cpp 3.88 0.80 4391 308 32384 llvm-g++-bc -O0 stepanov_v1p2.cpp 4.86 1.06 5726 340 32280 g++-4.2 -Os stepanov_v1p2.cpp 5.37 0.97 5770 340 32352 g++-4.1 -Os stepanov_v1p2.cpp 5.38 0.87 3972 304 32192 llvm-g++-bc -O2 stepanov_v1p2.cpp 5.88 0.88 4004 304 32192 llvm-g++-bc -Os stepanov_v1p2.cpp 25.60 0.73 8802 340 32352 g++-4.2 -O0 stepanov_v1p2.cpp 26.56 0.72 8918 340 32352 g++-4.1 -O0 stepanov_v1p2.cpp 30.56 0.69 10473 308 32420 llvm-g++ -O0 stepanov_v1p2.cpp 34.39 0.86 74167 1600 33408 icc -O0 Because you might want to sort such output according to your own criteria, you could call ./ccbench with --csv and load the resulting CSV (comma separated values) into your favourite spreadsheet or SQL database. -------------- next part -------------- A non-text attachment was scrubbed... Name: ccbench Type: application/x-python Size: 16612 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20070825/907b52e1/attachment.bin>