On 2009-09-16 11:42, Olivier Meurant wrote:> On Tue, Sep 15, 2009 at 5:35 PM, David Greene <dag at cray.com > <mailto:dag at cray.com>> wrote: > > > Remember, the goal here isn't to show how great LLVM is. It's to > get an > honest assessment of where we are at. Phoronix did us a big > favor. Getting > more details about his tests would help us even more. > > > It's exactly what I'm trying to do. I'm not even a LLVM contributor, > so I don't feel responsible for a good or bad performances of LLVM. :) > Anyway, thanks for pointing this target differences. After some hours > at recompiling gcc, here we are : > > Some information on the test platform : > > cat /etc/lsb-release > DISTRIB_ID=Ubuntu > DISTRIB_RELEASE=9.04 > DISTRIB_CODENAME=jaunty > DISTRIB_DESCRIPTION="Ubuntu 9.04" > > uname -a > Linux zaraki 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC > 2009 i686 GNU/Linux > > /proc/cpuinfo > dual core Intel(R) Core(TM)2 CPU X6800 @ 2.93GHz (cache size 4096) > > llvm-gcc -v > Using built-in specs. > Target: i686-pc-linux-gnu > Configured with: ../llvm-gcc4.2-2.5.source/configure > --prefix=/mounts/zion/disks/0/localhome/tbrethou/2.5/prerelease2/llvm-gcc/obj/../install > --program-prefix=llvm- > --enable-llvm=/localhome/tbrethou/2.5/prerelease2/llvm-2.5/ > --disable-bootstrap --enable-languages=c,c++,fortran > Thread model: posix > gcc version 4.2.1 (Based on Apple Inc. build 5636) (LLVM build 2.5) > (Binary from official x86 2.5 release) > > gcc -v > Using built-in specs. > Target: i686-pc-linux-gnu > Configured with: ./configure --prefix=/usr/local > Thread model: posix > gcc version 4.4.1 (GCC)Phoronix tested with gcc 4.2.1 (Apple's).> > The compilation line are : > gcc -o bin src.c -O3 > llvm-gcc -o bin src.c -O3The Makefile uses -O by default, perhaps that is what Phoronix used too.> > > Results : > LLVM average : 13553161.95 > gcc average : 14624441.45 > Average : gcc is better with 8% > > The full report : > "LLVM" "GCC" "Difference" "Difference %" > > "Run 1" 13842616 14693115.8 -850499.8 -6.14 > "Run 2" 13773202.9 14785734.9 -1012532 -7.35 > "Run 3" 13671309.3 14985114 -1313804.7 -9.61 > "Run 4" 12916939 14640392.9 -1723453.9 -13.34 > "Run 5" 13572133.3 14425575 -853441.7 -6.29 > "Run 6" 13810096.5 14694240.6 -884144.1 -6.4 > "Run 7" 13427435.8 14695094.6 -1267658.8 -9.44 > "Run 8" 14056659.9 14468968.4 -412308.5 -2.93 > "Run 9" 13699125.3 14373360.9 -674235.6 -4.92 > "Run 10 12762101.5 14482817.4 -1720715.9 -13.48 > > "Average" 13553161.95 14624441.45 -1071279.5 -7.99 > "Standard deviation" 412810.77 187898.68 > 430895.31 3.46 > > Is there again something wrong on this test ?No, it seems fine. I get different results on x86-64 though (with -O), the difference is very small (4% with 3% stddev, or 3% with 1.5% stddev). Maybe you should try to reproduce one of the other benchmarks that show a higher difference, according to phoronix JOhn the Ripper shown 40% performance drop, there was also a huge drop for blowfish. GCC 4.4.1 GCC 4.2.4 LLVM-GCC 2.5 LLVM-GCC 2.6 GCC-4.4/LLVM2.5 GCC-4.4/LLVM2.6 GCC-4.4/GCC-4.2 144663277 139072037 154747143 151467440 0.935 0.955 1.040 144413249 139042455 150949846 150322055 0.957 0.961 1.039 144915166 138131283 150630261 150650540 0.962 0.962 1.049 145602711 138587982 154993781 147281539 0.939 0.989 1.051 145547868 139224734 150650813 147998859 0.966 0.983 1.045 145719970 139224734 144055081 152474829 1.012 0.956 1.047 average 145143706.83 138880537.5 151004487.5 150032543.67 0.962 0.968 1.045 stdev 551855.54 435155.79 3961926.31 2009642.82 0.027 0.015 0.005 Using built-in specs. Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Debian 4.4.1-4' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4 --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-mpfr --enable-objc-gc --with-arch-32=i486 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.4.1 (Debian 4.4.1-4) llvm2.5: Using built-in specs. Target: x86_64-unknown-linux-gnu Configured with: ../llvm-gcc-4.2/configure --program-prefix=llvm- --enable-llvm=/home/edwin/llvm2.5/obj --enable-languages=c,c++,fortran Thread model: posix gcc version 4.2.1 (Based on Apple Inc. build 5636) (LLVM build) llvm2.6pre1: Using built-in specs. Target: x86_64-unknown-linux-gnu Configured with: ../llvm-gcc4.2-2.6.source/configure --program-prefix=llvm- --enable-languages=c,c++,fortran --enable-llvm=/home/edwin/llvm2.6/obj/ Thread model: posix gcc version 4.2.1 (Based on Apple Inc. build 5649) (LLVM build) Using built-in specs. Target: x86_64-linux-gnu Configured with: ../src/configure -v --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.2.4 (Debian 4.2.4-6) Best regards, --Edwin
Some additional tests :
With : -O2 -march=core2 for both gcc and llvm-gcc : LLVM is better with 10%.
LLVM GCC Difference Difference %
Run 1 13771597.2 13105010.6 666586.6 4.84
Run 2 13813420.8 12536327.1 1277093.7 9.25
Run 3 13769573.8 12124207.3 1645366.5 11.95
Run 4 13883222.6 12794159.1 1089063.5 7.84
Run 5 13890414.6 12105946 1784468.6 12.85
Run 6 13784627.3 11648245 2136382.3 15.5
Run 7 13920748.9 11781692.4 2139056.5 15.37
Run 8 13816843.9 12460451.3 1356392.6 9.82
Run 9 13864682.1 12959671.4 905010.7 6.53
Run 10 13849863.4 12964021.5 885841.9 6.4
Average 13836499.46 12447973.17 1388526.29 10.03
Standard deviation 53189.13 515638.56
522400.98 3.77
With : -O3 -march=core2 for both compiler : GCC is better with 5.75%
LLVM GCC Difference Difference %
Run 1 13790728.3 14902550.9 -1111822.6 -8.06
Run 2 13840499.3 14759093.2 -918593.9 -6.64
Run 3 13975018.5 14545983.5 -570965 -4.09
Run 4 13853799.5 15126446.4 -1272646.9 -9.19
Run 5 13993633.5 14824100.4 -830466.9 -5.93
Run 6 13903645.3 12768793.8 1134851.5 8.16
Run 7 13816401.7 15003783.8 -1187382.1 -8.59
Run 8 13735232.8 14976854.3 -1241621.5 -9.04
Run 9 13902380.7 14778159.5 -875778.8 -6.3
Run 10 13936761.8 15022553.5 -1085791.7 -7.79
Average 13874810.14 14670831.93 -796021.79 -5.75
Standard deviation 82443.34 688446.69
712123.9 5.14
Olivier.
2009/9/16 Török Edwin <edwintorok at gmail.com>
> On 2009-09-16 11:42, Olivier Meurant wrote:
> > On Tue, Sep 15, 2009 at 5:35 PM, David Greene <dag at cray.com
> > <mailto:dag at cray.com>> wrote:
> >
> >
> > Remember, the goal here isn't to show how great LLVM is.
It's to
> > get an
> > honest assessment of where we are at. Phoronix did us a big
> > favor. Getting
> > more details about his tests would help us even more.
> >
> >
> > It's exactly what I'm trying to do. I'm not even a LLVM
contributor,
> > so I don't feel responsible for a good or bad performances of
LLVM. :)
> > Anyway, thanks for pointing this target differences. After some hours
> > at recompiling gcc, here we are :
> >
> > Some information on the test platform :
> >
> > cat /etc/lsb-release
> > DISTRIB_ID=Ubuntu
> > DISTRIB_RELEASE=9.04
> > DISTRIB_CODENAME=jaunty
> > DISTRIB_DESCRIPTION="Ubuntu 9.04"
> >
> > uname -a
> > Linux zaraki 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC
> > 2009 i686 GNU/Linux
> >
> > /proc/cpuinfo
> > dual core Intel(R) Core(TM)2 CPU X6800 @ 2.93GHz (cache size 4096)
> >
> > llvm-gcc -v
> > Using built-in specs.
> > Target: i686-pc-linux-gnu
> > Configured with: ../llvm-gcc4.2-2.5.source/configure
> >
>
--prefix=/mounts/zion/disks/0/localhome/tbrethou/2.5/prerelease2/llvm-gcc/obj/../install
> > --program-prefix=llvm-
> > --enable-llvm=/localhome/tbrethou/2.5/prerelease2/llvm-2.5/
> > --disable-bootstrap --enable-languages=c,c++,fortran
> > Thread model: posix
> > gcc version 4.2.1 (Based on Apple Inc. build 5636) (LLVM build 2.5)
> > (Binary from official x86 2.5 release)
> >
> > gcc -v
> > Using built-in specs.
> > Target: i686-pc-linux-gnu
> > Configured with: ./configure --prefix=/usr/local
> > Thread model: posix
> > gcc version 4.4.1 (GCC)
>
> Phoronix tested with gcc 4.2.1 (Apple's).
>
> >
> > The compilation line are :
> > gcc -o bin src.c -O3
> > llvm-gcc -o bin src.c -O3
>
> The Makefile uses -O by default, perhaps that is what Phoronix used too.
>
> >
> >
> > Results :
> > LLVM average : 13553161.95
> > gcc average : 14624441.45
> > Average : gcc is better with 8%
> >
> > The full report :
> > "LLVM" "GCC" "Difference"
"Difference %"
> >
> > "Run 1" 13842616 14693115.8 -850499.8
-6.14
> > "Run 2" 13773202.9 14785734.9 -1012532
-7.35
> > "Run 3" 13671309.3 14985114 -1313804.7
-9.61
> > "Run 4" 12916939 14640392.9 -1723453.9
-13.34
> > "Run 5" 13572133.3 14425575 -853441.7
-6.29
> > "Run 6" 13810096.5 14694240.6 -884144.1 -6.4
> > "Run 7" 13427435.8 14695094.6 -1267658.8
-9.44
> > "Run 8" 14056659.9 14468968.4 -412308.5
-2.93
> > "Run 9" 13699125.3 14373360.9 -674235.6
-4.92
> > "Run 10 12762101.5 14482817.4 -1720715.9 -13.48
> >
> > "Average" 13553161.95 14624441.45 -1071279.5
-7.99
> > "Standard deviation" 412810.77 187898.68
> > 430895.31 3.46
> >
> > Is there again something wrong on this test ?
>
> No, it seems fine.
>
> I get different results on x86-64 though (with -O), the difference is
> very small (4% with 3% stddev, or 3% with 1.5% stddev).
>
> Maybe you should try to reproduce one of the other benchmarks that show
> a higher difference, according to phoronix JOhn the Ripper shown 40%
> performance drop, there was also a huge drop for blowfish.
>
>
>
> GCC 4.4.1 GCC 4.2.4 LLVM-GCC 2.5 LLVM-GCC 2.6
> GCC-4.4/LLVM2.5
> GCC-4.4/LLVM2.6 GCC-4.4/GCC-4.2
>
> 144663277 139072037 154747143 151467440
> 0.935 0.955 1.040
>
> 144413249 139042455 150949846 150322055
> 0.957 0.961 1.039
>
> 144915166 138131283 150630261 150650540
> 0.962 0.962 1.049
>
> 145602711 138587982 154993781 147281539
> 0.939 0.989 1.051
>
> 145547868 139224734 150650813 147998859
> 0.966 0.983 1.045
>
> 145719970 139224734 144055081 152474829
> 1.012 0.956 1.047
> average 145143706.83 138880537.5 151004487.5
> 150032543.67 0.962
> 0.968 1.045
> stdev 551855.54 435155.79 3961926.31 2009642.82
> 0.027 0.015 0.005
>
>
> Using built-in specs.
> Target: x86_64-linux-gnu
> Configured with: ../src/configure -v --with-pkgversion='Debian
4.4.1-4'
> --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs
> --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
> --enable-shared --enable-multiarch --enable-linker-build-id
> --with-system-zlib --libexecdir=/usr/lib --without-included-gettext
> --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4
> --program-suffix=-4.4 --enable-nls --enable-clocale=gnu
> --enable-libstdcxx-debug --enable-mpfr --enable-objc-gc
> --with-arch-32=i486 --with-tune=generic --enable-checking=release
> --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
> Thread model: posix
> gcc version 4.4.1 (Debian 4.4.1-4)
>
> llvm2.5:
> Using built-in specs.
> Target: x86_64-unknown-linux-gnu
> Configured with: ../llvm-gcc-4.2/configure --program-prefix=llvm-
> --enable-llvm=/home/edwin/llvm2.5/obj --enable-languages=c,c++,fortran
> Thread model: posix
> gcc version 4.2.1 (Based on Apple Inc. build 5636) (LLVM build)
>
> llvm2.6pre1:
> Using built-in specs.
> Target: x86_64-unknown-linux-gnu
> Configured with: ../llvm-gcc4.2-2.6.source/configure
> --program-prefix=llvm- --enable-languages=c,c++,fortran
> --enable-llvm=/home/edwin/llvm2.6/obj/
> Thread model: posix
> gcc version 4.2.1 (Based on Apple Inc. build 5649) (LLVM build)
>
> Using built-in specs.
> Target: x86_64-linux-gnu
> Configured with: ../src/configure -v
> --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
> --enable-shared --with-system-zlib --libexecdir=/usr/lib
> --without-included-gettext --enable-threads=posix --enable-nls
> --with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2
> --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc
> --enable-mpfr --with-tune=generic --enable-checking=release
> --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
> Thread model: posix
> gcc version 4.2.4 (Debian 4.2.4-6)
>
>
> Best regards,
> --Edwin
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20090916/b0ee0726/attachment.html>
2009/9/16 Olivier Meurant <meurant.olivier at gmail.com>:> Average 13836499.46 12447973.17 1388526.29 10.03 > Standard deviation 53189.13 515638.56 > 522400.98 3.77That was pretty much what I was expecting from the article... numbers. It doesn't matter who's best, you can't be best in all areas, but profiling must be done right. The standard deviation is, at least, one order of magnitude lower than the difference between the two averages, which means the measurements have a meaning and express some reality. There is little value in the results on the article. People had to re-do it properly again, anyway... Shouldn't it be part of the standard release? I mean, add profiling as an automated task before every big release, comparing to previous versions of LLVM and other important compilers. Not to be waving the results about, but to know the weakness and work on them on the next release. It might take a while to build such infrastructure, but it's a good thing to do, I guess. cheers, --renato Reclaim your digital rights, eliminate DRM, learn more at http://www.defectivebydesign.org/what_is_drm