thr3ads.net - llvm dev - [LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks [Sep 2009]

If this information is useful, please help other people find it:
Share via:

David Greene

2009-Sep-15 15:35 UTC

[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

On Tuesday 15 September 2009 08:16, Olivier Meurant wrote:
> llvm-gcc -v
> Using built-in specs.
> Target: i686-pc-linux-gnu
>
> gcc -v
> Using built-in specs.
> Target: i486-linux-gnu
Different targets.  It's no surprise 486 code would perform much worse than 
i686 code.  These compilers need to target the same architecture.  Ideally the 
Makefile would include appropriate -march/-mtune/whatever the gcc architecture 
switch of the day is.

All test reports should also include switches passed to the compiler for each 
run.

Right now this isn't a fair test.  It's practically meaningless.

Remember, the goal here isn't to show how great LLVM is.  It's to get an
honest assessment of where we are at.  Phoronix did us a big favor.  Getting
more details about his tests would help us even more.

For one thing, we (Cray) have a number of LLVM enhancements geared toward 
Opteron that make a big difference in performance.  I'm planning to
contribute
much of this to our open source community.  So there is definitely room for 
improvement to LLVM.  The biggest mistake we can make is to think we're the 
best.

                                 -Dave

罗勇刚(Yonggang Luo)

2009-Sep-15 16:28 UTC

head link

[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

Wow, that's beautiful: and thanks

2009/9/15, David Greene <dag at cray.com>:> On Tuesday 15 September 2009 08:16, Olivier Meurant wrote:
>
>> llvm-gcc -v
>> Using built-in specs.
>> Target: i686-pc-linux-gnu
>
>>
>> gcc -v
>> Using built-in specs.
>> Target: i486-linux-gnu
>
> Different targets.  It's no surprise 486 code would perform much worse
than
> i686 code.  These compilers need to target the same architecture.  Ideally
> the
> Makefile would include appropriate -march/-mtune/whatever the gcc
> architecture
> switch of the day is.
>
> All test reports should also include switches passed to the compiler for
> each
> run.
>
> Right now this isn't a fair test.  It's practically meaningless.
>
> Remember, the goal here isn't to show how great LLVM is.  It's to
get an
> honest assessment of where we are at.  Phoronix did us a big favor. 
Getting
> more details about his tests would help us even more.
>
> For one thing, we (Cray) have a number of LLVM enhancements geared toward
> Opteron that make a big difference in performance.  I'm planning to
> contribute
> much of this to our open source community.  So there is definitely room for
> improvement to LLVM.  The biggest mistake we can make is to think we're
the
> best.
>
>                                  -Dave
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-- 
从我的移动设备发送

         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo

Olivier Meurant

2009-Sep-16 08:42 UTC

head link

[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

On Tue, Sep 15, 2009 at 5:35 PM, David Greene <dag at cray.com> wrote:
>
> Remember, the goal here isn't to show how great LLVM is.  It's to
get an
> honest assessment of where we are at.  Phoronix did us a big favor.
>  Getting
> more details about his tests would help us even more.
>
It's exactly what I'm trying to do. I'm not even a LLVM contributor,
so I
don't feel responsible for a good or bad performances of LLVM. :)
Anyway, thanks for pointing this target differences. After some hours at
recompiling gcc, here we are :

Some information on the test platform :

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=9.04
DISTRIB_CODENAME=jaunty
DISTRIB_DESCRIPTION="Ubuntu 9.04"

uname -a
Linux zaraki 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC 2009
i686 GNU/Linux

/proc/cpuinfo
dual core Intel(R) Core(TM)2 CPU X6800  @ 2.93GHz (cache size 4096)

llvm-gcc -v
Using built-in specs.
Target: i686-pc-linux-gnu
Configured with: ../llvm-gcc4.2-2.5.source/configure
--prefix=/mounts/zion/disks/0/localhome/tbrethou/2.5/prerelease2/llvm-gcc/obj/../install
--program-prefix=llvm-
--enable-llvm=/localhome/tbrethou/2.5/prerelease2/llvm-2.5/
--disable-bootstrap --enable-languages=c,c++,fortran
Thread model: posix
gcc version 4.2.1 (Based on Apple Inc. build 5636) (LLVM build 2.5)
(Binary from official x86 2.5 release)

gcc -v
Using built-in specs.
Target: i686-pc-linux-gnu
Configured with: ./configure --prefix=/usr/local
Thread model: posix
gcc version 4.4.1 (GCC)

The compilation line are :
gcc -o bin src.c -O3
llvm-gcc -o bin src.c -O3


Results :
LLVM average : 13553161.95
gcc average : 14624441.45
Average : gcc is better with 8%

The full report :
        "LLVM"  "GCC"   "Difference"   
"Difference %"

"Run 1" 13842616        14693115.8      -850499.8       -6.14
"Run 2" 13773202.9      14785734.9      -1012532        -7.35
"Run 3" 13671309.3      14985114        -1313804.7      -9.61
"Run 4" 12916939        14640392.9      -1723453.9      -13.34
"Run 5" 13572133.3      14425575        -853441.7       -6.29
"Run 6" 13810096.5      14694240.6      -884144.1       -6.4
"Run 7" 13427435.8      14695094.6      -1267658.8      -9.44
"Run 8" 14056659.9      14468968.4      -412308.5       -2.93
"Run 9" 13699125.3      14373360.9      -674235.6       -4.92
"Run 10 12762101.5      14482817.4      -1720715.9      -13.48

"Average"       13553161.95     14624441.45     -1071279.5      -7.99
"Standard deviation"    412810.77       187898.68       430895.31     
3.46

Is there again something wrong on this test ?

Olivier.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20090916/17b14394/attachment.html>

Anton Korobeynikov

2009-Sep-16 09:14 UTC

head link

[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

Hello, Oliver
> The compilation line are :
> gcc -o bin src.c -O3
> llvm-gcc -o bin src.c -O3I'm not quite sure which target processor the compiler will optimize
code for. I'd suggest adding explicit -march=core2 here. Also, could
you please do a separate run with O2, not O3.

Thanks!

-- 
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University

Török Edwin

2009-Sep-16 09:22 UTC

head link

[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

On 2009-09-16 11:42, Olivier Meurant wrote:> On Tue, Sep 15, 2009 at 5:35 PM, David Greene <dag at cray.com
> <mailto:dag at cray.com>> wrote:
>
>
>     Remember, the goal here isn't to show how great LLVM is.  It's
to
>     get an
>     honest assessment of where we are at.  Phoronix did us a big
>     favor.  Getting
>     more details about his tests would help us even more.
>
>
> It's exactly what I'm trying to do. I'm not even a LLVM
contributor,
> so I don't feel responsible for a good or bad performances of LLVM. :)
> Anyway, thanks for pointing this target differences. After some hours
> at recompiling gcc, here we are :
>
> Some information on the test platform :
>
> cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=9.04
> DISTRIB_CODENAME=jaunty
> DISTRIB_DESCRIPTION="Ubuntu 9.04"
>
> uname -a
> Linux zaraki 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC
> 2009 i686 GNU/Linux
>
> /proc/cpuinfo
> dual core Intel(R) Core(TM)2 CPU X6800  @ 2.93GHz (cache size 4096)
>
> llvm-gcc -v
> Using built-in specs.
> Target: i686-pc-linux-gnu
> Configured with: ../llvm-gcc4.2-2.5.source/configure
>
--prefix=/mounts/zion/disks/0/localhome/tbrethou/2.5/prerelease2/llvm-gcc/obj/../install
> --program-prefix=llvm-
> --enable-llvm=/localhome/tbrethou/2.5/prerelease2/llvm-2.5/
> --disable-bootstrap --enable-languages=c,c++,fortran
> Thread model: posix
> gcc version 4.2.1 (Based on Apple Inc. build 5636) (LLVM build 2.5)
> (Binary from official x86 2.5 release)
>
> gcc -v
> Using built-in specs.
> Target: i686-pc-linux-gnu
> Configured with: ./configure --prefix=/usr/local
> Thread model: posix
> gcc version 4.4.1 (GCC)
Phoronix tested with gcc 4.2.1 (Apple's).
>
> The compilation line are :
> gcc -o bin src.c -O3
> llvm-gcc -o bin src.c -O3
The Makefile uses -O by default, perhaps that is what Phoronix used too.
>
>
> Results :
> LLVM average : 13553161.95
> gcc average : 14624441.45
> Average : gcc is better with 8%
>
> The full report :
>         "LLVM"  "GCC"   "Difference"   
"Difference %"
>
> "Run 1" 13842616        14693115.8      -850499.8       -6.14
> "Run 2" 13773202.9      14785734.9      -1012532        -7.35
> "Run 3" 13671309.3      14985114        -1313804.7      -9.61
> "Run 4" 12916939        14640392.9      -1723453.9      -13.34
> "Run 5" 13572133.3      14425575        -853441.7       -6.29
> "Run 6" 13810096.5      14694240.6      -884144.1       -6.4
> "Run 7" 13427435.8      14695094.6      -1267658.8      -9.44
> "Run 8" 14056659.9      14468968.4      -412308.5       -2.93
> "Run 9" 13699125.3      14373360.9      -674235.6       -4.92
> "Run 10 12762101.5      14482817.4      -1720715.9      -13.48
>
> "Average"       13553161.95     14624441.45     -1071279.5     
-7.99
> "Standard deviation"    412810.77       187898.68      
> 430895.31       3.46
>
> Is there again something wrong on this test ?
No, it seems fine.

I get different results on x86-64 though (with -O), the difference is
very small (4% with 3% stddev, or 3% with 1.5% stddev).

Maybe you should try to reproduce one of the other benchmarks that show
a higher difference, according to phoronix JOhn the Ripper shown 40%
performance drop, there was also a huge drop for blowfish.



	GCC 4.4.1 	GCC 4.2.4 	LLVM-GCC 2.5 	LLVM-GCC 2.6 	GCC-4.4/LLVM2.5
GCC-4.4/LLVM2.6 	GCC-4.4/GCC-4.2

	144663277 	139072037 	154747143 	151467440 	0.935 	0.955 	1.040

	144413249 	139042455 	150949846 	150322055 	0.957 	0.961 	1.039

	144915166 	138131283 	150630261 	150650540 	0.962 	0.962 	1.049

	145602711 	138587982 	154993781 	147281539 	0.939 	0.989 	1.051

	145547868 	139224734 	150650813 	147998859 	0.966 	0.983 	1.045

	145719970 	139224734 	144055081 	152474829 	1.012 	0.956 	1.047
average 	145143706.83 	138880537.5 	151004487.5 	150032543.67 	0.962
0.968 	1.045
stdev 	551855.54 	435155.79 	3961926.31 	2009642.82 	0.027 	0.015 	0.005


Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.4.1-4'
--with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
--enable-shared --enable-multiarch --enable-linker-build-id
--with-system-zlib --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4
--program-suffix=-4.4 --enable-nls --enable-clocale=gnu
--enable-libstdcxx-debug --enable-mpfr --enable-objc-gc
--with-arch-32=i486 --with-tune=generic --enable-checking=release
--build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.4.1 (Debian 4.4.1-4)

llvm2.5:
Using built-in specs.
Target: x86_64-unknown-linux-gnu
Configured with: ../llvm-gcc-4.2/configure --program-prefix=llvm-
--enable-llvm=/home/edwin/llvm2.5/obj --enable-languages=c,c++,fortran
Thread model: posix
gcc version 4.2.1 (Based on Apple Inc. build 5636) (LLVM build)

llvm2.6pre1:
Using built-in specs.
Target: x86_64-unknown-linux-gnu
Configured with: ../llvm-gcc4.2-2.6.source/configure
--program-prefix=llvm- --enable-languages=c,c++,fortran
--enable-llvm=/home/edwin/llvm2.6/obj/
Thread model: posix
gcc version 4.2.1 (Based on Apple Inc. build 5649) (LLVM build)

Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
--enable-shared --with-system-zlib --libexecdir=/usr/lib
--without-included-gettext --enable-threads=posix --enable-nls
--with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2
--enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc
--enable-mpfr --with-tune=generic --enable-checking=release
--build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.2.4 (Debian 4.2.4-6)


Best regards,
--Edwin

Seemingly Similar Threads

Search for more possibly parallel threads

llvm dev - Sep 2009 - [LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

[LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

Seemingly Similar Threads