llvm dev - Sep 2009 - [LLVMdev] FYI: Phoronix GCC vs. LLVM-GCC benchmarks

Olivier Meurant:
> I have run the john the ripper test.
> I have used the official archive (same version as phoronix) from 
> http://www.openwall.com/john/g/john-1.7.3.1.tar.bz2
> 
> To build with llvm-gcc, replace the line CC = gcc with CC = llvm-gcc.
> I have used the following command to build : make clean linux-x86-sse2 
> (seems to be the best on x86-32)
> The makefile invocation is "gcc -c -Wall -O2 -fomit-frame-pointer 
> -funroll-loops src.c" and "llvm-gcc -c -Wall -O2
-fomit-frame-pointer
> -funroll-loops src.c".
I don't know what you think about this, but shouldn't it be more 
meaningful to make these tests with -O3? I mean, we ought to make the 
comparisons with the highest level of optimization available for both of 
the compilers. It is difficult to compare an intermediate level.

Stefano

On Wed, Sep 16, 2009 at 3:46 PM, Stefano Delli Ponti <
stefano.delliponti at gmail.com> wrote:
> I don't know what you think about this, but shouldn't it be more
> meaningful to make these tests with -O3? I mean, we ought to make the
> comparisons with the highest level of optimization available for both of
> the compilers. It is difficult to compare an intermediate level.
>

A quick test confirms that the 40% difference stays even with an -O3 switch
on both compilers.
I just "choose" -O2 as it's the default of the original makefile.

Olivier.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20090916/c8de5bf6/attachment.html>

On Sep 16, 2009, at 6:46 AM, Stefano Delli Ponti wrote:
> Olivier Meurant:
>> I have run the john the ripper test.
>> I have used the official archive (same version as phoronix) from
>> http://www.openwall.com/john/g/john-1.7.3.1.tar.bz2
>>
>> To build with llvm-gcc, replace the line CC = gcc with CC = llvm-gcc.
>> I have used the following command to build : make clean linux-x86- 
>> sse2
>> (seems to be the best on x86-32)
>> The makefile invocation is "gcc -c -Wall -O2 -fomit-frame-pointer
>> -funroll-loops src.c" and "llvm-gcc -c -Wall -O2
-fomit-frame-pointer
>> -funroll-loops src.c".
>
> I don't know what you think about this, but shouldn't it be more
> meaningful to make these tests with -O3? I mean, we ought to make the
> comparisons with the highest level of optimization available for  
> both of
> the compilers. It is difficult to compare an intermediate level.
Comparing -O3 (and even -O4) is interesting, but we want all  
optimization levels to perform better than GCC :).  Lots of people use  
-O2 and -Os, so comparing against other compiler's -O2 and -Os levels  
is just as interesting as comparing -O3 vs -O3.

-Chris

Chris Lattner:
> Comparing -O3 (and even -O4) is interesting, but we want all  
> optimization levels to perform better than GCC :).  Lots of people use  
> -O2 and -Os, so comparing against other compiler's -O2 and -Os levels  
> is just as interesting as comparing -O3 vs -O3.
> 
My thinking was that, for instance, -02 for GCC and -02 for LLVM(-GCC) 
do not necessarily mean the same thing, they may be not /commensurable/.
But perhaps, my ignorance, you are saying that they _are_, that LLVM 
assigns the same types of optimizations as GCC to the different levels.

Regards,
Stefano

On 2009-09-16 17:32, Olivier Meurant wrote:
>
>
> On Wed, Sep 16, 2009 at 3:46 PM, Stefano Delli Ponti
> <stefano.delliponti at gmail.com <mailto:stefano.delliponti at
gmail.com>>
> wrote:
>
>     I don't know what you think about this, but shouldn't it be
more
>     meaningful to make these tests with -O3? I mean, we ought to make the
>     comparisons with the highest level of optimization available for
>     both of
>     the compilers. It is difficult to compare an intermediate level.
>
>
>
> A quick test confirms that the 40% difference stays even with an -O3
> switch on both compilers.
> I just "choose" -O2 as it's the default of the original
makefile.
It would be interesting to see if LLVM has improved since 2.5 on this
benchmark, could you try the prerelease here?
http://llvm.org/prereleases/2.6/

A 40% performance loss is bad, something should be done about it for
LLVM 2.7 (if it is too late for 2.6 already).

Best regards,
--Edwin