llvm dev - Nov 2011 - [LLVMdev] Performance Tracking

Hello Everyone,

I've been looking at benchmarks of LLVM recently, and overall they look
pretty good.  Aside from things that use OpenMP or benefit from
autovectorisation, Clang/LLVM and GCC seem to come fairly close, with no overall
winner.

But: there do seem to have been a number of performance regressions between 2.9
and 3.0:

http://openbenchmarking.org/result/1110178-AR-1110173AR66

Identifying where these were introduced is quite difficult.  I wonder if some of
the buildbots could be persuaded, after building clang, to build and run some
benchmark projects (e.g. perlbench, povray, flac / lame, whatever) and email
people if they introduce a performance regression, just as the current buildbots
email people who break the build.

David

On Nov 14, 2011, at 10:46 AM, David Chisnall wrote:
> Hello Everyone,
> 
> I've been looking at benchmarks of LLVM recently, and overall they look
pretty good.  Aside from things that use OpenMP or benefit from
autovectorisation, Clang/LLVM and GCC seem to come fairly close, with no overall
winner.
Nice. Thanks.
> 
> But: there do seem to have been a number of performance regressions between
2.9 and 3.0:
> 
> http://openbenchmarking.org/result/1110178-AR-1110173AR66
> 
> Identifying where these were introduced is quite difficult.  I wonder if
some of the buildbots could be persuaded, after building clang, to build and run
some benchmark projects (e.g. perlbench, povray, flac / lame, whatever) and
email people if they introduce a performance regression, just as the current
buildbots email people who break the build.
Step one would be to adopt these tests into the LLVM test harness. It will force
organizations that care about performance to start paying attention to these
benchmarks.

Evan
> 
> David
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

Le 14 novembre 2011 20:40, Evan Cheng <evan.cheng at apple.com> a écrit :
>
> On Nov 14, 2011, at 10:46 AM, David Chisnall wrote:
>
> > Hello Everyone,
> >
> > I've been looking at benchmarks of LLVM recently, and overall they
look
> pretty good.  Aside from things that use OpenMP or benefit from
> autovectorisation, Clang/LLVM and GCC seem to come fairly close, with no
> overall winner.
>
> Nice. Thanks.
>
> >
> > But: there do seem to have been a number of performance regressions
> between 2.9 and 3.0:
> >
> > http://openbenchmarking.org/result/1110178-AR-1110173AR66
> >
> > Identifying where these were introduced is quite difficult.  I wonder
if
> some of the buildbots could be persuaded, after building clang, to build
> and run some benchmark projects (e.g. perlbench, povray, flac / lame,
> whatever) and email people if they introduce a performance regression, just
> as the current buildbots email people who break the build.
>
> Step one would be to adopt these tests into the LLVM test harness. It will
> force organizations that care about performance to start paying attention
> to these benchmarks.
>
> Evan
>
> >
> > David
>
>
Many thanks David, it had been a while (6 months I guess) since the last
benchmark I saw and I was wondering how the new Clang/LLVM compared to GCC!

One comment though, the graphs are great, however the alternance of "less
is better"/"more is better" makes for a difficult read: it's
not obvious at
a glance which is performing better and it's difficult to get a quick
overview surveying the few graphs available.

Perhaps that introducing a relative performance would help: normalize on
the winner and indicate the factor or percentage next to the "losers"
?

Thanks for taking the time of doing this anyway, I'm looking forward to the
final article.

-- Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111116/fe16dd5b/attachment.html>