llvm dev - Oct 2011 - [LLVMdev] Comparison of optimizations

Hi,

I want to measure the performance of functions within a program, in
order to see whether or not a given set of optimizations are useful
(making the code faster in my case). It would be ideal to compare the
LLVM IR representation of the functions before and after the
optimizations, but the process has to be automatized so if there is no
code that can do this now then I guess it will be too hard to come up
with good heuristics for comparing the code with respect to execution
performance.
My current approach would be to use a profiler, such as
google-perftools, but the information acquired suffers from some
inaccuracy.
However I would like to hear your take on it and maybe if I should
implement the measurement mechanism in an interpreter (lli) that uses
a high resolution timer for entering and exiting functions. But it
depends on if the performance of the executed LLVM IR code in lli is
reflective on the performance from native compiled code, such that it
can be tested if a set of optimizations are useful or not. Or would it
be more desirable to make a pass that inserts code to activate and
deactivate a timer on function entry and exit?

Cheers, Michael

Maybe you could use the available profilers (i.e edge profiler, path
profiler...) to generate instrumented code, enabling and disabling the
optimizations you want to test, and then  make the comparison (assuming you
can actually disable the optimizations of interest without compromising
later passes). You could modify the profilers to collect information that is
more interesting for you as well.

On Sun, Oct 9, 2011 at 6:38 AM, bejer <bejerdk at gmail.com> wrote:
> Hi,
>
> I want to measure the performance of functions within a program, in
> order to see whether or not a given set of optimizations are useful
> (making the code faster in my case). It would be ideal to compare the
> LLVM IR representation of the functions before and after the
> optimizations, but the process has to be automatized so if there is no
> code that can do this now then I guess it will be too hard to come up
> with good heuristics for comparing the code with respect to execution
> performance.
> My current approach would be to use a profiler, such as
> google-perftools, but the information acquired suffers from some
> inaccuracy.
> However I would like to hear your take on it and maybe if I should
> implement the measurement mechanism in an interpreter (lli) that uses
> a high resolution timer for entering and exiting functions. But it
> depends on if the performance of the executed LLVM IR code in lli is
> reflective on the performance from native compiled code, such that it
> can be tested if a set of optimizations are useful or not. Or would it
> be more desirable to make a pass that inserts code to activate and
> deactivate a timer on function entry and exit?
>
> Cheers, Michael
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>


-- 
[Carolina Simões Gomes]
M.Sc. Student in Computing Science
University of Alberta, Canada
CAS Partner - IBM Toronto
+1 (780) 863-0155
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111009/9adca333/attachment.html>

bejer <bejerdk at gmail.com> writes:
> Hi,
>
> I want to measure the performance of functions within a program, in
> order to see whether or not a given set of optimizations are useful
> (making the code faster in my case). It would be ideal to compare the
> LLVM IR representation of the functions before and after the
> optimizations, but the process has to be automatized so if there is no
> code that can do this now then I guess it will be too hard to come up
> with good heuristics for comparing the code with respect to execution
> performance.
Camparing IR is going to be nearly useless.  It won't tell you much
about actual performance.
> My current approach would be to use a profiler, such as
> google-perftools, but the information acquired suffers from some
> inaccuracy.
What are you trying to measure that would require a profile?  The best
way to measure performance improvements of optimization is to compile
one version of the code without optimization and another with
optimization.  Run both and see which is faster.  It is the definitive
answer.
> However I would like to hear your take on it and maybe if I should
> implement the measurement mechanism in an interpreter (lli) that uses
> a high resolution timer for entering and exiting functions.
But you would be measuring interpreter performance, not performance of
the code.  Unless you will always run using the interpreter, this is
going to be misleading, at best.
> But it depends on if the performance of the executed LLVM IR code in
> lli is reflective on the performance from native compiled code, such
> that it can be tested if a set of optimizations are useful or not.
It is not.
> Or would it be more desirable to make a pass that inserts code to
> activate and deactivate a timer on function entry and exit?
Do you just want to time one function out of some benchmark?  No need to
make a pass.  Just add timers to the code.  But make sure that function
is really the performance dominator!

                              -Dave