llvm dev - Mar 2014 - [LLVMdev] RFC - Adding an optimization report facility?

The context of this is performance analysis of generated code. My interest
is to trace at a high-level the major decisions done by the various
optimizers. For instance, when the inliner decides to inline foo into bar,
or the loop unroller decides to unroll a loop N times, or the vectorizer
decides to vectorize a loop body.

Many of these details are usually available via -debug-only. However, this
has several limitations:

   1. The output is generally too detailed. Passes will often emit the
   result of their analysis, what failed, what worked, etc. This is often fine
   when debugging the pass itself, but it's too noisy for initial analysis.
   2. The output is unstructured and it often uses pass-specific lingo
   which may confuse someone who just started looking at it.
   3. It forces a debug+asserts build (or at least a build with -UNDEBUG).
   4. Only one pass at a time can be asked to produce debugging output.

Additionally, I don't think it's right to co-opt the -debug-only
facility
for implementing optimization reports. They have different purposes. An
optimization report should simply state what the pass did in fairly terse
way. This facilitates initial and comparative analysis. If I want to
compare what one compiler did versus the current version, it would be easy
to spot what decisions were made by each one.

Clearly, the quality of the information will depend on how many decisions
are reported. Not every difference in performance will be detected by
comparing optimization reports. But major differences are often due to
major passes making slightly different decisions (e.g., the inliner).

My intent is to introduce an optimization report option to LLVM which
passes will be able to use to indicate the major decisions they make.
Initially, I am tempted to mimic GCC's -fopt-info (
http://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-fopt-info-747
).

Before I get too far into this, do folks think this is a good idea? I'm
open to reasonable requests on how the facility should work, etc.


Thanks. Diego.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140306/50de3572/attachment.html>

On Thu, Mar 6, 2014 at 11:07 AM, Diego Novillo <dnovillo at google.com>
wrote:
> Before I get too far into this, do folks think this is a good idea? I'm
> open to reasonable requests on how the facility should work, etc.

Yes, and see Tobias's recent work to add "remark" diagnostics to
LLVM and
Clang. I think it is targeted at precisely these kinds of use cases.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140306/7f79fc74/attachment.html>

On Thu, Mar 6, 2014 at 11:07 AM, Diego Novillo <dnovillo at google.com>
wrote:
>
> The context of this is performance analysis of generated code. My interest
> is to trace at a high-level the major decisions done by the various
> optimizers. For instance, when the inliner decides to inline foo into bar,
> or the loop unroller decides to unroll a loop N times, or the vectorizer
> decides to vectorize a loop body.
>
> Many of these details are usually available via -debug-only. However, this
> has several limitations:
>
> The output is generally too detailed. Passes will often emit the result of
> their analysis, what failed, what worked, etc. This is often fine when
> debugging the pass itself, but it's too noisy for initial analysis.
> The output is unstructured and it often uses pass-specific lingo which may
> confuse someone who just started looking at it.
> It forces a debug+asserts build (or at least a build with -UNDEBUG).
> Only one pass at a time can be asked to produce debugging output.
>
> Additionally, I don't think it's right to co-opt the -debug-only
facility
> for implementing optimization reports. They have different purposes. An
> optimization report should simply state what the pass did in fairly terse
> way. This facilitates initial and comparative analysis. If I want to
compare
> what one compiler did versus the current version, it would be easy to spot
> what decisions were made by each one.
>
> Clearly, the quality of the information will depend on how many decisions
> are reported. Not every difference in performance will be detected by
> comparing optimization reports. But major differences are often due to
major
> passes making slightly different decisions (e.g., the inliner).
>
> My intent is to introduce an optimization report option to LLVM which
passes
> will be able to use to indicate the major decisions they make. Initially, I
> am tempted to mimic GCC's -fopt-info
>
(http://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-fopt-info-747).
>
> Before I get too far into this, do folks think this is a good idea? I'm
open
> to reasonable requests on how the facility should work, etc.
>
Yes please.

-eric
>
> Thanks. Diego.
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

----- Original Message -----
> From: "Chandler Carruth" <chandlerc at google.com>
> To: "Diego Novillo" <dnovillo at google.com>, "Tobias
Grosser" <tobias at grosser.es>
> Cc: "LLVM Developers Mailing List" <llvmdev at cs.uiuc.edu>
> Sent: Thursday, March 6, 2014 1:11:30 PM
> Subject: Re: [LLVMdev] RFC - Adding an optimization report facility?
> 
> 
> 
> 
> 
> 
> On Thu, Mar 6, 2014 at 11:07 AM, Diego Novillo < dnovillo at google.com
> > wrote:
> 
> 
> Before I get too far into this, do folks think this is a good idea?
> I'm open to reasonable requests on how the facility should work,
> etc.
> Yes, and see Tobias's recent work to add "remark" diagnostics
to LLVM
> and Clang. I think it is targeted at precisely these kinds of use
> cases.
+1

 -Hal
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

On 03/06/2014 08:07 PM, Diego Novillo wrote:
> The context of this is performance analysis of generated code. My interest
> is to trace at a high-level the major decisions done by the various
> optimizers. For instance, when the inliner decides to inline foo into bar,
> or the loop unroller decides to unroll a loop N times, or the vectorizer
> decides to vectorize a loop body.
>
> Many of these details are usually available via -debug-only. However, this
> has several limitations:
>
>     1. The output is generally too detailed. Passes will often emit the
>     result of their analysis, what failed, what worked, etc. This is often
fine
>     when debugging the pass itself, but it's too noisy for initial
analysis.
>     2. The output is unstructured and it often uses pass-specific lingo
>     which may confuse someone who just started looking at it.
>     3. It forces a debug+asserts build (or at least a build with -UNDEBUG).
>     4. Only one pass at a time can be asked to produce debugging output.
>
> Additionally, I don't think it's right to co-opt the -debug-only
facility
> for implementing optimization reports. They have different purposes. An
> optimization report should simply state what the pass did in fairly terse
> way. This facilitates initial and comparative analysis. If I want to
> compare what one compiler did versus the current version, it would be easy
> to spot what decisions were made by each one.
>
> Clearly, the quality of the information will depend on how many decisions
> are reported. Not every difference in performance will be detected by
> comparing optimization reports. But major differences are often due to
> major passes making slightly different decisions (e.g., the inliner).
>
> My intent is to introduce an optimization report option to LLVM which
> passes will be able to use to indicate the major decisions they make.
> Initially, I am tempted to mimic GCC's -fopt-info (
>
http://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-fopt-info-747
> ).
>
> Before I get too far into this, do folks think this is a good idea? I'm
> open to reasonable requests on how the facility should work, etc.
Hi Diego,

as others already pointed out, I committed the first piece of such an 
infrastructure in LLVM commit 202474 and clang commit 202475. This is
mostly the backend and printing infrastructure.

This should already be functional, even though Alp pointed out there are 
still some open issues with the c-bindings and the verify diagnostic 
consumer (I still need to look into those). The command line options to 
enable this are currently working, but still very basic. Several people 
had ideas how to improve on this, but we did not yet agree on a 
solution. If you are interested in contributing here, that would be 
great. I have myself a patch on the clang-commits mailing list ('tblgen: 
Modularize the diagnostic emitter') which prepares the command line 
infrastructure for possible changes, but the more complicated issue of 
finding a good interface without increasing complexity is still open. 
Also, we still need more in-tree users to
better test the infrastructure and to get an idea of what kind of 
reports we would like to emit (which again can help shaping the command 
line interface). Your thoughts and contributions are highly appreciated.

Cheers,
Tobias

On Thu, Mar 6, 2014 at 2:44 PM, Tobias Grosser <tobias at grosser.es>
wrote:

as others already pointed out, I committed the first piece of such
an
> infrastructure in LLVM commit 202474 and clang commit 202475. This is
> mostly the backend and printing infrastructure.
>

Thanks, Tobias.  I've browsed the two patches and I think they're going
to
be exactly what I need. IIUC, the patches add two new LLVM instructions
remark and note. These are inserted in the IL by the passes and the
compiler emits them as diagnostic if the right -W flag is enabled?

One question I have from the Clang patch. If I compile with -Weverything,
will this enable all warnings *and* remarks? In this context, I only want
to enable all remarks.  In fact, I want to enable a family of remarks: the
optimization remarks.

There will be other modifiers to these remarks as well:


   1. Report *missed* optimizations, instead of the successful ones.
   2. Increase verbosity of the report. This would be done using note
   nodes, I expect. But we may want varying degrees of verbosity.
   3. Group families of optimizations. For example, i want to report on all
   loop-related optimizations.
   4. IIRC, GCC's -fopt-info will also allow you to collect the reports
   into a separate text file. Not sure how useful I find this feature myself.


Thanks. Diego.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140306/3e9fa1ce/attachment.html>

On Mar 6, 2014, at 11:07 AM, Diego Novillo <dnovillo at google.com>
wrote:
> The context of this is performance analysis of generated code. My interest
is to trace at a high-level the major decisions done by the various optimizers.
For instance, when the inliner decides to inline foo into bar, or the loop
unroller decides to unroll a loop N times, or the vectorizer decides to
vectorize a loop body.
...
> Before I get too far into this, do folks think this is a good idea? I'm
open to reasonable requests on how the facility should work, etc.
This is a great idea, and many people would welcome it.  Please write up a
concrete proposals about how this will work, and we can iterate on that though.
> My intent is to introduce an optimization report option to LLVM which
passes will be able to use to indicate the major decisions they make. Initially,
I am tempted to mimic GCC's -fopt-info
(http://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-fopt-info-747).
I'm not sure if this is the best design or not (having never used it) - what
feedback have you heard from (non-compiler-hacker) people trying to use it?

IMO, the hard part of doing something like this is getting the user experience
right.  It does you no good to say "hey I unrolled a loop" if you
don't have enough location information to tell the user *which* loop got
unrolled.  The key is to give them actionable information, not just the output
of -debug :-)

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140306/9cf1c595/attachment.html>

----- Original Message -----
> From: "Chris Lattner" <clattner at apple.com>
> To: "Diego Novillo" <dnovillo at google.com>
> Cc: "LLVM Developers Mailing List" <llvmdev at cs.uiuc.edu>
> Sent: Thursday, March 6, 2014 5:54:02 PM
> Subject: Re: [LLVMdev] RFC - Adding an optimization report facility?
> 
> 
> On Mar 6, 2014, at 11:07 AM, Diego Novillo < dnovillo at google.com >
> wrote:
> 
> 
> 
> 
> 
> The context of this is performance analysis of generated code. My
> interest is to trace at a high-level the major decisions done by the
> various optimizers. For instance, when the inliner decides to inline
> foo into bar, or the loop unroller decides to unroll a loop N times,
> or the vectorizer decides to vectorize a loop body. ...
> 
> 
> 
> 
> 
> Before I get too far into this, do folks think this is a good idea?
> I'm open to reasonable requests on how the facility should work,
> etc.
> 
> This is a great idea, and many people would welcome it. Please write
> up a concrete proposals about how this will work, and we can iterate
> on that though.
> 
> 
> 
> 
> 
> My intent is to introduce an optimization report option to LLVM which
> passes will be able to use to indicate the major decisions they
> make. Initially, I am tempted to mimic GCC's -fopt-info (
>
http://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-fopt-info-747
> ).
> 
> 
> 
> I'm not sure if this is the best design or not (having never used it)
> - what feedback have you heard from (non-compiler-hacker) people
> trying to use it?
> 
> IMO, the hard part of doing something like this is getting the user
> experience right. It does you no good to say "hey I unrolled a
loop"
> if you don't have enough location information to tell the user
> *which* loop got unrolled. The key is to give them actionable
> information, not just the output of -debug :-)
My suggestion is that we start attaching loop-id metadata to loops in the
frontend, and then also start attaching 'srcloc' metadata, just like we
do for inline asm statements. This way we can pass back the information we need
to the frontend for it to identify the loop without too much trouble. There may
be a better long-term design, but this seems, at least, like an easy thing to do
in the short term.

 -Hal 
> 
> 
> -Chris
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

On Thu, Mar 6, 2014 at 3:54 PM, Chris Lattner <clattner at apple.com>
wrote:
> On Mar 6, 2014, at 11:07 AM, Diego Novillo <dnovillo at google.com>
wrote:
>
> The context of this is performance analysis of generated code. My interest
> is to trace at a high-level the major decisions done by the various
> optimizers. For instance, when the inliner decides to inline foo into bar,
> or the loop unroller decides to unroll a loop N times, or the vectorizer
> decides to vectorize a loop body.
>
> ...
>
> Before I get too far into this, do folks think this is a good idea? I'm
> open to reasonable requests on how the facility should work, etc.
>
>
> This is a great idea, and many people would welcome it.  Please write up a
> concrete proposals about how this will work, and we can iterate on that
> though.
>
> My intent is to introduce an optimization report option to LLVM which
> passes will be able to use to indicate the major decisions they make.
> Initially, I am tempted to mimic GCC's -fopt-info (
>
http://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-fopt-info-747
> ).
>
>
> I'm not sure if this is the best design or not (having never used it) -
> what feedback have you heard from (non-compiler-hacker) people trying to
> use it?
>
> IMO, the hard part of doing something like this is getting the user
> experience right.  It does you no good to say "hey I unrolled a
loop" if
> you don't have enough location information to tell the user *which*
loop
> got unrolled.  The key is to give them actionable information, not just the
> output of -debug :-)
>
yep, the information will be useless without source/location information.

David
>
> -Chris
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140306/dc2ede27/attachment.html>

On Thu, Mar 6, 2014 at 6:54 PM, Chris Lattner <clattner at apple.com>
wrote:
>
> This is a great idea, and many people would welcome it.  Please write up a
> concrete proposals about how this will work, and we can iterate on that
> though.
>
Yup, working on one. Will send it out in the coming days.


My intent is to introduce an optimization report option to LLVM
which
> passes will be able to use to indicate the major decisions they make.
> Initially, I am tempted to mimic GCC's -fopt-info (
>
http://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-fopt-info-747
> ).
>
>
> I'm not sure if this is the best design or not (having never used it) -
> what feedback have you heard from (non-compiler-hacker) people trying to
> use it?
>
Used heavily in our optimization team. It's no altogether different from
what I've seen in other compilers in the past. I don't intend to mimic
the
complete UI, I only want to convey the same information.


IMO, the hard part of doing something like this is getting the
user
> experience right.  It does you no good to say "hey I unrolled a
loop" if
> you don't have enough location information to tell the user *which*
loop
> got unrolled.  The key is to give them actionable information, not just the
> output of -debug :-)
>
Absolutely.  Every note must always be emitted with location information on
the instruction that generates it. In GCC, the compiler keeps track of
source LOCs at all times.  But in Clang, we will want to turn on
-gline-tables-only when reports are requested.

The UI is based exclusively on source LOCs.  We want to preserve the exact
same formatting used for warnings/errors so that we can feed these notices
back to editors, IDEs, etc.


Diego.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140307/71a623c7/attachment.html>

I have written down my thoughts and added some of the feedback from this
thread. The proposal is probably still sparse, but I would like to start
the discussion so I can shape it into final documentation when we are done
painting the bike shed.

https://docs.google.com/document/d/1FYUatSjZZO-zmFBxjOiuOzAy9mhHA8hqdvklZv68WuQ/edit?usp=sharing

Please add your feedback on the document itself (anyone should be able to
comment) or to this thread.

Tobias, I will look at your patches this week and start producing my own
changes on top of them.


Thanks.  Diego.


On Thu, Mar 6, 2014 at 2:07 PM, Diego Novillo <dnovillo at google.com>
wrote:
>
> The context of this is performance analysis of generated code. My interest
> is to trace at a high-level the major decisions done by the various
> optimizers. For instance, when the inliner decides to inline foo into bar,
> or the loop unroller decides to unroll a loop N times, or the vectorizer
> decides to vectorize a loop body.
>
> Many of these details are usually available via -debug-only. However, this
> has several limitations:
>
>    1. The output is generally too detailed. Passes will often emit the
>    result of their analysis, what failed, what worked, etc. This is often
fine
>    when debugging the pass itself, but it's too noisy for initial
analysis.
>    2. The output is unstructured and it often uses pass-specific lingo
>    which may confuse someone who just started looking at it.
>    3. It forces a debug+asserts build (or at least a build with -UNDEBUG).
>    4. Only one pass at a time can be asked to produce debugging output.
>
> Additionally, I don't think it's right to co-opt the -debug-only
facility
> for implementing optimization reports. They have different purposes. An
> optimization report should simply state what the pass did in fairly terse
> way. This facilitates initial and comparative analysis. If I want to
> compare what one compiler did versus the current version, it would be easy
> to spot what decisions were made by each one.
>
> Clearly, the quality of the information will depend on how many decisions
> are reported. Not every difference in performance will be detected by
> comparing optimization reports. But major differences are often due to
> major passes making slightly different decisions (e.g., the inliner).
>
> My intent is to introduce an optimization report option to LLVM which
> passes will be able to use to indicate the major decisions they make.
> Initially, I am tempted to mimic GCC's -fopt-info (
>
http://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-fopt-info-747
> ).
>
> Before I get too far into this, do folks think this is a good idea? I'm
> open to reasonable requests on how the facility should work, etc.
>
>
> Thanks. Diego.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140311/02e77f9c/attachment.html>

This feature will be very useful and I suggest taking a bigger perspective.

Not only the major decisions but also simple optimizations metrics like
#<opt_instances>, #spills etc can be useful for performance analysis,
comparison and tracking. In addition to metrics locality is helpful,
specifically a report at a function level. A good place to start and test the
design could be to issue current —stats per function. In practice the users of
these reports know the hot functions and like to drill down.

The reports themselves could offer various levels of verbosity for the user to
pick (and the implementer to decide which information to report at which level).

For performance tuning a compiler that also reports why it didn’t apply an
optimization can be very useful (like in “I didn’t vectorize because of this
dependency.” etc.).

-Gerolf


On Mar 11, 2014, at 2:39 PM, Diego Novillo <dnovillo at google.com> wrote:
> I have written down my thoughts and added some of the feedback from this
thread. The proposal is probably still sparse, but I would like to start the
discussion so I can shape it into final documentation when we are done painting
the bike shed.
> 
>
https://docs.google.com/document/d/1FYUatSjZZO-zmFBxjOiuOzAy9mhHA8hqdvklZv68WuQ/edit?usp=sharing
> 
> Please add your feedback on the document itself (anyone should be able to
comment) or to this thread.
> 
> Tobias, I will look at your patches this week and start producing my own
changes on top of them.
> 
> 
> Thanks.  Diego.
> 
> 
> On Thu, Mar 6, 2014 at 2:07 PM, Diego Novillo <dnovillo at
google.com> wrote:
> 
> The context of this is performance analysis of generated code. My interest
is to trace at a high-level the major decisions done by the various optimizers.
For instance, when the inliner decides to inline foo into bar, or the loop
unroller decides to unroll a loop N times, or the vectorizer decides to
vectorize a loop body.
> 
> Many of these details are usually available via -debug-only. However, this
has several limitations:
> The output is generally too detailed. Passes will often emit the result of
their analysis, what failed, what worked, etc. This is often fine when debugging
the pass itself, but it's too noisy for initial analysis.
> The output is unstructured and it often uses pass-specific lingo which may
confuse someone who just started looking at it.
> It forces a debug+asserts build (or at least a build with -UNDEBUG).
> Only one pass at a time can be asked to produce debugging output.
> Additionally, I don't think it's right to co-opt the -debug-only
facility for implementing optimization reports. They have different purposes. An
optimization report should simply state what the pass did in fairly terse way.
This facilitates initial and comparative analysis. If I want to compare what one
compiler did versus the current version, it would be easy to spot what decisions
were made by each one.
> 
> Clearly, the quality of the information will depend on how many decisions
are reported. Not every difference in performance will be detected by comparing
optimization reports. But major differences are often due to major passes making
slightly different decisions (e.g., the inliner).
> 
> My intent is to introduce an optimization report option to LLVM which
passes will be able to use to indicate the major decisions they make. Initially,
I am tempted to mimic GCC's -fopt-info
(http://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-fopt-info-747).
> 
> Before I get too far into this, do folks think this is a good idea? I'm
open to reasonable requests on how the facility should work, etc.
> 
> 
> Thanks. Diego.
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140311/c444c672/attachment.html>