llvm dev - Apr 2019 - Tool to help hunt down binary size regressions

Hi folks,

Sometimes I make a change to the compiler that causes a binary size
regression and I need to track down where that regression came from. One
relatively naive way of doing this is to take two build trees, one compiled
with the old compiler and another compiled with the new compiler, and then
compare the symbol sizes (st_size) in the final output files in each build
tree. The problem with this is that the results can be very noisy,
especially if your change is early in the compiler pipeline, since it can
tend to affect heuristics used by the inliner to make inlining decisions.

What we really want to see is a size delta that ignores any changes that
resulted from different inlining decisions having been made. In the past
I've done this by passing -fno-inline when compiling both build trees, but
this won't be entirely representative given how important inlining is to
optimization. What I later realised was that we can use the relocation list
of each section to help identify functions that ended up having the same
inlining decisions apply to them -- if a symbol's relocations point to
different symbols in the two build trees, it is likely that different
inlining decisions were made.

I wrote a quick and dirty tool in Go that takes two build trees and
compares the sizes of the symbols in any object files found recursively in
the build trees, while ignoring any symbols whose relocations point to
different symbols. It can be found here:
https://github.com/pcc/llvm-project/blob/sym-reloc-sizes/llgo/utils/benchcomp/main.go

And it can be used like this:
$ benchcomp sym_reloc_sizes builddir1 builddir2 | sort -n -k 4

If this seems useful I can see if I can find some time to contribute a tool
based on this idea to LLVM in some form.

Thanks,
-- 
-- 
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190416/11ba146f/attachment.html>

On Tue, 16 Apr 2019 at 22:11, Peter Collingbourne via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> Hi folks,
>
> Sometimes I make a change to the compiler that causes a binary size
regression and I need to track down where that regression came from. One
relatively naive way of doing this is to take two build trees, one compiled with
the old compiler and another compiled with the new compiler, and then compare
the symbol sizes (st_size) in the final output files in each build tree. The
problem with this is that the results can be very noisy, especially if your
change is early in the compiler pipeline, since it can tend to affect heuristics
used by the inliner to make inlining decisions.
>
> What we really want to see is a size delta that ignores any changes that
resulted from different inlining decisions having been made. In the past
I've done this by passing -fno-inline when compiling both build trees, but
this won't be entirely representative given how important inlining is to
optimization. What I later realised was that we can use the relocation list of
each section to help identify functions that ended up having the same inlining
decisions apply to them -- if a symbol's relocations point to different
symbols in the two build trees, it is likely that different inlining decisions
were made.
>
> I wrote a quick and dirty tool in Go that takes two build trees and
compares the sizes of the symbols in any object files found recursively in the
build trees, while ignoring any symbols whose relocations point to different
symbols. It can be found here:
>
https://github.com/pcc/llvm-project/blob/sym-reloc-sizes/llgo/utils/benchcomp/main.go
You might want to also take a look at the size-info tool from Vedant
Kumar (CCed)
<http://lists.llvm.org/pipermail/llvm-dev/2018-September/126501.html>
<https://github.com/vedantk/llvm-project/tree/sizeinfo>

Vedant: do you plan to push size-info upstream?

Best,

Alex

Hi,
> On Apr 17, 2019, at 7:09 AM, Alex Bradbury <asb at asbradbury.org>
wrote:
> 
> On Tue, 16 Apr 2019 at 22:11, Peter Collingbourne via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>> 
>> Hi folks,
>> 
>> Sometimes I make a change to the compiler that causes a binary size
regression and I need to track down where that regression came from. One
relatively naive way of doing this is to take two build trees, one compiled with
the old compiler and another compiled with the new compiler, and then compare
the symbol sizes (st_size) in the final output files in each build tree. The
problem with this is that the results can be very noisy, especially if your
change is early in the compiler pipeline, since it can tend to affect heuristics
used by the inliner to make inlining decisions.
>> 
>> What we really want to see is a size delta that ignores any changes
that resulted from different inlining decisions having been made. In the past
I've done this by passing -fno-inline when compiling both build trees, but
this won't be entirely representative given how important inlining is to
optimization. What I later realised was that we can use the relocation list of
each section to help identify functions that ended up having the same inlining
decisions apply to them -- if a symbol's relocations point to different
symbols in the two build trees, it is likely that different inlining decisions
were made.
>> 
>> I wrote a quick and dirty tool in Go that takes two build trees and
compares the sizes of the symbols in any object files found recursively in the
build trees, while ignoring any symbols whose relocations point to different
symbols. It can be found here:
>>
https://github.com/pcc/llvm-project/blob/sym-reloc-sizes/llgo/utils/benchcomp/main.go
> 
> You might want to also take a look at the size-info tool from Vedant
> Kumar (CCed)
<http://lists.llvm.org/pipermail/llvm-dev/2018-September/126501.html
<http://lists.llvm.org/pipermail/llvm-dev/2018-September/126501.html>>
> <https://github.com/vedantk/llvm-project/tree/sizeinfo
<https://github.com/vedantk/llvm-project/tree/sizeinfo>>
This tool does display inlining trees, but doesn't have support for ignoring
symbols with different inlining histories while diffing.

> Vedant: do you plan to push size-info upstream?
I'd like to upstream a size analysis library derived from the prototype, but
don't know when I'll have time to do this. If anyone's interested in
grabbing the code and running with it, I'd be happy to give reviews!

best,
vedant
> 
> Best,
> 
> Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190417/c99fcb62/attachment.html>