llvm dev - Jun 2019 - LLD/Link find out what triggers 80mb exe

I did try that yes. Didn't seem to make any difference (still 80ishmb)

On Fri, Jun 7, 2019, at 15:29, David Major wrote:
> Just to double check: are you calling lld-link with `-opt:ref`? It may or
may not be enabled by default, depending on other flags.
> 
> 
> On Fri, Jun 7, 2019 at 8:26 AM Carlo Kok via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>> __
>> Unfortunately, lld link doesn't do -map yet (and link.exe /map is
kinda useless; it shows what get linked in, but only refers to an object file,
not function name when it does so, like a single reference in that object file
triggers all it's dependencies)
>> 
>> Yeah I'm fairly sure it's caused by something I'm doing
wrong, I just have no idea how to find out. I thought -function-section would
have pretty much covered it.
>> 
>> On Fri, Jun 7, 2019, at 14:22, Rui Ueyama wrote:
>>> If you use only one function from a static library, and that adds
80 MiB to the executable, it is clear that that undefined symbol causes a lot of
other files to be loaded, so the question is why that function has such a large
(transitive) dependency and how to find the cause.
>>> 
>>> Unfortunately there's no exact command line option or a feature
to find it out. I have an idea to write a post-link analysis tool, which
processes a linker's debug output, to find out a "weak link" in a
dependency graph. If we can split a graph into two by removing a small number of
symbols (or vertices) from the graph, we can reduce a program size by removing
these symbols from the program. But that kind of analysis tool is not available
yet, and I don't know how powerful it would actually be.
>>> 
>>> There are a few options that may be useful when debugging the
issue.
>>> 
>>> `-v` makes lld to print out a file name as the linker loads a new
file.
>>> 
>>> `-Map=<filename>` makes lld to write an output file layout
information to a given file.
>>> 
>>> On Fri, Jun 7, 2019 at 8:59 PM Carlo Kok via llvm-dev <llvm-dev
at lists.llvm.org> wrote:
>>>> Is there any way to find out what causes a 80mb executable with
lld/link?
>>>> 
>>>> 
>>>> The object files are compiled with -function-sections and
-data-sections, the originating .lib is 270mb, but I'm calling 1 function in
it, which might call some stuff recursively, but I don't know how to find
out what exactly triggers the use. Is there a commandline option that shows
this?
>>>> 
>>>> (repro is 286 mb) : http://gofile.me/4Gflq/OOr8WKU12
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> 
>> _______________________________________________
>>  LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190607/bfc965bd/attachment.html>

You could try analyzing the PDB file using llvm-pdbutil.  We won't add any
symbols to the PDB that were eliminated by linker gc, so this should at
least allow you to get a good idea of what symbols are actually
contributing to the output.  It won't give you a full dependency graph like
Rui suggests, but it might be a good starting point.

On Fri, Jun 7, 2019 at 6:42 AM Carlo Kok via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I did try that yes. Didn't seem to make any difference (still 80ishmb)
>
> On Fri, Jun 7, 2019, at 15:29, David Major wrote:
>
> Just to double check: are you calling lld-link with `-opt:ref`? It may or
> may not be enabled by default, depending on other flags.
>
>
> On Fri, Jun 7, 2019 at 8:26 AM Carlo Kok via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>
> Unfortunately, lld link doesn't do -map yet (and link.exe /map is kinda
> useless; it shows what get linked in, but only refers to an object file,
> not function name when it does so, like a single reference in that object
> file triggers all it's dependencies)
>
> Yeah I'm fairly sure it's caused by something I'm doing wrong,
I just have
> no idea how to find out. I thought -function-section would have pretty much
> covered it.
>
> On Fri, Jun 7, 2019, at 14:22, Rui Ueyama wrote:
>
> If you use only one function from a static library, and that adds 80 MiB
> to the executable, it is clear that that undefined symbol causes a lot of
> other files to be loaded, so the question is why that function has such a
> large (transitive) dependency and how to find the cause.
>
> Unfortunately there's no exact command line option or a feature to find
it
> out. I have an idea to write a post-link analysis tool, which processes a
> linker's debug output, to find out a "weak link" in a
dependency graph. If
> we can split a graph into two by removing a small number of symbols (or
> vertices) from the graph, we can reduce a program size by removing these
> symbols from the program. But that kind of analysis tool is not available
> yet, and I don't know how powerful it would actually be.
>
> There are a few options that may be useful when debugging the issue.
>
> `-v` makes lld to print out a file name as the linker loads a new file.
>
> `-Map=<filename>` makes lld to write an output file layout
information to
> a given file.
>
> On Fri, Jun 7, 2019 at 8:59 PM Carlo Kok via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Is there any way to find out what causes a 80mb executable with lld/link?
>
>
> The object files are compiled with -function-sections and -data-sections,
> the originating .lib is 270mb, but I'm calling 1 function in it, which
> might call some stuff recursively, but I don't know how to find out
what
> exactly triggers the use. Is there a commandline option that shows this?
>
> (repro is 286 mb) : http://gofile.me/4Gflq/OOr8WKU12
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190607/59629297/attachment.html>

Oke so this turned out to be caused by the presence of .debug_info, which pulled
in the methods, and those pulled in the types. It seems just referencing a .lib
with .obj that have (dwarf) debug info makes it pull in a lot of stuff. But at
the very least, I now know what to avoid. Thanks!

On Fri, Jun 7, 2019, at 21:37, Zachary Turner wrote:
> You could try analyzing the PDB file using llvm-pdbutil. We won't add
any symbols to the PDB that were eliminated by linker gc, so this should at
least allow you to get a good idea of what symbols are actually contributing to
the output. It won't give you a full dependency graph like Rui suggests, but
it might be a good starting point.
> 
> On Fri, Jun 7, 2019 at 6:42 AM Carlo Kok via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>> __
>> I did try that yes. Didn't seem to make any difference (still
80ishmb)
>> 
>> On Fri, Jun 7, 2019, at 15:29, David Major wrote:
>>> Just to double check: are you calling lld-link with `-opt:ref`? It
may or may not be enabled by default, depending on other flags.
>>> 
>>> 
>>> On Fri, Jun 7, 2019 at 8:26 AM Carlo Kok via llvm-dev <llvm-dev
at lists.llvm.org> wrote:
>>>> __
>>>> Unfortunately, lld link doesn't do -map yet (and link.exe
/map is kinda useless; it shows what get linked in, but only refers to an object
file, not function name when it does so, like a single reference in that object
file triggers all it's dependencies)
>>>> 
>>>> Yeah I'm fairly sure it's caused by something I'm
doing wrong, I just have no idea how to find out. I thought -function-section
would have pretty much covered it.
>>>> 
>>>> On Fri, Jun 7, 2019, at 14:22, Rui Ueyama wrote:
>>>>> If you use only one function from a static library, and
that adds 80 MiB to the executable, it is clear that that undefined symbol
causes a lot of other files to be loaded, so the question is why that function
has such a large (transitive) dependency and how to find the cause.
>>>>> 
>>>>> Unfortunately there's no exact command line option or a
feature to find it out. I have an idea to write a post-link analysis tool, which
processes a linker's debug output, to find out a "weak link" in a
dependency graph. If we can split a graph into two by removing a small number of
symbols (or vertices) from the graph, we can reduce a program size by removing
these symbols from the program. But that kind of analysis tool is not available
yet, and I don't know how powerful it would actually be.
>>>>> 
>>>>> There are a few options that may be useful when debugging
the issue.
>>>>> 
>>>>> `-v` makes lld to print out a file name as the linker loads
a new file.
>>>>> 
>>>>> `-Map=<filename>` makes lld to write an output file
layout information to a given file.
>>>>> 
>>>>> On Fri, Jun 7, 2019 at 8:59 PM Carlo Kok via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>>>>>> Is there any way to find out what causes a 80mb
executable with lld/link?
>>>>>> 
>>>>>> 
>>>>>> The object files are compiled with -function-sections
and -data-sections, the originating .lib is 270mb, but I'm calling 1
function in it, which might call some stuff recursively, but I don't know
how to find out what exactly triggers the use. Is there a commandline option
that shows this?
>>>>>> 
>>>>>> (repro is 286 mb) : http://gofile.me/4Gflq/OOr8WKU12
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>> 
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> 
>> _______________________________________________
>>  LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190619/f5706209/attachment.html>