llvm dev - Nov 2013 - [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)

On Nov 12, 2013, at 6:11 PM, Chandler Carruth <chandlerc at google.com>
wrote:
> On Tue, Nov 12, 2013 at 6:07 PM, Manman Ren <manman.ren at gmail.com>
wrote:
> Hi Chandler,
> 
> I don't quite get why you think sharing is not buying us anything...
> It reduces the memory footprint of the source modules (there is sharing
among the source modules) and the number of MDNodes created for the destination
module (we do not need to re-create the MDNodes that can be shared).
> 
> The amount of sharing may not be that much but it still exists.
> 
> I had some experiments earlier on building clang with "-flto -g",
if we dis-allow sharing between source modules and destination module, the
memory footprint for MDNodes will increase by 15%.
> 
> So, in my naive view, we do something like the following:
> 
> 0) load a source module
> 1) load another source module
> 2) merge the second module into the first
> 3) delete the second module
> 4) while there are more source modules, goto 1
I'll describe how the darwin linker uses the LTO interface.  It may be
amenable to earlier module deletion.

1) The darwin linker mmap()s each input file.  If it is a bitcode file, it calls
   lto_module_create_from_memory()
then lto_module_get_num_symbols() and lto_module_get_symbol_*() to discover what
the module provides and needs.

2) After all object files are loaded (which means no undefined symbols are
left), the linker then calls:
  lto_codegen_create() and then in a for-loop calls lto_codegen_add_module() on
each module previously loaded.

3) After lto_codegen_compile() has returned, the linker does clean up and
deletes each module with lto_module_dispose().

It sounds like the linker could call lto_module_dispose() right after
lto_codegen_add_module() to help reduce the memory footprint.  That would be a
simple linker change.  A slightly larger linker change would be to immediately
call lto_codegen_add_module() right after lto_module_create_from_memory(), then
lto_module_dispose().  That is, never have any unmerged modules laying around.

I have no idea is these sort of changes work for the gold plugin.

-Nick

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131113/449ee352/attachment.html>

> It sounds like the linker could call lto_module_dispose() right after
> lto_codegen_add_module() to help reduce the memory footprint.  That would
be
> a simple linker change.  A slightly larger linker change would be to
> immediately call lto_codegen_add_module() right after
> lto_module_create_from_memory(), then lto_module_dispose().  That is, never
> have any unmerged modules laying around.
>
> I have no idea is these sort of changes work for the gold plugin.
The gold plugin calls lto_codegen_add_module/lto_module_dispose early.
So it looks like Chandler's idea would be a win for gold but a loss
for ld64 right now.

Cheers,
Rafael

On Wed, Nov 13, 2013 at 7:58 PM, Rafael Espíndola <
rafael.espindola at gmail.com> wrote:
> > It sounds like the linker could call lto_module_dispose() right after
> > lto_codegen_add_module() to help reduce the memory footprint.  That
> would be
> > a simple linker change.  A slightly larger linker change would be to
> > immediately call lto_codegen_add_module() right after
> > lto_module_create_from_memory(), then lto_module_dispose().  That is,
> never
> > have any unmerged modules laying around.
> >
> > I have no idea is these sort of changes work for the gold plugin.
>
> The gold plugin calls lto_codegen_add_module/lto_module_dispose early.
> So it looks like Chandler's idea would be a win for gold but a loss
> for ld64 right now.
>
Letting the module own MDNodes may not be a win for gold since it is going
to create multiple copies of MDNodes that could be shared with Context
owning MDNodes.

For example, with debug info type uniquing, the type nodes can be shared
across modules, but with module owning MDNodes, each module will create its
own copy of the type nodes. The advantage is that the MDNodes can be
deleted easily by deleting the module. It is not clearly a win to me.

Manman

>
> Cheers,
> Rafael
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131113/3ad93341/attachment.html>