Nick Kledzik
2013-Nov-13 21:55 UTC
[LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
On Nov 12, 2013, at 6:11 PM, Chandler Carruth <chandlerc at google.com> wrote:> On Tue, Nov 12, 2013 at 6:07 PM, Manman Ren <manman.ren at gmail.com> wrote: > Hi Chandler, > > I don't quite get why you think sharing is not buying us anything... > It reduces the memory footprint of the source modules (there is sharing among the source modules) and the number of MDNodes created for the destination module (we do not need to re-create the MDNodes that can be shared). > > The amount of sharing may not be that much but it still exists. > > I had some experiments earlier on building clang with "-flto -g", if we dis-allow sharing between source modules and destination module, the memory footprint for MDNodes will increase by 15%. > > So, in my naive view, we do something like the following: > > 0) load a source module > 1) load another source module > 2) merge the second module into the first > 3) delete the second module > 4) while there are more source modules, goto 1I'll describe how the darwin linker uses the LTO interface. It may be amenable to earlier module deletion. 1) The darwin linker mmap()s each input file. If it is a bitcode file, it calls lto_module_create_from_memory() then lto_module_get_num_symbols() and lto_module_get_symbol_*() to discover what the module provides and needs. 2) After all object files are loaded (which means no undefined symbols are left), the linker then calls: lto_codegen_create() and then in a for-loop calls lto_codegen_add_module() on each module previously loaded. 3) After lto_codegen_compile() has returned, the linker does clean up and deletes each module with lto_module_dispose(). It sounds like the linker could call lto_module_dispose() right after lto_codegen_add_module() to help reduce the memory footprint. That would be a simple linker change. A slightly larger linker change would be to immediately call lto_codegen_add_module() right after lto_module_create_from_memory(), then lto_module_dispose(). That is, never have any unmerged modules laying around. I have no idea is these sort of changes work for the gold plugin. -Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131113/449ee352/attachment.html>
Rafael Espíndola
2013-Nov-14 03:58 UTC
[LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
> It sounds like the linker could call lto_module_dispose() right after > lto_codegen_add_module() to help reduce the memory footprint. That would be > a simple linker change. A slightly larger linker change would be to > immediately call lto_codegen_add_module() right after > lto_module_create_from_memory(), then lto_module_dispose(). That is, never > have any unmerged modules laying around. > > I have no idea is these sort of changes work for the gold plugin.The gold plugin calls lto_codegen_add_module/lto_module_dispose early. So it looks like Chandler's idea would be a win for gold but a loss for ld64 right now. Cheers, Rafael
Manman Ren
2013-Nov-14 06:59 UTC
[LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
On Wed, Nov 13, 2013 at 7:58 PM, Rafael Espíndola < rafael.espindola at gmail.com> wrote:> > It sounds like the linker could call lto_module_dispose() right after > > lto_codegen_add_module() to help reduce the memory footprint. That > would be > > a simple linker change. A slightly larger linker change would be to > > immediately call lto_codegen_add_module() right after > > lto_module_create_from_memory(), then lto_module_dispose(). That is, > never > > have any unmerged modules laying around. > > > > I have no idea is these sort of changes work for the gold plugin. > > The gold plugin calls lto_codegen_add_module/lto_module_dispose early. > So it looks like Chandler's idea would be a win for gold but a loss > for ld64 right now. >Letting the module own MDNodes may not be a win for gold since it is going to create multiple copies of MDNodes that could be shared with Context owning MDNodes. For example, with debug info type uniquing, the type nodes can be shared across modules, but with module owning MDNodes, each module will create its own copy of the type nodes. The advantage is that the MDNodes can be deleted easily by deleting the module. It is not clearly a win to me. Manman> > Cheers, > Rafael >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131113/3ad93341/attachment.html>
Seemingly Similar Threads
- [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
- [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
- [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
- [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
- [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)