Manman Ren
2013-Nov-14 06:59 UTC
[LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
On Wed, Nov 13, 2013 at 7:58 PM, Rafael Espíndola < rafael.espindola at gmail.com> wrote:> > It sounds like the linker could call lto_module_dispose() right after > > lto_codegen_add_module() to help reduce the memory footprint. That > would be > > a simple linker change. A slightly larger linker change would be to > > immediately call lto_codegen_add_module() right after > > lto_module_create_from_memory(), then lto_module_dispose(). That is, > never > > have any unmerged modules laying around. > > > > I have no idea is these sort of changes work for the gold plugin. > > The gold plugin calls lto_codegen_add_module/lto_module_dispose early. > So it looks like Chandler's idea would be a win for gold but a loss > for ld64 right now. >Letting the module own MDNodes may not be a win for gold since it is going to create multiple copies of MDNodes that could be shared with Context owning MDNodes. For example, with debug info type uniquing, the type nodes can be shared across modules, but with module owning MDNodes, each module will create its own copy of the type nodes. The advantage is that the MDNodes can be deleted easily by deleting the module. It is not clearly a win to me. Manman> > Cheers, > Rafael >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131113/3ad93341/attachment.html>
Rafael Espíndola
2013-Nov-14 13:19 UTC
[LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
> Letting the module own MDNodes may not be a win for gold since it is going > to create multiple copies of MDNodes that could be shared with Context > owning MDNodes. > > For example, with debug info type uniquing, the type nodes can be shared > across modules, but with module owning MDNodes, each module will create its > own copy of the type nodes. The advantage is that the MDNodes can be deleted > easily by deleting the module. It is not clearly a win to me.But gold has at most 2 objects loaded at any time. Cheers, Rafael
Eric Christopher
2013-Nov-14 18:11 UTC
[LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
On Thu, Nov 14, 2013 at 5:19 AM, Rafael Espíndola <rafael.espindola at gmail.com> wrote:>> Letting the module own MDNodes may not be a win for gold since it is going >> to create multiple copies of MDNodes that could be shared with Context >> owning MDNodes. >> >> For example, with debug info type uniquing, the type nodes can be shared >> across modules, but with module owning MDNodes, each module will create its >> own copy of the type nodes. The advantage is that the MDNodes can be deleted >> easily by deleting the module. It is not clearly a win to me. > > But gold has at most 2 objects loaded at any time. >Agreed. Dave and I were chatting about this some and from my perspective the only disadvantage to this is the time to copy metadata from one place to the other. Peak memory usage will still go down since you could then free up each module as you added. The "uniquing" works basically the same way as adding the same node 400 times to the folding set - it just returns the same one each time. I guess I'm confused at where this 40% overhead of MDNodes is coming from though? Do we know what they are? -eric
Manman Ren
2013-Nov-14 18:14 UTC
[LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
On Thu, Nov 14, 2013 at 5:19 AM, Rafael Espíndola < rafael.espindola at gmail.com> wrote:> > Letting the module own MDNodes may not be a win for gold since it is > going > > to create multiple copies of MDNodes that could be shared with Context > > owning MDNodes. > > > > For example, with debug info type uniquing, the type nodes can be shared > > across modules, but with module owning MDNodes, each module will create > its > > own copy of the type nodes. The advantage is that the MDNodes can be > deleted > > easily by deleting the module. It is not clearly a win to me. > > But gold has at most 2 objects loaded at any time. >The problem is not how many objects co-exist, the problem is how many copies of the type nodes we create. We will create a new copy of the type nodes each time we load in a source module, even though the type nodes exist in the destination module. So if we have 700 source modules, assuming a type is shared across all 700 modules, the type nodes will be created 701 times. Cheers, Manman> > Cheers, > Rafael >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/2ee78d8a/attachment.html>
Stephen Checkoway
2013-Nov-14 23:01 UTC
[LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
On Nov 14, 2013, at 8:19 AM, Rafael Espíndola <rafael.espindola at gmail.com> wrote:> But gold has at most 2 objects loaded at any time.Are you sure about that? I haven't looked into it but while building Chromium with LTO, I get: ../../third_party/gold/gold64: fatal error: out of file descriptors and couldn't close any clang: error: linker command failed with exit code 1 (use -v to see invocation) ninja: build stopped: subcommand failed. [secdev:~/chromium/src/out/Release] (master) s$ ulimit -Sn 10000 Looking in /proc, it has 10013 open file descriptors. With only 2 objects loaded at a time, I'd expect many fewer file descriptors to be open. Maybe it only has 2 objects in memory at once but keeps all the file descriptors open? -- Stephen Checkoway
Reasonably Related Threads
- [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
- [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
- [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
- [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)
- [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)