llvm dev - Nov 2013 - [LLVMdev] Proposal: release MDNodes for source modules (LTO+debug info)

Hi All,

In LTO, we load in the source modules and link the source modules into a
destination module.
Lots of MDNodes are only used by the source modules, for example Xalan used
649MB for MDNodes after loading and linking, but the actual destination
module only has 393MB of MDNodes. There are 649-393MB (40% of 649MB) not
used.

MDNodes belong to the Context, deleting modules will not release the
MDNodes.

One possible solution is:

In LLVMContext, add “removeUnusedMDNodes" function
  It goes through OwnedModules and check if a MDNode is used by any of the
modules, if not remove it.
  One implementation is to mark a visited MDNode used when traversing the
module. After done traversing all modules, we can delete MDNodes in
MDNodeSet that are not marked.

In LTOCodeGenerator, add a vector of source modules that are added (these
source modules will be linked with DestroySource mode).
In LTOCodeGenerator:: compile_to_file, delete all source modules that are
linked in, then call LLVMContext::removeUnusedMDNodes
—> I can’t find a better place to call the function. When we
call compile_to_file, we should have done linking in all source modules.
Another possibility is to add a lto API so the linker can delete the source
modules and call the API to release MDNodes.

Other options are:
1> Using a different LLVMContext for the destination module, but it didn’t
work out since Linker was not designed to work with different LLVMContexts
for source vs destination.
2> removeUnusedMDNodes checks if a MDNode is used in a different way  (i.e
use_empty() && !hasValueHandler()), but it does not remove MDNodes that
form cycles.

Comments and suggestions are welcome.

Thanks,
Manman
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131112/d1714eca/attachment.html>

On Tue, Nov 12, 2013 at 4:29 PM, Manman Ren <manman.ren at gmail.com>
wrote:
> Hi All,
>
> In LTO, we load in the source modules and link the source modules into a
> destination module.
> Lots of MDNodes are only used by the source modules, for example Xalan
> used 649MB for MDNodes after loading and linking, but the actual
> destination module only has 393MB of MDNodes. There are 649-393MB (40% of
> 649MB) not used.
>
> MDNodes belong to the Context, deleting modules will not release the
> MDNodes.
>
> One possible solution is:
>
> In LLVMContext, add “removeUnusedMDNodes" function
>   It goes through OwnedModules and check if a MDNode is used by any of the
> modules, if not remove it.
>   One implementation is to mark a visited MDNode used when traversing the
> module. After done traversing all modules, we can delete MDNodes in
> MDNodeSet that are not marked.
>
> In LTOCodeGenerator, add a vector of source modules that are added (these
> source modules will be linked with DestroySource mode).
> In LTOCodeGenerator:: compile_to_file, delete all source modules that are
> linked in, then call LLVMContext::removeUnusedMDNodes
> —> I can’t find a better place to call the function. When we
> call compile_to_file, we should have done linking in all source modules.
> Another possibility is to add a lto API so the linker can delete the
> source modules and call the API to release MDNodes.
>
> Other options are:
> 1> Using a different LLVMContext for the destination module, but it
didn’t
> work out since Linker was not designed to work with different LLVMContexts
> for source vs destination.
> 2> removeUnusedMDNodes checks if a MDNode is used in a different way 
(i.e
> use_empty() && !hasValueHandler()), but it does not remove MDNodes
that
> form cycles.
>
3) Make the MDNode be owned by the module that uses it?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131112/22442859/attachment.html>

On Tue, Nov 12, 2013 at 4:38 PM, Chandler Carruth <chandlerc at
google.com>wrote:
>
> On Tue, Nov 12, 2013 at 4:29 PM, Manman Ren <manman.ren at gmail.com>
wrote:
>
>> Hi All,
>>
>>  In LTO, we load in the source modules and link the source modules into
a
>> destination module.
>> Lots of MDNodes are only used by the source modules, for example Xalan
>> used 649MB for MDNodes after loading and linking, but the actual
>> destination module only has 393MB of MDNodes. There are 649-393MB (40%
of
>> 649MB) not used.
>>
>> MDNodes belong to the Context, deleting modules will not release the
>> MDNodes.
>>
>> One possible solution is:
>>
>> In LLVMContext, add “removeUnusedMDNodes" function
>>   It goes through OwnedModules and check if a MDNode is used by any of
>> the modules, if not remove it.
>>   One implementation is to mark a visited MDNode used when traversing
the
>> module. After done traversing all modules, we can delete MDNodes in
>> MDNodeSet that are not marked.
>>
>> In LTOCodeGenerator, add a vector of source modules that are added
(these
>> source modules will be linked with DestroySource mode).
>> In LTOCodeGenerator:: compile_to_file, delete all source modules that
are
>> linked in, then call LLVMContext::removeUnusedMDNodes
>> —> I can’t find a better place to call the function. When we
>> call compile_to_file, we should have done linking in all source
modules.
>> Another possibility is to add a lto API so the linker can delete the
>> source modules and call the API to release MDNodes.
>>
>> Other options are:
>> 1> Using a different LLVMContext for the destination module, but it
>> didn’t work out since Linker was not designed to work with different
>> LLVMContexts for source vs destination.
>> 2> removeUnusedMDNodes checks if a MDNode is used in a different way
>>  (i.e use_empty() && !hasValueHandler()), but it does not
remove MDNodes
>> that form cycles.
>>
>
> 3) Make the MDNode be owned by the module that uses it?
>
MDNode is shared among modules so multiple modules can use it, if we
specify an owner for a MDNode, that will prevent sharing.

Manman
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131112/01bce254/attachment.html>