llvm dev - Oct 2014 - [LLVMdev] [RFC] Less memory and greater maintainability for debug info IR

> 
> On 2014 Oct 18, at 10:27, Sean Silva <chisophugis at gmail.com>
wrote:
> 
> Derp. My bad. It would be nice in the future if you communicated this
better in the OP. In the OP it sounds like you are doing this solely for memory,
since there is no mention of CPU time or the excessive callback-based RAUW
traffic.
It's clear that you found the OP misleading.  I focused this RFC on what
I thought the debug info maintainers would find most compelling.

FTR, it was there, but I admit I assumed (too much) prior familiarity
with the problem space in order to appreciate its import:
>> By leveraging the use-list
>> infrastructure for metadata operands -- i.e., only using value handles
>> for non-metadata operands -- we'll [...] increase
>> RAUW speed.
[snip]
>> 7. (Optional) Refactor `DebugMDNode` sub-classes to minimize RAUW
>>    traffic during bitcode serialization.  Now that metadata types are
>>    known, we can write debug info out in an order that makes it cheap
>>    to read back in.
>> 
>>    Note that using `MDUser` will make RAUW much cheaper, since
we're
>>    using the use-list infrastructure for most of them.  If RAUW
isn't
>>    showing up in a profile, I may skip this.
> On 2014 Oct 18, at 10:27, Sean Silva <chisophugis at gmail.com>
wrote:
> 
>> Since debug info IR is at the heart of the RAUW bottleneck, I looked
>> into its memory layout (it's a hog).  I started working on PR17891
>> because, besides improving the memory usage, the plan promised to
>> greatly reduce the number of nodes (indirectly reducing RAUW traffic).
>> 
>> In the context of `llvm-lto`, "stage 1" knocked memory usage
down from
>> ~5GB to ~3GB -- but didn't reduce the number of nodes.
> 
> Please put these numbers in context. In the OP you were talking about
15.3GB peak for llvm-lto. Why is ~5GB now the peak? Also in the OP, the
theoretical improvement, out of 15.3GB, was 2GB after stage 4. How are you
getting 2GB improvement out of ~5GB with only stage 1?
I'm talking variously about PR17891 and this proposal.  I suppose that
could be confusing.  "Stage 1" of PR17891 -- have a look at the PR for
context -- yielded 2.2GB reduction in peak memory usage in `llvm-lto`.
After that change, we're at 15.3GB peak in `llvm-lto`.

A conservative estimate of the allocated memory for debug info metadata,
based on counting live nodes and operands (post-change), is ~3GB.  Given
that "stage 1" of PR17891 dropped peak memory usage by 2.2GB, I assume
that the original cost was ~5GB.  This proposal drops the conservative
estimate by a further ~2GB to ~1GB.
>> As Bob suggested, please feel free to join the party!  Less work for me
>> to do later.
> 
> I'm planning on it.
Great!

>
> >> By leveraging the use-list
> >> infrastructure for metadata operands -- i.e., only using value
handles
> >> for non-metadata operands -- we'll [...] increase
> >> RAUW speed.

This is something I know next to nothing about - why is it that use-lists
would be applicable to metadata-to-metadata operands, but not for
non-metadata operands?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141019/f413ca1d/attachment.html>

> On Oct 19, 2014, at 7:00 PM, David Blaikie <dblaikie at gmail.com>
wrote:
> 
> >> By leveraging the use-list
> >> infrastructure for metadata operands -- i.e., only using value
handles
> >> for non-metadata operands -- we'll [...] increase
> >> RAUW speed.
> 
> This is something I know next to nothing about - why is it that use-lists
would be applicable to metadata-to-metadata operands, but not for non-metadata
operands?
Metadata isn't allowed to add uses to "regular" IR, since then it
would
influence optimizations.  Instead, it's required to refer to them via
value handles.

However, optimizations won't be affected by adding uses to metadata IR.