llvm dev - Mar 2016 - Possible Memory Savings for tools emitting large amounts of existing data through MC

On Mon, Feb 29, 2016 at 3:51 PM, Adrian Prantl <aprantl at apple.com>
wrote:
>
> On Feb 29, 2016, at 3:46 PM, David Blaikie <dblaikie at gmail.com>
wrote:
>
>
>
> On Mon, Feb 29, 2016 at 3:36 PM, Adrian Prantl <aprantl at apple.com>
wrote:
>
>>
>> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at
gmail.com> wrote:
>>
>> Just in case it interests anyone else, I'm playing around with
trying to
>> broaden the MCStreamer API to allow for emission of bytes without
copying
>> the contents into a local buffer first (either because you already have
a
>> buffer, or the bytes are already present in another file, etc) in
>> http://reviews.llvm.org/D17694 . In theory there's some overlap
with lld
>> here (no doubt it already does this sort of thing, but not in a way, I
>> assume, we could reuse from other tools at the moment) and my
motivation,
>> llvm-dwp, looks very much like "linking with a few extra
steps".
>>
>> But to check that these changes might be more generally applicable, I
>> thought I'd solicit data from anyone building tools that might be
memory
>> constrained as well.
>>
>> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
>>
>> Adrian/Fred - do you guys ever have trouble with memory usage of
>> llvm-dsymutil? Do you have an example you could provide that has high
>> memory usage, so I could see if any simple changes based on my
prototype MC
>> changes would help.
>>
>>
>> Since dsymutil processes object files one after another,
>>
>
> As does llvm-dwp. Think of llvm-dwp more like a linker with a few extra
> bits. But the MCStreamer API means any bytes you write to the streamer stay
> in memory until you "Finish" - so if you're dwp/linking large
enough
> inputs, you have them all in memory when you really don't need them.
For
> example, the dwp file I was generating is 7GB, but the tool with the memory
> improvements only has a high water mark of 2.3GB.
>
>
>> memory usage wasn’t really a problem so far, but you could try running
>> llvm-dsymutil on bin/clang for a larger example (takes about a minute
to
>> finish).
>>
>
> Was thinking of something more accessible to me, on a non-Darwin platform.
> Is there a way I can generate the dsym inputs across Clang on a non-Darwin
> platform? (what happens if I run dsymutil on my ELF object files?)
>
>
> At this point probably nothing. Dsymutil acts on STABS symbol table
> entries that are (I guess) not present in a typical ELF binary. Dsymutil
> also only implements MachO relocations and has lots of other things where
> the ELF implementation is missing. It’s probably not too much work to wire
> all this up, but so far nobody did it.
>
& no easy way for me to get a representative (or pathalogically large,
even) set of machO files to play with, I take it? It's no worries - just
figured I'd give it a go if it was convenient.

>
> -- adrian
>
>
>> A quick glance at dsymutil's code indicates it might benefit
slightly, at
>> least - in the string table emission, for example (it looks very
similar to
>> string table emission in dwp - just being able to reference the strings
in
>> the StringMap rather than copying them into MCStreamer could help (also
I
>> found using a DenseMap<StringRef to the memory mapped input helped
as well
>> - but that's a change you can make locally without any MCStreamer
>> improvements) - other parts might be trickier, and consist of parts of
>> referencable data (like the line table header)  and parts that are not
>> referencable (like their contents) - my prototype could be extended to
>> handle that)
>>
>>
>> -- adrian
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/38484a1c/attachment.html>

> On Feb 29, 2016, at 4:10 PM, David Blaikie <dblaikie at gmail.com>
wrote:
> 
> 
> 
> On Mon, Feb 29, 2016 at 3:51 PM, Adrian Prantl <aprantl at apple.com
<mailto:aprantl at apple.com>> wrote:
> 
>> On Feb 29, 2016, at 3:46 PM, David Blaikie <dblaikie at gmail.com
<mailto:dblaikie at gmail.com>> wrote:
>> 
>> 
>> 
>> On Mon, Feb 29, 2016 at 3:36 PM, Adrian Prantl <aprantl at apple.com
<mailto:aprantl at apple.com>> wrote:
>> 
>>> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at
gmail.com <mailto:dblaikie at gmail.com>> wrote:
>>> 
>>> Just in case it interests anyone else, I'm playing around with
trying to broaden the MCStreamer API to allow for emission of bytes without
copying the contents into a local buffer first (either because you already have
a buffer, or the bytes are already present in another file, etc) in
http://reviews.llvm.org/D17694 <http://reviews.llvm.org/D17694> . In
theory there's some overlap with lld here (no doubt it already does this
sort of thing, but not in a way, I assume, we could reuse from other tools at
the moment) and my motivation, llvm-dwp, looks very much like "linking with
a few extra steps".
>>> 
>>> But to check that these changes might be more generally applicable,
I thought I'd solicit data from anyone building tools that might be memory
constrained as well.
>>> 
>>> First that comes to mind (Eric suggested/mentioned) is
llvm-dsymutil.
>>> 
>>> Adrian/Fred - do you guys ever have trouble with memory usage of
llvm-dsymutil? Do you have an example you could provide that has high memory
usage, so I could see if any simple changes based on my prototype MC changes
would help.
>> 
>> Since dsymutil processes object files one after another,
>> 
>> As does llvm-dwp. Think of llvm-dwp more like a linker with a few extra
bits. But the MCStreamer API means any bytes you write to the streamer stay in
memory until you "Finish" - so if you're dwp/linking large enough
inputs, you have them all in memory when you really don't need them. For
example, the dwp file I was generating is 7GB, but the tool with the memory
improvements only has a high water mark of 2.3GB.
>>  
>> memory usage wasn’t really a problem so far, but you could try running
llvm-dsymutil on bin/clang for a larger example (takes about a minute to
finish).
>> 
>> Was thinking of something more accessible to me, on a non-Darwin
platform. Is there a way I can generate the dsym inputs across Clang on a
non-Darwin platform? (what happens if I run dsymutil on my ELF object files?)
> 
> At this point probably nothing. Dsymutil acts on STABS symbol table entries
that are (I guess) not present in a typical ELF binary. Dsymutil also only
implements MachO relocations and has lots of other things where the ELF
implementation is missing. It’s probably not too much work to wire all this up,
but so far nobody did it.
> 
> & no easy way for me to get a representative (or pathalogically large,
even) set of machO files to play with, I take it? It's no worries - just
figured I'd give it a go if it was convenient.
I can definitely go and grab you a clang build directory from one of the green
dragon bots for example; but all the paths are hardcoded so you’d have to
install them in the exact same location. In theory everything doing file access
should be handled by the LLVM low-level libraries, so this *could* work.

-- adrian
>  
> 
> -- adrian
> 
>>> 
>>> A quick glance at dsymutil's code indicates it might benefit
slightly, at least - in the string table emission, for example (it looks very
similar to string table emission in dwp - just being able to reference the
strings in the StringMap rather than copying them into MCStreamer could help
(also I found using a DenseMap<StringRef to the memory mapped input helped as
well - but that's a change you can make locally without any MCStreamer
improvements) - other parts might be trickier, and consist of parts of
referencable data (like the line table header)  and parts that are not
referencable (like their contents) - my prototype could be extended to handle
that)
>> 
>> -- adrian
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/49981cb2/attachment.html>

On Mon, Feb 29, 2016 at 4:17 PM, Adrian Prantl <aprantl at apple.com>
wrote:
>
> On Feb 29, 2016, at 4:10 PM, David Blaikie <dblaikie at gmail.com>
wrote:
>
>
>
> On Mon, Feb 29, 2016 at 3:51 PM, Adrian Prantl <aprantl at apple.com>
wrote:
>
>>
>> On Feb 29, 2016, at 3:46 PM, David Blaikie <dblaikie at
gmail.com> wrote:
>>
>>
>>
>> On Mon, Feb 29, 2016 at 3:36 PM, Adrian Prantl <aprantl at
apple.com> wrote:
>>
>>>
>>> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at
gmail.com> wrote:
>>>
>>> Just in case it interests anyone else, I'm playing around with
trying to
>>> broaden the MCStreamer API to allow for emission of bytes without
copying
>>> the contents into a local buffer first (either because you already
have a
>>> buffer, or the bytes are already present in another file, etc) in
>>> http://reviews.llvm.org/D17694 . In theory there's some overlap
with
>>> lld here (no doubt it already does this sort of thing, but not in a
way, I
>>> assume, we could reuse from other tools at the moment) and my
motivation,
>>> llvm-dwp, looks very much like "linking with a few extra
steps".
>>>
>>> But to check that these changes might be more generally applicable,
I
>>> thought I'd solicit data from anyone building tools that might
be memory
>>> constrained as well.
>>>
>>> First that comes to mind (Eric suggested/mentioned) is
llvm-dsymutil.
>>>
>>> Adrian/Fred - do you guys ever have trouble with memory usage of
>>> llvm-dsymutil? Do you have an example you could provide that has
high
>>> memory usage, so I could see if any simple changes based on my
prototype MC
>>> changes would help.
>>>
>>>
>>> Since dsymutil processes object files one after another,
>>>
>>
>> As does llvm-dwp. Think of llvm-dwp more like a linker with a few extra
>> bits. But the MCStreamer API means any bytes you write to the streamer
stay
>> in memory until you "Finish" - so if you're dwp/linking
large enough
>> inputs, you have them all in memory when you really don't need
them. For
>> example, the dwp file I was generating is 7GB, but the tool with the
memory
>> improvements only has a high water mark of 2.3GB.
>>
>>
>>> memory usage wasn’t really a problem so far, but you could try
running
>>> llvm-dsymutil on bin/clang for a larger example (takes about a
minute to
>>> finish).
>>>
>>
>> Was thinking of something more accessible to me, on a non-Darwin
>> platform. Is there a way I can generate the dsym inputs across Clang on
a
>> non-Darwin platform? (what happens if I run dsymutil on my ELF object
>> files?)
>>
>>
>> At this point probably nothing. Dsymutil acts on STABS symbol table
>> entries that are (I guess) not present in a typical ELF binary.
Dsymutil
>> also only implements MachO relocations and has lots of other things
where
>> the ELF implementation is missing. It’s probably not too much work to
wire
>> all this up, but so far nobody did it.
>>
>
> & no easy way for me to get a representative (or pathalogically large,
> even) set of machO files to play with, I take it? It's no worries -
just
> figured I'd give it a go if it was convenient.
>
>
> I can definitely go and grab you a clang build directory from one of the
> green dragon bots for example; but all the paths are hardcoded so you’d
> have to install them in the exact same location. In theory everything doing
> file access should be handled by the LLVM low-level libraries, so this
> *could* work.
>
If you like/have time, feel free to throw them up somewhere I can download
them from.


>
> -- adrian
>
>
>
>>
>> -- adrian
>>
>>
>>> A quick glance at dsymutil's code indicates it might benefit
slightly,
>>> at least - in the string table emission, for example (it looks very
similar
>>> to string table emission in dwp - just being able to reference the
strings
>>> in the StringMap rather than copying them into MCStreamer could
help (also
>>> I found using a DenseMap<StringRef to the memory mapped input
helped as
>>> well - but that's a change you can make locally without any
MCStreamer
>>> improvements) - other parts might be trickier, and consist of parts
of
>>> referencable data (like the line table header)  and parts that are
not
>>> referencable (like their contents) - my prototype could be extended
to
>>> handle that)
>>>
>>>
>>> -- adrian
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/c3ab404a/attachment.html>