llvm dev - Feb 2016 - Possible Memory Savings for tools emitting large amounts of existing data through MC

Just in case it interests anyone else, I'm playing around with trying to
broaden the MCStreamer API to allow for emission of bytes without copying
the contents into a local buffer first (either because you already have a
buffer, or the bytes are already present in another file, etc) in
http://reviews.llvm.org/D17694 . In theory there's some overlap with lld
here (no doubt it already does this sort of thing, but not in a way, I
assume, we could reuse from other tools at the moment) and my motivation,
llvm-dwp, looks very much like "linking with a few extra steps".

But to check that these changes might be more generally applicable, I
thought I'd solicit data from anyone building tools that might be memory
constrained as well.

First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.

Adrian/Fred - do you guys ever have trouble with memory usage of
llvm-dsymutil? Do you have an example you could provide that has high
memory usage, so I could see if any simple changes based on my prototype MC
changes would help.

A quick glance at dsymutil's code indicates it might benefit slightly, at
least - in the string table emission, for example (it looks very similar to
string table emission in dwp - just being able to reference the strings in
the StringMap rather than copying them into MCStreamer could help (also I
found using a DenseMap<StringRef to the memory mapped input helped as well
- but that's a change you can make locally without any MCStreamer
improvements) - other parts might be trickier, and consist of parts of
referencable data (like the line table header)  and parts that are not
referencable (like their contents) - my prototype could be extended to
handle that)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/91164401/attachment.html>

Hi David,

The way I imagined that we might want to extend the MCStreamer API (this
was motivated by DIEData) is by allowing clients to move bytes and fixups
into the MC layer.

This is the sort of API that I was imagining:

void MoveAndEmitFragment(SmallVectorImpl<char> &&Data,
                         SmallVectorImpl<MCFixup> &&Fixups);

Note that this mirrors the fields
MCEncodedFragmentWithContents::Contents and MCEncodedFragmentWithFixups::Fixups
and the arguments could be directly moved into the fields of a newly created
MCDataFragment.

Would that work for your use case?

Peter

On Mon, Feb 29, 2016 at 03:18:22PM -0800, David Blaikie via llvm-dev
wrote:
> Just in case it interests anyone else, I'm playing around with trying
to
> broaden the MCStreamer API to allow for emission of bytes without copying
> the contents into a local buffer first (either because you already have a
> buffer, or the bytes are already present in another file, etc) in
> http://reviews.llvm.org/D17694 . In theory there's some overlap with
lld
> here (no doubt it already does this sort of thing, but not in a way, I
> assume, we could reuse from other tools at the moment) and my motivation,
> llvm-dwp, looks very much like "linking with a few extra steps".
> 
> But to check that these changes might be more generally applicable, I
> thought I'd solicit data from anyone building tools that might be
memory
> constrained as well.
> 
> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
> 
> Adrian/Fred - do you guys ever have trouble with memory usage of
> llvm-dsymutil? Do you have an example you could provide that has high
> memory usage, so I could see if any simple changes based on my prototype MC
> changes would help.
> 
> A quick glance at dsymutil's code indicates it might benefit slightly,
at
> least - in the string table emission, for example (it looks very similar to
> string table emission in dwp - just being able to reference the strings in
> the StringMap rather than copying them into MCStreamer could help (also I
> found using a DenseMap<StringRef to the memory mapped input helped as
well
> - but that's a change you can make locally without any MCStreamer
> improvements) - other parts might be trickier, and consist of parts of
> referencable data (like the line table header)  and parts that are not
> referencable (like their contents) - my prototype could be extended to
> handle that)
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 
Peter

> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com>
wrote:
> 
> Just in case it interests anyone else, I'm playing around with trying
to broaden the MCStreamer API to allow for emission of bytes without copying the
contents into a local buffer first (either because you already have a buffer, or
the bytes are already present in another file, etc) in
http://reviews.llvm.org/D17694 <http://reviews.llvm.org/D17694> . In
theory there's some overlap with lld here (no doubt it already does this
sort of thing, but not in a way, I assume, we could reuse from other tools at
the moment) and my motivation, llvm-dwp, looks very much like "linking with
a few extra steps".
> 
> But to check that these changes might be more generally applicable, I
thought I'd solicit data from anyone building tools that might be memory
constrained as well.
> 
> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
> 
> Adrian/Fred - do you guys ever have trouble with memory usage of
llvm-dsymutil? Do you have an example you could provide that has high memory
usage, so I could see if any simple changes based on my prototype MC changes
would help.
Since dsymutil processes object files one after another, memory usage wasn’t
really a problem so far, but you could try running llvm-dsymutil on bin/clang
for a larger example (takes about a minute to finish).
> 
> A quick glance at dsymutil's code indicates it might benefit slightly,
at least - in the string table emission, for example (it looks very similar to
string table emission in dwp - just being able to reference the strings in the
StringMap rather than copying them into MCStreamer could help (also I found
using a DenseMap<StringRef to the memory mapped input helped as well - but
that's a change you can make locally without any MCStreamer improvements) -
other parts might be trickier, and consist of parts of referencable data (like
the line table header)  and parts that are not referencable (like their
contents) - my prototype could be extended to handle that)
-- adrian
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/b7d688ed/attachment.html>

On Mon, Feb 29, 2016 at 3:36 PM, Adrian Prantl <aprantl at apple.com>
wrote:
>
> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com>
wrote:
>
> Just in case it interests anyone else, I'm playing around with trying
to
> broaden the MCStreamer API to allow for emission of bytes without copying
> the contents into a local buffer first (either because you already have a
> buffer, or the bytes are already present in another file, etc) in
> http://reviews.llvm.org/D17694 . In theory there's some overlap with
lld
> here (no doubt it already does this sort of thing, but not in a way, I
> assume, we could reuse from other tools at the moment) and my motivation,
> llvm-dwp, looks very much like "linking with a few extra steps".
>
> But to check that these changes might be more generally applicable, I
> thought I'd solicit data from anyone building tools that might be
memory
> constrained as well.
>
> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
>
> Adrian/Fred - do you guys ever have trouble with memory usage of
> llvm-dsymutil? Do you have an example you could provide that has high
> memory usage, so I could see if any simple changes based on my prototype MC
> changes would help.
>
>
> Since dsymutil processes object files one after another,
>
As does llvm-dwp. Think of llvm-dwp more like a linker with a few extra
bits. But the MCStreamer API means any bytes you write to the streamer stay
in memory until you "Finish" - so if you're dwp/linking large
enough
inputs, you have them all in memory when you really don't need them. For
example, the dwp file I was generating is 7GB, but the tool with the memory
improvements only has a high water mark of 2.3GB.

> memory usage wasn’t really a problem so far, but you could try running
> llvm-dsymutil on bin/clang for a larger example (takes about a minute to
> finish).
>
Was thinking of something more accessible to me, on a non-Darwin platform.
Is there a way I can generate the dsym inputs across Clang on a non-Darwin
platform? (what happens if I run dsymutil on my ELF object files?)
>
> A quick glance at dsymutil's code indicates it might benefit slightly,
at
> least - in the string table emission, for example (it looks very similar to
> string table emission in dwp - just being able to reference the strings in
> the StringMap rather than copying them into MCStreamer could help (also I
> found using a DenseMap<StringRef to the memory mapped input helped as
well
> - but that's a change you can make locally without any MCStreamer
> improvements) - other parts might be trickier, and consist of parts of
> referencable data (like the line table header)  and parts that are not
> referencable (like their contents) - my prototype could be extended to
> handle that)
>
>
> -- adrian
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/f51d10a6/attachment.html>

On Mon, Feb 29, 2016 at 3:36 PM, Peter Collingbourne <peter at pcc.me.uk>
wrote:
> Hi David,
>
> The way I imagined that we might want to extend the MCStreamer API (this
> was motivated by DIEData) is by allowing clients to move bytes and fixups
> into the MC layer.
>
> This is the sort of API that I was imagining:
>
> void MoveAndEmitFragment(SmallVectorImpl<char> &&Data,
>                          SmallVectorImpl<MCFixup> &&Fixups);
>
> Note that this mirrors the fields
> MCEncodedFragmentWithContents::Contents and
> MCEncodedFragmentWithFixups::Fixups
> and the arguments could be directly moved into the fields of a newly
> created
> MCDataFragment.
>
> Would that work for your use case?
>
Not quite, unfortunately - the issue is that we're doing a task that is
essentially "linking + a bit" - so imagine linking a bunch of files
together, the final, say, debug_info.dwo section is made up of the
concatenation of all the debug_info.dwo sections of the inputs. So it's
fragmented and it's already available, memory mapped, never in a
SmallVector, etc.

>
> Peter
>
> On Mon, Feb 29, 2016 at 03:18:22PM -0800, David Blaikie via llvm-dev wrote:
> > Just in case it interests anyone else, I'm playing around with
trying to
> > broaden the MCStreamer API to allow for emission of bytes without
copying
> > the contents into a local buffer first (either because you already
have a
> > buffer, or the bytes are already present in another file, etc) in
> > http://reviews.llvm.org/D17694 . In theory there's some overlap
with lld
> > here (no doubt it already does this sort of thing, but not in a way, I
> > assume, we could reuse from other tools at the moment) and my
motivation,
> > llvm-dwp, looks very much like "linking with a few extra
steps".
> >
> > But to check that these changes might be more generally applicable, I
> > thought I'd solicit data from anyone building tools that might be
memory
> > constrained as well.
> >
> > First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
> >
> > Adrian/Fred - do you guys ever have trouble with memory usage of
> > llvm-dsymutil? Do you have an example you could provide that has high
> > memory usage, so I could see if any simple changes based on my
prototype
> MC
> > changes would help.
> >
> > A quick glance at dsymutil's code indicates it might benefit
slightly, at
> > least - in the string table emission, for example (it looks very
similar
> to
> > string table emission in dwp - just being able to reference the
strings
> in
> > the StringMap rather than copying them into MCStreamer could help
(also I
> > found using a DenseMap<StringRef to the memory mapped input helped
as
> well
> > - but that's a change you can make locally without any MCStreamer
> > improvements) - other parts might be trickier, and consist of parts of
> > referencable data (like the line table header)  and parts that are not
> > referencable (like their contents) - my prototype could be extended to
> > handle that)
>
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> --
> Peter
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/b9689273/attachment.html>

> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com>
wrote:
> 
> Just in case it interests anyone else, I'm playing around with trying
to broaden the MCStreamer API to allow for emission of bytes without copying the
contents into a local buffer first (either because you already have a buffer, or
the bytes are already present in another file, etc) in
http://reviews.llvm.org/D17694 <http://reviews.llvm.org/D17694> . In
theory there's some overlap with lld here (no doubt it already does this
sort of thing, but not in a way, I assume, we could reuse from other tools at
the moment) and my motivation, llvm-dwp, looks very much like "linking with
a few extra steps".
> 
> But to check that these changes might be more generally applicable, I
thought I'd solicit data from anyone building tools that might be memory
constrained as well.
> 
> First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil.
> 
> Adrian/Fred - do you guys ever have trouble with memory usage of
llvm-dsymutil? Do you have an example you could provide that has high memory
usage, so I could see if any simple changes based on my prototype MC changes
would help.
> 
> A quick glance at dsymutil's code indicates it might benefit slightly,
at least - in the string table emission, for example (it looks very similar to
string table emission in dwp - just being able to reference the strings in the
StringMap rather than copying them into MCStreamer could help (also I found
using a DenseMap<StringRef to the memory mapped input helped as well - but
that's a change you can make locally without any MCStreamer improvements) -
other parts might be trickier, and consist of parts of referencable data (like
the line table header)  and parts that are not referencable (like their
contents) - my prototype could be extended to handle that)
Yes, I agree dsymutil could see slight gains from the parts you mention. It
wouldn’t be groundbreaking though, the biggest contributor would always be the
DIE tree that we recreate completely.

Fred
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/638c8626/attachment.html>