David Blaikie via llvm-dev
2016-Feb-29 23:18 UTC
[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC
Just in case it interests anyone else, I'm playing around with trying to broaden the MCStreamer API to allow for emission of bytes without copying the contents into a local buffer first (either because you already have a buffer, or the bytes are already present in another file, etc) in http://reviews.llvm.org/D17694 . In theory there's some overlap with lld here (no doubt it already does this sort of thing, but not in a way, I assume, we could reuse from other tools at the moment) and my motivation, llvm-dwp, looks very much like "linking with a few extra steps". But to check that these changes might be more generally applicable, I thought I'd solicit data from anyone building tools that might be memory constrained as well. First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil. Adrian/Fred - do you guys ever have trouble with memory usage of llvm-dsymutil? Do you have an example you could provide that has high memory usage, so I could see if any simple changes based on my prototype MC changes would help. A quick glance at dsymutil's code indicates it might benefit slightly, at least - in the string table emission, for example (it looks very similar to string table emission in dwp - just being able to reference the strings in the StringMap rather than copying them into MCStreamer could help (also I found using a DenseMap<StringRef to the memory mapped input helped as well - but that's a change you can make locally without any MCStreamer improvements) - other parts might be trickier, and consist of parts of referencable data (like the line table header) and parts that are not referencable (like their contents) - my prototype could be extended to handle that) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/91164401/attachment.html>
Peter Collingbourne via llvm-dev
2016-Feb-29 23:36 UTC
[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC
Hi David, The way I imagined that we might want to extend the MCStreamer API (this was motivated by DIEData) is by allowing clients to move bytes and fixups into the MC layer. This is the sort of API that I was imagining: void MoveAndEmitFragment(SmallVectorImpl<char> &&Data, SmallVectorImpl<MCFixup> &&Fixups); Note that this mirrors the fields MCEncodedFragmentWithContents::Contents and MCEncodedFragmentWithFixups::Fixups and the arguments could be directly moved into the fields of a newly created MCDataFragment. Would that work for your use case? Peter On Mon, Feb 29, 2016 at 03:18:22PM -0800, David Blaikie via llvm-dev wrote:> Just in case it interests anyone else, I'm playing around with trying to > broaden the MCStreamer API to allow for emission of bytes without copying > the contents into a local buffer first (either because you already have a > buffer, or the bytes are already present in another file, etc) in > http://reviews.llvm.org/D17694 . In theory there's some overlap with lld > here (no doubt it already does this sort of thing, but not in a way, I > assume, we could reuse from other tools at the moment) and my motivation, > llvm-dwp, looks very much like "linking with a few extra steps". > > But to check that these changes might be more generally applicable, I > thought I'd solicit data from anyone building tools that might be memory > constrained as well. > > First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil. > > Adrian/Fred - do you guys ever have trouble with memory usage of > llvm-dsymutil? Do you have an example you could provide that has high > memory usage, so I could see if any simple changes based on my prototype MC > changes would help. > > A quick glance at dsymutil's code indicates it might benefit slightly, at > least - in the string table emission, for example (it looks very similar to > string table emission in dwp - just being able to reference the strings in > the StringMap rather than copying them into MCStreamer could help (also I > found using a DenseMap<StringRef to the memory mapped input helped as well > - but that's a change you can make locally without any MCStreamer > improvements) - other parts might be trickier, and consist of parts of > referencable data (like the line table header) and parts that are not > referencable (like their contents) - my prototype could be extended to > handle that)> _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- Peter
Adrian Prantl via llvm-dev
2016-Feb-29 23:36 UTC
[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC
> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com> wrote: > > Just in case it interests anyone else, I'm playing around with trying to broaden the MCStreamer API to allow for emission of bytes without copying the contents into a local buffer first (either because you already have a buffer, or the bytes are already present in another file, etc) in http://reviews.llvm.org/D17694 <http://reviews.llvm.org/D17694> . In theory there's some overlap with lld here (no doubt it already does this sort of thing, but not in a way, I assume, we could reuse from other tools at the moment) and my motivation, llvm-dwp, looks very much like "linking with a few extra steps". > > But to check that these changes might be more generally applicable, I thought I'd solicit data from anyone building tools that might be memory constrained as well. > > First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil. > > Adrian/Fred - do you guys ever have trouble with memory usage of llvm-dsymutil? Do you have an example you could provide that has high memory usage, so I could see if any simple changes based on my prototype MC changes would help.Since dsymutil processes object files one after another, memory usage wasn’t really a problem so far, but you could try running llvm-dsymutil on bin/clang for a larger example (takes about a minute to finish).> > A quick glance at dsymutil's code indicates it might benefit slightly, at least - in the string table emission, for example (it looks very similar to string table emission in dwp - just being able to reference the strings in the StringMap rather than copying them into MCStreamer could help (also I found using a DenseMap<StringRef to the memory mapped input helped as well - but that's a change you can make locally without any MCStreamer improvements) - other parts might be trickier, and consist of parts of referencable data (like the line table header) and parts that are not referencable (like their contents) - my prototype could be extended to handle that)-- adrian -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/b7d688ed/attachment.html>
David Blaikie via llvm-dev
2016-Feb-29 23:46 UTC
[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC
On Mon, Feb 29, 2016 at 3:36 PM, Adrian Prantl <aprantl at apple.com> wrote:> > On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com> wrote: > > Just in case it interests anyone else, I'm playing around with trying to > broaden the MCStreamer API to allow for emission of bytes without copying > the contents into a local buffer first (either because you already have a > buffer, or the bytes are already present in another file, etc) in > http://reviews.llvm.org/D17694 . In theory there's some overlap with lld > here (no doubt it already does this sort of thing, but not in a way, I > assume, we could reuse from other tools at the moment) and my motivation, > llvm-dwp, looks very much like "linking with a few extra steps". > > But to check that these changes might be more generally applicable, I > thought I'd solicit data from anyone building tools that might be memory > constrained as well. > > First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil. > > Adrian/Fred - do you guys ever have trouble with memory usage of > llvm-dsymutil? Do you have an example you could provide that has high > memory usage, so I could see if any simple changes based on my prototype MC > changes would help. > > > Since dsymutil processes object files one after another, >As does llvm-dwp. Think of llvm-dwp more like a linker with a few extra bits. But the MCStreamer API means any bytes you write to the streamer stay in memory until you "Finish" - so if you're dwp/linking large enough inputs, you have them all in memory when you really don't need them. For example, the dwp file I was generating is 7GB, but the tool with the memory improvements only has a high water mark of 2.3GB.> memory usage wasn’t really a problem so far, but you could try running > llvm-dsymutil on bin/clang for a larger example (takes about a minute to > finish). >Was thinking of something more accessible to me, on a non-Darwin platform. Is there a way I can generate the dsym inputs across Clang on a non-Darwin platform? (what happens if I run dsymutil on my ELF object files?)> > A quick glance at dsymutil's code indicates it might benefit slightly, at > least - in the string table emission, for example (it looks very similar to > string table emission in dwp - just being able to reference the strings in > the StringMap rather than copying them into MCStreamer could help (also I > found using a DenseMap<StringRef to the memory mapped input helped as well > - but that's a change you can make locally without any MCStreamer > improvements) - other parts might be trickier, and consist of parts of > referencable data (like the line table header) and parts that are not > referencable (like their contents) - my prototype could be extended to > handle that) > > > -- adrian >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/f51d10a6/attachment.html>
David Blaikie via llvm-dev
2016-Feb-29 23:47 UTC
[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC
On Mon, Feb 29, 2016 at 3:36 PM, Peter Collingbourne <peter at pcc.me.uk> wrote:> Hi David, > > The way I imagined that we might want to extend the MCStreamer API (this > was motivated by DIEData) is by allowing clients to move bytes and fixups > into the MC layer. > > This is the sort of API that I was imagining: > > void MoveAndEmitFragment(SmallVectorImpl<char> &&Data, > SmallVectorImpl<MCFixup> &&Fixups); > > Note that this mirrors the fields > MCEncodedFragmentWithContents::Contents and > MCEncodedFragmentWithFixups::Fixups > and the arguments could be directly moved into the fields of a newly > created > MCDataFragment. > > Would that work for your use case? >Not quite, unfortunately - the issue is that we're doing a task that is essentially "linking + a bit" - so imagine linking a bunch of files together, the final, say, debug_info.dwo section is made up of the concatenation of all the debug_info.dwo sections of the inputs. So it's fragmented and it's already available, memory mapped, never in a SmallVector, etc.> > Peter > > On Mon, Feb 29, 2016 at 03:18:22PM -0800, David Blaikie via llvm-dev wrote: > > Just in case it interests anyone else, I'm playing around with trying to > > broaden the MCStreamer API to allow for emission of bytes without copying > > the contents into a local buffer first (either because you already have a > > buffer, or the bytes are already present in another file, etc) in > > http://reviews.llvm.org/D17694 . In theory there's some overlap with lld > > here (no doubt it already does this sort of thing, but not in a way, I > > assume, we could reuse from other tools at the moment) and my motivation, > > llvm-dwp, looks very much like "linking with a few extra steps". > > > > But to check that these changes might be more generally applicable, I > > thought I'd solicit data from anyone building tools that might be memory > > constrained as well. > > > > First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil. > > > > Adrian/Fred - do you guys ever have trouble with memory usage of > > llvm-dsymutil? Do you have an example you could provide that has high > > memory usage, so I could see if any simple changes based on my prototype > MC > > changes would help. > > > > A quick glance at dsymutil's code indicates it might benefit slightly, at > > least - in the string table emission, for example (it looks very similar > to > > string table emission in dwp - just being able to reference the strings > in > > the StringMap rather than copying them into MCStreamer could help (also I > > found using a DenseMap<StringRef to the memory mapped input helped as > well > > - but that's a change you can make locally without any MCStreamer > > improvements) - other parts might be trickier, and consist of parts of > > referencable data (like the line table header) and parts that are not > > referencable (like their contents) - my prototype could be extended to > > handle that) > > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > -- > Peter >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/b9689273/attachment.html>
Frédéric Riss via llvm-dev
2016-Mar-01 04:36 UTC
[llvm-dev] Possible Memory Savings for tools emitting large amounts of existing data through MC
> On Feb 29, 2016, at 3:18 PM, David Blaikie <dblaikie at gmail.com> wrote: > > Just in case it interests anyone else, I'm playing around with trying to broaden the MCStreamer API to allow for emission of bytes without copying the contents into a local buffer first (either because you already have a buffer, or the bytes are already present in another file, etc) in http://reviews.llvm.org/D17694 <http://reviews.llvm.org/D17694> . In theory there's some overlap with lld here (no doubt it already does this sort of thing, but not in a way, I assume, we could reuse from other tools at the moment) and my motivation, llvm-dwp, looks very much like "linking with a few extra steps". > > But to check that these changes might be more generally applicable, I thought I'd solicit data from anyone building tools that might be memory constrained as well. > > First that comes to mind (Eric suggested/mentioned) is llvm-dsymutil. > > Adrian/Fred - do you guys ever have trouble with memory usage of llvm-dsymutil? Do you have an example you could provide that has high memory usage, so I could see if any simple changes based on my prototype MC changes would help. > > A quick glance at dsymutil's code indicates it might benefit slightly, at least - in the string table emission, for example (it looks very similar to string table emission in dwp - just being able to reference the strings in the StringMap rather than copying them into MCStreamer could help (also I found using a DenseMap<StringRef to the memory mapped input helped as well - but that's a change you can make locally without any MCStreamer improvements) - other parts might be trickier, and consist of parts of referencable data (like the line table header) and parts that are not referencable (like their contents) - my prototype could be extended to handle that)Yes, I agree dsymutil could see slight gains from the parts you mention. It wouldn’t be groundbreaking though, the biggest contributor would always be the DIE tree that we recreate completely. Fred -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160229/638c8626/attachment.html>
Reasonably Related Threads
- Possible Memory Savings for tools emitting large amounts of existing data through MC
- Possible Memory Savings for tools emitting large amounts of existing data through MC
- Possible Memory Savings for tools emitting large amounts of existing data through MC
- Possible Memory Savings for tools emitting large amounts of existing data through MC
- Possible Memory Savings for tools emitting large amounts of existing data through MC