Diego Novillo via llvm-dev
2015-Dec-10 20:56 UTC
[llvm-dev] Memory utilization problems in profile reader
On Wed, Dec 9, 2015 at 3:14 PM, Xinliang David Li <xinliangli at gmail.com> wrote:> Can you extract the relevant part of the heap profile data? >It's all profile data, actually. The heap utilization is massively dominated by the profile reader.> How large is the sample profile data fed to the compiler? > >For this run, the input file was 21Mb.> The indexed format profile size for clang is <100MB. The InstrProfRecord > for each function is read, used and discarded one at a time, so there > should not be problem as described. >Good. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151210/635bfd6f/attachment.html>
Diego Novillo via llvm-dev
2015-Dec-11 14:58 UTC
[llvm-dev] Memory utilization problems in profile reader
So, I traced it down to the DenseMaps in class FunctionSamples. I've replaced them with two std::vector, and the read operation causes the compiler to grow from 70Mb to 280Mb. With the DenseMaps, reading the profile causes the compiler to grow from 70Mb to 3Gb. Somehow the DenseMaps are causing a 10x growth factor. Those keys are probably an issue. Or perhaps we just need a different representation for sample records and call sites. On Thu, Dec 10, 2015 at 3:56 PM, Diego Novillo <dnovillo at google.com> wrote:> > > On Wed, Dec 9, 2015 at 3:14 PM, Xinliang David Li <xinliangli at gmail.com> > wrote: > >> Can you extract the relevant part of the heap profile data? >> > > It's all profile data, actually. The heap utilization is massively > dominated by the profile reader. > > > >> How large is the sample profile data fed to the compiler? >> >> > For this run, the input file was 21Mb. > > >> The indexed format profile size for clang is <100MB. The InstrProfRecord >> for each function is read, used and discarded one at a time, so there >> should not be problem as described. >> > > Good. >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151211/e44831b1/attachment.html>
Diego Novillo via llvm-dev
2015-Dec-11 15:53 UTC
[llvm-dev] Memory utilization problems in profile reader
On Fri, Dec 11, 2015 at 9:58 AM, Diego Novillo <dnovillo at google.com> wrote:> So, I traced it down to the DenseMaps in class FunctionSamples. I've > replaced them with two std::vector, and the read operation causes the > compiler to grow from 70Mb to 280Mb. With the DenseMaps, reading the > profile causes the compiler to grow from 70Mb to 3Gb. > > Somehow the DenseMaps are causing a 10x growth factor. Those keys are > probably an issue. Or perhaps we just need a different representation for > sample records and call sites. >Yes. In going through DenseMap's implementation I see that large values for keys will cause a lot of growth. And the documentation confirms it (someday I'll learn to read documentation first). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151211/ae465acd/attachment.html>
Xinliang David Li via llvm-dev
2015-Dec-11 17:04 UTC
[llvm-dev] Memory utilization problems in profile reader
On Fri, Dec 11, 2015 at 6:58 AM, Diego Novillo <dnovillo at google.com> wrote:> So, I traced it down to the DenseMaps in class FunctionSamples. I've > replaced them with two std::vector, and the read operation causes the > compiler to grow from 70Mb to 280Mb. With the DenseMaps, reading the > profile causes the compiler to grow from 70Mb to 3Gb. > >The growth from 70Mb to 280Mb for a ~20Mb profile is also alarming, IMO. David> Somehow the DenseMaps are causing a 10x growth factor. Those keys are > probably an issue. Or perhaps we just need a different representation for > sample records and call sites. > > On Thu, Dec 10, 2015 at 3:56 PM, Diego Novillo <dnovillo at google.com> > wrote: > >> >> >> On Wed, Dec 9, 2015 at 3:14 PM, Xinliang David Li <xinliangli at gmail.com> >> wrote: >> >>> Can you extract the relevant part of the heap profile data? >>> >> >> It's all profile data, actually. The heap utilization is massively >> dominated by the profile reader. >> >> >> >>> How large is the sample profile data fed to the compiler? >>> >>> >> For this run, the input file was 21Mb. >> >> >>> The indexed format profile size for clang is <100MB. The >>> InstrProfRecord for each function is read, used and discarded one at a >>> time, so there should not be problem as described. >>> >> >> Good. >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151211/fdee4d92/attachment.html>