llvm dev - Jun 2014 - [LLVMdev] Looking for a fix to memory leak in DWARF support

David,

Thanks for the quick response... 


No, at this point I am just getting into the issue... I assume it is a leak, but
no clear proof yet. I was hoping it was an obvious thing since I recall a
discussion about it a while ago... but maybe I am just confused.

Was your work for compressing DWARF data  motivated by a certain inefficiency in
debug info representation? Did it result in significant savings?

Thanks.

Sergei

---
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The
Linux Foundation

> -----Original Message-----
> From: David Blaikie [mailto:dblaikie at gmail.com]
> Sent: Friday, June 13, 2014 12:41 PM
> To: Sergei Larin
> Cc: LLVM Developers Mailing List
> Subject: Re: Looking for a fix to memory leak in DWARF support
> 
> Are you sure it's a leak, not just a large memory footprint?
> 
> A lot of things have changed. Heck, even the bugs that we've fixed
might not
> have existed two years ago (I've certainly created (and
> fixed) my share of bugs in debug info).
> 
> Best you start with a leak tracking tool and at least be able to point to
the
> major leak culprit (eg: "90% of the leaked memory was allocated at
<stack
> trace>")
> 
> On Fri, Jun 13, 2014 at 9:16 AM, Sergei Larin <slarin at
codeaurora.org> wrote:
> >
> > David, (and everyone else)
> >
> >   I am forced to do some maintenance work on a fairly old LLVM branch
> > (likely based on release 3.1) that among other issues has a major
> > problem with memory leak somewhere around DWARF debug support.
> > In fact customer is unable to build with -g at all - simply running
> > out of memory on their project...
> >
> >   I seem to remember that there has been a major fix related to it,
> > but finding it hard to pinpoint. So the appeal to the communal memory
> > - do you remember if  there was a single fix (since the middle of 2012
> > or so) that would have addressed such issue?
> >
> >   I know the code was drastically change since - I have seen this for
> > instance
> > llvm.org/viewvc/llvm-project/llvm/trunk/lib/MC/ELFObjectWriter.cpp?vie
> > w=log&
> > pathrev=205990
> >
> > ...but not sure this is what I am looking for, and whether there is
> > something less drastic than I can do to plug that leak.
> >
> > Any clue would be very much appreciated.
> >
> > Thanks.
> >
> > Sergei
> >
> > ---
> > Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> > hosted by The Linux Foundation
> >

On Fri, Jun 13, 2014 at 11:03 AM, Sergei Larin <slarin at codeaurora.org>
wrote:
>
> David,
>
> Thanks for the quick response...
>
>
> No, at this point I am just getting into the issue... I assume it is a
leak, but no clear proof yet. I was hoping it was an obvious thing since I
recall a discussion about it a while ago... but maybe I am just confused.
>
> Was your work for compressing DWARF data  motivated by a certain
inefficiency in debug info representation? Did it result in significant savings?
>
The compression isn't related to the IR level and some data is here on
that compressing:

https://gcc.gnu.org/wiki/DebugFission

The internal representation and code have change significantly since
then and there's no real way to help much. It may be a memory leak,
but I don't recall one that bad back then. How is your customer
compiling? LTO? Something else?

-eric
> Thanks.
>
> Sergei
>
> ---
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted
by The Linux Foundation
>
>
>> -----Original Message-----
>> From: David Blaikie [mailto:dblaikie at gmail.com]
>> Sent: Friday, June 13, 2014 12:41 PM
>> To: Sergei Larin
>> Cc: LLVM Developers Mailing List
>> Subject: Re: Looking for a fix to memory leak in DWARF support
>>
>> Are you sure it's a leak, not just a large memory footprint?
>>
>> A lot of things have changed. Heck, even the bugs that we've fixed
might not
>> have existed two years ago (I've certainly created (and
>> fixed) my share of bugs in debug info).
>>
>> Best you start with a leak tracking tool and at least be able to point
to the
>> major leak culprit (eg: "90% of the leaked memory was allocated at
<stack
>> trace>")
>>
>> On Fri, Jun 13, 2014 at 9:16 AM, Sergei Larin <slarin at
codeaurora.org> wrote:
>> >
>> > David, (and everyone else)
>> >
>> >   I am forced to do some maintenance work on a fairly old LLVM
branch
>> > (likely based on release 3.1) that among other issues has a major
>> > problem with memory leak somewhere around DWARF debug support.
>> > In fact customer is unable to build with -g at all - simply
running
>> > out of memory on their project...
>> >
>> >   I seem to remember that there has been a major fix related to
it,
>> > but finding it hard to pinpoint. So the appeal to the communal
memory
>> > - do you remember if  there was a single fix (since the middle of
2012
>> > or so) that would have addressed such issue?
>> >
>> >   I know the code was drastically change since - I have seen this
for
>> > instance
>> >
llvm.org/viewvc/llvm-project/llvm/trunk/lib/MC/ELFObjectWriter.cpp?vie
>> > w=log&
>> > pathrev=205990
>> >
>> > ...but not sure this is what I am looking for, and whether there
is
>> > something less drastic than I can do to plug that leak.
>> >
>> > Any clue would be very much appreciated.
>> >
>> > Thanks.
>> >
>> > Sergei
>> >
>> > ---
>> > Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
>> > hosted by The Linux Foundation
>> >
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

Thanks Eric,


They are doing LTO build but with some custom modifications (think a library at
a time as opposed to a whole program). I must admit, it is a rather large
application as well, so as expected, any inefficiencies are multiplied greatly.
>From little that I have seen so far, it looks like debug metadata for an IR
object linger behind once the object itself is eliminated (optimized). Rings a
bell?
Thanks.

Sergei

---
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The
Linux Foundation

> -----Original Message-----
> From: Eric Christopher [mailto:echristo at gmail.com]
> Sent: Friday, June 13, 2014 4:00 PM
> To: Sergei Larin
> Cc: David Blaikie; LLVM Developers Mailing List
> Subject: Re: [LLVMdev] Looking for a fix to memory leak in DWARF support
> 
> On Fri, Jun 13, 2014 at 11:03 AM, Sergei Larin <slarin at
codeaurora.org> wrote:
> >
> > David,
> >
> > Thanks for the quick response...
> >
> >
> > No, at this point I am just getting into the issue... I assume it is a
leak, but
> no clear proof yet. I was hoping it was an obvious thing since I recall a
> discussion about it a while ago... but maybe I am just confused.
> >
> > Was your work for compressing DWARF data  motivated by a certain
> inefficiency in debug info representation? Did it result in significant
savings?
> >
> 
> The compression isn't related to the IR level and some data is here on
that
> compressing:
> 
> https://gcc.gnu.org/wiki/DebugFission
> 
> The internal representation and code have change significantly since then
> and there's no real way to help much. It may be a memory leak, but I
don't
> recall one that bad back then. How is your customer compiling? LTO?
> Something else?
> 
> -eric
> 
> > Thanks.
> >
> > Sergei
> >
> > ---
> > Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> > hosted by The Linux Foundation
> >
> >
> >> -----Original Message-----
> >> From: David Blaikie [mailto:dblaikie at gmail.com]
> >> Sent: Friday, June 13, 2014 12:41 PM
> >> To: Sergei Larin
> >> Cc: LLVM Developers Mailing List
> >> Subject: Re: Looking for a fix to memory leak in DWARF support
> >>
> >> Are you sure it's a leak, not just a large memory footprint?
> >>
> >> A lot of things have changed. Heck, even the bugs that we've
fixed
> >> might not have existed two years ago (I've certainly created
(and
> >> fixed) my share of bugs in debug info).
> >>
> >> Best you start with a leak tracking tool and at least be able to
> >> point to the major leak culprit (eg: "90% of the leaked
memory was
> >> allocated at <stack
> >> trace>")
> >>
> >> On Fri, Jun 13, 2014 at 9:16 AM, Sergei Larin <slarin at
codeaurora.org>
> wrote:
> >> >
> >> > David, (and everyone else)
> >> >
> >> >   I am forced to do some maintenance work on a fairly old
LLVM
> >> > branch (likely based on release 3.1) that among other issues
has a
> >> > major problem with memory leak somewhere around DWARF debug
> support.
> >> > In fact customer is unable to build with -g at all - simply
running
> >> > out of memory on their project...
> >> >
> >> >   I seem to remember that there has been a major fix related
to it,
> >> > but finding it hard to pinpoint. So the appeal to the
communal
> >> > memory
> >> > - do you remember if  there was a single fix (since the
middle of
> >> > 2012 or so) that would have addressed such issue?
> >> >
> >> >   I know the code was drastically change since - I have seen
this
> >> > for instance
> >> >
llvm.org/viewvc/llvm-project/llvm/trunk/lib/MC/ELFObjectWriter.cpp?
> >> > vie
> >> > w=log&
> >> > pathrev=205990
> >> >
> >> > ...but not sure this is what I am looking for, and whether
there is
> >> > something less drastic than I can do to plug that leak.
> >> >
> >> > Any clue would be very much appreciated.
> >> >
> >> > Thanks.
> >> >
> >> > Sergei
> >> >
> >> > ---
> >> > Qualcomm Innovation Center, Inc. is a member of Code Aurora
Forum,
> >> > hosted by The Linux Foundation
> >> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On Fri, Jun 13, 2014 at 11:03 AM, Sergei Larin <slarin at codeaurora.org>
wrote:
>
> David,
>
> Thanks for the quick response...
>
>
> No, at this point I am just getting into the issue... I assume it is a
leak, but no clear proof yet. I was hoping it was an obvious thing since I
recall a discussion about it a while ago... but maybe I am just confused.
There have been some memory leaks fixed - the discussion you might be
recalling is the frontend leaks using DIBuilder. Addressed by changes
like this: http://llvm.org/viewvc/llvm-project?rev=208055&view=rev

But I don't think that was ever a crippling bug, so far as I recall.
We only found this when using leak detectors - not when compiling vast
swathes of code, etc.
> Was your work for compressing DWARF data  motivated by a certain
inefficiency in debug info representation? Did it result in significant savings?
compressing DWARF has little to do with memory leaks, etc - it's
related to the nature of DWARF on disk, that's where the compression
occurs (in the object (and dwo) files themselves). Yes, it does have
substantial savings, though I don't have the numbers at hand right
now. (string sections especially are compressed well).

- David
>
> Thanks.
>
> Sergei
>
> ---
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted
by The Linux Foundation
>
>
>> -----Original Message-----
>> From: David Blaikie [mailto:dblaikie at gmail.com]
>> Sent: Friday, June 13, 2014 12:41 PM
>> To: Sergei Larin
>> Cc: LLVM Developers Mailing List
>> Subject: Re: Looking for a fix to memory leak in DWARF support
>>
>> Are you sure it's a leak, not just a large memory footprint?
>>
>> A lot of things have changed. Heck, even the bugs that we've fixed
might not
>> have existed two years ago (I've certainly created (and
>> fixed) my share of bugs in debug info).
>>
>> Best you start with a leak tracking tool and at least be able to point
to the
>> major leak culprit (eg: "90% of the leaked memory was allocated at
<stack
>> trace>")
>>
>> On Fri, Jun 13, 2014 at 9:16 AM, Sergei Larin <slarin at
codeaurora.org> wrote:
>> >
>> > David, (and everyone else)
>> >
>> >   I am forced to do some maintenance work on a fairly old LLVM
branch
>> > (likely based on release 3.1) that among other issues has a major
>> > problem with memory leak somewhere around DWARF debug support.
>> > In fact customer is unable to build with -g at all - simply
running
>> > out of memory on their project...
>> >
>> >   I seem to remember that there has been a major fix related to
it,
>> > but finding it hard to pinpoint. So the appeal to the communal
memory
>> > - do you remember if  there was a single fix (since the middle of
2012
>> > or so) that would have addressed such issue?
>> >
>> >   I know the code was drastically change since - I have seen this
for
>> > instance
>> >
llvm.org/viewvc/llvm-project/llvm/trunk/lib/MC/ELFObjectWriter.cpp?vie
>> > w=log&
>> > pathrev=205990
>> >
>> > ...but not sure this is what I am looking for, and whether there
is
>> > something less drastic than I can do to plug that leak.
>> >
>> > Any clue would be very much appreciated.
>> >
>> > Thanks.
>> >
>> > Sergei
>> >
>> > ---
>> > Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
>> > hosted by The Linux Foundation
>> >
>

David,

  This certainly looks relevant - even if this is not the ultimate cause of the
issue. I will try it. Thank you.

  ...it also sounds very reasonable what Eric is describing:

" It's just an explosion of debug info because of our lack of uniquing
for type information."

It looks more like the version of tools I am dealing with was never ready to
handle LTO + debug efficiently, and it would be more like back porting a major
new feature to an older codebase :(  This sure looks like a fun project... Maybe
my time will be better used to convince the customer to upgrade :)

Thanks again for helping.

Sergei

---
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The
Linux Foundation

> -----Original Message-----
> From: David Blaikie [mailto:dblaikie at gmail.com]
> Sent: Friday, June 13, 2014 4:32 PM
> To: Sergei Larin
> Cc: LLVM Developers Mailing List
> Subject: Re: Looking for a fix to memory leak in DWARF support
> 
> On Fri, Jun 13, 2014 at 11:03 AM, Sergei Larin <slarin at
codeaurora.org> wrote:
> >
> > David,
> >
> > Thanks for the quick response...
> >
> >
> > No, at this point I am just getting into the issue... I assume it is a
leak, but
> no clear proof yet. I was hoping it was an obvious thing since I recall a
> discussion about it a while ago... but maybe I am just confused.
> 
> There have been some memory leaks fixed - the discussion you might be
> recalling is the frontend leaks using DIBuilder. Addressed by changes like
this:
> http://llvm.org/viewvc/llvm-project?rev=208055&view=rev
> 
> But I don't think that was ever a crippling bug, so far as I recall.
> We only found this when using leak detectors - not when compiling vast
> swathes of code, etc.
> 
> > Was your work for compressing DWARF data  motivated by a certain
> inefficiency in debug info representation? Did it result in significant
savings?
> 
> compressing DWARF has little to do with memory leaks, etc - it's
related to
> the nature of DWARF on disk, that's where the compression occurs (in
the
> object (and dwo) files themselves). Yes, it does have substantial savings,
> though I don't have the numbers at hand right now. (string sections
> especially are compressed well).
> 
> - David
> 
> >
> > Thanks.
> >
> > Sergei
> >
> > ---
> > Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> > hosted by The Linux Foundation
> >
> >
> >> -----Original Message-----
> >> From: David Blaikie [mailto:dblaikie at gmail.com]
> >> Sent: Friday, June 13, 2014 12:41 PM
> >> To: Sergei Larin
> >> Cc: LLVM Developers Mailing List
> >> Subject: Re: Looking for a fix to memory leak in DWARF support
> >>
> >> Are you sure it's a leak, not just a large memory footprint?
> >>
> >> A lot of things have changed. Heck, even the bugs that we've
fixed
> >> might not have existed two years ago (I've certainly created
(and
> >> fixed) my share of bugs in debug info).
> >>
> >> Best you start with a leak tracking tool and at least be able to
> >> point to the major leak culprit (eg: "90% of the leaked
memory was
> >> allocated at <stack
> >> trace>")
> >>
> >> On Fri, Jun 13, 2014 at 9:16 AM, Sergei Larin <slarin at
codeaurora.org>
> wrote:
> >> >
> >> > David, (and everyone else)
> >> >
> >> >   I am forced to do some maintenance work on a fairly old
LLVM
> >> > branch (likely based on release 3.1) that among other issues
has a
> >> > major problem with memory leak somewhere around DWARF debug
> support.
> >> > In fact customer is unable to build with -g at all - simply
running
> >> > out of memory on their project...
> >> >
> >> >   I seem to remember that there has been a major fix related
to it,
> >> > but finding it hard to pinpoint. So the appeal to the
communal
> >> > memory
> >> > - do you remember if  there was a single fix (since the
middle of
> >> > 2012 or so) that would have addressed such issue?
> >> >
> >> >   I know the code was drastically change since - I have seen
this
> >> > for instance
> >> >
llvm.org/viewvc/llvm-project/llvm/trunk/lib/MC/ELFObjectWriter.cpp?
> >> > vie
> >> > w=log&
> >> > pathrev=205990
> >> >
> >> > ...but not sure this is what I am looking for, and whether
there is
> >> > something less drastic than I can do to plug that leak.
> >> >
> >> > Any clue would be very much appreciated.
> >> >
> >> > Thanks.
> >> >
> >> > Sergei
> >> >
> >> > ---
> >> > Qualcomm Innovation Center, Inc. is a member of Code Aurora
Forum,
> >> > hosted by The Linux Foundation
> >> >
> >