llvm dev - Sep 2018 - [lldb-dev] [LLD] How to get rid of debug info of sections deleted by garbage collector

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Davide Italiano via llvm-dev
> Sent: Thursday, September 20, 2018 10:55 AM
> To: ramana.venkat83 at gmail.com; Cary Coutant
> Cc: llvm-dev; LLDB
> Subject: Re: [llvm-dev] [lldb-dev] [LLD] How to get rid of debug info of
> sections deleted by garbage collector
> 
> On Wed, Sep 19, 2018 at 8:35 PM Venkata Ramanaiah Nalamothu via
> lldb-dev <lldb-dev at lists.llvm.org> wrote:
> >
> > Hi,
> >
> > After compiling an example.cpp file with "-c
-ffunction-sections" and
> linking with "--gc-sections" (used ld.lld), I am still seeing
debug info
> for the sections deleted by garbage collector in the generated executable.
> >
> > Are there any compiler/linker options and/or other tools in LLVM to
get
> rid of the above mentioned unneeded debug info?
> >
> > If such options does not exist, what needs to be changed in the linker
> (lld)?
> >
> 
> It's not easy. It's also format dependent. I assume you're
talking
> about ELF here. In first approximation, the linker does not GC section
> marked without SHF_ALLOC. At some point we did an analysis and in
> practice it turns out most of them are debug info.
> I seem to recall that Cary Coutant had a proposal for ld.gold on how
> to reclaim them without breaking, but I can't find it easily (cc:ing
> him directly).
The short answer is: Nothing you can do currently.

I had a chat with some of the Sony linker guys last week about this.
Currently .debug_info is monolithic; we'd have to break it up in some
fashion that would correspond with the way .text is broken up with
-ffunction-sections, in such a way that the linker would automatically
paste the right pieces back together to form a syntactically correct 
.debug_info section in the final executable.  There are some gotchas
that would need to be designed correctly (e.g. reference from an
inlined-subprogram to its abstract instance) but it didn't seem like
the problems were insurmountable.

The ultimate design almost certainly requires agreement about what the
ELF pieces should look like, and a description in the DWARF spec so
that consumers (e.g. dumpers) of the .o files would understand about
the fragmented sections.  And then the linkers and dumpers have to
be modified to implement it all. :-)

Even without gc-sections, there is duplicate info to get rid of:
everything that ends up in a COMDAT, like template instantiations
and inline functions.  This is actually a much bigger win than
anything you'd see left behind by GC.
--paulr
> 
> Thanks,
> 
> --
> Davide
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Right. Technically we can get rid of debug info that corresponds to dead
sections, but in order to do that, you have to scan the entire debug info.
Debug info is actually one of only few pieces of information that the
linker has to have a special logic to merge them, and that is already slow.
IIUC, debug info for dead sections doesn't do any harm, so spending time in
the linker to get rid of it isn't probably worth the cost. If we really
need to do, I want a new mechanism as Paul wrote.

On Thu, Sep 20, 2018 at 8:57 AM via llvm-dev <llvm-dev at lists.llvm.org>
wrote:
>
> > -----Original Message-----
> > From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf
Of
> > Davide Italiano via llvm-dev
> > Sent: Thursday, September 20, 2018 10:55 AM
> > To: ramana.venkat83 at gmail.com; Cary Coutant
> > Cc: llvm-dev; LLDB
> > Subject: Re: [llvm-dev] [lldb-dev] [LLD] How to get rid of debug info
of
> > sections deleted by garbage collector
> >
> > On Wed, Sep 19, 2018 at 8:35 PM Venkata Ramanaiah Nalamothu via
> > lldb-dev <lldb-dev at lists.llvm.org> wrote:
> > >
> > > Hi,
> > >
> > > After compiling an example.cpp file with "-c
-ffunction-sections" and
> > linking with "--gc-sections" (used ld.lld), I am still
seeing debug info
> > for the sections deleted by garbage collector in the generated
> executable.
> > >
> > > Are there any compiler/linker options and/or other tools in LLVM
to get
> > rid of the above mentioned unneeded debug info?
> > >
> > > If such options does not exist, what needs to be changed in the
linker
> > (lld)?
> > >
> >
> > It's not easy. It's also format dependent. I assume you're
talking
> > about ELF here. In first approximation, the linker does not GC section
> > marked without SHF_ALLOC. At some point we did an analysis and in
> > practice it turns out most of them are debug info.
> > I seem to recall that Cary Coutant had a proposal for ld.gold on how
> > to reclaim them without breaking, but I can't find it easily
(cc:ing
> > him directly).
>
> The short answer is: Nothing you can do currently.
>
> I had a chat with some of the Sony linker guys last week about this.
> Currently .debug_info is monolithic; we'd have to break it up in some
> fashion that would correspond with the way .text is broken up with
> -ffunction-sections, in such a way that the linker would automatically
> paste the right pieces back together to form a syntactically correct
> .debug_info section in the final executable.  There are some gotchas
> that would need to be designed correctly (e.g. reference from an
> inlined-subprogram to its abstract instance) but it didn't seem like
> the problems were insurmountable.
>
> The ultimate design almost certainly requires agreement about what the
> ELF pieces should look like, and a description in the DWARF spec so
> that consumers (e.g. dumpers) of the .o files would understand about
> the fragmented sections.  And then the linkers and dumpers have to
> be modified to implement it all. :-)
>
> Even without gc-sections, there is duplicate info to get rid of:
> everything that ends up in a COMDAT, like template instantiations
> and inline functions.  This is actually a much bigger win than
> anything you'd see left behind by GC.
> --paulr
>
> >
> > Thanks,
> >
> > --
> > Davide
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180920/b6a734d5/attachment.html>

Thank you all for your time in responding to my query.

My understanding was also similar to what you all mentioned here but wanted
to check if there are any recent developments in solving this problem.

Thanks,
Ramana

On Thu, Sep 20, 2018 at 9:32 PM, Rui Ueyama <ruiu at google.com> wrote:
> Right. Technically we can get rid of debug info that corresponds to dead
> sections, but in order to do that, you have to scan the entire debug info.
> Debug info is actually one of only few pieces of information that the
> linker has to have a special logic to merge them, and that is already slow.
> IIUC, debug info for dead sections doesn't do any harm, so spending
time in
> the linker to get rid of it isn't probably worth the cost. If we really
> need to do, I want a new mechanism as Paul wrote.
>
> On Thu, Sep 20, 2018 at 8:57 AM via llvm-dev <llvm-dev at
lists.llvm.org>
> wrote:
>
>>
>> > -----Original Message-----
>> > From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On
Behalf Of
>> > Davide Italiano via llvm-dev
>> > Sent: Thursday, September 20, 2018 10:55 AM
>> > To: ramana.venkat83 at gmail.com; Cary Coutant
>> > Cc: llvm-dev; LLDB
>> > Subject: Re: [llvm-dev] [lldb-dev] [LLD] How to get rid of debug
info of
>> > sections deleted by garbage collector
>> >
>> > On Wed, Sep 19, 2018 at 8:35 PM Venkata Ramanaiah Nalamothu via
>> > lldb-dev <lldb-dev at lists.llvm.org> wrote:
>> > >
>> > > Hi,
>> > >
>> > > After compiling an example.cpp file with "-c
-ffunction-sections" and
>> > linking with "--gc-sections" (used ld.lld), I am still
seeing debug info
>> > for the sections deleted by garbage collector in the generated
>> executable.
>> > >
>> > > Are there any compiler/linker options and/or other tools in
LLVM to
>> get
>> > rid of the above mentioned unneeded debug info?
>> > >
>> > > If such options does not exist, what needs to be changed in
the linker
>> > (lld)?
>> > >
>> >
>> > It's not easy. It's also format dependent. I assume
you're talking
>> > about ELF here. In first approximation, the linker does not GC
section
>> > marked without SHF_ALLOC. At some point we did an analysis and in
>> > practice it turns out most of them are debug info.
>> > I seem to recall that Cary Coutant had a proposal for ld.gold on
how
>> > to reclaim them without breaking, but I can't find it easily
(cc:ing
>> > him directly).
>>
>> The short answer is: Nothing you can do currently.
>>
>> I had a chat with some of the Sony linker guys last week about this.
>> Currently .debug_info is monolithic; we'd have to break it up in
some
>> fashion that would correspond with the way .text is broken up with
>> -ffunction-sections, in such a way that the linker would automatically
>> paste the right pieces back together to form a syntactically correct
>> .debug_info section in the final executable.  There are some gotchas
>> that would need to be designed correctly (e.g. reference from an
>> inlined-subprogram to its abstract instance) but it didn't seem
like
>> the problems were insurmountable.
>>
>> The ultimate design almost certainly requires agreement about what the
>> ELF pieces should look like, and a description in the DWARF spec so
>> that consumers (e.g. dumpers) of the .o files would understand about
>> the fragmented sections.  And then the linkers and dumpers have to
>> be modified to implement it all. :-)
>>
>> Even without gc-sections, there is duplicate info to get rid of:
>> everything that ends up in a COMDAT, like template instantiations
>> and inline functions.  This is actually a much bigger win than
>> anything you'd see left behind by GC.
>> --paulr
>>
>> >
>> > Thanks,
>> >
>> > --
>> > Davide
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180921/63b8482b/attachment.html>