llvm dev - Nov 2020 - [LLD] Support DWARF64, debug_info "sorting"

Looks like there is an agreement that this path, modifying lld to order sections
using relocations, should be explored.
If Igor doesn't object, since he was primary one driving DWARF64 so far, I
would like to give it a shot at implementing and collecting some performance
numbers. 🙂

Alex

________________________________
From: James Henderson <jh7370.2008 at my.bristol.ac.uk>
Sent: Thursday, November 12, 2020 2:20 AM
To: Fangrui Song <maskray at google.com>
Cc: Alexander Yermolovich <ayermolo at fb.com>; Robinson, Paul
<paul.robinson at sony.com>; David Blaikie <dblaikie at gmail.com>;
Eric Christopher <echristo at gmail.com>; Igor Kudrin <ikudrin at
accesssoftek.com>; llvm-dev at lists.llvm.org <llvm-dev at
lists.llvm.org>
Subject: Re: [llvm-dev] [LLD] Support DWARF64, debug_info "sorting"



On Thu, 12 Nov 2020 at 02:10, Fangrui Song <maskray at
google.com<mailto:maskray at google.com>> wrote:
On 2020-11-12, Alexander Yermolovich wrote:
>Thanks for feedback.
>
>I agree with patch and numbers this will be a more concrete discussion, but
I wanted to judge overall receptiveness to this approach and see maybe there was
a better way.
>
>"Whilst the majority of objects will only have a single CU in them,
there will be exceptions (LTO-generated objects, -r merged objects etc), so we
do need to consider this approach."
>David can you elaborate under which conditions LTO-generated objects will
have a mix of DWARF32/64 in same .debug_info? Looking at how dwarf64 was
implemented same flag will be used for the entirety of the dwarf output, even if
multiple CUs are included.
Thinking about it, I wouldn't expect an LTO generated object itself to have
a mixture of DWARF32/64, although I guess the 32/64 bit state could be encoded
in the IR (I am not familiar enough with it to know if it actually is or not).
It might be necessary to find ways to configure LTO to generate DWARF64,
possibly via a link-time option.
>
>On one hand since this is only applicable for when DWARF64 is used, special
option would be the way to go. Although the user will need to be aware of yet
another LLD option. Maybe an error when relocations overflow occur can be
modified to display this option along with -fdebug-types-section
I am quite happy with the relocation approach under a linker option. I'd
still
want to know generic-abi folks's thoughts, though. James may have prepared
something
he wants to share with generic-abi:) Let's wait...

I hadn't prepared anything if I'm honest (though if there's
widespread agreement that this would be useful, I certainly can - it would have
other positive improvements too, reducing the need for tools to rely on section
names to identify debug data for example). It was more a case of bouncing ideas
off of people to see what they thought. Any discussion we have will probably
also need circulating on the DWARF mailing list too, since it is more a DWARF
issue than a gABI issue (unless the solution is a new section type). Further
refinements to this idea that might make it more appealing to the generic group:
`SHT_DEBUG` for the section type name, with the first N bytes of the sh_info
used to specify the variant of debug data it represents (e.g. 0x1 for DWARF, 0x2
for SOME_OTHER_STANDARD etc), and the remainder for use as flags as defined by
the standard (I'm thinking for DWARF you could encode the 64-bit/32-bit
state in there, possibly the section variant (info/rnglists/line etc) and the
DWARF version too), on the understanding that consumers like the linker
wouldn't combine sections in a potentially broken way. This has the
advantage that it could be retrofitted to the existing standard versions, but as
has been pointed out, this won't help those with linker scripts - that could
only be solved with a new DWARF standard and separate names for 64/32 bit
sections, at least if we wanted to avoid the linker needing to do anything
beyond reading the section header.

The relocation approach sounds like a reasonable solution for the current
situation - even if we do decide to go the route of changing producers to start
emitting a new section type/update the standard etc, it doesn't resolve the
problem people may currently face.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201113/fba9b02c/attachment.html>

On 13.11.2020 07:43, Alexander Yermolovich wrote:
> Looks like there is an agreement that this path, modifying lld to order 
> sections using relocations, should be explored.
> If Igor doesn't object, since he was primary one driving DWARF64 so
far,
> I would like to give it a shot at implementing and collecting some 
> performance numbers. 🙂
No objections, definitely. Please, go ahead.

-- 
Best Regards,
Igor Kudrin
C++ Developer, Access Softek, Inc.​

On Thu, Nov 12, 2020 at 4:43 PM Alexander Yermolovich <ayermolo at fb.com>
wrote:
>
> Looks like there is an agreement that this path, modifying lld to order
sections using relocations, should be explored.
> If Igor doesn't object, since he was primary one driving DWARF64 so
far,
I would like to give it a shot at implementing and collecting some
performance numbers.
>
> Alex
>
> ________________________________
> From: James Henderson <jh7370.2008 at my.bristol.ac.uk>
> Sent: Thursday, November 12, 2020 2:20 AM
> To: Fangrui Song <maskray at google.com>
> Cc: Alexander Yermolovich <ayermolo at fb.com>; Robinson, Paul <
paul.robinson at sony.com>; David Blaikie <dblaikie at gmail.com>; Eric
Christopher <echristo at gmail.com>; Igor Kudrin <ikudrin at
accesssoftek.com>;
llvm-dev at lists.llvm.org <llvm-dev at
lists.llvm.org>
> Subject: Re: [llvm-dev] [LLD] Support DWARF64, debug_info
"sorting"
I probably should have mentioned that I had started a prototype:) (And I
realized that I could use firstRelocation instead of dependentSections)
What I haven't felt comfortable with is the input section description
inconsistency
> https://sourceware.org/pipermail/binutils/2020-November/114099.html
>
> This behavior does add some inconsistency to the system:
>
> For an output section description .debug_info 0 : { *(.debug_info) } ,
should the linker sort DWARF32 and DWARF64 components? It it does, the
behavior will be inconsistent with other input section descriptions
*(foo)
>
> If there is a magic keyword, say, SORT_BY_MAGIC_DEBUG, and the internal
> linker script does something similar to
>
>   *(SORT_BY_MAGIC_DEBUG(.debug_info))
>
> then the system is still consistent.
I also started a thread on binutils side yesterday (sent in haste)
https://sourceware.org/pipermail/binutils/2020-November/114099.html
(We should give then a chance for design and hope for a common option, at
least getting a consensus even if the implementation on their side is of
low priority)
>
>
> On Thu, 12 Nov 2020 at 02:10, Fangrui Song <maskray at google.com>
wrote:
>
> On 2020-11-12, Alexander Yermolovich wrote:
> >Thanks for feedback.
> >
> >I agree with patch and numbers this will be a more concrete discussion,
but I wanted to judge overall receptiveness to this approach and see maybe
there was a better way.
> >
> >"Whilst the majority of objects will only have a single CU in
them,
there will be exceptions (LTO-generated objects, -r merged objects etc), so
we do need to consider this approach."
> >David can you elaborate under which conditions LTO-generated objects
will have a mix of DWARF32/64 in same .debug_info? Looking at how dwarf64
was implemented same flag will be used for the entirety of the dwarf
output, even if multiple CUs are included.
>
>
> Thinking about it, I wouldn't expect an LTO generated object itself to
have a mixture of DWARF32/64, although I guess the 32/64 bit state could be
encoded in the IR (I am not familiar enough with it to know if it actually
is or not). It might be necessary to find ways to configure LTO to generate
DWARF64, possibly via a link-time option.
>
> >
> >On one hand since this is only applicable for when DWARF64 is used,
special option would be the way to go. Although the user will need to be
aware of yet another LLD option. Maybe an error when relocations overflow
occur can be modified to display this option along with
-fdebug-types-section
>
> I am quite happy with the relocation approach under a linker option.
I'd
still
> want to know generic-abi folks's thoughts, though. James may have
prepared something
> he wants to share with generic-abi:) Let's wait...
>
>
> I hadn't prepared anything if I'm honest (though if there's
widespread
agreement that this would be useful, I certainly can - it would have other
positive improvements too, reducing the need for tools to rely on section
names to identify debug data for example). It was more a case of bouncing
ideas off of people to see what they thought. Any discussion we have will
probably also need circulating on the DWARF mailing list too, since it is
more a DWARF issue than a gABI issue (unless the solution is a new section
type). Further refinements to this idea that might make it more appealing
to the generic group: `SHT_DEBUG` for the section type name, with the first
N bytes of the sh_info used to specify the variant of debug data it
represents (e.g. 0x1 for DWARF, 0x2 for SOME_OTHER_STANDARD etc), and the
remainder for use as flags as defined by the standard (I'm thinking for
DWARF you could encode the 64-bit/32-bit state in there, possibly the
section variant (info/rnglists/line etc) and the DWARF version too), on the
understanding that consumers like the linker wouldn't combine sections in a
potentially broken way. This has the advantage that it could be retrofitted
to the existing standard versions, but as has been pointed out, this won't
help those with linker scripts - that could only be solved with a new DWARF
standard and separate names for 64/32 bit sections, at least if we wanted
to avoid the linker needing to do anything beyond reading the section
header.
>
> The relocation approach sounds like a reasonable solution for the current
situation - even if we do decide to go the route of changing producers to
start emitting a new section type/update the standard etc, it doesn't
resolve the problem people may currently face.



-- 
宋方睿
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201112/6a64018b/attachment.html>

On Thu, Nov 12, 2020 at 9:03 PM Fāng-ruì Sòng <maskray at google.com>
wrote:
> On Thu, Nov 12, 2020 at 4:43 PM Alexander Yermolovich <ayermolo at
fb.com>
> wrote:
> >
> > Looks like there is an agreement that this path, modifying lld to
order
> sections using relocations, should be explored.
> > If Igor doesn't object, since he was primary one driving DWARF64
so far,
> I would like to give it a shot at implementing and collecting some
> performance numbers.
> >
> > Alex
> >
> > ________________________________
> > From: James Henderson <jh7370.2008 at my.bristol.ac.uk>
> > Sent: Thursday, November 12, 2020 2:20 AM
> > To: Fangrui Song <maskray at google.com>
> > Cc: Alexander Yermolovich <ayermolo at fb.com>; Robinson, Paul
<
> paul.robinson at sony.com>; David Blaikie <dblaikie at gmail.com>;
Eric
> Christopher <echristo at gmail.com>; Igor Kudrin <ikudrin at
accesssoftek.com>;
> llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>
> > Subject: Re: [llvm-dev] [LLD] Support DWARF64, debug_info
"sorting"
>
> I probably should have mentioned that I had started a prototype:) (And I
> realized that I could use firstRelocation instead of dependentSections)
> What I haven't felt comfortable with is the input section description
> inconsistency
>
> > https://sourceware.org/pipermail/binutils/2020-November/114099.html
> >
> > This behavior does add some inconsistency to the system:
> >
> > For an output section description .debug_info 0 : { *(.debug_info) } ,
> should the linker sort DWARF32 and DWARF64 components? It it does, the
> behavior will be inconsistent with other input section descriptions *(foo)
> >
> > If there is a magic keyword, say, SORT_BY_MAGIC_DEBUG, and the
internal
> > linker script does something similar to
> >
> >   *(SORT_BY_MAGIC_DEBUG(.debug_info))
> >
> > then the system is still consistent.
>
> I also started a thread on binutils side yesterday (sent in haste)
> https://sourceware.org/pipermail/binutils/2020-November/114099.html
> (We should give then a chance for design and hope for a common option, at
> least getting a consensus even if the implementation on their side is of
> low priority)
>
Sent https://reviews.llvm.org/D91404 for the idea

> >
> >
> > On Thu, 12 Nov 2020 at 02:10, Fangrui Song <maskray at
google.com> wrote:
> >
> > On 2020-11-12, Alexander Yermolovich wrote:
> > >Thanks for feedback.
> > >
> > >I agree with patch and numbers this will be a more concrete
discussion,
> but I wanted to judge overall receptiveness to this approach and see maybe
> there was a better way.
> > >
> > >"Whilst the majority of objects will only have a single CU in
them,
> there will be exceptions (LTO-generated objects, -r merged objects etc), so
> we do need to consider this approach."
> > >David can you elaborate under which conditions LTO-generated
objects
> will have a mix of DWARF32/64 in same .debug_info? Looking at how dwarf64
> was implemented same flag will be used for the entirety of the dwarf
> output, even if multiple CUs are included.
> >
> >
> > Thinking about it, I wouldn't expect an LTO generated object
itself to
> have a mixture of DWARF32/64, although I guess the 32/64 bit state could be
> encoded in the IR (I am not familiar enough with it to know if it actually
> is or not). It might be necessary to find ways to configure LTO to generate
> DWARF64, possibly via a link-time option.
> >
> > >
> > >On one hand since this is only applicable for when DWARF64 is
used,
> special option would be the way to go. Although the user will need to be
> aware of yet another LLD option. Maybe an error when relocations overflow
> occur can be modified to display this option along with
> -fdebug-types-section
> >
> > I am quite happy with the relocation approach under a linker option.
I'd
> still
> > want to know generic-abi folks's thoughts, though. James may have
> prepared something
> > he wants to share with generic-abi:) Let's wait...
> >
> >
> > I hadn't prepared anything if I'm honest (though if
there's widespread
> agreement that this would be useful, I certainly can - it would have other
> positive improvements too, reducing the need for tools to rely on section
> names to identify debug data for example). It was more a case of bouncing
> ideas off of people to see what they thought. Any discussion we have will
> probably also need circulating on the DWARF mailing list too, since it is
> more a DWARF issue than a gABI issue (unless the solution is a new section
> type). Further refinements to this idea that might make it more appealing
> to the generic group: `SHT_DEBUG` for the section type name, with the first
> N bytes of the sh_info used to specify the variant of debug data it
> represents (e.g. 0x1 for DWARF, 0x2 for SOME_OTHER_STANDARD etc), and the
> remainder for use as flags as defined by the standard (I'm thinking for
> DWARF you could encode the 64-bit/32-bit state in there, possibly the
> section variant (info/rnglists/line etc) and the DWARF version too), on the
> understanding that consumers like the linker wouldn't combine sections
in a
> potentially broken way. This has the advantage that it could be retrofitted
> to the existing standard versions, but as has been pointed out, this
won't
> help those with linker scripts - that could only be solved with a new DWARF
> standard and separate names for 64/32 bit sections, at least if we wanted
> to avoid the linker needing to do anything beyond reading the section
> header.
> >
> > The relocation approach sounds like a reasonable solution for the
> current situation - even if we do decide to go the route of changing
> producers to start emitting a new section type/update the standard etc, it
> doesn't resolve the problem people may currently face.
>
>
>
> --
> 宋方睿
>

-- 
宋方睿
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201112/13391dd3/attachment-0001.html>