llvm dev - Nov 2020 - [LLD] Support DWARF64, debug_info "sorting"

I got replies from Nick Clifton and Michael Matz:
https://sourceware.org/pipermail/binutils/2020-November/114116.html
(and its reply).
I have mentioned (a) the difficulty of the
detecting-DWARF64-by-first-relocation approach and (b) the section
type approach  in my reply there
https://sourceware.org/pipermail/binutils/2020-November/114125.html

(a) My prototype has made me feel uneasy with this approach.

<quote>
In DWARF v4 or if .debug_str_offset is not used, it is a problem. A
heuristic is: if an input section in a file is marked DWARF64, we mark
all other .debug_* DWARF64. This makes me feel a bit uneasy because
for an output section description

   .debug_str 0 : { *(.debug_str) }

Now the behavior of `*` (or, if we invent a `SORT_*` keyword) is also
dependent on other output sections.
</quote>

(b)
* It needs a section type (either a gABI one or a SHT_GNU_* in GNU
ABI). Seeking for a gABI one is not that I think this is particularly
related to gABI but that I don't want Solaris (which LLVM also
supports) uses a different section type to unnecessarily cause
friction on our implementation
* It needs a clarification on multiple output section descriptions
with the same name.
* It needs a linker script feature to match input sections by type.

On Fri, Nov 13, 2020 at 9:52 AM David Blaikie <dblaikie at gmail.com>
wrote:
>
> On Fri, Nov 13, 2020 at 8:35 AM Wenlei He via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >
> > >  Thinking about it, I wouldn't expect an LTO generated object
itself to have a mixture of DWARF32/64, although I guess the 32/64 bit state
could be encoded in the IR (I am not familiar enough with it to know if it
actually is or not). It might be necessary to find ways to configure LTO to
generate DWARF64, possibly via a link-time option.
> >
> >
> >
> > I don’t think we need to encode dwarf32/64 in IR as attribute for each
module. We’re not going to emit mixed dwarf32/64 for merged LTO module anyways,
so allowing each module to express its dwarf setting would only introduce burden
for LTO to deal with inconsistency (warning?) among input modules. Having a
linker switch to pass the setting from driver to LTO sounds better to me.
>
> Usually the issue there is that existing build systems may be setup
> only to pass such flags to the compilations, and not to the link
> invocations - like DWARF version, we pass that down through IR, emit
> warnings/errors when two IR modules with different DWARF versions are
> linked together, and then emit only one (the higher, I believe) DWARF
> version out the other end.
>
> We aren't 100% consistent on this "anything you could do without
LTO,
> you shuold be able to do with LTO/passing the same flags to the same
> actions" kind of strategy (eg: type units and DWARF compression
aren't
> passed down through IR - if you want those you have to pass them to
> the link invocation (via the clang driver) yourself). So it's more
"is
> there a systemic use of these flags already for the compilation and
> would not supporting it there be a pain"? It's probably not for
> DWARF64, since we haven't had any flag to support it at the moment
> anyway.
>
> - Dave
>
> >
> >
> >
> > From: llvm-dev <llvm-dev-bounces at lists.llvm.org>
> > Date: Thursday, November 12, 2020 at 2:21 AM
> > To: Fangrui Song <maskray at google.com>
> > Cc: llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>
> > Subject: Re: [llvm-dev] [LLD] Support DWARF64, debug_info
"sorting"
> >
> >
> >
> >
> >
> > On Thu, 12 Nov 2020 at 02:10, Fangrui Song <maskray at
google.com> wrote:
> >
> > On 2020-11-12, Alexander Yermolovich wrote:
> > >Thanks for feedback.
> > >
> > >I agree with patch and numbers this will be a more concrete
discussion, but I wanted to judge overall receptiveness to this approach and see
maybe there was a better way.
> > >
> > >"Whilst the majority of objects will only have a single CU in
them, there will be exceptions (LTO-generated objects, -r merged objects etc),
so we do need to consider this approach."
> > >David can you elaborate under which conditions LTO-generated
objects will have a mix of DWARF32/64 in same .debug_info? Looking at how
dwarf64 was implemented same flag will be used for the entirety of the dwarf
output, even if multiple CUs are included.
> >
> >
> >
> > Thinking about it, I wouldn't expect an LTO generated object
itself to have a mixture of DWARF32/64, although I guess the 32/64 bit state
could be encoded in the IR (I am not familiar enough with it to know if it
actually is or not). It might be necessary to find ways to configure LTO to
generate DWARF64, possibly via a link-time option.
> >
> >
> >
> > >
> > >On one hand since this is only applicable for when DWARF64 is
used, special option would be the way to go. Although the user will need to be
aware of yet another LLD option. Maybe an error when relocations overflow occur
can be modified to display this option along with -fdebug-types-section
> >
> > I am quite happy with the relocation approach under a linker option.
I'd still
> > want to know generic-abi folks's thoughts, though. James may have
prepared something
> > he wants to share with generic-abi:) Let's wait...
> >
> >
> >
> > I hadn't prepared anything if I'm honest (though if
there's widespread agreement that this would be useful, I certainly can - it
would have other positive improvements too, reducing the need for tools to rely
on section names to identify debug data for example). It was more a case of
bouncing ideas off of people to see what they thought. Any discussion we have
will probably also need circulating on the DWARF mailing list too, since it is
more a DWARF issue than a gABI issue (unless the solution is a new section
type). Further refinements to this idea that might make it more appealing to the
generic group: `SHT_DEBUG` for the section type name, with the first N bytes of
the sh_info used to specify the variant of debug data it represents (e.g. 0x1
for DWARF, 0x2 for SOME_OTHER_STANDARD etc), and the remainder for use as flags
as defined by the standard (I'm thinking for DWARF you could encode the
64-bit/32-bit state in there, possibly the section variant (info/rnglists/line
etc) and the DWARF version too), on the understanding that consumers like the
linker wouldn't combine sections in a potentially broken way. This has the
advantage that it could be retrofitted to the existing standard versions, but as
has been pointed out, this won't help those with linker scripts - that could
only be solved with a new DWARF standard and separate names for 64/32 bit
sections, at least if we wanted to avoid the linker needing to do anything
beyond reading the section header.
> >
> >
> >
> > The relocation approach sounds like a reasonable solution for the
current situation - even if we do decide to go the route of changing producers
to start emitting a new section type/update the standard etc, it doesn't
resolve the problem people may currently face.
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


-- 
宋方睿

On Fri, Nov 13, 2020 at 11:05 AM Fāng-ruì Sòng <maskray at google.com>
wrote:
>
> I got replies from Nick Clifton and Michael Matz:
> https://sourceware.org/pipermail/binutils/2020-November/114116.html
> (and its reply).
> I have mentioned (a) the difficulty of the
> detecting-DWARF64-by-first-relocation approach and (b) the section
> type approach  in my reply there
> https://sourceware.org/pipermail/binutils/2020-November/114125.html
>
> (a) My prototype has made me feel uneasy with this approach.
>
> <quote>
> In DWARF v4 or if .debug_str_offset is not used, it is a problem. A
> heuristic is: if an input section in a file is marked DWARF64, we mark
> all other .debug_* DWARF64. This makes me feel a bit uneasy because
> for an output section description
>
>    .debug_str 0 : { *(.debug_str) }
>
> Now the behavior of `*` (or, if we invent a `SORT_*` keyword) is also
> dependent on other output sections.
> </quote>
>
> (b)
> * It needs a section type (either a gABI one or a SHT_GNU_* in GNU
> ABI). Seeking for a gABI one is not that I think this is particularly
> related to gABI but that I don't want Solaris (which LLVM also
> supports) uses a different section type to unnecessarily cause
> friction on our implementation
If I'm understawding you correrctly you're suggesting the sorting
behavior would only be implemented if the input object file had some
new attributes in it designating which sections are debug info
sections?

I don't think that's a viable solution to the problem at hand, then -
if someone is able to update their toolchain and rebuild objects with
new attributes, they can probably update the build configuration of
those objects to build them with DWARF64 instead, avoiding the mixed
32/64 problem. I think the solution we're looking for would have to
work with existing precompiled object files using DWARF32 that are in
the wild today, without modification.
> * It needs a clarification on multiple output section descriptions
> with the same name.
> * It needs a linker script feature to match input sections by type.
>
> On Fri, Nov 13, 2020 at 9:52 AM David Blaikie <dblaikie at gmail.com>
wrote:
> >
> > On Fri, Nov 13, 2020 at 8:35 AM Wenlei He via llvm-dev
> > <llvm-dev at lists.llvm.org> wrote:
> > >
> > > >  Thinking about it, I wouldn't expect an LTO generated
object itself to have a mixture of DWARF32/64, although I guess the 32/64 bit
state could be encoded in the IR (I am not familiar enough with it to know if it
actually is or not). It might be necessary to find ways to configure LTO to
generate DWARF64, possibly via a link-time option.
> > >
> > >
> > >
> > > I don’t think we need to encode dwarf32/64 in IR as attribute for
each module. We’re not going to emit mixed dwarf32/64 for merged LTO module
anyways, so allowing each module to express its dwarf setting would only
introduce burden for LTO to deal with inconsistency (warning?) among input
modules. Having a linker switch to pass the setting from driver to LTO sounds
better to me.
> >
> > Usually the issue there is that existing build systems may be setup
> > only to pass such flags to the compilations, and not to the link
> > invocations - like DWARF version, we pass that down through IR, emit
> > warnings/errors when two IR modules with different DWARF versions are
> > linked together, and then emit only one (the higher, I believe) DWARF
> > version out the other end.
> >
> > We aren't 100% consistent on this "anything you could do
without LTO,
> > you shuold be able to do with LTO/passing the same flags to the same
> > actions" kind of strategy (eg: type units and DWARF compression
aren't
> > passed down through IR - if you want those you have to pass them to
> > the link invocation (via the clang driver) yourself). So it's more
"is
> > there a systemic use of these flags already for the compilation and
> > would not supporting it there be a pain"? It's probably not
for
> > DWARF64, since we haven't had any flag to support it at the moment
> > anyway.
> >
> > - Dave
> >
> > >
> > >
> > >
> > > From: llvm-dev <llvm-dev-bounces at lists.llvm.org>
> > > Date: Thursday, November 12, 2020 at 2:21 AM
> > > To: Fangrui Song <maskray at google.com>
> > > Cc: llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org>
> > > Subject: Re: [llvm-dev] [LLD] Support DWARF64, debug_info
"sorting"
> > >
> > >
> > >
> > >
> > >
> > > On Thu, 12 Nov 2020 at 02:10, Fangrui Song <maskray at
google.com> wrote:
> > >
> > > On 2020-11-12, Alexander Yermolovich wrote:
> > > >Thanks for feedback.
> > > >
> > > >I agree with patch and numbers this will be a more concrete
discussion, but I wanted to judge overall receptiveness to this approach and see
maybe there was a better way.
> > > >
> > > >"Whilst the majority of objects will only have a single
CU in them, there will be exceptions (LTO-generated objects, -r merged objects
etc), so we do need to consider this approach."
> > > >David can you elaborate under which conditions LTO-generated
objects will have a mix of DWARF32/64 in same .debug_info? Looking at how
dwarf64 was implemented same flag will be used for the entirety of the dwarf
output, even if multiple CUs are included.
> > >
> > >
> > >
> > > Thinking about it, I wouldn't expect an LTO generated object
itself to have a mixture of DWARF32/64, although I guess the 32/64 bit state
could be encoded in the IR (I am not familiar enough with it to know if it
actually is or not). It might be necessary to find ways to configure LTO to
generate DWARF64, possibly via a link-time option.
> > >
> > >
> > >
> > > >
> > > >On one hand since this is only applicable for when DWARF64 is
used, special option would be the way to go. Although the user will need to be
aware of yet another LLD option. Maybe an error when relocations overflow occur
can be modified to display this option along with -fdebug-types-section
> > >
> > > I am quite happy with the relocation approach under a linker
option. I'd still
> > > want to know generic-abi folks's thoughts, though. James may
have prepared something
> > > he wants to share with generic-abi:) Let's wait...
> > >
> > >
> > >
> > > I hadn't prepared anything if I'm honest (though if
there's widespread agreement that this would be useful, I certainly can - it
would have other positive improvements too, reducing the need for tools to rely
on section names to identify debug data for example). It was more a case of
bouncing ideas off of people to see what they thought. Any discussion we have
will probably also need circulating on the DWARF mailing list too, since it is
more a DWARF issue than a gABI issue (unless the solution is a new section
type). Further refinements to this idea that might make it more appealing to the
generic group: `SHT_DEBUG` for the section type name, with the first N bytes of
the sh_info used to specify the variant of debug data it represents (e.g. 0x1
for DWARF, 0x2 for SOME_OTHER_STANDARD etc), and the remainder for use as flags
as defined by the standard (I'm thinking for DWARF you could encode the
64-bit/32-bit state in there, possibly the section variant (info/rnglists/line
etc) and the DWARF version too), on the understanding that consumers like the
linker wouldn't combine sections in a potentially broken way. This has the
advantage that it could be retrofitted to the existing standard versions, but as
has been pointed out, this won't help those with linker scripts - that could
only be solved with a new DWARF standard and separate names for 64/32 bit
sections, at least if we wanted to avoid the linker needing to do anything
beyond reading the section header.
> > >
> > >
> > >
> > > The relocation approach sounds like a reasonable solution for the
current situation - even if we do decide to go the route of changing producers
to start emitting a new section type/update the standard etc, it doesn't
resolve the problem people may currently face.
> > >
> > > _______________________________________________
> > > LLVM Developers mailing list
> > > llvm-dev at lists.llvm.org
> > > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> --
> 宋方睿

On Fri, Nov 13, 2020 at 11:17 AM David Blaikie <dblaikie at gmail.com>
wrote:
>
> On Fri, Nov 13, 2020 at 11:05 AM Fāng-ruì Sòng <maskray at
google.com> wrote:
> >
> > I got replies from Nick Clifton and Michael Matz:
> > https://sourceware.org/pipermail/binutils/2020-November/114116.html
> > (and its reply).
> > I have mentioned (a) the difficulty of the
> > detecting-DWARF64-by-first-relocation approach and (b) the section
> > type approach  in my reply there
> > https://sourceware.org/pipermail/binutils/2020-November/114125.html
> >
> > (a) My prototype has made me feel uneasy with this approach.
> >
> > <quote>
> > In DWARF v4 or if .debug_str_offset is not used, it is a problem. A
> > heuristic is: if an input section in a file is marked DWARF64, we mark
> > all other .debug_* DWARF64. This makes me feel a bit uneasy because
> > for an output section description
> >
> >    .debug_str 0 : { *(.debug_str) }
> >
> > Now the behavior of `*` (or, if we invent a `SORT_*` keyword) is also
> > dependent on other output sections.
> > </quote>
> >
> > (b)
> > * It needs a section type (either a gABI one or a SHT_GNU_* in GNU
> > ABI). Seeking for a gABI one is not that I think this is particularly
> > related to gABI but that I don't want Solaris (which LLVM also
> > supports) uses a different section type to unnecessarily cause
> > friction on our implementation
>
> If I'm understawding you correrctly you're suggesting the sorting
> behavior would only be implemented if the input object file had some
> new attributes in it designating which sections are debug info
> sections?
>
> I don't think that's a viable solution to the problem at hand, then
-
> if someone is able to update their toolchain and rebuild objects with
> new attributes, they can probably update the build configuration of
> those objects to build them with DWARF64 instead, avoiding the mixed
> 32/64 problem. I think the solution we're looking for would have to
> work with existing precompiled object files using DWARF32 that are in
> the wild today, without modification.
I know the "no-modification" requirement:) The first paragraph of
https://sourceware.org/pipermail/binutils/2020-November/114125.html
mentioned this.

The section type approach is used this way (in another paragraph):

<quote>
If we invent a keyword (say, TYPE) to match sections by type, we could use

   .debug_info     0 : { *(TYPE (SHT_PROGBITS)
.debug_info${RELOCATING+ .gnu.linkonce.wi.*}) }
   .debug_info     0 : { *(TYPE (SHT_GNU_DWARF64) .debug_info) }

or

   .debug_info     0 : { *(TYPE (SHT_PROGBITS)
.debug_info${RELOCATING+ .gnu.linkonce.wi.*} TYPE (SHT_GNU_DWARF64)
.debug_info) }
</quote>
> > * It needs a clarification on multiple output section descriptions
> > with the same name.
> > * It needs a linker script feature to match input sections by type.
> >
> > On Fri, Nov 13, 2020 at 9:52 AM David Blaikie <dblaikie at
gmail.com> wrote:
> > >
> > > On Fri, Nov 13, 2020 at 8:35 AM Wenlei He via llvm-dev
> > > <llvm-dev at lists.llvm.org> wrote:
> > > >
> > > > >  Thinking about it, I wouldn't expect an LTO
generated object itself to have a mixture of DWARF32/64, although I guess the
32/64 bit state could be encoded in the IR (I am not familiar enough with it to
know if it actually is or not). It might be necessary to find ways to configure
LTO to generate DWARF64, possibly via a link-time option.
> > > >
> > > >
> > > >
> > > > I don’t think we need to encode dwarf32/64 in IR as
attribute for each module. We’re not going to emit mixed dwarf32/64 for merged
LTO module anyways, so allowing each module to express its dwarf setting would
only introduce burden for LTO to deal with inconsistency (warning?) among input
modules. Having a linker switch to pass the setting from driver to LTO sounds
better to me.
> > >
> > > Usually the issue there is that existing build systems may be
setup
> > > only to pass such flags to the compilations, and not to the link
> > > invocations - like DWARF version, we pass that down through IR,
emit
> > > warnings/errors when two IR modules with different DWARF versions
are
> > > linked together, and then emit only one (the higher, I believe)
DWARF
> > > version out the other end.
> > >
> > > We aren't 100% consistent on this "anything you could do
without LTO,
> > > you shuold be able to do with LTO/passing the same flags to the
same
> > > actions" kind of strategy (eg: type units and DWARF
compression aren't
> > > passed down through IR - if you want those you have to pass them
to
> > > the link invocation (via the clang driver) yourself). So it's
more "is
> > > there a systemic use of these flags already for the compilation
and
> > > would not supporting it there be a pain"? It's probably
not for
> > > DWARF64, since we haven't had any flag to support it at the
moment
> > > anyway.
> > >
> > > - Dave
> > >
> > > >
> > > >
> > > >
> > > > From: llvm-dev <llvm-dev-bounces at lists.llvm.org>
> > > > Date: Thursday, November 12, 2020 at 2:21 AM
> > > > To: Fangrui Song <maskray at google.com>
> > > > Cc: llvm-dev at lists.llvm.org <llvm-dev at
lists.llvm.org>
> > > > Subject: Re: [llvm-dev] [LLD] Support DWARF64, debug_info
"sorting"
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, 12 Nov 2020 at 02:10, Fangrui Song <maskray at
google.com> wrote:
> > > >
> > > > On 2020-11-12, Alexander Yermolovich wrote:
> > > > >Thanks for feedback.
> > > > >
> > > > >I agree with patch and numbers this will be a more
concrete discussion, but I wanted to judge overall receptiveness to this
approach and see maybe there was a better way.
> > > > >
> > > > >"Whilst the majority of objects will only have a
single CU in them, there will be exceptions (LTO-generated objects, -r merged
objects etc), so we do need to consider this approach."
> > > > >David can you elaborate under which conditions
LTO-generated objects will have a mix of DWARF32/64 in same .debug_info? Looking
at how dwarf64 was implemented same flag will be used for the entirety of the
dwarf output, even if multiple CUs are included.
> > > >
> > > >
> > > >
> > > > Thinking about it, I wouldn't expect an LTO generated
object itself to have a mixture of DWARF32/64, although I guess the 32/64 bit
state could be encoded in the IR (I am not familiar enough with it to know if it
actually is or not). It might be necessary to find ways to configure LTO to
generate DWARF64, possibly via a link-time option.
> > > >
> > > >
> > > >
> > > > >
> > > > >On one hand since this is only applicable for when
DWARF64 is used, special option would be the way to go. Although the user will
need to be aware of yet another LLD option. Maybe an error when relocations
overflow occur can be modified to display this option along with
-fdebug-types-section
> > > >
> > > > I am quite happy with the relocation approach under a linker
option. I'd still
> > > > want to know generic-abi folks's thoughts, though. James
may have prepared something
> > > > he wants to share with generic-abi:) Let's wait...
> > > >
> > > >
> > > >
> > > > I hadn't prepared anything if I'm honest (though if
there's widespread agreement that this would be useful, I certainly can - it
would have other positive improvements too, reducing the need for tools to rely
on section names to identify debug data for example). It was more a case of
bouncing ideas off of people to see what they thought. Any discussion we have
will probably also need circulating on the DWARF mailing list too, since it is
more a DWARF issue than a gABI issue (unless the solution is a new section
type). Further refinements to this idea that might make it more appealing to the
generic group: `SHT_DEBUG` for the section type name, with the first N bytes of
the sh_info used to specify the variant of debug data it represents (e.g. 0x1
for DWARF, 0x2 for SOME_OTHER_STANDARD etc), and the remainder for use as flags
as defined by the standard (I'm thinking for DWARF you could encode the
64-bit/32-bit state in there, possibly the section variant (info/rnglists/line
etc) and the DWARF version too), on the understanding that consumers like the
linker wouldn't combine sections in a potentially broken way. This has the
advantage that it could be retrofitted to the existing standard versions, but as
has been pointed out, this won't help those with linker scripts - that could
only be solved with a new DWARF standard and separate names for 64/32 bit
sections, at least if we wanted to avoid the linker needing to do anything
beyond reading the section header.
> > > >
> > > >
> > > >
> > > > The relocation approach sounds like a reasonable solution
for the current situation - even if we do decide to go the route of changing
producers to start emitting a new section type/update the standard etc, it
doesn't resolve the problem people may currently face.
> > > >
> > > > _______________________________________________
> > > > LLVM Developers mailing list
> > > > llvm-dev at lists.llvm.org
> > > > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> >
> >
> > --
> > 宋方睿


-- 
宋方睿