Fāng-ruì Sòng via llvm-dev
2020-Nov-17 07:05 UTC
[llvm-dev] [LLD] Support DWARF64, debug_info "sorting"
On Mon, Nov 16, 2020 at 10:42 PM Igor Kudrin <ikudrin at accesssoftek.com> wrote:> > On 14.11.2020 3:42, Fāng-ruì Sòng wrote: > > For .debug_* in object files: > > > > DWARF32 -> SHT_PROGBITS (unchanged) > > DWARF64 -> SHT_DWARF64 or SHT_GNU_DWARF64 > > > > In LLD, we will need to allow mixed SHT_PROGBITS and SHT_DWARF64. If > > all input sections are SHT_DWARF64, the output section type probably > > should also be SHT_DWARF64. > > If mixed, SHT_PROGBITS. > > I am not really sure that we need a new section type. This gets a small > simplification in one part of the linker but adds much more burden of > supporting that to all tools and specs around. And that support would be > required for a rarely used feature, which certainly results that support > to be partial and sometimes erroneous.This is not a small simplification. If we consider how we would address .debug_str with the relocation approach, it would be quite contrived. A reply I just made https://lists.llvm.org/pipermail/llvm-dev/2020-November/146673.html detailed why I don't like the conceptual model: the behavior is dependent on other sections. I also felt bad as I had to do string comparison on ".debug_" (https://reviews.llvm.org/D91404) I've chatted with gdb folks: gdb (gdb/dwarf2/read.c) is agnostic about the section type. (They just cannot handle multiple .debug_info sections.) SHT_PROGBITS is a default (or "use this if nothing more specific exists") section type that probably no tool will specifically check the type. For many tools, they don't understand DWARF64 yet - there is little they need to adapt when they add DWARF64 support. Currently the only problem I can think of is readelf -S (from my understanding of https://sourceware.org/pipermail/binutils/2020-November/114116.html , Nick will be happy if I write a GNU readelf patch to make the section header table dump look good:) )> If we aim for clarity and not ambiguity, I would suggest considering the > following. The DWARF specs designed so that the data of DWARF32 and > DWARF64 formats can be intermixed together in one section. What section > type should be used for that combination? Should it be another new > section type, something like SHT_DWARF32_DWARF64? Probably not, that > should be a regular SHT_PROGBITS. That means that DWARF64 data can be > contained in a SHT_PROGBITS section. Consequently, the section which > contains DWARF64 data without DWARF32 data should also be SHT_PROGBITS, > because the latter is just a special case of the former.I suggest we follow the `canMergeToProgbits` logic in LLD: mixed SHT_DWARF64 and SHT_PROGBITS get SHT_PROGBITS. This is the existing rule for many other section types. Conceptually, the combined section should impose the rigid restriction when it is further combined with other sections. This is probably an extra argument that we don't need SHT_DWARF32.> > FWIW I started a generic-abi thread > > https://groups.google.com/g/generic-abi/c/i2Xio-47QdQ ("Reserve a > > section type value for DWARF64") to give stakeholders from other ELF > > operating systems a chance to participate in the design. I have paid > > attention to my wording: a new section type is **not decided yet** on > > LLVM/GNU binutils sides. Our discussions on llvm-dev/binutils will > > benefit from agreement/disagreement from generic-abi. > > Thank you for conducting all these discussions. I hope we will finally > find a way to bring the feature alive.Thanks to James and Pavel for quoting the standard. https://groups.google.com/g/generic-abi/c/i2Xio-47QdQ It seems that we've made progress on the thread and may be able to get a generic value. I am following up with binutils folks https://sourceware.org/pipermail/binutils/2020-November/ I hope we can get to a design which satisfies all parties.> -- > Best Regards, > Igor Kudrin > C++ Developer, Access Softek, Inc.-- 宋方睿
James Henderson via llvm-dev
2020-Nov-17 08:17 UTC
[llvm-dev] [LLD] Support DWARF64, debug_info "sorting"
Thinking about it, it would probably be wise to raise this discussion on the DWARF mailing list too. They might want to put an addendum in the spec/DWARF wiki/somewhere appropriate along the lines of "ELF support for DWARF64", which can be retroactively applied to existing standards. The committee may also have some thoughts on how tools are expected to work with DWARF64 and DWARF32 mixtures. James On Tue, 17 Nov 2020 at 07:06, Fāng-ruì Sòng <maskray at google.com> wrote:> On Mon, Nov 16, 2020 at 10:42 PM Igor Kudrin <ikudrin at accesssoftek.com> > wrote: > > > > On 14.11.2020 3:42, Fāng-ruì Sòng wrote: > > > For .debug_* in object files: > > > > > > DWARF32 -> SHT_PROGBITS (unchanged) > > > DWARF64 -> SHT_DWARF64 or SHT_GNU_DWARF64 > > > > > > In LLD, we will need to allow mixed SHT_PROGBITS and SHT_DWARF64. If > > > all input sections are SHT_DWARF64, the output section type probably > > > should also be SHT_DWARF64. > > > If mixed, SHT_PROGBITS. > > > > I am not really sure that we need a new section type. This gets a small > > simplification in one part of the linker but adds much more burden of > > supporting that to all tools and specs around. And that support would be > > required for a rarely used feature, which certainly results that support > > to be partial and sometimes erroneous. > > This is not a small simplification. If we consider how we would > address .debug_str with the relocation approach, > it would be quite contrived. A reply I just made > https://lists.llvm.org/pipermail/llvm-dev/2020-November/146673.html > detailed why I don't like the conceptual model: the behavior is > dependent on other sections. > I also felt bad as I had to do string comparison on ".debug_" > (https://reviews.llvm.org/D91404) > > I've chatted with gdb folks: gdb (gdb/dwarf2/read.c) is agnostic about > the section type. (They just cannot handle multiple .debug_info > sections.) > SHT_PROGBITS is a default (or "use this if nothing more specific > exists") section type that probably no tool will specifically check > the type. > For many tools, they don't understand DWARF64 yet - there is little > they need to adapt when they add DWARF64 support. > Currently the only problem I can think of is readelf -S (from my > understanding of > https://sourceware.org/pipermail/binutils/2020-November/114116.html , > Nick will be happy if I write a GNU readelf patch to make the section > header table dump look good:) ) > > > If we aim for clarity and not ambiguity, I would suggest considering the > > following. The DWARF specs designed so that the data of DWARF32 and > > DWARF64 formats can be intermixed together in one section. What section > > type should be used for that combination? Should it be another new > > section type, something like SHT_DWARF32_DWARF64? Probably not, that > > should be a regular SHT_PROGBITS. That means that DWARF64 data can be > > contained in a SHT_PROGBITS section. Consequently, the section which > > contains DWARF64 data without DWARF32 data should also be SHT_PROGBITS, > > because the latter is just a special case of the former. > > I suggest we follow the `canMergeToProgbits` logic in LLD: mixed > SHT_DWARF64 and SHT_PROGBITS get SHT_PROGBITS. This is the existing > rule for many other section types. > Conceptually, the combined section should impose the rigid restriction > when it is further combined with other sections. > This is probably an extra argument that we don't need SHT_DWARF32. > > > > FWIW I started a generic-abi thread > > > https://groups.google.com/g/generic-abi/c/i2Xio-47QdQ ("Reserve a > > > section type value for DWARF64") to give stakeholders from other ELF > > > operating systems a chance to participate in the design. I have paid > > > attention to my wording: a new section type is **not decided yet** on > > > LLVM/GNU binutils sides. Our discussions on llvm-dev/binutils will > > > benefit from agreement/disagreement from generic-abi. > > > > Thank you for conducting all these discussions. I hope we will finally > > find a way to bring the feature alive. > > Thanks to James and Pavel for quoting the standard. > https://groups.google.com/g/generic-abi/c/i2Xio-47QdQ > It seems that we've made progress on the thread and may be able to get > a generic value. > I am following up with binutils folks > https://sourceware.org/pipermail/binutils/2020-November/ > I hope we can get to a design which satisfies all parties. > > > > -- > > Best Regards, > > Igor Kudrin > > C++ Developer, Access Softek, Inc. > > > > -- > 宋方睿 >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201117/2b28f618/attachment.html>
Igor Kudrin via llvm-dev
2020-Nov-17 09:05 UTC
[llvm-dev] [EXTERNAL] Re: [LLD] Support DWARF64, debug_info "sorting"
On 17.11.2020 14:05, Fāng-ruì Sòng wrote:> On Mon, Nov 16, 2020 at 10:42 PM Igor Kudrin <ikudrin at accesssoftek.com> wrote: >> >> On 14.11.2020 3:42, Fāng-ruì Sòng wrote: >>> For .debug_* in object files: >>> >>> DWARF32 -> SHT_PROGBITS (unchanged) >>> DWARF64 -> SHT_DWARF64 or SHT_GNU_DWARF64 >>> >>> In LLD, we will need to allow mixed SHT_PROGBITS and SHT_DWARF64. If >>> all input sections are SHT_DWARF64, the output section type probably >>> should also be SHT_DWARF64. >>> If mixed, SHT_PROGBITS. >> >> I am not really sure that we need a new section type. This gets a small >> simplification in one part of the linker but adds much more burden of >> supporting that to all tools and specs around. And that support would be >> required for a rarely used feature, which certainly results that support >> to be partial and sometimes erroneous. > > This is not a small simplification. If we consider how we would > address .debug_str with the relocation approach, > it would be quite contrived. A reply I just made > https://lists.llvm.org/pipermail/llvm-dev/2020-November/146673.html > detailed why I don't like the conceptual model: the behavior is > dependent on other sections.I do not see real issues in the dependency on other sections. Section 7.4, p. 198, DWARFv5 states: "The 32-bit and 64-bit DWARF format conventions must not be intermixed within a single compilation unit." From this, we can conclude that if a ".debug_info" section contains only 64-bit relocations, all other debug info sections in the object file are in the 64-bit DWARF format. This can be simplified to checking just the first relocation of this section, as you have made in https://reviews.llvm.org/D91404, and then spreading the assessment to all other debug sections in the object file. Sure, this heuristic will not work for some clumsy partially linked relocatable objects with a mixture of DWARF64 and DWARF32 data, but I guess they can be ignored for the first shot, because, firstly, their creation is usually under full control of the user and, secondly, the heuristic can be extended to check all the relocations if that will be ever necessary.> I also felt bad as I had to do string comparison on ".debug_" > (https://reviews.llvm.org/D91404)Well, that is a common way to find sections with debugging information, no?> I've chatted with gdb folks: gdb (gdb/dwarf2/read.c) is agnostic about > the section type. (They just cannot handle multiple .debug_info > sections.) > SHT_PROGBITS is a default (or "use this if nothing more specific > exists") section type that probably no tool will specifically check > the type. > For many tools, they don't understand DWARF64 yet - there is little > they need to adapt when they add DWARF64 support. > Currently the only problem I can think of is readelf -S (from my > understanding of > https://sourceware.org/pipermail/binutils/2020-November/114116.html , > Nick will be happy if I write a GNU readelf patch to make the section > header table dump look good:) )My main concern is that the new type is mostly useless for everyone, except that it provides a small hint for the linker, which is not very important because the same information can be easily and reliably gathered in place. Is that hint really so useful that deserves special support on the ELF specs level? -- Best Regards, Igor Kudrin C++ Developer, Access Softek, Inc.
Fāng-ruì Sòng via llvm-dev
2020-Nov-18 00:49 UTC
[llvm-dev] [LLD] Support DWARF64, debug_info "sorting"
On 2020-11-17, Igor Kudrin wrote:> >On 17.11.2020 14:05, Fāng-ruì Sòng wrote: >>On Mon, Nov 16, 2020 at 10:42 PM Igor Kudrin <ikudrin at accesssoftek.com> wrote: >>> >>>On 14.11.2020 3:42, Fāng-ruì Sòng wrote: >>>>For .debug_* in object files: >>>> >>>>DWARF32 -> SHT_PROGBITS (unchanged) >>>>DWARF64 -> SHT_DWARF64 or SHT_GNU_DWARF64 >>>> >>>>In LLD, we will need to allow mixed SHT_PROGBITS and SHT_DWARF64. If >>>>all input sections are SHT_DWARF64, the output section type probably >>>>should also be SHT_DWARF64. >>>>If mixed, SHT_PROGBITS. >>> >>>I am not really sure that we need a new section type. This gets a small >>>simplification in one part of the linker but adds much more burden of >>>supporting that to all tools and specs around. And that support would be >>>required for a rarely used feature, which certainly results that support >>>to be partial and sometimes erroneous. >> >>This is not a small simplification. If we consider how we would >>address .debug_str with the relocation approach, >>it would be quite contrived. A reply I just made >>https://lists.llvm.org/pipermail/llvm-dev/2020-November/146673.html >>detailed why I don't like the conceptual model: the behavior is >>dependent on other sections. > >I do not see real issues in the dependency on other sections. Section >7.4, p. 198, DWARFv5 states: "The 32-bit and 64-bit DWARF format >conventions must not be intermixed within a single compilation unit." >From this, we can conclude that if a ".debug_info" section contains >only 64-bit relocations, all other debug info sections in the object >file are in the 64-bit DWARF format. This can be simplified to >checking just the first relocation of this section, as you have made >in https://reviews.llvm.org/D91404, and then spreading the assessment >to all other debug sections in the object file. Sure, this heuristic >will not work for some clumsy partially linked relocatable objects >with a mixture of DWARF64 and DWARF32 data, but I guess they can be >ignored for the first shot, because, firstly, their creation is >usually under full control of the user and, secondly, the heuristic >can be extended to check all the relocations if that will be ever >necessary.I don't like the "if .debug_info looks like DWARF64, mark .debug_str approach" because it makes a section's behavior dependent on other sections in the input file, diverging a lot from the current way existing output section descriptions/input section descriptions are handled. This is about system consistency that I care a lot and really don't want to break giving that we have compelling alternative designs.>>I also felt bad as I had to do string comparison on ".debug_" >>(https://reviews.llvm.org/D91404) > >Well, that is a common way to find sections with debugging information, no? > >>I've chatted with gdb folks: gdb (gdb/dwarf2/read.c) is agnostic about >>the section type. (They just cannot handle multiple .debug_info >>sections.) >>SHT_PROGBITS is a default (or "use this if nothing more specific >>exists") section type that probably no tool will specifically check >>the type. >>For many tools, they don't understand DWARF64 yet - there is little >>they need to adapt when they add DWARF64 support. >>Currently the only problem I can think of is readelf -S (from my >>understanding of >>https://sourceware.org/pipermail/binutils/2020-November/114116.html , >>Nick will be happy if I write a GNU readelf patch to make the section >>header table dump look good:) ) > >My main concern is that the new type is mostly useless for everyone, >except that it provides a small hint for the linker, which is not very >important because the same information can be easily and reliably >gathered in place. Is that hint really so useful that deserves special >support on the ELF specs level?It is not useless. Avoiding string comparison on ".debug_" is one thing (sometimes this can improve performance a bit). The linker complexity is another. As I mentioned in my previous reply, "SHT_PROGBITS+SHT_DWARF64 => SHT_DWARF64" makes the scheme automatically work with relocatable links. There are 0x60000000 generic section type values and 0x10000000 OS specific values. The resource is abundant. On https://groups.google.com/g/generic-abi/c/i2Xio-47QdQ , Cary Coutant has made an alternative proposal by adding a new section flag. Personally I prefer a section type a section flag with reasons explained by Ali Bahrami. People are still arguing but I appreciate that the theme of the discussion is that people acknowledge the use cases and agree to use an appropriate ELF-level thing, instead of adding ad-hoc rules to the linker.
Fāng-ruì Sòng via llvm-dev
2020-Nov-18 06:07 UTC
[llvm-dev] [LLD] Support DWARF64, debug_info "sorting"
In https://groups.google.com/g/generic-abi/c/i2Xio-47QdQ (you need to join the group before making a post) Cary Coutant raised yet another idea: whether we can use ".debug64" as the section prefix. I like the idea because of: * It is immediately obvious whether DWARF64 is used and whether DWARF32 is used along with DWARF64. * In a relocatable link mixing DWARF32 and DWARF64 sections, DWARF32 and DWARF64 sections will naturally not get mixed. (For a relocation based approach, if DWARF64 is the first input section, the output may appear as a "DWARF64" because the proposed approach only checks the first relocation) On the other hand, * It is (slightly) non-conformant because of the different section names. * Tooling support. Some commonly used consumers have recognized DWARF64. We'll have to teach these tools about new section names. The number of sections to recognize has doubled. This may result in a fair amount of complexity (DWARFContext/MCObjectFileInfo/llvm-dwarfdump -debug* options/ld.lld --gdb-index are things I can immediately think of). On balance I think this is not as good as the section type idea. On Tue, Nov 17, 2020 at 12:17 AM James Henderson <jh7370.2008 at my.bristol.ac.uk> wrote:> > Thinking about it, it would probably be wise to raise this discussion on the DWARF mailing list too. They might want to put an addendum in the spec/DWARF wiki/somewhere appropriate along the lines of "ELF support for DWARF64", which can be retroactively applied to existing standards. The committee may also have some thoughts on how tools are expected to work with DWARF64 and DWARF32 mixtures. > > James > > On Tue, 17 Nov 2020 at 07:06, Fāng-ruì Sòng <maskray at google.com> wrote: >> >> On Mon, Nov 16, 2020 at 10:42 PM Igor Kudrin <ikudrin at accesssoftek.com> wrote: >> > >> > On 14.11.2020 3:42, Fāng-ruì Sòng wrote: >> > > For .debug_* in object files: >> > > >> > > DWARF32 -> SHT_PROGBITS (unchanged) >> > > DWARF64 -> SHT_DWARF64 or SHT_GNU_DWARF64 >> > > >> > > In LLD, we will need to allow mixed SHT_PROGBITS and SHT_DWARF64. If >> > > all input sections are SHT_DWARF64, the output section type probably >> > > should also be SHT_DWARF64. >> > > If mixed, SHT_PROGBITS. >> > >> > I am not really sure that we need a new section type. This gets a small >> > simplification in one part of the linker but adds much more burden of >> > supporting that to all tools and specs around. And that support would be >> > required for a rarely used feature, which certainly results that support >> > to be partial and sometimes erroneous. >> >> This is not a small simplification. If we consider how we would >> address .debug_str with the relocation approach, >> it would be quite contrived. A reply I just made >> https://lists.llvm.org/pipermail/llvm-dev/2020-November/146673.html >> detailed why I don't like the conceptual model: the behavior is >> dependent on other sections. >> I also felt bad as I had to do string comparison on ".debug_" >> (https://reviews.llvm.org/D91404) >> >> I've chatted with gdb folks: gdb (gdb/dwarf2/read.c) is agnostic about >> the section type. (They just cannot handle multiple .debug_info >> sections.) >> SHT_PROGBITS is a default (or "use this if nothing more specific >> exists") section type that probably no tool will specifically check >> the type. >> For many tools, they don't understand DWARF64 yet - there is little >> they need to adapt when they add DWARF64 support. >> Currently the only problem I can think of is readelf -S (from my >> understanding of >> https://sourceware.org/pipermail/binutils/2020-November/114116.html , >> Nick will be happy if I write a GNU readelf patch to make the section >> header table dump look good:) ) >> >> > If we aim for clarity and not ambiguity, I would suggest considering the >> > following. The DWARF specs designed so that the data of DWARF32 and >> > DWARF64 formats can be intermixed together in one section. What section >> > type should be used for that combination? Should it be another new >> > section type, something like SHT_DWARF32_DWARF64? Probably not, that >> > should be a regular SHT_PROGBITS. That means that DWARF64 data can be >> > contained in a SHT_PROGBITS section. Consequently, the section which >> > contains DWARF64 data without DWARF32 data should also be SHT_PROGBITS, >> > because the latter is just a special case of the former. >> >> I suggest we follow the `canMergeToProgbits` logic in LLD: mixed >> SHT_DWARF64 and SHT_PROGBITS get SHT_PROGBITS. This is the existing >> rule for many other section types. >> Conceptually, the combined section should impose the rigid restriction >> when it is further combined with other sections. >> This is probably an extra argument that we don't need SHT_DWARF32. >> >> > > FWIW I started a generic-abi thread >> > > https://groups.google.com/g/generic-abi/c/i2Xio-47QdQ ("Reserve a >> > > section type value for DWARF64") to give stakeholders from other ELF >> > > operating systems a chance to participate in the design. I have paid >> > > attention to my wording: a new section type is **not decided yet** on >> > > LLVM/GNU binutils sides. Our discussions on llvm-dev/binutils will >> > > benefit from agreement/disagreement from generic-abi. >> > >> > Thank you for conducting all these discussions. I hope we will finally >> > find a way to bring the feature alive. >> >> Thanks to James and Pavel for quoting the standard. >> https://groups.google.com/g/generic-abi/c/i2Xio-47QdQ >> It seems that we've made progress on the thread and may be able to get >> a generic value. >> I am following up with binutils folks >> https://sourceware.org/pipermail/binutils/2020-November/ >> I hope we can get to a design which satisfies all parties. >> >> >> > -- >> > Best Regards, >> > Igor Kudrin >> > C++ Developer, Access Softek, Inc. >> >> >> >> -- >> 宋方睿-- 宋方睿
Cary Coutant via llvm-dev
2020-Nov-18 20:15 UTC
[llvm-dev] [LLD] Support DWARF64, debug_info "sorting"
> Thinking about it, it would probably be wise to raise this discussion on the DWARF mailing list too. They might want to put an addendum in the spec/DWARF wiki/somewhere appropriate along the lines of "ELF support for DWARF64", which can be retroactively applied to existing standards. The committee may also have some thoughts on how tools are expected to work with DWARF64 and DWARF32 mixtures.Eric, Paul, and I are members of the DWARF committee, so you can be assured that we will bring this issue up there! -cary -cary