Pavel Labath via llvm-dev
2020-Apr-24 13:21 UTC
[llvm-dev] [yaml2obj] GSoC-20: Add DWARF support to yaml2obj
(Reviving this thread because a code review reminded me I want to reply here. Sorry for the extremely long turnaround). On 31/03/2020 20:22, Fangrui Song wrote:> On 2020-03-31, Adrian Prantl via llvm-dev wrote: >> >> >>> On Mar 31, 2020, at 10:55 AM, David Blaikie <dblaikie at gmail.com> wrote: >>> >>> +1 to all that & cc'ing a few of the usual suspects as FYI >>> >>> On Tue, Mar 31, 2020 at 10:36 AM Pavel Labath via llvm-dev >>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> >>> For me personally, the ability to write/edit syntactically correct dwarf >>> easily is much more important than being able to generate "incorrect" >>> dwarf -- I'm perfectly happy to continue to write the latter in >>> assembly, but there is a lot that could be improved about the experience >>> of writing "correct" dwarf. Ideally, I'd have a mode where I can just >> >> Do we think that yaml2obj is the best format for this, or would >> high-level DWARF DIE assembler directives be a more useful abstraction >> level? If you think about the .loc directive, there is actually some >> prior art in assembler. >> >> -- adrian > > I also find YAML tests unwieldy but for some tests (especiall malformed) > we may have to use them because it is diffult for an assembly directive > to produce invalid output (invalid offset/relocation/string table/etc). > > An assembly syntax can be conciser than its YAML counterpart, e.g. to > describe a section: > > assembly: .section .foo,"a", at progbits > YAML: - Name: foo > Type: SHT_PROGBITS > Flags: [ SHF_ALLOC ] > > A symbol table entry is similar. A YAML entry usually takes several > lines of code. > > Another advantage of assembly syntax is that it is composable. To define a > local symbol: > > label: > > To make it global: > > .globl label > label: > > Some directives are more expressive, e.g. .file .loc > An assembler even supports some meta programming features (macros). The > syntax may be strange. > > We do need some composable directives to make DWARF tests easier to > write/read.I am agree with David that we shouldn't add first class DWARF-generation directives to the assembler just for the sake of writing tests. However, I do see the appeal of assembler metaprogramming, and I have used it on occasion when generating some DWARF. A separate utility with some (meta-)programming facilities could be interesting, though it would be an additional burden to maintain, and I am not sure it is really needed (in tests, repetition is often better than complex control flow). I am curious about your comment on invalid relocations et al. I can see how that is interesting for testing binary utilities (and I don't think anyone wants to take that away), but I am not sure how useful is that in the context of DWARF testing, except maybe in a couple of low-level DWARF tests (which could be done in the traditional elf yaml and the DWARF could be written as a blob of bytes). If you have some examples like that, I'd very much like to about it. regards, pl
Greg Clayton via llvm-dev
2020-Apr-26 21:17 UTC
[llvm-dev] [yaml2obj] GSoC-20: Add DWARF support to yaml2obj
Personally I generate DWARF with a python DWARF generator I wrote so I can make minimal test cases. I often can't get the compiler to emit what I need to test with DWARF. For example, I needed to create a compile unit with three functions: two of the functions need to look like they are dead stripped (with an address of zero) and they need to come first in the compile unit DW_AT_ranges. Also the compile unit needs to use a DW_AT_ranges attribute for the address ranges of each of the functions and I need to control the ordering. If I compile something like this from a source file, the compiler users a DW_AT_low_pc and DW_AT_high_pc and I have no DW_AT_ranges on the compile unit. I was reproducing a bug in "llvm-dwarfdump --verify" and I fixed it and in order to check in a test case for this that isn't the huge example I started with I need to create exact DWARF that is as I detailed above. I can't get a .S file for my linked executable, and producing such an executable in the first place is not easy. So I find generating hand crafted DWARF easier. Then I just obj2yaml it, and I have my reduced test case. So it can be hard to get the compiler to emit test cases that are close enough to what I need in order to modify them at the .S file level. That being said, the whole reason I wrote a DWARF generator was because the yaml stuff doesn't allow you to create a yaml file and edit it freely for reasons everyone else mentioned already (offsets mismatch, section sizes can't change easily, can't add attributes easily). Does anyone have a way to create the kind of binary in a .S file that I mentioned above? The DWARF I need is in https://reviews.llvm.org/D78782 <https://reviews.llvm.org/D78782> in the llvm/test/tools/obj2yaml/macho-DWARF-debug-ranges.yaml file on line 6 through line 31. Greg> On Apr 24, 2020, at 6:21 AM, Pavel Labath <pavel at labath.sk> wrote: > > (Reviving this thread because a code review reminded me I want to reply > here. Sorry for the extremely long turnaround). > > On 31/03/2020 20:22, Fangrui Song wrote: >> On 2020-03-31, Adrian Prantl via llvm-dev wrote: >>> >>> >>>> On Mar 31, 2020, at 10:55 AM, David Blaikie <dblaikie at gmail.com> wrote: >>>> >>>> +1 to all that & cc'ing a few of the usual suspects as FYI >>>> >>>> On Tue, Mar 31, 2020 at 10:36 AM Pavel Labath via llvm-dev >>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>>> >>>> >>>> For me personally, the ability to write/edit syntactically correct dwarf >>>> easily is much more important than being able to generate "incorrect" >>>> dwarf -- I'm perfectly happy to continue to write the latter in >>>> assembly, but there is a lot that could be improved about the experience >>>> of writing "correct" dwarf. Ideally, I'd have a mode where I can just >>> >>> Do we think that yaml2obj is the best format for this, or would >>> high-level DWARF DIE assembler directives be a more useful abstraction >>> level? If you think about the .loc directive, there is actually some >>> prior art in assembler. >>> >>> -- adrian >> >> I also find YAML tests unwieldy but for some tests (especiall malformed) >> we may have to use them because it is diffult for an assembly directive >> to produce invalid output (invalid offset/relocation/string table/etc). >> >> An assembly syntax can be conciser than its YAML counterpart, e.g. to >> describe a section: >> >> assembly: .section .foo,"a", at progbits >> YAML: - Name: foo >> Type: SHT_PROGBITS >> Flags: [ SHF_ALLOC ] >> >> A symbol table entry is similar. A YAML entry usually takes several >> lines of code. >> >> Another advantage of assembly syntax is that it is composable. To define a >> local symbol: >> >> label: >> >> To make it global: >> >> .globl label >> label: >> >> Some directives are more expressive, e.g. .file .loc >> An assembler even supports some meta programming features (macros). The >> syntax may be strange. >> >> We do need some composable directives to make DWARF tests easier to >> write/read. > > I am agree with David that we shouldn't add first class DWARF-generation > directives to the assembler just for the sake of writing tests. > > However, I do see the appeal of assembler metaprogramming, and I have > used it on occasion when generating some DWARF. A separate utility with > some (meta-)programming facilities could be interesting, though it would > be an additional burden to maintain, and I am not sure it is really > needed (in tests, repetition is often better than complex control flow). > > I am curious about your comment on invalid relocations et al. I can see > how that is interesting for testing binary utilities (and I don't think > anyone wants to take that away), but I am not sure how useful is that in > the context of DWARF testing, except maybe in a couple of low-level > DWARF tests (which could be done in the traditional elf yaml and the > DWARF could be written as a blob of bytes). If you have some examples > like that, I'd very much like to about it. > > regards, > pl-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200426/90e73041/attachment.html>
James Henderson via llvm-dev
2020-Apr-27 08:10 UTC
[llvm-dev] [yaml2obj] GSoC-20: Add DWARF support to yaml2obj
I believe the compiler will generate a .debug_ranges section if you use -ffunction-sections, since the addresses of sections will be non-contiguous. From there, you should be able to edit the .debug_ranges assembly as needed (replace references to symbols with 0s in the .debug_ranges content) to get the exact behaviour you want (I'm assuming you don't want to have to hand-edit a .debug_abbrev/.debug_info data structure to manually create a .debug_ranges). On Sun, 26 Apr 2020 at 22:17, Greg Clayton via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Personally I generate DWARF with a python DWARF generator I wrote so I can > make minimal test cases. I often can't get the compiler to emit what I need > to test with DWARF. For example, I needed to create a compile unit with > three functions: two of the functions need to look like they are dead > stripped (with an address of zero) and they need to come first in the > compile unit DW_AT_ranges. Also the compile unit needs to use a > DW_AT_ranges attribute for the address ranges of each of the functions and > I need to control the ordering. If I compile something like this from a > source file, the compiler users a DW_AT_low_pc and DW_AT_high_pc and I have > no DW_AT_ranges on the compile unit. > > I was reproducing a bug in "llvm-dwarfdump --verify" and I fixed it and in > order to check in a test case for this that isn't the huge example I > started with I need to create exact DWARF that is as I detailed above. I > can't get a .S file for my linked executable, and producing such an > executable in the first place is not easy. So I find generating hand > crafted DWARF easier. Then I just obj2yaml it, and I have my reduced test > case. So it can be hard to get the compiler to emit test cases that are > close enough to what I need in order to modify them at the .S file level. > > That being said, the whole reason I wrote a DWARF generator was because > the yaml stuff doesn't allow you to create a yaml file and edit it freely > for reasons everyone else mentioned already (offsets mismatch, section > sizes can't change easily, can't add attributes easily). > > Does anyone have a way to create the kind of binary in a .S file that I > mentioned above? The DWARF I need is in https://reviews.llvm.org/D78782 in > the llvm/test/tools/obj2yaml/macho-DWARF-debug-ranges.yaml file on line 6 > through line 31. > > Greg > > > On Apr 24, 2020, at 6:21 AM, Pavel Labath <pavel at labath.sk> wrote: > > (Reviving this thread because a code review reminded me I want to reply > here. Sorry for the extremely long turnaround). > > On 31/03/2020 20:22, Fangrui Song wrote: > > On 2020-03-31, Adrian Prantl via llvm-dev wrote: > > > > On Mar 31, 2020, at 10:55 AM, David Blaikie <dblaikie at gmail.com> wrote: > > +1 to all that & cc'ing a few of the usual suspects as FYI > > On Tue, Mar 31, 2020 at 10:36 AM Pavel Labath via llvm-dev > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org > <llvm-dev at lists.llvm.org>>> wrote: > > > For me personally, the ability to write/edit syntactically correct dwarf > easily is much more important than being able to generate "incorrect" > dwarf -- I'm perfectly happy to continue to write the latter in > assembly, but there is a lot that could be improved about the experience > of writing "correct" dwarf. Ideally, I'd have a mode where I can just > > > Do we think that yaml2obj is the best format for this, or would > high-level DWARF DIE assembler directives be a more useful abstraction > level? If you think about the .loc directive, there is actually some > prior art in assembler. > > -- adrian > > > I also find YAML tests unwieldy but for some tests (especiall malformed) > we may have to use them because it is diffult for an assembly directive > to produce invalid output (invalid offset/relocation/string table/etc). > > An assembly syntax can be conciser than its YAML counterpart, e.g. to > describe a section: > > assembly: .section .foo,"a", at progbits > YAML: - Name: foo > Type: SHT_PROGBITS > Flags: [ SHF_ALLOC ] > > A symbol table entry is similar. A YAML entry usually takes several > lines of code. > > Another advantage of assembly syntax is that it is composable. To define a > local symbol: > > label: > > To make it global: > > .globl label > label: > > Some directives are more expressive, e.g. .file .loc > An assembler even supports some meta programming features (macros). The > syntax may be strange. > > We do need some composable directives to make DWARF tests easier to > write/read. > > > I am agree with David that we shouldn't add first class DWARF-generation > directives to the assembler just for the sake of writing tests. > > However, I do see the appeal of assembler metaprogramming, and I have > used it on occasion when generating some DWARF. A separate utility with > some (meta-)programming facilities could be interesting, though it would > be an additional burden to maintain, and I am not sure it is really > needed (in tests, repetition is often better than complex control flow). > > I am curious about your comment on invalid relocations et al. I can see > how that is interesting for testing binary utilities (and I don't think > anyone wants to take that away), but I am not sure how useful is that in > the context of DWARF testing, except maybe in a couple of low-level > DWARF tests (which could be done in the traditional elf yaml and the > DWARF could be written as a blob of bytes). If you have some examples > like that, I'd very much like to about it. > > regards, > pl > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200427/5feeef21/attachment.html>