So Dehao and I have been dealing with some of the nitty gritty details of debug info with ThinLTO, specifically with Fission(Split DWARF). This applies to LTO as well, so I won't single out ThinLTO here. 1) Multiple CUs in a .dwo file Clang/LLVM produces a CU for each original source file - these CUs are kept through IR linking (thin or full) and produced as distinct CUs in the resulting DWARF. This preserves semantics as much as possible - eg: file-local functions and types (those in anonymous namespaces or declared file-static) con co-exist even if their names collide. GDB does good things with this, except with Fission. Binutils DWP produces usable DWPs from DWO files with multiple CUs 2) Cross-CU references This is where it gets trickier. LLVM produces cross-CU references (DW_FORM_ref_addr) to refer to types or functions defined in one CU used from another. This only happens with (thin or plain) LTO. LLVM's had this for a while. This helps fully express interesting cases outlined in (1) - it's possible that a file-local function is inlined into another CU - by using ref_addr the semantics described in (1) can be preserved, while also explaining the inlining that has occurred. (similar cases can arise with intra-CU type references too) GDB handles this, except with Fission (I mean, it doesn't get this far - but even with patches to handle (1)+Fission, it's still not enough - needs more work) Binutils DWP and the DWP format in general... maybe can't cope with this. Talking about (2)+DWP: It looks like this may require adjusting at least the semantics, if not the syntax of the indexes in the DWP file. I've started a thread on dwarf-discuss to start trying to hash that out. So, to cut a long story short: Fission+LTO is currently unusable and may take a while to make the DWARF side of this usable. In the interim, I'd like to propose adding a flag to LLVM to support something not very nice: When merging modules (or importing things into a module in ThinLTO), do not maintain separate CUs but redirect the cu pointer in any DISubprogram (& probably DIGlobalVariable too - I forget if ThinLTO can import them) metadata to refer to the original CU in the destination module. (in the full LTO case, it'd also be necessary to import any retained types) This flag could be passed to the task doing the merging/importing, or could be passed to the frontend compile and stashed in per-CU metadata (where it would be respected when importing/merging from that CU). The intent would be to support this flag until the DWARF functionality can be decided on and implemented - and kept for some time after that for backwards compatibility for those who want to use Fission but haven't got the latest toolchains with the fixes/improvements in them. How's this sound to everyone? Reasonable? Unreasonable? Need more details about the impact of these changes, etc? (I have examples with DWARF output, GDB behavior, etc) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170503/b5d1a8df/attachment.html>
> On May 3, 2017, at 2:00 PM, David Blaikie <dblaikie at gmail.com> wrote: > > So Dehao and I have been dealing with some of the nitty gritty details of debug info with ThinLTO, specifically with Fission(Split DWARF). > > This applies to LTO as well, so I won't single out ThinLTO here. > > 1) Multiple CUs in a .dwo file > Clang/LLVM produces a CU for each original source file - these CUs are kept through IR linking (thin or full) and produced as distinct CUs in the resulting DWARF. > This preserves semantics as much as possible - eg: file-local functions and types (those in anonymous namespaces or declared file-static) con co-exist even if their names collide. > GDB does good things with this, except with Fission.Can you lay out the terminology you are using here? Is Fission as used here different/more than creating .dwo files?> Binutils DWP produces usable DWPs from DWO files with multiple CUs > > 2) Cross-CU references > This is where it gets trickier. > LLVM produces cross-CU references (DW_FORM_ref_addr) to refer to types or functions defined in one CU used from another. This only happens with (thin or plain) LTO. LLVM's had this for a while. > This helps fully express interesting cases outlined in (1) - it's possible that a file-local function is inlined into another CU - by using ref_addr the semantics described in (1) can be preserved, while also explaining the inlining that has occurred. (similar cases can arise with intra-CU type references too) > GDB handles this, except with Fission (I mean, it doesn't get this far - but even with patches to handle (1)+Fission, it's still not enough - needs more work) > Binutils DWP and the DWP format in general... maybe can't cope with this. > > Talking about (2)+DWP:You mean (2)+DWO+DWP?> It looks like this may require adjusting at least the semantics, if not the syntax of the indexes in the DWP file. I've started a thread on dwarf-discuss to start trying to hash that out. > > > > So, to cut a long story short: Fission+LTO is currently unusable and may take a while to make the DWARF side of this usable. > > In the interim, I'd like to propose adding a flag to LLVM to support something not very nice: When merging modules (or importing things into a module in ThinLTO), do not maintain separate CUs but redirect the cu pointer in any DISubprogram (& probably DIGlobalVariable too - I forget if ThinLTO can import them) metadata to refer to the original CU in the destination module. (in the full LTO case, it'd also be necessary to import any retained types) > > This flag could be passed to the task doing the merging/importing, or could be passed to the frontend compile and stashed in per-CU metadata (where it would be respected when importing/merging from that CU). > > The intent would be to support this flag until the DWARF functionality can be decided on and implemented - and kept for some time after that for backwards compatibility for those who want to use Fission but haven't got the latest toolchains with the fixes/improvements in them. > > > > How's this sound to everyone? Reasonable? Unreasonable? Need more details about the impact of these changes, etc? (I have examples with DWARF output, GDB behavior, etc)If you can post an example, that would help. thanks, adrian
On Wed, May 3, 2017 at 2:09 PM Adrian Prantl <aprantl at apple.com> wrote:> > > On May 3, 2017, at 2:00 PM, David Blaikie <dblaikie at gmail.com> wrote: > > > > So Dehao and I have been dealing with some of the nitty gritty details > of debug info with ThinLTO, specifically with Fission(Split DWARF). > > > > This applies to LTO as well, so I won't single out ThinLTO here. > > > > 1) Multiple CUs in a .dwo file > > Clang/LLVM produces a CU for each original source file - these CUs are > kept through IR linking (thin or full) and produced as distinct CUs in the > resulting DWARF. > > This preserves semantics as much as possible - eg: file-local functions > and types (those in anonymous namespaces or declared file-static) con > co-exist even if their names collide. > > GDB does good things with this, except with Fission. > > Can you lay out the terminology you are using here? Is Fission as used > here different/more than creating .dwo files? >Fission as used in that sentence means .dwo files and anything beyond (.dwp files - https://gcc.gnu.org/wiki/DebugFissionDWP ). GDB warns/errors/complains if two CUs are in a single .dwo, and ignores all but the first.> > > Binutils DWP produces usable DWPs from DWO files with multiple CUs > > > > 2) Cross-CU references > > This is where it gets trickier. > > LLVM produces cross-CU references (DW_FORM_ref_addr) to refer to types > or functions defined in one CU used from another. This only happens with > (thin or plain) LTO. LLVM's had this for a while. > > This helps fully express interesting cases outlined in (1) - it's > possible that a file-local function is inlined into another CU - by using > ref_addr the semantics described in (1) can be preserved, while also > explaining the inlining that has occurred. (similar cases can arise with > intra-CU type references too) > > GDB handles this, except with Fission (I mean, it doesn't get this far - > but even with patches to handle (1)+Fission, it's still not enough - needs > more work) > > Binutils DWP and the DWP format in general... maybe can't cope with this. > > > > Talking about (2)+DWP: > > You mean (2)+DWO+DWP? >DWP is a package containing the contents of multiple DWOs - so, yes and no? Not sure how to answer. (2)+DWO could be made to work without changing much in the contents/format/etc, I think - consumers could be made to understand the 'obvious' form (no change to producers I think would be needed). But once you go to a DWP file, then there are repreesntational problems that make it currently impossible to use cross-CU references. The semantics (& possibly the syntax) of DWP files would have to change to enable this functionality.> > > It looks like this may require adjusting at least the semantics, if not > the syntax of the indexes in the DWP file. I've started a thread on > dwarf-discuss to start trying to hash that out. > > > > > > > > So, to cut a long story short: Fission+LTO is currently unusable and may > take a while to make the DWARF side of this usable. > > > > In the interim, I'd like to propose adding a flag to LLVM to support > something not very nice: When merging modules (or importing things into a > module in ThinLTO), do not maintain separate CUs but redirect the cu > pointer in any DISubprogram (& probably DIGlobalVariable too - I forget if > ThinLTO can import them) metadata to refer to the original CU in the > destination module. (in the full LTO case, it'd also be necessary to import > any retained types) > > > > This flag could be passed to the task doing the merging/importing, or > could be passed to the frontend compile and stashed in per-CU metadata > (where it would be respected when importing/merging from that CU). > > > > The intent would be to support this flag until the DWARF functionality > can be decided on and implemented - and kept for some time after that for > backwards compatibility for those who want to use Fission but haven't got > the latest toolchains with the fixes/improvements in them. > > > > > > > > How's this sound to everyone? Reasonable? Unreasonable? Need more > details about the impact of these changes, etc? (I have examples with DWARF > output, GDB behavior, etc) > > If you can post an example, that would help. >OK, so, here's an example of how GDB, without fission, benefits from the two distinct CUs (rather than bundling all the types & functions into a single CU): given: a.cpp: namespace { struct foo { int i; }; } static void f1() { } void f2() { f1(); } b.cpp: namespace { struct foo { float f; }; } static void f1() { } void f2(); int main() { f1(); f2(); } $ clang++ {a,b}.cpp -g -c -emit-llvm -S $ llvm-link {a,b}.ll -S -o ab.ll $ clang++-tot ab.ll $ gdb a.out (gdb) start ..., main () at b.cpp:11 11 f1(); (gdb) ptype foo type = struct (anonymous namespace)::foo { float f; } (gdb) p &f1 $1 = (void (*)(void)) 0x400530 <f1()> (gdb) n 12 f2(); (gdb) s f2 () at a.cpp:10 10 f1(); (gdb) ptype foo type = struct (anonymous namespace)::foo { int i; } (gdb) p &f1 $2 = (void (*)(void)) 0x400500 <f1()> So in that case, the foo type and f1 function are properly identified depending on the context in which the expression is evaluated. If the types and subprograms are tied into the same CU (with a bit of manual IR editing), two distinct types and functions are emitted into the DWARF, but GDB only finds the first one, every time, in both contexts. To look into the larger motivation, cross-CU inlining and how it interacts with Fission: a.cpp: __attribute__((optnone)) void f1() { } __attribute__((always_inline)) void f2() { f1(); }; b.cpp: void f2(); void f3() { f2(); } c.cpp: void f3(); int main() { f3(); } $ clang++ {a,b}.cpp -g -c -emit-llvm -S $ llvm-link {a,b}.ll -S -o ab.ll $ clang++-tot c.cpp ab.ll -gsplit-dwarf $ dwp {c,ab}.dwo -o cab.dwp This produces two dwo files - one per object file (c.dwo, ab.dwo). If we look at the contents, they make a fair bit of sense: 0x0b: DW_TAG_compile_unit [1] * 0x19: DW_TAG_subprogram [2] 0x25: DW_TAG_subprogram [3] 0x31: DW_TAG_subprogram [4] 0x43: DW_TAG_compile_unit [1] * 0x51: DW_TAG_subprogram [5] * 0x5d: DW_TAG_inlined_subroutine [6] DW_AT_abstract_origin [DW_FORM_ref_addr] (0x31 "_Z2f2v") The ref_addr can be resolved relative to this whole .dwo file & the abstract subprogram is found. Yay. So GDB can't currently handle this: Could not find DWO CU ab.dwo(0x32dd6d7121dd1d9a) referenced by CU at offset 0x66 [in module a.out] And that's after some local fixes I have so it doesn't error out on the two-CUs-in-a-dwo situation (that fix at least allows the first example (local foo/f1, etc) to work with GDB - but fixing ref_addr to resolve correctly would require deeper changes I haven't figured out/prototyped yet). So, up until this point I was pretty satisfied there would be a way forward... Then I hit DWP files. DWP files represent multiple DWO files stacked together - mostly to deduplicate strings and type units as a sort of archival (like dsym) format. The way DWPs work, roughly (full/more details here: https://gcc.gnu.org/wiki/DebugFissionDWP ) is that all the sections are concatenated together, except the debug_str and debug_str_offsets sections which have special handling to do what you'd expect - deduplicate strings, and adjust the str_offsets to point to the right offsets in the deduplicated string section. The other thing that DWP has, is an index, or two. cu_index and tu_index. We'll just look at cu_index. A cu_index for the cab.dwp above, is: Index Signature INFO ABBR LINE STR_OFF ----- ------------------ -------- -------- -------- -------- 2 0x7bd765349b7e7631 [2d, 65) [38, ae) [11, 22) [14, 3c) 8 0x66f4e160661d2687 [00, 2d) [00, 38) [00, 11) [00, 14) 11 0x32dd6d7121dd1d9a [65, 98) [38, ae) [11, 22) [14, 3c) What this is for is to tell the consumer, which portions of each section relate to which CUs - since the DIEs in the CU don't contain any relocations, for example the abbrev offset in the CU header is probably zero. It must be resolved relative to the ABBR section in the above table to find the chunk of the debug_abbrev that came from that CU. So, this is all good and well, except that the INFO range isn't the whole range of the debug info from the DWO file - it's /just/ the range of this CU. Which means there's no way to resolve the ref_addr relative to the whole range, you don't know where it starts. So this leads us to a few options - the 'simplest', that changes the semantics but not the syntax of the table - would be to widen the INFO range to cover the whole portion that came from the original DWO file. Consumers would have to be adjusted to cope with the fact that the INFO contribution for a given DWO ID would be a range that contains the CU somewhere, along with other CUs - and they'd have to search (in full LTO, this would mean searching through /all/ the CUs). This doesn't cover all the use cases I'd have in mind, but it would at least be possible to implement & support ref_addr. The full depth of design choices I'll likely leave to the dwarf-discuss thread, hopefully. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170503/c9b46c56/attachment.html>
Rafael Avila de Espindola via llvm-dev
2017-May-04 14:22 UTC
[llvm-dev] DWARF Fission + ThinLTO
David Blaikie via llvm-dev <llvm-dev at lists.llvm.org> writes:> So Dehao and I have been dealing with some of the nitty gritty details of > debug info with ThinLTO, specifically with Fission(Split DWARF). > > This applies to LTO as well, so I won't single out ThinLTO here.When are the .dwo files produced? If at link time, what is the advantage of using fission? Normally the advantage is the faster link, but if it is the linker producing them, what is left? Cheers, Rafael
On Thu, May 4, 2017 at 7:22 AM, Rafael Avila de Espindola via llvm-dev < llvm-dev at lists.llvm.org> wrote:> David Blaikie via llvm-dev <llvm-dev at lists.llvm.org> writes: > > > So Dehao and I have been dealing with some of the nitty gritty details of > > debug info with ThinLTO, specifically with Fission(Split DWARF). > > > > This applies to LTO as well, so I won't single out ThinLTO here. > > When are the .dwo files produced? If at link time, what is the advantage > of using fission? Normally the advantage is the faster link, but if it > is the linker producing them, what is left? >For ThinLTO they are produced by the backends, and only the split .o part needs to be linked. I think that benefit would occur with in-process ThinLTO as well (the .o files handed back to the linker are smaller), but for us where we have a distributed ThinLTO build, the backends are producing split .o and .dwo files and so the final native link process gets all the normal advantages of fission. For regular LTO, I suppose the same benefit could occur - the backend produces a split .o and .dwo file and only passes the split .o file to the link. Teresa> Cheers, > Rafael > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-- Teresa Johnson | Software Engineer | tejohnson at google.com | 408-460-2413 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170504/e560bf0e/attachment.html>