llvm dev - Jul 2018 - [DWARF] De-segregating type units and compile units

Hello DWARF fans,

I've just posted a set of four refactoring patches for DebugInfo/DWARF,
which move in the direction of handling DWARF v4 or v5 type units and
compile units more coherently.

In DWARF v4, type units and compile units are strictly segregated into
the .debug_types and .debug_info sections, respectively.  This division
was pretty ingrained into how DebugInfo/DWARF handled the units.

In DWARF v5, type units and compile units are all in the .debug_info
section, and the .debug_types section is obsolete.  So, we need to have
a container than can handle both kinds of units in a straightforward way.

The refactoring replaces a pair of collections templated on the unit type
with a single collection whose elements are base-class pointers; thus it
can contain elements that are a mix of unit kinds.  Everything that really
mattered was either already virtual or was straightforward to convert to
generic handling.  I think I needed only one type-based conditional, on
top of what already existed.  The code doesn't *quite* support DWARF v5
type units in the .debug_info section, but the new starting point should
make that straightforward.

In a mixed v4/v5 executable, we can pretend the .debug_types sections
are all really .debug_info sections, and tack them onto the end of the
vector of units that already handles a mix of compile and type units.
This is also how the LLDB DWARF parser handles this case, so it's at
least consistent in principle (even if the actual code differs).

The existing distinction between "normal" and split (DWO) units
remains;
that has to do with what information exists in which object files, and
is not fundamentally changing in DWARF v5.

Patch 1: De-templatize DWARFUnitSection
         https://reviews.llvm.org/D49741
Patch 2: The TU collection doesn't need to be a deque
         https://reviews.llvm.org/D49742
Patch 3: Rename DWARFUnitSection to DWARFUnitVector
         https://reviews.llvm.org/D49743
Patch 4: Unify handling of type and compile units
         https://reviews.llvm.org/D49744

Thanks,
--paulr

A few general design discussions that cross multiple patches, so figure
it's worth discussing here:

The DWARFUnitSection is currently (before your changes) representing the
contents from a single section - and most/all of the classes in
libDebugInfoDWARF use "parse" to initialize their complete state. So
it's a
bit novel/new/different/worth considering (maybe renaming the function or
the like) what it means to make this class now represent data from multiple
sections and to be documentend/intended to be used to aggregate the parsing
result from multiple sections (both multiple sections with the same name
(debug_types currently, or in v5 debug_info comdat sections for type
units)). One observable part of the matter of which sections come from is
the section relative offsets (you'll see the difference in dumping a file
with at least 2 types with type units enabled, with and without split DWARF
- without split DWARF comdat sections are used, so each types section
offsets are zerod at the start of each type unit because it's a separate
section (even though it has the same section name) - whereas in split
DWARF, in the .dwo file the offsets are across the whole, singular,
debug_types.dwo section)

As for virtuality in the unit hierarchy - reckon it's worth it, or maybe
just use a union/conditional for the dumping differences (& collapse/remove
the type distinction between type units and compile units - just have
DWARFUnit as the only unit class)? (since it's just the type_signature V
DWO_id (maybe we could generalize that to "id") and type_offset
that's
different between them).

On Tue, Jul 24, 2018 at 10:40 AM via llvm-dev <llvm-dev at lists.llvm.org>
wrote:
> Hello DWARF fans,
>
> I've just posted a set of four refactoring patches for DebugInfo/DWARF,
> which move in the direction of handling DWARF v4 or v5 type units and
> compile units more coherently.
>
> In DWARF v4, type units and compile units are strictly segregated into
> the .debug_types and .debug_info sections, respectively.  This division
> was pretty ingrained into how DebugInfo/DWARF handled the units.
>
> In DWARF v5, type units and compile units are all in the .debug_info
> section, and the .debug_types section is obsolete.  So, we need to have
> a container than can handle both kinds of units in a straightforward way.
>
> The refactoring replaces a pair of collections templated on the unit type
> with a single collection whose elements are base-class pointers; thus it
> can contain elements that are a mix of unit kinds.  Everything that really
> mattered was either already virtual or was straightforward to convert to
> generic handling.  I think I needed only one type-based conditional, on
> top of what already existed.  The code doesn't *quite* support DWARF v5
> type units in the .debug_info section, but the new starting point should
> make that straightforward.
>
> In a mixed v4/v5 executable, we can pretend the .debug_types sections
> are all really .debug_info sections, and tack them onto the end of the
> vector of units that already handles a mix of compile and type units.
> This is also how the LLDB DWARF parser handles this case, so it's at
> least consistent in principle (even if the actual code differs).
>
> The existing distinction between "normal" and split (DWO) units
remains;
> that has to do with what information exists in which object files, and
> is not fundamentally changing in DWARF v5.
>
> Patch 1: De-templatize DWARFUnitSection
>          https://reviews.llvm.org/D49741
> Patch 2: The TU collection doesn't need to be a deque
>          https://reviews.llvm.org/D49742
> Patch 3: Rename DWARFUnitSection to DWARFUnitVector
>          https://reviews.llvm.org/D49743
> Patch 4: Unify handling of type and compile units
>          https://reviews.llvm.org/D49744
>
> Thanks,
> --paulr
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180725/bd3eae87/attachment.html>

Here's a broader precis of my thinking in taking the library in this
direction.

Prior to v4, DWARF kept information of different natures in different
object-file sections.  So, .debug_line had the line tables, .debug_macinfo had
the macro information, .debug_info had the DIEs, and so on.  In some cases,
information in a single section was subdivided according to compilation unit
(.debug_info and .debug_line in particular) but the information in each section
was all of the same nature.

In v4, DWARF added .debug_types, which was different in two ways.  First, there
might be multiple object-file sections with the same name (the COMDATs), and
second, the nature of the information was much the same as in .debug_info. 
There was a distinction, however, in that .debug_info contained only compile
units and .debug_types contained only type units.  There's also the aspect
that the LLVM parser is oriented toward the needs of an object-file dumper,
which encouraged thinking about the information in object-file terms, i.e. by
section.  Hence, type units were supported by using a container of
section-oriented classes, each element being able to handle multiple units.

In v5, DWARF corrected the mistake of putting type units in a differently named
object-file section.  So, now we can have one or more .debug_info sections, each
of which can contain one or more compile and/or type units.  But of course
.debug_types still exists if there is some linked object file containing a mix
of v4 and 5 compilations.

In parallel, there is increased interest in making the LLVM and LLDB parsers
converge, and if the decision goes with using the LLVM parser in LLDB, the LLVM
parser needs to better accommodate the needs of a debugger rather than just a
dumper.  In that case, the important distinction is not the object-file section,
but the nature of the information.  That is, pasting together all the DIE data
from .debug_info and .debug_types into a single manageable whole is what LLDB
wants, because the source section is irrelevant.  And in fact, within the past
year LLDB's parser was modified to understand .debug_types, and models it as
being appended to .debug_info for navigation purposes.

Of course the LLVM dumper still thinks about things in object-file section
terms, so it's worth being able to look at the DIE information on a
per-section basis.

I think I have achieved that with the refactoring.  The DWARF data that is in
the nature of DIEs is managed by one container, making it easier for LLDB to
look at it as a unified whole.  The container can provide iterators that
distinguish between different section names, allowing the dumper to retain a
section-oriented view.  The functional patch that enables multiple .debug_info
sections for DWARF v5 should be not a whole lot more than changing the parse
methods to iterate over possibly multiple instances of object-file sections with
that name.

If there are bugs in how intra-section offsets are reported and handled, then
it's worth knowing; what that means is that we don't have existing tests
that demonstrate the offsets are done correctly.  Happy to improve testing and
fix any problems found.

If there is bikeshedding about the names of methods, that's fine too. 
Renaming DWARFUnitVector::parse[DWO] to parse[DWO]Section is obviously simple
and can be folded into the same patch that renamed the class.

As for merging DWARFTypeUnit and DWARFCompileUnit into DWARFUnit and eliminating
the distinction at the class level, I haven't really looked at that.  There
are a couple reasons to put that off.  First, Jan Kratochvil has been doing LLDB
work in the direction of more subclasses, not fewer, i.e. putting partial and
imported units into their own subclasses.  I haven't looked closely at his
patches.  Second, I've had interest internally in getting partial units to
work reasonably so we can put them into COMDATs and trivially deduplicate the
debug info for linkonce functions.  If a separate subclass would help there,
merging others would make less sense.

So I'm not fundamentally opposed to merging the unit classes, but I think it
can be deferred with little problem while we work out the other aspects.

Other questions/comments always welcome.
--paulr

From: David Blaikie [mailto:dblaikie at gmail.com]
Sent: Wednesday, July 25, 2018 3:36 PM
To: Robinson, Paul
Cc: llvm-dev at lists.llvm.org
Subject: Re: [llvm-dev] [DWARF] De-segregating type units and compile units

A few general design discussions that cross multiple patches, so figure it's
worth discussing here:

The DWARFUnitSection is currently (before your changes) representing the
contents from a single section - and most/all of the classes in
libDebugInfoDWARF use "parse" to initialize their complete state. So
it's a bit novel/new/different/worth considering (maybe renaming the
function or the like) what it means to make this class now represent data from
multiple sections and to be documentend/intended to be used to aggregate the
parsing result from multiple sections (both multiple sections with the same name
(debug_types currently, or in v5 debug_info comdat sections for type units)).
One observable part of the matter of which sections come from is the section
relative offsets (you'll see the difference in dumping a file with at least
2 types with type units enabled, with and without split DWARF - without split
DWARF comdat sections are used, so each types section offsets are zerod at the
start of each type unit because it's a separate section (even though it has
the same section name) - whereas in split DWARF, in the .dwo file the offsets
are across the whole, singular, debug_types.dwo section)

As for virtuality in the unit hierarchy - reckon it's worth it, or maybe
just use a union/conditional for the dumping differences (& collapse/remove
the type distinction between type units and compile units - just have DWARFUnit
as the only unit class)? (since it's just the type_signature V DWO_id (maybe
we could generalize that to "id") and type_offset that's different
between them).

On Tue, Jul 24, 2018 at 10:40 AM via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
Hello DWARF fans,

I've just posted a set of four refactoring patches for DebugInfo/DWARF,
which move in the direction of handling DWARF v4 or v5 type units and
compile units more coherently.

In DWARF v4, type units and compile units are strictly segregated into
the .debug_types and .debug_info sections, respectively.  This division
was pretty ingrained into how DebugInfo/DWARF handled the units.

In DWARF v5, type units and compile units are all in the .debug_info
section, and the .debug_types section is obsolete.  So, we need to have
a container than can handle both kinds of units in a straightforward way.

The refactoring replaces a pair of collections templated on the unit type
with a single collection whose elements are base-class pointers; thus it
can contain elements that are a mix of unit kinds.  Everything that really
mattered was either already virtual or was straightforward to convert to
generic handling.  I think I needed only one type-based conditional, on
top of what already existed.  The code doesn't *quite* support DWARF v5
type units in the .debug_info section, but the new starting point should
make that straightforward.

In a mixed v4/v5 executable, we can pretend the .debug_types sections
are all really .debug_info sections, and tack them onto the end of the
vector of units that already handles a mix of compile and type units.
This is also how the LLDB DWARF parser handles this case, so it's at
least consistent in principle (even if the actual code differs).

The existing distinction between "normal" and split (DWO) units
remains;
that has to do with what information exists in which object files, and
is not fundamentally changing in DWARF v5.

Patch 1: De-templatize DWARFUnitSection
         https://reviews.llvm.org/D49741
Patch 2: The TU collection doesn't need to be a deque
         https://reviews.llvm.org/D49742
Patch 3: Rename DWARFUnitSection to DWARFUnitVector
         https://reviews.llvm.org/D49743
Patch 4: Unify handling of type and compile units
         https://reviews.llvm.org/D49744

Thanks,
--paulr

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180726/c21e49a2/attachment-0001.html>