llvm dev - Jan 2018 - Adding DWARF5 accelerator table support to llvm

Hello all,

In <https://reviews.llvm.org/D41986#977215> it was brought up that
there are at least two parties interested in having DWARF5 accelerator
tables implemented, so I'm writing this email to see if there's anyone
else interested in this topic, and to try to synchronize our efforts.

Our interest for this stems from a desire to make dwarf parsing fast
on non-apple targets (specifically android). As you may know, apple
targets already use a non-standard accelerator table format, which was
a precursor to the dwarf5 one. Due to other differences in debug info
deployment model, the apple tables are not directly applicable to
other targets, so instead of trying to make them fit, we chose to go
with the standard approach.

I personally will have some time to work on this this quarter or two,
and my plan is roughly the following:
1. add .debug_names support to llvm-dwarfdump via the DebugInfo
library (to enable testing of the table generation)
2. add .debug_names generation support (not enabled by default)
3. add .debug_names support to lldb
4. validate all three things work together
5. hook up .debug_names to clang's -glldb flag.
6. add .debug_names support to lld (accelerator table merging)

Right now I have (1) roughly implemented, and I think I'll be able to
put it up for review in a couple of days (although I expect the review
will go through several revisions before being accepted). I also have
a very basic implementation of (2), but this is still quite far from
being upstreamable.

So, my question is whether anyone is planning to work, or maybe
working already on dwarf5 accelerator tables? Help with reviewing
patches would also be greatly appreciated. If you have any questions
or concerns, let me know.

regards,
Pavel

Hello, I hope I will have time to help you with that. I discussed
dwarfv5 .debug_names implementation with involved party from RH. Anyway
even if can't help much could you keep me in the loop please?


On 01/17/2018 07:13 PM, Pavel Labath via llvm-dev wrote:
> Hello all,
>
> In <https://reviews.llvm.org/D41986#977215> it was brought up that
> there are at least two parties interested in having DWARF5 accelerator
> tables implemented, so I'm writing this email to see if there's
anyone
> else interested in this topic, and to try to synchronize our efforts.
>
> Our interest for this stems from a desire to make dwarf parsing fast
> on non-apple targets (specifically android). As you may know, apple
> targets already use a non-standard accelerator table format, which was
> a precursor to the dwarf5 one. Due to other differences in debug info
> deployment model, the apple tables are not directly applicable to
> other targets, so instead of trying to make them fit, we chose to go
> with the standard approach.
>
> I personally will have some time to work on this this quarter or two,
> and my plan is roughly the following:
> 1. add .debug_names support to llvm-dwarfdump via the DebugInfo
> library (to enable testing of the table generation)
> 2. add .debug_names generation support (not enabled by default)
> 3. add .debug_names support to lldb
> 4. validate all three things work together
> 5. hook up .debug_names to clang's -glldb flag.
> 6. add .debug_names support to lld (accelerator table merging)
>
> Right now I have (1) roughly implemented, and I think I'll be able to
> put it up for review in a couple of days (although I expect the review
> will go through several revisions before being accepted). I also have
> a very basic implementation of (2), but this is still quite far from
> being upstreamable.
>
> So, my question is whether anyone is planning to work, or maybe
> working already on dwarf5 accelerator tables? Help with reviewing
> patches would also be greatly appreciated. If you have any questions
> or concerns, let me know.
>
> regards,
> Pavel
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-- 
Best Regards,

Victor Leschuk | Software Engineer | Access Softek

Hi Pavel,

As mentioned by Adrian in the comment you linked, I too am looking at DWARFv5
accelerator tables in LLVM. 

To give you some background: my motivation is that I want to upstream support
for (Apple style) accelerator tables in llvm-dsymutil, which is currently
missing because the way they are generated is slightly different. As this
requires making changes the current code, I wanted to consider the impact on
the standard's tables. I have some old patches from Fred (Riss) that take
different
approaches to this, but I have nothing concrete yet for the v5 part.

Ultimately my goal is to share as much logic as possible between the Apple and
DWARFv5 implementations, both on the reading and generation side of things.

Your timeline seems sensible. Adding support in llvm-dwarfdump will be very
useful. I look forward to reviewing this. 

How do you feel about creating a [WIP] diff for (2). I'm very excited about
bundling our efforts on this. Let me know what I can do if there's anything
else I can help with. 

Cheers,
Jonas
> On 17 Jan 2018, at 16:40, Victor Leschuk <vleschuk at
accesssoftek.com> wrote:
> 
> Hello, I hope I will have time to help you with that. I discussed
> dwarfv5 .debug_names implementation with involved party from RH. Anyway
> even if can't help much could you keep me in the loop please?
> 
> 
> On 01/17/2018 07:13 PM, Pavel Labath via llvm-dev wrote:
>> Hello all,
>> 
>> In <https://reviews.llvm.org/D41986#977215> it was brought up
that
>> there are at least two parties interested in having DWARF5 accelerator
>> tables implemented, so I'm writing this email to see if there's
anyone
>> else interested in this topic, and to try to synchronize our efforts.
>> 
>> Our interest for this stems from a desire to make dwarf parsing fast
>> on non-apple targets (specifically android). As you may know, apple
>> targets already use a non-standard accelerator table format, which was
>> a precursor to the dwarf5 one. Due to other differences in debug info
>> deployment model, the apple tables are not directly applicable to
>> other targets, so instead of trying to make them fit, we chose to go
>> with the standard approach.
>> 
>> I personally will have some time to work on this this quarter or two,
>> and my plan is roughly the following:
>> 1. add .debug_names support to llvm-dwarfdump via the DebugInfo
>> library (to enable testing of the table generation)
>> 2. add .debug_names generation support (not enabled by default)
>> 3. add .debug_names support to lldb
>> 4. validate all three things work together
>> 5. hook up .debug_names to clang's -glldb flag.
>> 6. add .debug_names support to lld (accelerator table merging)
>> 
>> Right now I have (1) roughly implemented, and I think I'll be able
to
>> put it up for review in a couple of days (although I expect the review
>> will go through several revisions before being accepted). I also have
>> a very basic implementation of (2), but this is still quite far from
>> being upstreamable.
>> 
>> So, my question is whether anyone is planning to work, or maybe
>> working already on dwarf5 accelerator tables? Help with reviewing
>> patches would also be greatly appreciated. If you have any questions
>> or concerns, let me know.
>> 
>> regards,
>> Pavel
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> -- 
> Best Regards,
> 
> Victor Leschuk | Software Engineer | Access Softek
>

> On Jan 17, 2018, at 8:13 AM, Pavel Labath via lldb-dev <lldb-dev at
lists.llvm.org> wrote:
> 
> Hello all,
> 
> In <https://reviews.llvm.org/D41986#977215> it was brought up that
> there are at least two parties interested in having DWARF5 accelerator
> tables implemented, so I'm writing this email to see if there's
anyone
> else interested in this topic, and to try to synchronize our efforts.
> 
> Our interest for this stems from a desire to make dwarf parsing fast
> on non-apple targets (specifically android). As you may know, apple
> targets already use a non-standard accelerator table format, which was
> a precursor to the dwarf5 one. Due to other differences in debug info
> deployment model, the apple tables are not directly applicable to
> other targets, so instead of trying to make them fit, we chose to go
> with the standard approach.
> 
> I personally will have some time to work on this this quarter or two,
> and my plan is roughly the following:
> 1. add .debug_names support to llvm-dwarfdump via the DebugInfo
> library (to enable testing of the table generation)
> 2. add .debug_names generation support (not enabled by default)
> 3. add .debug_names support to lldb
> 4. validate all three things work together
> 5. hook up .debug_names to clang's -glldb flag.
> 6. add .debug_names support to lld (accelerator table merging)
> 
> Right now I have (1) roughly implemented, and I think I'll be able to
> put it up for review in a couple of days (although I expect the review
> will go through several revisions before being accepted). I also have
> a very basic implementation of (2), but this is still quite far from
> being upstreamable.
> 
> So, my question is whether anyone is planning to work, or maybe
> working already on dwarf5 accelerator tables? Help with reviewing
> patches would also be greatly appreciated. If you have any questions
> or concerns, let me know.

This sounds great. I would also vote for a few additions to the work:
- modify llvm-dwarfdump or llvm-dsymutil to be able to compute and add the Apple
and DWARF accelerator tables to existing object files that have DWARF in them.
As we are working out the bugs we might end up with compiler bugs where not
everything was added to the accelerator tables that was required. llvm-dsymutil
has a "--update" option that allows it to update and re-create the
Apple accelerator tables in a dSYM file. It would be nice to have this
functionality so that when we pull legacy DWARF files from a build server, we
have a simple way to update them with the needed indexing info so we can debug
them as quickly as new DWARF files.

Within LLDB, the DWARF parser will need to abstract the indexing lookups. Right
now we have each function that accesses the index doing:

  if (m_using_apple_tables) {
    if (m_apple_types_ap.get()) {
      const char *name_cstr = name.GetCString();
      m_apple_types_ap->FindByName(name_cstr, die_offsets);
    }
  } else {
    if (!m_indexed)
      Index();

    m_type_index.Find(name, die_offsets);
  }

We should abstract these out into a indexing base class and have one for manual
DWARF indexing, one for Apple accelerator table and one for .debug_names. I look
forward to seeing this in LLDB!

Greg

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180117/4f4650d3/attachment.html>

> -----Original Message-----
> From: Pavel Labath [mailto:labath at google.com]
> Sent: Wednesday, January 17, 2018 8:14 AM
> To: jdevlieghere at apple.com; LLVM Dev; LLDB; David Blaikie; Robinson,
Paul
> Subject: Adding DWARF5 accelerator table support to llvm
> 
> Hello all,
> 
> In <https://reviews.llvm.org/D41986#977215> it was brought up that
> there are at least two parties interested in having DWARF5 accelerator
> tables implemented, so I'm writing this email to see if there's
anyone
> else interested in this topic, and to try to synchronize our efforts.
> 
> Our interest for this stems from a desire to make dwarf parsing fast
> on non-apple targets (specifically android). As you may know, apple
> targets already use a non-standard accelerator table format, which was
> a precursor to the dwarf5 one. Due to other differences in debug info
> deployment model, the apple tables are not directly applicable to
> other targets, so instead of trying to make them fit, we chose to go
> with the standard approach.
> 
> I personally will have some time to work on this this quarter or two,
> and my plan is roughly the following:
> 1. add .debug_names support to llvm-dwarfdump via the DebugInfo
> library (to enable testing of the table generation)
> 2. add .debug_names generation support (not enabled by default)
> 3. add .debug_names support to lldb
> 4. validate all three things work together
> 5. hook up .debug_names to clang's -glldb flag.
> 6. add .debug_names support to lld (accelerator table merging)
> 
> Right now I have (1) roughly implemented, and I think I'll be able to
> put it up for review in a couple of days (although I expect the review
> will go through several revisions before being accepted). I also have
> a very basic implementation of (2), but this is still quite far from
> being upstreamable.
> 
> So, my question is whether anyone is planning to work, or maybe
> working already on dwarf5 accelerator tables? Help with reviewing
> patches would also be greatly appreciated. If you have any questions
> or concerns, let me know.
Hi Pavel,
This would not interfere/duplicate anything Sony is doing in the near
future.  I think having the accelerator tables available for our
debugger team to play with would be nice, and I will certainly try to
spend some time on reviews.

FTR, next thing on the Sony list will be the new range-list/loc-list
format.  We're really hoping to make `-gdwarf-5` a viable thing for
debuggers to try out by LLVM 7.0.  It won't have "everything" but
the
basic set of sections should be in place and be syntactically correct.

Thanks,
--paulr
> 
> regards,
> Pavel

On Wed, 17 Jan 2018 17:13:36 +0100, Pavel Labath via lldb-dev
wrote:
> so I'm writing this email to see if there's anyone
> else interested in this topic, and to try to synchronize our efforts.
I am sure interested in DWARF-5 .debug_names.  I wrote its producer+consumer
for GDB (but not producing/using DW_IDX_DIE_offset as GDB cannot use it).

> 1. add .debug_names support to llvm-dwarfdump via the DebugInfo
> library (to enable testing of the table generation)
FYI FSF binutils readelf can read .debug_names already for some possible
format cross-check (to prevent multiple incompatible implementations).

> 2. add .debug_names generation support (not enabled by default)
+
> I also have a very basic implementation of (2), but this is still quite far
> from being upstreamable.
Originally I expected I will reuse the GDB .debug_names producer for LLVM:
	https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commitdiff;h=437afbb81e3a04cbd933fc534a50c686eaa63b19
	https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commitdiff;h=8af5c486ea6153bb84b9257def4e5faa4bc72421

But I see you were faster.


Jan

On 30 January 2018 at 16:39, Jan Kratochvil <jan.kratochvil at redhat.com>
wrote:
> On Wed, 17 Jan 2018 17:13:36 +0100, Pavel Labath via lldb-dev wrote:
>> so I'm writing this email to see if there's anyone
>> else interested in this topic, and to try to synchronize our efforts.
>
> I am sure interested in DWARF-5 .debug_names.  I wrote its
producer+consumer
> for GDB (but not producing/using DW_IDX_DIE_offset as GDB cannot use it).
>
>
>> 1. add .debug_names support to llvm-dwarfdump via the DebugInfo
>> library (to enable testing of the table generation)
>
> FYI FSF binutils readelf can read .debug_names already for some possible
> format cross-check (to prevent multiple incompatible implementations).
>
>
>> 2. add .debug_names generation support (not enabled by default)
> +
>> I also have a very basic implementation of (2), but this is still quite
far
>> from being upstreamable.
>
> Originally I expected I will reuse the GDB .debug_names producer for LLVM:
>        
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commitdiff;h=437afbb81e3a04cbd933fc534a50c686eaa63b19
>        
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commitdiff;h=8af5c486ea6153bb84b9257def4e5faa4bc72421
>
> But I see you were faster.
>
>
> Jan
Thanks for the heads-up. I will study these to make sure we are compatible.


PS: I think people who wrote on this thread are CC'ed on the patch
already, but in case anyone is interested, I have created
<https://reviews.llvm.org/D42740> to implement a unicode-aware
case-insensitive hash according to the dwarf spec.