llvm dev - Mar 2020 - Adding accelerator tables to existing linked DWARF files

I am looking to create a tool that can add Apple or DWARF5 accelerator tables to
fully linked executables that contain DWARF. This will help us benchmark how
much accelerator tables can improve the debugging experience as debuggers
don't need to manually index all of the debug info during debugging.

Looking at how accelerator tables are currently emitted, they seem to be built
up as DWARF is being created or linked, and then emitted using a subclass of
DWARFEmitter. The only subclass if this right now that I see is one in dsymutil
which ends up emitting everything using an AsmPrinter by eventually
emitAppleAccelTable(...) from llvm/include/llvm/CodeGen/AccelTable.h.

I spoke briefly with Shoaib on this subject and he suggested adding code to
llvm-objcopy. I briefly looked through the code and from what I can tell,
llvm-objcopy doesn't seem to have any DWARF abilities other than compressing
DWARF sections. If we do add functionality to llvm-objcopy, are we ok pulling in
DebugInfoDWARF and the LLVM object model? AFAICT the code for this tools has its
own object file layer which doesn't match the full layer inside of LLVM
(llvm::ObjectFile and DWARFContext). Also, no AsmPrinter objects are used in
this codebase either.

Looking at lld sources, is seems to use the DebugInfoDWARF library to some
extent already. Not sure if this tool uses the standard LLVM object model or has
all of its own emitters. Does lld use AsmPrinter at all? I don't see any
mention of it in there.

dsymutil has a --update feature which seems to load all of the DWARF and pretend
to link it all the while generating the new accelerator table data, but I fear
using this would pull it way too much code (AsmPrinter, all targets required to
load all object files types, the standard llvm object file model (not the lld or
llvm-objcopy versions), targets, etc).

My initial thoughts are:
1 - load a DWARFContext and iterate through the DWARF and build accelerator
table data
2 - create the sections for the accelerator tables and either keep in memory or
save to disk
3 - call functions to add the newly created sections to the binary

#1 should be easy as long as I can use a DWARFContext from DebugInfoDWARF. 
#2 might need to be re-implemented using something other than an AsmPrinter?
#3 can use llvm-objcopy code if needed since it can add sections?

Any advice on how this can or should be implemented would be appreciated from
anyone with experience.

Greg Clayton

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200228/1cd4f9f8/attachment.html>

On 2020-02-28, Greg Clayton via llvm-dev wrote:
>I am looking to create a tool that can add Apple or DWARF5 accelerator
tables to fully linked executables that contain DWARF. This will help us
benchmark how much accelerator tables can improve the debugging experience as
debuggers don't need to manually index all of the debug info during
debugging.
Is it for ELF, Mach-O, wasm, COFF, or any of the combinations?
>Looking at how accelerator tables are currently emitted, they seem to be
built up as DWARF is being created or linked, and then emitted using a subclass
of DWARFEmitter. The only subclass if this right now that I see is one in
dsymutil which ends up emitting everything using an AsmPrinter by eventually
emitAppleAccelTable(...) from llvm/include/llvm/CodeGen/AccelTable.h.
>
>I spoke briefly with Shoaib on this subject and he suggested adding code to
llvm-objcopy. I briefly looked through the code and from what I can tell,
llvm-objcopy doesn't seem to have any DWARF abilities other than compressing
DWARF sections. If we do add functionality to llvm-objcopy, are we ok pulling in
DebugInfoDWARF and the LLVM object model? AFAICT the code for this tools has its
own object file layer which doesn't match the full layer inside of LLVM
(llvm::ObjectFile and DWARFContext). Also, no AsmPrinter objects are used in
this codebase either.
llvm-objcopy supports various ad-hoc binary manipulation features where each
feature does a very
simple task. Neither llvm-objcopy nor GNU objcopy knows DWARF. --strip-debug,
--compress-debug-sections, --add-gnu-debuglink and --only-keep-debug have
"debug" in their names but
these features don't need to parse DWARF. (GNU objcopy has a --debugging but
that only works for
a.out and coff, not elf).

Do we have a more suitable tool for such debugging functionality? dsymutil for
ELF?
>Looking at lld sources, is seems to use the DebugInfoDWARF library to some
extent already. Not sure if this tool uses the standard LLVM object model or has
all of its own emitters. Does lld use AsmPrinter at all? I don't see any
mention of it in there.
--gdb-index and diagnostics (line tables) use DebugInfoDWARF. I have a plan to
implement .debug_names, which is similar to --gdb-index.
>dsymutil has a --update feature which seems to load all of the DWARF and
pretend to link it all the while generating the new accelerator table data, but
I fear using this would pull it way too much code (AsmPrinter, all targets
required to load all object files types, the standard llvm object file model
(not the lld or llvm-objcopy versions), targets, etc).
>
>My initial thoughts are:
>1 - load a DWARFContext and iterate through the DWARF and build accelerator
table data
>2 - create the sections for the accelerator tables and either keep in memory
or save to disk
>3 - call functions to add the newly created sections to the binary
>
>#1 should be easy as long as I can use a DWARFContext from DebugInfoDWARF.
>#2 might need to be re-implemented using something other than an AsmPrinter?
>#3 can use llvm-objcopy code if needed since it can add sections?
>
>Any advice on how this can or should be implemented would be appreciated
from anyone with experience.
>
>Greg Clayton
>
>_______________________________________________
>LLVM Developers mailing list
>llvm-dev at lists.llvm.org
>https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

FWIW I'm with Ray here :)

-eric

On Fri, Feb 28, 2020 at 11:25 PM Fangrui Song via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 2020-02-28, Greg Clayton via llvm-dev wrote:
> >I am looking to create a tool that can add Apple or DWARF5 accelerator
> tables to fully linked executables that contain DWARF. This will help us
> benchmark how much accelerator tables can improve the debugging experience
> as debuggers don't need to manually index all of the debug info during
> debugging.
> Is it for ELF, Mach-O, wasm, COFF, or any of the combinations?
>
> >Looking at how accelerator tables are currently emitted, they seem to
be
> built up as DWARF is being created or linked, and then emitted using a
> subclass of DWARFEmitter. The only subclass if this right now that I see is
> one in dsymutil which ends up emitting everything using an AsmPrinter by
> eventually emitAppleAccelTable(...) from
> llvm/include/llvm/CodeGen/AccelTable.h.
> >
> >I spoke briefly with Shoaib on this subject and he suggested adding
code
> to llvm-objcopy. I briefly looked through the code and from what I can
> tell, llvm-objcopy doesn't seem to have any DWARF abilities other than
> compressing DWARF sections. If we do add functionality to llvm-objcopy, are
> we ok pulling in DebugInfoDWARF and the LLVM object model? AFAICT the code
> for this tools has its own object file layer which doesn't match the
full
> layer inside of LLVM (llvm::ObjectFile and DWARFContext). Also, no
> AsmPrinter objects are used in this codebase either.
>
> llvm-objcopy supports various ad-hoc binary manipulation features where
> each feature does a very
> simple task. Neither llvm-objcopy nor GNU objcopy knows DWARF.
> --strip-debug,
> --compress-debug-sections, --add-gnu-debuglink and --only-keep-debug have
> "debug" in their names but
> these features don't need to parse DWARF. (GNU objcopy has a
--debugging
> but that only works for
> a.out and coff, not elf).
>
> Do we have a more suitable tool for such debugging functionality? dsymutil
> for ELF?
>
> >Looking at lld sources, is seems to use the DebugInfoDWARF library to
> some extent already. Not sure if this tool uses the standard LLVM object
> model or has all of its own emitters. Does lld use AsmPrinter at all? I
> don't see any mention of it in there.
>
> --gdb-index and diagnostics (line tables) use DebugInfoDWARF. I have a
> plan to implement .debug_names, which is similar to --gdb-index.
>
> >dsymutil has a --update feature which seems to load all of the DWARF
and
> pretend to link it all the while generating the new accelerator table data,
> but I fear using this would pull it way too much code (AsmPrinter, all
> targets required to load all object files types, the standard llvm object
> file model (not the lld or llvm-objcopy versions), targets, etc).
> >
> >My initial thoughts are:
> >1 - load a DWARFContext and iterate through the DWARF and build
> accelerator table data
> >2 - create the sections for the accelerator tables and either keep in
> memory or save to disk
> >3 - call functions to add the newly created sections to the binary
> >
> >#1 should be easy as long as I can use a DWARFContext from
DebugInfoDWARF.
> >#2 might need to be re-implemented using something other than an
> AsmPrinter?
> >#3 can use llvm-objcopy code if needed since it can add sections?
> >
> >Any advice on how this can or should be implemented would be
appreciated
> from anyone with experience.
> >
> >Greg Clayton
> >
>
> >_______________________________________________
> >LLVM Developers mailing list
> >llvm-dev at lists.llvm.org
> >https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200302/20b6246b/attachment.html>

> On Feb 28, 2020, at 11:25 PM, Fangrui Song via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> On 2020-02-28, Greg Clayton via llvm-dev wrote:
>> I am looking to create a tool that can add Apple or DWARF5 accelerator
tables to fully linked executables that contain DWARF. This will help us
benchmark how much accelerator tables can improve the debugging experience as
debuggers don't need to manually index all of the debug info during
debugging.
> Is it for ELF, Mach-O, wasm, COFF, or any of the combinations?
Yes, for any object files that LLVM currently supports. But I am looking to
support ELF first as MachO already has these tables available since dsymutil
already creates either Apple or DWARF accelerator tables. COFF and Wasm can come
later.
> 
>> Looking at how accelerator tables are currently emitted, they seem to
be built up as DWARF is being created or linked, and then emitted using a
subclass of DWARFEmitter. The only subclass if this right now that I see is one
in dsymutil which ends up emitting everything using an AsmPrinter by eventually
emitAppleAccelTable(...) from llvm/include/llvm/CodeGen/AccelTable.h.
>> 
>> I spoke briefly with Shoaib on this subject and he suggested adding
code to llvm-objcopy. I briefly looked through the code and from what I can
tell, llvm-objcopy doesn't seem to have any DWARF abilities other than
compressing DWARF sections. If we do add functionality to llvm-objcopy, are we
ok pulling in DebugInfoDWARF and the LLVM object model? AFAICT the code for this
tools has its own object file layer which doesn't match the full layer
inside of LLVM (llvm::ObjectFile and DWARFContext). Also, no AsmPrinter objects
are used in this codebase either.
> 
> llvm-objcopy supports various ad-hoc binary manipulation features where
each feature does a very
> simple task. Neither llvm-objcopy nor GNU objcopy knows DWARF.
--strip-debug,
> --compress-debug-sections, --add-gnu-debuglink and --only-keep-debug have
"debug" in their names but
> these features don't need to parse DWARF. (GNU objcopy has a
--debugging but that only works for
> a.out and coff, not elf).
> 
> Do we have a more suitable tool for such debugging functionality? dsymutil
for ELF?
dsymutil is such a tool, but it uses the llvm::ObjectFile layer and the llvm
targets, so if you open a file that contains "armv7" architecture and
the ARM target hasn't been built into your bistro, it will fail to open this
binary with an error that says:

No available targets are compatible with triple "arm-unknown-unknown"

I ran into this with a recent gsym patch that is trying to fix the buildbots for
testing, but it fails when the ARM targets are not enabled and I try to load the
DWARF from an object file:

http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast/builds/63473/steps/test-check-all/logs/stdio
<http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast/builds/63473/steps/test-check-all/logs/stdio>

And this is part of the reason for this email. I would love to not require
llvm-gsymutil to require all LLVM targets to be there. DebugInfoDWARF
doesn't need the targets, it just needs to know address byte size and
endianness and it can parse the debug info in the DWARF.

So my main question stands: do we want all tools that must manipulate DWARF to
require the llvm::ObjectFile layer and all of the targets to be enabled just so
that the object files can be parsed, or do we want to make lighter layer
available, akin to what llvm-objcopy has, so more tools can take advantage of
this lighter weight layer.
> 
>> Looking at lld sources, is seems to use the DebugInfoDWARF library to
some extent already. Not sure if this tool uses the standard LLVM object model
or has all of its own emitters. Does lld use AsmPrinter at all? I don't see
any mention of it in there.
> 
> --gdb-index and diagnostics (line tables) use DebugInfoDWARF. I have a plan
to implement .debug_names, which is similar to --gdb-index.
That is great, and we will share code for this of course between the tool I
write and the modifications to lld. Does lld use the llvm::ObjectFile layer? or
does it have its own lighter weight layer?
> 
>> dsymutil has a --update feature which seems to load all of the DWARF
and pretend to link it all the while generating the new accelerator table data,
but I fear using this would pull it way too much code (AsmPrinter, all targets
required to load all object files types, the standard llvm object file model
(not the lld or llvm-objcopy versions), targets, etc).
>> 
>> My initial thoughts are:
>> 1 - load a DWARFContext and iterate through the DWARF and build
accelerator table data
>> 2 - create the sections for the accelerator tables and either keep in
memory or save to disk
>> 3 - call functions to add the newly created sections to the binary
>> 
>> #1 should be easy as long as I can use a DWARFContext from
DebugInfoDWARF.
>> #2 might need to be re-implemented using something other than an
AsmPrinter?
>> #3 can use llvm-objcopy code if needed since it can add sections?
>> 
>> Any advice on how this can or should be implemented would be
appreciated from anyone with experience.
>> 
>> Greg Clayton
>> 
> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200302/27d62eef/attachment.html>