thr3ads.net - llvm dev - [llvm-dev] DWARF Generator [Nov 2016]

If this information is useful, please help other people find it:
Share via:

Greg Clayton via llvm-dev

2016-Nov-17 23:12 UTC

[llvm-dev] DWARF Generator

I have recently been modifying the DWARF parser and have more patches planned
and I want to be able to add unit tests that test the internal llvm DWARF APIs
to ensure they continue to work and also validate the changes that I am making.
There are not many DWARF unit tests other than very simple ones that test DWARF
forms currently. I would like to expand this to include many more tests.

I had submitted a patch that I aborted as it was too large. One of the issues
with the patch was a stand alone DWARF generator that can turn a few API calls
into the section data required for the DWARFContextInMemory class to be able to
load DWARF from. The idea is to generate a small blurb of DWARF, parse it using
our built in DWARF parser and validate that the API calls we do when consuming
the DWARF match what we expect. The original stand along DWARF generator class
is in unittests/DebugInfo/DWARF/DWARFGenerator2.{h,cpp} in the patch attached.
The original review suggested that I try to use the AsmPrinter and many of its
associated classes to generate the DWARF. I attempted to do so and the
AsmPrinter version is in lib/CodeGen/DwarfGenerator.{h,cpp} in the patch
attached. This AsmPrinter based code steals code from the DwarfLinker.cpp.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dwarfgen.patch
Type: application/octet-stream
Size: 90291 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161117/a50b2c03/attachment.obj>
-------------- next part --------------


I am having trouble getting things to work with the AsmPrinter. I was able to
get simple DWARF to be emitted with the AsmPrinter version of the DWARF
generator with code like:


   initLLVM();
   DwarfGen DG;
   Triple Triple("x86_64--");
   StringRef Path("/tmp/test.elf");
   bool DwarfInitSuccess = DG.init(Triple, Path);
   EXPECT_TRUE(DwarfInitSuccess);
   uint16_t Version = 4;
   uint8_t AddrSize = 8;
   DwarfGenCU &CU = DG.appendCompileUnit(Version, AddrSize);
   DwarfGenDIE CUDie = CU.getUnitDIE();

   CUDie.addAttribute(DW_AT_name, DW_FORM_strp, "/tmp/main.c");
   CUDie.addAttribute(DW_AT_language, DW_FORM_data2, DW_LANG_C);

   DwarfGenDIE SubprogramDie = CUDie.addChild(DW_TAG_subprogram);
   SubprogramDie.addAttribute(DW_AT_name, DW_FORM_strp, "main");
   SubprogramDie.addAttribute(DW_AT_low_pc, DW_FORM_addr, 0x1000U);
   SubprogramDie.addAttribute(DW_AT_high_pc, DW_FORM_addr, 0x2000U);

   DwarfGenDIE IntDie = CUDie.addChild(DW_TAG_base_type);
   IntDie.addAttribute(DW_AT_name, DW_FORM_strp, "int");
   IntDie.addAttribute(DW_AT_encoding, DW_FORM_data1, DW_ATE_signed);
   IntDie.addAttribute(DW_AT_byte_size, DW_FORM_data1, 4);

   DwarfGenDIE ArgcDie = SubprogramDie.addChild(DW_TAG_formal_parameter);
   ArgcDie.addAttribute(DW_AT_name, DW_FORM_strp, "argc");
   //ArgcDie.addAttribute(DW_AT_type, DW_FORM_ref_addr, IntDie); // Crashes
here...

   DG.generate();

   auto Obj = object::ObjectFile::createObjectFile(Path);
   if (Obj) {
     DWARFContextInMemory DwarfContext(*Obj.get().getBinary());
     uint32_t NumCUs = DwarfContext.getNumCompileUnits();
     for (uint32_t i=0; i<NumCUs; ++i) {
       DWARFCompileUnit *U = DwarfContext.getCompileUnitAtIndex(i);
       if (U)
         U->getUnitDIE(false)->dump(llvm::outs(), U, -1u);
     }
   }


But things fall down if I try to uncomment the DW_FORM_ref_addr line above. The
problem is that AsmPrinter really expects a full stack of stuff to be there and
expects people to use the DwarfDebug class and all of its associated classes.
These associated classes really want to use the "DI" objects
(DICompileUnit, etc) so to create a compile unit we would need to create
DICompileUnit object and then make a AsmPrinter/DwarfCompileUnit. That stack is
pretty heavy and requires the code shown above to create many many classes just
to represent the simple output we wish to emit. Another downside of the
AsmPrinter method is we don't know which targets people are going to build
into their binaries and thus we don't know which triples we will be able to
use when generating DWARF info. Adrian Prantl attempted to help me get things
working over here and we kept running into roadblocks.

I wanted to pass this patch along in case someone wants to take a look at how we
can possibly fix the lib/CodeGen/DwarfGenerator.cpp and
lib/CodeGen/DwarfGenerator.h. The code that sets up all the required classes for
the AsmPrinter method is in the DwarfGen class from
lib/CodeGen/DwarfGenerator.cpp in the following function:

bool DwarfGen::init(Triple TheTriple, StringRef OutputFilename);

The code in this function was looted from existing DwarfLinker.cpp code. This
functions requires a valid triple and that triple is used to create a lot of the
classes required to make the AsmPrinter. I am not sure if any other code uses
the AsmPrinter like this besides the DwarfLinker.cpp code and that code uses its
own magic to actually link the DWARF. It does reuse some of the functions as I
did, but the DwarfLinker doesn't use any of the DwarfDebug, DwarfCompileUnit
or any of the classes that the compiler/assembler uses when making DWARF. The
amount of work required for refactoring the AsmPrinter exceeds the time I am
going to have, but I would still like to have DWARF API testing in the unit
tests.

So my question is if anyone would have objections to using the stand along DWARF
generator in unittests/DebugInfo/DWARF until we can later get the YAML tools to
be able to produce DWARF and we can switch to testing the DWARF data that way?
Chris Bieneman has expressed interest in getting a DWARF/YAML layer going.

My reasoning is:
- I want to be able to test DWARF APIs we have to ensure they work correctly as
there are no Dwarf API tests right now. I will be adding code that changes many
things in the DWARF parser and it will be essential to verify that there are no
regressions in the DWARF APIs.
- Not sure which targets would be built into LLVM so it might be hard to write
tests that cover 32/64 bit addresses and all the variants if we have to do
things legally via AsmPrinter and valid targets
- Not enough time to modify AsmPrinter to not require the full DebugInfo stack
and the classes that it uses (llvm::DwarfCompileUnit which must use
llvm::DICompileUnit, llvm::DIE class which uses many local classes that all
depend on  the full DwarfDebug stack).

I made a large effort to try and get things working with the AsmPrinter, so I
wanted everyone to know that I tried to get that solution working. Let me know
what you anyone thinks.

Greg Clayton

David Blaikie via llvm-dev

2016-Nov-17 23:40 UTC

head link

[llvm-dev] DWARF Generator

On Thu, Nov 17, 2016 at 3:12 PM Greg Clayton via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I have recently been modifying the DWARF parser and have more patches
> planned and I want to be able to add unit tests that test the internal llvm
> DWARF APIs to ensure they continue to work and also validate the changes
> that I am making. There are not many DWARF unit tests other than very
> simple ones that test DWARF forms currently. I would like to expand this to
> include many more tests.
>
> I had submitted a patch that I aborted as it was too large. One of the
> issues with the patch was a stand alone DWARF generator that can turn a few
> API calls into the section data required for the DWARFContextInMemory class
> to be able to load DWARF from. The idea is to generate a small blurb of
> DWARF, parse it using our built in DWARF parser and validate that the API
> calls we do when consuming the DWARF match what we expect. The original
> stand along DWARF generator class is in
> unittests/DebugInfo/DWARF/DWARFGenerator2.{h,cpp} in the patch attached.
> The original review suggested that I try to use the AsmPrinter and many of
> its associated classes to generate the DWARF. I attempted to do so and the
> AsmPrinter version is in lib/CodeGen/DwarfGenerator.{h,cpp} in the patch
> attached. This AsmPrinter based code steals code from the DwarfLinker.cpp.
>
>
>
> I am having trouble getting things to work with the AsmPrinter. I was able
> to get simple DWARF to be emitted with the AsmPrinter version of the DWARF
> generator with code like:
>
>
>    initLLVM();
>    DwarfGen DG;
>    Triple Triple("x86_64--");
>    StringRef Path("/tmp/test.elf");
>    bool DwarfInitSuccess = DG.init(Triple, Path);
>    EXPECT_TRUE(DwarfInitSuccess);
>    uint16_t Version = 4;
>    uint8_t AddrSize = 8;
>    DwarfGenCU &CU = DG.appendCompileUnit(Version, AddrSize);
>    DwarfGenDIE CUDie = CU.getUnitDIE();
>
>    CUDie.addAttribute(DW_AT_name, DW_FORM_strp, "/tmp/main.c");
>    CUDie.addAttribute(DW_AT_language, DW_FORM_data2, DW_LANG_C);
>
>    DwarfGenDIE SubprogramDie = CUDie.addChild(DW_TAG_subprogram);
>    SubprogramDie.addAttribute(DW_AT_name, DW_FORM_strp, "main");
>    SubprogramDie.addAttribute(DW_AT_low_pc, DW_FORM_addr, 0x1000U);
>    SubprogramDie.addAttribute(DW_AT_high_pc, DW_FORM_addr, 0x2000U);
>
>    DwarfGenDIE IntDie = CUDie.addChild(DW_TAG_base_type);
>    IntDie.addAttribute(DW_AT_name, DW_FORM_strp, "int");
>    IntDie.addAttribute(DW_AT_encoding, DW_FORM_data1, DW_ATE_signed);
>    IntDie.addAttribute(DW_AT_byte_size, DW_FORM_data1, 4);
>
>    DwarfGenDIE ArgcDie = SubprogramDie.addChild(DW_TAG_formal_parameter);
>    ArgcDie.addAttribute(DW_AT_name, DW_FORM_strp, "argc");
>    //ArgcDie.addAttribute(DW_AT_type, DW_FORM_ref_addr, IntDie); //
> Crashes here...
>
>    DG.generate();
>
>    auto Obj = object::ObjectFile::createObjectFile(Path);
>    if (Obj) {
>      DWARFContextInMemory DwarfContext(*Obj.get().getBinary());
>      uint32_t NumCUs = DwarfContext.getNumCompileUnits();
>      for (uint32_t i=0; i<NumCUs; ++i) {
>        DWARFCompileUnit *U = DwarfContext.getCompileUnitAtIndex(i);
>        if (U)
>          U->getUnitDIE(false)->dump(llvm::outs(), U, -1u);
>      }
>    }
>
>
> But things fall down if I try to uncomment the DW_FORM_ref_addr line
> above. The problem is that AsmPrinter really expects a full stack of stuff
> to be there and expects people to use the DwarfDebug class and all of its
> associated classes. These associated classes really want to use the
"DI"
> objects (DICompileUnit, etc) so to create a compile unit we would need to
> create DICompileUnit object and then make a AsmPrinter/DwarfCompileUnit.
> That stack is pretty heavy and requires the code shown above to create many
> many classes just to represent the simple output we wish to emit. Another
> downside of the AsmPrinter method is we don't know which targets people
are
> going to build into their binaries and thus we don't know which triples
we
> will be able to use when generating DWARF info. Adrian Prantl attempted to
> help me get things working over here and we kept running into roadblocks.
>
It'd be great to have more detail about the roadblocks you hit to better
understand how bad/what the issues are.

Even if we end up adding another set of code to generate DWARF (which I'd
really like to avoid) we'd want to, at some point, coalesce them back
together. Given the goal is to try to coalesce the DWARF parsing code in
LLDB and LLVM, it'd seem unfortunate if that effort just created another
similar (or larger) amount of work for DWARF generation.

> I wanted to pass this patch along in case someone wants to take a look at
> how we can possibly fix the lib/CodeGen/DwarfGenerator.cpp and
> lib/CodeGen/DwarfGenerator.h. The code that sets up all the required
> classes for the AsmPrinter method is in the DwarfGen class from
> lib/CodeGen/DwarfGenerator.cpp in the following function:
>
> bool DwarfGen::init(Triple TheTriple, StringRef OutputFilename);
>
> The code in this function was looted from existing DwarfLinker.cpp code.
> This functions requires a valid triple and that triple is used to create a
> lot of the classes required to make the AsmPrinter. I am not sure if any
> other code uses the AsmPrinter like this besides the DwarfLinker.cpp code
> and that code uses its own magic to actually link the DWARF. It does reuse
> some of the functions as I did, but the DwarfLinker doesn't use any of
the
> DwarfDebug, DwarfCompileUnit or any of the classes that the
> compiler/assembler uses when making DWARF.

What's the DwarfLinker code missing that you need? If that code is
generating essentially arbitrary DWARF, what's blocking using the same
technique for generating DWARF for parsing tests?

> The amount of work required for refactoring the AsmPrinter exceeds the
> time I am going to have, but I would still like to have DWARF API testing
> in the unit tests.
>
> So my question is if anyone would have objections to using the stand along
> DWARF generator in unittests/DebugInfo/DWARF until we can later get the
> YAML tools to be able to produce DWARF and we can switch to testing the
> DWARF data that way? Chris Bieneman has expressed interest in getting a
> DWARF/YAML layer going.
>
Those tools would still want to use pretty similar (conceptually)
abstractions to LLVM's codegen and llvm-dsymutil. I'd still strongly
prefer
to generalize/keep common APIs here - or better understand why it's not
practical now (& what it will take/how we make sure we have a plan and
resources to get there eventually).

> My reasoning is:
> - I want to be able to test DWARF APIs we have to ensure they work
> correctly as there are no Dwarf API tests right now. I will be adding code
> that changes many things in the DWARF parser and it will be essential to
> verify that there are no regressions in the DWARF APIs.
> - Not sure which targets would be built into LLVM so it might be hard to
> write tests that cover 32/64 bit addresses and all the variants if we have
> to do things legally via AsmPrinter and valid targets
>
Seems like it might be plausible to refactor out whatever features of the
AsmPrinter these APIs require (so we just harvest that data out of
AsmPrinter and pass it down in a struct, say - so that other users can pass
their own struct without needing an AsmPrinter). Though, again, interested
to know how dsymutil is working in these situations.

> - Not enough time to modify AsmPrinter to not require the full DebugInfo
> stack and the classes that it uses (llvm::DwarfCompileUnit which must use
> llvm::DICompileUnit, llvm::DIE class which uses many local classes that all
> depend on  the full DwarfDebug stack).
>
Will you have time at some later date to come back and revisit this? It's
understandable that we may choose to incur short term technical debt with
an understanding that it will be paid off in some timely manner. It'd be
less desirable if there's no such plan/possibility and we incur a fairly
clear case of technical debt (redundant DWARF generation libraries -
especially when this effort is to remove a redundant DWARF parser).

> I made a large effort to try and get things working with the AsmPrinter,
> so I wanted everyone to know that I tried to get that solution working. Let
> me know what you anyone thinks.
>
> Greg Clayton
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161117/343a1677/attachment.html>

Robinson, Paul via llvm-dev

2016-Nov-18 00:13 UTC

head link

[llvm-dev] DWARF Generator

It looks like DIE.cpp needs DwarfDebug (via AsmPrinter) for basically two
things: finding location-list entries, because the location list is managed by
DwarfDebug; and, handling DW_FORM_ref_addr, which requires (a) the DWARF
version, because the ref_addr size depends on the version, and (b) looking at
other units, because units are managed by DwarfDebug.

It might be feasible to refactor DwarfDebug into (broadly speaking) the
MD-to-DWARF translation part and the hunks-of-DWARF management part.  Then the
MD translator and the DWARF-generator could use the hunks-of-DWARF API to
achieve their respective goals.  I spent about 30 seconds thinking about that,
so the opinion might not be worth much.
--paulr

From: David Blaikie [mailto:dblaikie at gmail.com]
Sent: Thursday, November 17, 2016 3:40 PM
To: Greg Clayton; llvm-dev at lists.llvm.org; Robinson, Paul; Eric Christopher;
Adrian Prantl
Subject: Re: [llvm-dev] DWARF Generator


On Thu, Nov 17, 2016 at 3:12 PM Greg Clayton via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
I have recently been modifying the DWARF parser and have more patches planned
and I want to be able to add unit tests that test the internal llvm DWARF APIs
to ensure they continue to work and also validate the changes that I am making.
There are not many DWARF unit tests other than very simple ones that test DWARF
forms currently. I would like to expand this to include many more tests.

I had submitted a patch that I aborted as it was too large. One of the issues
with the patch was a stand alone DWARF generator that can turn a few API calls
into the section data required for the DWARFContextInMemory class to be able to
load DWARF from. The idea is to generate a small blurb of DWARF, parse it using
our built in DWARF parser and validate that the API calls we do when consuming
the DWARF match what we expect. The original stand along DWARF generator class
is in unittests/DebugInfo/DWARF/DWARFGenerator2.{h,cpp} in the patch attached.
The original review suggested that I try to use the AsmPrinter and many of its
associated classes to generate the DWARF. I attempted to do so and the
AsmPrinter version is in lib/CodeGen/DwarfGenerator.{h,cpp} in the patch
attached. This AsmPrinter based code steals code from the DwarfLinker.cpp.



I am having trouble getting things to work with the AsmPrinter. I was able to
get simple DWARF to be emitted with the AsmPrinter version of the DWARF
generator with code like:


   initLLVM();
   DwarfGen DG;
   Triple Triple("x86_64--");
   StringRef Path("/tmp/test.elf");
   bool DwarfInitSuccess = DG.init(Triple, Path);
   EXPECT_TRUE(DwarfInitSuccess);
   uint16_t Version = 4;
   uint8_t AddrSize = 8;
   DwarfGenCU &CU = DG.appendCompileUnit(Version, AddrSize);
   DwarfGenDIE CUDie = CU.getUnitDIE();

   CUDie.addAttribute(DW_AT_name, DW_FORM_strp, "/tmp/main.c");
   CUDie.addAttribute(DW_AT_language, DW_FORM_data2, DW_LANG_C);

   DwarfGenDIE SubprogramDie = CUDie.addChild(DW_TAG_subprogram);
   SubprogramDie.addAttribute(DW_AT_name, DW_FORM_strp, "main");
   SubprogramDie.addAttribute(DW_AT_low_pc, DW_FORM_addr, 0x1000U);
   SubprogramDie.addAttribute(DW_AT_high_pc, DW_FORM_addr, 0x2000U);

   DwarfGenDIE IntDie = CUDie.addChild(DW_TAG_base_type);
   IntDie.addAttribute(DW_AT_name, DW_FORM_strp, "int");
   IntDie.addAttribute(DW_AT_encoding, DW_FORM_data1, DW_ATE_signed);
   IntDie.addAttribute(DW_AT_byte_size, DW_FORM_data1, 4);

   DwarfGenDIE ArgcDie = SubprogramDie.addChild(DW_TAG_formal_parameter);
   ArgcDie.addAttribute(DW_AT_name, DW_FORM_strp, "argc");
   //ArgcDie.addAttribute(DW_AT_type, DW_FORM_ref_addr, IntDie); // Crashes
here...

   DG.generate();

   auto Obj = object::ObjectFile::createObjectFile(Path);
   if (Obj) {
     DWARFContextInMemory DwarfContext(*Obj.get().getBinary());
     uint32_t NumCUs = DwarfContext.getNumCompileUnits();
     for (uint32_t i=0; i<NumCUs; ++i) {
       DWARFCompileUnit *U = DwarfContext.getCompileUnitAtIndex(i);
       if (U)
         U->getUnitDIE(false)->dump(llvm::outs(), U, -1u);
     }
   }


But things fall down if I try to uncomment the DW_FORM_ref_addr line above. The
problem is that AsmPrinter really expects a full stack of stuff to be there and
expects people to use the DwarfDebug class and all of its associated classes.
These associated classes really want to use the "DI" objects
(DICompileUnit, etc) so to create a compile unit we would need to create
DICompileUnit object and then make a AsmPrinter/DwarfCompileUnit. That stack is
pretty heavy and requires the code shown above to create many many classes just
to represent the simple output we wish to emit. Another downside of the
AsmPrinter method is we don't know which targets people are going to build
into their binaries and thus we don't know which triples we will be able to
use when generating DWARF info. Adrian Prantl attempted to help me get things
working over here and we kept running into roadblocks.

It'd be great to have more detail about the roadblocks you hit to better
understand how bad/what the issues are.

Even if we end up adding another set of code to generate DWARF (which I'd
really like to avoid) we'd want to, at some point, coalesce them back
together. Given the goal is to try to coalesce the DWARF parsing code in LLDB
and LLVM, it'd seem unfortunate if that effort just created another similar
(or larger) amount of work for DWARF generation.

I wanted to pass this patch along in case someone wants to take a look at how we
can possibly fix the lib/CodeGen/DwarfGenerator.cpp and
lib/CodeGen/DwarfGenerator.h. The code that sets up all the required classes for
the AsmPrinter method is in the DwarfGen class from
lib/CodeGen/DwarfGenerator.cpp in the following function:

bool DwarfGen::init(Triple TheTriple, StringRef OutputFilename);

The code in this function was looted from existing DwarfLinker.cpp code. This
functions requires a valid triple and that triple is used to create a lot of the
classes required to make the AsmPrinter. I am not sure if any other code uses
the AsmPrinter like this besides the DwarfLinker.cpp code and that code uses its
own magic to actually link the DWARF. It does reuse some of the functions as I
did, but the DwarfLinker doesn't use any of the DwarfDebug, DwarfCompileUnit
or any of the classes that the compiler/assembler uses when making DWARF.

What's the DwarfLinker code missing that you need? If that code is
generating essentially arbitrary DWARF, what's blocking using the same
technique for generating DWARF for parsing tests?

The amount of work required for refactoring the AsmPrinter exceeds the time I am
going to have, but I would still like to have DWARF API testing in the unit
tests.

So my question is if anyone would have objections to using the stand along DWARF
generator in unittests/DebugInfo/DWARF until we can later get the YAML tools to
be able to produce DWARF and we can switch to testing the DWARF data that way?
Chris Bieneman has expressed interest in getting a DWARF/YAML layer going.

Those tools would still want to use pretty similar (conceptually) abstractions
to LLVM's codegen and llvm-dsymutil. I'd still strongly prefer to
generalize/keep common APIs here - or better understand why it's not
practical now (& what it will take/how we make sure we have a plan and
resources to get there eventually).

My reasoning is:
- I want to be able to test DWARF APIs we have to ensure they work correctly as
there are no Dwarf API tests right now. I will be adding code that changes many
things in the DWARF parser and it will be essential to verify that there are no
regressions in the DWARF APIs.
- Not sure which targets would be built into LLVM so it might be hard to write
tests that cover 32/64 bit addresses and all the variants if we have to do
things legally via AsmPrinter and valid targets

Seems like it might be plausible to refactor out whatever features of the
AsmPrinter these APIs require (so we just harvest that data out of AsmPrinter
and pass it down in a struct, say - so that other users can pass their own
struct without needing an AsmPrinter). Though, again, interested to know how
dsymutil is working in these situations.

- Not enough time to modify AsmPrinter to not require the full DebugInfo stack
and the classes that it uses (llvm::DwarfCompileUnit which must use
llvm::DICompileUnit, llvm::DIE class which uses many local classes that all
depend on  the full DwarfDebug stack).

Will you have time at some later date to come back and revisit this? It's
understandable that we may choose to incur short term technical debt with an
understanding that it will be paid off in some timely manner. It'd be less
desirable if there's no such plan/possibility and we incur a fairly clear
case of technical debt (redundant DWARF generation libraries - especially when
this effort is to remove a redundant DWARF parser).

I made a large effort to try and get things working with the AsmPrinter, so I
wanted everyone to know that I tried to get that solution working. Let me know
what you anyone thinks.

Greg Clayton

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161118/0d9c9ce2/attachment.html>

Greg Clayton via llvm-dev

2016-Nov-18 01:00 UTC

head link

[llvm-dev] DWARF Generator

> On Nov 17, 2016, at 3:40 PM, David Blaikie <dblaikie at gmail.com>
wrote:
> 
> 
> 
> On Thu, Nov 17, 2016 at 3:12 PM Greg Clayton via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> I have recently been modifying the DWARF parser and have more patches
planned and I want to be able to add unit tests that test the internal llvm
DWARF APIs to ensure they continue to work and also validate the changes that I
am making. There are not many DWARF unit tests other than very simple ones that
test DWARF forms currently. I would like to expand this to include many more
tests.
> 
> I had submitted a patch that I aborted as it was too large. One of the
issues with the patch was a stand alone DWARF generator that can turn a few API
calls into the section data required for the DWARFContextInMemory class to be
able to load DWARF from. The idea is to generate a small blurb of DWARF, parse
it using our built in DWARF parser and validate that the API calls we do when
consuming the DWARF match what we expect. The original stand along DWARF
generator class is in unittests/DebugInfo/DWARF/DWARFGenerator2.{h,cpp} in the
patch attached. The original review suggested that I try to use the AsmPrinter
and many of its associated classes to generate the DWARF. I attempted to do so
and the AsmPrinter version is in lib/CodeGen/DwarfGenerator.{h,cpp} in the patch
attached. This AsmPrinter based code steals code from the DwarfLinker.cpp.
> 
> 
> 
> I am having trouble getting things to work with the AsmPrinter. I was able
to get simple DWARF to be emitted with the AsmPrinter version of the DWARF
generator with code like:
> 
> 
>    initLLVM();
>    DwarfGen DG;
>    Triple Triple("x86_64--");
>    StringRef Path("/tmp/test.elf");
>    bool DwarfInitSuccess = DG.init(Triple, Path);
>    EXPECT_TRUE(DwarfInitSuccess);
>    uint16_t Version = 4;
>    uint8_t AddrSize = 8;
>    DwarfGenCU &CU = DG.appendCompileUnit(Version, AddrSize);
>    DwarfGenDIE CUDie = CU.getUnitDIE();
> 
>    CUDie.addAttribute(DW_AT_name, DW_FORM_strp, "/tmp/main.c");
>    CUDie.addAttribute(DW_AT_language, DW_FORM_data2, DW_LANG_C);
> 
>    DwarfGenDIE SubprogramDie = CUDie.addChild(DW_TAG_subprogram);
>    SubprogramDie.addAttribute(DW_AT_name, DW_FORM_strp, "main");
>    SubprogramDie.addAttribute(DW_AT_low_pc, DW_FORM_addr, 0x1000U);
>    SubprogramDie.addAttribute(DW_AT_high_pc, DW_FORM_addr, 0x2000U);
> 
>    DwarfGenDIE IntDie = CUDie.addChild(DW_TAG_base_type);
>    IntDie.addAttribute(DW_AT_name, DW_FORM_strp, "int");
>    IntDie.addAttribute(DW_AT_encoding, DW_FORM_data1, DW_ATE_signed);
>    IntDie.addAttribute(DW_AT_byte_size, DW_FORM_data1, 4);
> 
>    DwarfGenDIE ArgcDie = SubprogramDie.addChild(DW_TAG_formal_parameter);
>    ArgcDie.addAttribute(DW_AT_name, DW_FORM_strp, "argc");
>    //ArgcDie.addAttribute(DW_AT_type, DW_FORM_ref_addr, IntDie); // Crashes
here...
> 
>    DG.generate();
> 
>    auto Obj = object::ObjectFile::createObjectFile(Path);
>    if (Obj) {
>      DWARFContextInMemory DwarfContext(*Obj.get().getBinary());
>      uint32_t NumCUs = DwarfContext.getNumCompileUnits();
>      for (uint32_t i=0; i<NumCUs; ++i) {
>        DWARFCompileUnit *U = DwarfContext.getCompileUnitAtIndex(i);
>        if (U)
>          U->getUnitDIE(false)->dump(llvm::outs(), U, -1u);
>      }
>    }
> 
> 
> But things fall down if I try to uncomment the DW_FORM_ref_addr line above.
The problem is that AsmPrinter really expects a full stack of stuff to be there
and expects people to use the DwarfDebug class and all of its associated
classes. These associated classes really want to use the "DI" objects
(DICompileUnit, etc) so to create a compile unit we would need to create
DICompileUnit object and then make a AsmPrinter/DwarfCompileUnit. That stack is
pretty heavy and requires the code shown above to create many many classes just
to represent the simple output we wish to emit. Another downside of the
AsmPrinter method is we don't know which targets people are going to build
into their binaries and thus we don't know which triples we will be able to
use when generating DWARF info. Adrian Prantl attempted to help me get things
working over here and we kept running into roadblocks.
> 
> It'd be great to have more detail about the roadblocks you hit to
better understand how bad/what the issues are.
A few blocks:

- DIEString doesn't support DW_FORM_string. DW_FORM_string support might
have been pulled so that we never emit it from clang, but we would want to have
a unit test that covers being able to read an inlined C string from a DIE.
Support won't be that hard to add, but we might not want it so that people
can't use it by accident and make less efficient DWARF.
- Asserts, asserts, asserts. As we tried to emit DWARF, we got an asserts in
bool AsmPrinter::doInitialization(Module &M). On the first line:

  MMI = getAnalysisIfAvailable<MachineModuleInfo>();

This asserts if you use the AsmPrinter the way the DwarfLinker and the
AsmPrinter based DwarfGen does if you call this. You must call this to generate
the DebugDwarf. If you get past this by installing a Pass then we assert at:

  GCModuleInfo *MI = getAnalysisIfAvailable<GCModuleInfo>();
  assert(MI && "AsmPrinter didn't require GCModuleInfo?");

If we don't have this, we don't get a DwarfDebug.
> 
> Even if we end up adding another set of code to generate DWARF (which
I'd really like to avoid) we'd want to, at some point, coalesce them
back together. Given the goal is to try to coalesce the DWARF parsing code in
LLDB and LLVM, it'd seem unfortunate if that effort just created another
similar (or larger) amount of work for DWARF generation.
This DWARF generator could just live in the unittests/DebugInfo/DWARF directory
so it wouldn't pollute anything in LLVM it we do choose to use
it.>  
> I wanted to pass this patch along in case someone wants to take a look at
how we can possibly fix the lib/CodeGen/DwarfGenerator.cpp and
lib/CodeGen/DwarfGenerator.h. The code that sets up all the required classes for
the AsmPrinter method is in the DwarfGen class from
lib/CodeGen/DwarfGenerator.cpp in the following function:
> 
> bool DwarfGen::init(Triple TheTriple, StringRef OutputFilename);
> 
> The code in this function was looted from existing DwarfLinker.cpp code.
This functions requires a valid triple and that triple is used to create a lot
of the classes required to make the AsmPrinter. I am not sure if any other code
uses the AsmPrinter like this besides the DwarfLinker.cpp code and that code
uses its own magic to actually link the DWARF. It does reuse some of the
functions as I did, but the DwarfLinker doesn't use any of the DwarfDebug,
DwarfCompileUnit or any of the classes that the compiler/assembler uses when
making DWARF.
> 
> What's the DwarfLinker code missing that you need? If that code is
generating essentially arbitrary DWARF, what's blocking using the same
technique for generating DWARF for parsing tests?
They don't use any of the DwarfDebug, DwarfCompileUnit classes. They also
don't use any of the DI classes when making up the debug info. So both the
DWARF linker and the generator have similar needs: make DWARF that isn't
tied too closely to the clang internal classes and DI
classes.>  
> The amount of work required for refactoring the AsmPrinter exceeds the time
I am going to have, but I would still like to have DWARF API testing in the unit
tests.
> 
> So my question is if anyone would have objections to using the stand along
DWARF generator in unittests/DebugInfo/DWARF until we can later get the YAML
tools to be able to produce DWARF and we can switch to testing the DWARF data
that way? Chris Bieneman has expressed interest in getting a DWARF/YAML layer
going.
> 
> Those tools would still want to use pretty similar (conceptually)
abstractions to LLVM's codegen and llvm-dsymutil. I'd still strongly
prefer to generalize/keep common APIs here - or better understand why it's
not practical now (& what it will take/how we make sure we have a plan and
resources to get there eventually).
>  
> My reasoning is:
> - I want to be able to test DWARF APIs we have to ensure they work
correctly as there are no Dwarf API tests right now. I will be adding code that
changes many things in the DWARF parser and it will be essential to verify that
there are no regressions in the DWARF APIs.
> - Not sure which targets would be built into LLVM so it might be hard to
write tests that cover 32/64 bit addresses and all the variants if we have to do
things legally via AsmPrinter and valid targets
> 
> Seems like it might be plausible to refactor out whatever features of the
AsmPrinter these APIs require (so we just harvest that data out of AsmPrinter
and pass it down in a struct, say - so that other users can pass their own
struct without needing an AsmPrinter). Though, again, interested to know how
dsymutil is working in these situations.
I can try that method if indeed the only places that use the DwarfDebug are the
DW_FORM_ref_addr and location lists. I'll let you know how that
goes.>  
> - Not enough time to modify AsmPrinter to not require the full DebugInfo
stack and the classes that it uses (llvm::DwarfCompileUnit which must use
llvm::DICompileUnit, llvm::DIE class which uses many local classes that all
depend on  the full DwarfDebug stack).
> 
> Will you have time at some later date to come back and revisit this?
It's understandable that we may choose to incur short term technical debt
with an understanding that it will be paid off in some timely manner. It'd
be less desirable if there's no such plan/possibility and we incur a fairly
clear case of technical debt (redundant DWARF generation libraries - especially
when this effort is to remove a redundant DWARF parser).
Not sure anyone else will need to generate DWARF manually. The two clients
currently are the DWARF unittests and the DwarfLinker. The DwarfLinker worked
around these issues. If the AsmPrinter wasn't such an integral part of the
entire compiler stack, I could take a stab at refactoring it, but I don't
believe I am the right person to do this at this point as I have no experience
or knowledge of the various ways that this class is used, or how it interacts
with other support classes (DwarfDebug, and many many other classes).

Things that still worry me:
- not being able to generate DWARF for 32/64 if targets are missing
- DIEString not supporting DW_FORM_string. I can add support, but I don't
know if we want it as if we add it people might start using it.
- hacking around asserts by constructing classes and copying code from places
that properly use the AsmPrinter that way it is supposed to be used so that we
can use it in a way that it wasn't designed to be used.
>  
> I made a large effort to try and get things working with the AsmPrinter, so
I wanted everyone to know that I tried to get that solution working. Let me know
what you anyone thinks.
> 
> Greg Clayton
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Apparently Analagous Threads

Search for more reasonably related threads

llvm dev - Nov 2016 - DWARF Generator

[llvm-dev] DWARF Generator

[llvm-dev] DWARF Generator

[llvm-dev] DWARF Generator

[llvm-dev] DWARF Generator

Apparently Analagous Threads