llvm dev - Jan 2013 - [LLVMdev] RFC: Improving our DWARF (and ELF) emission testing capabilities

>> 1. llvm-dwarfdump: the best approach when it works. But unfortunately
>> lib/DebugInfo supports only a (small) subset of DWARF. Tricky sections
>> like debug_frame aren't supported.
>
> Ideally I'd like to see support added whenever a code change is made
> to a feature - so long as we hold ourselves to a "test new
changes"
> that can gate/encourage the necessary feature support in
> llvm-dwarfdump.
>
> Since no one's likely to go back & write a bunch of regression
tests
> for all the existing code it seems premature to add new features to
> llvm-dwarfdump before there's a use-case. It does sometimes mean bug
> fixes appear to be costly because they include adding the missing test
> infrastructure support, but that's essentially where the cost is
> anyway.
See test/MC/ELF/cfi-register.s for a test I consider unmaintainable
since it just matches an elf-dump and requires manual decoding of the
data for every change and addition. When tests are too hard to write,
fewer tests get written.
>> 2. Relying of assembly directive emissions (i.e. .cfi_*), which is
>> cumbersome and misses a lot of things like actual DWARF encoding.
>
> I'm not sure what you mean by "actual DWARF encoding" here.
> (disclaimer: I've only recently started dabbling with debug info, so I
> may be missing obvious things)
I mean that it doesn't test the whole way, and there's quite a bit of
DWARF-related functionality in MC. So when a test relies on matching
directives in ASM output, there's quite a bit of code in MC it doesn't
exercise.
>
>> 3. Using elf-dump and examining the raw binary dumps. This makes tests
>> nearly unmaintainable.
>>
>> The latter is also why IMHO our ELF emission in general isn't well
>> tested. elf-dump is just too rudimentary and relies on simple (=dumb)
>> binary contents dumps.
>>
>> The long-term solution for DWARF would be to enhance lib/DebugInfo to
>> the point where it can handle all interesting DWARF sections. But this
>> is a lofty goal, since DWARF parsing is notoriously hard and this
>> would require a large investment of time and effort. And in the
>> meantime, we just don't write good enough tests (and enough of
them)
>> for this very important feature.
>
> Are there particular recent commits you've been concerned about the
> test quality of? I've been trying to keep an eye on this but, again,
> don't necessarily fully understand the ramifications of some changes.
See basically every test employing elf-dump for non-trivial things.
>
>> Therefore, as an interim stage, I propose to adopt some external tool
>> that parses DWARF and emits decoded textual dumps which makes tests
>> easy to write.
>>
>> Concretely, I have a pure Python library named pyelftools
>> (https://bitbucket.org/eliben/pyelftools) which provides comprehensive
>> ELF and DWARF parsing capabilities and has a dumper that's fully
>> compatible with the readelf command. Using pyelftools would allow us
>> to immediately improve the quality of our tests, and as lib/DebugInfo
>> matures llvm-dwarfdump can gradually replace the dumper without
>> changing the actual tests.
>
> I would be a little hesitant about test execution performance if
> involved invoking new python processes for each debug info test. But
> numbers could convince me. Beyond that I can't rationally claim any
> particular need to support llvm-dwarfdump as the tool of choice over
> any 3rd party tool.
This is already done with elf-dump (a Python script) which is used for
a lot of tests for lack better options.


Eli

> > I would be a little hesitant about test execution performance if
> > involved invoking new python processes for each debug info test. But
> > numbers could convince me. Beyond that I can't rationally claim
any
> > particular need to support llvm-dwarfdump as the tool of choice over
> > any 3rd party tool.
>
> This is already done with elf-dump (a Python script) which is used for
> a lot of tests for lack better options.
>
>
Those tests should be done with llvm-objdump these days. If they can't be
then llvm-objdump should be extended.

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130118/1ecda6a2/attachment.html>

>>> 2. Relying of assembly directive emissions (i.e. .cfi_*), which is
>>> cumbersome and misses a lot of things like actual DWARF encoding.
>>
>> I'm not sure what you mean by "actual DWARF encoding"
here.
>> (disclaimer: I've only recently started dabbling with debug info,
so I
>> may be missing obvious things)
>
> I mean that it doesn't test the whole way, and there's quite a bit
of
> DWARF-related functionality in MC. So when a test relies on matching
> directives in ASM output, there's quite a bit of code in MC it
doesn't
> exercise.
Hmmm. "Proper" testing would exercise each component involved, as well
as possibly longer paths that maybe are not exactly the sum of the parts.
Debug info changes are quite likely to involve most or all of:
- Clang's C/C++ to IR (metadata)
- LLVM's IR to assembler source
- assembler source to object file
- LLVM's IR to object file (which partly bypasses or can be different
  from the previous two paths)
Properly speaking they should each get their own tests.
Not to mention a unit-test (or debuginfo-test) to exercise the complete
Clang -> object (-> debugger) sequence.

I try to be good about this, but as a developer I find that sort of
thing tedious. Which mostly proves that I suck at QA, and have to depend
on reviewers to keep me on the straight and narrow.  This works to the
extent that those reviewers are willing to be critical of my efforts,
and insist on adequate (instead of minimal) testing.  But testing is
an art unto itself, and most developers aren't good at it.

--paulr

On Tue, Jan 22, 2013 at 3:23 PM, Robinson, Paul
<Paul.Robinson at am.sony.com> wrote:
>>>> 2. Relying of assembly directive emissions (i.e. .cfi_*), which
is
>>>> cumbersome and misses a lot of things like actual DWARF
encoding.
>>>
>>> I'm not sure what you mean by "actual DWARF encoding"
here.
>>> (disclaimer: I've only recently started dabbling with debug
info, so I
>>> may be missing obvious things)
>>
>> I mean that it doesn't test the whole way, and there's quite a
bit of
>> DWARF-related functionality in MC. So when a test relies on matching
>> directives in ASM output, there's quite a bit of code in MC it
doesn't
>> exercise.
>
> Hmmm. "Proper" testing would exercise each component involved, as
well
> as possibly longer paths that maybe are not exactly the sum of the parts.
> Debug info changes are quite likely to involve most or all of:
> - Clang's C/C++ to IR (metadata)
Simple enough: changes to Clang require tests in Clang (OK, so there's
no mechanism in place to avoid tests in Clang testing optimization,
for example, but we try to be good about it)
> - LLVM's IR to assembler source
> - assembler source to object file
> - LLVM's IR to object file (which partly bypasses or can be different
>   from the previous two paths)
& these generally go in LLVM - yeah, they could be separate, but I'd
expect the assembler and object file emission to be tested separately
already - the only benefit to testing particular IR->object &
separately testing particular IR->assembly is probably not worthwhile.
If we could test against the precursor to those outputs then we might
get the advantage of only having the right tests fail for the right
reasons (debug info tests wouldn't fail when we broken the
assembler/object emission).
> Properly speaking they should each get their own tests.
> Not to mention a unit-test (or debuginfo-test) to exercise the complete
> Clang -> object (-> debugger) sequence.
Just because it makes me twitch (though I admit debating test taxonomy
terminology verges on a religious topic): these tests are the
antithesis of unit tests. Taking source code, compiling it with
clang/LLVM, loading it in a debugger and interacting with the debugger
is a scenario test.

A unit test would be API level, say building IR by calling Clang APIs
& then passing it into DebugInfo generation & watching the MI calls
that resulted (preferably stubbing them out in some way).
> I try to be good about this, but as a developer I find that sort of
> thing tedious. Which mostly proves that I suck at QA, and have to depend
> on reviewers to keep me on the straight and narrow.  This works to the
> extent that those reviewers are willing to be critical of my efforts,
> and insist on adequate (instead of minimal) testing.  But testing is
> an art unto itself, and most developers aren't good at it.
We do what we can (because we must).

- David