[LLVMdev] Proposal: Improved regression test support for RuntimeDyld/MCJIT.

Lang Hames
2014-06-24T18:36:37Z

It may help the discussion if I actually attach the correct patch:

- Lang.




On Tue, Jun 24, 2014 at 10:57 AM, Lang Hames <lhames at gmail.com> wrote:

> Hi All,
>
> Dropping in a relevant discussion that Dave and I had off list:
>
> From Dave: "In my mind it's mostly that we have a fairly general
purpose
> tool for checking things and we do that by separating output from
> verification that has proved pretty valuable in other areas (by having
> general purpose output we can use it for better debugging/investigation,
> rather than just for writing test cases, for example - I can run
> llvm-rtdyld with the disassembly mode to see what the state in-memory looks
> like, with a -verify mode I can't do that)"
>
> My argument against testing RuntimeDyld via examining textual output is
> that sensibly rendering RuntimeDyld's memory would require considerable
> work (see discussion below), and I don't think anybody actually wants
to
> look at it. People have requested the ability to examine the output of the
> JIT, and I'm happy to add that feature if it hasn't been added
already
> (I'll have check), but the appropriate way to do that is to dump the
object
> files produced by the JIT before they're passed to RuntimeDyld, not to
try
> to render RuntimeDyld's memory.
>
> As for why dumping RuntimeDyld's memory is a lot of work: RuntimeDyld
> really only knows about blobs of bytes (one blob per section in the object
> file) and addresses of symbols. Consider, for example, how the following C
> struct looks in assembly and to RuntimeDyld:
>
> In C:
>
> struct {
>   unsigned a;
>   void* b;
>   unsigned c;
> } x = {1, &foo, 2};
>
> In pseudo-asm:
>
> .section __data:
> x:
>   .long 1
>   .quad foo
>   .long 2
>
> To RuntimeDyld:
>
> x: 0x01 0x00 0x00 0x00 0xEF 0xBE 0xAD 0xDE 0xCE 0xFA 0xED 0xFE 0x02 0x00
> 0x00 0x00
>
>
> RuntimeDyld doesn't know where the fields of x start or end, or indeed
> where x itself ends. We could add extra labels to the assembly code
> to delineate the fields and teach RuntimeDyld to render the bytes between
> each symbol, and how to recognize common integer sizes and render them
> appropriately (so that you don't have to write expressions to
reassemble
> ints from the individual bytes). But all of that complicates both the test
> cases themselves  and RuntimeDyld. None of those features would be
> available to a client who just wants to inspect the JIT's memory, since
the
> code-generator doesn't (and IMHO shouldn't) pepper its output with
> extraneous labels on the off-chance that someone wants to pretty-print
> RuntimeDyld's memory.
>
> From Dave: "in many other cases we've invented canonical textual
> representations to check against (objdump, dwarfdump, readelf, etc)."
>
> In each of those cases there was a desire for people to be able to
> visually inspect the information for understanding/correctness. Once you
> have that functionality, inspecting dumps with FileCheck makes much more
> sense. In RuntimeDyld's case all the interesting information is
contained
> in the object file, which can already be rendered with existing tools. The
> only thing RuntimeDyld changes is some byte values in memory. Being able to
> inspect them when something goes wrong tells you nothing interesting about
> why they were wrong, so textually rendering isn't helpful.
>
> I think directly checking the invariants makes more sense here, rather
> than going via text.
>
> Cheers,
> Lang.
>
>
>
>
> On Mon, Jun 23, 2014 at 8:54 PM, Lang Hames <lhames at gmail.com>
wrote:
>
>> Hi Dave,
>>
>> Jim Grosbach asked the same question, so you're in good company.
With
>> hindsight I think it was a mistake to say "FileCheck
workflow". What I
>> really meant was that this system plays well with lit. Not that your
>> question about using FileCheck would have been any less valid.
>>
>> I did consider using FileCheck for this, but decided it was the wrong
>> approach. The fundamental reason is that there's no demand for
textually
>> rendering RuntimeDyld's memory, and developing a textual renderer
so that
>> we could output text just to pattern match and re-assemble ints in
>> FileCheck would be a lot of pain for (as far as I can see) no gain.
>>
>> If there's a desire for FileCheck to support expression evaluation
we
>> could flesh out this evaluator and make it available as a support
library
>> that both FileCheck and RuntimeDyld could use.
>>
>> You (and independently Nick Kledzik) do raise the really useful idea of
>> leveraging the disassembler though. I like the idea of adding some
special
>> syntax to disassemble an instruction at a label and use one of its
>> immediates. That would eliminate a lot of the bit bashing that would
have
>> been required on instruction sets with tricky immediate encodings (E.g.
>> ARM). Something like:
>>
>> # rtdyld-check: @test_inst[0] = foo - (test_inst + 5)
>> test_inst:
>>   callq foo
>>
>> Cheers,
>> Lang.
>>
>> > On Jun 23, 2014, at 5:01 PM, David Blaikie <dblaikie at
gmail.com> wrote:
>> >
>> >> On Mon, Jun 23, 2014 at 3:01 PM, Lang Hames <lhames at
gmail.com> wrote:
>> >> Hi Everyone,
>> >>
>> >> For your consideration: A proposal to improve regression test
support
>> for
>> >> RuntimeDyld.
>> >
>> > Thanks for working on this, Lang. It's great to see.
>> >
>> >> Short version: We can make RuntimeDyld far more testable by
adding a
>> trivial
>> >> pointer-expression language that allows us to describe how
memory
>> should
>> >> look post-relocation. Jump down to "The Proposal"
for details.
>> >
>> > I've been trying to puzzle over what this would look like with
a
>> > possibly more general feature*.
>> >
>> > What would testing look like if we had a rtdyld dumping mode that
>> > printed the disassembly of the relocated machine code, and a
symbol
>> > table (or just inserted the labels for the symbols into the
>> > disassembly?).
>> >
>> > I understand we'd need to beef up FileCheck with slightly more
>> > arithmetic operations - but is it really so much (& would they
be so
>> > useless for other tests) that it's not worth putting it there?
>> >
>> > To take your example, here's my vague idea of what it might
look like
>> > to use a dump+FileCheck. The dump would look something like:
>> >
>> > (obviously I don't know, nor for this purpose care, how big
the
>> > instructions are, just that they have distinct addresses, etc)
>> >
>> >  0x42: bar:
>> >  0x42:   retq
>> >  0x43: foo:
>> >  0x43:   callq 0x42
>> >  0x44: inst1:
>> >  0x44:   retq
>> >
>> > And the FileCheck equivalent of
>> >
>> >  # rtdyld-check: *{4}(inst1 - 4) = (bar - inst1) & 0xffffffff
>> >
>> > would be something like:
>> >
>> >  CHECK: [[CALL_ADDR:.*]]: bar:
>> >  CHECK: callq [[CALL_ADDR]]
>> >
>> > Which, I suppose, depends on disassembler working correctly, not
sure
>> > if that's high risk/complicated.
>> >
>> > Alternatively - could llvm-rtdyld just print a simple description
of
>> > relocations its applied and the location of symbols? (similar to a
>> > static display of relocations like llvm-objdump -r) then FileCheck
>> > that.
>> >
>> > * all that said, a feature like you've proposed/implemented
isn't
>> > without precedent - clang's -verify is very similar to what
you've got
>> > here
>> >
>> >
>> >> Long version:
>> >>
>> >> Background:
>> >>
>> >> For those unfamiliar with it, RuntimeDyld a component of
MCJIT, LLVM's
>> JIT
>> >> compiler infrastructure. MCJIT produces an object file in
memory for
>> each
>> >> module that is JIT'd. RuntimeDyld's job is to apply
all the relocations
>> >> necessary to make the code in the object file runnable. In
other words,
>> >> RuntimeDyld is acting as both the static and dynamic linker
for the
>> JIT.
>> >>
>> >> The Problem:
>> >>
>> >> We can't directly test RuntimeDyld at the moment. We
currently infer
>> the
>> >> correctness of RuntimeDyld indirectly from the success of the
MCJIT
>> >> regression tests - if they pass, we assume RuntimeDyld must
have done
>> its
>> >> job right. That's far from an ideal. The biggest issues
with it are:
>> >>
>> >> (1) Each platform is testing only its own relocations and no
others.
>> I.e.
>> >> X86 testers are testing X86 relocations only. ARM testers are
testing
>> ARM
>> >> relocations only. If someone running on X86 breaks a
relocation for
>> ARM they
>> >> won't see the error in their regression test run -
they'll have to wait
>> >> until an ARM buildbot breaks before they realize anything is
wrong.
>> Fixes
>> >> for platforms that you don't have access to are difficult
to test -
>> all you
>> >> can do is eyeball disassembled memory and see if everything
looks
>> sane. This
>> >> is not much fun.
>> >>
>> >> (2) Relocations are produced by CodeGen from IR, rather than
described
>> >> directly. That's a lot of machinery to have between the
test-case and
>> the
>> >> final result. It is difficult to know what relocations each IR
>> regression
>> >> test is testing (and they're often incidental - we
don't have a
>> dedicated
>> >> relocation test set). This also means that if/when the code
generator
>> >> produces different relocation types the existing tests will
keep on
>> passing
>> >> but will silently stop testing the thing they used to test.
>> >>
>> >> The Proposal:
>> >>
>> >> (1) We provide a mechanism for describing how pieces of
relocated
>> memory
>> >> should look immediately prior to execution, and then inspect
the memory
>> >> rather than executing it. This addresses point (1) above:
Tests for any
>> >> platform can be loaded, linked and verified on any platform.
If you're
>> >> coding on X86 and you break an ARM relocation you'll know
about it
>> >> immediately.
>> >>
>> >> (2) RuntimeDyld test cases should be written in assembly,
rather than
>> IR.
>> >> This addresses point (2) above - we can cut the code
generators out and
>> >> guarantee that we're testing what we're interested in.
>> >>
>> >> The way to do this is to introduce a simple pointer expression
>> language.
>> >> This should be able to express things like: "The
immediate for this
>> call
>> >> points at symbol foo".
>> >>
>> >> Symbolically, what I have in mind would look something like:
>> >>
>> >>        // some asm ...
>> >> # assert *(inst1 + 1) = foo
>> >> inst1:
>> >>        callq   foo
>> >>        // some asm...
>> >>
>> >> Here we add the "inst1" label to give us a address
from which we can
>> get at
>> >> the immediate for the call. The " + 1" expression
skips the call
>> opcode (we
>> >> know the size of the opcode ahead of time, since this is
assembly and
>> so
>> >> target-specific).
>> >>
>> >> To verify that constraints expressed in this language hold, we
can add
>> an
>> >> expression evaluator to the llvm-rtdyld utility, which is a
>> command-line
>> >> interface to RuntimeDyld.
>> >>
>> >> I find these things are easier to discuss in the concrete, so
I've
>> attached
>> >> a basic implementation of this idea. The following discussion
is in
>> terms of
>> >> my patch, but I'm very open to tweaking all this.
>> >>
>> >> The language I've implemented is:
>> >>
>> >> test = expr '=' expr
>> >>
>> >> expr = '*{' number '}' load_addr_expr
>> >>     | binary_expr
>> >>     | '(' expr ')'
>> >>     | symbol
>> >>     | number
>> >>
>> >> load_addr_expr = symbol
>> >>               | '(' symbol '+' number
')'
>> >>               | '(' symbol '-' number
')'
>> >>
>> >> binary_expr = expr '+' expr
>> >>            | expr '-' expr
>> >>            | expr '&' expr
>> >>            | expr '|' expr
>> >>            | expr '<<' expr
>> >>            | expr '>>' expr
>> >>
>> >> This expression language supports simple pointer arithmetic,
shifting,
>> >> masking and loading. All values are internally held as 64-bit
>> unsigneds,
>> >> since RuntimeDlyd is designed to support cross-platform
linking,
>> including
>> >> linking for 64-bit targets from a 32-bit host. I think the
only
>> stand-out
>> >> wart is the *{#size}<addr> syntax for loads. This comes
from the fact
>> that
>> >> immediates aren't always 64-bits, so it's not safe to
do a 64-bit
>> load: you
>> >> could read past the end of allocated memory. The #size field
indicates
>> how
>> >> many bytes to read.
>> >>
>> >> This patch adds a "-verify" option to llvm-rtdyld to
attach the
>> expression
>> >> evaluator to a RuntimeDyld instance after linking. When
-verify is
>> passed,
>> >> llvm-rtdyld does not execute any code. Files containing rules
are
>> passed via
>> >> "-check=<filename>" arguments, and rules are
read from any line
>> prefixed
>> >> with the string "# rtdyld-check: ". The intended
workflow is modeled
>> on the
>> >> FileCheck regression tests.
>> >>
>> >> Here's an example of what a test case for a test for an
x86-64
>> PC-relative
>> >> MACHO_VANILLA relocation would look like:
>> >>
>> >> ; RUN: clang -triple x86_64-apple-macosx10.9.0 -c -o foo.o %s
>> >> ; RUN: llvm-rtdyld -verify -check=foo.s foo.o
>> >> ; RUN: rm foo.o
>> >> ;
>> >> ; Test an x86-64 PC-relative MACHO_VANILLA relocation.
>> >>
>> >>        .text
>> >>        .globl  bar
>> >>        .align  16, 0x90
>> >> bar:
>> >>        retq
>> >>
>> >>        .globl  foo
>> >>        .align  16, 0x90
>> >> foo:
>> >> # rtdyld-check: *{4}(inst1 - 4) = (bar - inst1) &
0xffffffff
>> >>        callq   bar
>> >> inst1:
>> >>        retq
>> >>
>> >>
>> >> With this system, we could write targeted regression tests for
every
>> >> relocation type on every platform, and test them on any
system.
>> Failures
>> >> would immediately identify which target and relocation type
broke.
>> >>
>> >> I think this system would massively improve the testability of
the
>> >> RuntimeDyld layer, which is good news in light of the
increased usage
>> MCJIT
>> >> is getting these days.
>> >>
>> >> Please let me know what you think. Comments and critiques are
very
>> welcome,
>> >> both of the language and the proposed workflow.
>> >>
>> >> Cheers,
>> >> Lang.
>> >>
>> >> TL;DR: lhames responds to dblaikie's incessant demand for
test cases.
>> ;)
>> >>
>> >> _______________________________________________
>> >> LLVM Developers mailing list
>> >> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>> >>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140624/7875f665/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rtdyld-testing-expressions.patch
Type: application/octet-stream
Size: 22627 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140624/7875f665/attachment.obj>