[LLVMdev] Proposal: Improved regression test support for RuntimeDyld/MCJIT.

Lang Hames
2014-06-24T03:54:03Z

Hi Dave,

Jim Grosbach asked the same question, so you're in good company. With
hindsight I think it was a mistake to say "FileCheck workflow". What I
really meant was that this system plays well with lit. Not that your question
about using FileCheck would have been any less valid.

I did consider using FileCheck for this, but decided it was the wrong approach.
The fundamental reason is that there's no demand for textually rendering
RuntimeDyld's memory, and developing a textual renderer so that we could
output text just to pattern match and re-assemble ints in FileCheck would be a
lot of pain for (as far as I can see) no gain.

If there's a desire for FileCheck to support expression evaluation we could
flesh out this evaluator and make it available as a support library that both
FileCheck and RuntimeDyld could use.

You (and independently Nick Kledzik) do raise the really useful idea of
leveraging the disassembler though. I like the idea of adding some special
syntax to disassemble an instruction at a label and use one of its immediates.
That would eliminate a lot of the bit bashing that would have been required on
instruction sets with tricky immediate encodings (E.g. ARM). Something like:

# rtdyld-check: @test_inst[0] = foo - (test_inst + 5)
test_inst:
  callq foo

Cheers,
Lang.

> On Jun 23, 2014, at 5:01 PM, David Blaikie <dblaikie at gmail.com>
wrote:
>
>> On Mon, Jun 23, 2014 at 3:01 PM, Lang Hames <lhames at gmail.com>
wrote:
>> Hi Everyone,
>>
>> For your consideration: A proposal to improve regression test support
for
>> RuntimeDyld.
>
> Thanks for working on this, Lang. It's great to see.
>
>> Short version: We can make RuntimeDyld far more testable by adding a
trivial
>> pointer-expression language that allows us to describe how memory
should
>> look post-relocation. Jump down to "The Proposal" for
details.
>
> I've been trying to puzzle over what this would look like with a
> possibly more general feature*.
>
> What would testing look like if we had a rtdyld dumping mode that
> printed the disassembly of the relocated machine code, and a symbol
> table (or just inserted the labels for the symbols into the
> disassembly?).
>
> I understand we'd need to beef up FileCheck with slightly more
> arithmetic operations - but is it really so much (& would they be so
> useless for other tests) that it's not worth putting it there?
>
> To take your example, here's my vague idea of what it might look like
> to use a dump+FileCheck. The dump would look something like:
>
> (obviously I don't know, nor for this purpose care, how big the
> instructions are, just that they have distinct addresses, etc)
>
>  0x42: bar:
>  0x42:   retq
>  0x43: foo:
>  0x43:   callq 0x42
>  0x44: inst1:
>  0x44:   retq
>
> And the FileCheck equivalent of
>
>  # rtdyld-check: *{4}(inst1 - 4) = (bar - inst1) & 0xffffffff
>
> would be something like:
>
>  CHECK: [[CALL_ADDR:.*]]: bar:
>  CHECK: callq [[CALL_ADDR]]
>
> Which, I suppose, depends on disassembler working correctly, not sure
> if that's high risk/complicated.
>
> Alternatively - could llvm-rtdyld just print a simple description of
> relocations its applied and the location of symbols? (similar to a
> static display of relocations like llvm-objdump -r) then FileCheck
> that.
>
> * all that said, a feature like you've proposed/implemented isn't
> without precedent - clang's -verify is very similar to what you've
got
> here
>
>
>> Long version:
>>
>> Background:
>>
>> For those unfamiliar with it, RuntimeDyld a component of MCJIT,
LLVM's JIT
>> compiler infrastructure. MCJIT produces an object file in memory for
each
>> module that is JIT'd. RuntimeDyld's job is to apply all the
relocations
>> necessary to make the code in the object file runnable. In other words,
>> RuntimeDyld is acting as both the static and dynamic linker for the
JIT.
>>
>> The Problem:
>>
>> We can't directly test RuntimeDyld at the moment. We currently
infer the
>> correctness of RuntimeDyld indirectly from the success of the MCJIT
>> regression tests - if they pass, we assume RuntimeDyld must have done
its
>> job right. That's far from an ideal. The biggest issues with it
are:
>>
>> (1) Each platform is testing only its own relocations and no others.
I.e.
>> X86 testers are testing X86 relocations only. ARM testers are testing
ARM
>> relocations only. If someone running on X86 breaks a relocation for ARM
they
>> won't see the error in their regression test run - they'll have
to wait
>> until an ARM buildbot breaks before they realize anything is wrong.
Fixes
>> for platforms that you don't have access to are difficult to test -
all you
>> can do is eyeball disassembled memory and see if everything looks sane.
This
>> is not much fun.
>>
>> (2) Relocations are produced by CodeGen from IR, rather than described
>> directly. That's a lot of machinery to have between the test-case
and the
>> final result. It is difficult to know what relocations each IR
regression
>> test is testing (and they're often incidental - we don't have a
dedicated
>> relocation test set). This also means that if/when the code generator
>> produces different relocation types the existing tests will keep on
passing
>> but will silently stop testing the thing they used to test.
>>
>> The Proposal:
>>
>> (1) We provide a mechanism for describing how pieces of relocated
memory
>> should look immediately prior to execution, and then inspect the memory
>> rather than executing it. This addresses point (1) above: Tests for any
>> platform can be loaded, linked and verified on any platform. If
you're
>> coding on X86 and you break an ARM relocation you'll know about it
>> immediately.
>>
>> (2) RuntimeDyld test cases should be written in assembly, rather than
IR.
>> This addresses point (2) above - we can cut the code generators out and
>> guarantee that we're testing what we're interested in.
>>
>> The way to do this is to introduce a simple pointer expression
language.
>> This should be able to express things like: "The immediate for
this call
>> points at symbol foo".
>>
>> Symbolically, what I have in mind would look something like:
>>
>>        // some asm ...
>> # assert *(inst1 + 1) = foo
>> inst1:
>>        callq   foo
>>        // some asm...
>>
>> Here we add the "inst1" label to give us a address from which
we can get at
>> the immediate for the call. The " + 1" expression skips the
call opcode (we
>> know the size of the opcode ahead of time, since this is assembly and
so
>> target-specific).
>>
>> To verify that constraints expressed in this language hold, we can add
an
>> expression evaluator to the llvm-rtdyld utility, which is a
command-line
>> interface to RuntimeDyld.
>>
>> I find these things are easier to discuss in the concrete, so I've
attached
>> a basic implementation of this idea. The following discussion is in
terms of
>> my patch, but I'm very open to tweaking all this.
>>
>> The language I've implemented is:
>>
>> test = expr '=' expr
>>
>> expr = '*{' number '}' load_addr_expr
>>     | binary_expr
>>     | '(' expr ')'
>>     | symbol
>>     | number
>>
>> load_addr_expr = symbol
>>               | '(' symbol '+' number ')'
>>               | '(' symbol '-' number ')'
>>
>> binary_expr = expr '+' expr
>>            | expr '-' expr
>>            | expr '&' expr
>>            | expr '|' expr
>>            | expr '<<' expr
>>            | expr '>>' expr
>>
>> This expression language supports simple pointer arithmetic, shifting,
>> masking and loading. All values are internally held as 64-bit
unsigneds,
>> since RuntimeDlyd is designed to support cross-platform linking,
including
>> linking for 64-bit targets from a 32-bit host. I think the only
stand-out
>> wart is the *{#size}<addr> syntax for loads. This comes from the
fact that
>> immediates aren't always 64-bits, so it's not safe to do a
64-bit load: you
>> could read past the end of allocated memory. The #size field indicates
how
>> many bytes to read.
>>
>> This patch adds a "-verify" option to llvm-rtdyld to attach
the expression
>> evaluator to a RuntimeDyld instance after linking. When -verify is
passed,
>> llvm-rtdyld does not execute any code. Files containing rules are
passed via
>> "-check=<filename>" arguments, and rules are read from
any line prefixed
>> with the string "# rtdyld-check: ". The intended workflow is
modeled on the
>> FileCheck regression tests.
>>
>> Here's an example of what a test case for a test for an x86-64
PC-relative
>> MACHO_VANILLA relocation would look like:
>>
>> ; RUN: clang -triple x86_64-apple-macosx10.9.0 -c -o foo.o %s
>> ; RUN: llvm-rtdyld -verify -check=foo.s foo.o
>> ; RUN: rm foo.o
>> ;
>> ; Test an x86-64 PC-relative MACHO_VANILLA relocation.
>>
>>        .text
>>        .globl  bar
>>        .align  16, 0x90
>> bar:
>>        retq
>>
>>        .globl  foo
>>        .align  16, 0x90
>> foo:
>> # rtdyld-check: *{4}(inst1 - 4) = (bar - inst1) & 0xffffffff
>>        callq   bar
>> inst1:
>>        retq
>>
>>
>> With this system, we could write targeted regression tests for every
>> relocation type on every platform, and test them on any system.
Failures
>> would immediately identify which target and relocation type broke.
>>
>> I think this system would massively improve the testability of the
>> RuntimeDyld layer, which is good news in light of the increased usage
MCJIT
>> is getting these days.
>>
>> Please let me know what you think. Comments and critiques are very
welcome,
>> both of the language and the proposed workflow.
>>
>> Cheers,
>> Lang.
>>
>> TL;DR: lhames responds to dblaikie's incessant demand for test
cases. ;)
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>