llvm dev - Aug 2020 - ORC JIT Weekly #19 -- Relocatable object level mocking with llvm-jitlink.

Hi All,

There was no update last week -- I'm still trying to get back into a
regular schedule.

Open-source changes since the last update were:

(1) Some bug fixes for JITLink MachO / arm64 support (PAGE21/PAGEOFF12 now
handle addends correctly).
(2) llvm-jitlink now supports loading archives as well as relocatable
objects.
(3) llvm-jitlink now supports basic object-file level mocking and testing.

That last one is unusual so I want to expand on it. The llvm-jitlink tool
acquired a couple of new options to support this feature:
"--phony-externals" which resolves all otherwise unresolved symbols to
null
instead of erroring, and "-harness" which allows a set of objects to
be
designated as a test harness, with the regular object files implicitly
treated as objects to be tested. If the -harness option is passed
llvm-jitlink installs a JITLink plugin to make some modifications to the
usual linker rules:

-- All symbols referenced by the harness object(s) are promoted to default
visibility so that the harness can directly access static symbols inside
the test object(s)
-- All symbols defined by the harness object(s) override (rather than
clashing with) symbols defined in the test object(s).

With these modifications in place we can selectively test functions in an
object file by mocking their callees. For example, suppose we have an
object file, test_code.o, compiled from the following C source (which we do
not have access to):

void irrelevant_function() { irrelevant_external(); }

int function_to_mock(int X) {
  return /* some function of X */;
}

static void function_to_test() {
  ...
  int Y = function_to_mock();
  printf("Y is %i\n", Y);
}

We would like to know how function_to_test behaves when we change the
behavior of function_to_mock. We can do this by writing a test harness:

void function_to_test();

int function_to_mock(int X) {
  printf("used mock utility function\n");
  return 42;
}

int main(int argc, char *argv[]) {
  function_to_test():
  return 0;
}

This harness code could not be linked against test_code.o under normal
circumstances: function_to_test is static and will fail to resolve from the
harness code, function_to_mock is a duplicate definition, and
irrelevant_external is undefined. Our new llvm-jitlink options take care of
these issues however, and running the test looks like:

% clang -c -o test_code_harness.o test_code_harness.c
% llvm-jitlink -phony-externals test_code.o -harness test_code_harness.o
used mock utility function
Y is 42

This utility may be of interest to people who want to perform some very
late testing on build products to verify that compiled code behaves as
expected. On basic C test cases it works reasonably well. C++ test cases
are another matter: Any code involving classes tends to have a lot of
non-trivial surface area (e.g. vtables) that would require great care to
mock. Perhaps we need a new tool to tackle that problem.

In the meantime, if nothing else this provides a neat example of how
JITLink plugins can be used to modify the default linker behavior.

Regards,
Lang.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200809/7f5ea64c/attachment.html>

Hi Lang

Thanks for explaining the new mocking and testing feature in the
llvm-jitlink tool in such great detail! Indeed, it sounds like an
interesting by-product of your jitlink efforts.

"a lot of non-trivial surface area" is a very polite description of
the
C++ symbols disaster :)

Best
Stefan

On 10/08/2020 08:26, Lang Hames wrote:
> Hi All,
>
> There was no update last week -- I'm still trying to get back into a
> regular schedule.
>
> Open-source changes since the last update were:
>
> (1) Some bug fixes for JITLink MachO / arm64 support
> (PAGE21/PAGEOFF12 now handle addends correctly).
> (2) llvm-jitlink now supports loading archives as well as relocatable
> objects.
> (3) llvm-jitlink now supports basic object-file level mocking and testing.
>
> That last one is unusual so I want to expand on it. The llvm-jitlink
> tool acquired a couple of new options to support this feature:
> "--phony-externals" which resolves all otherwise unresolved
symbols to
> null instead of erroring, and "-harness" which allows a set of
objects
> to be designated as a test harness, with the regular object files
> implicitly treated as objects to be tested. If the -harness option is
> passed llvm-jitlink installs a JITLink plugin to make some
> modifications to the usual linker rules:
>
> -- All symbols referenced by the harness object(s) are promoted to
> default visibility so that the harness can directly access static
> symbols inside the test object(s)
> -- All symbols defined by the harness object(s) override (rather than
> clashing with) symbols defined in the test object(s).
>
> With these modifications in place we can selectively test functions in
> an object file by mocking their callees. For example, suppose we have
> an object file, test_code.o, compiled from the following C source
> (which we do not have access to):
>
> void irrelevant_function() { irrelevant_external(); }
>
> int function_to_mock(int X) {
>   return /* some function of X */;
> }
>
> static void function_to_test() {
>   ...
>   int Y = function_to_mock();
>   printf("Y is %i\n", Y);
> }
>
> We would like to know how function_to_test behaves when we change the
> behavior of function_to_mock. We can do this by writing a test harness:
>
> void function_to_test();
>
> int function_to_mock(int X) {
>   printf("used mock utility function\n");
>   return 42;
> }
>
> int main(int argc, char *argv[]) {
>   function_to_test():
>   return 0; 
> }
>
> This harness code could not be linked against test_code.o under normal
> circumstances: function_to_test is static and will fail to resolve
> from the harness code, function_to_mock is a duplicate definition, and
> irrelevant_external is undefined. Our new llvm-jitlink options take
> care of these issues however, and running the test looks like:
>
> % clang -c -o test_code_harness.o test_code_harness.c
> % llvm-jitlink -phony-externals test_code.o -harness test_code_harness.o
> used mock utility function
> Y is 42
>
> This utility may be of interest to people who want to perform some
> very late testing on build products to verify that compiled code
> behaves as expected. On basic C test cases it works reasonably well.
> C++ test cases are another matter: Any code involving classes tends to
> have a lot of non-trivial surface area (e.g. vtables) that would
> require great care to mock. Perhaps we need a new tool to tackle that
> problem.
>
> In the meantime, if nothing else this provides a neat example of how
> JITLink plugins can be used to modify the default linker behavior.
>
> Regards,
> Lang. 
-- 
https://flowcrypt.com/pub/stefan.graenitz at gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200816/2617e70b/attachment.html>

Hi Stefan,

"a lot of non-trivial surface area" is a very polite description of
the C++
> symbols disaster :)

Indeed. I think tooling may make this tractable: E.g. if debug info is
available then perhaps we can make a tool to synthesize ASTs for classes
that the user wants to mock. I haven't had a chance to think through the
ideas in detail yet though.

-- Lang.

On Sun, Aug 16, 2020 at 1:32 PM Stefan Gränitz <stefan.graenitz at
gmail.com>
wrote:
> Hi Lang
>
> Thanks for explaining the new mocking and testing feature in the
> llvm-jitlink tool in such great detail! Indeed, it sounds like an
> interesting by-product of your jitlink efforts.
>
> "a lot of non-trivial surface area" is a very polite description
of the
> C++ symbols disaster :)
>
> Best
> Stefan
>
> On 10/08/2020 08:26, Lang Hames wrote:
>
> Hi All,
>
> There was no update last week -- I'm still trying to get back into a
> regular schedule.
>
> Open-source changes since the last update were:
>
> (1) Some bug fixes for JITLink MachO / arm64 support (PAGE21/PAGEOFF12 now
> handle addends correctly).
> (2) llvm-jitlink now supports loading archives as well as relocatable
> objects.
> (3) llvm-jitlink now supports basic object-file level mocking and testing.
>
> That last one is unusual so I want to expand on it. The llvm-jitlink tool
> acquired a couple of new options to support this feature:
> "--phony-externals" which resolves all otherwise unresolved
symbols to null
> instead of erroring, and "-harness" which allows a set of objects
to be
> designated as a test harness, with the regular object files implicitly
> treated as objects to be tested. If the -harness option is passed
> llvm-jitlink installs a JITLink plugin to make some modifications to the
> usual linker rules:
>
> -- All symbols referenced by the harness object(s) are promoted to default
> visibility so that the harness can directly access static symbols inside
> the test object(s)
> -- All symbols defined by the harness object(s) override (rather than
> clashing with) symbols defined in the test object(s).
>
> With these modifications in place we can selectively test functions in an
> object file by mocking their callees. For example, suppose we have an
> object file, test_code.o, compiled from the following C source (which we do
> not have access to):
>
> void irrelevant_function() { irrelevant_external(); }
>
> int function_to_mock(int X) {
>   return /* some function of X */;
> }
>
> static void function_to_test() {
>   ...
>   int Y = function_to_mock();
>   printf("Y is %i\n", Y);
> }
>
> We would like to know how function_to_test behaves when we change the
> behavior of function_to_mock. We can do this by writing a test harness:
>
> void function_to_test();
>
> int function_to_mock(int X) {
>   printf("used mock utility function\n");
>   return 42;
> }
>
> int main(int argc, char *argv[]) {
>   function_to_test():
>   return 0;
> }
>
> This harness code could not be linked against test_code.o under normal
> circumstances: function_to_test is static and will fail to resolve from the
> harness code, function_to_mock is a duplicate definition, and
> irrelevant_external is undefined. Our new llvm-jitlink options take care of
> these issues however, and running the test looks like:
>
> % clang -c -o test_code_harness.o test_code_harness.c
> % llvm-jitlink -phony-externals test_code.o -harness test_code_harness.o
> used mock utility function
> Y is 42
>
> This utility may be of interest to people who want to perform some very
> late testing on build products to verify that compiled code behaves as
> expected. On basic C test cases it works reasonably well. C++ test cases
> are another matter: Any code involving classes tends to have a lot of
> non-trivial surface area (e.g. vtables) that would require great care to
> mock. Perhaps we need a new tool to tackle that problem.
>
> In the meantime, if nothing else this provides a neat example of how
> JITLink plugins can be used to modify the default linker behavior.
>
> Regards,
> Lang.
>
> -- https://flowcrypt.com/pub/stefan.graenitz at gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200816/0ef9919d/attachment-0001.html>