llvm dev - Apr 2018 - Wrapping lit commands for device testing

Hi,

Tests in compiler-rt run under both iOS simulators and Android devices, which
requires “wrapping” the run line with the “adb” command for Android, and adding
a prefix / setting some environment variables for simulators/
In case of Android this is done by changing the compilation command to produce a
wrapper which already calls “adb”, and in case of iOS simulators by explicitly
adding the “%run” command in front of run lines.
The solution is far from perfect though: many tests (e.g. libFuzzer) extensively
rely on using arbitrary shell commands to create and manipulate temporary text
files.
Performing those commands in a different environment is done in a very ad-hoc
manner: from what I see, for sanitizer tests those temporary files are then
usually references in the environment variables present in the run command,
which are then parsed to carry those files into the executable environment.
Doing this for libFuzzer would be very difficult and would require a lot of
modifications to test.

I would prefer a more transparent approach: if LIT configuration was to provide
a hook to perform arbitrary replacements/wrapping on each of the executed lines,
the wrapping could be performed without modifying tests at all.

My implementation preference would be adding a method  `def wrap(self, s)` (str
-> str) to the format class (lit/formats/*), and then using it (if non-empty)
on each line being run.
I think that is preferable to a method on a `config` object, since line
transformation is a very heavyweight operation, akin to other methods available
on the `format` class (test discovery semantics and test execution semantics).
Would there be any objections against such an implementation plan?

Regards,
George

Hello George,

I would be very much interested in a (generic) solution that would allow
running tests on a remote host/device. My intended use case for this is
lldb+android, and I am mostly interested in googletest-style tests. In
fact, I proposed something vaguely similar to this (but limited to gtest)
last year <http://lists.llvm.org/pipermail/llvm-dev/2017-May/113370.html>.
However, due to some internal reshuffles, this idea never got past the
prototype stage. Nonetheless, I would still like do something like this
eventually (even better if someone else does it for me :) ).

I would love to hear more about your proposal, as my head is currently
filled with questions about how this would work. (I admit I don't know
anything about how libFuzzer is tested.) It's not clear to me whether you
intend to implement a general purpose remote test execution framework, or a
solution tailored to the needs of libFuzzer. Do you automatically handle
copying the inputs to the remote target? How do you identify what the
inputs are? Do you copy over any llvm tools executed in the run lines? What
about their dependent libraries? etc.

Maybe if you could give an example of a typical test you intend to support
and how it would work, a lot of these questions would be answered. If this
turns out to be something that could be reused for our use case, I would be
willing to chip in to help make it happen.

regards,
pavel

On Mon, 23 Apr 2018 at 15:09, George Karpenkov via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> Hi,
> Tests in compiler-rt run under both iOS simulators and Android devices,
which requires “wrapping” the run line with the “adb” command for Android,
and adding a prefix / setting some environment variables for
simulators/
> In case of Android this is done by changing the compilation command to
produce a wrapper which already calls “adb”, and in case of iOS simulators
by explicitly adding the “%run” command in front of run
lines.
> The solution is far from perfect though: many tests (e.g. libFuzzer)
extensively rely on using arbitrary shell commands to create and manipulate
temporary text files.
> Performing those commands in a different environment is done in a very
ad-hoc manner: from what I see, for sanitizer tests those temporary files
are then usually references in the environment variables present in the run
command, which are then parsed to carry those files into the executable
environment.
> Doing this for libFuzzer would be very difficult and would require a lot
of modifications to test.
> I would prefer a more transparent approach: if LIT configuration was to
provide a hook to perform arbitrary replacements/wrapping on each of the
executed lines, the wrapping could be performed without modifying tests at
all.
> My implementation preference would be adding a method  `def wrap(self,
s)` (str -> str) to the format class (lit/formats/*), and then using it (if
non-empty) on each line being run.
> I think that is preferable to a method on a `config` object, since line
transformation is a very heavyweight operation, akin to other methods
available on the `format` class (test discovery semantics and test
execution semantics).
> Would there be any objections against such an implementation plan?
> Regards,
> George
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

Hi Pavel,

I was thinking about a generic proposal for a while,
but then I’ve realized that a much simpler heuristic suffices for all of my
needs:

 - Prepend all commands which should execute on the device with %run, which runs
a wrapper python script
 - Inside that script use a simple regexp to find all paths
 - sync all paths to the device
 - run the executable on the device
 - sync all paths form the device back to host
> On Apr 23, 2018, at 8:09 AM, Pavel Labath <labath at google.com>
wrote:
> 
> Hello George,
> 
> I would be very much interested in a (generic) solution that would allow
> running tests on a remote host/device. My intended use case for this is
> lldb+android, and I am mostly interested in googletest-style tests.
I’m not sure about those, all my problems were in LIT tests.
> In
> fact, I proposed something vaguely similar to this (but limited to gtest)
> last year
<http://lists.llvm.org/pipermail/llvm-dev/2017-May/113370.html>.
> However, due to some internal reshuffles, this idea never got past the
> prototype stage. Nonetheless, I would still like do something like this
> eventually (even better if someone else does it for me :) ).
> 
> I would love to hear more about your proposal, as my head is currently
> filled with questions about how this would work. (I admit I don't know
> anything about how libFuzzer is tested.) It's not clear to me whether
you
> intend to implement a general purpose remote test execution framework, or a
> solution tailored to the needs of libFuzzer.
Seemed generic enough for most of compiler-rt.
> Do you automatically handle
> copying the inputs to the remote target? How do you identify what the
> inputs are?
Yes, the simple heuristic above seems to work here.
> Do you copy over any llvm tools executed in the run lines?
I copy the command directly after %run wrapper, but not FileCheck/etc.
%run syncs it over to the device.
> What
> about their dependent libraries? etc.
Dynamically linked? Mostly we don’t have any, and for those tests where we do,
some rpath substitution magic is required.

> Maybe if you could give an example of a typical test you intend to support
> and how it would work, a lot of these questions would be answered. If this
> turns out to be something that could be reused for our use case, I would be
> willing to chip in to help make it happen.
Basically, any of libFuzzer tests with %run prepended, e.g.
https://reviews.llvm.org/D45235

George
> 
> regards,
> pavel
> 
> On Mon, 23 Apr 2018 at 15:09, George Karpenkov via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
> 
>> Hi,
> 
>> Tests in compiler-rt run under both iOS simulators and Android devices,
> which requires “wrapping” the run line with the “adb” command for Android,
> and adding a prefix / setting some environment variables for simulators/
>> In case of Android this is done by changing the compilation command to
> produce a wrapper which already calls “adb”, and in case of iOS simulators
> by explicitly adding the “%run” command in front of run lines.
>> The solution is far from perfect though: many tests (e.g. libFuzzer)
> extensively rely on using arbitrary shell commands to create and manipulate
> temporary text files.
>> Performing those commands in a different environment is done in a very
> ad-hoc manner: from what I see, for sanitizer tests those temporary files
> are then usually references in the environment variables present in the run
> command, which are then parsed to carry those files into the executable
> environment.
>> Doing this for libFuzzer would be very difficult and would require a
lot
> of modifications to test.
> 
>> I would prefer a more transparent approach: if LIT configuration was to
> provide a hook to perform arbitrary replacements/wrapping on each of the
> executed lines, the wrapping could be performed without modifying tests at
> all.
> 
>> My implementation preference would be adding a method  `def wrap(self,
> s)` (str -> str) to the format class (lit/formats/*), and then using it
(if
> non-empty) on each line being run.
>> I think that is preferable to a method on a `config` object, since line
> transformation is a very heavyweight operation, akin to other methods
> available on the `format` class (test discovery semantics and test
> execution semantics).
>> Would there be any objections against such an implementation plan?
> 
>> Regards,
>> George
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev