llvm dev - Sep 2014 - [LLVMdev] QEMU testing for LIT execution tests

So I started prototyping remote testing for libcxx over the weekend:
   https://github.com/jroelofs/libcxx/tree/remote_test

The SSHExecutor isn't quite finished yet, but this should give an idea of
what
I've got in mind.

Does this look generic enough to work for other projects under the llvm-umbrella
(LNT, test-suite, compiler_rt, libcxxabi, etc)?  Are there use cases that you 
see where this will not work?


Jon

On 8/25/14, 6:48 PM, Greg Fitzgerald wrote:
> Hi Jon,
>
> +1 on adding support to run cross-compiled libcxx tests.  +Marshall
>
> To add cross-compilation support to the compiler-rt test suite, I
> added '%run' variable to the 'RUN:' lines in each lit test.
For
> example:
>
>
https://github.com/llvm-mirror/compiler-rt/blob/master/test/asan/TestCases/atexit_stats.cc
>
>
> and substituted '%run' with 'config.emulator' in the lit
configuration file:
>
> https://github.com/llvm-mirror/compiler-rt/blob/master/test/lit.common.cfg
>
>
> which is set via CMake option COMPILER_RT_EMULATOR:
>
>
https://github.com/llvm-mirror/compiler-rt/blob/master/test/lit.common.configured.in
>
>
> To configure the build to use QEMU (or adb, scp, etc):
>
> $ cmake -DCOMPILER_RT_EMULATOR=qemu-aarch64
>
> If you need additional parameters, you can either add them to the
> COMPILER_RT_EMULATOR string or add a wrapper shell script.
>
>
> To ensure new tests include '%run', I added the script
'litlint.py'
> and invoke it within the CMake build:
>
>
https://github.com/llvm-mirror/compiler-rt/blob/master/lib/sanitizer_common/scripts/litlint.py
>
>
> Let me know if you have any questions.  Feel free to include me on
> Phabricator reviews.
>
> -Greg
>
>
> On Fri, Aug 22, 2014 at 7:30 AM, Renato Golin <renato.golin at
linaro.org> wrote:
>> On 22 August 2014 01:48, Jim Grosbach <grosbach at apple.com>
wrote:
>>> No specific comments at the moment, but I do want to say that I
very much believe this is worth pursuing. Better testing infrastructure for
remote targets will be an absolutely fantastic improvement. Thank you and Dan
both for pushing forward on the topic.
>>
>> +1
>>
>> I remember Greg  (cc'd) did some of that in compiler-rt (or was it
>> sanitizers?). He's not active in LLVM any more, but maybe you guys
>> could share some thoughts.
>>
>> cheers,
>> --renato
-- 
Jon Roelofs
jonathan at codesourcery.com
CodeSourcery / Mentor Embedded

Hi Jon,

Compiler-rt, libcxx, libcxxabi should all use the same
cross-compilation testing strategy.  Compiler-rt already has a working
solution.  If it is inadequate for libcxx or libcxxabi, can we start
by addressing those deficiencies?

Thanks,
Greg


On Mon, Sep 8, 2014 at 10:11 AM, Jonathan Roelofs
<jonathan at codesourcery.com> wrote:
> So I started prototyping remote testing for libcxx over the weekend:
>   https://github.com/jroelofs/libcxx/tree/remote_test
>
> The SSHExecutor isn't quite finished yet, but this should give an idea
of
> what I've got in mind.
>
> Does this look generic enough to work for other projects under the
> llvm-umbrella (LNT, test-suite, compiler_rt, libcxxabi, etc)?  Are there
use
> cases that you see where this will not work?
>
>
> Jon
>
>
> On 8/25/14, 6:48 PM, Greg Fitzgerald wrote:
>>
>> Hi Jon,
>>
>> +1 on adding support to run cross-compiled libcxx tests.  +Marshall
>>
>> To add cross-compilation support to the compiler-rt test suite, I
>> added '%run' variable to the 'RUN:' lines in each lit
test.  For
>> example:
>>
>>
>>
https://github.com/llvm-mirror/compiler-rt/blob/master/test/asan/TestCases/atexit_stats.cc
>>
>>
>> and substituted '%run' with 'config.emulator' in the
lit configuration
>> file:
>>
>>
https://github.com/llvm-mirror/compiler-rt/blob/master/test/lit.common.cfg
>>
>>
>> which is set via CMake option COMPILER_RT_EMULATOR:
>>
>>
>>
https://github.com/llvm-mirror/compiler-rt/blob/master/test/lit.common.configured.in
>>
>>
>> To configure the build to use QEMU (or adb, scp, etc):
>>
>> $ cmake -DCOMPILER_RT_EMULATOR=qemu-aarch64
>>
>> If you need additional parameters, you can either add them to the
>> COMPILER_RT_EMULATOR string or add a wrapper shell script.
>>
>>
>> To ensure new tests include '%run', I added the script
'litlint.py'
>> and invoke it within the CMake build:
>>
>>
>>
https://github.com/llvm-mirror/compiler-rt/blob/master/lib/sanitizer_common/scripts/litlint.py
>>
>>
>> Let me know if you have any questions.  Feel free to include me on
>> Phabricator reviews.
>>
>> -Greg
>>
>>
>> On Fri, Aug 22, 2014 at 7:30 AM, Renato Golin <renato.golin at
linaro.org>
>> wrote:
>>>
>>> On 22 August 2014 01:48, Jim Grosbach <grosbach at apple.com>
wrote:
>>>>
>>>> No specific comments at the moment, but I do want to say that I
very
>>>> much believe this is worth pursuing. Better testing
infrastructure for
>>>> remote targets will be an absolutely fantastic improvement.
Thank you and
>>>> Dan both for pushing forward on the topic.
>>>
>>>
>>> +1
>>>
>>> I remember Greg  (cc'd) did some of that in compiler-rt (or was
it
>>> sanitizers?). He's not active in LLVM any more, but maybe you
guys
>>> could share some thoughts.
>>>
>>> cheers,
>>> --renato
>
>
> --
> Jon Roelofs
> jonathan at codesourcery.com
> CodeSourcery / Mentor Embedded

On 9/8/14, 12:39 PM, Greg Fitzgerald wrote:
> Hi Jon,
>
> Compiler-rt, libcxx, libcxxabi should all use the same
> cross-compilation testing strategy.  Compiler-rt already has a working
> solution.  If it is inadequate for libcxx or libcxxabi, can we start
> by addressing those deficiencies?
One issue to start with is that none of the libcxx/libcxxabi tests have RUN 
lines. All of the tests are either: *.fail.cpp ones that fail to compile, or 
*.pass.cpp ones that are expected to exit with a return status of 0 without 
failing an asserts. Granted, this is more of a test semantic issue, but there 
are a *lot* of tests to update in order to converge on compiler_rt's
'RUN: ' format.

[Sidenote: I don't see any 'RUN:' lines on the builtins unit tests.
How does it
work for those ones?]

Another issue, which Dan Albert recently pointed out to me, is that there are 
several libcxx tests which would require copying in of test data (in addition to
test executables).  I think that this necessitates inventing a new LIT syntax, 
something like '// DEPENDS-ON: filename.dat' in order to inform the
lit.cfg that
the dependent file also needs to be copied to the remote target.

A third issue is that I don't think the current '%run' solution
works for full
Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever on HOST, and 
the tests themselves on TARGET, where {BUILD,HOST,TARGET} are all different 
machines).  This use case is of particular interest to me. I suppose %run could 
be renamed %run_target, add %run_host, and then have an implied no-op %run_build
on commands that don't have %run_target or %run_host. Perhaps someone might
want
to have BUILD be a different machine than where LIT runs, in which case 
%run_build would have to be explicit (but I'm not really interested in that
use
case).


Cheers,
Jon
>
> Thanks,
> Greg
>
>
-- 
Jon Roelofs
jonathan at codesourcery.com
CodeSourcery / Mentor Embedded