llvm dev - May 2020 - How to execute cross-compiled libcxx tests on a remote target?

Hi Louis,

I saw that you were the one who implemented the new testing system for 
libcxx, so I figured that you might be able to help me out with my problem.

I'm working on a downstream target and we're now in the process of 
adding support for libcxx. We would like to leverage the existing libcxx 
test suite to make sure that the implementation is working correctly. 
The problem is that our target is a bare-metal target and we need to 
cross-compile the tests and execute them remotely. There isn't really 
any documentation for how to do it, but I came across an old email 
thread from 2015 which said to implement an executor and use that. 
However, it seems like those executors are not used with the new test 
format any longer (except for a check whether to use `run.py` or `ssh.py`).

Also, in the original Phabricator review where you first introduced the 
new format, you said:

 >As a side effect of this design, configuration files for the test 
suite can be as simple as:
 >
 >config.substitutions.append(('%{cxx}',
'<path-to-compiler>'))
 >config.substitutions.append(('%{compile_flags}',
'<flags>'))
 >config.substitutions.append(('%{link_flags}',
'<flags>'))
 >config.substitutions.append(('%{exec}',
'<script-to-execute>'))
 >
 >This should allow storing lit.cfg files for various configurations 
directly in the repository

However I'm not sure what exactly I need to do here. Do I need to write 
my own `lit.site.cfg` and point CMake to it? If possible I'd like to 
reuse the default- and auto-generated config as much as possible.

We have a script which can execute a cross-compiled binary on our remote 
target. I just want the test suite to call this script. Could you give 
me some help on what I need to do to make this work?

Cheers,

Dominik

-- 
----------------------------------------------------------------------
Dominik Montada                   Email: dominik.montada at hightec-rt.com
HighTec EDV-Systeme GmbH          Phone: +49 681 92613 19
Europaallee 19                    Fax:   +49-681-92613-26
D-66113 Saarbrücken               WWW: http://www.hightec-rt.com

Managing Director: Vera Strothmann
Register Court: Saarbrücken, HRB 10445, VAT ID: DE 138344222

This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient please notify the sender immediately
and destroy this e-mail. Any unauthorised copying, disclosure or
distribution of the material in this e-mail is strictly forbidden.
---

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5409 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200528/740cbe75/attachment.bin>

BCC: LLVM Dev

Hi,
> On May 28, 2020, at 06:19, Dominik Montada <dominik.montada at
hightec-rt.com> wrote:
> 
> Hi Louis,
> 
> I saw that you were the one who implemented the new testing system for
libcxx, so I figured that you might be able to help me out with my problem.
> 
> I'm working on a downstream target and we're now in the process of
adding support for libcxx. We would like to leverage the existing libcxx test
suite to make sure that the implementation is working correctly. The problem is
that our target is a bare-metal target and we need to cross-compile the tests
and execute them remotely. There isn't really any documentation for how to
do it, but I came across an old email thread from 2015 which said to implement
an executor and use that. However, it seems like those executors are not used
with the new test format any longer (except for a check whether to use `run.py`
or `ssh.py`).
> 
> Also, in the original Phabricator review where you first introduced the new
format, you said:
> 
> >As a side effect of this design, configuration files for the test suite
can be as simple as:
> >
> >config.substitutions.append(('%{cxx}',
'<path-to-compiler>'))
> >config.substitutions.append(('%{compile_flags}',
'<flags>'))
> >config.substitutions.append(('%{link_flags}',
'<flags>'))
> >config.substitutions.append(('%{exec}',
'<script-to-execute>'))
> >
> >This should allow storing lit.cfg files for various configurations
directly in the repository
> 
> However I'm not sure what exactly I need to do here. Do I need to write
my own `lit.site.cfg` and point CMake to it? If possible I'd like to reuse
the default- and auto-generated config as much as possible.
> 
> We have a script which can execute a cross-compiled binary on our remote
target. I just want the test suite to call this script. Could you give me some
help on what I need to do to make this work?
As of 96e6cbbf941d0f937b7e823433d4c222967a1817 (committed today), you should be
able to use:

    cmake [...] -DLIBCXX_EXECUTOR="<path-to-script>"

and that script will be used to run executables in the test suite. Look at how
ssh.py and run.py handle their arguments for the arguments that should be
accepted by your script, however it doesn't really differ much from old
style executors. I'll work on documenting those in the libcxx docs.

Please reach out again if you run into issues with the test suite, I'll help
you resolve them. I'd like everyone to move to the new format, which is
simpler and more flexible.

Cheers,
Louis