llvm dev - Sep 2019 - Help with RISCV and QEMU in llvm testsuite lit testing

Hi,

I am not sure if this is the right forum. Please direct me to the
appropriate place if it isn't so.
Please keep in CC as i am not subscribed to this mailing list.

I am trying to test riscv llvm tools in QEMU using llvm testsuite. As a
trial i am trying only the Single Source C Regression folder. The steps
that i took are :

]$ cmake -DCMAKE_C_COMPILER=/opt/riscv-tools/bin/clang
-DCMAKE_C_FLAGS="--target=riscv32 -march=rv32i -mabi=ilp32"
-DCMAKE_CXX_COMPILER=/opt/riscv-tools/bin/clang
-DCMAKE_CXX_FLAGS="--target=riscv32 -march=rv32i -mabi=ilp32"
-DCMAKE_EXE_LINKER_FLAGS="-lm"
-DTEST_SUITE_RUN_UNDER=/opt/qemu/bin/qemu-riscv32
-DTEST_SUITE_COLLECT_STATS=OFF -DTEST_SUITE_USER_MODE_EMULATION=ON
-DTEST_SUITE_USE_PERF=ON -DARCH="riscv"
-DTEST_SUITE_SUBDIRS="SingleSource/Regression/C"
-C../llvm-test-suite/cmake/caches/O3.cmake ../llvm-test-suite

]$ make -j 12 VERBOSE=1

]$ /opt/riscv-tools/bin/llvm-lit -j 12 -o results.json .

I see that all the tests are failing. When i try the tests manually i could
see that results produced are same as that of the reference output. When
verbose mode is enabled i see that *.test.out files are not being produced.
This is the file that gets compared with the reference file.

FAIL: test-suite ::
SingleSource/Regression/C/Regression-C-2003-06-16-InvalidInitializer.test
(1 of 35)
******************** TEST 'test-suite ::
SingleSource/Regression/C/Regression-C-2003-06-16-InvalidInitializer.test'
FAILED ********************

/òpt/llvm-test/build/tools/timeit --limit-core 0 --limit-cpu 7200 --timeout
7200 --limit-file-size 104857600 --limit-rss-size 838860800
--redirect-input /dev/null --chdir
/òpt/llvm-test/build/SingleSource/Regression/C --summary
/òpt/llvm-test/build/SingleSource/Regression/C/Output/Regression-C-2003-06-16-InvalidInitializer.test.time
/opt/riscv/install-qemu/bin/qemu-riscv32
/òpt/llvm-test/build/SingleSource/Regression/C/Regression-C-2003-06-16-InvalidInitializer
cd /òpt/llvm-test/build/SingleSource/Regression/C ;
/òpt/llvm-test/build/tools/fpcmp
/òpt/llvm-test/build/SingleSource/Regression/C/Output/Regression-C-2003-06-16-InvalidInitializer.test.out
2003-06-16-InvalidInitializer.reference_output

+ cd /òpt/llvm-test/build/SingleSource/Regression/C
+ /òpt/llvm-test/build/tools/fpcmp
/òpt/llvm-test/build/SingleSource/Regression/C/Output/Regression-C-2003-06-16-InvalidInitializer.test.out
2003-06-16-InvalidInitializer.reference_output
/òpt/llvm-test/build/tools/fpcmp: error: unable to open
'/òpt/llvm-test/build/SingleSource/Regression/C/Output/Regression-C-2003-06-16-InvalidInitializer.test.out'


I am not sure who is responsible for the *.test.out file. I am assuming
that this should be generated by QEMU. But I see that there is "exit
0" in
the reference files that are not produced in the QEMU output. Should qemu
be run via a wrapper that captures the output and "exit 0" in a
*.test.out
file ?

Any help is appreciated.

Thanks for your time.

Regards,
Shafi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190925/8a1d311f/attachment.html>

Am Mi., 25. Sept. 2019 um 17:04 Uhr schrieb Mohamed Shafi via llvm-dev
<llvm-dev at lists.llvm.org>:
> I am not sure who is responsible for the *.test.out file. I am assuming
that this should be generated by QEMU. But I see that there is "exit
0" in the reference files that are not produced in the QEMU output. Should
qemu be run via a wrapper that captures the output and "exit 0" in a
*.test.out file ?
The "exit 0" should be produced by the tools/timeit.sh (since you are
using  -DTEST_SUITE_USE_PERF=ON) and has a cmdline switch
--append-exitstatus. The command line for it is produced by
litsupport/modules/timeit.py. No idea why it does not work in your
case, but I hope this help for looking into the issue.

Michael

On Thu, 26 Sep 2019 at 23:02, Michael Kruse <llvmdev at meinersbur.de>
wrote:
>
> Am Mi., 25. Sept. 2019 um 17:04 Uhr schrieb Mohamed Shafi via llvm-dev
> <llvm-dev at lists.llvm.org>:
> > I am not sure who is responsible for the *.test.out file. I am
assuming that this should be generated by QEMU. But I see that there is
"exit 0" in the reference files that are not produced in the QEMU
output. Should qemu be run via a wrapper that captures the output and "exit
0" in a *.test.out file ?
>
> The "exit 0" should be produced by the tools/timeit.sh (since you
are
> using  -DTEST_SUITE_USE_PERF=ON) and has a cmdline switch
> --append-exitstatus. The command line for it is produced by
> litsupport/modules/timeit.py. No idea why it does not work in your
> case, but I hope this help for looking into the issue.
>
> Michael
Thanks for the response. What you have pointed out is correct. Since i
have enabled TEST_SUITE_USER_MODE_EMULATION the comparison with the
reference file is done with a *.out file rather than with the
simulator output directly. The appropriate out file with the simulator
output and the exit code will get generated if timeit.sh is invoked
with --append-exitstatus and --redirect-output options. timeit.sh gets
invoked by the timeit.py file with these options when
config.traditional_output is true. Interestingly lit.local.cfg in
test-suite\SingleSource has this enabled. But for some reason which i
still haven't found out, timeit.sh is invoked as though
config.traditional_output is false. From what i see this issue should
crop up when RISCV target (or a QEMU target for that matter) is tested
with llvm test suite. It would be of great help if some one could
point out what i am doing wrong.

Regards,
Shafi