llvm dev - Aug 2019 - RFC: Adding GCC C Torture Suite to External Test Suites

TL;DR: I am proposing to add the GCC C Torture suite [1], as an additional
external source of tests for the “nightly” test suite. If you are willing to
review the patch, it is here: https://reviews.llvm.org/D66887

Background:

While working on the RISC-V backend, we have found it useful to use additional
test suites beyond the in-tree Clang and LLVM tests and the LLVM nightly tests,
in order to ensure the compiler is correct. Internally at lowRISC, we have been
running the GCC C Torture suite [1] using custom scripts in order to ensure the
backend can handle these tests as well.

The main advantage is that the torture suite provide a corpus of simple
executable tests that can relatively easily be minimised to identify the sources
of bugs or regressions. During the bringup of the RISC-V backend, the executable
torture tests were the main litmus test for backend correctness.

We are not the only backend to have found this test suite helpful, it is noted
in the WebAssembly backend that they have found the GCC Torture suite helpful
during their backend bring-up as well [2].

The GCC C Torture Suite is composed of four sets of tests. The set we have found
most useful is the “execute” tests, which are single-source-file test cases that
should be able to be compiled, linked, and run without issues. The next most
useful group are the “compile” tests, which should be able to be compiled
without issues. There is more documentation about the test suites here, under
“gcc.c-torture” [3].

Implementation:

The LLVM Nightly test suite has support for external test suites like SPEC,
where there are licencing difficulties with the tests, but where the llvm
project would still like to be able to run these tests as part of our general
compiler testing and benchmarking.

I have prepared a patch to llvm-test-suite [0], which adds support for the
“execute” tests in the GCC C Torture Suite. At the moment, this patch only
covers the CMake build system, which I am more familiar with.

Limitations:

The main limitation at the moment (beyond lack of Makefile support) is that the
GCC torture suite has a lot of tests that rely on internal features of GCC, or
GNU extensions that clang has not implemented (like nested functions). At the
moment the patch includes a long blacklist of test cases that do not run with
clang. This is sufficient for the moment, but could of course be improved - the
DejaGnu comments in the test cases allow them to add target-specific C Flags,
and to skip tests that rely on internal features.

A promising direction on this point is the work by some Embecosm developers on a
version of the GCC Test Suite that is explicit about which test cases need GCC
internal features [4]. This has included both modernising tests, and adding
better DejaGnu filters and options.

In the future, it would be nice to be able to generate the list of
compiler-compatible target-specific test cases (and their cflags) from the
DejaGnu information, and use this information within the CMake test suite. Most
optimal would be the ability to generate this info, and save the generated info
into the test suite so that test runners don’t need to run an additional tool
during configuration.

Summary:

I am proposing to add the GCC C Torture suite as an additional external test
suite. This will allow us to run these extra test cases regularly and within
existing LLVM test suite infrastructure. I welcome volunteers to review [0], and
input from anyone else who has been using the torture suite in compiler
development or has relevant out-of-tree work.

[0]: https://reviews.llvm.org/D66887 
[1]: https://gcc.gnu.org/viewcvs/gcc/trunk/gcc/testsuite/gcc.c-torture/
[2]:
https://github.com/llvm/llvm-project/blob/master/llvm/lib/Target/WebAssembly/README.txt
[3]: https://gcc.gnu.org/onlinedocs/gccint/C-Tests.html 
[4]: https://github.com/embecosm/gcc-for-llvm-testing 

--
Sam Elliott
Software Developer - LLVM
lowRISC CIC
selliott at lowrisc.org
--

On 8/30/19 10:18 AM, Sam Elliott via llvm-dev wrote:
> TL;DR: I am proposing to add the GCC C Torture suite [1], as an additional
external source of tests for the “nightly” test suite. If you are willing to
review the patch, it is here: https://reviews.llvm.org/D66887
>
> Background:
>
> While working on the RISC-V backend, we have found it useful to use
additional test suites beyond the in-tree Clang and LLVM tests and the LLVM
nightly tests, in order to ensure the compiler is correct. Internally at
lowRISC, we have been running the GCC C Torture suite [1] using custom scripts
in order to ensure the backend can handle these tests as well.
>
> The main advantage is that the torture suite provide a corpus of simple
executable tests that can relatively easily be minimised to identify the sources
of bugs or regressions. During the bringup of the RISC-V backend, the executable
torture tests were the main litmus test for backend correctness.
>
> We are not the only backend to have found this test suite helpful, it is
noted in the WebAssembly backend that they have found the GCC Torture suite
helpful during their backend bring-up as well [2].
>
> The GCC C Torture Suite is composed of four sets of tests. The set we have
found most useful is the “execute” tests, which are single-source-file test
cases that should be able to be compiled, linked, and run without issues. The
next most useful group are the “compile” tests, which should be able to be
compiled without issues. There is more documentation about the test suites here,
under “gcc.c-torture” [3].
>
> Implementation:
>
> The LLVM Nightly test suite has support for external test suites like SPEC,
where there are licencing difficulties with the tests,

Given the issues with blacklisting tests, etc. I wonder whether it would 
be more useful, and acceptable, to import the tests themselves. Are the 
tests all, to the best of your knowledge, GPLv3?

  -Hal

>   but where the llvm project would still like to be able to run these tests
as part of our general compiler testing and benchmarking.
>
> I have prepared a patch to llvm-test-suite [0], which adds support for the
“execute” tests in the GCC C Torture Suite. At the moment, this patch only
covers the CMake build system, which I am more familiar with.
>
> Limitations:
>
> The main limitation at the moment (beyond lack of Makefile support) is that
the GCC torture suite has a lot of tests that rely on internal features of GCC,
or GNU extensions that clang has not implemented (like nested functions). At the
moment the patch includes a long blacklist of test cases that do not run with
clang. This is sufficient for the moment, but could of course be improved - the
DejaGnu comments in the test cases allow them to add target-specific C Flags,
and to skip tests that rely on internal features.
>
> A promising direction on this point is the work by some Embecosm developers
on a version of the GCC Test Suite that is explicit about which test cases need
GCC internal features [4]. This has included both modernising tests, and adding
better DejaGnu filters and options.
>
> In the future, it would be nice to be able to generate the list of
compiler-compatible target-specific test cases (and their cflags) from the
DejaGnu information, and use this information within the CMake test suite. Most
optimal would be the ability to generate this info, and save the generated info
into the test suite so that test runners don’t need to run an additional tool
during configuration.
>
> Summary:
>
> I am proposing to add the GCC C Torture suite as an additional external
test suite. This will allow us to run these extra test cases regularly and
within existing LLVM test suite infrastructure. I welcome volunteers to review
[0], and input from anyone else who has been using the torture suite in compiler
development or has relevant out-of-tree work.
>
> [0]: https://reviews.llvm.org/D66887
> [1]: https://gcc.gnu.org/viewcvs/gcc/trunk/gcc/testsuite/gcc.c-torture/
> [2]:
https://github.com/llvm/llvm-project/blob/master/llvm/lib/Target/WebAssembly/README.txt
> [3]: https://gcc.gnu.org/onlinedocs/gccint/C-Tests.html
> [4]: https://github.com/embecosm/gcc-for-llvm-testing
>
> --
> Sam Elliott
> Software Developer - LLVM
> lowRISC CIC
> selliott at lowrisc.org
> --
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

On Fri, 30 Aug 2019 at 17:34, Finkel, Hal J. via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
>
> On 8/30/19 10:18 AM, Sam Elliott via llvm-dev wrote:
> > TL;DR: I am proposing to add the GCC C Torture suite [1], as an
additional external source of tests for the “nightly” test suite. If you are
willing to review the patch, it is here: https://reviews.llvm.org/D66887
> >
> > Background:
> >
> > While working on the RISC-V backend, we have found it useful to use
additional test suites beyond the in-tree Clang and LLVM tests and the LLVM
nightly tests, in order to ensure the compiler is correct. Internally at
lowRISC, we have been running the GCC C Torture suite [1] using custom scripts
in order to ensure the backend can handle these tests as well.
> >
> > The main advantage is that the torture suite provide a corpus of
simple executable tests that can relatively easily be minimised to identify the
sources of bugs or regressions. During the bringup of the RISC-V backend, the
executable torture tests were the main litmus test for backend correctness.
> >
> > We are not the only backend to have found this test suite helpful, it
is noted in the WebAssembly backend that they have found the GCC Torture suite
helpful during their backend bring-up as well [2].
> >
> > The GCC C Torture Suite is composed of four sets of tests. The set we
have found most useful is the “execute” tests, which are single-source-file test
cases that should be able to be compiled, linked, and run without issues. The
next most useful group are the “compile” tests, which should be able to be
compiled without issues. There is more documentation about the test suites here,
under “gcc.c-torture” [3].
> >
> > Implementation:
> >
> > The LLVM Nightly test suite has support for external test suites like
SPEC, where there are licencing difficulties with the tests,
>
>
> Given the issues with blacklisting tests, etc. I wonder whether it would
> be more useful, and acceptable, to import the tests themselves. Are the
> tests all, to the best of your knowledge, GPLv3?
Yes, the tests are, by our understanding, GPLv3. Importing could be a
reasonable option - but how do you see that interacting with the need
for blacklisting tests? Are you imagining that we just wouldn't import
the GCC-only tests into the repo? Obviously there'd still be potential
architecture-specific blacklisting, and hassle around tests that need
custom options.

Best,

Alex