llvm dev - Aug 2019 - RFC: Adding GCC C Torture Suite to External Test Suites

On Fri, 30 Aug 2019 at 17:34, Finkel, Hal J. via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
>
> On 8/30/19 10:18 AM, Sam Elliott via llvm-dev wrote:
> > TL;DR: I am proposing to add the GCC C Torture suite [1], as an
additional external source of tests for the “nightly” test suite. If you are
willing to review the patch, it is here: https://reviews.llvm.org/D66887
> >
> > Background:
> >
> > While working on the RISC-V backend, we have found it useful to use
additional test suites beyond the in-tree Clang and LLVM tests and the LLVM
nightly tests, in order to ensure the compiler is correct. Internally at
lowRISC, we have been running the GCC C Torture suite [1] using custom scripts
in order to ensure the backend can handle these tests as well.
> >
> > The main advantage is that the torture suite provide a corpus of
simple executable tests that can relatively easily be minimised to identify the
sources of bugs or regressions. During the bringup of the RISC-V backend, the
executable torture tests were the main litmus test for backend correctness.
> >
> > We are not the only backend to have found this test suite helpful, it
is noted in the WebAssembly backend that they have found the GCC Torture suite
helpful during their backend bring-up as well [2].
> >
> > The GCC C Torture Suite is composed of four sets of tests. The set we
have found most useful is the “execute” tests, which are single-source-file test
cases that should be able to be compiled, linked, and run without issues. The
next most useful group are the “compile” tests, which should be able to be
compiled without issues. There is more documentation about the test suites here,
under “gcc.c-torture” [3].
> >
> > Implementation:
> >
> > The LLVM Nightly test suite has support for external test suites like
SPEC, where there are licencing difficulties with the tests,
>
>
> Given the issues with blacklisting tests, etc. I wonder whether it would
> be more useful, and acceptable, to import the tests themselves. Are the
> tests all, to the best of your knowledge, GPLv3?
Yes, the tests are, by our understanding, GPLv3. Importing could be a
reasonable option - but how do you see that interacting with the need
for blacklisting tests? Are you imagining that we just wouldn't import
the GCC-only tests into the repo? Obviously there'd still be potential
architecture-specific blacklisting, and hassle around tests that need
custom options.

Best,

Alex

On 8/30/19 2:21 PM, Alex Bradbury wrote:
> On Fri, 30 Aug 2019 at 17:34, Finkel, Hal J. via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>>
>> On 8/30/19 10:18 AM, Sam Elliott via llvm-dev wrote:
>>> TL;DR: I am proposing to add the GCC C Torture suite [1], as an
additional external source of tests for the “nightly” test suite. If you are
willing to review the patch, it is here: https://reviews.llvm.org/D66887
>>>
>>> Background:
>>>
>>> While working on the RISC-V backend, we have found it useful to use
additional test suites beyond the in-tree Clang and LLVM tests and the LLVM
nightly tests, in order to ensure the compiler is correct. Internally at
lowRISC, we have been running the GCC C Torture suite [1] using custom scripts
in order to ensure the backend can handle these tests as well.
>>>
>>> The main advantage is that the torture suite provide a corpus of
simple executable tests that can relatively easily be minimised to identify the
sources of bugs or regressions. During the bringup of the RISC-V backend, the
executable torture tests were the main litmus test for backend correctness.
>>>
>>> We are not the only backend to have found this test suite helpful,
it is noted in the WebAssembly backend that they have found the GCC Torture
suite helpful during their backend bring-up as well [2].
>>>
>>> The GCC C Torture Suite is composed of four sets of tests. The set
we have found most useful is the “execute” tests, which are single-source-file
test cases that should be able to be compiled, linked, and run without issues.
The next most useful group are the “compile” tests, which should be able to be
compiled without issues. There is more documentation about the test suites here,
under “gcc.c-torture” [3].
>>>
>>> Implementation:
>>>
>>> The LLVM Nightly test suite has support for external test suites
like SPEC, where there are licencing difficulties with the tests,
>>
>> Given the issues with blacklisting tests, etc. I wonder whether it
would
>> be more useful, and acceptable, to import the tests themselves. Are the
>> tests all, to the best of your knowledge, GPLv3?
> Yes, the tests are, by our understanding, GPLv3. Importing could be a
> reasonable option - but how do you see that interacting with the need
> for blacklisting tests? Are you imagining that we just wouldn't import
> the GCC-only tests into the repo?

That was, indeed, what I had in mind, but...

>   Obviously there'd still be potential
> architecture-specific blacklisting, and hassle around tests that need
> custom options.

In general, it seems like keeping any blacklists and special options in 
sync with the tests will be easiest if the configurations and the tests 
are together in the repository.

I'm just curious what your thoughts are on which will be better in the 
long term.

Thanks again,

Hal

>
> Best,
>
> Alex
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

There are 1500 tests total, and about 100 on the platform-agnostic blacklist.
Alex and I do not think this is an onerous burden for maintenance, either as an
external test suite or if the test suite is imported.

In the long term, if we import the tests, we know we will have to do updates
when the Embecosm work lands, and beyond that updates can be more sporadic. It’s
not clear to me how much harder these updates will be than if the test suite
remains external.

We would welcome more views as to whether this suite should be imported or
should be an external test suite.

It would also be useful to understand the status of the CMake vs Makefile build
system in the nightly test suite, to understand whether we need to support
Makefiles, or whether Makefiles will be disappearing from the nightly tests soon
(as they have already from the main project repo).

Sam

> On 30 Aug 2019, at 9:22PM, Finkel, Hal J. <hfinkel at anl.gov> wrote:
> 
> 
> On 8/30/19 2:21 PM, Alex Bradbury wrote:
>> On Fri, 30 Aug 2019 at 17:34, Finkel, Hal J. via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>>> 
>>> On 8/30/19 10:18 AM, Sam Elliott via llvm-dev wrote:
>>>> TL;DR: I am proposing to add the GCC C Torture suite [1], as an
additional external source of tests for the “nightly” test suite. If you are
willing to review the patch, it is here: https://reviews.llvm.org/D66887
>>>> 
>>>> Background:
>>>> 
>>>> While working on the RISC-V backend, we have found it useful to
use additional test suites beyond the in-tree Clang and LLVM tests and the LLVM
nightly tests, in order to ensure the compiler is correct. Internally at
lowRISC, we have been running the GCC C Torture suite [1] using custom scripts
in order to ensure the backend can handle these tests as well.
>>>> 
>>>> The main advantage is that the torture suite provide a corpus
of simple executable tests that can relatively easily be minimised to identify
the sources of bugs or regressions. During the bringup of the RISC-V backend,
the executable torture tests were the main litmus test for backend correctness.
>>>> 
>>>> We are not the only backend to have found this test suite
helpful, it is noted in the WebAssembly backend that they have found the GCC
Torture suite helpful during their backend bring-up as well [2].
>>>> 
>>>> The GCC C Torture Suite is composed of four sets of tests. The
set we have found most useful is the “execute” tests, which are
single-source-file test cases that should be able to be compiled, linked, and
run without issues. The next most useful group are the “compile” tests, which
should be able to be compiled without issues. There is more documentation about
the test suites here, under “gcc.c-torture” [3].
>>>> 
>>>> Implementation:
>>>> 
>>>> The LLVM Nightly test suite has support for external test
suites like SPEC, where there are licencing difficulties with the tests,
>>> 
>>> Given the issues with blacklisting tests, etc. I wonder whether it
would
>>> be more useful, and acceptable, to import the tests themselves. Are
the
>>> tests all, to the best of your knowledge, GPLv3?
>> Yes, the tests are, by our understanding, GPLv3. Importing could be a
>> reasonable option - but how do you see that interacting with the need
>> for blacklisting tests? Are you imagining that we just wouldn't
import
>> the GCC-only tests into the repo?
> 
> 
> That was, indeed, what I had in mind, but...
> 
> 
>>  Obviously there'd still be potential
>> architecture-specific blacklisting, and hassle around tests that need
>> custom options.
> 
> 
> In general, it seems like keeping any blacklists and special options in 
> sync with the tests will be easiest if the configurations and the tests 
> are together in the repository.
> 
> I'm just curious what your thoughts are on which will be better in the 
> long term.
> 
> Thanks again,
> 
> Hal
> 
> 
>> 
>> Best,
>> 
>> Alex
> 
> -- 
> Hal Finkel
> Lead, Compiler Technology and Programming Languages
> Leadership Computing Facility
> Argonne National Laboratory
--
Sam Elliott
Software Developer - LLVM
lowRISC CIC
selliott at lowrisc.org
--