llvm dev - Sep 2019 - RFC: Adding GCC C Torture Suite to External Test Suites

There are 1500 tests total, and about 100 on the platform-agnostic blacklist.
Alex and I do not think this is an onerous burden for maintenance, either as an
external test suite or if the test suite is imported.

In the long term, if we import the tests, we know we will have to do updates
when the Embecosm work lands, and beyond that updates can be more sporadic. It’s
not clear to me how much harder these updates will be than if the test suite
remains external.

We would welcome more views as to whether this suite should be imported or
should be an external test suite.

It would also be useful to understand the status of the CMake vs Makefile build
system in the nightly test suite, to understand whether we need to support
Makefiles, or whether Makefiles will be disappearing from the nightly tests soon
(as they have already from the main project repo).

Sam

> On 30 Aug 2019, at 9:22PM, Finkel, Hal J. <hfinkel at anl.gov> wrote:
> 
> 
> On 8/30/19 2:21 PM, Alex Bradbury wrote:
>> On Fri, 30 Aug 2019 at 17:34, Finkel, Hal J. via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>>> 
>>> On 8/30/19 10:18 AM, Sam Elliott via llvm-dev wrote:
>>>> TL;DR: I am proposing to add the GCC C Torture suite [1], as an
additional external source of tests for the “nightly” test suite. If you are
willing to review the patch, it is here: https://reviews.llvm.org/D66887
>>>> 
>>>> Background:
>>>> 
>>>> While working on the RISC-V backend, we have found it useful to
use additional test suites beyond the in-tree Clang and LLVM tests and the LLVM
nightly tests, in order to ensure the compiler is correct. Internally at
lowRISC, we have been running the GCC C Torture suite [1] using custom scripts
in order to ensure the backend can handle these tests as well.
>>>> 
>>>> The main advantage is that the torture suite provide a corpus
of simple executable tests that can relatively easily be minimised to identify
the sources of bugs or regressions. During the bringup of the RISC-V backend,
the executable torture tests were the main litmus test for backend correctness.
>>>> 
>>>> We are not the only backend to have found this test suite
helpful, it is noted in the WebAssembly backend that they have found the GCC
Torture suite helpful during their backend bring-up as well [2].
>>>> 
>>>> The GCC C Torture Suite is composed of four sets of tests. The
set we have found most useful is the “execute” tests, which are
single-source-file test cases that should be able to be compiled, linked, and
run without issues. The next most useful group are the “compile” tests, which
should be able to be compiled without issues. There is more documentation about
the test suites here, under “gcc.c-torture” [3].
>>>> 
>>>> Implementation:
>>>> 
>>>> The LLVM Nightly test suite has support for external test
suites like SPEC, where there are licencing difficulties with the tests,
>>> 
>>> Given the issues with blacklisting tests, etc. I wonder whether it
would
>>> be more useful, and acceptable, to import the tests themselves. Are
the
>>> tests all, to the best of your knowledge, GPLv3?
>> Yes, the tests are, by our understanding, GPLv3. Importing could be a
>> reasonable option - but how do you see that interacting with the need
>> for blacklisting tests? Are you imagining that we just wouldn't
import
>> the GCC-only tests into the repo?
> 
> 
> That was, indeed, what I had in mind, but...
> 
> 
>>  Obviously there'd still be potential
>> architecture-specific blacklisting, and hassle around tests that need
>> custom options.
> 
> 
> In general, it seems like keeping any blacklists and special options in 
> sync with the tests will be easiest if the configurations and the tests 
> are together in the repository.
> 
> I'm just curious what your thoughts are on which will be better in the 
> long term.
> 
> Thanks again,
> 
> Hal
> 
> 
>> 
>> Best,
>> 
>> Alex
> 
> -- 
> Hal Finkel
> Lead, Compiler Technology and Programming Languages
> Leadership Computing Facility
> Argonne National Laboratory
--
Sam Elliott
Software Developer - LLVM
lowRISC CIC
selliott at lowrisc.org
--

On 9/3/19 7:19 AM, Sam Elliott wrote:
> There are 1500 tests total, and about 100 on the platform-agnostic
blacklist. Alex and I do not think this is an onerous burden for maintenance,
either as an external test suite or if the test suite is imported.
>
> In the long term, if we import the tests, we know we will have to do
updates when the Embecosm work lands, and beyond that updates can be more
sporadic. It’s not clear to me how much harder these updates will be than if the
test suite remains external.
>
> We would welcome more views as to whether this suite should be imported or
should be an external test suite.

I lean toward importing - I suspect we'll get better coverage on 
buildbots, and just in general more people will end up using the tests, 
than if it is external. I'm also curious what other people think.

  -Hal

>
> It would also be useful to understand the status of the CMake vs Makefile
build system in the nightly test suite, to understand whether we need to support
Makefiles, or whether Makefiles will be disappearing from the nightly tests soon
(as they have already from the main project repo).
>
> Sam
>
>
>> On 30 Aug 2019, at 9:22PM, Finkel, Hal J. <hfinkel at anl.gov>
wrote:
>>
>>
>> On 8/30/19 2:21 PM, Alex Bradbury wrote:
>>> On Fri, 30 Aug 2019 at 17:34, Finkel, Hal J. via llvm-dev
>>> <llvm-dev at lists.llvm.org> wrote:
>>>> On 8/30/19 10:18 AM, Sam Elliott via llvm-dev wrote:
>>>>> TL;DR: I am proposing to add the GCC C Torture suite [1],
as an additional external source of tests for the “nightly” test suite. If you
are willing to review the patch, it is here: https://reviews.llvm.org/D66887
>>>>>
>>>>> Background:
>>>>>
>>>>> While working on the RISC-V backend, we have found it
useful to use additional test suites beyond the in-tree Clang and LLVM tests and
the LLVM nightly tests, in order to ensure the compiler is correct. Internally
at lowRISC, we have been running the GCC C Torture suite [1] using custom
scripts in order to ensure the backend can handle these tests as well.
>>>>>
>>>>> The main advantage is that the torture suite provide a
corpus of simple executable tests that can relatively easily be minimised to
identify the sources of bugs or regressions. During the bringup of the RISC-V
backend, the executable torture tests were the main litmus test for backend
correctness.
>>>>>
>>>>> We are not the only backend to have found this test suite
helpful, it is noted in the WebAssembly backend that they have found the GCC
Torture suite helpful during their backend bring-up as well [2].
>>>>>
>>>>> The GCC C Torture Suite is composed of four sets of tests.
The set we have found most useful is the “execute” tests, which are
single-source-file test cases that should be able to be compiled, linked, and
run without issues. The next most useful group are the “compile” tests, which
should be able to be compiled without issues. There is more documentation about
the test suites here, under “gcc.c-torture” [3].
>>>>>
>>>>> Implementation:
>>>>>
>>>>> The LLVM Nightly test suite has support for external test
suites like SPEC, where there are licencing difficulties with the tests,
>>>> Given the issues with blacklisting tests, etc. I wonder whether
it would
>>>> be more useful, and acceptable, to import the tests themselves.
Are the
>>>> tests all, to the best of your knowledge, GPLv3?
>>> Yes, the tests are, by our understanding, GPLv3. Importing could be
a
>>> reasonable option - but how do you see that interacting with the
need
>>> for blacklisting tests? Are you imagining that we just wouldn't
import
>>> the GCC-only tests into the repo?
>>
>> That was, indeed, what I had in mind, but...
>>
>>
>>>   Obviously there'd still be potential
>>> architecture-specific blacklisting, and hassle around tests that
need
>>> custom options.
>>
>> In general, it seems like keeping any blacklists and special options in
>> sync with the tests will be easiest if the configurations and the tests
>> are together in the repository.
>>
>> I'm just curious what your thoughts are on which will be better in
the
>> long term.
>>
>> Thanks again,
>>
>> Hal
>>
>>
>>> Best,
>>>
>>> Alex
>> -- 
>> Hal Finkel
>> Lead, Compiler Technology and Programming Languages
>> Leadership Computing Facility
>> Argonne National Laboratory
> --
> Sam Elliott
> Software Developer - LLVM
> lowRISC CIC
> selliott at lowrisc.org
> --
>
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

Op di 3 sep. 2019 om 18:36 schreef Finkel, Hal J. via llvm-dev <
llvm-dev at lists.llvm.org>:
> On 9/3/19 7:19 AM, Sam Elliott wrote:
> > There are 1500 tests total, and about 100 on the platform-agnostic
> blacklist. Alex and I do not think this is an onerous burden for
> maintenance, either as an external test suite or if the test suite is
> imported.
> >
> > In the long term, if we import the tests, we know we will have to do
> updates when the Embecosm work lands, and beyond that updates can be more
> sporadic. It’s not clear to me how much harder these updates will be than
> if the test suite remains external.
> >
> > We would welcome more views as to whether this suite should be
imported
> or should be an external test suite.
>
>
> I lean toward importing - I suspect we'll get better coverage on
> buildbots, and just in general more people will end up using the tests,
> than if it is external. I'm also curious what other people think.
>
>   -Hal
>
I also thought that importing the tests will result in them being run far
more regularly.
I wonder in how far regressions happen in these tests after a backend has
been brought up with it. I.e. once all these tests are made to pass, do
they later still capture regressions from time to time, or do they pretty
much always keep on passing after?
If they do catch regressions later on, the value of running them more
frequently (e.g. on buildbots) goes up quite a bit.

Maybe the only reason I could think of to not import them is if they would
take a long time to run - making buildbots slower. Is there any data on how
long it takes to run these tests?

Thanks,

Kristof
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190903/722aec9b/attachment.html>