llvm dev - Sep 2019 - RFC: Adding GCC C Torture Suite to External Test Suites

Op di 3 sep. 2019 om 18:36 schreef Finkel, Hal J. via llvm-dev <
llvm-dev at lists.llvm.org>:
> On 9/3/19 7:19 AM, Sam Elliott wrote:
> > There are 1500 tests total, and about 100 on the platform-agnostic
> blacklist. Alex and I do not think this is an onerous burden for
> maintenance, either as an external test suite or if the test suite is
> imported.
> >
> > In the long term, if we import the tests, we know we will have to do
> updates when the Embecosm work lands, and beyond that updates can be more
> sporadic. It’s not clear to me how much harder these updates will be than
> if the test suite remains external.
> >
> > We would welcome more views as to whether this suite should be
imported
> or should be an external test suite.
>
>
> I lean toward importing - I suspect we'll get better coverage on
> buildbots, and just in general more people will end up using the tests,
> than if it is external. I'm also curious what other people think.
>
>   -Hal
>
I also thought that importing the tests will result in them being run far
more regularly.
I wonder in how far regressions happen in these tests after a backend has
been brought up with it. I.e. once all these tests are made to pass, do
they later still capture regressions from time to time, or do they pretty
much always keep on passing after?
If they do catch regressions later on, the value of running them more
frequently (e.g. on buildbots) goes up quite a bit.

Maybe the only reason I could think of to not import them is if they would
take a long time to run - making buildbots slower. Is there any data on how
long it takes to run these tests?

Thanks,

Kristof
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190903/722aec9b/attachment.html>

On Tue, 3 Sep 2019 at 18:37, Kristof Beyls <kristof.beyls at gmail.com>
wrote:
> Op di 3 sep. 2019 om 18:36 schreef Finkel, Hal J. via llvm-dev <llvm-dev
at lists.llvm.org>:
>>
>> On 9/3/19 7:19 AM, Sam Elliott wrote:
>> > There are 1500 tests total, and about 100 on the platform-agnostic
blacklist. Alex and I do not think this is an onerous burden for maintenance,
either as an external test suite or if the test suite is imported.
>> >
>> > In the long term, if we import the tests, we know we will have to
do updates when the Embecosm work lands, and beyond that updates can be more
sporadic. It’s not clear to me how much harder these updates will be than if the
test suite remains external.
>> >
>> > We would welcome more views as to whether this suite should be
imported or should be an external test suite.
>>
>>
>> I lean toward importing - I suspect we'll get better coverage on
>> buildbots, and just in general more people will end up using the tests,
>> than if it is external. I'm also curious what other people think.
>>
>>   -Hal
>
>
> I also thought that importing the tests will result in them being run far
more regularly.
> I wonder in how far regressions happen in these tests after a backend has
been brought up with it. I.e. once all these tests are made to pass, do they
later still capture regressions from time to time, or do they pretty much always
keep on passing after?
> If they do catch regressions later on, the value of running them more
frequently (e.g. on buildbots) goes up quite a bit.
>
> Maybe the only reason I could think of to not import them is if they would
take a long time to run - making buildbots slower. Is there any data on how long
it takes to run these tests?
They're fast to build+run. Compiling and executing all ~1400
non-masked tests on an i9-9900k takes less than a minute (~40s):
* That's using a release build of Clang (will be slower for Debug+Asserts)
* Compiling tests with O2 targeting RISC-V
* Running tests using qemu-user

I typically run a whole bunch of ISA variant + opt level + ABI
variants. You'd obviously hope that regressions don't happen, but
it's
a useful sanity check to complement the in-tree unit tests. It's
helped me catch things when reviewing patches from others and
developing my own.

I also tend to generate .s, check them into a git repo and use git
diff to check for unexpected changes in output.

My general feeling is that if a regression or unexpected code change
(e.g. code size regression) can be found in one of the torture suite
tests then it's great news (vs seeing tthe same problem in a larger
program) - you've already got a fairly small and easy to understand
input so it's usually pretty easy to minimise and get to the root of
the problem.

Best,

Alex

Given the replies so far have been tending towards importing rather than having
the test suite be external, I have updated the patch, here:
https://reviews.llvm.org/D66887

I would welcome reviewers, noting that the patch is huge, and really the
important parts for this review are the build scripts.

I have not added Makefile support for these tests. I still don’t know the status
of the Makefile build system - whether it’s staying around permanently, or
whether it’s due to be removed like it was from the main llvm repos.

Sam
> On 3 Sep 2019, at 8:21 pm, Alex Bradbury <asb at lowrisc.org> wrote:
> 
> On Tue, 3 Sep 2019 at 18:37, Kristof Beyls <kristof.beyls at
gmail.com> wrote:
>> Op di 3 sep. 2019 om 18:36 schreef Finkel, Hal J. via llvm-dev
<llvm-dev at lists.llvm.org>:
>>> 
>>> On 9/3/19 7:19 AM, Sam Elliott wrote:
>>>> There are 1500 tests total, and about 100 on the
platform-agnostic blacklist. Alex and I do not think this is an onerous burden
for maintenance, either as an external test suite or if the test suite is
imported.
>>>> 
>>>> In the long term, if we import the tests, we know we will have
to do updates when the Embecosm work lands, and beyond that updates can be more
sporadic. It’s not clear to me how much harder these updates will be than if the
test suite remains external.
>>>> 
>>>> We would welcome more views as to whether this suite should be
imported or should be an external test suite.
>>> 
>>> 
>>> I lean toward importing - I suspect we'll get better coverage
on
>>> buildbots, and just in general more people will end up using the
tests,
>>> than if it is external. I'm also curious what other people
think.
>>> 
>>>  -Hal
>> 
>> 
>> I also thought that importing the tests will result in them being run
far more regularly.
>> I wonder in how far regressions happen in these tests after a backend
has been brought up with it. I.e. once all these tests are made to pass, do they
later still capture regressions from time to time, or do they pretty much always
keep on passing after?
>> If they do catch regressions later on, the value of running them more
frequently (e.g. on buildbots) goes up quite a bit.
>> 
>> Maybe the only reason I could think of to not import them is if they
would take a long time to run - making buildbots slower. Is there any data on
how long it takes to run these tests?
> 
> They're fast to build+run. Compiling and executing all ~1400
> non-masked tests on an i9-9900k takes less than a minute (~40s):
> * That's using a release build of Clang (will be slower for
Debug+Asserts)
> * Compiling tests with O2 targeting RISC-V
> * Running tests using qemu-user
> 
> I typically run a whole bunch of ISA variant + opt level + ABI
> variants. You'd obviously hope that regressions don't happen, but
it's
> a useful sanity check to complement the in-tree unit tests. It's
> helped me catch things when reviewing patches from others and
> developing my own.
> 
> I also tend to generate .s, check them into a git repo and use git
> diff to check for unexpected changes in output.
> 
> My general feeling is that if a regression or unexpected code change
> (e.g. code size regression) can be found in one of the torture suite
> tests then it's great news (vs seeing tthe same problem in a larger
> program) - you've already got a fairly small and easy to understand
> input so it's usually pretty easy to minimise and get to the root of
> the problem.
> 
> Best,
> 
> Alex
--
Sam Elliott
Software Developer - LLVM
lowRISC CIC
selliott at lowrisc.org
--