llvm dev - Oct 2019 - [Openmp-dev] [cfe-dev] RFC: End-to-end testing

Renato Golin <rengolin at gmail.com> writes:
> On Wed, 16 Oct 2019 at 21:00, David Greene <greened at obbligato.org>
wrote:
>> Can you elaborate?  I'm talking about very small tests targeted to
>> generate a specific instruction or small number of instructions.
>> Vectorization isn't the best example.  Something like verifying FMA
>> generation is a better example.
>
> To check that instructions are generated from source, a two-step test
> is the best approach:
>  - Verify that Clang emits different IR for different options, or the
> right IR for a new functionality
>  - Verify that the affected targets (or at least two of the main ones)
> can take that IR and generate the right asm
Yes, of course we have tests like that.  We have found they are not
always sufficient.
> If you want to do the test in Clang all the way to asm, you need to
> make sure the back-end is built. Clang is not always build with all
> back-ends, possibly even none.
Right, which is why we have things like REQUIRES: x86-registered-target.
> To do that in the back-end, you'd have to rely on Clang being built,
> which is not always true.
Sure.
> Hacking our test infrastructure to test different things when a
> combination of components is built, especially after they start to
> merge after being in a monorepo, will complicate tests and increase
> the likelihood that some tests will never be run by CI and bit rot.
>From other discussion, it sounds like at least some people are open to
asm tests under clang.  I think that should be fine.  But there are
probably other kinds of end-to-end tests that should not live under
clang.
> On the test-suite, you can guarantee that the whole toolchain is
> available: Front and back end of the compilers, assemblers (if
> necessary), linkers, libraries, etc.
>
> Writing a small source file per test, as you would in Clang/LLVM,
> running LIT and FileCheck, and *always* running it in the TS would be
> trivial.
How often would such tests be run as part of test-suite?

Honestly, it's not really clear to me exactly which bots cover what, how
often they run and so on.  Is there a document somewhere describing the
setup?

                     -David

On Thu, 17 Oct 2019 at 18:10, David Greene <greened at obbligato.org>
wrote:
> From other discussion, it sounds like at least some people are open to
> asm tests under clang.  I think that should be fine.  But there are
> probably other kinds of end-to-end tests that should not live under
> clang.
That is my position as well. Some tests, especially similar to
existing ones, are fine.

But if we really want to do do complete tests and stress more than
just grepping a couple of instructions, should be in a better suited
place.
> How often would such tests be run as part of test-suite?
Every time the TS is executed. Some good work has been put on it to
run with CMake etc, so it should be trivial to to run that before
commits, but it *does* require more than just "make check-all".

On CI, a number of bots run those as often as they can, non-stop.
> Honestly, it's not really clear to me exactly which bots cover what,
how
> often they run and so on.  Is there a document somewhere describing the
> setup?
Not really. The main Buildbot page is a mess and the system is very
old. There is a round table at the dev meeting to discuss the path
forward.

This is not the first, though. We have been discussing this for a
number of years, but getting people / companies to commit to testing
is not trivial.

I created a page for the Arm bots (after many incarnations, it ended
up here: http://ex40-01.tcwglab.linaro.org/) to make that simpler. But
that wouldn't scale, nor it fixes the real problems.

Renato Golin <rengolin at gmail.com> writes:
> On Thu, 17 Oct 2019 at 18:10, David Greene <greened at obbligato.org>
wrote:
>> From other discussion, it sounds like at least some people are open to
>> asm tests under clang.  I think that should be fine.  But there are
>> probably other kinds of end-to-end tests that should not live under
>> clang.
>
> That is my position as well. Some tests, especially similar to
> existing ones, are fine.
Ok.
> But if we really want to do do complete tests and stress more than
> just grepping a couple of instructions, should be in a better suited
> place.
That's probably true.
>> How often would such tests be run as part of test-suite?
>
> Every time the TS is executed. Some good work has been put on it to
> run with CMake etc, so it should be trivial to to run that before
> commits, but it *does* require more than just "make check-all".
I have been viewing test-suite as a kind of second-level/backup testing
that catches things not flagged by "make check-all."  Is that a
reasonable interpretation?  I was hoping to get some end-to-end tests
under "make check-all" because that's easier for developers to run
in
their workflows.
> On CI, a number of bots run those as often as they can, non-stop.
>
>> Honestly, it's not really clear to me exactly which bots cover
what, how
>> often they run and so on.  Is there a document somewhere describing the
>> setup?
>
> Not really. The main Buildbot page is a mess and the system is very
> old. There is a round table at the dev meeting to discuss the path
> forward.
Yeah, I saw that.  I will see if I can attend.  There are some conflicts
we have to work out here.
> This is not the first, though. We have been discussing this for a
> number of years, but getting people / companies to commit to testing
> is not trivial.
Is there a proposal somewhere of what companies would be expected to do?
It's difficult for us engineers to talk to management without a concrete
set of expectations, resource requirements, etc.
> I created a page for the Arm bots (after many incarnations, it ended
> up here: http://ex40-01.tcwglab.linaro.org/) to make that simpler. But
> that wouldn't scale, nor it fixes the real problems.
Nice!  That's much better.  Yes, it won't scale but it's much
clearer
about what is being run.

                         -David