llvm dev - Oct 2019 - [cfe-dev] RFC: End-to-end testing

> On Oct 9, 2019, at 16:12, David Greene via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Mehdi AMINI via cfe-dev <cfe-dev at lists.llvm.org> writes:
> 
>>> I have a bit of concern about this sort of thing - worrying
it'll lead to
>>> people being less cautious about writing the more isolated tests.
>>> 
>> 
>> I have the same concern. I really believe we need to be careful about
>> testing at the right granularity to keep things both modular and the
>> testing maintainable (for instance checking vectorized ASM from a C++
>> source through clang has always been considered a bad FileCheck
practice).
>> (Not saying that there is no space for better integration testing in
some
>> areas).
> 
> I absolutely disagree about vectorization tests.  We have seen
> vectorization loss in clang even though related LLVM lit tests pass,
> because something else in the clang pipeline changed that caused the
> vectorizer to not do its job.  We need both kinds of tests.  There are
> many asm tests of value beyond vectorization and they should include
> component and well as end-to-end tests.

Have you considered alternatives to checking the assembly for ensuring
vectorization or other transformations? For example, instead of checking the
assembly, we could check LLVM’s statistics or optimization remarks. If you want
to ensure a loop got vectorized, you could check the loop-vectorize remarks,
which should give you the position of the loop in the source and
vectorization/interleave factor used. There are few other things that could go
wrong later on that would prevent vector instruction selection, but I think it
should be sufficient to guard against most cases where we loose vectorization
and should be much more robust to unrelated changes. If there are additional
properties you want to ensure, they potentially could be added to the remark as
well.

This idea of leveraging statistics and optimization remarks to track the impact
of changes on overall optimization results is nothing new and I think several
people already discussed it in various forms. For regular benchmark runs, in
addition to tracking the existing benchmarks, we could also track selected
optimization remarks (e.g. loop-vectorize, but not necessarily noisy ones like
gvn) and statistics. Comparing those run-to-run could potentially highlight new
end-to-end issues on a much larger scale, across all existing benchmarks
integrated in test-suite. We might be able to detect loss in vectorization
pro-actively, instead of requiring someone to file a bug report and then we add
an isolated test after the fact.

But building something like this would be much more work of course….

Cheers,
Florian

On Thu, 10 Oct 2019 at 10:56, Florian Hahn <florian_hahn at apple.com>
wrote:
> Have you considered alternatives to checking the assembly for ensuring
vectorization or other transformations? For example, instead of checking the
assembly, we could check LLVM’s statistics or optimization remarks. If you want
to ensure a loop got vectorized, you could check the loop-vectorize remarks,
which should give you the position of the loop in the source and
vectorization/interleave factor used. There are few other things that could go
wrong later on that would prevent vector instruction selection, but I think it
should be sufficient to guard against most cases where we loose vectorization
and should be much more robust to unrelated changes. If there are additional
properties you want to ensure, they potentially could be added to the remark as
well.
We used to have lots of them, at least in the initial implementation
of the loop vectoriser (I know, many years ago).

The thread has enough points not to repeat here, but I guess the main
point is to make sure we don't duplicate tests, increasing CI cost
(especially on slower hardware).

I'd recommend trying to move any e2e tests into the test-suite and
make it easier to run, and leave specific tests only in the repo (to
guarantee independence of components).

The last thing we want is to create direct paths from front-ends to
back-ends and make LLVM IR transformation less flexible.

cheers,
--renato

Florian Hahn via cfe-dev <cfe-dev at lists.llvm.org> writes:
> Have you considered alternatives to checking the assembly for ensuring
> vectorization or other transformations? For example, instead of
> checking the assembly, we could check LLVM’s statistics or
> optimization remarks.
Yes, absolutely.  We have tests that do things like that.  I don't want
to focus on the asm bit, that's just one type of test.  The larger issue
is end-to-end tests that ensure the compiler and other tools are working
correctly, be it from checking messages, statistics, asm or something
else.
> This idea of leveraging statistics and optimization remarks to track
> the impact of changes on overall optimization results is nothing new
> and I think several people already discussed it in various forms. For
> regular benchmark runs, in addition to tracking the existing
> benchmarks, we could also track selected optimization remarks
> (e.g. loop-vectorize, but not necessarily noisy ones like gvn) and
> statistics. Comparing those run-to-run could potentially highlight new
> end-to-end issues on a much larger scale, across all existing
> benchmarks integrated in test-suite. We might be able to detect loss
> in vectorization pro-actively, instead of requiring someone to file a
> bug report and then we add an isolated test after the fact.
That's an interesting idea!  I would love to get more use out of
test-suite.

                       -David

Renato Golin via cfe-dev <cfe-dev at lists.llvm.org> writes:
> I'd recommend trying to move any e2e tests into the test-suite and
> make it easier to run, and leave specific tests only in the repo (to
> guarantee independence of components).
That would be a shame.  Where is test-suite run right now?  Are there
bots?  How are regressions reported?
> The last thing we want is to create direct paths from front-ends to
> back-ends and make LLVM IR transformation less flexible.
I'm not sure I follow.  Can you explain this a bit?

                    -David