llvm dev - Oct 2019 - [cfe-dev] [Openmp-dev] RFC: End-to-end testing

On Fri, 11 Oct 2019 at 18:02, David Greene via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> >> We have to support many different systems and those systems are
always
> >> changing (new processors, new BIOS, new OS, etc.).  Performance
can vary
> >> widely day to day from factors completely outside the
compiler's
> >> control.  As the performance changes you have to keep updating the
tests
> >> to expect the new performance numbers.  Relying on performance
> >> measurements to ensure something like vectorization is happening
just
> >> isn't reliable in our experience.
> >
> > Could you compare performance with vectorization turned on and off?
>
> That might catch more things but now you're running tests twice and it
> still won't catch some cases.
Precisely.

In my experience, benchmarks numbers need to reset on most (if not
all) system changes, that's why we keep our benchmark machines *very*
stable (ie. outdated). Testing multiple configurations need multiple
baselines, combinatorial explosion and all that.

For clarity, I didn't mean "make e2e tests *only* run tests and check
for performance", that would be a *very* poor substitute for the tests
you proposed.

The idea to have extra checks in the test-suite has circulated many
years ago when a similar proposal was put forward, but IIRC, the
piece-wise LIT tests we already have were deemed good enough for the
cases we wanted to cover.

But in the test-suite, we have more than just the compiler. We have
libraries (run-time, language), tools (linkers, assemblers) and the
environment. Those can affect the quality of the code (as you mention
earlier).

We need to test that, but we can't do a good job in the LIT side (how
do you control libraries and other tools? it can get crazy ugly). The
sanitizer tests are a good example on how weird it gets executing the
code, grepping for output and relying on runtime system libraries to
"get right".

So, in a way, we could just stop the test-suite discussion and do like
the sanitizers. If people are ok with this, don't let me stop you. :)

But if we have some consensus on doing a clean job, then I would
actually like to have that kind of intermediary check (diagnostics,
warnings, etc) on most test-suite tests, which would cover at least
the main vectorisation issues. Later, we could add more analysis
tools, if we want.

It would be as simple as adding CHECK lines on the execution of the
compilation process (in CMake? Make? wrapper?) and keep the check
files with the tests / per file.

I think we're on the same page regarding almost everything, but
perhaps I haven't been clear enough on the main point, which I think
it's pretty simple. :)

--renato

Renato Golin via Openmp-dev <openmp-dev at lists.llvm.org> writes:
> But if we have some consensus on doing a clean job, then I would
> actually like to have that kind of intermediary check (diagnostics,
> warnings, etc) on most test-suite tests, which would cover at least
> the main vectorisation issues. Later, we could add more analysis
> tools, if we want.
I think this makes a lot of sense.
> It would be as simple as adding CHECK lines on the execution of the
> compilation process (in CMake? Make? wrapper?) and keep the check
> files with the tests / per file.
Yep.
> I think we're on the same page regarding almost everything, but
> perhaps I haven't been clear enough on the main point, which I think
> it's pretty simple. :)
Personally, I still find source-to-asm tests to be highly valuable and I
don't think we need test-suite for that.  Such tests don't (usually)
depend on system libraries (headers may occasionally be an issue but I
would argue that the test is too fragile in that case).

So maybe we separate concerns.  Use test-suite to do the kind of
system-level testing you've discussed but still allow some tests in a
monorepo top-level directory that test across components but don't
depend on system configurations.

If people really object to a top-level monorepo test directory I guess
they could go into test-suite but that makes it much more cumbersome to
run what really should be very simple tests.

                   -David

On Wed, Oct 16, 2019 at 12:54 PM David Greene via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> Renato Golin via Openmp-dev <openmp-dev at lists.llvm.org> writes:
>
> > But if we have some consensus on doing a clean job, then I would
> > actually like to have that kind of intermediary check (diagnostics,
> > warnings, etc) on most test-suite tests, which would cover at least
> > the main vectorisation issues. Later, we could add more analysis
> > tools, if we want.
>
> I think this makes a lot of sense.
>
> > It would be as simple as adding CHECK lines on the execution of the
> > compilation process (in CMake? Make? wrapper?) and keep the check
> > files with the tests / per file.
>
> Yep.
>
> > I think we're on the same page regarding almost everything, but
> > perhaps I haven't been clear enough on the main point, which I
think
> > it's pretty simple. :)
>
> Personally, I still find source-to-asm tests to be highly valuable and I
> don't think we need test-suite for that.  Such tests don't
(usually)
> depend on system libraries (headers may occasionally be an issue but I
> would argue that the test is too fragile in that case).
>
> So maybe we separate concerns.  Use test-suite to do the kind of
> system-level testing you've discussed but still allow some tests in a
> monorepo top-level directory that test across components but don't
> depend on system configurations.
>
I'm inclined to the direction suggested by others that the monorepo is
orthogonal to this issue and top level tests might not be the right thing.

lldb already does end-to-end testing in its tests, for instance.

Clang does in some tests (the place I always hit is anything that's
configured API-wise on the MCContext - there's no way to test that
configuration on the clang boundary, so the only test that we can write is
one that tests the effect of that API/programmatic configuration done by
clang to the MCContext (function sections, for instance) - in some cases
I've just skipped the testing, in others I've written the end-to-end
test
in clang (& an LLVM test for the functionality that uses llvm-mc or
similar)).

> If people really object to a top-level monorepo test directory I guess
> they could go into test-suite but that makes it much more cumbersome to
> run what really should be very simple tests.
>
>                    -David
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191016/bf3bded8/attachment.html>

On Wed, Oct 16, 2019 at 12:54 PM David Greene via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> Renato Golin via Openmp-dev <openmp-dev at lists.llvm.org> writes:
>
> > But if we have some consensus on doing a clean job, then I would
> > actually like to have that kind of intermediary check (diagnostics,
> > warnings, etc) on most test-suite tests, which would cover at least
> > the main vectorisation issues. Later, we could add more analysis
> > tools, if we want.
>
> I think this makes a lot of sense.
>
> > It would be as simple as adding CHECK lines on the execution of the
> > compilation process (in CMake? Make? wrapper?) and keep the check
> > files with the tests / per file.
>
> Yep.
>
> > I think we're on the same page regarding almost everything, but
> > perhaps I haven't been clear enough on the main point, which I
think
> > it's pretty simple. :)
>
> Personally, I still find source-to-asm tests to be highly valuable and I
> don't think we need test-suite for that.  Such tests don't
(usually)
> depend on system libraries (headers may occasionally be an issue but I
> would argue that the test is too fragile in that case).
>
> So maybe we separate concerns.  Use test-suite to do the kind of
> system-level testing you've discussed but still allow some tests in a
> monorepo top-level directory that test across components but don't
> depend on system configurations.
>
> If people really object to a top-level monorepo test directory I guess
> they could go into test-suite but that makes it much more cumbersome to
> run what really should be very simple tests.

The main thing I see that will justify push-back on such test is the
maintenance: you need to convince everyone that every component in LLVM
must also maintain (update, fix, etc.) the tests that are in other
components (clang, flang, other future subproject, etc.). Changing the
vectorizer in the middle-end may require now to understand the kind of
update a test written in Fortran (or Haskell?) is checking with some
Hexagon assembly. This is a non-trivial burden when you compute the full
matrix of possible frontend and backends.
Even if you write very small tests for checking vectorization, what is
next? What about unrolling, inlining, loop-fusion, etc. ? Why would we stop
the end-to-end FileCheck testing to vectorization?

So the monorepo vs the test-suite seems like a false dichotomy: if such
tests don't make it in the monorepo it will be (I believe) because folks
won't want to maintain them. Putting them "elsewhere" is fine but
it does
not solve the question of the maintenance of the tests.

-- 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191016/a5331739/attachment.html>