llvm dev - Oct 2019 - [Openmp-dev] [cfe-dev] RFC: End-to-end testing

On Wed, Oct 16, 2019 at 12:54 PM David Greene via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> Renato Golin via Openmp-dev <openmp-dev at lists.llvm.org> writes:
>
> > But if we have some consensus on doing a clean job, then I would
> > actually like to have that kind of intermediary check (diagnostics,
> > warnings, etc) on most test-suite tests, which would cover at least
> > the main vectorisation issues. Later, we could add more analysis
> > tools, if we want.
>
> I think this makes a lot of sense.
>
> > It would be as simple as adding CHECK lines on the execution of the
> > compilation process (in CMake? Make? wrapper?) and keep the check
> > files with the tests / per file.
>
> Yep.
>
> > I think we're on the same page regarding almost everything, but
> > perhaps I haven't been clear enough on the main point, which I
think
> > it's pretty simple. :)
>
> Personally, I still find source-to-asm tests to be highly valuable and I
> don't think we need test-suite for that.  Such tests don't
(usually)
> depend on system libraries (headers may occasionally be an issue but I
> would argue that the test is too fragile in that case).
>
> So maybe we separate concerns.  Use test-suite to do the kind of
> system-level testing you've discussed but still allow some tests in a
> monorepo top-level directory that test across components but don't
> depend on system configurations.
>
I'm inclined to the direction suggested by others that the monorepo is
orthogonal to this issue and top level tests might not be the right thing.

lldb already does end-to-end testing in its tests, for instance.

Clang does in some tests (the place I always hit is anything that's
configured API-wise on the MCContext - there's no way to test that
configuration on the clang boundary, so the only test that we can write is
one that tests the effect of that API/programmatic configuration done by
clang to the MCContext (function sections, for instance) - in some cases
I've just skipped the testing, in others I've written the end-to-end
test
in clang (& an LLVM test for the functionality that uses llvm-mc or
similar)).

> If people really object to a top-level monorepo test directory I guess
> they could go into test-suite but that makes it much more cumbersome to
> run what really should be very simple tests.
>
>                    -David
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191016/bf3bded8/attachment.html>

> I'm inclined to the direction suggested by others that the monorepo is
> orthogonal to this issue and top level tests might not be the right thing.
>
> lldb already does end-to-end testing in its tests, for instance.
>
> Clang does in some tests (the place I always hit is anything that's
> configured API-wise on the MCContext - there's no way to test that
> configuration on the clang boundary, so the only test that we can write is
> one that tests the effect of that API/programmatic configuration done by
> clang to the MCContext (function sections, for instance) - in some cases
> I've just skipped the testing, in others I've written the
end-to-end test
> in clang (& an LLVM test for the functionality that uses llvm-mc or
> similar)).
I'd be totally happy putting such tests under clang.  This whole
discussion was spurred by D68230 where some noted that previous
discussion had determined we didn't want source-to-asm tests in clang
and the test update script explicitly forbade it.

If we're saying we want to reverse that decision, I'm very glad!

                    -David

On Wed, Oct 16, 2019 at 6:05 PM David Greene <dag at cray.com> wrote:
> > I'm inclined to the direction suggested by others that the
monorepo is
> > orthogonal to this issue and top level tests might not be the right
> thing.
> >
> > lldb already does end-to-end testing in its tests, for instance.
> >
> > Clang does in some tests (the place I always hit is anything
that's
> > configured API-wise on the MCContext - there's no way to test that
> > configuration on the clang boundary, so the only test that we can
write
> is
> > one that tests the effect of that API/programmatic configuration done
by
> > clang to the MCContext (function sections, for instance) - in some
cases
> > I've just skipped the testing, in others I've written the
end-to-end test
> > in clang (& an LLVM test for the functionality that uses llvm-mc
or
> > similar)).
>
> I'd be totally happy putting such tests under clang.  This whole
> discussion was spurred by D68230 where some noted that previous
> discussion had determined we didn't want source-to-asm tests in clang
> and the test update script explicitly forbade it.
>
> If we're saying we want to reverse that decision, I'm very glad!
>
Unfortunately LLVM's community is by no means a monolith, so my opinion
here doesn't mean whoever expressed their opinion there has changed their
mind.

& I generally agree that end-to-end testing should be very limited - but
there are already some end-to-end-ish tests in clang and I don't think
they're entirely wrong there. I don't know much about the vectorization
tests - but any test that requires a tool to maintain/generate makes me a
bit skeptical and doubly-so if we were testing all of those end-to-end too.
(I'd expect maybe one or two sample/example end-to-end tests, to test
certain integration points, but exhaustive testing would usually be left to
narrower tests (so if you have one subsystem with three codepaths {1, 2, 3}
and another subsystem with 3 codepaths {A, B, C}, you don't test the full
combination of {1, 2, 3} X {A, B, C} (9 tests), you test each set
separately, and maybe one representative sample end-to-end (so you end up
with maybe 7-8 tests))

Possible I know so little about the vectorization issues in particular that
my thoughts on testing don't line up with the realities of that particular
domain.

- Dave

>
>                     -David
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191016/5fe6e142/attachment.html>