[llvm-dev] [RFC] Cross-project lit test suite

Mehdi AMINI via llvm-dev
2021-01-26T00:19:38Z

On Mon, Jan 25, 2021 at 12:16 PM David Blaikie via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> On Mon, Jan 25, 2021 at 3:36 AM James Henderson via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >
> > Dear all,
> >
> >
> > Recently, I and a number of my colleagues have run into cases where we
> would like the ability to write tests that involve components from multiple
> LLVM projects, for example using both clang and LLD. Similarly, I have seen
> a few instances recently where tests would ideally make use of LLD but only
> to help generate input objects for testing an LLVM tool, such as
> llvm-symbolizer (see for example https://reviews.llvm.org/D88988).
> Currently, there is no location where lit tests that use both clang and LLD
> can be put, whilst the llvm-symbolizer cases I’ve hit are testing
> llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I
> therefore have prototyped a lit test suite that would be part of the
> monorepo, and which can support tests that use elements from multiple
> projects - see https://reviews.llvm.org/D95339. Tests could be added to
> this suite as needed. The suite is modelled as an additional top-level
> directory, and is enabled by enabling the “cross-project-tests” project in
> CMake. I have initially added support for both LLD and clang. At
> configuration time, the tests that require LLD or clang will only be
> enabled when the respective projects are enabled, so that developers
> continue to benefit from the subset of tests that are applicable for the
> projects they are building. Note that I am not especially familiar with
> CMake or lit, so this code may not be perfect, but it should be sufficient
> to demonstrate what it can do.
> >
> >
> >
> > One could argue that these sorts of tests should belong in the
external
> (to the monorepo) test-suite, but this is a) quite distant from the
> existing testing, and therefore easily forgotten, delaying potential
> feedback for any breakages/resulting in potentially duplicate testing etc,
> and b) is not as easy to set up and run (owing to the fact that it isn’t
> part of the monorepo, isn’t connected to check-all etc), therefore making
> it harder for developers to maintain the tests. Back in October 2019, there
> was an extensive discussion on end-to-end testing and how to write them
> (starting from
> https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html). The
> suggestion was that these tests would be lit-based and run as part of
> check-all, and would not be inside the clang tree, although there was some
> opposition. This concluded with a round table. Unfortunately, I am unaware
> of what the conclusion of that round table conversation was, so it’s
> possible that what I am proposing is redundant/being worked on by someone
> else. Additionally, I don’t consider all classes of tests that the proposed
> lit suite would be useful for to be “end-to-end” testing. For example,
> llvm-symbolizer is usually used on linked output containing debug
> information. Usually tests that consume objects that have debug data in
> them rely on assembly that has been written by hand or generated by clang
> prior to commit, with a limited set able to make use of yaml2obj to
> generate the debug data instead. However, the output of these approaches is
> typically not a fully linked output (yaml2obj output can be made to look
> like one, but getting all the addresses to match up in a maintainable
> manner makes this approach not particularly viable). Being able to use LLD
> to link the object file produced would make the test significantly more
> readable, much as using llvm-mc and assembly to generate test inputs is
> more preferable to using prebuilt binaries. Such a test is ultimately not
> really any more an end-to-end test than an llvm-symbolizer test that just
> uses the object produced by the assembler directly.
> >
> >
> >
> > What do people think?
>
> Some concerns (the usual: Things should be tested in isolation, things
> should be tested independently - but end to end tests have some value
> too), but generally seems good.
>

Indeed this is a usual concern: such tests shouldn't be seen as replacing
isolated lit tests ("unit tests").
But I have another question about the cost of maintenance here: are we
gonna revert patches to either project when one of the integration tests
fails?
What about integration tests that require to be updated manually when
changing another component?
I find the cost of maintenance of end-to-end tests is often hard to carry
over, especially since they are only supplementing and not replacing lit
"unit-tests".

Best,

--
Mehdi



>
> Though perhaps debuginfo-tests (this presumably already supports the
> multiple-subproject mechanical isssue you're discussing?) could be
> generalized/renamed to be all our cross-project lit testing
> (Essentially an in-monorepo, lit-only, high-reliability/not-flakey/etc
> version of the test-suite).
>
> - Dave
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210125/0e0e45bc/attachment.html>