[llvm-dev] [RFC] Cross-project lit test suite

James Henderson via llvm-dev
Tue, 26 Jan 2021 12:50:13 -0000

On Tue, 26 Jan 2021 at 00:28, David Blaikie <dblaikie at gmail.com> wrote:

>
>
> On Mon, Jan 25, 2021 at 4:20 PM Mehdi AMINI <joker.eph at gmail.com>
wrote:
>
>>
>>
>> On Mon, Jan 25, 2021 at 12:16 PM David Blaikie via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> On Mon, Jan 25, 2021 at 3:36 AM James Henderson via llvm-dev
>>> <llvm-dev at lists.llvm.org> wrote:
>>> >
>>> > Dear all,
>>> >
>>> >
>>> > Recently, I and a number of my colleagues have run into cases
where we
>>> would like the ability to write tests that involve components from
multiple
>>> LLVM projects, for example using both clang and LLD. Similarly, I
have seen
>>> a few instances recently where tests would ideally make use of LLD
but only
>>> to help generate input objects for testing an LLVM tool, such as
>>> llvm-symbolizer (see for example https://reviews.llvm.org/D88988).
>>> Currently, there is no location where lit tests that use both clang
and LLD
>>> can be put, whilst the llvm-symbolizer cases I’ve hit are testing
>>> llvm-symbolizer (and not LLD), so don’t really fit in the LLD test
suite. I
>>> therefore have prototyped a lit test suite that would be part of
the
>>> monorepo, and which can support tests that use elements from
multiple
>>> projects - see https://reviews.llvm.org/D95339. Tests could be
added to
>>> this suite as needed. The suite is modelled as an additional
top-level
>>> directory, and is enabled by enabling the “cross-project-tests”
project in
>>> CMake. I have initially added support for both LLD and clang. At
>>> configuration time, the tests that require LLD or clang will only
be
>>> enabled when the respective projects are enabled, so that
developers
>>> continue to benefit from the subset of tests that are applicable
for the
>>> projects they are building. Note that I am not especially familiar
with
>>> CMake or lit, so this code may not be perfect, but it should be
sufficient
>>> to demonstrate what it can do.
>>> >
>>> >
>>> >
>>> > One could argue that these sorts of tests should belong in the
>>> external (to the monorepo) test-suite, but this is a) quite distant
from
>>> the existing testing, and therefore easily forgotten, delaying
potential
>>> feedback for any breakages/resulting in potentially duplicate
testing etc,
>>> and b) is not as easy to set up and run (owing to the fact that it
isn’t
>>> part of the monorepo, isn’t connected to check-all etc), therefore
making
>>> it harder for developers to maintain the tests. Back in October
2019, there
>>> was an extensive discussion on end-to-end testing and how to write
them
>>> (starting from
>>> https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html).
The
>>> suggestion was that these tests would be lit-based and run as part
of
>>> check-all, and would not be inside the clang tree, although there
was some
>>> opposition. This concluded with a round table. Unfortunately, I am
unaware
>>> of what the conclusion of that round table conversation was, so
it’s
>>> possible that what I am proposing is redundant/being worked on by
someone
>>> else. Additionally, I don’t consider all classes of tests that the
proposed
>>> lit suite would be useful for to be “end-to-end” testing. For
example,
>>> llvm-symbolizer is usually used on linked output containing debug
>>> information. Usually tests that consume objects that have debug
data in
>>> them rely on assembly that has been written by hand or generated by
clang
>>> prior to commit, with a limited set able to make use of yaml2obj to
>>> generate the debug data instead. However, the output of these
approaches is
>>> typically not a fully linked output (yaml2obj output can be made to
look
>>> like one, but getting all the addresses to match up in a
maintainable
>>> manner makes this approach not particularly viable). Being able to
use LLD
>>> to link the object file produced would make the test significantly
more
>>> readable, much as using llvm-mc and assembly to generate test
inputs is
>>> more preferable to using prebuilt binaries. Such a test is
ultimately not
>>> really any more an end-to-end test than an llvm-symbolizer test
that just
>>> uses the object produced by the assembler directly.
>>> >
>>> >
>>> >
>>> > What do people think?
>>>
>>> Some concerns (the usual: Things should be tested in isolation,
things
>>> should be tested independently - but end to end tests have some
value
>>> too), but generally seems good.
>>>
>>
>> Indeed this is a usual concern: such tests shouldn't be seen as
replacing
>> isolated lit tests ("unit tests").
>>
>
I completely agree. Indeed, the llvm-symbolizer test referenced in the
review I'd class as an isolated test - LLD was just being used to generate
the input for the test. The key here is that the thing being tested was the
llvm-symbolizer behaviour, and not the linker behaviour. As mentioned, this
isn't really different to how llvm-mc or llc might be used to convert some
input source (asm/IR etc) into something the tool under test wants to be
run on. Potentially, changes in those tools might break things, but as long
as the input is specific enough, this shouldn't happen often.

But I have another question about the cost of maintenance here: are
we
>> gonna revert patches to either project when one of the integration
tests
>> fails?
>>
>
> Possibly, yeah. If they demonstrate a bug.
>

That would be my intention - these tests should be classed as first-class
citizens as much as any other lit test. They're just unit tests that use
other components (LLD, clang etc) to generate inputs.


>
>> What about integration tests that require to be updated manually when
>> changing another component?
>>
>
> If they need to be updated, because their failure isn't representative
of
> a bug, yes.
>

Hopefully these sorts of failures would occur only infrequently. As noted,
my aim here isn't to provide a place in opensource LLVM to do integration
testing, but rather unit tests that just can't sit in the corresponding lit
area, so input variability should be minimal.

That being said, the proposed suite could be used for integration testing,
if the community agreed such testing belonged in the monorepo - indeed, I
have plans for some downstream integration tests that would make use of
this if it lands - but that isn't the goal here.

>
>
>> I find the cost of maintenance of end-to-end tests is often hard to
carry
>> over, especially since they are only supplementing and not replacing
lit
>> "unit-tests".
>>
>
> One of the nice thing about end to end tests (as with all tests, if
> designed carefully - eg: don't take some arbitrary code, compile it
with
> optimizations, and expect a very specific backtrace - optimizations might
> lead to different line table/stack frame details (if some code was merged,
> or moved, it might lose or gain a specific source file/line)) is that they
> can be pretty resilient to implementation details, so less likely to need
> updating due to changes in implementation details. If someone changes the
> output format of llvm-symbolizer these would require updating and I think
> it'd be reasonable to expect that to be updated/not left failing.
>

Right, just as we would change other tests for tools where the output
format changed.


> - Dave
>
>
>>
>> Best,
>>
>> --
>> Mehdi
>>
>>
>>
>>>
>>> Though perhaps debuginfo-tests (this presumably already supports
the
>>> multiple-subproject mechanical isssue you're discussing?) could
be
>>> generalized/renamed to be all our cross-project lit testing
>>> (Essentially an in-monorepo, lit-only,
high-reliability/not-flakey/etc
>>> version of the test-suite).
>>>
>>>
The existing debug-info test area looks like it could work if it was more
generalised. It looks like we'd want the ability to have tests that would
still work without clang/lldb being built, and similarly which could use
LLD, but I don't think those are insurmountable issues - it would just
require taking some of the ideas from my existing prototype (or equivalent
from elsewhere) and merging them in.


> - Dave
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210126/5d5d95d0/attachment-0001.html>