llvm dev - Feb 2021 - [RFC] Cross-project lit test suite

James Henderson via llvm-dev <llvm-dev at lists.llvm.org> writes:
> Currently, there is no location where lit tests that use both clang and LLD
> can be put, whilst the llvm-symbolizer cases I’ve hit are testing
> llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I
> therefore have prototyped a lit test suite that would be part of the
> monorepo, and which can support tests that use elements from multiple
> projects - see https://reviews.llvm.org/D95339. Tests could be added to
> this suite as needed. The suite is modelled as an additional top-level
> directory, and is enabled by enabling the “cross-project-tests” project in
> CMake.
This is fantastic!
> Back in October 2019, there was an extensive discussion on end-to-end
> testing and how to write them (starting from
> https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html).
> The suggestion was that these tests would be lit-based and run as part
> of check-all, and would not be inside the clang tree, although there
> was some opposition. This concluded with a round table. Unfortunately,
> I am unaware of what the conclusion of that round table conversation
> was, so it’s possible that what I am proposing is redundant/being
> worked on by someone else.
I started that thread and IIRC we ended up with the suggestion that such
tests should live in test-suite.  As you noted having tests separated
from the monorepo is less than ideal.  I haven't done anything with this
conclusion yet, mostly due to lack of time.  If your proposal gains
traction I would like to see if we could build end-to-end testing on top
of it.
> Additionally, I don’t consider all classes of tests that the proposed
> lit suite would be useful for to be “end-to-end” testing.
Agreed.  There are various classes of tests that could make use of your
proposed layout, one of which is "end-to-end."  Your proposal
doesn't
provide end-to-end testing per se, but it does make adding end-to-end
tests later on more straightforward.

             -David

> -----Original Message-----
> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of
David Greene
> via llvm-dev
> Sent: Wednesday, January 27, 2021 10:29 AM
> To: jh7370.2008 at my.bristol.ac.uk; llvm-dev <llvm-dev at
lists.llvm.org>
> Subject: Re: [llvm-dev] [RFC] Cross-project lit test suite
> 
> James Henderson via llvm-dev <llvm-dev at lists.llvm.org> writes:
> 
> > Currently, there is no location where lit tests that use both clang
and
> LLD
> > can be put, whilst the llvm-symbolizer cases I’ve hit are testing
> > llvm-symbolizer (and not LLD), so don’t really fit in the LLD test
> suite. I
> > therefore have prototyped a lit test suite that would be part of the
> > monorepo, and which can support tests that use elements from multiple
> > projects - see
> https://urldefense.com/v3/__https://reviews.llvm.org/D95339__;!!JmoZiZGBv3
> RvKRSx!vWidWrbKJid6-eIKVUT-dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqKCvkqW4A$
> . Tests could be added to
> > this suite as needed. The suite is modelled as an additional top-level
> > directory, and is enabled by enabling the “cross-project-tests”
project
> in
> > CMake.
> 
> This is fantastic!
> 
> > Back in October 2019, there was an extensive discussion on end-to-end
> > testing and how to write them (starting from
> > https://urldefense.com/v3/__https://lists.llvm.org/pipermail/cfe-
> dev/2019-October/063509.html__;!!JmoZiZGBv3RvKRSx!vWidWrbKJid6-eIKVUT-
> dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqJU8k_M_Q$ ).
> > The suggestion was that these tests would be lit-based and run as part
> > of check-all, and would not be inside the clang tree, although there
> > was some opposition. This concluded with a round table. Unfortunately,
> > I am unaware of what the conclusion of that round table conversation
> > was, so it’s possible that what I am proposing is redundant/being
> > worked on by someone else.
> 
> I started that thread and IIRC we ended up with the suggestion that such
> tests should live in test-suite.  As you noted having tests separated
> from the monorepo is less than ideal.  I haven't done anything with
this
> conclusion yet, mostly due to lack of time.  If your proposal gains
> traction I would like to see if we could build end-to-end testing on top
> of it.
> 
> > Additionally, I don’t consider all classes of tests that the proposed
> > lit suite would be useful for to be “end-to-end” testing.
> 
> Agreed.  There are various classes of tests that could make use of your
> proposed layout, one of which is "end-to-end."  Your proposal
doesn't
> provide end-to-end testing per se, but it does make adding end-to-end
> tests later on more straightforward.
> 
>              -David
I think a useful distinction here is that lit tests are generally very
focused on a specific feature/function, where test-suite has a much
broader scope.  Another slice at it would be that lit tests tend to be
"regression" tests, while test-suite is more of an
"integration" suite.

I am not a QA person so I may be abusing some of these terms, but that's
how I look at it.  Sometimes writing that focused lit test ends up
depending on multiple tools, and the cross-project lit suite would be a
good place to drop those; debuginfo-tests is a prime example.
--paulr

Given that the debuginfo tests already have cross-project dependencies, I
figured I'd try adapting them instead. I've updated
https://reviews.llvm.org/D95339 accordingly. Ideally, I think making the
existing debug-info tests a subdirectory, and renaming the top-level
directory, might be a good idea, but I haven't really come to any
conclusions about that yet.

I also found that several of the existing debuginfo-test tests fail for me.
Are these tests expected to work on Windows? If so, are there any slightly
more unusual prerequisites that I might be missing?

What do people think?

James

On Wed, 27 Jan 2021 at 15:40, <paul.robinson at sony.com> wrote:
>
>
> > -----Original Message-----
> > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of
David
> Greene
> > via llvm-dev
> > Sent: Wednesday, January 27, 2021 10:29 AM
> > To: jh7370.2008 at my.bristol.ac.uk; llvm-dev <llvm-dev at
lists.llvm.org>
> > Subject: Re: [llvm-dev] [RFC] Cross-project lit test suite
> >
> > James Henderson via llvm-dev <llvm-dev at lists.llvm.org>
writes:
> >
> > > Currently, there is no location where lit tests that use both
clang and
> > LLD
> > > can be put, whilst the llvm-symbolizer cases I’ve hit are testing
> > > llvm-symbolizer (and not LLD), so don’t really fit in the LLD
test
> > suite. I
> > > therefore have prototyped a lit test suite that would be part of
the
> > > monorepo, and which can support tests that use elements from
multiple
> > > projects - see
> >
> https://urldefense.com/v3/__https://reviews.llvm.org/D95339__;!!JmoZiZGBv3
> >
> RvKRSx!vWidWrbKJid6-eIKVUT-dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqKCvkqW4A$
> > . Tests could be added to
> > > this suite as needed. The suite is modelled as an additional
top-level
> > > directory, and is enabled by enabling the “cross-project-tests”
project
> > in
> > > CMake.
> >
> > This is fantastic!
> >
> > > Back in October 2019, there was an extensive discussion on
end-to-end
> > > testing and how to write them (starting from
> > > https://urldefense.com/v3/__https://lists.llvm.org/pipermail/cfe-
> > dev/2019-October/063509.html__;!!JmoZiZGBv3RvKRSx!vWidWrbKJid6-eIKVUT-
> > dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqJU8k_M_Q$ ).
> > > The suggestion was that these tests would be lit-based and run as
part
> > > of check-all, and would not be inside the clang tree, although
there
> > > was some opposition. This concluded with a round table.
Unfortunately,
> > > I am unaware of what the conclusion of that round table
conversation
> > > was, so it’s possible that what I am proposing is redundant/being
> > > worked on by someone else.
> >
> > I started that thread and IIRC we ended up with the suggestion that
such
> > tests should live in test-suite.  As you noted having tests separated
> > from the monorepo is less than ideal.  I haven't done anything
with this
> > conclusion yet, mostly due to lack of time.  If your proposal gains
> > traction I would like to see if we could build end-to-end testing on
top
> > of it.
> >
> > > Additionally, I don’t consider all classes of tests that the
proposed
> > > lit suite would be useful for to be “end-to-end” testing.
> >
> > Agreed.  There are various classes of tests that could make use of
your
> > proposed layout, one of which is "end-to-end."  Your
proposal doesn't
> > provide end-to-end testing per se, but it does make adding end-to-end
> > tests later on more straightforward.
> >
> >              -David
>
> I think a useful distinction here is that lit tests are generally very
> focused on a specific feature/function, where test-suite has a much
> broader scope.  Another slice at it would be that lit tests tend to be
> "regression" tests, while test-suite is more of an
"integration" suite.
>
> I am not a QA person so I may be abusing some of these terms, but
that's
> how I look at it.  Sometimes writing that focused lit test ends up
> depending on multiple tools, and the cross-project lit suite would be a
> good place to drop those; debuginfo-tests is a prime example.
> --paulr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210205/026d6d67/attachment.html>