James Henderson via llvm-dev
2021-Feb-05 15:37 UTC
[llvm-dev] [RFC] Cross-project lit test suite
Given that the debuginfo tests already have cross-project dependencies, I figured I'd try adapting them instead. I've updated https://reviews.llvm.org/D95339 accordingly. Ideally, I think making the existing debug-info tests a subdirectory, and renaming the top-level directory, might be a good idea, but I haven't really come to any conclusions about that yet. I also found that several of the existing debuginfo-test tests fail for me. Are these tests expected to work on Windows? If so, are there any slightly more unusual prerequisites that I might be missing? What do people think? James On Wed, 27 Jan 2021 at 15:40, <paul.robinson at sony.com> wrote:> > > > -----Original Message----- > > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of David > Greene > > via llvm-dev > > Sent: Wednesday, January 27, 2021 10:29 AM > > To: jh7370.2008 at my.bristol.ac.uk; llvm-dev <llvm-dev at lists.llvm.org> > > Subject: Re: [llvm-dev] [RFC] Cross-project lit test suite > > > > James Henderson via llvm-dev <llvm-dev at lists.llvm.org> writes: > > > > > Currently, there is no location where lit tests that use both clang and > > LLD > > > can be put, whilst the llvm-symbolizer cases I’ve hit are testing > > > llvm-symbolizer (and not LLD), so don’t really fit in the LLD test > > suite. I > > > therefore have prototyped a lit test suite that would be part of the > > > monorepo, and which can support tests that use elements from multiple > > > projects - see > > > https://urldefense.com/v3/__https://reviews.llvm.org/D95339__;!!JmoZiZGBv3 > > > RvKRSx!vWidWrbKJid6-eIKVUT-dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqKCvkqW4A$ > > . Tests could be added to > > > this suite as needed. The suite is modelled as an additional top-level > > > directory, and is enabled by enabling the “cross-project-tests” project > > in > > > CMake. > > > > This is fantastic! > > > > > Back in October 2019, there was an extensive discussion on end-to-end > > > testing and how to write them (starting from > > > https://urldefense.com/v3/__https://lists.llvm.org/pipermail/cfe- > > dev/2019-October/063509.html__;!!JmoZiZGBv3RvKRSx!vWidWrbKJid6-eIKVUT- > > dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqJU8k_M_Q$ ). > > > The suggestion was that these tests would be lit-based and run as part > > > of check-all, and would not be inside the clang tree, although there > > > was some opposition. This concluded with a round table. Unfortunately, > > > I am unaware of what the conclusion of that round table conversation > > > was, so it’s possible that what I am proposing is redundant/being > > > worked on by someone else. > > > > I started that thread and IIRC we ended up with the suggestion that such > > tests should live in test-suite. As you noted having tests separated > > from the monorepo is less than ideal. I haven't done anything with this > > conclusion yet, mostly due to lack of time. If your proposal gains > > traction I would like to see if we could build end-to-end testing on top > > of it. > > > > > Additionally, I don’t consider all classes of tests that the proposed > > > lit suite would be useful for to be “end-to-end” testing. > > > > Agreed. There are various classes of tests that could make use of your > > proposed layout, one of which is "end-to-end." Your proposal doesn't > > provide end-to-end testing per se, but it does make adding end-to-end > > tests later on more straightforward. > > > > -David > > I think a useful distinction here is that lit tests are generally very > focused on a specific feature/function, where test-suite has a much > broader scope. Another slice at it would be that lit tests tend to be > "regression" tests, while test-suite is more of an "integration" suite. > > I am not a QA person so I may be abusing some of these terms, but that's > how I look at it. Sometimes writing that focused lit test ends up > depending on multiple tools, and the cross-project lit suite would be a > good place to drop those; debuginfo-tests is a prime example. > --paulr > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210205/026d6d67/attachment.html>
David Blaikie via llvm-dev
2021-Feb-05 18:50 UTC
[llvm-dev] [RFC] Cross-project lit test suite
Reid and Amy might have some context for Windows (though I don't know if any Windows folks have done much with this test suite). On Fri, Feb 5, 2021 at 7:38 AM James Henderson via llvm-dev <llvm-dev at lists.llvm.org> wrote:> > Given that the debuginfo tests already have cross-project dependencies, I figured I'd try adapting them instead. I've updated https://reviews.llvm.org/D95339 accordingly. Ideally, I think making the existing debug-info tests a subdirectory, and renaming the top-level directory, might be a good idea, but I haven't really come to any conclusions about that yet. > > I also found that several of the existing debuginfo-test tests fail for me. Are these tests expected to work on Windows? If so, are there any slightly more unusual prerequisites that I might be missing? > > What do people think? > > James > > On Wed, 27 Jan 2021 at 15:40, <paul.robinson at sony.com> wrote: >> >> >> >> > -----Original Message----- >> > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of David Greene >> > via llvm-dev >> > Sent: Wednesday, January 27, 2021 10:29 AM >> > To: jh7370.2008 at my.bristol.ac.uk; llvm-dev <llvm-dev at lists.llvm.org> >> > Subject: Re: [llvm-dev] [RFC] Cross-project lit test suite >> > >> > James Henderson via llvm-dev <llvm-dev at lists.llvm.org> writes: >> > >> > > Currently, there is no location where lit tests that use both clang and >> > LLD >> > > can be put, whilst the llvm-symbolizer cases I’ve hit are testing >> > > llvm-symbolizer (and not LLD), so don’t really fit in the LLD test >> > suite. I >> > > therefore have prototyped a lit test suite that would be part of the >> > > monorepo, and which can support tests that use elements from multiple >> > > projects - see >> > https://urldefense.com/v3/__https://reviews.llvm.org/D95339__;!!JmoZiZGBv3 >> > RvKRSx!vWidWrbKJid6-eIKVUT-dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqKCvkqW4A$ >> > . Tests could be added to >> > > this suite as needed. The suite is modelled as an additional top-level >> > > directory, and is enabled by enabling the “cross-project-tests” project >> > in >> > > CMake. >> > >> > This is fantastic! >> > >> > > Back in October 2019, there was an extensive discussion on end-to-end >> > > testing and how to write them (starting from >> > > https://urldefense.com/v3/__https://lists.llvm.org/pipermail/cfe- >> > dev/2019-October/063509.html__;!!JmoZiZGBv3RvKRSx!vWidWrbKJid6-eIKVUT- >> > dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqJU8k_M_Q$ ). >> > > The suggestion was that these tests would be lit-based and run as part >> > > of check-all, and would not be inside the clang tree, although there >> > > was some opposition. This concluded with a round table. Unfortunately, >> > > I am unaware of what the conclusion of that round table conversation >> > > was, so it’s possible that what I am proposing is redundant/being >> > > worked on by someone else. >> > >> > I started that thread and IIRC we ended up with the suggestion that such >> > tests should live in test-suite. As you noted having tests separated >> > from the monorepo is less than ideal. I haven't done anything with this >> > conclusion yet, mostly due to lack of time. If your proposal gains >> > traction I would like to see if we could build end-to-end testing on top >> > of it. >> > >> > > Additionally, I don’t consider all classes of tests that the proposed >> > > lit suite would be useful for to be “end-to-end” testing. >> > >> > Agreed. There are various classes of tests that could make use of your >> > proposed layout, one of which is "end-to-end." Your proposal doesn't >> > provide end-to-end testing per se, but it does make adding end-to-end >> > tests later on more straightforward. >> > >> > -David >> >> I think a useful distinction here is that lit tests are generally very >> focused on a specific feature/function, where test-suite has a much >> broader scope. Another slice at it would be that lit tests tend to be >> "regression" tests, while test-suite is more of an "integration" suite. >> >> I am not a QA person so I may be abusing some of these terms, but that's >> how I look at it. Sometimes writing that focused lit test ends up >> depending on multiple tools, and the cross-project lit suite would be a >> good place to drop those; debuginfo-tests is a prime example. >> --paulr >> > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev