Mehdi AMINI via llvm-dev
2021-Jan-26 00:19 UTC
[llvm-dev] [RFC] Cross-project lit test suite
On Mon, Jan 25, 2021 at 12:16 PM David Blaikie via llvm-dev < llvm-dev at lists.llvm.org> wrote:> On Mon, Jan 25, 2021 at 3:36 AM James Henderson via llvm-dev > <llvm-dev at lists.llvm.org> wrote: > > > > Dear all, > > > > > > Recently, I and a number of my colleagues have run into cases where we > would like the ability to write tests that involve components from multiple > LLVM projects, for example using both clang and LLD. Similarly, I have seen > a few instances recently where tests would ideally make use of LLD but only > to help generate input objects for testing an LLVM tool, such as > llvm-symbolizer (see for example https://reviews.llvm.org/D88988). > Currently, there is no location where lit tests that use both clang and LLD > can be put, whilst the llvm-symbolizer cases I’ve hit are testing > llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I > therefore have prototyped a lit test suite that would be part of the > monorepo, and which can support tests that use elements from multiple > projects - see https://reviews.llvm.org/D95339. Tests could be added to > this suite as needed. The suite is modelled as an additional top-level > directory, and is enabled by enabling the “cross-project-tests” project in > CMake. I have initially added support for both LLD and clang. At > configuration time, the tests that require LLD or clang will only be > enabled when the respective projects are enabled, so that developers > continue to benefit from the subset of tests that are applicable for the > projects they are building. Note that I am not especially familiar with > CMake or lit, so this code may not be perfect, but it should be sufficient > to demonstrate what it can do. > > > > > > > > One could argue that these sorts of tests should belong in the external > (to the monorepo) test-suite, but this is a) quite distant from the > existing testing, and therefore easily forgotten, delaying potential > feedback for any breakages/resulting in potentially duplicate testing etc, > and b) is not as easy to set up and run (owing to the fact that it isn’t > part of the monorepo, isn’t connected to check-all etc), therefore making > it harder for developers to maintain the tests. Back in October 2019, there > was an extensive discussion on end-to-end testing and how to write them > (starting from > https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html). The > suggestion was that these tests would be lit-based and run as part of > check-all, and would not be inside the clang tree, although there was some > opposition. This concluded with a round table. Unfortunately, I am unaware > of what the conclusion of that round table conversation was, so it’s > possible that what I am proposing is redundant/being worked on by someone > else. Additionally, I don’t consider all classes of tests that the proposed > lit suite would be useful for to be “end-to-end” testing. For example, > llvm-symbolizer is usually used on linked output containing debug > information. Usually tests that consume objects that have debug data in > them rely on assembly that has been written by hand or generated by clang > prior to commit, with a limited set able to make use of yaml2obj to > generate the debug data instead. However, the output of these approaches is > typically not a fully linked output (yaml2obj output can be made to look > like one, but getting all the addresses to match up in a maintainable > manner makes this approach not particularly viable). Being able to use LLD > to link the object file produced would make the test significantly more > readable, much as using llvm-mc and assembly to generate test inputs is > more preferable to using prebuilt binaries. Such a test is ultimately not > really any more an end-to-end test than an llvm-symbolizer test that just > uses the object produced by the assembler directly. > > > > > > > > What do people think? > > Some concerns (the usual: Things should be tested in isolation, things > should be tested independently - but end to end tests have some value > too), but generally seems good. >Indeed this is a usual concern: such tests shouldn't be seen as replacing isolated lit tests ("unit tests"). But I have another question about the cost of maintenance here: are we gonna revert patches to either project when one of the integration tests fails? What about integration tests that require to be updated manually when changing another component? I find the cost of maintenance of end-to-end tests is often hard to carry over, especially since they are only supplementing and not replacing lit "unit-tests". Best, -- Mehdi> > Though perhaps debuginfo-tests (this presumably already supports the > multiple-subproject mechanical isssue you're discussing?) could be > generalized/renamed to be all our cross-project lit testing > (Essentially an in-monorepo, lit-only, high-reliability/not-flakey/etc > version of the test-suite). > > - Dave > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210125/0e0e45bc/attachment.html>
David Blaikie via llvm-dev
2021-Jan-26 00:28 UTC
[llvm-dev] [RFC] Cross-project lit test suite
On Mon, Jan 25, 2021 at 4:20 PM Mehdi AMINI <joker.eph at gmail.com> wrote:> > > On Mon, Jan 25, 2021 at 12:16 PM David Blaikie via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >> On Mon, Jan 25, 2021 at 3:36 AM James Henderson via llvm-dev >> <llvm-dev at lists.llvm.org> wrote: >> > >> > Dear all, >> > >> > >> > Recently, I and a number of my colleagues have run into cases where we >> would like the ability to write tests that involve components from multiple >> LLVM projects, for example using both clang and LLD. Similarly, I have seen >> a few instances recently where tests would ideally make use of LLD but only >> to help generate input objects for testing an LLVM tool, such as >> llvm-symbolizer (see for example https://reviews.llvm.org/D88988). >> Currently, there is no location where lit tests that use both clang and LLD >> can be put, whilst the llvm-symbolizer cases I’ve hit are testing >> llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I >> therefore have prototyped a lit test suite that would be part of the >> monorepo, and which can support tests that use elements from multiple >> projects - see https://reviews.llvm.org/D95339. Tests could be added to >> this suite as needed. The suite is modelled as an additional top-level >> directory, and is enabled by enabling the “cross-project-tests” project in >> CMake. I have initially added support for both LLD and clang. At >> configuration time, the tests that require LLD or clang will only be >> enabled when the respective projects are enabled, so that developers >> continue to benefit from the subset of tests that are applicable for the >> projects they are building. Note that I am not especially familiar with >> CMake or lit, so this code may not be perfect, but it should be sufficient >> to demonstrate what it can do. >> > >> > >> > >> > One could argue that these sorts of tests should belong in the external >> (to the monorepo) test-suite, but this is a) quite distant from the >> existing testing, and therefore easily forgotten, delaying potential >> feedback for any breakages/resulting in potentially duplicate testing etc, >> and b) is not as easy to set up and run (owing to the fact that it isn’t >> part of the monorepo, isn’t connected to check-all etc), therefore making >> it harder for developers to maintain the tests. Back in October 2019, there >> was an extensive discussion on end-to-end testing and how to write them >> (starting from >> https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html). The >> suggestion was that these tests would be lit-based and run as part of >> check-all, and would not be inside the clang tree, although there was some >> opposition. This concluded with a round table. Unfortunately, I am unaware >> of what the conclusion of that round table conversation was, so it’s >> possible that what I am proposing is redundant/being worked on by someone >> else. Additionally, I don’t consider all classes of tests that the proposed >> lit suite would be useful for to be “end-to-end” testing. For example, >> llvm-symbolizer is usually used on linked output containing debug >> information. Usually tests that consume objects that have debug data in >> them rely on assembly that has been written by hand or generated by clang >> prior to commit, with a limited set able to make use of yaml2obj to >> generate the debug data instead. However, the output of these approaches is >> typically not a fully linked output (yaml2obj output can be made to look >> like one, but getting all the addresses to match up in a maintainable >> manner makes this approach not particularly viable). Being able to use LLD >> to link the object file produced would make the test significantly more >> readable, much as using llvm-mc and assembly to generate test inputs is >> more preferable to using prebuilt binaries. Such a test is ultimately not >> really any more an end-to-end test than an llvm-symbolizer test that just >> uses the object produced by the assembler directly. >> > >> > >> > >> > What do people think? >> >> Some concerns (the usual: Things should be tested in isolation, things >> should be tested independently - but end to end tests have some value >> too), but generally seems good. >> > > Indeed this is a usual concern: such tests shouldn't be seen as replacing > isolated lit tests ("unit tests"). > But I have another question about the cost of maintenance here: are we > gonna revert patches to either project when one of the integration tests > fails? >Possibly, yeah. If they demonstrate a bug.> What about integration tests that require to be updated manually when > changing another component? >If they need to be updated, because their failure isn't representative of a bug, yes.> I find the cost of maintenance of end-to-end tests is often hard to carry > over, especially since they are only supplementing and not replacing lit > "unit-tests". >One of the nice thing about end to end tests (as with all tests, if designed carefully - eg: don't take some arbitrary code, compile it with optimizations, and expect a very specific backtrace - optimizations might lead to different line table/stack frame details (if some code was merged, or moved, it might lose or gain a specific source file/line)) is that they can be pretty resilient to implementation details, so less likely to need updating due to changes in implementation details. If someone changes the output format of llvm-symbolizer these would require updating and I think it'd be reasonable to expect that to be updated/not left failing. - Dave> > Best, > > -- > Mehdi > > > >> >> Though perhaps debuginfo-tests (this presumably already supports the >> multiple-subproject mechanical isssue you're discussing?) could be >> generalized/renamed to be all our cross-project lit testing >> (Essentially an in-monorepo, lit-only, high-reliability/not-flakey/etc >> version of the test-suite). >> >> - Dave >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210125/21bed3f6/attachment.html>