James Henderson via llvm-dev
2021-Jan-25 11:36 UTC
[llvm-dev] [RFC] Cross-project lit test suite
Dear all, Recently, I and a number of my colleagues have run into cases where we would like the ability to write tests that involve components from multiple LLVM projects, for example using both clang and LLD. Similarly, I have seen a few instances recently where tests would ideally make use of LLD but only to help generate input objects for testing an LLVM tool, such as llvm-symbolizer (see for example https://reviews.llvm.org/D88988). Currently, there is no location where lit tests that use both clang and LLD can be put, whilst the llvm-symbolizer cases I’ve hit are testing llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I therefore have prototyped a lit test suite that would be part of the monorepo, and which can support tests that use elements from multiple projects - see https://reviews.llvm.org/D95339. Tests could be added to this suite as needed. The suite is modelled as an additional top-level directory, and is enabled by enabling the “cross-project-tests” project in CMake. I have initially added support for both LLD and clang. At configuration time, the tests that require LLD or clang will only be enabled when the respective projects are enabled, so that developers continue to benefit from the subset of tests that are applicable for the projects they are building. Note that I am not especially familiar with CMake or lit, so this code may not be perfect, but it should be sufficient to demonstrate what it can do. One could argue that these sorts of tests should belong in the external (to the monorepo) test-suite, but this is a) quite distant from the existing testing, and therefore easily forgotten, delaying potential feedback for any breakages/resulting in potentially duplicate testing etc, and b) is not as easy to set up and run (owing to the fact that it isn’t part of the monorepo, isn’t connected to check-all etc), therefore making it harder for developers to maintain the tests. Back in October 2019, there was an extensive discussion on end-to-end testing and how to write them (starting from https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html). The suggestion was that these tests would be lit-based and run as part of check-all, and would not be inside the clang tree, although there was some opposition. This concluded with a round table. Unfortunately, I am unaware of what the conclusion of that round table conversation was, so it’s possible that what I am proposing is redundant/being worked on by someone else. Additionally, I don’t consider all classes of tests that the proposed lit suite would be useful for to be “end-to-end” testing. For example, llvm-symbolizer is usually used on linked output containing debug information. Usually tests that consume objects that have debug data in them rely on assembly that has been written by hand or generated by clang prior to commit, with a limited set able to make use of yaml2obj to generate the debug data instead. However, the output of these approaches is typically not a fully linked output (yaml2obj output can be made to look like one, but getting all the addresses to match up in a maintainable manner makes this approach not particularly viable). Being able to use LLD to link the object file produced would make the test significantly more readable, much as using llvm-mc and assembly to generate test inputs is more preferable to using prebuilt binaries. Such a test is ultimately not really any more an end-to-end test than an llvm-symbolizer test that just uses the object produced by the assembler directly. What do people think? James -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210125/dbd7ccfd/attachment-0001.html>
Tom Stellard via llvm-dev
2021-Jan-25 15:27 UTC
[llvm-dev] [RFC] Cross-project lit test suite
On 1/25/21 3:36 AM, James Henderson via llvm-dev wrote:> Dear all, > > > Recently, I and a number of my colleagues have run into cases where we would like the ability to write tests that involve components from multiple LLVM projects, for example using both clang and LLD. Similarly, I have seen a few instances recently where tests would ideally make use of LLD but only to help generate input objects for testing an LLVM tool, such as llvm-symbolizer (see for example https://reviews.llvm.org/D88988). Currently, there is no location where lit tests that use both clang and LLD can be put, whilst the llvm-symbolizer cases I’ve hit are testing llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I therefore have prototyped a lit test suite that would be part of the monorepo, and which can support tests that use elements from multiple projects - see https://reviews.llvm.org/D95339. Tests could be added to this suite as needed. The suite is modelled as an additional top-level directory, and is enabled by enabling the > “cross-project-tests” project in CMake. I have initially added support for both LLD and clang. At configuration time, the tests that require LLD or clang will only be enabled when the respective projects are enabled, so that developers continue to benefit from the subset of tests that are applicable for the projects they are building. Note that I am not especially familiar with CMake or lit, so this code may not be perfect, but it should be sufficient to demonstrate what it can do. > > One could argue that these sorts of tests should belong in the external (to the monorepo) test-suite, but this is a) quite distant from the existing testing, and therefore easily forgotten, delaying potential feedback for any breakages/resulting in potentially duplicate testing etc, and b) is not as easy to set up and run (owing to the fact that it isn’t part of the monorepo, isn’t connected to check-all etc), therefore making it harder for developers to maintain the tests. Back in October 2019, there was an extensive discussion on end-to-end testing and how to write them (starting from https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html). The suggestion was that these tests would be lit-based and run as part of check-all, and would not be inside the clang tree, although there was some opposition. This concluded with a round table. Unfortunately, I am unaware of what the conclusion of that round table conversation was, so it’s possible that what I am proposing > is redundant/being worked on by someone else. Additionally, I don’t consider all classes of tests that the proposed lit suite would be useful for to be “end-to-end” testing. For example, llvm-symbolizer is usually used on linked output containing debug information. Usually tests that consume objects that have debug data in them rely on assembly that has been written by hand or generated by clang prior to commit, with a limited set able to make use of yaml2obj to generate the debug data instead. However, the output of these approaches is typically not a fully linked output (yaml2obj output can be made to look like one, but getting all the addresses to match up in a maintainable manner makes this approach not particularly viable). Being able to use LLD to link the object file produced would make the test significantly more readable, much as using llvm-mc and assembly to generate test inputs is more preferable to using prebuilt binaries. Such a test is ultimately not really any > more an end-to-end test than an llvm-symbolizer test that just uses the object produced by the assembler directly. > > What do people think? >Is this similar to what you are looking for? https://github.com/opencollab/llvm-toolchain-integration-test-suite/ -Tom> James > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >
David Blaikie via llvm-dev
2021-Jan-25 20:16 UTC
[llvm-dev] [RFC] Cross-project lit test suite
On Mon, Jan 25, 2021 at 3:36 AM James Henderson via llvm-dev <llvm-dev at lists.llvm.org> wrote:> > Dear all, > > > Recently, I and a number of my colleagues have run into cases where we would like the ability to write tests that involve components from multiple LLVM projects, for example using both clang and LLD. Similarly, I have seen a few instances recently where tests would ideally make use of LLD but only to help generate input objects for testing an LLVM tool, such as llvm-symbolizer (see for example https://reviews.llvm.org/D88988). Currently, there is no location where lit tests that use both clang and LLD can be put, whilst the llvm-symbolizer cases I’ve hit are testing llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I therefore have prototyped a lit test suite that would be part of the monorepo, and which can support tests that use elements from multiple projects - see https://reviews.llvm.org/D95339. Tests could be added to this suite as needed. The suite is modelled as an additional top-level directory, and is enabled by enabling the “cross-project-tests” project in CMake. I have initially added support for both LLD and clang. At configuration time, the tests that require LLD or clang will only be enabled when the respective projects are enabled, so that developers continue to benefit from the subset of tests that are applicable for the projects they are building. Note that I am not especially familiar with CMake or lit, so this code may not be perfect, but it should be sufficient to demonstrate what it can do. > > > > One could argue that these sorts of tests should belong in the external (to the monorepo) test-suite, but this is a) quite distant from the existing testing, and therefore easily forgotten, delaying potential feedback for any breakages/resulting in potentially duplicate testing etc, and b) is not as easy to set up and run (owing to the fact that it isn’t part of the monorepo, isn’t connected to check-all etc), therefore making it harder for developers to maintain the tests. Back in October 2019, there was an extensive discussion on end-to-end testing and how to write them (starting from https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html). The suggestion was that these tests would be lit-based and run as part of check-all, and would not be inside the clang tree, although there was some opposition. This concluded with a round table. Unfortunately, I am unaware of what the conclusion of that round table conversation was, so it’s possible that what I am proposing is redundant/being worked on by someone else. Additionally, I don’t consider all classes of tests that the proposed lit suite would be useful for to be “end-to-end” testing. For example, llvm-symbolizer is usually used on linked output containing debug information. Usually tests that consume objects that have debug data in them rely on assembly that has been written by hand or generated by clang prior to commit, with a limited set able to make use of yaml2obj to generate the debug data instead. However, the output of these approaches is typically not a fully linked output (yaml2obj output can be made to look like one, but getting all the addresses to match up in a maintainable manner makes this approach not particularly viable). Being able to use LLD to link the object file produced would make the test significantly more readable, much as using llvm-mc and assembly to generate test inputs is more preferable to using prebuilt binaries. Such a test is ultimately not really any more an end-to-end test than an llvm-symbolizer test that just uses the object produced by the assembler directly. > > > > What do people think?Some concerns (the usual: Things should be tested in isolation, things should be tested independently - but end to end tests have some value too), but generally seems good. Though perhaps debuginfo-tests (this presumably already supports the multiple-subproject mechanical isssue you're discussing?) could be generalized/renamed to be all our cross-project lit testing (Essentially an in-monorepo, lit-only, high-reliability/not-flakey/etc version of the test-suite). - Dave
David Greene via llvm-dev
2021-Jan-27 15:28 UTC
[llvm-dev] [RFC] Cross-project lit test suite
James Henderson via llvm-dev <llvm-dev at lists.llvm.org> writes:> Currently, there is no location where lit tests that use both clang and LLD > can be put, whilst the llvm-symbolizer cases I’ve hit are testing > llvm-symbolizer (and not LLD), so don’t really fit in the LLD test suite. I > therefore have prototyped a lit test suite that would be part of the > monorepo, and which can support tests that use elements from multiple > projects - see https://reviews.llvm.org/D95339. Tests could be added to > this suite as needed. The suite is modelled as an additional top-level > directory, and is enabled by enabling the “cross-project-tests” project in > CMake.This is fantastic!> Back in October 2019, there was an extensive discussion on end-to-end > testing and how to write them (starting from > https://lists.llvm.org/pipermail/cfe-dev/2019-October/063509.html). > The suggestion was that these tests would be lit-based and run as part > of check-all, and would not be inside the clang tree, although there > was some opposition. This concluded with a round table. Unfortunately, > I am unaware of what the conclusion of that round table conversation > was, so it’s possible that what I am proposing is redundant/being > worked on by someone else.I started that thread and IIRC we ended up with the suggestion that such tests should live in test-suite. As you noted having tests separated from the monorepo is less than ideal. I haven't done anything with this conclusion yet, mostly due to lack of time. If your proposal gains traction I would like to see if we could build end-to-end testing on top of it.> Additionally, I don’t consider all classes of tests that the proposed > lit suite would be useful for to be “end-to-end” testing.Agreed. There are various classes of tests that could make use of your proposed layout, one of which is "end-to-end." Your proposal doesn't provide end-to-end testing per se, but it does make adding end-to-end tests later on more straightforward. -David