Mehdi AMINI via llvm-dev
2019-Oct-17 01:00 UTC
[llvm-dev] [cfe-dev] [Openmp-dev] RFC: End-to-end testing
On Wed, Oct 16, 2019 at 12:54 PM David Greene via cfe-dev < cfe-dev at lists.llvm.org> wrote:> Renato Golin via Openmp-dev <openmp-dev at lists.llvm.org> writes: > > > But if we have some consensus on doing a clean job, then I would > > actually like to have that kind of intermediary check (diagnostics, > > warnings, etc) on most test-suite tests, which would cover at least > > the main vectorisation issues. Later, we could add more analysis > > tools, if we want. > > I think this makes a lot of sense. > > > It would be as simple as adding CHECK lines on the execution of the > > compilation process (in CMake? Make? wrapper?) and keep the check > > files with the tests / per file. > > Yep. > > > I think we're on the same page regarding almost everything, but > > perhaps I haven't been clear enough on the main point, which I think > > it's pretty simple. :) > > Personally, I still find source-to-asm tests to be highly valuable and I > don't think we need test-suite for that. Such tests don't (usually) > depend on system libraries (headers may occasionally be an issue but I > would argue that the test is too fragile in that case). > > So maybe we separate concerns. Use test-suite to do the kind of > system-level testing you've discussed but still allow some tests in a > monorepo top-level directory that test across components but don't > depend on system configurations. > > If people really object to a top-level monorepo test directory I guess > they could go into test-suite but that makes it much more cumbersome to > run what really should be very simple tests.The main thing I see that will justify push-back on such test is the maintenance: you need to convince everyone that every component in LLVM must also maintain (update, fix, etc.) the tests that are in other components (clang, flang, other future subproject, etc.). Changing the vectorizer in the middle-end may require now to understand the kind of update a test written in Fortran (or Haskell?) is checking with some Hexagon assembly. This is a non-trivial burden when you compute the full matrix of possible frontend and backends. Even if you write very small tests for checking vectorization, what is next? What about unrolling, inlining, loop-fusion, etc. ? Why would we stop the end-to-end FileCheck testing to vectorization? So the monorepo vs the test-suite seems like a false dichotomy: if such tests don't make it in the monorepo it will be (I believe) because folks won't want to maintain them. Putting them "elsewhere" is fine but it does not solve the question of the maintenance of the tests. -- Mehdi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191016/a5331739/attachment.html>
David Greene via llvm-dev
2019-Oct-17 15:00 UTC
[llvm-dev] [cfe-dev] [Openmp-dev] RFC: End-to-end testing
Mehdi AMINI via llvm-dev <llvm-dev at lists.llvm.org> writes:> The main thing I see that will justify push-back on such test is the > maintenance: you need to convince everyone that every component in LLVM > must also maintain (update, fix, etc.) the tests that are in other > components (clang, flang, other future subproject, etc.). Changing the > vectorizer in the middle-end may require now to understand the kind of > update a test written in Fortran (or Haskell?) is checking with some > Hexagon assembly. This is a non-trivial burden when you compute the full > matrix of possible frontend and backends.That's true. But don't we want to make sure the complete compiler works as expected? And don't we want to be alerted as soon as possible if something breaks? To my knowledge we have very few end-to-end tests of the type I've been thinking about. That worries me.> Even if you write very small tests for checking vectorization, what is > next? What about unrolling, inlining, loop-fusion, etc. ? Why would we stop > the end-to-end FileCheck testing to vectorization?I actually think vectorization is probably lower on the concern list for end-to-end testing than more focused things like FMA generation, prefetching and so on. This is because there isn't a lot after the vectorization pass that can be mess up vectorization. Once something is vectorized, it is likely to stay vectorized. On the other hand, I have for example frequently seen prefetches dropped or poorly scheduled by code long after the prefetch got inserted into the IR.> So the monorepo vs the test-suite seems like a false dichotomy: if such > tests don't make it in the monorepo it will be (I believe) because folks > won't want to maintain them. Putting them "elsewhere" is fine but it does > not solve the question of the maintenance of the tests.Agree 100%. -David
Finkel, Hal J. via llvm-dev
2019-Oct-17 15:38 UTC
[llvm-dev] [cfe-dev] [Openmp-dev] RFC: End-to-end testing
On 10/17/19 10:00 AM, David Greene via cfe-dev wrote:> Mehdi AMINI via llvm-dev <llvm-dev at lists.llvm.org> writes: > >> The main thing I see that will justify push-back on such test is the >> maintenance: you need to convince everyone that every component in LLVM >> must also maintain (update, fix, etc.) the tests that are in other >> components (clang, flang, other future subproject, etc.). Changing the >> vectorizer in the middle-end may require now to understand the kind of >> update a test written in Fortran (or Haskell?) is checking with some >> Hexagon assembly. This is a non-trivial burden when you compute the full >> matrix of possible frontend and backends.That's true, but at some point we really do just need to work together to make changes. If some necessary group of people become unresponsive, then we'll need to deal with that, but just not knowing whether the compiler works as intended seems worse.> That's true. But don't we want to make sure the complete compiler works > as expected? And don't we want to be alerted as soon as possible if > something breaks? To my knowledge we have very few end-to-end tests of > the type I've been thinking about. That worries me.I agree. We really should have more end-to-end testing for cases where we have end-to-end contracts. If we provide a pragma to ask for vectorization, or loop unrolling, or whatever, then we should test "end to end" for whatever that means from the beginning of the contract (i.e., the place where the request is asserted) to the end (i.e., the place where to can confirm that the user will observe the intended behavior) - this might mean checking assembly or it might mean checking end-stage IR, etc. There are other cases where, even if there's no pragma, we know what the optimal output is and we can test for it. We've had plenty of cases where changes to the pass pipeline, instcombine, etc. have caused otherwise reasonably-well-covered components to stop behaving as expected in the context of the complete pipeline. Vectorization is a good example of this, but is not the only such example. As I recall, other loop optimizations (unrolling, idiom recognition, etc.) have also had these problems over time> >> Even if you write very small tests for checking vectorization, what is >> next? What about unrolling, inlining, loop-fusion, etc. ? Why would we stop >> the end-to-end FileCheck testing to vectorization? > I actually think vectorization is probably lower on the concern list for > end-to-end testing than more focused things like FMA generation, > prefetching and so on.In my experience, these are about equal. Vectorization being later means that fewer things can mess things up afterwards (although there still is all of codegen), but more things can mess things up beforehand. -Hal> This is because there isn't a lot after the > vectorization pass that can be mess up vectorization. Once something is > vectorized, it is likely to stay vectorized. On the other hand, I have > for example frequently seen prefetches dropped or poorly scheduled by > code long after the prefetch got inserted into the IR. > >> So the monorepo vs the test-suite seems like a false dichotomy: if such >> tests don't make it in the monorepo it will be (I believe) because folks >> won't want to maintain them. Putting them "elsewhere" is fine but it does >> not solve the question of the maintenance of the tests. > Agree 100%. > > -David > _______________________________________________ > cfe-dev mailing list > cfe-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev-- Hal Finkel Lead, Compiler Technology and Programming Languages Leadership Computing Facility Argonne National Laboratory