Romero, Nichols A. via llvm-dev
2020-Dec-09 17:06 UTC
[llvm-dev] RFC Adding Fortran tests to LLVM Test Suite
Renato, See replies below. From: Renato Golin <rengolin at gmail.com> Date: Sunday, December 6, 2020 at 6:22 AM To: Nichols Romero <naromero at anl.gov> Cc: Johannes Doerfert <johannesdoerfert at gmail.com>, via llvm-dev <llvm-dev at lists.llvm.org> Subject: Re: [llvm-dev] RFC Adding Fortran tests to LLVM Test Suite On Sat, 5 Dec 2020 at 06:15, Romero, Nichols A. <naromero at anl.gov<mailto:naromero at anl.gov>> wrote: As I was planning this out in my head, I do think in the first differential we would add the CMake plumbing and a simple opens source program or two (e.g. hand-coded GEMM or something along those lines). SPEC would come in the next batch. That sounds like a good initial plan. Do you also plan to add more OSS applications, benchmarks and tests right after SPEC? Yes, I am working on identifying some OSS applications and benchmarks. The main reason for the test-suite is not benchmark, but end-to-end regression testing on multiple architectures. We can only claim to support languages and targets if we can compile entire applications, run them and get the expected results. So having a set of language tests and some real world applications on the test-suite will be required for us to claim Fortran support of any kind. Agreed. Those are also your best friends in making sure all the work you've done on the front-end doesn't regress on any target, from release to release, or during normal development. That's why I'm pushing to have those tests and applications as soon as possible. It's for Flang's own benefit. On a related note, are there any buildbots running the llvm-test-suite or are folks just running llvm-test-suite manually? Plenty, on both testing and benchmark modes. It's also part of the release process on all architectures. Are they in some public dashboard like this one? http://lab.llvm.org:8011/#/console You should add support for both testing and benchmark modes, so that the benchmarks (not SPEC) get regression tested as well as be able to report and compare performance numbers. The goal is to have an analogous set-up to what is done now for C/C++. I am just getting up to speed now on it. cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201209/3bf6c160/attachment-0001.html>
Renato Golin via llvm-dev
2020-Dec-09 17:32 UTC
[llvm-dev] RFC Adding Fortran tests to LLVM Test Suite
On Wed, 9 Dec 2020 at 17:06, Romero, Nichols A. <naromero at anl.gov> wrote:> Yes, I am working on identifying some OSS applications and benchmarks. >Awesome! Are they in some public dashboard like this one?> http://lab.llvm.org:8011/#/console >Yes, though it's not really obvious (or even easy) to find that out. :( You can look at the configuration file: https://github.com/llvm/llvm-zorg/blob/master/buildbot/osuosl/master/config/builders.py If the builder is using `ClangBuilder`, just check the ones that have the option: `runTestSuite=True` I can see Arm, AArch64, PPC, IBM-Z, X86. The ones with `testsuite_flags=['--benchmarking-only']` will be running in benchmark mode and submitting the results to the LNT server. Other builder types may be doing it, but I don't know about those. The goal is to have an analogous set-up to what is done now for C/C++. I am> just getting up to speed now on it. >I want to make sure I'm not trying to push you, just giving you the runes that aren't obvious when you first start working on it. We have a number of people (including me, many times) excited to add tests or infrastructure to the test-suite only to give up half-way because of the mess it is. Folks really did a good number on it in the past few years (CMake support, statistical analysis, benchmark mode), so it's probably a lot easier now than when I had to work on it. Good luck and thanks for taking on such a thankless task. :) cheers, --renato>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20201209/2d214af6/attachment.html>