llvm dev - Dec 2020 - RFC Adding Fortran tests to LLVM Test Suite

Renato,

Sorry for not replying right away. After the Thanksgiving break, I was in
meeting for most of this week and only now catching up on e-mail.

Thanks for raising your concerns and point taken.

I am new to the llvm-test-suite and spent most of the day looking through it.

As I was planning this out in my head, I do think in the first differential we
would add the CMake plumbing and a simple opens source program or two (e.g.
hand-coded GEMM or something along those lines). SPEC would come in the next
batch.

On a related note, are there any buildbots running the llvm-test-suite or are
folks just running llvm-test-suite manually?

From: Renato Golin <rengolin at gmail.com>
Date: Thursday, November 26, 2020 at 4:38 AM
To: Johannes Doerfert <johannesdoerfert at gmail.com>
Cc: via llvm-dev <llvm-dev at lists.llvm.org>, Nichols Romero <naromero
at anl.gov>
Subject: Re: [llvm-dev] RFC Adding Fortran tests to LLVM Test Suite

I don't disagree with your roadmap. If I'm reading correctly, SPEC is
only the first benchmark, not the first program.

My point was to add the language tests, and perhaps one small program as a
benchmark, to test the infrastructure. SPEC could come in the same batch, to
show that the CMake glue works for all parts.

I wouldn't add CMake glue with SPEC only, as a first step. That's all
I'm saying.

Cheers,
Renato

On Wed, 25 Nov 2020, 23:46 Johannes Doerfert, <johannesdoerfert at
gmail.com<mailto:johannesdoerfert at gmail.com>> wrote:

On 11/25/20 2:33 PM, Renato Golin wrote:
> On Wed, 25 Nov 2020 at 18:10, Johannes Doerfert <johannesdoerfert at
gmail.com<mailto:johannesdoerfert at gmail.com>>
> wrote:
>
>>> If you only add infrastructure to build Fortran programs inside
SPEC,
>> then
>>> your change would be biased towards an external benchmark that is
private
>>> to some companies.
>> That doesn't make any sense to me.
>> Nobody suggested to change anything "inside SPEC".
>>
> Good part of your reply assumes I meant what you say above. I didn't.
>
> We're talking past each other. Let me try again.
>
> As I said on my original reply, I'm very supportive of the initiative
to
> add Fortran to the test suite. To add tests, benchmarks and openmp. This is
> very good news.
>
> But the test-suite doesn't have a core ownership, a group that has a
plan
> and implements all the parts of a bigger design goal. For many years we
> have tried to unify tests and benchmarks, Kristof did a great job rallying
> people around and so many other people contributed, but once it
"works",
> people stop paying attention.
>
> I just want to make sure that the overall support for Fortran in the
> test-suite is focused on building tests, benchmarks and other tools that
> are available upstream to all users.
>
> If adding Fortran support on the existing SPEC scripts is orthogonal, then
> it shouldn't be part of this discussion. If it's not, then it
shouldn't be
> the main driver for the rest of the infrastructure.
>
>> Public build-bots will start building those tests and benchmarks
>> (remember,
>>> it's not just benchmarks in there), and you'll need some
time to adjust
>>> strategy until it all works across the board.
>> Strategy: If you don't set it up to run Fortran codes, it
won't.
>>
> I'm going to take this as a tongue-in-cheek comment. The reducionism
here
> isn't really helpful.
>
> Fortran is just the language, but there are architectures and operating
> systems that need adjusting, too.
>
> Fortran benchmark support in the LLVM Test Suite, and literally
>> everything else mentioned in the initial RFC, is beneficial to the
>> community. SPEC support is not something harmful.
>>
> We definitely agree on that.
>
> How did you come to that conclusion after the initial RFC explicitly listed
>> other benchmarks and apps we want to include in to the test suite?
>>
> The original RFC was very clear. Your response was less so.
>
> On my reply to the RFC, I said I worry that we're focusing on SPEC too
> early. I'd rather make sure it works upstream before adding SPEC to the
> mix.
>
> The reason I tried to convey (and clearly failed) is that the test-suite
> isn't a robust and well designed infrastructure, but a patch-work from
> different approaches over the years, which seems to "work fine"
with what
> we have.
>
> I may have read that wrong, but it sounded to me as if you were defending
> the prioritisation of SPEC "and some micro benchmarks" over the
rest of the
> proposal.
>
> I think that's a mistake, because it risks being the main thing that
gets
> added and then not much else comes later (priorities change, etc).
>
> If my interpretation is wrong, I apologise and we can ignore our past
> exchange. I'm still very supportive of this RFC. :)
The way I understand you emails is that you argue against the roadmap
because
it lists SPEC as a first proper benchmark/app. This is actually on purpose:

SPEC is a well tested external benchmark suite with existing support in the
LLVM test suite and it allows for stable results with existing compilers. We
know what compiler works with SPEC, we know the expected outputs, we
know how
to select different input sizes, we know how to glue it to the Test
suite, etc.

The alternative is to bring in new benchmarks/apps which have multiple other
challenges, as you noted before. In order for us to test the support of the
Fortran plumbing with non-trivial programs, SPEC seems like an ideal
candidate.

I say this because Nick was working on compiling existing benchmarks and
apps
with Flang (sema only) and that often entails dealing with complex
undocumented
and unmaintained build systems. That is on top of potential issues wrt.
nuerical
stability, non-standard compliant code, ...

Don't get me wrong, adding other benchmarks is already part of the road
map we are
committed to. We recently added the C/C++ proxy apps, and we are working on
Parallelism/OpenMP (+offloading) support. This is not a one-off effort.

Please also note that we asked in the mail for benchmark/app ideas so we
know
what to look at next. We are certainly committed to work on this well
past SPEC
support. I know that is true for the ANL people, DOE people, and I'm
very certain
also for the wider Flang community.

~ Johannes


P.S. We're heading into a long Thanksgiving weekend, unclear how
reactive I'll be
      the next two days. I hope you'll also have a nice and relaxing
weekend :)

>
> cheers,
> --renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201205/4655d9ac/attachment.html>

On Sat, 5 Dec 2020 at 06:15, Romero, Nichols A. <naromero at anl.gov>
wrote:
> As I was planning this out in my head, I do think in the first
> differential we would add the CMake plumbing and a simple opens source
> program or two (e.g. hand-coded GEMM or something along those lines). SPEC
> would come in the next batch.
>
That sounds like a good initial plan. Do you also plan to add more OSS
applications, benchmarks and tests right after SPEC?

The main reason for the test-suite is not benchmark, but end-to-end
regression testing on multiple architectures. We can only claim to support
languages and targets if we can compile entire applications, run them and
get the expected results.

So having a set of language tests and some real world applications on the
test-suite will be required for us to claim Fortran support of any kind.

Those are also your best friends in making sure all the work you've done on
the front-end doesn't regress on any target, from release to release, or
during normal development.

That's why I'm pushing to have those tests and applications as soon as
possible. It's for Flang's own benefit.

On a related note, are there any buildbots running the llvm-test-suite
or
> are folks just running llvm-test-suite manually?
>
Plenty, on both testing and benchmark modes. It's also part of the release
process on all architectures.

You should add support for both testing and benchmark modes, so that the
benchmarks (not SPEC) get regression tested as well as be able to report
and compare performance numbers.

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201206/396b262e/attachment.html>