llvm dev - Mar 2019 - RFC for f18+runtimes in LLVM

"Finkel, Hal J. via llvm-dev" <llvm-dev at lists.llvm.org>
writes:
>     And then there is also the argument for reusing Clang tooling,
>     which David Greene keeps making, though that idea does not seem to
>     get a lot of interest.
>     
>
> I disagree. There's been a lot of interest in modeling Flang's
tooling
> after Clang's infrastructure, and refactoring for direct reuse where
> possible. In general, refactoring for code reuse is our default -
> developing similar functionality without reusing existing, related
> code is what, in general, requires specific justification.
My experience with this question has been the same as Petr's.  I keep
asking about it and keep getting radio silence from f18 developers.
Some other people (including Petr) have jumped into the discussion but
frankly, I have not seen a lot of interest in f18 tooling from the f18
developers and I'm very concerned about that.

f18 development so far has been opaque.  There's a biweekly call (which
I'm sure many people such as me simply can't participate in) and meeting
minutes from the calls, but there is really no convenient design
discussion or any other feedback mechanism into the project.  Several
people have tried to discuss reusing bits of clang, but again no
response from the actual developers.

There are Differentials posted for Fortran debug handling in LLVM and
the last update on them was Nov. 28.  As far as I can tell the status on
them is:

https://reviews.llvm.org/D54327 - Looks ready to land, awaiting approval
  from someone.  Who is the someone?

https://reviews.llvm.org/D54043 - Paul Robinson did some groundwork
  to aid the effort.  No updates since.

https://reviews.llvm.org/D54114 - No updates since Nov. 6.  Some
  questions awaiting response.

I know from personal experience how difficult it is to get big changes
approved, so I don't fault the f18 team for the lengthy process to get
these in.  However, D54043 is particularly concerning because we haven't
heard anything on it for a couple of months since a community member
went through some effort to get foundational support in so f18 could
make progress.

This *feels* like a giant code dump to me.  I am sure that's not the
intent of the f18 developers but Steve S. posted his note a few days ago
and there have been plenty of responses...except by the f18 developers.
Given my experience on flang-dev I'm unfortunately not surprised by
this.  There needs to be a lot more community engagement around f18.

I was really excited about the f18 project when it was announced.  I
helped set up a roundtable at LLVMDev for it.  We had some good
discussion.  But that discussion has stopped.  I'm somewhat less excited
about the project now and that's a shame.

                           -David

Hi David,

* Using Clang ASTs for Fortran --

We're very happy with the f18 AST data structure.  It provides type safety
to the level that with few exceptions will statically check that only legal ASTs
can be constructed. (The exceptions are when the parser doesn't know how to
resolve a construct exactly, e.g. statement expressions. In those cases, the
static checking allows only the ambiguous choices.) The ability to create tree
walkers that can be statically checked for completeness is a boon to correctness
and safety not only when the code is first written but more importantly when it
is extended.

The advantage of using clang's ASTs for Fortran would be a common mechanism
for dealing with ASTs.  It doesn't buy an implementation of say, clang-tidy,
for Fortran.  Either Fortran and C ASTs structs would overlay, perhaps creating
some confusion because the semantics of Fortran and C/C++ are different, or else
the Fortran ASTs would extent the set of ASTs, again creating some confusion for
developer's who wouldn't necessarily have a clear idea which structs
belonged to which language and what, if any, subtle differences in semantics are
implied by the origin language.

At the developer's conference we hoped that the mechanisms around f18 ASTs
would be roughly similar to clang's mechanism so that a tool writer would
have a common understand for how to get started with an infrastructure for
building a specific tool.   We have not done anything specific to support tool
development other than to structure our code and libraries in a way that we
think will be amenable to tool development.

That said, we've engaged with a handful of Fortran tool authors to discuss
how f18 can help them with their work.  The feedback has been positive -- keep
doing what you're doing -- but with one exception I don't think any of
these developers have engaged with the code yet.

I look forward to hearing more from Petr & his progressing using clang's
ASTs for Fortran.

* Queued Fortran debugging changes for llvm

Yes, we have three reviews in the queue for changes that extend llvm debug
metadata to support Fortran constructs in DWARF.   The feedback requested that
one of the changes be implemented differently, which we have yet to budget time
to reimplement.  The other feedback was for us to think harder about mapping
Fortran to existing C constructs; our experience says this approach won't
work because, for example, Fortran has much richer array representation than C
(negative bounds, dynamic bounds, bounds that are associated with variables
& not encapsulated are part of the array structure). We also want to stay
compatible with the existing Fortran aware debuggers that expect a certain shape
of DWARF.  This issue is on use to explain ourselves more clearly.

* Big code dump

Sorry it feels that way to you.  Most of our design communication happens via
github; if you are watching the project you'll see when new documents and
new code is posted.  As far as I know, anyone can participate in a code review;
if that is not the case please let me know & I'll do what I can to open
it up.

Most of what we are working on has been in flight for a long time.  Two of the
longest running tasks -- type declarations and symbols resolution and expression
analysis, semantics, and folding -- have been documented for a long time &
have seen many commits as the implementation progresses.  In the last month, the
control flow graph documentation and header files have been open for comments in
a pull request.  We did get questions and comments (via email) from a few
people.

* Dev calls

David, sorry you can't make the flang calls.  It's a split between a
read-out of progress with various issue, technical and non-technical, with time
for others to bring up issues.  ARM runs a developer-centric meeting too; we
participate.  Neither focuses heavily on f18.

Perhaps is my mistake to judge interest in f18 by traffic on github and the
flang-dev mailing list.  I'm happy to host one time or recurring call to
discuss f18.  One of the action items from last July's introductory call was
to schedule another call to discuss how arrays would be represented in the CFG
and in llvm IR; there was lively discussion of various options during the July
call.

On 3/1/19, 11:46 AM, "llvm-dev on behalf of David Greene via
llvm-dev" <llvm-dev-bounces at lists.llvm.org on behalf of llvm-dev at
lists.llvm.org> wrote:

    "Finkel, Hal J. via llvm-dev" <llvm-dev at lists.llvm.org>
writes:
    
    >     And then there is also the argument for reusing Clang tooling,
    >     which David Greene keeps making, though that idea does not seem to
    >     get a lot of interest.
    >     
    >
    > I disagree. There's been a lot of interest in modeling Flang's
tooling
    > after Clang's infrastructure, and refactoring for direct reuse
where
    > possible. In general, refactoring for code reuse is our default -
    > developing similar functionality without reusing existing, related
    > code is what, in general, requires specific justification.
    
    My experience with this question has been the same as Petr's.  I keep
    asking about it and keep getting radio silence from f18 developers.
    Some other people (including Petr) have jumped into the discussion but
    frankly, I have not seen a lot of interest in f18 tooling from the f18
    developers and I'm very concerned about that.
    
    f18 development so far has been opaque.  There's a biweekly call (which
    I'm sure many people such as me simply can't participate in) and
meeting
    minutes from the calls, but there is really no convenient design
    discussion or any other feedback mechanism into the project.  Several
    people have tried to discuss reusing bits of clang, but again no
    response from the actual developers.
    
    There are Differentials posted for Fortran debug handling in LLVM and
    the last update on them was Nov. 28.  As far as I can tell the status on
    them is:
    
    https://reviews.llvm.org/D54327 - Looks ready to land, awaiting approval
      from someone.  Who is the someone?
    
    https://reviews.llvm.org/D54043 - Paul Robinson did some groundwork
      to aid the effort.  No updates since.
    
    https://reviews.llvm.org/D54114 - No updates since Nov. 6.  Some
      questions awaiting response.
    
    I know from personal experience how difficult it is to get big changes
    approved, so I don't fault the f18 team for the lengthy process to get
    these in.  However, D54043 is particularly concerning because we haven't
    heard anything on it for a couple of months since a community member
    went through some effort to get foundational support in so f18 could
    make progress.
    
    This *feels* like a giant code dump to me.  I am sure that's not the
    intent of the f18 developers but Steve S. posted his note a few days ago
    and there have been plenty of responses...except by the f18 developers.
    Given my experience on flang-dev I'm unfortunately not surprised by
    this.  There needs to be a lot more community engagement around f18.
    
    I was really excited about the f18 project when it was announced.  I
    helped set up a roundtable at LLVMDev for it.  We had some good
    discussion.  But that discussion has stopped.  I'm somewhat less excited
    about the project now and that's a shame.
    
                               -David
    _______________________________________________
    LLVM Developers mailing list
    llvm-dev at lists.llvm.org
    https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
    


-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may
contain
confidential information.  Any unauthorized review, use, disclosure or
distribution
is prohibited.  If you are not the intended recipient, please contact the sender
by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Steve, thanks for your thoughtful reply!

Stephen Scalpone via llvm-dev <llvm-dev at lists.llvm.org> writes:
> The advantage of using clang's ASTs for Fortran would be a common
> mechanism for dealing with ASTs.  It doesn't buy an implementation of
> say, clang-tidy, for Fortran.  Either Fortran and C ASTs structs would
> overlay, perhaps creating some confusion because the semantics of
> Fortran and C/C++ are different, or else the Fortran ASTs would extent
> the set of ASTs, again creating some confusion for developer's who
> wouldn't necessarily have a clear idea which structs belonged to which
> language and what, if any, subtle differences in semantics are implied
> by the origin language.
I'm certainly out of my domain here so I'll let Petr and others with
experience drive this discussion.  I understand that simply extending
clang's AST won't magically get tooling working, but it seems to me it
would be a good foundation.

I was imagining that semantic differences would either result in
different AST node types for C/C++ and Fortran where necessary, or
annotations/attributes on existing AST node types.  Those may or may not
result in LLVM IR annotations/intrinsics/etc. (thinking in particular of
the array representation issue mentioned below).
> That said, we've engaged with a handful of Fortran tool authors to
> discuss how f18 can help them with their work.  The feedback has been
> positive -- keep doing what you're doing -- but with one exception I
> don't think any of these developers have engaged with the code yet.
It's good to hear tool developers are engaged!  Are any of them involed
with clang tooling as well?  If so it would be great to get their
insight.
> Yes, we have three reviews in the queue for changes that extend llvm
> debug metadata to support Fortran constructs in DWARF.  The feedback
> requested that one of the changes be implemented differently, which we
> have yet to budget time to reimplement.  The other feedback was for us
> to think harder about mapping Fortran to existing C constructs; our
> experience says this approach won't work because, for example, Fortran
> has much richer array representation than C (negative bounds, dynamic
> bounds, bounds that are associated with variables & not encapsulated
> are part of the array structure). We also want to stay compatible with
> the existing Fortran aware debuggers that expect a certain shape of
> DWARF.  This issue is on use to explain ourselves more clearly.
Understood.
> Sorry it feels that way to you.  Most of our design communication
> happens via github; if you are watching the project you'll see when
> new documents and new code is posted.  As far as I know, anyone can
> participate in a code review; if that is not the case please let me
> know & I'll do what I can to open it up.
>
> Most of what we are working on has been in flight for a long time.
> Two of the longest running tasks -- type declarations and symbols
> resolution and expression analysis, semantics, and folding -- have
> been documented for a long time & have seen many commits as the
> implementation progresses.  In the last month, the control flow graph
> documentation and header files have been open for comments in a pull
> request.  We did get questions and comments (via email) from a few
> people.
I think the main issue is that the LLVM community doesn't really operate
that way.  Design discussion happens on mailing lists.  There's been
some discussion about moving to GitHub PRs but even if a decision is
made to do so, the earliest it could happen is late this fall (1 year
after the monorepo publication).  Even then, I expect higher-level
design work will still happen on the mailing lists.

So when I see "flang-dev" as a mailing list, I expect discussion
about,
well, flang development.  :)

I'm hopeful that this RFC will spur some discussion about the current
design.  In the meantime, I'll see if I can subscribe to the GitHub
repository and get notifications.
> Perhaps is my mistake to judge interest in f18 by traffic on github
> and the flang-dev mailing list.  
I think there is a lot of interest!
> I'm happy to host one time or recurring call to discuss f18.  One of
> the action items from last July's introductory call was to schedule
> another call to discuss how arrays would be represented in the CFG and
> in llvm IR; there was lively discussion of various options during the
> July call.
That very discussion came to my mind too.  It's one of the most critical
design questions.

                            -David

> On Mar 1, 2019, at 11:45 AM, David Greene via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> There are Differentials posted for Fortran debug handling in LLVM and
> the last update on them was Nov. 28.  As far as I can tell the status on
> them is:
> 
> https://reviews.llvm.org/D54327 - Looks ready to land, awaiting approval
>  from someone.  Who is the someone?
> https://reviews.llvm.org/D54043 - Paul Robinson did some groundwork
>  to aid the effort.  No updates since.
> 
> https://reviews.llvm.org/D54114 - No updates since Nov. 6.  Some
>  questions awaiting response.
Because of the large volume of patches that need to be reviewed it often happens
that a review can get stuck. The accepted best practice is to ping the review
thread, e.g. once per week to force it to the top of the review's queue and
specifically call out what questions need answering, etc...

-- adrian

> On Mar 1, 2019, at 2:36 PM, Adrian Prantl via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> 
> 
>> On Mar 1, 2019, at 11:45 AM, David Greene via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>> 
>> There are Differentials posted for Fortran debug handling in LLVM and
>> the last update on them was Nov. 28.  As far as I can tell the status
on
>> them is:
>> 
>> https://reviews.llvm.org/D54327 - Looks ready to land, awaiting
approval
>> from someone.  Who is the someone?
> 
>> https://reviews.llvm.org/D54043 - Paul Robinson did some groundwork
>> to aid the effort.  No updates since.
>> 
>> https://reviews.llvm.org/D54114 - No updates since Nov. 6.  Some
>> questions awaiting response.
> 
> Because of the large volume of patches that need to be reviewed it often
happens that a review can get stuck. The accepted best practice is to ping the
review thread, e.g. once per week to force it to the top of the review's
queue and specifically call out what questions need answering, etc...
Looks like I replied to quickly... After reading through these reviews all of
them are (as was actually pointed out earlier in this thread) stuck on the patch
authors answering questions and not waiting on reviewers.

-- adrian

> * Queued Fortran debugging changes for llvm
> 
> Yes, we have three reviews in the queue for changes that extend llvm debug
> metadata to support Fortran constructs in DWARF.   The feedback requested
> that one of the changes be implemented differently, which we have yet to
> budget time to reimplement.  The other feedback was for us to think harder
> about mapping Fortran to existing C constructs; our experience says this
> approach won't work because, for example, Fortran has much richer array
> representation than C (negative bounds, dynamic bounds, bounds that are
> associated with variables & not encapsulated are part of the array
> structure). We also want to stay compatible with the existing Fortran
> aware debuggers that expect a certain shape of DWARF.  This issue is on
> use to explain ourselves more clearly.
Thanks for explaining; I had wondered about the dead air.

It is certainly the case that when you have worked hard on something and
you post your wonderful implementation, you're not exactly expecting 
significant push-back and won't have allocated plenty of time for 
engaging with the people who think it should work differently!  But this 
is in the nature of working with an open-source community.  Working with 
"upstream" has a definite cost, but ultimately also a significant
benefit.

Please keep engaged, clearly there's a lot of interest here.
--paulr