llvm dev - Dec 2019 - Flang landing in the monorepo

Hi Eric,

Apologies, I failed to disambiguate clearly, because there are multiple projects
named flang. I was referring to the "new" flang, whose repository is
currently found at https://github.com/flang-compiler/f18. It will land in the
monorepo under a directory called "/flang/".

f18 has been approved to join, for reference see "[llvm-dev] f18 is
accepted as part of LLVM project!", Chriss Lattner, April 10 2019:
http://lists.llvm.org/pipermail/llvm-dev/2019-April/131703.html.

I would like to emphasize that it will not be turned on by default in the llvm
build.

Regards,

- Peter

On 18/12/2019 01:19, Eric Christopher wrote:
Hi Peter,

I have a few concerns and questions so far:

 - Supported C++ compilers: It looks like nothing too recent has been used to
test? And the flang page still has version 7 listed as the "latest
llvm".
 - "It will be closely aligned with LLVM best practices and written in the
style of LLVM and clang":
    - The directory structure is quite different from clang's directory
structure.
    - IR generation still appears to be text string based?
    - I didn't see a single reference to ADT or any of llvm's libraries
on a cursory look through the f18 directory (grep -ri ADT and grep -ri llvm)
    - It looks like flang is a C based project and f18 is C++? Looking at the
flang directory itself for IR generation is quite confusing and while named C++
in some places is actually just C with few abstractions?

I haven't done a full reading of the code, this is just a starting point for
discussion and review.

Mostly it seems I have a lot of questions and so getting an updated idea of what
you expect to merge, what the sources are, and more would be a first start I
think. If some of my concerns hold through discussion I can't see this being
merged as-is.

Thanks!

-eric

On Tue, Dec 17, 2019 at 1:31 PM Peter Waller via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
Hi All,

The flang project (a Fortran compiler) is getting ready to join the
monorepo. We intend to preserve the existing history by rewriting the
existing commits as a linear series of commits on top of llvm-project.

I understand the flang community would like to do this before the LLVM
10 branch in due in mid January, so please speak up soon if you see
anything needing fixing in what I write below.

I've taken into account the discussion raised during the MLIR landing
discussion found at
http://lists.llvm.org/pipermail/llvm-dev/2019-November/136813.html. As
with MLIR, we rewrite the commits so that flang's work all appears to
happen in the flang directory, starting with llvm-project master as it
appears today. The topology of the f18 history was fairly interesting,
which is why I ended up writing a new program to rewrite it rather than
using an existing one.

=== Key links

* Resulting tree of the rewrite:
https://github.com/peterwaller-arm/f18/tree/rewritten-history-v2-llvm-project-merge

* Rewritten history, with flang commits applied on top of llvm-project
master:
https://github.com/peterwaller-arm/f18/commits/rewritten-history-v2-llvm-project-merge

* The history rewriting program is published here:
   https://github.com/flang-compiler/f18/pull/854

* Latest mailing list discussion of rewrite on flang-dev:
http://lists.llvm.org/pipermail/flang-dev/2019-December/000122.html

=== Additional considerations

* Existing references to pull request and issue numbers are rewritten so
that they point at the originals as, e.g. flang-compiler/f18#123. This
prevents those patches from generating bogus references to Issues/PRs of
llvm-project if/when those appear in the llvm-project repository.

* Developers using the llvm-project repo, when they pull after this
push, will see 2,700ish commits appear on the tip. These will follow on
as normal commits from wherever master is at the time of the push. The
fetch takes 40s and I see my ".git" directory grow by approximately
90MiB when I simulate this.

* Rewriting and validating the rewritten f18 history is sufficiently
fast that I don't think it will be necessary to pause commits to LLVM.
The script runs in a few seconds. Before this is done though, I think
new commits should no longer be accepted on the original repository.

* You can simulate the experience of the fresh merge with `git remote
add peterwaller-arm https://github.com/peterwaller-arm/f18 && time git
fetch peterwaller-arm rewritten-history-v2-llvm-project-merge`, and then
look at the peterwaller-arm/rewritten-history-v2-llvm-project-merge
branch with git log.

* Remember that you can restrict the "git log" output to what you are
interested in by specifying a directory, e.g. `git log clang/`.

That's all for now. Season's greetings!

- Peter

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191218/e4bf6470/attachment.html>

Hi Peter,

Yes, I looked through those sources and a number of my questions around
which clang versions have been supported and directory structure. I think
the only difference is removing the direct questions about earlier flang,
but I still don't see code generation or uses of llvm libraries that would
conform to "written in the style of LLVM and clang". Can you perhaps
point
me to where I'm missing these things?

Chris's earlier acceptance aside I don't see any evidence of code review
as
part of that and so I'd expect we'd see more here.

Thanks.

-eric

On Wed, Dec 18, 2019 at 1:44 AM Peter Waller <Peter.Waller at arm.com>
wrote:
> Hi Eric,
>
> Apologies, I failed to disambiguate clearly, because there are multiple
> projects named flang. I was referring to the "new" flang, whose
repository
> is currently found at https://github.com/flang-compiler/f18. It will land
> in the monorepo under a directory called "/flang/".
>
> f18 has been approved to join, for reference see "[llvm-dev] f18 is
> accepted as part of LLVM project!", Chriss Lattner, April 10 2019:
> http://lists.llvm.org/pipermail/llvm-dev/2019-April/131703.html.
>
> I would like to emphasize that it will not be turned on by default in the
> llvm build.
>
> Regards,
>
> - Peter
> On 18/12/2019 01:19, Eric Christopher wrote:
>
> Hi Peter,
>
> I have a few concerns and questions so far:
>
>  - Supported C++ compilers: It looks like nothing too recent has been used
> to test? And the flang page still has version 7 listed as the "latest
llvm".
>  - "It will be closely aligned with LLVM best practices and written in
the
> style of LLVM and clang":
>     - The directory structure is quite different from clang's directory
> structure.
>     - IR generation still appears to be text string based?
>     - I didn't see a single reference to ADT or any of llvm's
libraries on
> a cursory look through the f18 directory (grep -ri ADT and grep -ri llvm)
>     - It looks like flang is a C based project and f18 is C++? Looking at
> the flang directory itself for IR generation is quite confusing and while
> named C++ in some places is actually just C with few abstractions?
>
> I haven't done a full reading of the code, this is just a starting
point
> for discussion and review.
>
> Mostly it seems I have a lot of questions and so getting an updated idea
> of what you expect to merge, what the sources are, and more would be a
> first start I think. If some of my concerns hold through discussion I
can't
> see this being merged as-is.
>
> Thanks!
>
> -eric
>
> On Tue, Dec 17, 2019 at 1:31 PM Peter Waller via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi All,
>>
>> The flang project (a Fortran compiler) is getting ready to join the
>> monorepo. We intend to preserve the existing history by rewriting the
>> existing commits as a linear series of commits on top of llvm-project.
>>
>> I understand the flang community would like to do this before the LLVM
>> 10 branch in due in mid January, so please speak up soon if you see
>> anything needing fixing in what I write below.
>>
>> I've taken into account the discussion raised during the MLIR
landing
>> discussion found at
>> http://lists.llvm.org/pipermail/llvm-dev/2019-November/136813.html. As
>> with MLIR, we rewrite the commits so that flang's work all appears
to
>> happen in the flang directory, starting with llvm-project master as it
>> appears today. The topology of the f18 history was fairly interesting,
>> which is why I ended up writing a new program to rewrite it rather than
>> using an existing one.
>>
>> === Key links
>>
>> * Resulting tree of the rewrite:
>>
>>
https://github.com/peterwaller-arm/f18/tree/rewritten-history-v2-llvm-project-merge
>>
>> * Rewritten history, with flang commits applied on top of llvm-project
>> master:
>>
>>
https://github.com/peterwaller-arm/f18/commits/rewritten-history-v2-llvm-project-merge
>>
>> * The history rewriting program is published here:
>>    https://github.com/flang-compiler/f18/pull/854
>>
>> * Latest mailing list discussion of rewrite on flang-dev:
>> http://lists.llvm.org/pipermail/flang-dev/2019-December/000122.html
>>
>> === Additional considerations
>>
>> * Existing references to pull request and issue numbers are rewritten
so
>> that they point at the originals as, e.g. flang-compiler/f18#123. This
>> prevents those patches from generating bogus references to Issues/PRs
of
>> llvm-project if/when those appear in the llvm-project repository.
>>
>> * Developers using the llvm-project repo, when they pull after this
>> push, will see 2,700ish commits appear on the tip. These will follow on
>> as normal commits from wherever master is at the time of the push. The
>> fetch takes 40s and I see my ".git" directory grow by
approximately
>> 90MiB when I simulate this.
>>
>> * Rewriting and validating the rewritten f18 history is sufficiently
>> fast that I don't think it will be necessary to pause commits to
LLVM.
>> The script runs in a few seconds. Before this is done though, I think
>> new commits should no longer be accepted on the original repository.
>>
>> * You can simulate the experience of the fresh merge with `git remote
>> add peterwaller-arm https://github.com/peterwaller-arm/f18 &&
time git
>> fetch peterwaller-arm rewritten-history-v2-llvm-project-merge`, and
then
>> look at the peterwaller-arm/rewritten-history-v2-llvm-project-merge
>> branch with git log.
>>
>> * Remember that you can restrict the "git log" output to what
you are
>> interested in by specifying a directory, e.g. `git log clang/`.
>>
>> That's all for now. Season's greetings!
>>
>> - Peter
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191218/383d21c9/attachment.html>

On 18/12/2019 21:49, Eric Christopher wrote:
> Yes, I looked through those sources and a number of my questions 
> around which clang versions have been supported and directory 
> structure. I think the only difference is removing the direct 
> questions about earlier flang, but I still don't see code generation 
> or uses of llvm libraries that would conform to "written in the style 
> of LLVM and clang". Can you perhaps point me to where I'm missing 
> these things?
I can't speak with authority on all of these issues.

In terms of clang versions, I understand that clang version 7 and 8 are 
currently supported. We would expect it to work with newer LLVM 
versions, and
the readme is currently out of date. The intent is that it will work 
with all branches of LLVM and the community will build up CI to protect 
this.

It's worth mentioning that merging flang in at this point does not 
affect the existing LLVM build in any way. Patches to integrate the 
build system are expected in the near future, and be subject to the 
normal LLVM code review processes.

I understand that code generation is a work in progress and is expected 
to start landing in the not too distant future. Other people (Steve 
Scalpone, cc'd, and others) can perhaps speak to this more than me.

In terms of using LLVM ADTs, etc, I expect that once flang is part of 
the monorepo, there will be a greater usage of those
things.
> Chris's earlier acceptance aside I don't see any evidence of code 
> review as part of that and so I'd expect we'd see more here.
Code review has been happening all along in the the f18 github repository:
https://github.com/flang-compiler/f18/pulls

I am operating on the assumption that the code will land with no 
additional review by members of the LLVM community, and future code 
review will happen with the same mechanisms that the wider LLVM project use.

Regards,

- Peter