llvm dev - Dec 2019 - Flang landing in the monorepo

Hi All,

The flang project (a Fortran compiler) is getting ready to join the 
monorepo. We intend to preserve the existing history by rewriting the 
existing commits as a linear series of commits on top of llvm-project.

I understand the flang community would like to do this before the LLVM 
10 branch in due in mid January, so please speak up soon if you see 
anything needing fixing in what I write below.

I've taken into account the discussion raised during the MLIR landing 
discussion found at 
http://lists.llvm.org/pipermail/llvm-dev/2019-November/136813.html. As 
with MLIR, we rewrite the commits so that flang's work all appears to 
happen in the flang directory, starting with llvm-project master as it 
appears today. The topology of the f18 history was fairly interesting, 
which is why I ended up writing a new program to rewrite it rather than 
using an existing one.

=== Key links

* Resulting tree of the rewrite:
https://github.com/peterwaller-arm/f18/tree/rewritten-history-v2-llvm-project-merge

* Rewritten history, with flang commits applied on top of llvm-project 
master:
https://github.com/peterwaller-arm/f18/commits/rewritten-history-v2-llvm-project-merge

* The history rewriting program is published here:
   https://github.com/flang-compiler/f18/pull/854

* Latest mailing list discussion of rewrite on flang-dev:
http://lists.llvm.org/pipermail/flang-dev/2019-December/000122.html

=== Additional considerations

* Existing references to pull request and issue numbers are rewritten so 
that they point at the originals as, e.g. flang-compiler/f18#123. This 
prevents those patches from generating bogus references to Issues/PRs of 
llvm-project if/when those appear in the llvm-project repository.

* Developers using the llvm-project repo, when they pull after this 
push, will see 2,700ish commits appear on the tip. These will follow on 
as normal commits from wherever master is at the time of the push. The 
fetch takes 40s and I see my ".git" directory grow by approximately 
90MiB when I simulate this.

* Rewriting and validating the rewritten f18 history is sufficiently 
fast that I don't think it will be necessary to pause commits to LLVM. 
The script runs in a few seconds. Before this is done though, I think 
new commits should no longer be accepted on the original repository.

* You can simulate the experience of the fresh merge with `git remote 
add peterwaller-arm https://github.com/peterwaller-arm/f18 && time git 
fetch peterwaller-arm rewritten-history-v2-llvm-project-merge`, and then 
look at the peterwaller-arm/rewritten-history-v2-llvm-project-merge 
branch with git log.

* Remember that you can restrict the "git log" output to what you are 
interested in by specifying a directory, e.g. `git log clang/`.

That's all for now. Season's greetings!

- Peter

On 12/17/2019 01:30 PM, Peter Waller via llvm-dev wrote:
> Hi All,
> 
> The flang project (a Fortran compiler) is getting ready to join the 
> monorepo. We intend to preserve the existing history by rewriting the 
> existing commits as a linear series of commits on top of llvm-project.
> 
> I understand the flang community would like to do this before the LLVM 
> 10 branch in due in mid January, so please speak up soon if you see 
> anything needing fixing in what I write below.
> 
> I've taken into account the discussion raised during the MLIR landing 
> discussion found at 
> http://lists.llvm.org/pipermail/llvm-dev/2019-November/136813.html. As 
> with MLIR, we rewrite the commits so that flang's work all appears to 
> happen in the flang directory, starting with llvm-project master as it 
> appears today. The topology of the f18 history was fairly interesting, 
> which is why I ended up writing a new program to rewrite it rather than 
> using an existing one.
> 
> === Key links
> 
> * Resulting tree of the rewrite:
>
https://github.com/peterwaller-arm/f18/tree/rewritten-history-v2-llvm-project-merge
> 
> * Rewritten history, with flang commits applied on top of llvm-project 
> master:
>
https://github.com/peterwaller-arm/f18/commits/rewritten-history-v2-llvm-project-merge
> 
> * The history rewriting program is published here:
>    https://github.com/flang-compiler/f18/pull/854
> 
> * Latest mailing list discussion of rewrite on flang-dev:
> http://lists.llvm.org/pipermail/flang-dev/2019-December/000122.html
> 
> === Additional considerations
> 
> * Existing references to pull request and issue numbers are rewritten so 
> that they point at the originals as, e.g. flang-compiler/f18#123. This 
> prevents those patches from generating bogus references to Issues/PRs of 
> llvm-project if/when those appear in the llvm-project repository.
> 
> * Developers using the llvm-project repo, when they pull after this 
> push, will see 2,700ish commits appear on the tip. These will follow on 
> as normal commits from wherever master is at the time of the push. The 
> fetch takes 40s and I see my ".git" directory grow by
approximately
> 90MiB when I simulate this.
> 
I think we may want to disable the commit emailer before all these
new commits are pushed.  Once you are ready to push, can you pick a
specific time and date for the push and then coordinate with Mike (cc'd)
and myself, so we can avoid spamming the mail server.

Thanks,
Tom
> * Rewriting and validating the rewritten f18 history is sufficiently 
> fast that I don't think it will be necessary to pause commits to LLVM. 
> The script runs in a few seconds. Before this is done though, I think 
> new commits should no longer be accepted on the original repository.
> 
> * You can simulate the experience of the fresh merge with `git remote 
> add peterwaller-arm https://github.com/peterwaller-arm/f18 && time
git
> fetch peterwaller-arm rewritten-history-v2-llvm-project-merge`, and then 
> look at the peterwaller-arm/rewritten-history-v2-llvm-project-merge 
> branch with git log.
> 
> * Remember that you can restrict the "git log" output to what you
are
> interested in by specifying a directory, e.g. `git log clang/`.
> 
> That's all for now. Season's greetings!
> 
> - Peter
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

I think it would probably make the most sense to land this as a single
merge-commit (from the 2700-commit rewritten history you've created) onto
llvm-project master, rather than as 2700 individual toplevel commits to
master. (Which means: disable the merge-commit prohibition in the github
configuration, temporarily, push this commit, and then enable it again).


On Tue, Dec 17, 2019 at 5:10 PM Tom Stellard via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 12/17/2019 01:30 PM, Peter Waller via llvm-dev wrote:
> > Hi All,
> >
> > The flang project (a Fortran compiler) is getting ready to join the
> > monorepo. We intend to preserve the existing history by rewriting the
> > existing commits as a linear series of commits on top of llvm-project.
> >
> > I understand the flang community would like to do this before the LLVM
> > 10 branch in due in mid January, so please speak up soon if you see
> > anything needing fixing in what I write below.
> >
> > I've taken into account the discussion raised during the MLIR
landing
> > discussion found at
> > http://lists.llvm.org/pipermail/llvm-dev/2019-November/136813.html. As
> > with MLIR, we rewrite the commits so that flang's work all appears
to
> > happen in the flang directory, starting with llvm-project master as it
> > appears today. The topology of the f18 history was fairly interesting,
> > which is why I ended up writing a new program to rewrite it rather
than
> > using an existing one.
> >
> > === Key links
> >
> > * Resulting tree of the rewrite:
> >
>
https://github.com/peterwaller-arm/f18/tree/rewritten-history-v2-llvm-project-merge
> >
> > * Rewritten history, with flang commits applied on top of llvm-project
> > master:
> >
>
https://github.com/peterwaller-arm/f18/commits/rewritten-history-v2-llvm-project-merge
> >
> > * The history rewriting program is published here:
> >    https://github.com/flang-compiler/f18/pull/854
> >
> > * Latest mailing list discussion of rewrite on flang-dev:
> > http://lists.llvm.org/pipermail/flang-dev/2019-December/000122.html
> >
> > === Additional considerations
> >
> > * Existing references to pull request and issue numbers are rewritten
so
> > that they point at the originals as, e.g. flang-compiler/f18#123. This
> > prevents those patches from generating bogus references to Issues/PRs
of
> > llvm-project if/when those appear in the llvm-project repository.
> >
> > * Developers using the llvm-project repo, when they pull after this
> > push, will see 2,700ish commits appear on the tip. These will follow
on
> > as normal commits from wherever master is at the time of the push. The
> > fetch takes 40s and I see my ".git" directory grow by
approximately
> > 90MiB when I simulate this.
> >
>
> I think we may want to disable the commit emailer before all these
> new commits are pushed.  Once you are ready to push, can you pick a
> specific time and date for the push and then coordinate with Mike
(cc'd)
> and myself, so we can avoid spamming the mail server.
>
> Thanks,
> Tom
>
> > * Rewriting and validating the rewritten f18 history is sufficiently
> > fast that I don't think it will be necessary to pause commits to
LLVM.
> > The script runs in a few seconds. Before this is done though, I think
> > new commits should no longer be accepted on the original repository.
> >
> > * You can simulate the experience of the fresh merge with `git remote
> > add peterwaller-arm https://github.com/peterwaller-arm/f18 &&
time git
> > fetch peterwaller-arm rewritten-history-v2-llvm-project-merge`, and
then
> > look at the peterwaller-arm/rewritten-history-v2-llvm-project-merge
> > branch with git log.
> >
> > * Remember that you can restrict the "git log" output to
what you are
> > interested in by specifying a directory, e.g. `git log clang/`.
> >
> > That's all for now. Season's greetings!
> >
> > - Peter
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191217/e165d3ad/attachment.html>

Hi Peter,

I have a few concerns and questions so far:

 - Supported C++ compilers: It looks like nothing too recent has been used
to test? And the flang page still has version 7 listed as the "latest
llvm".
 - "It will be closely aligned with LLVM best practices and written in the
style of LLVM and clang":
    - The directory structure is quite different from clang's directory
structure.
    - IR generation still appears to be text string based?
    - I didn't see a single reference to ADT or any of llvm's libraries
on
a cursory look through the f18 directory (grep -ri ADT and grep -ri llvm)
    - It looks like flang is a C based project and f18 is C++? Looking at
the flang directory itself for IR generation is quite confusing and while
named C++ in some places is actually just C with few abstractions?

I haven't done a full reading of the code, this is just a starting point
for discussion and review.

Mostly it seems I have a lot of questions and so getting an updated idea of
what you expect to merge, what the sources are, and more would be a first
start I think. If some of my concerns hold through discussion I can't see
this being merged as-is.

Thanks!

-eric

On Tue, Dec 17, 2019 at 1:31 PM Peter Waller via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi All,
>
> The flang project (a Fortran compiler) is getting ready to join the
> monorepo. We intend to preserve the existing history by rewriting the
> existing commits as a linear series of commits on top of llvm-project.
>
> I understand the flang community would like to do this before the LLVM
> 10 branch in due in mid January, so please speak up soon if you see
> anything needing fixing in what I write below.
>
> I've taken into account the discussion raised during the MLIR landing
> discussion found at
> http://lists.llvm.org/pipermail/llvm-dev/2019-November/136813.html. As
> with MLIR, we rewrite the commits so that flang's work all appears to
> happen in the flang directory, starting with llvm-project master as it
> appears today. The topology of the f18 history was fairly interesting,
> which is why I ended up writing a new program to rewrite it rather than
> using an existing one.
>
> === Key links
>
> * Resulting tree of the rewrite:
>
>
https://github.com/peterwaller-arm/f18/tree/rewritten-history-v2-llvm-project-merge
>
> * Rewritten history, with flang commits applied on top of llvm-project
> master:
>
>
https://github.com/peterwaller-arm/f18/commits/rewritten-history-v2-llvm-project-merge
>
> * The history rewriting program is published here:
>    https://github.com/flang-compiler/f18/pull/854
>
> * Latest mailing list discussion of rewrite on flang-dev:
> http://lists.llvm.org/pipermail/flang-dev/2019-December/000122.html
>
> === Additional considerations
>
> * Existing references to pull request and issue numbers are rewritten so
> that they point at the originals as, e.g. flang-compiler/f18#123. This
> prevents those patches from generating bogus references to Issues/PRs of
> llvm-project if/when those appear in the llvm-project repository.
>
> * Developers using the llvm-project repo, when they pull after this
> push, will see 2,700ish commits appear on the tip. These will follow on
> as normal commits from wherever master is at the time of the push. The
> fetch takes 40s and I see my ".git" directory grow by
approximately
> 90MiB when I simulate this.
>
> * Rewriting and validating the rewritten f18 history is sufficiently
> fast that I don't think it will be necessary to pause commits to LLVM.
> The script runs in a few seconds. Before this is done though, I think
> new commits should no longer be accepted on the original repository.
>
> * You can simulate the experience of the fresh merge with `git remote
> add peterwaller-arm https://github.com/peterwaller-arm/f18 && time
git
> fetch peterwaller-arm rewritten-history-v2-llvm-project-merge`, and then
> look at the peterwaller-arm/rewritten-history-v2-llvm-project-merge
> branch with git log.
>
> * Remember that you can restrict the "git log" output to what you
are
> interested in by specifying a directory, e.g. `git log clang/`.
>
> That's all for now. Season's greetings!
>
> - Peter
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191217/f43f6459/attachment.html>

How about [flang] :-) The reviewed-on tag in the commit messages has
the issue number so no information is lost.

[flang] Merge pull request flang-compiler#862 from flang...

Original-commit: flang-compiler/f18 at 8d2b296
Reviewed-on: flang-compiler#862



On Tue, Dec 17, 2019 at 1:31 PM Peter Waller via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> Hi All,
>
> The flang project (a Fortran compiler) is getting ready to join the
> monorepo. We intend to preserve the existing history by rewriting the
> existing commits as a linear series of commits on top of llvm-project.
>
> I understand the flang community would like to do this before the LLVM
> 10 branch in due in mid January, so please speak up soon if you see
> anything needing fixing in what I write below.
>
> I've taken into account the discussion raised during the MLIR landing
> discussion found at
> http://lists.llvm.org/pipermail/llvm-dev/2019-November/136813.html. As
> with MLIR, we rewrite the commits so that flang's work all appears to
> happen in the flang directory, starting with llvm-project master as it
> appears today. The topology of the f18 history was fairly interesting,
> which is why I ended up writing a new program to rewrite it rather than
> using an existing one.
>
> === Key links
>
> * Resulting tree of the rewrite:
>
https://github.com/peterwaller-arm/f18/tree/rewritten-history-v2-llvm-project-merge
>
> * Rewritten history, with flang commits applied on top of llvm-project
> master:
>
https://github.com/peterwaller-arm/f18/commits/rewritten-history-v2-llvm-project-merge
>
> * The history rewriting program is published here:
>    https://github.com/flang-compiler/f18/pull/854
>
> * Latest mailing list discussion of rewrite on flang-dev:
> http://lists.llvm.org/pipermail/flang-dev/2019-December/000122.html
>
> === Additional considerations
>
> * Existing references to pull request and issue numbers are rewritten so
> that they point at the originals as, e.g. flang-compiler/f18#123. This
> prevents those patches from generating bogus references to Issues/PRs of
> llvm-project if/when those appear in the llvm-project repository.
>
> * Developers using the llvm-project repo, when they pull after this
> push, will see 2,700ish commits appear on the tip. These will follow on
> as normal commits from wherever master is at the time of the push. The
> fetch takes 40s and I see my ".git" directory grow by
approximately
> 90MiB when I simulate this.
>
> * Rewriting and validating the rewritten f18 history is sufficiently
> fast that I don't think it will be necessary to pause commits to LLVM.
> The script runs in a few seconds. Before this is done though, I think
> new commits should no longer be accepted on the original repository.
>
> * You can simulate the experience of the fresh merge with `git remote
> add peterwaller-arm https://github.com/peterwaller-arm/f18 && time
git
> fetch peterwaller-arm rewritten-history-v2-llvm-project-merge`, and then
> look at the peterwaller-arm/rewritten-history-v2-llvm-project-merge
> branch with git log.
>
> * Remember that you can restrict the "git log" output to what you
are
> interested in by specifying a directory, e.g. `git log clang/`.
>
> That's all for now. Season's greetings!
>
> - Peter
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Hi Eric,

Apologies, I failed to disambiguate clearly, because there are multiple projects
named flang. I was referring to the "new" flang, whose repository is
currently found at https://github.com/flang-compiler/f18. It will land in the
monorepo under a directory called "/flang/".

f18 has been approved to join, for reference see "[llvm-dev] f18 is
accepted as part of LLVM project!", Chriss Lattner, April 10 2019:
http://lists.llvm.org/pipermail/llvm-dev/2019-April/131703.html.

I would like to emphasize that it will not be turned on by default in the llvm
build.

Regards,

- Peter

On 18/12/2019 01:19, Eric Christopher wrote:
Hi Peter,

I have a few concerns and questions so far:

 - Supported C++ compilers: It looks like nothing too recent has been used to
test? And the flang page still has version 7 listed as the "latest
llvm".
 - "It will be closely aligned with LLVM best practices and written in the
style of LLVM and clang":
    - The directory structure is quite different from clang's directory
structure.
    - IR generation still appears to be text string based?
    - I didn't see a single reference to ADT or any of llvm's libraries
on a cursory look through the f18 directory (grep -ri ADT and grep -ri llvm)
    - It looks like flang is a C based project and f18 is C++? Looking at the
flang directory itself for IR generation is quite confusing and while named C++
in some places is actually just C with few abstractions?

I haven't done a full reading of the code, this is just a starting point for
discussion and review.

Mostly it seems I have a lot of questions and so getting an updated idea of what
you expect to merge, what the sources are, and more would be a first start I
think. If some of my concerns hold through discussion I can't see this being
merged as-is.

Thanks!

-eric

On Tue, Dec 17, 2019 at 1:31 PM Peter Waller via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
Hi All,

The flang project (a Fortran compiler) is getting ready to join the
monorepo. We intend to preserve the existing history by rewriting the
existing commits as a linear series of commits on top of llvm-project.

I understand the flang community would like to do this before the LLVM
10 branch in due in mid January, so please speak up soon if you see
anything needing fixing in what I write below.

I've taken into account the discussion raised during the MLIR landing
discussion found at
http://lists.llvm.org/pipermail/llvm-dev/2019-November/136813.html. As
with MLIR, we rewrite the commits so that flang's work all appears to
happen in the flang directory, starting with llvm-project master as it
appears today. The topology of the f18 history was fairly interesting,
which is why I ended up writing a new program to rewrite it rather than
using an existing one.

=== Key links

* Resulting tree of the rewrite:
https://github.com/peterwaller-arm/f18/tree/rewritten-history-v2-llvm-project-merge

* Rewritten history, with flang commits applied on top of llvm-project
master:
https://github.com/peterwaller-arm/f18/commits/rewritten-history-v2-llvm-project-merge

* The history rewriting program is published here:
   https://github.com/flang-compiler/f18/pull/854

* Latest mailing list discussion of rewrite on flang-dev:
http://lists.llvm.org/pipermail/flang-dev/2019-December/000122.html

=== Additional considerations

* Existing references to pull request and issue numbers are rewritten so
that they point at the originals as, e.g. flang-compiler/f18#123. This
prevents those patches from generating bogus references to Issues/PRs of
llvm-project if/when those appear in the llvm-project repository.

* Developers using the llvm-project repo, when they pull after this
push, will see 2,700ish commits appear on the tip. These will follow on
as normal commits from wherever master is at the time of the push. The
fetch takes 40s and I see my ".git" directory grow by approximately
90MiB when I simulate this.

* Rewriting and validating the rewritten f18 history is sufficiently
fast that I don't think it will be necessary to pause commits to LLVM.
The script runs in a few seconds. Before this is done though, I think
new commits should no longer be accepted on the original repository.

* You can simulate the experience of the fresh merge with `git remote
add peterwaller-arm https://github.com/peterwaller-arm/f18 && time git
fetch peterwaller-arm rewritten-history-v2-llvm-project-merge`, and then
look at the peterwaller-arm/rewritten-history-v2-llvm-project-merge
branch with git log.

* Remember that you can restrict the "git log" output to what you are
interested in by specifying a directory, e.g. `git log clang/`.

That's all for now. Season's greetings!

- Peter

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191218/e4bf6470/attachment.html>

On 18/12/2019 04:48, Aaron Smith wrote:
> How about [flang] :-) The reviewed-on tag in the commit messages has
> the issue number so no information is lost.
>
> [flang] Merge pull request flang-compiler#862 from flang...
>
> Original-commit: flang-compiler/f18 at 8d2b296
> Reviewed-on: flang-compiler#862
Hi,

Adding the pull request number to the individual commits was requested 
in https://github.com/flang-compiler/f18/pull/854#discussion_r355516596, 
but granted since then we also chose to keep the empty merge commits for 
their commit message metadata as well, so maybe it is surplus to 
requirements now. I've put your suggestion over on the pull request for 
the history rewriting program. 
https://github.com/flang-compiler/f18/pull/854#issuecomment-566958262

Regards,

- Peter

On 17/12/2019 22:10, Tom Stellard wrote:
>
> I think we may want to disable the commit emailer before all these
> new commits are pushed.  Once you are ready to push, can you pick a
> specific time and date for the push and then coordinate with Mike
(cc'd)
> and myself, so we can avoid spamming the mail server.
Hi Tom,

Great suggestion. I've put it on our checklist, which I'm recording on 
https://github.com/flang-compiler/f18/issues/876.

Regards,

- Peter