llvm dev - Nov 2019 - MLIR landing in the monorepo

Hi,

(bcc: mlir at tensorflow.org FYI)

I am following-up on the integration of MLIR in LLVM as a subproject (Re:
http://lists.llvm.org/pipermail/llvm-dev/2019-October/135687.html ).

We're aiming to integrate into the monorepo next month. Right now our
intent is for MLIR to live in a top-level directory in parallel to clang,
lldb, lld, etc.
Our top option for the integration is to perform a git subtree merge to
bring the MLIR history into the monorepo, here is a prototype:
https://github.com/joker-eph/llvm-project-with-mlir

If you're curious to try it, at the moment it needs a specific CMake
invocation:

cmake -G Ninja ../llvm/ -DLLVM_TARGETS_TO_BUILD="host"
-DLLVM_EXTERNAL_PROJECTS=mlir -DLLVM_EXTERNAL_MLIR_SOURCE_DIR={path to
repo}/mlir/

We'll hook into -DLLVM_ENABLE_PROJECTS after landing.

Let me know if you have any comment about this!

Cheers,

-- 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191115/3cc8ead9/attachment.html>

On Fri, Nov 15, 2019 at 5:03 AM Mehdi AMINI via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi,
>
> (bcc: mlir at tensorflow.org FYI)
>
> I am following-up on the integration of MLIR in LLVM as a subproject (Re:
> http://lists.llvm.org/pipermail/llvm-dev/2019-October/135687.html ).
>
> We're aiming to integrate into the monorepo next month. Right now our
> intent is for MLIR to live in a top-level directory in parallel to clang,
> lldb, lld, etc.
>
Sounds right.

> Our top option for the integration is to perform a git subtree merge to
> bring the MLIR history into the monorepo, here is a prototype:
> https://github.com/joker-eph/llvm-project-with-mlir
>
I'll note that this would be the very-first merge commit on master. I'm
not
opposed to this, but others may be. (To allow this, we'd temporarily
reconfigure github to allow pushing merge-commits, for this one commit, and
then disable it again.)

However, another issue is that subtree merges have really weird artifacts
when trying to look through history, with e.g. git log <filename>. I think
I'd really prefer to avoid utilizing a subtree-merge for this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191115/fe0be7f3/attachment-0001.html>

On Fri, Nov 15, 2019 at 8:54 AM James Y Knight <jyknight at google.com>
wrote:
>
>
> On Fri, Nov 15, 2019 at 5:03 AM Mehdi AMINI via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi,
>>
>> (bcc: mlir at tensorflow.org FYI)
>>
>> I am following-up on the integration of MLIR in LLVM as a subproject
(Re:
>> http://lists.llvm.org/pipermail/llvm-dev/2019-October/135687.html ).
>>
>> We're aiming to integrate into the monorepo next month. Right now
our
>> intent is for MLIR to live in a top-level directory in parallel to
clang,
>> lldb, lld, etc.
>>
>
> Sounds right.
>
>
>> Our top option for the integration is to perform a git subtree merge to
>> bring the MLIR history into the monorepo, here is a prototype:
>> https://github.com/joker-eph/llvm-project-with-mlir
>>
>
> I'll note that this would be the very-first merge commit on master.
I'm
> not opposed to this, but others may be. (To allow this, we'd
temporarily
> reconfigure github to allow pushing merge-commits, for this one commit, and
> then disable it again.)
>
> However, another issue is that subtree merges have really weird artifacts
> when trying to look through history, with e.g. git log <filename>. I
think
> I'd really prefer to avoid utilizing a subtree-merge for this.
>
Right: `git blame` work well though. The alternative would be to have a
commit to move the files under a “mlir” directory and then do a normal
merge?
I am interested to hear about other advices here :)

Thanks,

Mehdi
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191115/d6f2c7b1/attachment-0001.html>

Since you are going to rewrite the mlir history anyway, you can
probably delete accidentally checked in large files if any.

* I don't know whether the file CONTRIBUTING.md is still appropriate,
at least for the Code of Conduct, LLVM has its own version.
* g3doc/ seems a very Google specific name. Does `docs/` work?
* bindings/python/pybind.cpp - does it have to be an in-tree plugin?
* The Apache 2 license headers are verbose. LLVM uses
SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception.

On Fri, Nov 15, 2019 at 2:03 AM Mehdi AMINI via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> Hi,
>
> (bcc: mlir at tensorflow.org FYI)
>
> I am following-up on the integration of MLIR in LLVM as a subproject (Re:
http://lists.llvm.org/pipermail/llvm-dev/2019-October/135687.html ).
>
> We're aiming to integrate into the monorepo next month. Right now our
intent is for MLIR to live in a top-level directory in parallel to clang, lldb,
lld, etc.
> Our top option for the integration is to perform a git subtree merge to
bring the MLIR history into the monorepo, here is a prototype:
https://github.com/joker-eph/llvm-project-with-mlir
>
> If you're curious to try it, at the moment it needs a specific CMake
invocation:
>
> cmake -G Ninja ../llvm/ -DLLVM_TARGETS_TO_BUILD="host"
-DLLVM_EXTERNAL_PROJECTS=mlir -DLLVM_EXTERNAL_MLIR_SOURCE_DIR={path to
repo}/mlir/
>
> We'll hook into -DLLVM_ENABLE_PROJECTS after landing.
>
> Let me know if you have any comment about this!
>
> Cheers,
>
> --
> Mehdi
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


-- 
宋方睿

On Fri, Nov 15, 2019 at 10:58 AM Fāng-ruì Sòng <maskray at google.com>
wrote:
> Since you are going to rewrite the mlir history anyway, you can
> probably delete accidentally checked in large files if any.
>
Good point, I checked and this is the largest file in the history of the
repo as far as I can tell:
https://github.com/joker-eph/llvm-project-with-mlir/blob/master/mlir/g3doc/includes/img/view-operation.svg
(155kB)
>
> * I don't know whether the file CONTRIBUTING.md is still appropriate,
> at least for the Code of Conduct, LLVM has its own version.
> * g3doc/ seems a very Google specific name. Does `docs/` work?
> * bindings/python/pybind.cpp - does it have to be an in-tree plugin?
> * The Apache 2 license headers are verbose. LLVM uses
> SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception.
>
Absolutely! All of these points (except the python bindings) are on my TODO
list of things to do at the time of the merge. At the moment there is just
a script continuously performing the merge in my test repository so this is
an exact view of the current state of the public repo.
I could also try to rewrite the license in the header in all the history of
the repository, but I'm not sure it won't be brittle in practice, I was
planning to do an update to all the files before pushing.

I'll send the final version of the repo next month, I can CC you if
you'd
like to review this before we push it?

For the python bindings, these are intended to provide some equivalent
facility to the LLVM python bindings:
https://github.com/joker-eph/llvm-project-with-mlir/tree/master/llvm/bindings/python
; while they need some work at the moment, I think we will want to have
bindings though.

-- 
Mehdi


>
> On Fri, Nov 15, 2019 at 2:03 AM Mehdi AMINI via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >
> > Hi,
> >
> > (bcc: mlir at tensorflow.org FYI)
> >
> > I am following-up on the integration of MLIR in LLVM as a subproject
> (Re: http://lists.llvm.org/pipermail/llvm-dev/2019-October/135687.html ).
> >
> > We're aiming to integrate into the monorepo next month. Right now
our
> intent is for MLIR to live in a top-level directory in parallel to clang,
> lldb, lld, etc.
> > Our top option for the integration is to perform a git subtree merge
to
> bring the MLIR history into the monorepo, here is a prototype:
> https://github.com/joker-eph/llvm-project-with-mlir
> >
> > If you're curious to try it, at the moment it needs a specific
CMake
> invocation:
> >
> > cmake -G Ninja ../llvm/ -DLLVM_TARGETS_TO_BUILD="host"
> -DLLVM_EXTERNAL_PROJECTS=mlir -DLLVM_EXTERNAL_MLIR_SOURCE_DIR={path to
> repo}/mlir/
> >
> > We'll hook into -DLLVM_ENABLE_PROJECTS after landing.
> >
> > Let me know if you have any comment about this!
> >
> > Cheers,
> >
> > --
> > Mehdi
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> --
> 宋方睿
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191115/84fa300d/attachment-0001.html>

James Y Knight via llvm-dev <llvm-dev at lists.llvm.org> writes:
> However, another issue is that subtree merges have really weird artifacts
> when trying to look through history, with e.g. git log <filename>. I
think
> I'd really prefer to avoid utilizing a subtree-merge for this.
I agree.  Subtree-merges are really strange when walking back through
history.  If someone checks out an early commit from mlir, all they will
see in their working directory is the mlir sources.  All other
components will have disappeared.

I you want to maintain history, you can look at the scripts James, I and
others worked on for the LLVM git migration:

https://github.com/jyknight/llvm-git-migration

In particular, have a look at this pull request:

https://github.com/jyknight/llvm-git-migration/pull/6

This provides a tool to graft an existing git repository into a
subdirectory of the monorepo.  It has the option of rewriting commits so
that checkouts of those commits will maintain the other subprojects
alongside.  That functionality isn't well tested but I'd be happy to
work on it some more if needed.

I think this could work well for MLIR.

                      -David