llvm dev - Jun 2020 - [Incubation] Request to incubate mlir-npcomp

Per the recent (seeming) consensus regarding incubating new projects under
the LLVM organization, I would like to trial the process by requesting to
incubate mlir-npcomp <https://github.com/google/mlir-npcomp>. The project
is still quite young and has been primarily developed part time by myself
and Sean Silva over the last ~2 months. We set it up following discussion
of a Numpy/Scipy op set
<https://llvm.discourse.group/t/numpy-scipy-op-set/768> and also in
conjunction with a proving ground for high level dialects/transforms for
lowering from "numpy-aligned" frontends (e.g. sometimes labeled
TCF/TCP).

When I obtained permission from my employer (Google) to open source the
effort, it was with the understanding that it was being developed with an
eye towards eventual inclusion (in some fashion) under the LLVM umbrella.
As such, we set up licensing and repository organization in such a way as
to facilitate this down the road. We originally started it as a fork of the
LLVM repository, but transitioned to the MLIR standalone template
<https://github.com/llvm/llvm-project/tree/master/mlir/examples/standalone>,
and we found it more productive to iterate out of tree in this fashion,
bumping to the latest LLVM version every week or so as needed (note: the
ability to exist out of tree for MLIR dependent projects is actually quite
good, and the more of us who do it, the better it becomes).

As noted in the README, there are multiple directions that this project
could go in the future, possibly growing to be a full python/numpy compiler
frontend or just being a source of dialects, transforms and interface code:
we just need to keep building it to find out, and I am open to multiple
outcomes as things progress. In addition, there has been some inquiries
from the community regarding aligning this work with the LLVM foundation
explicitly, as it is an easier entity for contributors to align with.

So, as the first one walking through the door, what is the process we would
like to follow? I'm happy to provide more information/discussion, but
I'd
also be happy if with just an LGTM and someone creating an
"mlir-npcomp"
repository under the LLVM GitHub organization and working the rest out as
we go.

Thanks.
- Stella
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200622/9ad3f189/attachment-0001.html>

Personally, I'm very excited about this as it will make it much easier for
us to contribute to npcomp and I see the TCF/TCP dialects as being very
important for MLIR.

+1

Steve

On Mon, Jun 22, 2020 at 11:08 PM Stella Laurenzo <stellaraccident at
gmail.com>
wrote:
> Per the recent (seeming) consensus regarding incubating new projects under
> the LLVM organization, I would like to trial the process by requesting to
> incubate mlir-npcomp <https://github.com/google/mlir-npcomp>. The
project
> is still quite young and has been primarily developed part time by myself
> and Sean Silva over the last ~2 months. We set it up following discussion
> of a Numpy/Scipy op set
> <https://llvm.discourse.group/t/numpy-scipy-op-set/768> and also in
> conjunction with a proving ground for high level dialects/transforms for
> lowering from "numpy-aligned" frontends (e.g. sometimes labeled
TCF/TCP).
>
> When I obtained permission from my employer (Google) to open source the
> effort, it was with the understanding that it was being developed with an
> eye towards eventual inclusion (in some fashion) under the LLVM umbrella.
> As such, we set up licensing and repository organization in such a way as
> to facilitate this down the road. We originally started it as a fork of the
> LLVM repository, but transitioned to the MLIR standalone template
>
<https://github.com/llvm/llvm-project/tree/master/mlir/examples/standalone>,
> and we found it more productive to iterate out of tree in this fashion,
> bumping to the latest LLVM version every week or so as needed (note: the
> ability to exist out of tree for MLIR dependent projects is actually quite
> good, and the more of us who do it, the better it becomes).
>
> As noted in the README, there are multiple directions that this project
> could go in the future, possibly growing to be a full python/numpy compiler
> frontend or just being a source of dialects, transforms and interface code:
> we just need to keep building it to find out, and I am open to multiple
> outcomes as things progress. In addition, there has been some inquiries
> from the community regarding aligning this work with the LLVM foundation
> explicitly, as it is an easier entity for contributors to align with.
>
> So, as the first one walking through the door, what is the process we
> would like to follow? I'm happy to provide more information/discussion,
but
> I'd also be happy if with just an LGTM and someone creating an
> "mlir-npcomp" repository under the LLVM GitHub organization and
working the
> rest out as we go.
>
> Thanks.
> - Stella
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200623/985dc08c/attachment.html>

On Jun 22, 2020, at 11:07 PM, Stella Laurenzo via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> Per the recent (seeming) consensus regarding incubating new projects under
the LLVM organization, I would like to trial the process by requesting to
incubate mlir-npcomp <https://github.com/google/mlir-npcomp>. The project
is still quite young and has been primarily developed part time by myself and
Sean Silva over the last ~2 months. We set it up following discussion of a
Numpy/Scipy op set <https://llvm.discourse.group/t/numpy-scipy-op-set/768>
and also in conjunction with a proving ground for high level dialects/transforms
for lowering from "numpy-aligned" frontends (e.g. sometimes labeled
TCF/TCP).
Awesome.
> So, as the first one walking through the door, what is the process we would
like to follow? I'm happy to provide more information/discussion, but
I'd also be happy if with just an LGTM and someone creating an
"mlir-npcomp" repository under the LLVM GitHub organization and
working the rest out as we go.
I feel like we have general consensus that something like an incubator process
is a good idea, but I think we should converge on adding a new section to the
LLVM Developer Policy that describe what an incubator project is, and outline
the requirements (following the policy etc).

Could you help put together a draft of a patch?  If not, I can take a look at
this this weekend.

-Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200623/6f158e40/attachment-0001.html>

On Tue, Jun 23, 2020, 11:04 PM Chris Lattner <clattner at nondot.org>
wrote:
> On Jun 22, 2020, at 11:07 PM, Stella Laurenzo via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Per the recent (seeming) consensus regarding incubating new projects under
> the LLVM organization, I would like to trial the process by requesting to
> incubate mlir-npcomp <https://github.com/google/mlir-npcomp>. The
project
> is still quite young and has been primarily developed part time by myself
> and Sean Silva over the last ~2 months. We set it up following discussion
> of a Numpy/Scipy op set
> <https://llvm.discourse.group/t/numpy-scipy-op-set/768> and also in
> conjunction with a proving ground for high level dialects/transforms for
> lowering from "numpy-aligned" frontends (e.g. sometimes labeled
TCF/TCP).
>
>
> Awesome.
>
> So, as the first one walking through the door, what is the process we
> would like to follow? I'm happy to provide more information/discussion,
but
> I'd also be happy if with just an LGTM and someone creating an
> "mlir-npcomp" repository under the LLVM GitHub organization and
working the
> rest out as we go.
>
>
> I feel like we have general consensus that something like an incubator
> process is a good idea, but I think we should converge on adding a new
> section to the LLVM Developer Policy that describe what an incubator
> project is, and outline the requirements (following the policy etc).
>
> Could you help put together a draft of a patch?  If not, I can take a look
> at this this weekend.
>
Yes, I can do that and can likely get a draft out before the weekend (but
may do it this weekend).

> -Chris
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200624/b5f56320/attachment.html>

On Tue, Jun 23, 2020 at 2:40 PM Stella Laurenzo via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> We originally started it as a fork of the LLVM repository, but transitioned
to the MLIR standalone template, and we found it more productive to iterate out
of tree in this fashion, bumping to the latest LLVM version every week or so as
needed (note: the ability to exist out of tree for MLIR dependent projects is
actually quite good, and the more of us who do it, the better it becomes).
How do you deal with the problem of using the "right" LLVM version? As
somebody who spends a significant amount of time on a project that is
open-source but out-of-tree -- and for good reasons that mean we're
unlikely to want to incubate in this fashion -- I find this to be a
major problem.

If the goal of incubation is to eventually become part of the
llvm-project monorepo, I feel that being inside the monorepo should be
a goal early on. This would make your project more inclusive, as
others will automatically have the right LLVM version -- they don't
have to follow some syncing mechanism that you may have tooling for
inside of Google but which isn't available outside. You can always
"bump to the latest LLVM version every week or so" by doing a merge
commit.

Cheers,
Nicolai


-- 
Lerne, wie die Welt wirklich ist,
aber vergiss niemals, wie sie sein sollte.

On Wed, Jun 24, 2020 at 9:54 AM Nicolai Hähnle via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Tue, Jun 23, 2020 at 2:40 PM Stella Laurenzo via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > We originally started it as a fork of the LLVM repository, but
> transitioned to the MLIR standalone template, and we found it more
> productive to iterate out of tree in this fashion, bumping to the latest
> LLVM version every week or so as needed (note: the ability to exist out of
> tree for MLIR dependent projects is actually quite good, and the more of us
> who do it, the better it becomes).
>
> How do you deal with the problem of using the "right" LLVM
version? As
> somebody who spends a significant amount of time on a project that is
> open-source but out-of-tree -- and for good reasons that mean we're
> unlikely to want to incubate in this fashion -- I find this to be a
> major problem.
>
> If the goal of incubation is to eventually become part of the
> llvm-project monorepo, I feel that being inside the monorepo should be
> a goal early on. This would make your project more inclusive, as
> others will automatically have the right LLVM version -- they don't
> have to follow some syncing mechanism that you may have tooling for
> inside of Google but which isn't available outside.

There is nothing specific to Google, quite the opposite actually: the
"update once a week" here is purely OSS and does not work inside
Google.


> You can always
> "bump to the latest LLVM version every week or so" by doing a
merge
> commit.
>
I'm not sure I perceive how having code that is a "read-only"
mirror (the
LLVM monorepo) mixed within the project repo would be helping to be "more
inclusive"?
Not duplicating the monorepo helps to ensure that you don't diverge from
the rest of LLVM by patching it (you're losing flexibility in the
development of course, but then shouldn't this just be in the monorepo in
the first place?)

But it also seems to me that we're spreading the discussion on the
"right
template" between the incubator thread and this particular proposal, which
does not help to keep track.

-- 
Mehdi

>
> Cheers,
> Nicolai
>
>
> --
> Lerne, wie die Welt wirklich ist,
> aber vergiss niemals, wie sie sein sollte.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200624/4bcb6f22/attachment.html>

On Wed, Jun 24, 2020 at 9:54 AM Nicolai Hähnle <nhaehnle at gmail.com>
wrote:
> On Tue, Jun 23, 2020 at 2:40 PM Stella Laurenzo via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > We originally started it as a fork of the LLVM repository, but
> transitioned to the MLIR standalone template, and we found it more
> productive to iterate out of tree in this fashion, bumping to the latest
> LLVM version every week or so as needed (note: the ability to exist out of
> tree for MLIR dependent projects is actually quite good, and the more of us
> who do it, the better it becomes).
>
> How do you deal with the problem of using the "right" LLVM
version? As
> somebody who spends a significant amount of time on a project that is
> open-source but out-of-tree -- and for good reasons that mean we're
> unlikely to want to incubate in this fashion -- I find this to be a
> major problem.
>
> If the goal of incubation is to eventually become part of the
> llvm-project monorepo, I feel that being inside the monorepo should be
> a goal early on.

Actually that would be a big problem in practice, because it means that
either:
1. random changes in the monorepo can put the incubator into an unbuildable
state
2. people changing the monorepo need to somehow build and test and fix
incubator projects

Currently, in npcomp, we have a monorepo hash that we bump periodically.
That means that people can follow our README and build our project at any
point by checking out the right monorepo revision. Npcomp developers have
the responsibility of fixing our own code as LLVM updates.

-- Sean Silva

> This would make your project more inclusive, as
> others will automatically have the right LLVM version -- they don't
> have to follow some syncing mechanism that you may have tooling for
> inside of Google but which isn't available outside. You can always
> "bump to the latest LLVM version every week or so" by doing a
merge
> commit.
>
> Cheers,
> Nicolai
>
>
> --
> Lerne, wie die Welt wirklich ist,
> aber vergiss niemals, wie sie sein sollte.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200624/cbd3398d/attachment.html>

> On Jun 24, 2020, at 9:53 AM, Nicolai Hähnle via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> On Tue, Jun 23, 2020 at 2:40 PM Stella Laurenzo via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> We originally started it as a fork of the LLVM repository, but
transitioned to the MLIR standalone template, and we found it more productive to
iterate out of tree in this fashion, bumping to the latest LLVM version every
week or so as needed (note: the ability to exist out of tree for MLIR dependent
projects is actually quite good, and the more of us who do it, the better it
becomes).
> 
> How do you deal with the problem of using the "right" LLVM
version? As
> somebody who spends a significant amount of time on a project that is
> open-source but out-of-tree -- and for good reasons that mean we're
> unlikely to want to incubate in this fashion -- I find this to be a
> major problem.

I’m contributing to an external project based on MLIR (which should become
public soon).  That project is using the LLVM monorepo as a git submodule,
allowing us to update it and track at project-specific times.  It seems to work
well.

-Chris

On Wed, Jun 24, 2020 at 2:37 AM Chris Lattner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Jun 22, 2020, at 11:07 PM, Stella Laurenzo via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Per the recent (seeming) consensus regarding incubating new projects under
> the LLVM organization, I would like to trial the process by requesting to
> incubate mlir-npcomp <https://github.com/google/mlir-npcomp>. The
project
> is still quite young and has been primarily developed part time by myself
> and Sean Silva over the last ~2 months. We set it up following discussion
> of a Numpy/Scipy op set
> <https://llvm.discourse.group/t/numpy-scipy-op-set/768> and also in
> conjunction with a proving ground for high level dialects/transforms for
> lowering from "numpy-aligned" frontends (e.g. sometimes labeled
TCF/TCP).
>
>
> Awesome.
>
> So, as the first one walking through the door, what is the process we
> would like to follow? I'm happy to provide more information/discussion,
but
> I'd also be happy if with just an LGTM and someone creating an
> "mlir-npcomp" repository under the LLVM GitHub organization and
working the
> rest out as we go.
>
>
> I feel like we have general consensus that something like an incubator
> process is a good idea, but I think we should converge on adding a new
> section to the LLVM Developer Policy that describe what an incubator
> project is, and outline the requirements (following the policy etc).
>
> Could you help put together a draft of a patch?  If not, I can take a look
> at this this weekend.
>
Now that the proposal landed, we could resume the discussion on this
proposal?

I haven't been involved with mlir-npcomp directly so far, but I'm very
interested in seeing this moving forward: it has a lot of potential as an
incubator project!

-- 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200709/7943a0c9/attachment.html>