llvm dev - Jan 2021 - [RFC] Move py-mlir-release to new top-level repo in the LLVM org

Hi folks, I would like to propose that we create a new top-level repo in
the LLVM organization for organizing the Python MLIR Releases (both daily
and official numbered releases, whenever we are ready for such a thing) and
corresponding pushes to package repositories, etc.

I have prototyped such a release process in a personal repo:
https://github.com/stellaraccident/mlir-py-release

Additional development on that release process is currently blocked on more
work on the shared library organization in LLVM (discussed here
https://lists.llvm.org/pipermail/llvm-dev/2021-January/147567.html and
being worked on independently) but it is useful as is and a reasonable
starting point for further work.

I would propose that we just fork my current repo into a new one in the
LLVM organization and then take the necessary steps to get
credentials/permissions/secrets set up in the new context.

Some answers to questions that may come up:

   - *Why should this be a repo separate from llvm-project? *These kind of
   automation repos tend to have a lot of "garbage" commits that I
think is
   best if they do not pollute the main repo (and also don't face contention
   on automatic jobs bumping things, etc). They also tend to require special
   permissions and secrets that we will want to more tightly control. They
   also make use of other GitHub features that it seems like we would like not
   polluting the main development flow ("Releases" tab, Actions, etc).
Also,
   this is the kind of thing that tends to get revised en-masse periodically,
   and again, it would be good to not pollute the monorepo.
   - *Why not land this in llvm-zorg? *llvm-zorg claims to be for "LLVM
   Testing Infrastructure" and seems well scoped to that statement. What I
am
   managing above is periodic, automated release tooling based on open-source
   CI systems (currently GitHub Actions), which are fairly standardized across
   the Python releasing community, easy to set up, etc.
   - *What ultimately will the code in this repo do?*
      - Have periodic GitHub actions to select new LLVM revisions and
      schedule daily/snapshot releases.
      - Have manual actions for triggering official, numbered releases.
      - Facilities for building Python wheels for PyPi and house any
      additional metadata/automation needed for anaconda.
      - Builds releases for all supported operating systems (currently
      Linux/CentOS7/manylinux2014, MacOS, and Windows) and supported Python
      versions (currently 3.6, 3.7, 3.8, 3.9).
      - Publish release artifacts on the Releases tab for daily/snapshot
      releases.
      - Provide a stable reference point for downstream projects that
      extend MLIR-Python and need to produce version-matched artifacts of their
      own.
   - *Could this graduate to be more than "MLIR" python?* Maybe. I
chose
   the name because that is what I am focused on and didn't want to grab too
   much land. But there is nothing stopping this from becoming automation for
   general LLVM monorepo+incubator Python releasing.
   - *What if we don't do this?*
      - *Option A:* We keep running this in a private repo with the
      disclaimer that is currently at the top: "Note that this is a
prototype of
      a real MLIR release process being run by a member of the community. These
      are not official releases of the LLVM Foundation in any way, and they are
      likely only going to be useful to people actually working on LLVM/MLIR
      until we get things productionized." We would miss opportunities for
      convergence with other projects and would cause things to fragment.
      - *Option B: *We only publish Python bindings in official LLVM
      release packages, and only for the Python version they are built with. We
      don't release Python binaries through normal package management
channels.

Opinions?
- Stella
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210107/72221aa3/attachment.html>

Thanks for working on this Stella!

I'm wondering about the versioning: these packages that we'd publish
continuously are necessarily "unstable" to some extent. How do we
handle
the versioning and the alignment with LLVM releases?

Otherwise I'm fine with a new repo for this, in particular since it
involves GitHub action and these can't be isolated (in terms of
notification and such) other than at the repository boundary.

For the LLVM releases, would we need to branch in this repo as well?

Thanks,

-- 
Mehdi


On Thu, Jan 7, 2021 at 10:56 AM Stella Laurenzo via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi folks, I would like to propose that we create a new top-level repo in
> the LLVM organization for organizing the Python MLIR Releases (both daily
> and official numbered releases, whenever we are ready for such a thing) and
> corresponding pushes to package repositories, etc.
>
> I have prototyped such a release process in a personal repo:
> https://github.com/stellaraccident/mlir-py-release
>
> Additional development on that release process is currently blocked on
> more work on the shared library organization in LLVM (discussed here
> https://lists.llvm.org/pipermail/llvm-dev/2021-January/147567.html and
> being worked on independently) but it is useful as is and a reasonable
> starting point for further work.
>
> I would propose that we just fork my current repo into a new one in the
> LLVM organization and then take the necessary steps to get
> credentials/permissions/secrets set up in the new context.
>
> Some answers to questions that may come up:
>
>    - *Why should this be a repo separate from llvm-project? *These kind
>    of automation repos tend to have a lot of "garbage" commits
that I think is
>    best if they do not pollute the main repo (and also don't face
contention
>    on automatic jobs bumping things, etc). They also tend to require
special
>    permissions and secrets that we will want to more tightly control. They
>    also make use of other GitHub features that it seems like we would like
not
>    polluting the main development flow ("Releases" tab, Actions,
etc). Also,
>    this is the kind of thing that tends to get revised en-masse
periodically,
>    and again, it would be good to not pollute the monorepo.
>    - *Why not land this in llvm-zorg? *llvm-zorg claims to be for
"LLVM
>    Testing Infrastructure" and seems well scoped to that statement.
What I am
>    managing above is periodic, automated release tooling based on
open-source
>    CI systems (currently GitHub Actions), which are fairly standardized
across
>    the Python releasing community, easy to set up, etc.
>    - *What ultimately will the code in this repo do?*
>       - Have periodic GitHub actions to select new LLVM revisions and
>       schedule daily/snapshot releases.
>       - Have manual actions for triggering official, numbered releases.
>       - Facilities for building Python wheels for PyPi and house any
>       additional metadata/automation needed for anaconda.
>       - Builds releases for all supported operating systems (currently
>       Linux/CentOS7/manylinux2014, MacOS, and Windows) and supported Python
>       versions (currently 3.6, 3.7, 3.8, 3.9).
>       - Publish release artifacts on the Releases tab for daily/snapshot
>       releases.
>       - Provide a stable reference point for downstream projects that
>       extend MLIR-Python and need to produce version-matched artifacts of
their
>       own.
>    - *Could this graduate to be more than "MLIR" python?* Maybe.
I chose
>    the name because that is what I am focused on and didn't want to
grab too
>    much land. But there is nothing stopping this from becoming automation
for
>    general LLVM monorepo+incubator Python releasing.
>    - *What if we don't do this?*
>       - *Option A:* We keep running this in a private repo with the
>       disclaimer that is currently at the top: "Note that this is a
prototype of
>       a real MLIR release process being run by a member of the community.
These
>       are not official releases of the LLVM Foundation in any way, and they
are
>       likely only going to be useful to people actually working on
LLVM/MLIR
>       until we get things productionized." We would miss opportunities
for
>       convergence with other projects and would cause things to fragment.
>       - *Option B: *We only publish Python bindings in official LLVM
>       release packages, and only for the Python version they are built
with. We
>       don't release Python binaries through normal package management
channels.
>
> Opinions?
> - Stella
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210107/0711b9b0/attachment-0001.html>

On 1/7/21 10:55 AM, Stella Laurenzo via llvm-dev wrote:
> Hi folks, I would like to propose that we create a new top-level repo in 
> the LLVM organization for organizing the Python MLIR Releases (both 
> daily and official numbered releases, whenever we are ready for such a 
> thing) and corresponding pushes to package repositories, etc.
> 
For those of use that are unfamiliar, can you explain what the "Python 
MLIR Releases" are?
> I have prototyped such a release process in a personal repo: 
> https://github.com/stellaraccident/mlir-py-release
> 
> Additional development on that release process is currently blocked on 
> more work on the shared library organization in LLVM (discussed here 
> https://lists.llvm.org/pipermail/llvm-dev/2021-January/147567.html and 
> being worked on independently) but it is useful as is and a reasonable 
> starting point for further work.
> 
> I would propose that we just fork my current repo into a new one in the 
> LLVM organization and then take the necessary steps to get 
> credentials/permissions/secrets set up in the new context.
> 
> Some answers to questions that may come up:
> 
>   * *Why should this be a repo separate from llvm-project? *These kind
>     of automation repos tend to have a lot of "garbage" commits
that I
>     think is best if they do not pollute the main repo (and also don't
>     face contention on automatic jobs bumping things, etc). They also
>     tend to require special permissions and secrets that we will want to
>     more tightly control. They also make use of other GitHub features
>     that it seems like we would like not polluting the main development
>     flow ("Releases" tab, Actions, etc). Also, this is the kind
of thing
>     that tends to get revised en-masse periodically, and again, it would
>     be good to not pollute the monorepo.
There really aren't many files in this repo, do you anticipate it 
growing significantly?
>   * *Why not land this in llvm-zorg? *llvm-zorg claims to be for "LLVM
>     Testing Infrastructure" and seems well scoped to that statement.
>     What I am managing above is periodic, automated release tooling
>     based on open-source CI systems (currently GitHub Actions), which
>     are fairly standardized across the Python releasing community, easy
>     to set up, etc.
llvm-zorg also handles generating the websites.  My personal opinion is 
that it would be OK to try to do this in llvm-zorg, but you're probably 
better off asking Galina about that.  I guess the downside of using 
llvm-zorg is you don't get the releases tab.

Why did you choose to write the checkout_repo.py script in python rather 
than using the GitHub checkout action, or writing your own custom action?
>   * *What ultimately will the code in this repo do?*
>       o Have periodic GitHub actions to select new LLVM revisions and
>         schedule daily/snapshot releases.
Do you have any idea of much of the GitHub actions resources this would 
use?  e.g. how many hours per week per Operating System?
>       o Have manual actions for triggering official, numbered releases.
>       o Facilities for building Python wheels for PyPi and house any
>         additional metadata/automation needed for anaconda.
>       o Builds releases for all supported operating systems (currently
>         Linux/CentOS7/manylinux2014, MacOS, and Windows) and supported
>         Python versions (currently 3.6, 3.7, 3.8, 3.9).
>       o Publish release artifacts on the Releases tab for daily/snapshot
>         releases.
>       o Provide a stable reference point for downstream projects that
>         extend MLIR-Python and need to produce version-matched artifacts
>         of their own.
>   * *Could this graduate to be more than "MLIR" python?* Maybe. I
chose
>     the name because that is what I am focused on and didn't want to
>     grab too much land. But there is nothing stopping this from becoming
>     automation for general LLVM monorepo+incubator Python releasing.
I think it would be great to generalize this.  I would also like to 
automate parts the main LLVM release, and there seems to be some overlap 
with what you are doing.

-Tom
>   * *What if we don't do this?*
>       o *Option A:* We keep running this in a private repo with the
>         disclaimer that is currently at the top: "Note that this is a
>         prototype of a real MLIR release process being run by a member
>         of the community. These are not official releases of the LLVM
>         Foundation in any way, and they are likely only going to be
>         useful to people actually working on LLVM/MLIR until we get
>         things productionized." We would miss opportunities for
>         convergence with other projects and would cause things to fragment.
>       o *Option B: *We only publish Python bindings in official LLVM
>         release packages, and only for the Python version they are built
>         with. We don't release Python binaries through normal package
>         management channels.
> 
> Opinions?
> - Stella
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>