llvm dev - Dec 2021 - [Release-testers] RFC: New Automated Release Workflow (using Issues and Pull Requests)

Hi,

Here is a proposal for a new automated workflow for managing parts of the
release
process.  I've been experimenting with this over the past few releases and
now that we have migrated to GitHub issues, it would be possible for us to
implement this in the main repo.

The workflow is pretty straight forward, but it does use pull requests.  My
idea is to enable pull requests for only this automated workflow and not
for general development (i.e. We would still use Phabricator for code review).
Let me know what you think about this:


# Workflow

* On an existing issue or a newly created issue, a user who wants to backport
one or more commits to the release branch adds a comment:

/cherry-pick <commit_sha> <..>

* This starts a GitHub Action job that attempts to cherry-pick the commit(s)
to the current release branch.

* If the commit(s) can be cherry-picked cleanly, then the GitHub Action:
     * Pushes the result of the cherry-pick to a branch in the
       llvmbot/llvm-project repo called issue<n>, where n is the number of
the
       GitHub Issue that launched the Action.

     * Adds this comment on the issue: /branch
llvmbot/llvm-project/issue<n>

     * Creates a pull request from llvmbot/llvm-project/issue<n> to
       llvm/llvm-project/release/XX.x

     * Adds a comment on the issue: /pull-request #<n>
       where n is the number of the pull request.

* If the commit(s) can't be cherry-picked cleanly, then the GitHub Action
job adds
the release:cherry-pick-failed label to the issue and adds a comment:
"Failed to cherry-pick <commit_sha> <..>" along with a
link to the failing
Action.

* If a user has manually cherry-picked the fixes, resolved the conflicts, and
pushed the result to a branch on github, they can automatically create a pull
request by adding this comment to an issue: /branch
<user>/<repo>/<branch>

* Once a pull request has been created, this launches more GitHub Actions
to run pre-commit tests.

* Once the tests complete successfully and the changes have been approved
by the release manager, the pull request can me merged into the release branch.

* After the pull request is merged, a GitHub Action automatically closes the
associated issue.

Some Examples:

Cherry-pick success: https://github.com/tstellar/llvm-project/issues/729
Cherry-pick failure: https://github.com/tstellar/llvm-project/issues/730
Manual Branch comment: https://github.com/tstellar/llvm-project/issues/710


# Motivation

Why do this?  The goal is to make the release process more efficient and
transparent.
With this new workflow, users can get automatic and immediate feedback when a
commit
they want backported doesn't apply cleanly or introduces some test failures.
With
the current process, these kinds of issues are communicated by the release
manager,
and it can be days or even weeks before a problem is discovered and communicated
back
to the users.

Another advantage of this workflow is it introduces pre-commit CI to the release
branch,
which is important for the stability of the branch and the releases, but also
gives
the project an opportunity to experiment with new CI workflows in a way that
does not disrupt development on the main branch.

# Implementation

If this proposal is accepted, I would plan to implement this for the LLVM 14
release cycle based
on the following proof of concept that I have been testing for the last few
releases:

https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml

Thanks,
Tom

Hi,





On Fri, Dec 17, 2021 at 1:15 PM Tom Stellard via Release-testers <
release-testers at lists.llvm.org> wrote:
> Hi,
>
> Here is a proposal for a new automated workflow for managing parts of the
> release
> process.  I've been experimenting with this over the past few releases
and
> now that we have migrated to GitHub issues, it would be possible for us to
> implement this in the main repo.
>
> The workflow is pretty straight forward, but it does use pull requests.  My
> idea is to enable pull requests for only this automated workflow and not
> for general development (i.e. We would still use Phabricator for code
> review).
> Let me know what you think about this:
>
>
> # Workflow
>
> * On an existing issue or a newly created issue, a user who wants to
> backport
> one or more commits to the release branch adds a comment:
>
> /cherry-pick <commit_sha> <..>
>
> * This starts a GitHub Action job that attempts to cherry-pick the
> commit(s)
> to the current release branch.
>
> * If the commit(s) can be cherry-picked cleanly, then the GitHub Action:
>      * Pushes the result of the cherry-pick to a branch in the
>        llvmbot/llvm-project repo called issue<n>, where n is the
number of
> the
>        GitHub Issue that launched the Action.
>
>      * Adds this comment on the issue: /branch
> llvmbot/llvm-project/issue<n>
>
>      * Creates a pull request from llvmbot/llvm-project/issue<n> to
>        llvm/llvm-project/release/XX.x
>
>      * Adds a comment on the issue: /pull-request #<n>
>        where n is the number of the pull request.
>
> * If the commit(s) can't be cherry-picked cleanly, then the GitHub
Action
> job adds
> the release:cherry-pick-failed label to the issue and adds a comment:
> "Failed to cherry-pick <commit_sha> <..>" along with
a link to the failing
> Action.
>
> * If a user has manually cherry-picked the fixes, resolved the conflicts,
> and
> pushed the result to a branch on github, they can automatically create a
> pull
> request by adding this comment to an issue: /branch
<user>/<repo>/<branch>
>
> * Once a pull request has been created, this launches more GitHub Actions
> to run pre-commit tests.
>
> * Once the tests complete successfully and the changes have been approved
> by the release manager, the pull request can me merged into the release
> branch.
>
> * After the pull request is merged, a GitHub Action automatically closes
> the
> associated issue.
>
> Some Examples:
>
> Cherry-pick success: https://github.com/tstellar/llvm-project/issues/729
> Cherry-pick
> <https://github.com/tstellar/llvm-project/issues/729Cherry-pick>
failure:
> https://github.com/tstellar/llvm-project/issues/730
> Manual Branch comment: https://github.com/tstellar/llvm-project/issues/710


Since your workflow can trigger actions from comments in the issues, why do
you need pull-requests at all? Can't you trigger the pre-merge testing
action on the branch from the issue? Then you can "approve" it with a
*/merge
LGTM* comment in the issue directly and let the action merge it for example.


>
>
>
> # Motivation
>
> Why do this?  The goal is to make the release process more efficient and
> transparent.
> With this new workflow, users can get automatic and immediate feedback
> when a commit
> they want backported doesn't apply cleanly or introduces some test
> failures.  With
> the current process, these kinds of issues are communicated by the release
> manager,
> and it can be days or even weeks before a problem is discovered and
> communicated back
> to the users.
>
> Another advantage of this workflow is it introduces pre-commit CI to the
> release branch,
> which is important for the stability of the branch and the releases, but
> also gives
> the project an opportunity to experiment with new CI workflows in a way
> that
> does not disrupt development on the main branch.
>
> # Implementation
>
> If this proposal is accepted, I would plan to implement this for the LLVM
> 14 release cycle based
> on the following proof of concept that I have been testing for the last
> few releases:
>
>
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
>
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
>
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
>
> Thanks,
> Tom
>
> _______________________________________________
> Release-testers mailing list
> Release-testers at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/release-testers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211217/2d819180/attachment.html>

Sounds pretty good to me - wouldn't mind knowing more about/a good summary
of the effects of this on project/repo/etc notifications that Mehdi's
mentioning. (be good to have a write up of the expected impact/options to
then discuss - from the thread so far I understand some general/high level
concerns, but it's not clear to me exactly how it plays out)

On Fri, Dec 17, 2021 at 1:15 PM Tom Stellard via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi,
>
> Here is a proposal for a new automated workflow for managing parts of the
> release
> process.  I've been experimenting with this over the past few releases
and
> now that we have migrated to GitHub issues, it would be possible for us to
> implement this in the main repo.
>
> The workflow is pretty straight forward, but it does use pull requests.  My
> idea is to enable pull requests for only this automated workflow and not
> for general development (i.e. We would still use Phabricator for code
> review).
> Let me know what you think about this:
>
>
> # Workflow
>
> * On an existing issue or a newly created issue, a user who wants to
> backport
> one or more commits to the release branch adds a comment:
>
> /cherry-pick <commit_sha> <..>
>
> * This starts a GitHub Action job that attempts to cherry-pick the
> commit(s)
> to the current release branch.
>
> * If the commit(s) can be cherry-picked cleanly, then the GitHub Action:
>      * Pushes the result of the cherry-pick to a branch in the
>        llvmbot/llvm-project repo called issue<n>, where n is the
number of
> the
>        GitHub Issue that launched the Action.
>
>      * Adds this comment on the issue: /branch
> llvmbot/llvm-project/issue<n>
>
>      * Creates a pull request from llvmbot/llvm-project/issue<n> to
>        llvm/llvm-project/release/XX.x
>
>      * Adds a comment on the issue: /pull-request #<n>
>        where n is the number of the pull request.
>
> * If the commit(s) can't be cherry-picked cleanly, then the GitHub
Action
> job adds
> the release:cherry-pick-failed label to the issue and adds a comment:
> "Failed to cherry-pick <commit_sha> <..>" along with
a link to the failing
> Action.
>
> * If a user has manually cherry-picked the fixes, resolved the conflicts,
> and
> pushed the result to a branch on github, they can automatically create a
> pull
> request by adding this comment to an issue: /branch
<user>/<repo>/<branch>
>
> * Once a pull request has been created, this launches more GitHub Actions
> to run pre-commit tests.
>
> * Once the tests complete successfully and the changes have been approved
> by the release manager, the pull request can me merged into the release
> branch.
>
> * After the pull request is merged, a GitHub Action automatically closes
> the
> associated issue.
>
> Some Examples:
>
> Cherry-pick success: https://github.com/tstellar/llvm-project/issues/729
> Cherry-pick
> <https://github.com/tstellar/llvm-project/issues/729Cherry-pick>
failure:
> https://github.com/tstellar/llvm-project/issues/730
> Manual Branch comment: https://github.com/tstellar/llvm-project/issues/710
>
>
> # Motivation
>
> Why do this?  The goal is to make the release process more efficient and
> transparent.
> With this new workflow, users can get automatic and immediate feedback
> when a commit
> they want backported doesn't apply cleanly or introduces some test
> failures.  With
> the current process, these kinds of issues are communicated by the release
> manager,
> and it can be days or even weeks before a problem is discovered and
> communicated back
> to the users.
>
> Another advantage of this workflow is it introduces pre-commit CI to the
> release branch,
> which is important for the stability of the branch and the releases, but
> also gives
> the project an opportunity to experiment with new CI workflows in a way
> that
> does not disrupt development on the main branch.
>
> # Implementation
>
> If this proposal is accepted, I would plan to implement this for the LLVM
> 14 release cycle based
> on the following proof of concept that I have been testing for the last
> few releases:
>
>
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
>
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
>
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
>
> Thanks,
> Tom
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211217/0cbfe248/attachment.html>

On Fri, 17 Dec 2021 at 21:15, Tom Stellard via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> * On an existing issue or a newly created issue, a user who wants to
> backport
> one or more commits to the release branch adds a comment:
>
> /cherry-pick <commit_sha> <..>
>
Hi Tom,

Would this be *any* user or users with certain permissions in the repo
(like code owners, release managers)?

Ignoring malicious action, *any* user creating a cherry-pick at any time,
may create confusion if two users are trying to pick changes that need
multiple (non-sequential) commits each.

An alternative would be to build a branch off the release branch (ex.
"release-x.y.z-$username") and pick the commits on that branch, run
the
pre-commit tests, and then merge to the release branch if it's all green.

Because the merge is atomic, and the tests passed on the alternative
branch, the probability of the release branch breaking is lower.

Of course, interaction between the users' branches can still break, and
well, further tests that are not present in the pre-commit tests, can also.

But with atomic merges of cherry-picks in a linear sequence will also make
it easier to bisect in case anything goes wrong with the release candidate.

If only a subset of users can merge, then they'd do one at a time and this
problem wouldn't be a big issue and we'd avoid a complicated
infrastructure
setup.

Does that make sense?

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211223/a3c99b57/attachment.html>

On 12/17/21 13:15, Tom Stellard wrote:
> Hi,
> 
> Here is a proposal for a new automated workflow for managing parts of the
release
> process.  I've been experimenting with this over the past few releases
and
> now that we have migrated to GitHub issues, it would be possible for us to
> implement this in the main repo.
> 
> The workflow is pretty straight forward, but it does use pull requests.  My
> idea is to enable pull requests for only this automated workflow and not
> for general development (i.e. We would still use Phabricator for code
review).
> Let me know what you think about this:
> 
Hi,

Thanks for the feedback on this.  I've posted a patch to implement this
proposal:
https://reviews.llvm.org/D117386.  The only change is that the pull requests
will
not be in the llvm/llvm-project repo, but will instead be in
llvmbot/llvm-project.
This will reduce the number of notifications and avoid confusion about whether
or
not we are using pull requests for this project.

-Tom
> 
> # Workflow
> 
> * On an existing issue or a newly created issue, a user who wants to
backport
> one or more commits to the release branch adds a comment:
> 
> /cherry-pick <commit_sha> <..>
> 
> * This starts a GitHub Action job that attempts to cherry-pick the
commit(s)
> to the current release branch.
> 
> * If the commit(s) can be cherry-picked cleanly, then the GitHub Action:
>      * Pushes the result of the cherry-pick to a branch in the
>        llvmbot/llvm-project repo called issue<n>, where n is the
number of the
>        GitHub Issue that launched the Action.
> 
>      * Adds this comment on the issue: /branch
llvmbot/llvm-project/issue<n>
> 
>      * Creates a pull request from llvmbot/llvm-project/issue<n> to
>        llvm/llvm-project/release/XX.x
> 
>      * Adds a comment on the issue: /pull-request #<n>
>        where n is the number of the pull request.
> 
> * If the commit(s) can't be cherry-picked cleanly, then the GitHub
Action job adds
> the release:cherry-pick-failed label to the issue and adds a comment:
> "Failed to cherry-pick <commit_sha> <..>" along with
a link to the failing
> Action.
> 
> * If a user has manually cherry-picked the fixes, resolved the conflicts,
and
> pushed the result to a branch on github, they can automatically create a
pull
> request by adding this comment to an issue: /branch
<user>/<repo>/<branch>
> 
> * Once a pull request has been created, this launches more GitHub Actions
> to run pre-commit tests.
> 
> * Once the tests complete successfully and the changes have been approved
> by the release manager, the pull request can me merged into the release
branch.
> 
> * After the pull request is merged, a GitHub Action automatically closes
the
> associated issue.
> 
> Some Examples:
> 
> Cherry-pick success: https://github.com/tstellar/llvm-project/issues/729
> Cherry-pick failure: https://github.com/tstellar/llvm-project/issues/730
> Manual Branch comment: https://github.com/tstellar/llvm-project/issues/710
> 
> 
> # Motivation
> 
> Why do this?  The goal is to make the release process more efficient and
transparent.
> With this new workflow, users can get automatic and immediate feedback when
a commit
> they want backported doesn't apply cleanly or introduces some test
failures.  With
> the current process, these kinds of issues are communicated by the release
manager,
> and it can be days or even weeks before a problem is discovered and
communicated back
> to the users.
> 
> Another advantage of this workflow is it introduces pre-commit CI to the
release branch,
> which is important for the stability of the branch and the releases, but
also gives
> the project an opportunity to experiment with new CI workflows in a way
that
> does not disrupt development on the main branch.
> 
> # Implementation
> 
> If this proposal is accepted, I would plan to implement this for the LLVM
14 release cycle based
> on the following proof of concept that I have been testing for the last few
releases:
> 
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
> 
> Thanks,
> Tom