llvm dev - Dec 2021 - RFC: New Automated Release Workflow (using Issues and Pull Requests)

On Fri, Dec 17, 2021 at 6:38 PM Tom Stellard <tstellar at redhat.com>
wrote:
> On 12/17/21 16:47, David Blaikie wrote:
> > Sounds pretty good to me - wouldn't mind knowing more about/a good
> summary of the effects of this on project/repo/etc notifications that
> Mehdi's mentioning. (be good to have a write up of the expected
> impact/options to then discuss - from the thread so far I understand some
> general/high level concerns, but it's not clear to me exactly how it
plays
> out)
> >
>
> The impact is really going to depend on the person and what notification
> preferences they
> have/want.  If you are already watching the repo with the default
> settings, then you probably
> won't notice much of a difference given the current volume of
> notifications.
>
I think I'm on the default settings - which does currently mean a
notification for every issue update, which is a lot. Given that
llvm-bugs at email.llvm.org has been re-enabled, sending mail only on issue
creation, I & others might opt back in to that behavior by disabling the
baseline "notify on everything" to "notify only on issues I'm
mentioned in".

I guess currently the only email that github is generating is one email per
issue update. We don't have any pull requests, so there aren't any
emails
for that, yeah?

So this new strategy might add a few more back-and-forth on each
cherrypick issue (for those using llvm-bugs & disabling general issue
notifications, this will not be relevant to them - there won't be more
issues created, just more comments on existing issues). But there will be
some more emails generated related to the pull requests themselves, I
guess? So each cherrypick goes from 2 emails to llvm-bugs (the issue
creation and closure) to, how many? 4 (2 for llvm-bugs and I guess at least
2 for the pull request - one to make the request and one to close it -
maybe a couple more status ones along the way?)

> If people want to give their notification preferences, I can try to look
> at how
> this change will impact specific configurations.
>
@Mehdi AMINI <joker.eph at gmail.com> - are there particular scenarios you
have in mind that'd be good to work through?

>
> -Tom
>
>
> > On Fri, Dec 17, 2021 at 1:15 PM Tom Stellard via llvm-dev <
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
> >
> >     Hi,
> >
> >     Here is a proposal for a new automated workflow for managing parts
> of the release
> >     process.  I've been experimenting with this over the past few
> releases and
> >     now that we have migrated to GitHub issues, it would be possible
for
> us to
> >     implement this in the main repo.
> >
> >     The workflow is pretty straight forward, but it does use pull
> requests.  My
> >     idea is to enable pull requests for only this automated workflow
and
> not
> >     for general development (i.e. We would still use Phabricator for
> code review).
> >     Let me know what you think about this:
> >
> >
> >     # Workflow
> >
> >     * On an existing issue or a newly created issue, a user who wants
to
> backport
> >     one or more commits to the release branch adds a comment:
> >
> >     /cherry-pick <commit_sha> <..>
> >
> >     * This starts a GitHub Action job that attempts to cherry-pick the
> commit(s)
> >     to the current release branch.
> >
> >     * If the commit(s) can be cherry-picked cleanly, then the GitHub
> Action:
> >           * Pushes the result of the cherry-pick to a branch in the
> >             llvmbot/llvm-project repo called issue<n>, where n
is the
> number of the
> >             GitHub Issue that launched the Action.
> >
> >           * Adds this comment on the issue: /branch
> llvmbot/llvm-project/issue<n>
> >
> >           * Creates a pull request from
llvmbot/llvm-project/issue<n> to
> >             llvm/llvm-project/release/XX.x
> >
> >           * Adds a comment on the issue: /pull-request #<n>
> >             where n is the number of the pull request.
> >
> >     * If the commit(s) can't be cherry-picked cleanly, then the
GitHub
> Action job adds
> >     the release:cherry-pick-failed label to the issue and adds a
comment:
> >     "Failed to cherry-pick <commit_sha> <..>"
along with a link to the
> failing
> >     Action.
> >
> >     * If a user has manually cherry-picked the fixes, resolved the
> conflicts, and
> >     pushed the result to a branch on github, they can automatically
> create a pull
> >     request by adding this comment to an issue: /branch
> <user>/<repo>/<branch>
> >
> >     * Once a pull request has been created, this launches more GitHub
> Actions
> >     to run pre-commit tests.
> >
> >     * Once the tests complete successfully and the changes have been
> approved
> >     by the release manager, the pull request can me merged into the
> release branch.
> >
> >     * After the pull request is merged, a GitHub Action automatically
> closes the
> >     associated issue.
> >
> >     Some Examples:
> >
> >     Cherry-pick success:
> https://github.com/tstellar/llvm-project/issues/729
> >     Cherry-pick <
> https://github.com/tstellar/llvm-project/issues/729Cherry-pick> failure:
> https://github.com/tstellar/llvm-project/issues/730 <
> https://github.com/tstellar/llvm-project/issues/730>
> >     Manual Branch comment:
> https://github.com/tstellar/llvm-project/issues/710 <
> https://github.com/tstellar/llvm-project/issues/710>
> >
> >
> >     # Motivation
> >
> >     Why do this?  The goal is to make the release process more
efficient
> and transparent.
> >     With this new workflow, users can get automatic and immediate
> feedback when a commit
> >     they want backported doesn't apply cleanly or introduces some
test
> failures.  With
> >     the current process, these kinds of issues are communicated by the
> release manager,
> >     and it can be days or even weeks before a problem is discovered
and
> communicated back
> >     to the users.
> >
> >     Another advantage of this workflow is it introduces pre-commit CI
to
> the release branch,
> >     which is important for the stability of the branch and the
releases,
> but also gives
> >     the project an opportunity to experiment with new CI workflows in
a
> way that
> >     does not disrupt development on the main branch.
> >
> >     # Implementation
> >
> >     If this proposal is accepted, I would plan to implement this for
the
> LLVM 14 release cycle based
> >     on the following proof of concept that I have been testing for the
> last few releases:
> >
> >
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
> <
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
> >
> >
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
> <
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
> >
> >
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
> <
>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
> >
> >
> >     Thanks,
> >     Tom
> >
> >     _______________________________________________
> >     LLVM Developers mailing list
> >     llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
> >     https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211218/f7f61225/attachment.html>

On Sat, Dec 18, 2021 at 3:04 PM David Blaikie <dblaikie at gmail.com>
wrote:
>
>
> On Fri, Dec 17, 2021 at 6:38 PM Tom Stellard <tstellar at redhat.com>
wrote:
>
>> On 12/17/21 16:47, David Blaikie wrote:
>> > Sounds pretty good to me - wouldn't mind knowing more about/a
good
>> summary of the effects of this on project/repo/etc notifications that
>> Mehdi's mentioning. (be good to have a write up of the expected
>> impact/options to then discuss - from the thread so far I understand
some
>> general/high level concerns, but it's not clear to me exactly how
it plays
>> out)
>> >
>>
>> The impact is really going to depend on the person and what
notification
>> preferences they
>> have/want.  If you are already watching the repo with the default
>> settings, then you probably
>> won't notice much of a difference given the current volume of
>> notifications.
>>
>
> I think I'm on the default settings - which does currently mean a
> notification for every issue update, which is a lot. Given that
> llvm-bugs at email.llvm.org has been re-enabled, sending mail only on issue
> creation, I & others might opt back in to that behavior by disabling
the
> baseline "notify on everything" to "notify only on issues
I'm mentioned in".
>
> I guess currently the only email that github is generating is one email
> per issue update. We don't have any pull requests, so there aren't
any
> emails for that, yeah?
>
> So this new strategy might add a few more back-and-forth on each
> cherrypick issue (for those using llvm-bugs & disabling general issue
> notifications, this will not be relevant to them - there won't be more
> issues created, just more comments on existing issues). But there will be
> some more emails generated related to the pull requests themselves, I
> guess? So each cherrypick goes from 2 emails to llvm-bugs (the issue
> creation and closure) to, how many? 4 (2 for llvm-bugs and I guess at least
> 2 for the pull request - one to make the request and one to close it -
> maybe a couple more status ones along the way?)
>
>
>> If people want to give their notification preferences, I can try to
look
>> at how
>> this change will impact specific configurations.
>>
>
> @Mehdi AMINI <joker.eph at gmail.com> - are there particular
scenarios you
> have in mind that'd be good to work through?
>
It's more the unknown that I'm cautious about here. In particular the
recent experience with the issues' notifications lend me to think we likely
should have more thorough consideration to how things are fitting as a
whole before adding more things on GitHub. You mentioned above how emails
are issued for these new pull-requests, so I'd like to see a better
characterization of this (quantity and example of such emails) as well as
some documentation (what are the developer workflows, how to manage
notifications configuration on GitHub, and client-side email filtering,
etc.).

Using a separate repo if possible alleviates these constraints by sharding
things early, which seems to fit better into how GitHub organizes things: a
repo seems a really a unit of itself, and everything seems "scoped" at
this
level (notification, permissions, etc.).

-- 
Mehdi



>
>
>>
>> -Tom
>>
>>
>> > On Fri, Dec 17, 2021 at 1:15 PM Tom Stellard via llvm-dev <
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>> >
>> >     Hi,
>> >
>> >     Here is a proposal for a new automated workflow for managing
parts
>> of the release
>> >     process.  I've been experimenting with this over the past
few
>> releases and
>> >     now that we have migrated to GitHub issues, it would be
possible
>> for us to
>> >     implement this in the main repo.
>> >
>> >     The workflow is pretty straight forward, but it does use pull
>> requests.  My
>> >     idea is to enable pull requests for only this automated
workflow
>> and not
>> >     for general development (i.e. We would still use Phabricator
for
>> code review).
>> >     Let me know what you think about this:
>> >
>> >
>> >     # Workflow
>> >
>> >     * On an existing issue or a newly created issue, a user who
wants
>> to backport
>> >     one or more commits to the release branch adds a comment:
>> >
>> >     /cherry-pick <commit_sha> <..>
>> >
>> >     * This starts a GitHub Action job that attempts to cherry-pick
the
>> commit(s)
>> >     to the current release branch.
>> >
>> >     * If the commit(s) can be cherry-picked cleanly, then the
GitHub
>> Action:
>> >           * Pushes the result of the cherry-pick to a branch in
the
>> >             llvmbot/llvm-project repo called issue<n>, where
n is the
>> number of the
>> >             GitHub Issue that launched the Action.
>> >
>> >           * Adds this comment on the issue: /branch
>> llvmbot/llvm-project/issue<n>
>> >
>> >           * Creates a pull request from
llvmbot/llvm-project/issue<n> to
>> >             llvm/llvm-project/release/XX.x
>> >
>> >           * Adds a comment on the issue: /pull-request #<n>
>> >             where n is the number of the pull request.
>> >
>> >     * If the commit(s) can't be cherry-picked cleanly, then
the GitHub
>> Action job adds
>> >     the release:cherry-pick-failed label to the issue and adds a
>> comment:
>> >     "Failed to cherry-pick <commit_sha>
<..>" along with a link to the
>> failing
>> >     Action.
>> >
>> >     * If a user has manually cherry-picked the fixes, resolved the
>> conflicts, and
>> >     pushed the result to a branch on github, they can
automatically
>> create a pull
>> >     request by adding this comment to an issue: /branch
>> <user>/<repo>/<branch>
>> >
>> >     * Once a pull request has been created, this launches more
GitHub
>> Actions
>> >     to run pre-commit tests.
>> >
>> >     * Once the tests complete successfully and the changes have
been
>> approved
>> >     by the release manager, the pull request can me merged into
the
>> release branch.
>> >
>> >     * After the pull request is merged, a GitHub Action
automatically
>> closes the
>> >     associated issue.
>> >
>> >     Some Examples:
>> >
>> >     Cherry-pick success:
>> https://github.com/tstellar/llvm-project/issues/729
>> >     Cherry-pick <
>> https://github.com/tstellar/llvm-project/issues/729Cherry-pick>
failure:
>> https://github.com/tstellar/llvm-project/issues/730 <
>> https://github.com/tstellar/llvm-project/issues/730>
>> >     Manual Branch comment:
>> https://github.com/tstellar/llvm-project/issues/710 <
>> https://github.com/tstellar/llvm-project/issues/710>
>> >
>> >
>> >     # Motivation
>> >
>> >     Why do this?  The goal is to make the release process more
>> efficient and transparent.
>> >     With this new workflow, users can get automatic and immediate
>> feedback when a commit
>> >     they want backported doesn't apply cleanly or introduces
some test
>> failures.  With
>> >     the current process, these kinds of issues are communicated by
the
>> release manager,
>> >     and it can be days or even weeks before a problem is
discovered and
>> communicated back
>> >     to the users.
>> >
>> >     Another advantage of this workflow is it introduces pre-commit
CI
>> to the release branch,
>> >     which is important for the stability of the branch and the
>> releases, but also gives
>> >     the project an opportunity to experiment with new CI workflows
in a
>> way that
>> >     does not disrupt development on the main branch.
>> >
>> >     # Implementation
>> >
>> >     If this proposal is accepted, I would plan to implement this
for
>> the LLVM 14 release cycle based
>> >     on the following proof of concept that I have been testing for
the
>> last few releases:
>> >
>> >
>>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
>> <
>>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
>> >
>> >
>>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
>> <
>>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
>> >
>> >
>>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
>> <
>>
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
>> >
>> >
>> >     Thanks,
>> >     Tom
>> >
>> >     _______________________________________________
>> >     LLVM Developers mailing list
>> >     llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>> >     https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211218/23691c6d/attachment.html>

On 12/18/21 15:04, David Blaikie wrote:
> 
> 
> On Fri, Dec 17, 2021 at 6:38 PM Tom Stellard <tstellar at redhat.com
<mailto:tstellar at redhat.com>> wrote:
> 
>     On 12/17/21 16:47, David Blaikie wrote:
>      > Sounds pretty good to me - wouldn't mind knowing more about/a
good summary of the effects of this on project/repo/etc notifications that
Mehdi's mentioning. (be good to have a write up of the expected
impact/options to then discuss - from the thread so far I understand some
general/high level concerns, but it's not clear to me exactly how it plays
out)
>      >
> 
>     The impact is really going to depend on the person and what
notification preferences they
>     have/want.  If you are already watching the repo with the default
settings, then you probably
>     won't notice much of a difference given the current volume of
notifications.
> 
> 
> I think I'm on the default settings - which does currently mean a
notification for every issue update, which is a lot. Given that llvm-bugs at
email.llvm.org <mailto:llvm-bugs at email.llvm.org> has been re-enabled,
sending mail only on issue creation, I & others might opt back in to that
behavior by disabling the baseline "notify on everything" to
"notify only on issues I'm mentioned in".
> 
> I guess currently the only email that github is generating is one email per
issue update. We don't have any pull requests, so there aren't any
emails for that, yeah?
> 
> So this new strategy might add a few more back-and-forth on each
cherrypick issue (for those using llvm-bugs & disabling general issue
notifications, this will not be relevant to them - there won't be more
issues created, just more comments on existing issues). But there will be some
more emails generated related to the pull requests themselves, I guess? So each
cherrypick goes from 2 emails to llvm-bugs (the issue creation and closure) to,
how many? 4 (2 for llvm-bugs and I guess at least 2 for the pull request - one
to make the request and one to close it - maybe a couple more status ones along
the way?)
> 
I think the number of net new comments on issues will be very minimal or none at
all.  The
automated comments that are created by this process are replacing comments that
I'm already making
manually.

So 2+ for pull requests is probably a good estimate.  I still need to figure out
how many notifications
get generated for Actions with the default settings.

--Tom
   
>     If people want to give their notification preferences, I can try to
look at how
>     this change will impact specific configurations.
> 
> 
> @Mehdi AMINI <mailto:joker.eph at gmail.com> - are there particular
scenarios you have in mind that'd be good to work through?
> 
> 
>     -Tom
> 
> 
>      > On Fri, Dec 17, 2021 at 1:15 PM Tom Stellard via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
<mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>>> wrote:
>      >
>      >     Hi,
>      >
>      >     Here is a proposal for a new automated workflow for managing
parts of the release
>      >     process.  I've been experimenting with this over the past
few releases and
>      >     now that we have migrated to GitHub issues, it would be
possible for us to
>      >     implement this in the main repo.
>      >
>      >     The workflow is pretty straight forward, but it does use pull
requests.  My
>      >     idea is to enable pull requests for only this automated
workflow and not
>      >     for general development (i.e. We would still use Phabricator
for code review).
>      >     Let me know what you think about this:
>      >
>      >
>      >     # Workflow
>      >
>      >     * On an existing issue or a newly created issue, a user who
wants to backport
>      >     one or more commits to the release branch adds a comment:
>      >
>      >     /cherry-pick <commit_sha> <..>
>      >
>      >     * This starts a GitHub Action job that attempts to
cherry-pick the commit(s)
>      >     to the current release branch.
>      >
>      >     * If the commit(s) can be cherry-picked cleanly, then the
GitHub Action:
>      >           * Pushes the result of the cherry-pick to a branch in
the
>      >             llvmbot/llvm-project repo called issue<n>,
where n is the number of the
>      >             GitHub Issue that launched the Action.
>      >
>      >           * Adds this comment on the issue: /branch
llvmbot/llvm-project/issue<n>
>      >
>      >           * Creates a pull request from
llvmbot/llvm-project/issue<n> to
>      >             llvm/llvm-project/release/XX.x
>      >
>      >           * Adds a comment on the issue: /pull-request #<n>
>      >             where n is the number of the pull request.
>      >
>      >     * If the commit(s) can't be cherry-picked cleanly, then
the GitHub Action job adds
>      >     the release:cherry-pick-failed label to the issue and adds a
comment:
>      >     "Failed to cherry-pick <commit_sha>
<..>" along with a link to the failing
>      >     Action.
>      >
>      >     * If a user has manually cherry-picked the fixes, resolved
the conflicts, and
>      >     pushed the result to a branch on github, they can
automatically create a pull
>      >     request by adding this comment to an issue: /branch
<user>/<repo>/<branch>
>      >
>      >     * Once a pull request has been created, this launches more
GitHub Actions
>      >     to run pre-commit tests.
>      >
>      >     * Once the tests complete successfully and the changes have
been approved
>      >     by the release manager, the pull request can me merged into
the release branch.
>      >
>      >     * After the pull request is merged, a GitHub Action
automatically closes the
>      >     associated issue.
>      >
>      >     Some Examples:
>      >
>      >     Cherry-pick success:
https://github.com/tstellar/llvm-project/issues/729
<https://github.com/tstellar/llvm-project/issues/729>
>      >     Cherry-pick
<https://github.com/tstellar/llvm-project/issues/729Cherry-pick
<https://github.com/tstellar/llvm-project/issues/729Cherry-pick>>
failure: https://github.com/tstellar/llvm-project/issues/730
<https://github.com/tstellar/llvm-project/issues/730>
<https://github.com/tstellar/llvm-project/issues/730
<https://github.com/tstellar/llvm-project/issues/730>>
>      >     Manual Branch comment:
https://github.com/tstellar/llvm-project/issues/710
<https://github.com/tstellar/llvm-project/issues/710>
<https://github.com/tstellar/llvm-project/issues/710
<https://github.com/tstellar/llvm-project/issues/710>>
>      >
>      >
>      >     # Motivation
>      >
>      >     Why do this?  The goal is to make the release process more
efficient and transparent.
>      >     With this new workflow, users can get automatic and immediate
feedback when a commit
>      >     they want backported doesn't apply cleanly or introduces
some test failures.  With
>      >     the current process, these kinds of issues are communicated
by the release manager,
>      >     and it can be days or even weeks before a problem is
discovered and communicated back
>      >     to the users.
>      >
>      >     Another advantage of this workflow is it introduces
pre-commit CI to the release branch,
>      >     which is important for the stability of the branch and the
releases, but also gives
>      >     the project an opportunity to experiment with new CI
workflows in a way that
>      >     does not disrupt development on the main branch.
>      >
>      >     # Implementation
>      >
>      >     If this proposal is accepted, I would plan to implement this
for the LLVM 14 release cycle based
>      >     on the following proof of concept that I have been testing
for the last few releases:
>      >
>      >
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
<https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml>
<https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
<https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml>>
>      >
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
<https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml>
<https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
<https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml>>
>      >
https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
<https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml>
<https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
<https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml>>
>      >
>      >     Thanks,
>      >     Tom
>      >
>      >     _______________________________________________
>      >     LLVM Developers mailing list
>      > llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>>
>      > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>>
>      >
>