llvm dev - Nov 2019 - RFC: Using GitHub Actions for CI testing on the release/* branches

Hi,

I would like to start using GitHub Actions[1] for CI testing on the release/*
branches.  As far as I know we don't have any buildbots listening to the
release branches, and I think GitHub Actions are a good way for us to quickly
bring-up some CI jobs there.

My proposal is to start by adding two post-commit CI jobs to the release/9.x
branch.
One for building and testing (ninja checka-all) llvm/clang/lld on Linux,
Windows, and Mac, and another for detecting ABI changes since the 9.0.0 release.

I have already implemented these two CI jobs in my llvm-project fork on
GitHub[2][3],
but in order to get these running in the main repository, I would need to:

1. Create a new repository in the LLVM organization called 'actions' for
storing some custom
builds steps for our CI jobs (see [4]).
2. Commit yaml CI definitions to the .github/workflows directory in the
release/9.x
branch.

In the future, I would also like to add buil and tests jobs for other
sub-projects
once I am able to get those working.

In addition to being used for post-commit testing, having these CI definitions
in the
main tree will make it easier for me (or anyone) to do pre-commit testing for
the
release branch in a personal fork.  It will also allow me to experiment with
some new
workflows to help make managing the releases much easier.

I think this will be a good way to test Actions in a low traffic environment to
see if they are something we would want to use for CI on the master branch.

Given that we are close to the end of the 9.0.1 cycle, unless there are any
strong objections, I would like to get this enabled by Mon Nov 18, to maximize
its
usefulness.  Let me know what you think.

Thanks,
Tom

[1] https://github.com/features/actions
[2]
https://github.com/tstellar/llvm-project/commit/952d80e8509ecc95797b2ddbf1af40abad2dcf4e/checks?check_suite_id=305765621
[3]
https://github.com/tstellar/llvm-project/commit/6d74f1b81632ef081dffa1e0c0434f47d4954423/checks?check_suite_id=303074176
[4] https://github.com/tstellar/actions

Not having given it deep thought/analysis, nor understanding much of the
GIT infrastructure here, but: Sounds good to me, for whatever that's worth
:)

On Mon, Nov 11, 2019 at 4:32 PM Tom Stellard via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi,
>
> I would like to start using GitHub Actions[1] for CI testing on the
> release/*
> branches.  As far as I know we don't have any buildbots listening to
the
> release branches, and I think GitHub Actions are a good way for us to
> quickly
> bring-up some CI jobs there.
>
> My proposal is to start by adding two post-commit CI jobs to the
> release/9.x branch.
> One for building and testing (ninja checka-all) llvm/clang/lld on Linux,
> Windows, and Mac, and another for detecting ABI changes since the 9.0.0
> release.
>
> I have already implemented these two CI jobs in my llvm-project fork on
> GitHub[2][3],
> but in order to get these running in the main repository, I would need to:
>
> 1. Create a new repository in the LLVM organization called
'actions' for
> storing some custom
> builds steps for our CI jobs (see [4]).
> 2. Commit yaml CI definitions to the .github/workflows directory in the
> release/9.x
> branch.
>
> In the future, I would also like to add buil and tests jobs for other
> sub-projects
> once I am able to get those working.
>
> In addition to being used for post-commit testing, having these CI
> definitions in the
> main tree will make it easier for me (or anyone) to do pre-commit testing
> for the
> release branch in a personal fork.  It will also allow me to experiment
> with some new
> workflows to help make managing the releases much easier.
>
> I think this will be a good way to test Actions in a low traffic
> environment to
> see if they are something we would want to use for CI on the master branch.
>
> Given that we are close to the end of the 9.0.1 cycle, unless there are any
> strong objections, I would like to get this enabled by Mon Nov 18, to
> maximize its
> usefulness.  Let me know what you think.
>
> Thanks,
> Tom
>
> [1] https://github.com/features/actions
> [2]
>
https://github.com/tstellar/llvm-project/commit/952d80e8509ecc95797b2ddbf1af40abad2dcf4e/checks?check_suite_id=305765621
> [3]
>
https://github.com/tstellar/llvm-project/commit/6d74f1b81632ef081dffa1e0c0434f47d4954423/checks?check_suite_id=303074176
> [4] https://github.com/tstellar/actions
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191111/55c0c1bd/attachment.html>

Although I've not had any Github Action experience, and despite being one
who has raised objections to other parts of adopting github features, this
sounds like a sensible idea. It seems odd to me that we don't already have
some form of CI for our releases, so anything that improves on that is
great.

James

On Tue, 12 Nov 2019 at 00:34, David Blaikie via lldb-dev <
lldb-dev at lists.llvm.org> wrote:
> Not having given it deep thought/analysis, nor understanding much of the
> GIT infrastructure here, but: Sounds good to me, for whatever that's
worth
> :)
>
> On Mon, Nov 11, 2019 at 4:32 PM Tom Stellard via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi,
>>
>> I would like to start using GitHub Actions[1] for CI testing on the
>> release/*
>> branches.  As far as I know we don't have any buildbots listening
to the
>> release branches, and I think GitHub Actions are a good way for us to
>> quickly
>> bring-up some CI jobs there.
>>
>> My proposal is to start by adding two post-commit CI jobs to the
>> release/9.x branch.
>> One for building and testing (ninja checka-all) llvm/clang/lld on
Linux,
>> Windows, and Mac, and another for detecting ABI changes since the 9.0.0
>> release.
>>
>> I have already implemented these two CI jobs in my llvm-project fork on
>> GitHub[2][3],
>> but in order to get these running in the main repository, I would need
to:
>>
>> 1. Create a new repository in the LLVM organization called
'actions' for
>> storing some custom
>> builds steps for our CI jobs (see [4]).
>> 2. Commit yaml CI definitions to the .github/workflows directory in the
>> release/9.x
>> branch.
>>
>> In the future, I would also like to add buil and tests jobs for other
>> sub-projects
>> once I am able to get those working.
>>
>> In addition to being used for post-commit testing, having these CI
>> definitions in the
>> main tree will make it easier for me (or anyone) to do pre-commit
testing
>> for the
>> release branch in a personal fork.  It will also allow me to experiment
>> with some new
>> workflows to help make managing the releases much easier.
>>
>> I think this will be a good way to test Actions in a low traffic
>> environment to
>> see if they are something we would want to use for CI on the master
>> branch.
>>
>> Given that we are close to the end of the 9.0.1 cycle, unless there are
>> any
>> strong objections, I would like to get this enabled by Mon Nov 18, to
>> maximize its
>> usefulness.  Let me know what you think.
>>
>> Thanks,
>> Tom
>>
>> [1] https://github.com/features/actions
>> [2]
>>
https://github.com/tstellar/llvm-project/commit/952d80e8509ecc95797b2ddbf1af40abad2dcf4e/checks?check_suite_id=305765621
>> [3]
>>
https://github.com/tstellar/llvm-project/commit/6d74f1b81632ef081dffa1e0c0434f47d4954423/checks?check_suite_id=303074176
>> [4] https://github.com/tstellar/actions
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191112/52114dfb/attachment.html>

On Tue, Nov 12, 2019 at 1:32 AM Tom Stellard via lldb-dev
<lldb-dev at lists.llvm.org> wrote:
>
> Hi,
>
> I would like to start using GitHub Actions[1] for CI testing on the
release/*
> branches.  As far as I know we don't have any buildbots listening to
the
> release branches, and I think GitHub Actions are a good way for us to
quickly
> bring-up some CI jobs there.
>
> My proposal is to start by adding two post-commit CI jobs to the
release/9.x branch.
> One for building and testing (ninja checka-all) llvm/clang/lld on Linux,
> Windows, and Mac, and another for detecting ABI changes since the 9.0.0
release.
>
> I have already implemented these two CI jobs in my llvm-project fork on
GitHub[2][3],
> but in order to get these running in the main repository, I would need to:
>
> 1. Create a new repository in the LLVM organization called
'actions' for storing some custom
> builds steps for our CI jobs (see [4]).
> 2. Commit yaml CI definitions to the .github/workflows directory in the
release/9.x
> branch.
>
> In the future, I would also like to add buil and tests jobs for other
sub-projects
> once I am able to get those working.
>
> In addition to being used for post-commit testing, having these CI
definitions in the
> main tree will make it easier for me (or anyone) to do pre-commit testing
for the
> release branch in a personal fork.  It will also allow me to experiment
with some new
> workflows to help make managing the releases much easier.
>
> I think this will be a good way to test Actions in a low traffic
environment to
> see if they are something we would want to use for CI on the master branch.
>
> Given that we are close to the end of the 9.0.1 cycle, unless there are any
> strong objections, I would like to get this enabled by Mon Nov 18, to
maximize its
> usefulness.  Let me know what you think.
>
> Thanks,
> Tom
>
> [1] https://github.com/features/actions
> [2]
https://github.com/tstellar/llvm-project/commit/952d80e8509ecc95797b2ddbf1af40abad2dcf4e/checks?check_suite_id=305765621
> [3]
https://github.com/tstellar/llvm-project/commit/6d74f1b81632ef081dffa1e0c0434f47d4954423/checks?check_suite_id=303074176
> [4] https://github.com/tstellar/actions
Sounds great to me!

Hey Tom,

That sounds really useful. Would it be possible to include LLDB as
well? We have a subset of tests (unit & lit) that can be run without
Python/SWIG by passing LLDB_DISABLE_PYTHON=ON to CMake.

Thanks,
Jonas

On Tue, Nov 12, 2019 at 2:35 AM Hans Wennborg via cfe-dev
<cfe-dev at lists.llvm.org> wrote:
>
> On Tue, Nov 12, 2019 at 1:32 AM Tom Stellard via lldb-dev
> <lldb-dev at lists.llvm.org> wrote:
> >
> > Hi,
> >
> > I would like to start using GitHub Actions[1] for CI testing on the
release/*
> > branches.  As far as I know we don't have any buildbots listening
to the
> > release branches, and I think GitHub Actions are a good way for us to
quickly
> > bring-up some CI jobs there.
> >
> > My proposal is to start by adding two post-commit CI jobs to the
release/9.x branch.
> > One for building and testing (ninja checka-all) llvm/clang/lld on
Linux,
> > Windows, and Mac, and another for detecting ABI changes since the
9.0.0 release.
> >
> > I have already implemented these two CI jobs in my llvm-project fork
on GitHub[2][3],
> > but in order to get these running in the main repository, I would need
to:
> >
> > 1. Create a new repository in the LLVM organization called
'actions' for storing some custom
> > builds steps for our CI jobs (see [4]).
> > 2. Commit yaml CI definitions to the .github/workflows directory in
the release/9.x
> > branch.
> >
> > In the future, I would also like to add buil and tests jobs for other
sub-projects
> > once I am able to get those working.
> >
> > In addition to being used for post-commit testing, having these CI
definitions in the
> > main tree will make it easier for me (or anyone) to do pre-commit
testing for the
> > release branch in a personal fork.  It will also allow me to
experiment with some new
> > workflows to help make managing the releases much easier.
> >
> > I think this will be a good way to test Actions in a low traffic
environment to
> > see if they are something we would want to use for CI on the master
branch.
> >
> > Given that we are close to the end of the 9.0.1 cycle, unless there
are any
> > strong objections, I would like to get this enabled by Mon Nov 18, to
maximize its
> > usefulness.  Let me know what you think.
> >
> > Thanks,
> > Tom
> >
> > [1] https://github.com/features/actions
> > [2]
https://github.com/tstellar/llvm-project/commit/952d80e8509ecc95797b2ddbf1af40abad2dcf4e/checks?check_suite_id=305765621
> > [3]
https://github.com/tstellar/llvm-project/commit/6d74f1b81632ef081dffa1e0c0434f47d4954423/checks?check_suite_id=303074176
> > [4] https://github.com/tstellar/actions
>
> Sounds great to me!
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

+1, I'll be really curious to know how this works out.

Philip

On 11/11/19 4:32 PM, Tom Stellard via cfe-dev wrote:
> Hi,
>
> I would like to start using GitHub Actions[1] for CI testing on the
release/*
> branches.  As far as I know we don't have any buildbots listening to
the
> release branches, and I think GitHub Actions are a good way for us to
quickly
> bring-up some CI jobs there.
>
> My proposal is to start by adding two post-commit CI jobs to the
release/9.x branch.
> One for building and testing (ninja checka-all) llvm/clang/lld on Linux,
> Windows, and Mac, and another for detecting ABI changes since the 9.0.0
release.
>
> I have already implemented these two CI jobs in my llvm-project fork on
GitHub[2][3],
> but in order to get these running in the main repository, I would need to:
>
> 1. Create a new repository in the LLVM organization called
'actions' for storing some custom
> builds steps for our CI jobs (see [4]).
> 2. Commit yaml CI definitions to the .github/workflows directory in the
release/9.x
> branch.
>
> In the future, I would also like to add buil and tests jobs for other
sub-projects
> once I am able to get those working.
>
> In addition to being used for post-commit testing, having these CI
definitions in the
> main tree will make it easier for me (or anyone) to do pre-commit testing
for the
> release branch in a personal fork.  It will also allow me to experiment
with some new
> workflows to help make managing the releases much easier.
>
> I think this will be a good way to test Actions in a low traffic
environment to
> see if they are something we would want to use for CI on the master branch.
>
> Given that we are close to the end of the 9.0.1 cycle, unless there are any
> strong objections, I would like to get this enabled by Mon Nov 18, to
maximize its
> usefulness.  Let me know what you think.
>
> Thanks,
> Tom
>
> [1] https://github.com/features/actions
> [2]
https://github.com/tstellar/llvm-project/commit/952d80e8509ecc95797b2ddbf1af40abad2dcf4e/checks?check_suite_id=305765621
> [3]
https://github.com/tstellar/llvm-project/commit/6d74f1b81632ef081dffa1e0c0434f47d4954423/checks?check_suite_id=303074176
> [4] https://github.com/tstellar/actions
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

On 11/11/2019 04:32 PM, Tom Stellard wrote:
> Hi,
> 
> I would like to start using GitHub Actions[1] for CI testing on the
release/*
> branches.  As far as I know we don't have any buildbots listening to
the
> release branches, and I think GitHub Actions are a good way for us to
quickly
> bring-up some CI jobs there.
> 
> My proposal is to start by adding two post-commit CI jobs to the
release/9.x branch.
> One for building and testing (ninja checka-all) llvm/clang/lld on Linux,
> Windows, and Mac, and another for detecting ABI changes since the 9.0.0
release.
> 
> I have already implemented these two CI jobs in my llvm-project fork on
GitHub[2][3],
> but in order to get these running in the main repository, I would need to:
> 
> 1. Create a new repository in the LLVM organization called
'actions' for storing some custom
> builds steps for our CI jobs (see [4]).
> 2. Commit yaml CI definitions to the .github/workflows directory in the
release/9.x
> branch.
> 
> In the future, I would also like to add buil and tests jobs for other
sub-projects
> once I am able to get those working.
> 
> In addition to being used for post-commit testing, having these CI
definitions in the
> main tree will make it easier for me (or anyone) to do pre-commit testing
for the
> release branch in a personal fork.  It will also allow me to experiment
with some new
> workflows to help make managing the releases much easier.
> 
> I think this will be a good way to test Actions in a low traffic
environment to
> see if they are something we would want to use for CI on the master branch.
> 
> Given that we are close to the end of the 9.0.1 cycle, unless there are any
> strong objections, I would like to get this enabled by Mon Nov 18, to
maximize its
> usefulness.  Let me know what you think.
> 
The CI tests are running now on the stable branch.  Here is the
first run [1].  Passing commits[2] now have a green check mark in
the web UI that you can click on to view the results.

These tests have already caught 1 bug that did not appear in my local
builds, so they are proving to be very useful.

Now that this is in place, I'm going to start looking into some more
automation
around backporting fixes to the stable branch.  I think we could do something
like
filing backport requests as a GitHub issues and then have that issue
automatically
trigger a pull request and do some pre-commit testing.  I'm still trying to
learn
what is technically possible with GitHub and if anyone has any ideas or
suggestions,
let me know.  I'm hoping to have something in place for the 10.0.1 release
cycle.

Thanks,
Tom

[1]
https://github.com/llvm/llvm-project/commit/6c86aa62d502ac2df58e207d4792544cc96d79bd/checks?check_suite_id=316648311
[2]
https://github.com/llvm/llvm-project/commit/6c86aa62d502ac2df58e207d4792544cc96d79bd
> Thanks,
> Tom
> 
> [1] https://github.com/features/actions
> [2]
https://github.com/tstellar/llvm-project/commit/952d80e8509ecc95797b2ddbf1af40abad2dcf4e/checks?check_suite_id=305765621
> [3]
https://github.com/tstellar/llvm-project/commit/6d74f1b81632ef081dffa1e0c0434f47d4954423/checks?check_suite_id=303074176
> [4] https://github.com/tstellar/actions
>

> Hi Tom,
>
> This sounds very interesting! +1 from me.
> What does this imply for the release testers?
> Would it be possible / desirable for us to add self-hosted runners for
> other architectures? I'm assuming GitHub only provides x86, right? I
> totally missed any details about that in their docs, but it seems to
> be implied here [1]. I'm not saying we should go all-possible-arches
> from the start, we can definitely try it out only for x86 this time
> around if you think that would be the best approach.
>
Nothing changes for release testers at this point.  The main goal here is
to get some post-commit testing so we can catch issues in the branch early.

Longer term I think have self-hosted runners would be great, because it would
allow us to fully automate the testing and uploading we do when a release gets
tagged.

You are correct that GitHub only provides x86 machines, however, they don't
have enough disk space (14GB) to be able to run the test-release.sh script, so
adding x86 self-hosted builders with more disk space would still be very useful.

-Tom
 
>
> [1]
https://github.blog/2019-11-05-self-hosted-runners-for-github-actions-is-now-in-beta/
>
> Thanks,
> Diana

Hi Tom,

First of: I'm a big fan of adding more automatic tests to find bugs. Great
work!

On the other hand, I think we should consider creating some sort of test
strategy for the LLVM project:

   - What tests do we expect users to run before uploading patches?
   - What tests do we expect users to run before merging?
   - What tests do we run after merging?
   - What failures must be fixed, what failures can be ignored?
   - What do we check for on the build bots?
   - What do we check for on the release branches?
   - What do we check for on the pre-merge tests?
   - Which CI tools do we want to use (github, Jenkins, bulidbot, ...)?
   - What about running clang-tidy and clang-format?
   - What CMake configurations should we check? (Release/Debug, assertions,
   ...)
   - Do we want to run tests with Sanitizers?
   - Which of these systems do we expect users to monitor?

I suppose it would be good to have a document that summarizes all of this
so that we

   1. do not test the same thing twice and
   2. do not miss any important checks.

Does something like this exist?
Is anyone working on this?

Best,
Christian

On Wed, Nov 20, 2019 at 7:26 AM Tom Stellard via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> > Hi Tom,
> >
> > This sounds very interesting! +1 from me.
> > What does this imply for the release testers?
> > Would it be possible / desirable for us to add self-hosted runners for
> > other architectures? I'm assuming GitHub only provides x86, right?
I
> > totally missed any details about that in their docs, but it seems to
> > be implied here [1]. I'm not saying we should go
all-possible-arches
> > from the start, we can definitely try it out only for x86 this time
> > around if you think that would be the best approach.
> >
>
> Nothing changes for release testers at this point.  The main goal here is
> to get some post-commit testing so we can catch issues in the branch early.
>
> Longer term I think have self-hosted runners would be great, because it
> would
> allow us to fully automate the testing and uploading we do when a release
> gets
> tagged.
>
> You are correct that GitHub only provides x86 machines, however, they
don't
> have enough disk space (14GB) to be able to run the test-release.sh
> script, so
> adding x86 self-hosted builders with more disk space would still be very
> useful.
>
> -Tom
>
> >
> > [1]
>
https://github.blog/2019-11-05-self-hosted-runners-for-github-actions-is-now-in-beta/
> >
> > Thanks,
> > Diana
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

-- 
Best,
Christian
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191121/02172ea9/attachment.html>