llvm dev - Dec 2019 - RFC: Using GitHub Actions for CI testing on the release/* branches

Please forgive the incorrect threading on this reply to Tom Stellard's RFC.
> I would like to start using GitHub Actions[1] for CI testing on the
release/*
> branches.  As far as I know we don't have any buildbots listening to
the
> release branches, and I think GitHub Actions are a good way for us to
> quickly bring-up some CI jobs there.
Personally, I feel that Tom's proof of concept, is more important than we
seem to be giving him credit for.

As of this writing, the Github actions system permits all comers, six hours
of CPU time per build platform.  Due to this long CPU allotment, AFAIK,
Github is one of the few CIs in town that lets anyone build and smoke llvm
for free.

Consider the workflow of someone who has never worked on llvm before.  They
will probably fork the monorepo on Github, in order to fix bugs or add a
feature or such.  At the moment they do this, they get a built-in workflow
that will sanity-check their builds on several important targets.  Zero
braining involved.

Giving Joe Programmer a CI system that magically smoke tests llvm, out of
the box, after he forks the repo, is a compelling reason to make something
like Tom's system a standard part of llvm master.

Concerns might be raised that llvm is "preferring" one CI system over
another.  Some thoughts about that.  First, because the monorepo's on
Github, you'll end up going to github.com anyway to get your first pull.
Second, nothing about Github actions precludes supporting other CI systems
in the future.

Thanks for your kind consideration.

-- 
---

John Byrd
Gigantic Software
2321 E 4th Street
Suite C #429
Santa Ana, CA  92705-3862
http://www.giganticsoftware.com
T: (949) 892-3526 F: (206) 309-0850
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191211/0da239ed/attachment.html>

I think everyone agrees that LLVM needs better CI, automatic pre-commit
testing of various platforms, etc. This is not rocket science, it is
standard practice for 2019 software engineering.

I think my concern is that LLVM could prove to be too big and require too
many resources for github's infrastructure. How many patches go into LLVM a
day, and how many build and test jobs does GitHub allow users to run
concurrently before being throttled?

You may have seen that Christian Kuhnel has been working on a pre-commit
testing bot that integrates with Phab:
https://reviews.llvm.org/p/merge_guards_bot/
https://github.com/google/llvm-premerge-checks/blob/master/docs/user_doc.md
I hope that ends up being the way forward and suits Tom's original release
testing goals.

On Wed, Dec 11, 2019 at 11:43 PM John Byrd via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Please forgive the incorrect threading on this reply to Tom Stellard's
RFC.
>
> > I would like to start using GitHub Actions[1] for CI testing on the
> release/*
> > branches.  As far as I know we don't have any buildbots listening
to the
> > release branches, and I think GitHub Actions are a good way for us to
> > quickly bring-up some CI jobs there.
>
> Personally, I feel that Tom's proof of concept, is more important than
we
> seem to be giving him credit for.
>
> As of this writing, the Github actions system permits all comers, six
> hours of CPU time per build platform.  Due to this long CPU allotment,
> AFAIK, Github is one of the few CIs in town that lets anyone build and
> smoke llvm for free.
>
> Consider the workflow of someone who has never worked on llvm before.
> They will probably fork the monorepo on Github, in order to fix bugs or add
> a feature or such.  At the moment they do this, they get a built-in
> workflow that will sanity-check their builds on several important targets.
> Zero braining involved.
>
> Giving Joe Programmer a CI system that magically smoke tests llvm, out of
> the box, after he forks the repo, is a compelling reason to make something
> like Tom's system a standard part of llvm master.
>
> Concerns might be raised that llvm is "preferring" one CI system
over
> another.  Some thoughts about that.  First, because the monorepo's on
> Github, you'll end up going to github.com anyway to get your first
pull.
> Second, nothing about Github actions precludes supporting other CI systems
> in the future.
>
> Thanks for your kind consideration.
>
> --
> ---
>
> John Byrd
> Gigantic Software
> 2321 E 4th Street
> Suite C #429
> Santa Ana, CA  92705-3862
> http://www.giganticsoftware.com
> T: (949) 892-3526 F: (206) 309-0850
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191212/344ccbd2/attachment.html>

>
> I think my concern is that LLVM could prove to be too big and require too
> many resources for github's infrastructure. How many patches go into
LLVM a
> day, and how many build and test jobs does GitHub allow users to run
> concurrently before being throttled?
>
A big project like LLVM, should almost never rely on the free tier of a CI
service. I am neither for nor against Github actions (I personally use them
for my project and they work out well, but the caching story needs more
work), but Github actions at least allows for self hosted runners that
allows us to move the load away from their infrastructure to ours.

https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners#about-self-hosted-runners


On Thu, Dec 12, 2019 at 1:53 PM Reid Kleckner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I think everyone agrees that LLVM needs better CI, automatic pre-commit
> testing of various platforms, etc. This is not rocket science, it is
> standard practice for 2019 software engineering.
>
> I think my concern is that LLVM could prove to be too big and require too
> many resources for github's infrastructure. How many patches go into
LLVM a
> day, and how many build and test jobs does GitHub allow users to run
> concurrently before being throttled?
>
> You may have seen that Christian Kuhnel has been working on a pre-commit
> testing bot that integrates with Phab:
> https://reviews.llvm.org/p/merge_guards_bot/
> https://github.com/google/llvm-premerge-checks/blob/master/docs/user_doc.md
> I hope that ends up being the way forward and suits Tom's original
release
> testing goals.
>
> On Wed, Dec 11, 2019 at 11:43 PM John Byrd via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Please forgive the incorrect threading on this reply to Tom
>> Stellard's RFC.
>>
>> > I would like to start using GitHub Actions[1] for CI testing on
the
>> release/*
>> > branches.  As far as I know we don't have any buildbots
listening to the
>> > release branches, and I think GitHub Actions are a good way for us
to
>> > quickly bring-up some CI jobs there.
>>
>> Personally, I feel that Tom's proof of concept, is more important
than we
>> seem to be giving him credit for.
>>
>> As of this writing, the Github actions system permits all comers, six
>> hours of CPU time per build platform.  Due to this long CPU allotment,
>> AFAIK, Github is one of the few CIs in town that lets anyone build and
>> smoke llvm for free.
>>
>> Consider the workflow of someone who has never worked on llvm before.
>> They will probably fork the monorepo on Github, in order to fix bugs or
add
>> a feature or such.  At the moment they do this, they get a built-in
>> workflow that will sanity-check their builds on several important
targets.
>> Zero braining involved.
>>
>> Giving Joe Programmer a CI system that magically smoke tests llvm, out
of
>> the box, after he forks the repo, is a compelling reason to make
something
>> like Tom's system a standard part of llvm master.
>>
>> Concerns might be raised that llvm is "preferring" one CI
system over
>> another.  Some thoughts about that.  First, because the monorepo's
on
>> Github, you'll end up going to github.com anyway to get your first
>> pull.  Second, nothing about Github actions precludes supporting other
CI
>> systems in the future.
>>
>> Thanks for your kind consideration.
>>
>> --
>> ---
>>
>> John Byrd
>> Gigantic Software
>> 2321 E 4th Street
>> Suite C #429
>> Santa Ana, CA  92705-3862
>> http://www.giganticsoftware.com
>> T: (949) 892-3526 F: (206) 309-0850
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191213/a7e4eac0/attachment.html>

On 12/12/2019 10:53 AM, Reid Kleckner via llvm-dev
wrote:
> I think everyone agrees that LLVM needs better CI, automatic pre-commit
testing of various platforms, etc. This is not rocket science, it is standard
practice for 2019 software engineering.
> 
> I think my concern is that LLVM could prove to be too big and require too
many resources for github's infrastructure. How many patches go into LLVM a
day, and how many build and test jobs does GitHub allow users to run
concurrently before being throttled?
> 
The key point from John's email is that the GitHub CI resources are easily
accessible for everyone.  So even if we aren't using it as a project for CI,
having working CI definitions available so people can pre-test changes in
their own forks is really valuable.

-Tom
> You may have seen that Christian Kuhnel has been working on a pre-commit
testing bot that integrates with Phab:
> https://reviews.llvm.org/p/merge_guards_bot/ 
> https://github.com/google/llvm-premerge-checks/blob/master/docs/user_doc.md
> I hope that ends up being the way forward and suits Tom's original
release testing goals.
> 
> On Wed, Dec 11, 2019 at 11:43 PM John Byrd via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> 
>     Please forgive the incorrect threading on this reply to Tom
Stellard's RFC.
> 
>     > I would like to start using GitHub Actions[1] for CI testing on
the release/*
>     > branches.  As far as I know we don't have any buildbots
listening to the
>     > release branches, and I think GitHub Actions are a good way for us
to
>     > quickly bring-up some CI jobs there.  
> 
>     Personally, I feel that Tom's proof of concept, is more important
than we seem to be giving him credit for.
> 
>     As of this writing, the Github actions system permits all comers, six
hours of CPU time per build platform.  Due to this long CPU allotment, AFAIK,
Github is one of the few CIs in town that lets anyone build and smoke llvm for
free.
> 
>     Consider the workflow of someone who has never worked on llvm before. 
They will probably fork the monorepo on Github, in order to fix bugs or add a
feature or such.  At the moment they do this, they get a built-in workflow that
will sanity-check their builds on several important targets.  Zero braining
involved.
> 
>     Giving Joe Programmer a CI system that magically smoke tests llvm, out
of the box, after he forks the repo, is a compelling reason to make something
like Tom's system a standard part of llvm master.
> 
>     Concerns might be raised that llvm is "preferring" one CI
system over another.  Some thoughts about that.  First, because the
monorepo's on Github, you'll end up going to github.com
<http://github.com> anyway to get your first pull.  Second, nothing about
Github actions precludes supporting other CI systems in the future.
> 
>     Thanks for your kind consideration.
> 
>     -- 
>     ---
> 
>     John Byrd
>     Gigantic Software
>     2321 E 4th Street
>     Suite C #429
>     Santa Ana, CA  92705-3862
>     http://www.giganticsoftware.com
>     T: (949) 892-3526 <tel:%28949%29%20892-3526> F: (206) 309-0850
<tel:%28206%29%20309-0850>
>     _______________________________________________
>     LLVM Developers mailing list
>     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>     https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

> I think my concern is that LLVM could prove to be too big and require too
many resources for github's infrastructure. How many patches go into LLVM a
day, and how many build and test jobs does GitHub allow users to run
concurrently before being throttled?

Please, no one confuse "what should be LLVM's official CIs" with
"what can
we do to make it easier for individuals and small companies to experiment
with small changes in their own forks."

Current usage limits for the free Github actions tier are:

- You can execute up to 20 workflows concurrently per repository.
- You can execute up to 1000 API requests in an hour across all actions
within a repository.
- Each job in a workflow can run for up to 6 hours of execution time.
- The number of jobs you can run concurrently across all repositories in
your account depends on your GitHub plan.
- You can have a maximum concurrent set of 5 maximum concurrent MacOS jobs.

So, this isn't enough juice to replace all the LLVM buildbots, but it is
enough to have CI on *your personal fork* of LLVM.  And having CI on
everyone's personal fork, makes it that much more likely that your
Phabricator patches will be correct the first time.

Best of all, if you don't like Github's CI, you can completely ignore it
and proceed with your current workflow.

On Thu, Dec 12, 2019 at 10:53 AM Reid Kleckner <rnk at google.com> wrote:
> I think everyone agrees that LLVM needs better CI, automatic pre-commit
> testing of various platforms, etc. This is not rocket science, it is
> standard practice for 2019 software engineering.
>
> I think my concern is that LLVM could prove to be too big and require too
> many resources for github's infrastructure. How many patches go into
LLVM a
> day, and how many build and test jobs does GitHub allow users to run
> concurrently before being throttled?
>
> You may have seen that Christian Kuhnel has been working on a pre-commit
> testing bot that integrates with Phab:
> https://reviews.llvm.org/p/merge_guards_bot/
> https://github.com/google/llvm-premerge-checks/blob/master/docs/user_doc.md
> I hope that ends up being the way forward and suits Tom's original
release
> testing goals.
>
> On Wed, Dec 11, 2019 at 11:43 PM John Byrd via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Please forgive the incorrect threading on this reply to Tom
>> Stellard's RFC.
>>
>> > I would like to start using GitHub Actions[1] for CI testing on
the
>> release/*
>> > branches.  As far as I know we don't have any buildbots
listening to the
>> > release branches, and I think GitHub Actions are a good way for us
to
>> > quickly bring-up some CI jobs there.
>>
>> Personally, I feel that Tom's proof of concept, is more important
than we
>> seem to be giving him credit for.
>>
>> As of this writing, the Github actions system permits all comers, six
>> hours of CPU time per build platform.  Due to this long CPU allotment,
>> AFAIK, Github is one of the few CIs in town that lets anyone build and
>> smoke llvm for free.
>>
>> Consider the workflow of someone who has never worked on llvm before.
>> They will probably fork the monorepo on Github, in order to fix bugs or
add
>> a feature or such.  At the moment they do this, they get a built-in
>> workflow that will sanity-check their builds on several important
targets.
>> Zero braining involved.
>>
>> Giving Joe Programmer a CI system that magically smoke tests llvm, out
of
>> the box, after he forks the repo, is a compelling reason to make
something
>> like Tom's system a standard part of llvm master.
>>
>> Concerns might be raised that llvm is "preferring" one CI
system over
>> another.  Some thoughts about that.  First, because the monorepo's
on
>> Github, you'll end up going to github.com anyway to get your first
>> pull.  Second, nothing about Github actions precludes supporting other
CI
>> systems in the future.
>>
>> Thanks for your kind consideration.
>>
>> --
>> ---
>>
>> John Byrd
>> Gigantic Software
>> 2321 E 4th Street
>> Suite C #429
>> Santa Ana, CA  92705-3862
>> http://www.giganticsoftware.com
>> T: (949) 892-3526 F: (206) 309-0850
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-- 
---

John Byrd
Gigantic Software
2321 E 4th Street
Suite C #429
Santa Ana, CA  92705-3862
http://www.giganticsoftware.com
T: (949) 892-3526 F: (206) 309-0850
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191213/02bfb643/attachment.html>