llvm dev - Oct 2019 - [Openmp-dev] [cfe-dev] RFC: End-to-end testing

Renato Golin <rengolin at gmail.com> writes:
> On Thu, 17 Oct 2019 at 18:10, David Greene <greened at obbligato.org>
wrote:
>> From other discussion, it sounds like at least some people are open to
>> asm tests under clang.  I think that should be fine.  But there are
>> probably other kinds of end-to-end tests that should not live under
>> clang.
>
> That is my position as well. Some tests, especially similar to
> existing ones, are fine.
Ok.
> But if we really want to do do complete tests and stress more than
> just grepping a couple of instructions, should be in a better suited
> place.
That's probably true.
>> How often would such tests be run as part of test-suite?
>
> Every time the TS is executed. Some good work has been put on it to
> run with CMake etc, so it should be trivial to to run that before
> commits, but it *does* require more than just "make check-all".
I have been viewing test-suite as a kind of second-level/backup testing
that catches things not flagged by "make check-all."  Is that a
reasonable interpretation?  I was hoping to get some end-to-end tests
under "make check-all" because that's easier for developers to run
in
their workflows.
> On CI, a number of bots run those as often as they can, non-stop.
>
>> Honestly, it's not really clear to me exactly which bots cover
what, how
>> often they run and so on.  Is there a document somewhere describing the
>> setup?
>
> Not really. The main Buildbot page is a mess and the system is very
> old. There is a round table at the dev meeting to discuss the path
> forward.
Yeah, I saw that.  I will see if I can attend.  There are some conflicts
we have to work out here.
> This is not the first, though. We have been discussing this for a
> number of years, but getting people / companies to commit to testing
> is not trivial.
Is there a proposal somewhere of what companies would be expected to do?
It's difficult for us engineers to talk to management without a concrete
set of expectations, resource requirements, etc.
> I created a page for the Arm bots (after many incarnations, it ended
> up here: http://ex40-01.tcwglab.linaro.org/) to make that simpler. But
> that wouldn't scale, nor it fixes the real problems.
Nice!  That's much better.  Yes, it won't scale but it's much
clearer
about what is being run.

                         -David

On Fri, 18 Oct 2019 at 16:30, David Greene <greened at obbligato.org>
wrote:
> I have been viewing test-suite as a kind of second-level/backup testing
> that catches things not flagged by "make check-all."  Is that a
> reasonable interpretation?  I was hoping to get some end-to-end tests
> under "make check-all" because that's easier for developers
to run in
> their workflows.
It is a common understanding, which makes the test-suite less useful,
but that's not really relevant.

No one needs to the test-suite as part of their development processes,
because we have bots for that.

If you have decent piece-wise tests in Clang/LLVM, you really don't
need end-to-end tests in Clang/LLVM, because the test-suite will run
on bots and you will be told if you break them.

Most errors will be picked up by piece-wise tests, and the minority
where e2e make a difference can be reactive, rather than pro-active
fixing.
> Is there a proposal somewhere of what companies would be expected to do?
> It's difficult for us engineers to talk to management without a
concrete
> set of expectations, resource requirements, etc.
There were talks about upgrading Buildbot (the service), moving to
Jenkins or something else (Travis?). None of them have the majority of
the community behind, and that's the main problem.

IIRC, the arguments (definitely outdated, probably wrong) were:

Buildbot:
 - Pros: we already have a big infra based on it, it's passable, an
upgrade could ameliorate a lot of problems without creating many new
ones.
 - Cons: it's old tech and requires extensive coding to make it work

Jenkins:
 - Pros: Most companies use that already, it's more modetn than
Apple's GreenBot is based on that, lots of plugins and expertise in
the community
 - Cons: It requires Java running on the client, which not all targets
like. Alternatives require a separate server to run as a slave and
connect to targets.

Travis:
 - Pros: It's natively compatible with Github (is is still the case?)
and it could be the easiest to connect with our new repo for CI
 - Cons: less expertise, I guess, and other things I don't really know.

> Nice!  That's much better.  Yes, it won't scale but it's much
clearer
> about what is being run.
Perhaps adding a new column as to what components we test in each one
would be nice.

--renato

On Fri, Oct 18, 2019 at 8:53 AM Renato Golin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Fri, 18 Oct 2019 at 16:30, David Greene <greened at obbligato.org>
wrote:
> > I have been viewing test-suite as a kind of second-level/backup
testing
> > that catches things not flagged by "make check-all."  Is
that a
> > reasonable interpretation?  I was hoping to get some end-to-end tests
> > under "make check-all" because that's easier for
developers to run in
> > their workflows.
>
> It is a common understanding, which makes the test-suite less useful,
> but that's not really relevant.
>
> No one needs to the test-suite as part of their development processes,
> because we have bots for that.
>
> If you have decent piece-wise tests in Clang/LLVM, you really don't
> need end-to-end tests in Clang/LLVM, because the test-suite will run
> on bots and you will be told if you break them.
>
> Most errors will be picked up by piece-wise tests, and the minority
> where e2e make a difference can be reactive, rather than pro-active
> fixing.
>
> > Is there a proposal somewhere of what companies would be expected to
do?
> > It's difficult for us engineers to talk to management without a
concrete
> > set of expectations, resource requirements, etc.
>
> There were talks about upgrading Buildbot (the service), moving to
> Jenkins or something else (Travis?). None of them have the majority of
> the community behind, and that's the main problem.
>
> IIRC, the arguments (definitely outdated, probably wrong) were:
>
> Buildbot:
>  - Pros: we already have a big infra based on it, it's passable, an
> upgrade could ameliorate a lot of problems without creating many new
> ones.
>  - Cons: it's old tech and requires extensive coding to make it work
>
> Jenkins:
>  - Pros: Most companies use that already, it's more modetn than
> Apple's GreenBot is based on that, lots of plugins and expertise in
> the community
>  - Cons: It requires Java running on the client, which not all targets
> like. Alternatives require a separate server to run as a slave and
> connect to targets.
>
> Travis:
>  - Pros: It's natively compatible with Github (is is still the case?)
> and it could be the easiest to connect with our new repo for CI
>  - Cons: less expertise, I guess, and other things I don't really know.
>
This is good summary! I just want to put some new options like BuildKite
(we've been experimenting with this on MLIR: https://buildkite.com/mlir/ ),
and GitHub Actions (Still in Beta: https://github.com/features/actions , I
haven't had time to play with it). There are also paying options like
CircleCI and TeamCity, but they seem out-of-scope for us I think?
It'd be interesting to collect a more complete list of CI tools
free-for-open-source :)

-- 
Mehdi


>
>
> > Nice!  That's much better.  Yes, it won't scale but it's
much clearer
> > about what is being run.
>
> Perhaps adding a new column as to what components we test in each one
> would be nice.
>
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191019/3814c063/attachment.html>