llvm dev - Jul 2020 - [RFC] Compiled regression tests.

Am Di., 30. Juni 2020 um 15:58 Uhr schrieb Chris Lattner <clattner at
nondot.org>:
> One of the major and important pieces of the LLVM design is how its testing
infrastructure works.  The choice to use a small number of tools (llc, opt, etc)
is important for multiple reasons:
>
> 1) Link time of executables is a significant problem, particularly in large
scale builds.
You can use dynamic linking. Unfortunately, an LLVM dynamic library is
not (yet?) supported on Windows, so we need a static linking fallback.
If this is the only issue, I'd work on a solution (at least with
gtest) that works on Windows as well.

> 2) This encourages investing in testing tools (see, e.g. the recent
improvements to FileCheck etc)
Google test also is a testing tool that is worth investing into, such
as more expressive ASSERT macros as in the RFC.

I think it makes it even easier to start new tools that begin being
used within a single test only when it does not seem worth adding a
new executable or FileCheck option.

> 3) It reduces/factors the number of users of API surface area, making it
easier to do large scale refactoring etc.
FileCheck makes string output (including llvm::dbgs()) part of the
interface, which becomes not only harder to change, but also harder to
extend (new lines/tags to not match existing CHECK lines).

In contrast, refactoring using tools such as
clang-refactor/clang-format is not different then refactoring the
source of LLVM itself.

In the interest of downstream users, having a well checked API surface
should be more important.

> 4) It encourages the development of textual interfaces to libraries, which
aids with understandability and helps reinforce stronger interfaces (llvm-mc is
one example of this, LLVM IR text syntax is another).
While I don't disagree with good textual presentation, I think these
should be designed for human understandability and consistency, not
for machine processing.

> 5) Depending on the details, this can make the build dependence graph more
serialized.
I don't see why this would be that case.

> Unit tests are very widely used across the industry, and it is certainly
true that they are fully general and more flexible.  This makes them attractive,
but it is a trap.  I’d really rather we don’t go down this route, and maintain
the approach of only using unit tests for very low level things like apis in ADT
etc.
Note that we already have unittests for non low-level APIs such as
passes (VPlan, LICM, Unrolling, ...)

Can you elaborate on what the trap is?


Michael

On Jun 30, 2020, at 10:42 PM, Michael Kruse <llvmdev at meinersbur.de>
wrote:
> 
> Am Di., 30. Juni 2020 um 15:58 Uhr schrieb Chris Lattner <clattner at
nondot.org>:
>> One of the major and important pieces of the LLVM design is how its
testing infrastructure works.  The choice to use a small number of tools (llc,
opt, etc) is important for multiple reasons:
>> 
>> 1) Link time of executables is a significant problem, particularly in
large scale builds.
> 
> You can use dynamic linking. Unfortunately, an LLVM dynamic library is
> not (yet?) supported on Windows, so we need a static linking fallback.
> If this is the only issue, I'd work on a solution (at least with
> gtest) that works on Windows as well.
Agreed - this is something I investigated early on in LLVM’s evolution (and one
of the reasons the libraries are all dynamically linkable).

The problem with this is that you shift time from the static linker to the
dynamic linker - specifically making the launch time of the tests really slow. 
This is even on Linux, which has a pretty efficient loader.
>> 2) This encourages investing in testing tools (see, e.g. the recent
improvements to FileCheck etc)
> 
> Google test also is a testing tool that is worth investing into, such
> as more expressive ASSERT macros as in the RFC.
Yes, I’m very familiar with it, we use gtest in LLVM already.
>> 3) It reduces/factors the number of users of API surface area, making
it easier to do large scale refactoring etc.
> 
> FileCheck makes string output (including llvm::dbgs()) part of the
> interface, which becomes not only harder to change, but also harder to
> extend (new lines/tags to not match existing CHECK lines).
Yes, agreed.  The reason this is ok is that a change to a DBGS string breaks
tests of (e.g.) the pass that it lives in.  API changes affect everything
touching the API, which unit tests generally expose much more broadly.
>> Unit tests are very widely used across the industry, and it is
certainly true that they are fully general and more flexible.  This makes them
attractive, but it is a trap.  I’d really rather we don’t go down this route,
and maintain the approach of only using unit tests for very low level things
like apis in ADT etc.
> 
> Note that we already have unittests for non low-level APIs such as
> passes (VPlan, LICM, Unrolling, ...)
> 
> Can you elaborate on what the trap is?
I am also opposed to those - I think they have exactly the same problems
described above, and we should invest in better testing infra to support them.

-Chris

Am Mi., 1. Juli 2020 um 15:07 Uhr schrieb Chris Lattner <clattner at
nondot.org>:
>
> On Jun 30, 2020, at 10:42 PM, Michael Kruse <llvmdev at
meinersbur.de> wrote:
> >
> > Am Di., 30. Juni 2020 um 15:58 Uhr schrieb Chris Lattner <clattner
at nondot.org>:
> >> One of the major and important pieces of the LLVM design is how
its testing infrastructure works.  The choice to use a small number of tools
(llc, opt, etc) is important for multiple reasons:
> >>
> >> 1) Link time of executables is a significant problem, particularly
in large scale builds.
> >
> > You can use dynamic linking. Unfortunately, an LLVM dynamic library is
> > not (yet?) supported on Windows, so we need a static linking fallback.
> > If this is the only issue, I'd work on a solution (at least with
> > gtest) that works on Windows as well.
>
> Agreed - this is something I investigated early on in LLVM’s evolution (and
one of the reasons the libraries are all dynamically linkable).
>
> The problem with this is that you shift time from the static linker to the
dynamic linker - specifically making the launch time of the tests really slow. 
This is even on Linux, which has a pretty efficient loader.
>
> >> 2) This encourages investing in testing tools (see, e.g. the
recent improvements to FileCheck etc)
> >
> > Google test also is a testing tool that is worth investing into, such
> > as more expressive ASSERT macros as in the RFC.
>
> Yes, I’m very familiar with it, we use gtest in LLVM already.
>
> >> 3) It reduces/factors the number of users of API surface area,
making it easier to do large scale refactoring etc.
> >
> > FileCheck makes string output (including llvm::dbgs()) part of the
> > interface, which becomes not only harder to change, but also harder to
> > extend (new lines/tags to not match existing CHECK lines).
>
> Yes, agreed.  The reason this is ok is that a change to a DBGS string
breaks tests of (e.g.) the pass that it lives in.  API changes affect everything
touching the API, which unit tests generally expose much more broadly.
>
> >> Unit tests are very widely used across the industry, and it is
certainly true that they are fully general and more flexible.  This makes them
attractive, but it is a trap.  I’d really rather we don’t go down this route,
and maintain the approach of only using unit tests for very low level things
like apis in ADT etc.
> >
> > Note that we already have unittests for non low-level APIs such as
> > passes (VPlan, LICM, Unrolling, ...)
> >
> > Can you elaborate on what the trap is?
>
> I am also opposed to those - I think they have exactly the same problems
described above, and we should invest in better testing infra to support them.
Except for the linking issue, I don't see them being problems or even
being advantages.

You seem to acknowledge that these kind of unit tests are widely used
for supposedly good reasons. Why is LLVM special?

Michael