llvm dev - Feb 2016 - [cfe-dev] Testing Best Practices/Goals (in the context of compiler-rt)

On Wed, Feb 10, 2016 at 2:11 PM, Justin Bogner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> David Blaikie via cfe-dev <cfe-dev at lists.llvm.org> writes:
> > Recently had a bit of a digression in a review thread related to some
> tests
> > going in to compiler-rt (
> >
>
http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html
> > ) and there seems to be some disconnect at least between my
expectations
> > and reality. So I figured I'd have a bit of a discussion out here
on the
> > dev lists where there's a bit more visibility.
> >
> > My basic expectation is that the lit tests in any LLVM project except
the
> > test-suite are targeted tests intended to test only the functionality
in
> > the project. This seems like a pretty well accepted doctrine across
most
> > LLVM projects - most visibly in Clang, where we make a concerted
effort
> not
> > to have tests that execute LLVM optimizations, etc.
> >
> > There are exceptions/middle ground to this - DIBuilder is in LLVM, but
> > essentially tested in Clang rather than writing LLVM unit tests.
It's
> > somewhat unavoidable that any of the IR building code (IRBuilder,
> > DIBuilder, IR asm printing, etc) is 'tested' incidentally in
Clang in
> > process of testing Clang's IR generation. But these are seen as
> incidental,
> > not intentionally trying to cover LLVM with Clang tests (we don't
add a
> > Clang test if we add a new feature to IRBuilder just to test the
> IRBuilder).
> >
> > Another case with some middle ground are things like linker tests and
> > objdump, dwarfdump, etc - in theory to isolate the test we would
checkin
> > binaries (or the textual object representation lld had for a while,
etc)
> to
> > test those tools. Some tests instead checkin assembly and assemble it
> with
> > llvm-mc. Again, not to cover llvm-mc, but on the assumption that
llvm-mc
> is
> > tested, and just using it as a tool to make tests easier to maintain.
> >
> > So I was surprised to find that the compiler-rt lit tests seem to
diverge
> > from this philosophy & contain more intentional end-to-end tests
(eg:
> > adding a test there when making a fix to Clang to add a counter to a
> > function that was otherwise missing a counter - I'd expect that to
be
> > tested in Clang and that there would already be coverage in
compiler-rt
> for
> > "if a function has a counter, does compiler-rt do the right thing
with
> that
> > counter" (testing whatever code in compiler-rt needs to be
tested)).
> >
> > Am I off base here? Are compiler-rt's tests fundamentally
different to
> the
> > rest of the LLVM project? Why? Should they continue to be?
>
> I think there's a bit of grey area in terms testing the runtime -
> generally it's pretty hard to use the runtime without a fairly
> end-to-end test, so tests of the runtime often end up looking pretty
> close to an end-to-end test.
>
> That said, I don't think that should be used as an excuse to sneak
> arbitrary end-to-end tests into compiler-rt. We should absolutely write
> tests in clang and llvm that we're inputting what we expect to the
> runtime and try to keep the tests in compiler-rt as focused on just
> exercising the runtime code as possible.
>
Yes, we should not use compiler-rt tests as an excuse of not adding
clang/LLVM test. The latter should always be added if possible -- they are
platform independent and is the first level of defense.  runtime test's
focus is also more on the runtime lib itself and interaction between
 runtime, compiler, binutils and other tools.

David
>
> IIUC, the correct place for integration tests in general is somewhere
> like test-suite, but I think it's a bit intimidating to some people to
> add new tests there (Are there docs on this?). I suspect some of the
> profiling related tests in compiler-rt are doing a bit much and should
> graduate to a spot in the test-suite (but I don't have time to
volunteer
> to do the work, unfortunately).
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160210/7c4524dc/attachment.html>

I mostly agree with what Richard and Justin said. Adding a few notes about
the general strategy we use:

(1) lit tests which look "end-to-end" proved to be way more convenient
for
testing runtime libraries than unit tests. We do have
the latter, and use them to provide test coverage for utility functions,
but we quite often accompany fix to the runtime library with
"end-to-end" small reproducer extracted from the real-world code that
exposed the issue.
Incidentally, this tests a whole lot of other functionality: Clang driver,
frontend, LLVM passes, etc, but it's not the intent of the test.
These tests are sometimes platform-specific and poorly portable, but they
are more reliable (we make the same steps as the
user of the compiler), and serve the purpose of documentation.

(2) If we change LLVM instrumentation - we add a test to LLVM. If we change
Clang code generation or driver behavior - we add
a test to Clang. No excuses here.

(3) Sometimes we still add a compiler-rt test for the change in LLVM or
Clang: e.g. if we enhance Clang frontend to teach UBSan
to detecting yet another kind of overflow, it makes sense to add a test to
UBSan test-suite that demonstrates it, in addition to
Clang test verifying that we emit a call to UBSan runtime. Also,
compiler-rt test would allow us to verify that the actual error report
we present to the user is sane.

(4) True, we're intimidated by test-suite :) I feel that current use of
compiler-rt test suite to check compiler-rt libs better follows
the doctrine described by David. Also, there's significant complexity in
compiler-rt test suite that narrows the tests executed
to those supported by current host.

On Wed, Feb 10, 2016 at 2:33 PM, Xinliang David Li via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
>
>
> On Wed, Feb 10, 2016 at 2:11 PM, Justin Bogner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> David Blaikie via cfe-dev <cfe-dev at lists.llvm.org> writes:
>> > Recently had a bit of a digression in a review thread related to
some
>> tests
>> > going in to compiler-rt (
>> >
>>
http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html
>> > ) and there seems to be some disconnect at least between my
expectations
>> > and reality. So I figured I'd have a bit of a discussion out
here on the
>> > dev lists where there's a bit more visibility.
>> >
>> > My basic expectation is that the lit tests in any LLVM project
except
>> the
>> > test-suite are targeted tests intended to test only the
functionality in
>> > the project. This seems like a pretty well accepted doctrine
across most
>> > LLVM projects - most visibly in Clang, where we make a concerted
effort
>> not
>> > to have tests that execute LLVM optimizations, etc.
>> >
>> > There are exceptions/middle ground to this - DIBuilder is in LLVM,
but
>> > essentially tested in Clang rather than writing LLVM unit tests.
It's
>> > somewhat unavoidable that any of the IR building code (IRBuilder,
>> > DIBuilder, IR asm printing, etc) is 'tested' incidentally
in Clang in
>> > process of testing Clang's IR generation. But these are seen
as
>> incidental,
>> > not intentionally trying to cover LLVM with Clang tests (we
don't add a
>> > Clang test if we add a new feature to IRBuilder just to test the
>> IRBuilder).
>> >
>> > Another case with some middle ground are things like linker tests
and
>> > objdump, dwarfdump, etc - in theory to isolate the test we would
checkin
>> > binaries (or the textual object representation lld had for a
while,
>> etc) to
>> > test those tools. Some tests instead checkin assembly and assemble
it
>> with
>> > llvm-mc. Again, not to cover llvm-mc, but on the assumption that
>> llvm-mc is
>> > tested, and just using it as a tool to make tests easier to
maintain.
>> >
>> > So I was surprised to find that the compiler-rt lit tests seem to
>> diverge
>> > from this philosophy & contain more intentional end-to-end
tests (eg:
>> > adding a test there when making a fix to Clang to add a counter to
a
>> > function that was otherwise missing a counter - I'd expect
that to be
>> > tested in Clang and that there would already be coverage in
compiler-rt
>> for
>> > "if a function has a counter, does compiler-rt do the right
thing with
>> that
>> > counter" (testing whatever code in compiler-rt needs to be
tested)).
>> >
>> > Am I off base here? Are compiler-rt's tests fundamentally
different to
>> the
>> > rest of the LLVM project? Why? Should they continue to be?
>>
>> I think there's a bit of grey area in terms testing the runtime -
>> generally it's pretty hard to use the runtime without a fairly
>> end-to-end test, so tests of the runtime often end up looking pretty
>> close to an end-to-end test.
>>
>> That said, I don't think that should be used as an excuse to sneak
>> arbitrary end-to-end tests into compiler-rt. We should absolutely write
>> tests in clang and llvm that we're inputting what we expect to the
>> runtime and try to keep the tests in compiler-rt as focused on just
>> exercising the runtime code as possible.
>>
>
> Yes, we should not use compiler-rt tests as an excuse of not adding
> clang/LLVM test. The latter should always be added if possible -- they are
> platform independent and is the first level of defense.  runtime test's
> focus is also more on the runtime lib itself and interaction between
>  runtime, compiler, binutils and other tools.
>
> David
>
>>
>> IIUC, the correct place for integration tests in general is somewhere
>> like test-suite, but I think it's a bit intimidating to some people
to
>> add new tests there (Are there docs on this?). I suspect some of the
>> profiling related tests in compiler-rt are doing a bit much and should
>> graduate to a spot in the test-suite (but I don't have time to
volunteer
>> to do the work, unfortunately).
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
>

-- 
Alexey Samsonov
vonosmas at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160210/6e58a10a/attachment.html>

On Wed, Feb 10, 2016 at 2:33 PM, Xinliang David Li via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
>
>
> On Wed, Feb 10, 2016 at 2:11 PM, Justin Bogner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> David Blaikie via cfe-dev <cfe-dev at lists.llvm.org> writes:
>> > Recently had a bit of a digression in a review thread related to
some
>> tests
>> > going in to compiler-rt (
>> >
>>
http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html
>> > ) and there seems to be some disconnect at least between my
expectations
>> > and reality. So I figured I'd have a bit of a discussion out
here on the
>> > dev lists where there's a bit more visibility.
>> >
>> > My basic expectation is that the lit tests in any LLVM project
except
>> the
>> > test-suite are targeted tests intended to test only the
functionality in
>> > the project. This seems like a pretty well accepted doctrine
across most
>> > LLVM projects - most visibly in Clang, where we make a concerted
effort
>> not
>> > to have tests that execute LLVM optimizations, etc.
>> >
>> > There are exceptions/middle ground to this - DIBuilder is in LLVM,
but
>> > essentially tested in Clang rather than writing LLVM unit tests.
It's
>> > somewhat unavoidable that any of the IR building code (IRBuilder,
>> > DIBuilder, IR asm printing, etc) is 'tested' incidentally
in Clang in
>> > process of testing Clang's IR generation. But these are seen
as
>> incidental,
>> > not intentionally trying to cover LLVM with Clang tests (we
don't add a
>> > Clang test if we add a new feature to IRBuilder just to test the
>> IRBuilder).
>> >
>> > Another case with some middle ground are things like linker tests
and
>> > objdump, dwarfdump, etc - in theory to isolate the test we would
checkin
>> > binaries (or the textual object representation lld had for a
while,
>> etc) to
>> > test those tools. Some tests instead checkin assembly and assemble
it
>> with
>> > llvm-mc. Again, not to cover llvm-mc, but on the assumption that
>> llvm-mc is
>> > tested, and just using it as a tool to make tests easier to
maintain.
>> >
>> > So I was surprised to find that the compiler-rt lit tests seem to
>> diverge
>> > from this philosophy & contain more intentional end-to-end
tests (eg:
>> > adding a test there when making a fix to Clang to add a counter to
a
>> > function that was otherwise missing a counter - I'd expect
that to be
>> > tested in Clang and that there would already be coverage in
compiler-rt
>> for
>> > "if a function has a counter, does compiler-rt do the right
thing with
>> that
>> > counter" (testing whatever code in compiler-rt needs to be
tested)).
>> >
>> > Am I off base here? Are compiler-rt's tests fundamentally
different to
>> the
>> > rest of the LLVM project? Why? Should they continue to be?
>>
>> I think there's a bit of grey area in terms testing the runtime -
>> generally it's pretty hard to use the runtime without a fairly
>> end-to-end test, so tests of the runtime often end up looking pretty
>> close to an end-to-end test.
>>
>> That said, I don't think that should be used as an excuse to sneak
>> arbitrary end-to-end tests into compiler-rt. We should absolutely write
>> tests in clang and llvm that we're inputting what we expect to the
>> runtime and try to keep the tests in compiler-rt as focused on just
>> exercising the runtime code as possible.
>>
>
> Yes, we should not use compiler-rt tests as an excuse of not adding
> clang/LLVM test. The latter should always be added if possible -- they are
> platform independent and is the first level of defense.  runtime test's
> focus is also more on the runtime lib itself and interaction between
>  runtime, compiler, binutils and other tools.
>
This latter bit is where there's some disagreement.

The LLVM project generally expects the "make check" style tests to be
fast
and narrow in scope - LLVM tests test LLVM, Clang tests Clang, and it would
seem compiler-rt tests should be just about testing the runtime lib,
specifically. Sure, that means making sure it can handle the diversity of
input/output, etc. But if two inputs to Clang produce the same output we
only write one LLVM test for that output and I would expect the same of
compiler-rt - if two inputs to Clang produce the same sort of output (a
function with a counter, in the example we started with) I'd expect only
one test for compiler-rt, to check that it can handle a function with a
counter.

(generally: if a change wasn't made to compiler-rt, I wouldn't expect a
test to be added (except to makeup for cases of missing compiler-rt
coverage))

>
> David
>
>>
>> IIUC, the correct place for integration tests in general is somewhere
>> like test-suite, but I think it's a bit intimidating to some people
to
>> add new tests there (Are there docs on this?). I suspect some of the
>> profiling related tests in compiler-rt are doing a bit much and should
>> graduate to a spot in the test-suite (but I don't have time to
volunteer
>> to do the work, unfortunately).
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160211/172b6ae2/attachment.html>

On Wed, Feb 10, 2016 at 3:55 PM, Alexey Samsonov via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> I mostly agree with what Richard and Justin said. Adding a few notes about
> the general strategy we use:
>
> (1) lit tests which look "end-to-end" proved to be way more
convenient for
> testing runtime libraries than unit tests.
>
We do have
> the latter, and use them to provide test coverage for utility functions,
> but we quite often accompany fix to the runtime library with
> "end-to-end" small reproducer extracted from the real-world code
that
> exposed the issue.
> Incidentally, this tests a whole lot of other functionality: Clang driver,
> frontend, LLVM passes, etc, but it's not the intent of the test.
>
Indeed - this is analogous to the tests for, say, LLD that use llvm-mc to
produce the inputs rather than checking in object files. That area is open
to some discussion as to just how many tools we should rope in/how isolated
we should make tests (eg: maybe building the json object file format was
going too far towards isolation? Not clear - opinions differ). But the
point of the test is to test the compiler-rt functionality that was
added/removed/modified.

I think most people are in agreement with that, while acknowledging the
fuzzy line about how isolated we might be.

> These tests are sometimes platform-specific and poorly portable, but they
> are more reliable (we make the same steps as the
> user of the compiler), and serve the purpose of documentation.
>
> (2) If we change LLVM instrumentation - we add a test to LLVM. If we
> change Clang code generation or driver behavior - we add
> a test to Clang. No excuses here.
>
> (3) Sometimes we still add a compiler-rt test for the change in LLVM or
> Clang: e.g. if we enhance Clang frontend to teach UBSan
> to detecting yet another kind of overflow, it makes sense to add a test to
> UBSan test-suite that demonstrates it, in addition to
> Clang test verifying that we emit a call to UBSan runtime. Also,
> compiler-rt test would allow us to verify that the actual error report
> we present to the user is sane.
>
This bit ^ is a bit unclear to me. If there was no change to the UBSan
runtime, and the code generated by Clang is equivalent/similar to an
existing use of the UBSan runtime - what is it that the new compiler-rt
test is providing? (perhaps you could give a concrete example you had in
mind to look at?)

> (4) True, we're intimidated by test-suite :) I feel that current use of
> compiler-rt test suite to check compiler-rt libs better follows
> the doctrine described by David.
>
Which David? ;) (I guess David Li, not me)

I think maybe what could be worth doing would be separating out the
broader/intentionally "end to end" sort of tests from the ones
intended to
test compiler-rt in relative isolation.

Most importantly, I'd expect only the latter to run in a "make
check-all"
run, as we do for Clang/LLVM, etc.

> Also, there's significant complexity in compiler-rt test suite that
> narrows the tests executed
> to those supported by current host.
>


>
> On Wed, Feb 10, 2016 at 2:33 PM, Xinliang David Li via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>>
>>
>> On Wed, Feb 10, 2016 at 2:11 PM, Justin Bogner via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> David Blaikie via cfe-dev <cfe-dev at lists.llvm.org> writes:
>>> > Recently had a bit of a digression in a review thread related
to some
>>> tests
>>> > going in to compiler-rt (
>>> >
>>>
http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20160208/330759.html
>>> > ) and there seems to be some disconnect at least between my
>>> expectations
>>> > and reality. So I figured I'd have a bit of a discussion
out here on
>>> the
>>> > dev lists where there's a bit more visibility.
>>> >
>>> > My basic expectation is that the lit tests in any LLVM project
except
>>> the
>>> > test-suite are targeted tests intended to test only the
functionality
>>> in
>>> > the project. This seems like a pretty well accepted doctrine
across
>>> most
>>> > LLVM projects - most visibly in Clang, where we make a
concerted
>>> effort not
>>> > to have tests that execute LLVM optimizations, etc.
>>> >
>>> > There are exceptions/middle ground to this - DIBuilder is in
LLVM, but
>>> > essentially tested in Clang rather than writing LLVM unit
tests. It's
>>> > somewhat unavoidable that any of the IR building code
(IRBuilder,
>>> > DIBuilder, IR asm printing, etc) is 'tested'
incidentally in Clang in
>>> > process of testing Clang's IR generation. But these are
seen as
>>> incidental,
>>> > not intentionally trying to cover LLVM with Clang tests (we
don't add a
>>> > Clang test if we add a new feature to IRBuilder just to test
the
>>> IRBuilder).
>>> >
>>> > Another case with some middle ground are things like linker
tests and
>>> > objdump, dwarfdump, etc - in theory to isolate the test we
would
>>> checkin
>>> > binaries (or the textual object representation lld had for a
while,
>>> etc) to
>>> > test those tools. Some tests instead checkin assembly and
assemble it
>>> with
>>> > llvm-mc. Again, not to cover llvm-mc, but on the assumption
that
>>> llvm-mc is
>>> > tested, and just using it as a tool to make tests easier to
maintain.
>>> >
>>> > So I was surprised to find that the compiler-rt lit tests seem
to
>>> diverge
>>> > from this philosophy & contain more intentional end-to-end
tests (eg:
>>> > adding a test there when making a fix to Clang to add a
counter to a
>>> > function that was otherwise missing a counter - I'd expect
that to be
>>> > tested in Clang and that there would already be coverage in
>>> compiler-rt for
>>> > "if a function has a counter, does compiler-rt do the
right thing with
>>> that
>>> > counter" (testing whatever code in compiler-rt needs to
be tested)).
>>> >
>>> > Am I off base here? Are compiler-rt's tests fundamentally
different to
>>> the
>>> > rest of the LLVM project? Why? Should they continue to be?
>>>
>>> I think there's a bit of grey area in terms testing the runtime
-
>>> generally it's pretty hard to use the runtime without a fairly
>>> end-to-end test, so tests of the runtime often end up looking
pretty
>>> close to an end-to-end test.
>>>
>>> That said, I don't think that should be used as an excuse to
sneak
>>> arbitrary end-to-end tests into compiler-rt. We should absolutely
write
>>> tests in clang and llvm that we're inputting what we expect to
the
>>> runtime and try to keep the tests in compiler-rt as focused on just
>>> exercising the runtime code as possible.
>>>
>>
>> Yes, we should not use compiler-rt tests as an excuse of not adding
>> clang/LLVM test. The latter should always be added if possible -- they
are
>> platform independent and is the first level of defense.  runtime
test's
>> focus is also more on the runtime lib itself and interaction between
>>  runtime, compiler, binutils and other tools.
>>
>> David
>>
>>>
>>> IIUC, the correct place for integration tests in general is
somewhere
>>> like test-suite, but I think it's a bit intimidating to some
people to
>>> add new tests there (Are there docs on this?). I suspect some of
the
>>> profiling related tests in compiler-rt are doing a bit much and
should
>>> graduate to a spot in the test-suite (but I don't have time to
volunteer
>>> to do the work, unfortunately).
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>>
>
>
> --
> Alexey Samsonov
> vonosmas at gmail.com
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160211/e5b0b09a/attachment.html>