[LLVMdev] Bad test health

David Blaikie
Thu, 13 Feb 2014 17:38:44 -0000

On Wed, Feb 12, 2014 at 9:21 PM, Owen Anderson <resistor at mac.com>
wrote:

>
> On Feb 12, 2014, at 7:19 PM, Nico Rieck <nico.rieck at gmail.com>
wrote:
>
> Oh, and some tests don't even call FileCheck in the first
> place!
>
>
> That's probably because it's a regression test for a crasher.
It's
> checking that the compiler exited successfully.
>

I assume Nico means it's a test with CHECK lines but doesn't invoke
FileCheck.

Though personally I object to tests that don't check anything other than
"doesn't crash" - if a certain example previously crashed and now
doesn't,
there was some codepath that was untested (its output unconstrained). The
crash just revealed the untested codepath, testing that it doesn't crash
still leaves the behavior mostly unconstrained to "do anything other than
crashing" which seems insufficient. We should take the opportunity to test
the untested behavior that's been discovered.

- David
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140213/74200409/attachment.html>