> to a testsuite, we can use them for validation. But I wouldn't want to > require a validation to pass some set of random tests that shifts each test > cycle.This is easy to fix: just specify a starting seed for the PRNG. However I think you should get past your prejudice against tests that shift each cycle, since changing tests have the advantage of increased test coverage. Different parts of a test suite have different purposes, and of course random programs would not replace any part of the existing collection of fixed test cases. I woudn't be making this argument if I hadn't seen for myself how one week random testing gives you nothing, the next week a whole pile of previously unknown failures. Alternatively we are working to generalize our program generator a bit so that it does a DFS or BFS to generate all programs smaller than some size bound (obviously we need to fudge on integer constants, for example by picking from a predetermined set of interesting constants). Once we do this it may be worth adding the resulting test programs to LLVM's test suite. John Regehr
>> to a testsuite, we can use them for validation. But I wouldn't want to >> require a validation to pass some set of random tests that shifts each test >> cycle. > > This is easy to fix: just specify a starting seed for the PRNG....which defeats much of the point of random testing.> However I think you should get past your prejudice against tests that > shift each cycle, since changing tests have the advantage of increased > test coverage. Different parts of a test suite have different purposes, > and of course random programs would not replace any part of the existing > collection of fixed test cases. I woudn't be making this argument if I > hadn't seen for myself how one week random testing gives you nothing, the > next week a whole pile of previously unknown failures.I don't think anyone is arguing against the utility of random test generation, the issue is that the results aren't really appropriate for validation where you are trying to make a comparison between builds. A system I've used previously was to run and report randomly generated tests along with validation testing, but not consider the random test results when tagging a build as valid. Instead, when random tests failed, the generated test case was saved, and could be added (preferably in a reduced form) as a static test for future validation runs. That way you get the benefits of random testing without spurious changes in validation status dependent on randomly generated tests.
> validation runs. That way you get the benefits of random testing > without spurious changes in validation status dependent on randomly > generated tests.Sounds great. I'm not trying to push random testing on people who don't want it, but I think it is useful to the LLVM project and it's not clear that I have the resources to keep doing it myself indefinitely. The tools are, I think, to the point where they can be pushed into an automated build/test loop, which is what I'm aiming for. If this testing is done continuously and for several targets then more regressions can be squashed while they're fresh. John Regehr