> Indeed, but can't an analysis find at least one value for each variable > where the behavior is not undefined? > Such a value must exist, or the entire function is useless if it always > has undefined behavior.Good point :).> Sure, testing on 1 such value (or a random) value won't prove that the > result is correct, but may help finding trivial > miscompilations like the neon_helper case.Are you absolutely sure it's a miscompilation? I have already shot myself in the foot a couple times on the GCC mailing list or bugzilla by pointing out a bug that turned out to be code with subtle undefined behavior...> Alternatively a testcase could be manually constructed for the top 10 > functions in the size comparison charts, > and see whether they are miscompiled. Repeat until top 10 has no > miscompilations.Tell you what: if I get enough test cases like this, I'll write the test harness supporting it. I don't have time to do this kind of code inspection myself. There has been talk (I don't remember where) about a Clang option for detecting undefined behavior. Is there any progress on this? This could be used to enable automated random testing. John
On Jan 20, 2010, at 12:49 PM, John Regehr wrote:> > There has been talk (I don't remember where) about a Clang option for > detecting undefined behavior. Is there any progress on this? This > could > be used to enable automated random testing.-fcatch-undefined-behavior: http://clang.llvm.org/docs/UsersManual.html#codegen Right now it only catches out of range shifts and simple array out of bound issues, not all undefined behavior. -Chris
On 01/20/2010 10:49 PM, John Regehr wrote:>> Indeed, but can't an analysis find at least one value for each variable >> where the behavior is not undefined? >> Such a value must exist, or the entire function is useless if it always >> has undefined behavior. > > Good point :). > >> Sure, testing on 1 such value (or a random) value won't prove that the >> result is correct, but may help finding trivial >> miscompilations like the neon_helper case. > > Are you absolutely sure it's a miscompilation? I have already shot > myself in the foot a couple times on the GCC mailing list or bugzilla > by pointing out a bug that turned out to be code with subtle undefined > behavior...Well if it is not then it is a qemu bug, so it is a bug in either case, you just have to report it to another bugzilla ;) The code does conversions by assigning to one union member and reading from another. AFAIK that was a GCC language extension, maybe they don't support it in the latest release, or accidentaly broke it. I don't know. Someone should reduce a testcase for gcc-head to see exactly what it is about. My gcc (4.4) doesn't miscompile it. Either way I'd rather see a warning from gcc when it decides to optimize the entire function away.> >> Alternatively a testcase could be manually constructed for the top 10 >> functions in the size comparison charts, >> and see whether they are miscompiled. Repeat until top 10 has no >> miscompilations. > > Tell you what: if I get enough test cases like this, I'll write the > test harness supporting it. I don't have time to do this kind of code > inspection myself.Makes sense.> > There has been talk (I don't remember where) about a Clang option for > detecting undefined behavior. Is there any progress on this? This > could be used to enable automated random testing.*Yes, -fcatch-undefined-behavior. http://clang.llvm.org/docs/UsersManual.html#codegen * Best regards, --Edwin
> *Yes, -fcatch-undefined-behavior. > http://clang.llvm.org/docs/UsersManual.html#codegenThanks guys. My understanding of the situation is that for meaningful automated testing, the protection from undefined behavior has to cover all problems that actually occur in the code under test. But I'll keep checking on this... John
> Right now it only catches out of range shifts and simple array out of > bound issues, not all undefined behavior.Besides the obvious memory safety stuff, my list of top undefined behaviors to catch would be: - multiple updates to objects between sequence points - integer overflows - use-after-death of stack variables - use of uninitialized stack variables - const/volatile violations Some of these will be no fun to implement. But the resulting tool would be enormously valuable. John