Hi Torok-> Could you also add a main() for each of these files, and do > a very simple test that the optimized functions actually work?Unfortunately, testing isolated C functions is much harder than just passing them random data! Consider this function: int foo (int x, int y) { return x+y; } The behavior of foo() is undefined when x+y overflows. If course it is trivial to come up with similar examples based on shifts, multiplies and divides, etc. A potential solution is "under-constrained execution": http://www.stanford.edu/~engler/issta07v-engler.pdf I will bug Dawson and Daniel and see if I can get ahold of some code for this. John
On 01/20/2010 10:10 PM, John Regehr wrote:> Hi Torok- > >> Could you also add a main() for each of these files, and do >> a very simple test that the optimized functions actually work? > > Unfortunately, testing isolated C functions is much harder than just > passing them random data! > > Consider this function: > > int foo (int x, int y) { return x+y; } > > The behavior of foo() is undefined when x+y overflows. If course it > is trivial to come up with similar examples based on shifts, > multiplies and divides, etc.Indeed, but can't an analysis find at least one value for each variable where the behavior is not undefined? Such a value must exist, or the entire function is useless if it always has undefined behavior. Sure, testing on 1 such value (or a random) value won't prove that the result is correct, but may help finding trivial miscompilations like the neon_helper case. Alternatively a testcase could be manually constructed for the top 10 functions in the size comparison charts, and see whether they are miscompiled. Repeat until top 10 has no miscompilations.> > A potential solution is "under-constrained execution": > > http://www.stanford.edu/~engler/issta07v-engler.pdf > > I will bug Dawson and Daniel and see if I can get ahold of some code > for this.Although EXE isn't, KLEE is publicly available. Best regards, --Edwin
> Indeed, but can't an analysis find at least one value for each variable > where the behavior is not undefined? > Such a value must exist, or the entire function is useless if it always > has undefined behavior.Good point :).> Sure, testing on 1 such value (or a random) value won't prove that the > result is correct, but may help finding trivial > miscompilations like the neon_helper case.Are you absolutely sure it's a miscompilation? I have already shot myself in the foot a couple times on the GCC mailing list or bugzilla by pointing out a bug that turned out to be code with subtle undefined behavior...> Alternatively a testcase could be manually constructed for the top 10 > functions in the size comparison charts, > and see whether they are miscompiled. Repeat until top 10 has no > miscompilations.Tell you what: if I get enough test cases like this, I'll write the test harness supporting it. I don't have time to do this kind of code inspection myself. There has been talk (I don't remember where) about a Clang option for detecting undefined behavior. Is there any progress on this? This could be used to enable automated random testing. John