llvm dev - Jun 2013 - [LLVMdev] [icFuzz] Help needed with analyzing randomly generated tests that fail on clang 3.4 trunk

The tests by design are syntactically correct, semantically correct, and have
deterministic output.

-moh

From: Nick Lewycky [mailto:nlewycky at google.com]
Sent: Monday, June 24, 2013 4:14 PM
To: Haghighat, Mohammad R
Cc: llvmdev at cs.uiuc.edu
Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly generated
tests that fail on clang 3.4 trunk

On 24 June 2013 16:10, Haghighat, Mohammad R <mohammad.r.haghighat at
intel.com<mailto:mohammad.r.haghighat at intel.com>> wrote:
Hi,

I just submitted a bug report with a package containing 107 small test cases
that fail on the latest LLVM/clang 3.4 main trunk (184563). Included are test
sources, compilation commands, test input files, and results at -O0 and -O2 when
applicable.

http://llvm.org/bugs/show_bug.cgi?id=16431


These tests have been automatically generated by an internal tool at Intel, the
Intel Compiler fuzzer, icFuzz. The tests are typically very small. For example,
for the following simple loop (test t5702) on MacOS X, clang at -O2 generates a
binary that crashes:


// Test Loop Interchange
for (j = 2; j < 76; j++) {
    for (jm = 1; jm < 30; jm++) {
        h[j-1][jm-1] = j + 83;
    }
}

The tests are put in to two categories
- tests that have different runtime outputs when compiled at -O0 and -O2 (this
category also includes runtime crashes)

Are these tests generated in a manner such that they have a very low probability
of using undefined behaviour?

Nick

- tests that cause infinite loops in the Clang optimizer

Many of these failing tests could be due to the same bug, thus a much smaller
number of root problems are expected.

Any help with triaging these bugs would be highly appreciated.

Thanks,
-moh



_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130624/3fbe8876/attachment.html>

On Mon, Jun 24, 2013 at 11:20:09PM +0000, Haghighat, Mohammad R
wrote:
> The tests by design are syntactically correct, semantically correct, and
have deterministic output.
> 
> -moh
Hi,
I wanted to believe you had a randomized code generator that could cover the
entire
valid input space. (smiles ;) But the test cited was not generated by a
randomized
code generator. It is way too structured. Even so, it could have been generated
by a
partially randomized code generator that uses valid code idioms with some
aspects
randomized. Interesting to think about it.

I wrote a fully randomized Java bytecode sequence generator once. It covered the
entire
valid input space. I used it to test a Java hardware accelerator chip that had
just been
taped out. It found quite a few bugs, where 4 of them were fatal to the chip.

Writing such a tool for a general purpose compiler would be quite a bit more
difficult, but it is doable. Imagine releasing compilers that are bug free,
at least for a given language and a particular platform. It's doable, and I
wonder
why corporations don't invest in it.

enjoy,
Karen
-- 
Karen Shaeffer
Neuralscape, Mountain View, CA 94040

Hi Karen,

Thanks much for your comment and for sharing of your experience. icFuzz has a
core that is "really" random, but does not cover the entire C space.
The tool was designed from scratch to be extensible, and comes with a couple of
extensions that target some of compiler optimizations optimizing compilers
typically do: CSE, loop interchange, vectorization, etc. But even in the case of
extensions, other than the structure of the extension, most of the details are
highly parametric and configurable. The chances of a totally random test meeting
all the criteria for certain optimizations to kick in is lower than the case in
guided test-generation. I actually have the x-y charts where x is the number of
generated tests and y is the number of fails for both completely random tests as
well as random+guided tests, and the curve of guided tests is significantly
above that of random tests. Also, the test generator is restricted to the
"unsigned int" type where the C++ semantics are precisely defined even
in the cases of overflow. Also, features that have implementation-dependent
behavior are excluded by design (e.g., expressions including side effects during
their evaluation, array out of bounds access, etc.).

But you are right in that the generated code has some structure ;) 

Cheers,
-moh

-----Original Message-----
From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On
Behalf Of Karen Shaeffer
Sent: Monday, June 24, 2013 5:17 PM
To: llvmdev at cs.uiuc.edu
Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly generated
tests that fail on clang 3.4 trunk

On Mon, Jun 24, 2013 at 11:20:09PM +0000, Haghighat, Mohammad R
wrote:
> The tests by design are syntactically correct, semantically correct, and
have deterministic output.
> 
> -moh
Hi,
I wanted to believe you had a randomized code generator that could cover the
entire
valid input space. (smiles ;) But the test cited was not generated by a
randomized
code generator. It is way too structured. Even so, it could have been generated
by a
partially randomized code generator that uses valid code idioms with some
aspects
randomized. Interesting to think about it.

I wrote a fully randomized Java bytecode sequence generator once. It covered the
entire
valid input space. I used it to test a Java hardware accelerator chip that had
just been
taped out. It found quite a few bugs, where 4 of them were fatal to the chip.

Writing such a tool for a general purpose compiler would be quite a bit more
difficult, but it is doable. Imagine releasing compilers that are bug free,
at least for a given language and a particular platform. It's doable, and I
wonder
why corporations don't invest in it.

enjoy,
Karen
-- 
Karen Shaeffer
Neuralscape, Mountain View, CA 94040
_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev