llvm dev - Jun 2013 - [LLVMdev] [icFuzz] Help needed with analyzing randomly generated tests that fail on clang 3.4 trunk

Hi,

I just submitted a bug report with a package containing 107 small test cases
that fail on the latest LLVM/clang 3.4 main trunk (184563). Included are test
sources, compilation commands, test input files, and results at -O0 and -O2 when
applicable.

http://llvm.org/bugs/show_bug.cgi?id=16431


These tests have been automatically generated by an internal tool at Intel, the
Intel Compiler fuzzer, icFuzz. The tests are typically very small. For example,
for the following simple loop (test t5702) on MacOS X, clang at -O2 generates a
binary that crashes:


// Test Loop Interchange
for (j = 2; j < 76; j++) {
    for (jm = 1; jm < 30; jm++) {
        h[j-1][jm-1] = j + 83;
    }
}

The tests are put in to two categories
- tests that have different runtime outputs when compiled at -O0 and -O2 (this
category also includes runtime crashes)
- tests that cause infinite loops in the Clang optimizer

Many of these failing tests could be due to the same bug, thus a much smaller
number of root problems are expected.

Any help with triaging these bugs would be highly appreciated.

Thanks,
-moh


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130624/f087759c/attachment.html>

On 24 June 2013 16:10, Haghighat, Mohammad R <mohammad.r.haghighat at
intel.com
> wrote:
>  Hi,****
>
> ** **
>
> I just submitted a bug report with a package containing 107 small test
> cases that fail on the latest LLVM/clang 3.4 main trunk (184563). Included
> are test sources, compilation commands, test input files, and results at
> –O0 and –O2 when applicable.****
>
> ** **
>
> http://llvm.org/bugs/show_bug.cgi?id=16431****
>
> ** **
>
> These tests have been automatically generated by an internal tool at
> Intel, the Intel Compiler fuzzer, icFuzz. The tests are typically very
> small. For example, for the following simple loop (test t5702) on MacOS
> X, clang at –O2 generates a binary that crashes:****
>
> ** **
>
> // Test Loop Interchange****
>
> for (j = 2; j < 76; j++) {****
>
>     for (jm = 1; jm < 30; jm++) {****
>
>         h[j-1][jm-1] = j + 83;****
>
>     }****
>
> }****
>
> ** **
>
> The tests are put in to two categories ****
>
> - tests that have different runtime outputs when compiled at -O0 and -O2
> (this category also includes runtime crashes)
>
Are these tests generated in a manner such that they have a very low
probability of using undefined behaviour?

Nick

>  ****
>
> - tests that cause infinite loops in the Clang optimizer ****
>
> ** **
>
> Many of these failing tests could be due to the same bug, thus a much
> smaller number of root problems are expected. ****
>
> ** **
>
> Any help with triaging these bugs would be highly appreciated. ****
>
> ** **
>
> Thanks,****
>
> -moh****
>
> ** **
>
> ** **
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130624/87df64cc/attachment.html>

The tests by design are syntactically correct, semantically correct, and have
deterministic output.

-moh

From: Nick Lewycky [mailto:nlewycky at google.com]
Sent: Monday, June 24, 2013 4:14 PM
To: Haghighat, Mohammad R
Cc: llvmdev at cs.uiuc.edu
Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly generated
tests that fail on clang 3.4 trunk

On 24 June 2013 16:10, Haghighat, Mohammad R <mohammad.r.haghighat at
intel.com<mailto:mohammad.r.haghighat at intel.com>> wrote:
Hi,

I just submitted a bug report with a package containing 107 small test cases
that fail on the latest LLVM/clang 3.4 main trunk (184563). Included are test
sources, compilation commands, test input files, and results at -O0 and -O2 when
applicable.

http://llvm.org/bugs/show_bug.cgi?id=16431


These tests have been automatically generated by an internal tool at Intel, the
Intel Compiler fuzzer, icFuzz. The tests are typically very small. For example,
for the following simple loop (test t5702) on MacOS X, clang at -O2 generates a
binary that crashes:


// Test Loop Interchange
for (j = 2; j < 76; j++) {
    for (jm = 1; jm < 30; jm++) {
        h[j-1][jm-1] = j + 83;
    }
}

The tests are put in to two categories
- tests that have different runtime outputs when compiled at -O0 and -O2 (this
category also includes runtime crashes)

Are these tests generated in a manner such that they have a very low probability
of using undefined behaviour?

Nick

- tests that cause infinite loops in the Clang optimizer

Many of these failing tests could be due to the same bug, thus a much smaller
number of root problems are expected.

Any help with triaging these bugs would be highly appreciated.

Thanks,
-moh



_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130624/3fbe8876/attachment.html>

Hi Moh,

Thanks for this. I’m really glad to see the work you’re doing in this area and
believe it will be extremely helpful in improving the quality of the compiler.

-Jim

On Jun 24, 2013, at 4:10 PM, Haghighat, Mohammad R <mohammad.r.haghighat at
intel.com> wrote:
> Hi,
>  
> I just submitted a bug report with a package containing 107 small test
cases that fail on the latest LLVM/clang 3.4 main trunk (184563). Included are
test sources, compilation commands, test input files, and results at –O0 and –O2
when applicable.
>  
> http://llvm.org/bugs/show_bug.cgi?id=16431
>  
> These tests have been automatically generated by an internal tool at Intel,
the Intel Compiler fuzzer, icFuzz. The tests are typically very small. For
example, for the following simple loop (test t5702) on MacOS X, clang at –O2
generates a binary that crashes:
>  
> // Test Loop Interchange
> for (j = 2; j < 76; j++) {
>     for (jm = 1; jm < 30; jm++) {
>         h[j-1][jm-1] = j + 83;
>     }
> }
>  
> The tests are put in to two categories
> - tests that have different runtime outputs when compiled at -O0 and -O2
(this category also includes runtime crashes)
> - tests that cause infinite loops in the Clang optimizer
>  
> Many of these failing tests could be due to the same bug, thus a much
smaller number of root problems are expected.
>  
> Any help with triaging these bugs would be highly appreciated.
>  
> Thanks,
> -moh
>  
>  
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130625/af72110f/attachment.html>

----- Original Message -----
> 
> Hi Moh,
> 
> 
> Thanks for this. I’m really glad to see the work you’re doing in this
> area and believe it will be extremely helpful in improving the
> quality of the compiler.
> 
> 
> -Jim
> 
> 
> 
> On Jun 24, 2013, at 4:10 PM, Haghighat, Mohammad R <
> mohammad.r.haghighat at intel.com > wrote:
> 
> 
> 
> 
> 
> Hi,
> 
> I just submitted a bug report with a package containing 107 small
> test cases that fail on the latest LLVM/clang 3.4 main trunk
> (184563). Included are test sources, compilation commands, test
> input files, and results at –O0 and –O2 when applicable.
> 
> http://llvm.org/bugs/show_bug.cgi?id=16431
> 
> These tests have been automatically generated by an internal tool at
> Intel, the Intel Compiler fuzzer, icFuzz. The tests are typically
> very small. For example, for the following simple loop (test t5702)
> on MacOS X, clang at –O2 generates a binary that crashes:
> 
> // Test Loop Interchange
> for (j = 2; j < 76; j++) {
> for (jm = 1; jm < 30; jm++) {
> h[j-1][jm-1] = j + 83;
> }
> }
> 
> The tests are put in to two categories
> - tests that have different runtime outputs when compiled at -O0 and
> -O2 (this category also includes runtime crashes)
> - tests that cause infinite loops in the Clang optimizer
> 
> Many of these failing tests could be due to the same bug, thus a much
> smaller number of root problems are expected.
> 
> Any help with triaging these bugs would be highly appreciated.
I've gone through all of the miscompile cases, used bugpoint to reduce them,
and opened individual PRs for several distinct bugs. So far we have: PR16455
(loop vectorizer), PR16457 (sccp), PR16460 (instcombine). Thanks again for doing
this! Do you plan on repeating this testing on a regular basis? Can it be
automated?

 -Hal
> 
> Thanks,
> -moh
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory