llvm dev - Jun 2013 - [LLVMdev] [icFuzz] Help needed with analyzing randomly generated tests that fail on clang 3.4 trunk

----- Original Message -----
> 
> Hi Moh,
> 
> 
> Thanks for this. I’m really glad to see the work you’re doing in this
> area and believe it will be extremely helpful in improving the
> quality of the compiler.
> 
> 
> -Jim
> 
> 
> 
> On Jun 24, 2013, at 4:10 PM, Haghighat, Mohammad R <
> mohammad.r.haghighat at intel.com > wrote:
> 
> 
> 
> 
> 
> Hi,
> 
> I just submitted a bug report with a package containing 107 small
> test cases that fail on the latest LLVM/clang 3.4 main trunk
> (184563). Included are test sources, compilation commands, test
> input files, and results at –O0 and –O2 when applicable.
> 
> http://llvm.org/bugs/show_bug.cgi?id=16431
> 
> These tests have been automatically generated by an internal tool at
> Intel, the Intel Compiler fuzzer, icFuzz. The tests are typically
> very small. For example, for the following simple loop (test t5702)
> on MacOS X, clang at –O2 generates a binary that crashes:
> 
> // Test Loop Interchange
> for (j = 2; j < 76; j++) {
> for (jm = 1; jm < 30; jm++) {
> h[j-1][jm-1] = j + 83;
> }
> }
> 
> The tests are put in to two categories
> - tests that have different runtime outputs when compiled at -O0 and
> -O2 (this category also includes runtime crashes)
> - tests that cause infinite loops in the Clang optimizer
> 
> Many of these failing tests could be due to the same bug, thus a much
> smaller number of root problems are expected.
> 
> Any help with triaging these bugs would be highly appreciated.
I've gone through all of the miscompile cases, used bugpoint to reduce them,
and opened individual PRs for several distinct bugs. So far we have: PR16455
(loop vectorizer), PR16457 (sccp), PR16460 (instcombine). Thanks again for doing
this! Do you plan on repeating this testing on a regular basis? Can it be
automated?

 -Hal
> 
> Thanks,
> -moh
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

Great job, Hal! 

Sure. I'd be happy to run icFuzz and report the fails once these bugs are
fixed and thereafter whenever people want new runs. Obviously, this can be
automated, but the problem is that icFuzz is not currently open sourced. Once
there's a bug in the compiler, there's really no limit in the number of
failing tests that can be generated, so it's more productive to run the
generator after the previously reported bugs are fixed.

We've also seen cases where the results of "clang -O2" are
different on Mac vs. Linux/Windows.

Just let me know when you want a new run.

Cheers,
-moh

-----Original Message-----
From: Hal Finkel [mailto:hfinkel at anl.gov] 
Sent: Wednesday, June 26, 2013 7:35 AM
To: Haghighat, Mohammad R
Cc: llvmdev at cs.uiuc.edu; Jim Grosbach
Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly generated
tests that fail on clang 3.4 trunk

----- Original Message -----
> 
> Hi Moh,
> 
> 
> Thanks for this. I’m really glad to see the work you’re doing in this
> area and believe it will be extremely helpful in improving the
> quality of the compiler.
> 
> 
> -Jim
> 
> 
> 
> On Jun 24, 2013, at 4:10 PM, Haghighat, Mohammad R <
> mohammad.r.haghighat at intel.com > wrote:
> 
> 
> 
> 
> 
> Hi,
> 
> I just submitted a bug report with a package containing 107 small
> test cases that fail on the latest LLVM/clang 3.4 main trunk
> (184563). Included are test sources, compilation commands, test
> input files, and results at –O0 and –O2 when applicable.
> 
> http://llvm.org/bugs/show_bug.cgi?id=16431
> 
> These tests have been automatically generated by an internal tool at
> Intel, the Intel Compiler fuzzer, icFuzz. The tests are typically
> very small. For example, for the following simple loop (test t5702)
> on MacOS X, clang at –O2 generates a binary that crashes:
> 
> // Test Loop Interchange
> for (j = 2; j < 76; j++) {
> for (jm = 1; jm < 30; jm++) {
> h[j-1][jm-1] = j + 83;
> }
> }
> 
> The tests are put in to two categories
> - tests that have different runtime outputs when compiled at -O0 and
> -O2 (this category also includes runtime crashes)
> - tests that cause infinite loops in the Clang optimizer
> 
> Many of these failing tests could be due to the same bug, thus a much
> smaller number of root problems are expected.
> 
> Any help with triaging these bugs would be highly appreciated.
I've gone through all of the miscompile cases, used bugpoint to reduce them,
and opened individual PRs for several distinct bugs. So far we have: PR16455
(loop vectorizer), PR16457 (sccp), PR16460 (instcombine). Thanks again for doing
this! Do you plan on repeating this testing on a regular basis? Can it be
automated?

 -Hal
> 
> Thanks,
> -moh
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

----- Original Message -----
> Great job, Hal!
> 
> Sure. I'd be happy to run icFuzz and report the fails once these bugs
> are fixed and thereafter whenever people want new runs. Obviously,
> this can be automated, but the problem is that icFuzz is not
> currently open sourced.
I would be happy to see this open sourced, but I think that we can work
something out regardless.

Also, once we get the current set of things resolved, I think it would be useful
to test running with:

 -- -O3, LTO (-O4 or -flto),
 -- -fslp-vectorize, -fslp-vectorize-aggressive (which are actually separate
optimizations)
 -- -ffast-math (if you can do floating point with tolerances, or at least
-ffinite-math-only), -fno-math-errno
 (and there are obviously a whole bunch of non-default code-generation and
target options).

Is it feasible to set up runs with different flags?
> Once there's a bug in the compiler, there's
> really no limit in the number of failing tests that can be
> generated, so it's more productive to run the generator after the
> previously reported bugs are fixed.
Agreed.
> 
> We've also seen cases where the results of "clang -O2" are
different
> on Mac vs. Linux/Windows.
I recall an issue related to default settings for FP, and differences with libm
implementation. Are there non-floating-point cases?
> 
> Just let me know when you want a new run.
Will do!

 -Hal
> 
> Cheers,
> -moh
> 
> -----Original Message-----
> From: Hal Finkel [mailto:hfinkel at anl.gov]
> Sent: Wednesday, June 26, 2013 7:35 AM
> To: Haghighat, Mohammad R
> Cc: llvmdev at cs.uiuc.edu; Jim Grosbach
> Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly
> generated tests that fail on clang 3.4 trunk
> 
> ----- Original Message -----
> > 
> > Hi Moh,
> > 
> > 
> > Thanks for this. I’m really glad to see the work you’re doing in
> > this
> > area and believe it will be extremely helpful in improving the
> > quality of the compiler.
> > 
> > 
> > -Jim
> > 
> > 
> > 
> > On Jun 24, 2013, at 4:10 PM, Haghighat, Mohammad R <
> > mohammad.r.haghighat at intel.com > wrote:
> > 
> > 
> > 
> > 
> > 
> > Hi,
> > 
> > I just submitted a bug report with a package containing 107 small
> > test cases that fail on the latest LLVM/clang 3.4 main trunk
> > (184563). Included are test sources, compilation commands, test
> > input files, and results at –O0 and –O2 when applicable.
> > 
> > http://llvm.org/bugs/show_bug.cgi?id=16431
> > 
> > These tests have been automatically generated by an internal tool
> > at
> > Intel, the Intel Compiler fuzzer, icFuzz. The tests are typically
> > very small. For example, for the following simple loop (test t5702)
> > on MacOS X, clang at –O2 generates a binary that crashes:
> > 
> > // Test Loop Interchange
> > for (j = 2; j < 76; j++) {
> > for (jm = 1; jm < 30; jm++) {
> > h[j-1][jm-1] = j + 83;
> > }
> > }
> > 
> > The tests are put in to two categories
> > - tests that have different runtime outputs when compiled at -O0
> > and
> > -O2 (this category also includes runtime crashes)
> > - tests that cause infinite loops in the Clang optimizer
> > 
> > Many of these failing tests could be due to the same bug, thus a
> > much
> > smaller number of root problems are expected.
> > 
> > Any help with triaging these bugs would be highly appreciated.
> 
> I've gone through all of the miscompile cases, used bugpoint to
> reduce them, and opened individual PRs for several distinct bugs. So
> far we have: PR16455 (loop vectorizer), PR16457 (sccp), PR16460
> (instcombine). Thanks again for doing this! Do you plan on repeating
> this testing on a regular basis? Can it be automated?
> 
>  -Hal
> 
> > 
> > Thanks,
> > -moh
> > 
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > 
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > 
> 
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory