llvm dev - Jul 2013 - [LLVMdev] [icFuzz] Help needed with analyzing randomly generated tests that fail on clang 3.4 trunk

Hal,

I ran the failing tests in the attachment to the bug 16431 on the latest clang
trunk (version 3.4 trunk 187225).
http://llvm.org/bugs/show_bug.cgi?id=16431

The following tests still fail:
              Tests in diff: t10236  t12206  t2581   t6734   t7788   t7820  
t8069   t9982
All tests in InfLoopInClang: t19193  t22300  t25903  t27872  t33143  t8543

Meanwhile, I'll launch a new run of icFuzz and will post the results later.

-moh

-----Original Message-----
From: Hal Finkel [mailto:hfinkel at anl.gov] 
Sent: Friday, July 26, 2013 8:33 AM
To: Haghighat, Mohammad R
Cc: llvmdev at cs.uiuc.edu
Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly generated
tests that fail on clang 3.4 trunk

----- Original Message -----
> ----- Original Message -----
> > Great job, Hal!
> > 
> > Sure. I'd be happy to run icFuzz and report the fails once these
> > bugs
> > are fixed and thereafter whenever people want new runs. Obviously,
> > this can be automated, but the problem is that icFuzz is not
> > currently open sourced.
> 
> I would be happy to see this open sourced, but I think that we can
> work something out regardless.
> 
> Also, once we get the current set of things resolved, I think it
> would be useful to test running with:
> 
>  -- -O3, LTO (-O4 or -flto),
>  -- -fslp-vectorize, -fslp-vectorize-aggressive (which are actually
>  separate optimizations)
>  -- -ffast-math (if you can do floating point with tolerances, or at
>  least -ffinite-math-only), -fno-math-errno
>  (and there are obviously a whole bunch of non-default
>  code-generation and target options).
> 
> Is it feasible to set up runs with different flags?
> 
> > Once there's a bug in the compiler, there's
> > really no limit in the number of failing tests that can be
> > generated, so it's more productive to run the generator after the
> > previously reported bugs are fixed.
> 
> Agreed.
> 
> > 
> > We've also seen cases where the results of "clang -O2"
are
> > different
> > on Mac vs. Linux/Windows.
> 
> I recall an issue related to default settings for FP, and differences
> with libm implementation. Are there non-floating-point cases?
> 
> > 
> > Just let me know when you want a new run.
> 
> Will do!
Mohammad,

Can you please re-run these now? I know that the original loop-vectorizer bugs
causing the miscompiles have been fixed, and the others also seem to have been
resolved as well.

Thanks again,
Hal
> 
>  -Hal
> 
> > 
> > Cheers,
> > -moh
> > 
> > -----Original Message-----
> > From: Hal Finkel [mailto:hfinkel at anl.gov]
> > Sent: Wednesday, June 26, 2013 7:35 AM
> > To: Haghighat, Mohammad R
> > Cc: llvmdev at cs.uiuc.edu; Jim Grosbach
> > Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly
> > generated tests that fail on clang 3.4 trunk
> > 
> > ----- Original Message -----
> > > 
> > > Hi Moh,
> > > 
> > > 
> > > Thanks for this. I’m really glad to see the work you’re doing in
> > > this
> > > area and believe it will be extremely helpful in improving the
> > > quality of the compiler.
> > > 
> > > 
> > > -Jim
> > > 
> > > 
> > > 
> > > On Jun 24, 2013, at 4:10 PM, Haghighat, Mohammad R <
> > > mohammad.r.haghighat at intel.com > wrote:
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Hi,
> > > 
> > > I just submitted a bug report with a package containing 107 small
> > > test cases that fail on the latest LLVM/clang 3.4 main trunk
> > > (184563). Included are test sources, compilation commands, test
> > > input files, and results at –O0 and –O2 when applicable.
> > > 
> > > http://llvm.org/bugs/show_bug.cgi?id=16431
> > > 
> > > These tests have been automatically generated by an internal tool
> > > at
> > > Intel, the Intel Compiler fuzzer, icFuzz. The tests are typically
> > > very small. For example, for the following simple loop (test
> > > t5702)
> > > on MacOS X, clang at –O2 generates a binary that crashes:
> > > 
> > > // Test Loop Interchange
> > > for (j = 2; j < 76; j++) {
> > > for (jm = 1; jm < 30; jm++) {
> > > h[j-1][jm-1] = j + 83;
> > > }
> > > }
> > > 
> > > The tests are put in to two categories
> > > - tests that have different runtime outputs when compiled at -O0
> > > and
> > > -O2 (this category also includes runtime crashes)
> > > - tests that cause infinite loops in the Clang optimizer
> > > 
> > > Many of these failing tests could be due to the same bug, thus a
> > > much
> > > smaller number of root problems are expected.
> > > 
> > > Any help with triaging these bugs would be highly appreciated.
> > 
> > I've gone through all of the miscompile cases, used bugpoint to
> > reduce them, and opened individual PRs for several distinct bugs.
> > So
> > far we have: PR16455 (loop vectorizer), PR16457 (sccp), PR16460
> > (instcombine). Thanks again for doing this! Do you plan on
> > repeating
> > this testing on a regular basis? Can it be automated?
> > 
> >  -Hal
> > 
> > > 
> > > Thanks,
> > > -moh
> > > 
> > > _______________________________________________
> > > LLVM Developers mailing list
> > > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > > 
> > > _______________________________________________
> > > LLVM Developers mailing list
> > > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > > 
> > 
> > --
> > Hal Finkel
> > Assistant Computational Scientist
> > Leadership Computing Facility
> > Argonne National Laboratory
> > 
> 
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

Hal,

Just posted a package containing 214 small tests showing bugs in the latest
Clang (3.4 trunk 187225) on MacOS X when compiled at -O2.
http://llvm.org/bugs/show_bug.cgi?id=16431

These are new tests different from the previously posted ones, but their root
causes could be the same as before or could actually be new bugs.

Cheers,
-moh

-----Original Message-----
From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On
Behalf Of Haghighat, Mohammad R
Sent: Friday, July 26, 2013 8:19 PM
To: Hal Finkel
Cc: llvmdev at cs.uiuc.edu
Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly generated
tests that fail on clang 3.4 trunk

Hal,

I ran the failing tests in the attachment to the bug 16431 on the latest clang
trunk (version 3.4 trunk 187225).
http://llvm.org/bugs/show_bug.cgi?id=16431

The following tests still fail:
              Tests in diff: t10236  t12206  t2581   t6734   t7788   t7820  
t8069   t9982
All tests in InfLoopInClang: t19193  t22300  t25903  t27872  t33143  t8543

Meanwhile, I'll launch a new run of icFuzz and will post the results later.

-moh

-----Original Message-----
From: Hal Finkel [mailto:hfinkel at anl.gov]
Sent: Friday, July 26, 2013 8:33 AM
To: Haghighat, Mohammad R
Cc: llvmdev at cs.uiuc.edu
Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly generated
tests that fail on clang 3.4 trunk

----- Original Message -----
> ----- Original Message -----
> > Great job, Hal!
> > 
> > Sure. I'd be happy to run icFuzz and report the fails once these 
> > bugs are fixed and thereafter whenever people want new runs. 
> > Obviously, this can be automated, but the problem is that icFuzz is 
> > not currently open sourced.
> 
> I would be happy to see this open sourced, but I think that we can 
> work something out regardless.
> 
> Also, once we get the current set of things resolved, I think it would 
> be useful to test running with:
> 
>  -- -O3, LTO (-O4 or -flto),
>  -- -fslp-vectorize, -fslp-vectorize-aggressive (which are actually  
> separate optimizations)
>  -- -ffast-math (if you can do floating point with tolerances, or at  
> least -ffinite-math-only), -fno-math-errno  (and there are obviously a 
> whole bunch of non-default  code-generation and target options).
> 
> Is it feasible to set up runs with different flags?
> 
> > Once there's a bug in the compiler, there's really no limit in
the
> > number of failing tests that can be generated, so it's more 
> > productive to run the generator after the previously reported bugs 
> > are fixed.
> 
> Agreed.
> 
> > 
> > We've also seen cases where the results of "clang -O2"
are different
> > on Mac vs. Linux/Windows.
> 
> I recall an issue related to default settings for FP, and differences 
> with libm implementation. Are there non-floating-point cases?
> 
> > 
> > Just let me know when you want a new run.
> 
> Will do!
Mohammad,

Can you please re-run these now? I know that the original loop-vectorizer bugs
causing the miscompiles have been fixed, and the others also seem to have been
resolved as well.

Thanks again,
Hal
> 
>  -Hal
> 
> > 
> > Cheers,
> > -moh
> > 
> > -----Original Message-----
> > From: Hal Finkel [mailto:hfinkel at anl.gov]
> > Sent: Wednesday, June 26, 2013 7:35 AM
> > To: Haghighat, Mohammad R
> > Cc: llvmdev at cs.uiuc.edu; Jim Grosbach
> > Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly 
> > generated tests that fail on clang 3.4 trunk
> > 
> > ----- Original Message -----
> > > 
> > > Hi Moh,
> > > 
> > > 
> > > Thanks for this. I’m really glad to see the work you’re doing in 
> > > this area and believe it will be extremely helpful in improving 
> > > the quality of the compiler.
> > > 
> > > 
> > > -Jim
> > > 
> > > 
> > > 
> > > On Jun 24, 2013, at 4:10 PM, Haghighat, Mohammad R < 
> > > mohammad.r.haghighat at intel.com > wrote:
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Hi,
> > > 
> > > I just submitted a bug report with a package containing 107 small
> > > test cases that fail on the latest LLVM/clang 3.4 main trunk 
> > > (184563). Included are test sources, compilation commands, test 
> > > input files, and results at –O0 and –O2 when applicable.
> > > 
> > > http://llvm.org/bugs/show_bug.cgi?id=16431
> > > 
> > > These tests have been automatically generated by an internal tool
> > > at Intel, the Intel Compiler fuzzer, icFuzz. The tests are 
> > > typically very small. For example, for the following simple loop 
> > > (test
> > > t5702)
> > > on MacOS X, clang at –O2 generates a binary that crashes:
> > > 
> > > // Test Loop Interchange
> > > for (j = 2; j < 76; j++) {
> > > for (jm = 1; jm < 30; jm++) {
> > > h[j-1][jm-1] = j + 83;
> > > }
> > > }
> > > 
> > > The tests are put in to two categories
> > > - tests that have different runtime outputs when compiled at -O0 
> > > and
> > > -O2 (this category also includes runtime crashes)
> > > - tests that cause infinite loops in the Clang optimizer
> > > 
> > > Many of these failing tests could be due to the same bug, thus a 
> > > much smaller number of root problems are expected.
> > > 
> > > Any help with triaging these bugs would be highly appreciated.
> > 
> > I've gone through all of the miscompile cases, used bugpoint to 
> > reduce them, and opened individual PRs for several distinct bugs.
> > So
> > far we have: PR16455 (loop vectorizer), PR16457 (sccp), PR16460 
> > (instcombine). Thanks again for doing this! Do you plan on repeating 
> > this testing on a regular basis? Can it be automated?
> > 
> >  -Hal
> > 
> > > 
> > > Thanks,
> > > -moh
> > > 
> > > _______________________________________________
> > > LLVM Developers mailing list
> > > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu 
> > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > > 
> > > _______________________________________________
> > > LLVM Developers mailing list
> > > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > > 
> > 
> > --
> > Hal Finkel
> > Assistant Computational Scientist
> > Leadership Computing Facility
> > Argonne National Laboratory
> > 
> 
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
--
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

----- Original Message -----
> Hal,
> 
> Just posted a package containing 214 small tests showing bugs in the
> latest Clang (3.4 trunk 187225) on MacOS X when compiled at -O2.
> http://llvm.org/bugs/show_bug.cgi?id=16431
> 
> These are new tests different from the previously posted ones, but
> their root causes could be the same as before or could actually be
> new bugs.
Great, thanks! I'll go through them.

 -Hal
> 
> Cheers,
> -moh
> 
> -----Original Message-----
> From: llvmdev-bounces at cs.uiuc.edu
> [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Haghighat,
> Mohammad R
> Sent: Friday, July 26, 2013 8:19 PM
> To: Hal Finkel
> Cc: llvmdev at cs.uiuc.edu
> Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly
> generated tests that fail on clang 3.4 trunk
> 
> Hal,
> 
> I ran the failing tests in the attachment to the bug 16431 on the
> latest clang trunk (version 3.4 trunk 187225).
> http://llvm.org/bugs/show_bug.cgi?id=16431
> 
> The following tests still fail:
>               Tests in diff: t10236  t12206  t2581   t6734   t7788
>                 t7820   t8069   t9982
> All tests in InfLoopInClang: t19193  t22300  t25903  t27872  t33143
>  t8543
> 
> Meanwhile, I'll launch a new run of icFuzz and will post the results
> later.
> 
> -moh
> 
> -----Original Message-----
> From: Hal Finkel [mailto:hfinkel at anl.gov]
> Sent: Friday, July 26, 2013 8:33 AM
> To: Haghighat, Mohammad R
> Cc: llvmdev at cs.uiuc.edu
> Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing randomly
> generated tests that fail on clang 3.4 trunk
> 
> ----- Original Message -----
> > ----- Original Message -----
> > > Great job, Hal!
> > > 
> > > Sure. I'd be happy to run icFuzz and report the fails once
these
> > > bugs are fixed and thereafter whenever people want new runs.
> > > Obviously, this can be automated, but the problem is that icFuzz
> > > is
> > > not currently open sourced.
> > 
> > I would be happy to see this open sourced, but I think that we can
> > work something out regardless.
> > 
> > Also, once we get the current set of things resolved, I think it
> > would
> > be useful to test running with:
> > 
> >  -- -O3, LTO (-O4 or -flto),
> >  -- -fslp-vectorize, -fslp-vectorize-aggressive (which are actually
> > separate optimizations)
> >  -- -ffast-math (if you can do floating point with tolerances, or
> >  at
> > least -ffinite-math-only), -fno-math-errno  (and there are
> > obviously a
> > whole bunch of non-default  code-generation and target options).
> > 
> > Is it feasible to set up runs with different flags?
> > 
> > > Once there's a bug in the compiler, there's really no
limit in
> > > the
> > > number of failing tests that can be generated, so it's more
> > > productive to run the generator after the previously reported
> > > bugs
> > > are fixed.
> > 
> > Agreed.
> > 
> > > 
> > > We've also seen cases where the results of "clang
-O2" are
> > > different
> > > on Mac vs. Linux/Windows.
> > 
> > I recall an issue related to default settings for FP, and
> > differences
> > with libm implementation. Are there non-floating-point cases?
> > 
> > > 
> > > Just let me know when you want a new run.
> > 
> > Will do!
> 
> Mohammad,
> 
> Can you please re-run these now? I know that the original
> loop-vectorizer bugs causing the miscompiles have been fixed, and
> the others also seem to have been resolved as well.
> 
> Thanks again,
> Hal
> 
> > 
> >  -Hal
> > 
> > > 
> > > Cheers,
> > > -moh
> > > 
> > > -----Original Message-----
> > > From: Hal Finkel [mailto:hfinkel at anl.gov]
> > > Sent: Wednesday, June 26, 2013 7:35 AM
> > > To: Haghighat, Mohammad R
> > > Cc: llvmdev at cs.uiuc.edu; Jim Grosbach
> > > Subject: Re: [LLVMdev] [icFuzz] Help needed with analyzing
> > > randomly
> > > generated tests that fail on clang 3.4 trunk
> > > 
> > > ----- Original Message -----
> > > > 
> > > > Hi Moh,
> > > > 
> > > > 
> > > > Thanks for this. I’m really glad to see the work you’re
doing
> > > > in
> > > > this area and believe it will be extremely helpful in
improving
> > > > the quality of the compiler.
> > > > 
> > > > 
> > > > -Jim
> > > > 
> > > > 
> > > > 
> > > > On Jun 24, 2013, at 4:10 PM, Haghighat, Mohammad R <
> > > > mohammad.r.haghighat at intel.com > wrote:
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > Hi,
> > > > 
> > > > I just submitted a bug report with a package containing 107
> > > > small
> > > > test cases that fail on the latest LLVM/clang 3.4 main trunk
> > > > (184563). Included are test sources, compilation commands,
test
> > > > input files, and results at –O0 and –O2 when applicable.
> > > > 
> > > > http://llvm.org/bugs/show_bug.cgi?id=16431
> > > > 
> > > > These tests have been automatically generated by an internal
> > > > tool
> > > > at Intel, the Intel Compiler fuzzer, icFuzz. The tests are
> > > > typically very small. For example, for the following simple
> > > > loop
> > > > (test
> > > > t5702)
> > > > on MacOS X, clang at –O2 generates a binary that crashes:
> > > > 
> > > > // Test Loop Interchange
> > > > for (j = 2; j < 76; j++) {
> > > > for (jm = 1; jm < 30; jm++) {
> > > > h[j-1][jm-1] = j + 83;
> > > > }
> > > > }
> > > > 
> > > > The tests are put in to two categories
> > > > - tests that have different runtime outputs when compiled at
> > > > -O0
> > > > and
> > > > -O2 (this category also includes runtime crashes)
> > > > - tests that cause infinite loops in the Clang optimizer
> > > > 
> > > > Many of these failing tests could be due to the same bug,
thus
> > > > a
> > > > much smaller number of root problems are expected.
> > > > 
> > > > Any help with triaging these bugs would be highly
appreciated.
> > > 
> > > I've gone through all of the miscompile cases, used bugpoint
to
> > > reduce them, and opened individual PRs for several distinct bugs.
> > > So
> > > far we have: PR16455 (loop vectorizer), PR16457 (sccp), PR16460
> > > (instcombine). Thanks again for doing this! Do you plan on
> > > repeating
> > > this testing on a regular basis? Can it be automated?
> > > 
> > >  -Hal
> > > 
> > > > 
> > > > Thanks,
> > > > -moh
> > > > 
> > > > _______________________________________________
> > > > LLVM Developers mailing list
> > > > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> > > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > > > 
> > > > _______________________________________________
> > > > LLVM Developers mailing list
> > > > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > > > 
> > > 
> > > --
> > > Hal Finkel
> > > Assistant Computational Scientist
> > > Leadership Computing Facility
> > > Argonne National Laboratory
> > > 
> > 
> > --
> > Hal Finkel
> > Assistant Computational Scientist
> > Leadership Computing Facility
> > Argonne National Laboratory
> > 
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > 
> 
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory