llvm dev - Dec 2018 - Testing compiler reliability using Csmith

Hello everyone!

We are working on writing a paper about testing the reliability of C compilers
by using Csmith (a random C99 program generator).

A previous testing effort, using Csmith, found 202 LLVM bugs, which represented
2% of all reported bugs at that time (PDF:
https://www.flux.utah.edu/download?uid=114
<https://www.flux.utah.edu/download?uid=114>): . However, after this paper
was published we are unaware of any further testing using Csmith, and we would
like to ask you if you are aware of any such efforts or further results.

Best regards,
Radu Ometita,
Functional compilers engineer @IOHK
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181207/c445e837/attachment.html>

Hello,

+ Zhendong, who’s done a lot of work on automated testing of llvm.
> On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hello everyone!
> 
> We are working on writing a paper about testing the reliability of C
compilers by using Csmith (a random C99 program generator).
> 
> A previous testing effort, using Csmith, found 202 LLVM bugs, which
represented 2% of all reported bugs at that time (PDF:
https://www.flux.utah.edu/download?uid=114
<https://www.flux.utah.edu/download?uid=114>): . However, after this paper
was published we are unaware of any further testing using Csmith, and we would
like to ask you if you are aware of any such efforts or further results.
Just speaking for myself here, I use Csmith as part of my pre-commit testing.

vedant
> 
> Best regards,
> Radu Ometita,
> Functional compilers engineer @IOHK
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181207/794b16af/attachment.html>

Thanks, Vedant.  Yes, we have done a lot of testing of Clang/LLVM (and GCC)
in the past several years (more details at
https://people.inf.ethz.ch/suz/emi/index.html):

[GCC/LLVM bugs: *1,602* (total) / *1,007* (fixed)]
[Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2
<http://tinyurl.com/oxlkmjc>, link3 <http://tinyurl.com/jd3jnl3>,
link4
<http://tinyurl.com/zs2826l>, link5 <https://tinyurl.com/yb7okzqd>),
LLVM (
link1 <http://tinyurl.com/qj8x5ry>, link2
<http://tinyurl.com/nc9celr>,
link3 <http://tinyurl.com/qy58smb>, link4
<http://tinyurl.com/z7y8rwo>,
link5 <https://tinyurl.com/y7wdf5b4>)]

Best,
Zhendong

--------
Zhendong Su
Professor, Department of Computer Science, ETH Zurich
Office: CAB E 64.1, Tel: +41 44 633 77 72
https://people.inf.ethz.ch/suz/
--------

On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com> wrote:
> Hello,
>
> + Zhendong, who’s done a lot of work on automated testing of llvm.
>
> On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hello everyone!
>
> We are working on writing a paper about testing the reliability of C
> compilers by using Csmith (a random C99 program generator).
>
> A previous testing effort, using Csmith, found 202 LLVM bugs, which
> represented 2% of all reported bugs at that time (PDF:
> https://www.flux.utah.edu/download?uid=114): . However, after this paper
> was published we are unaware of any further testing using Csmith, and we
> would like to ask you if you are aware of any such efforts or further
> results.
>
>
> Just speaking for myself here, I use Csmith as part of my pre-commit
> testing.
>
> vedant
>
>
> Best regards,
> Radu Ometita,
> Functional compilers engineer @IOHK
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181207/c71bda15/attachment.html>