Zhendong Su via llvm-dev
2018-Dec-07 19:04 UTC
[llvm-dev] Testing compiler reliability using Csmith
Thanks, Vedant. Yes, we have done a lot of testing of Clang/LLVM (and GCC) in the past several years (more details at https://people.inf.ethz.ch/suz/emi/index.html): [GCC/LLVM bugs: *1,602* (total) / *1,007* (fixed)] [Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2 <http://tinyurl.com/oxlkmjc>, link3 <http://tinyurl.com/jd3jnl3>, link4 <http://tinyurl.com/zs2826l>, link5 <https://tinyurl.com/yb7okzqd>), LLVM ( link1 <http://tinyurl.com/qj8x5ry>, link2 <http://tinyurl.com/nc9celr>, link3 <http://tinyurl.com/qy58smb>, link4 <http://tinyurl.com/z7y8rwo>, link5 <https://tinyurl.com/y7wdf5b4>)] Best, Zhendong -------- Zhendong Su Professor, Department of Computer Science, ETH Zurich Office: CAB E 64.1, Tel: +41 44 633 77 72 https://people.inf.ethz.ch/suz/ -------- On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com> wrote:> Hello, > > + Zhendong, who’s done a lot of work on automated testing of llvm. > > On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > Hello everyone! > > We are working on writing a paper about testing the reliability of C > compilers by using Csmith (a random C99 program generator). > > A previous testing effort, using Csmith, found 202 LLVM bugs, which > represented 2% of all reported bugs at that time (PDF: > https://www.flux.utah.edu/download?uid=114): . However, after this paper > was published we are unaware of any further testing using Csmith, and we > would like to ask you if you are aware of any such efforts or further > results. > > > Just speaking for myself here, I use Csmith as part of my pre-commit > testing. > > vedant > > > Best regards, > Radu Ometita, > Functional compilers engineer @IOHK > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181207/c71bda15/attachment.html>
Philip Reames via llvm-dev
2018-Dec-07 19:19 UTC
[llvm-dev] Testing compiler reliability using Csmith
There's also a couple other active fuzzing efforts running concurrently:
* OSSFuzz is continuously running a set of codegen and individual pass
fuzzers. A bit more information can be found here:
http://llvm.org/docs/FuzzingLLVM.html#continuously-running-and-finding-bugs
* We (Azul Systems) have a continuously running Java fuzzer which
exercises LLVM through our Falcon compiler, and regularly finds
regressions and occasional deep long lurking issues. We don't have
a public bug tracker for this, but a sizable portion of our upstream
bug fixing activity is driven by the output of this tool.
Philip
On 12/7/18 11:04 AM, Zhendong Su via llvm-dev wrote:> Thanks, Vedant. Yes, we have done a lot of testing of Clang/LLVM (and
> GCC) in the past several years (more details at
> https://people.inf.ethz.ch/suz/emi/index.html):
>
> [GCC/LLVM bugs: *1,602* (total) / *1,007* (fixed)]
> [Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2
> <http://tinyurl.com/oxlkmjc>, link3
<http://tinyurl.com/jd3jnl3>,
> link4 <http://tinyurl.com/zs2826l>, link5
> <https://tinyurl.com/yb7okzqd>), LLVM (link1
> <http://tinyurl.com/qj8x5ry>, link2
<http://tinyurl.com/nc9celr>,
> link3 <http://tinyurl.com/qy58smb>, link4
> <http://tinyurl.com/z7y8rwo>, link5
<https://tinyurl.com/y7wdf5b4>)]
>
> Best,
> Zhendong
>
> --------
> Zhendong Su
> Professor, Department of Computer Science, ETH Zurich
> Office: CAB E 64.1, Tel: +41 44 633 77 72
> https://people.inf.ethz.ch/suz/
> --------
>
> On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com
> <mailto:vsk at apple.com>> wrote:
>
> Hello,
>
> + Zhendong, who’s done a lot of work on automated testing of llvm.
>
>> On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev
>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>
>> Hello everyone!
>>
>> We are working on writing a paper about testing the reliability
>> of C compilers by using Csmith (a random C99 program generator).
>>
>> A previous testing effort, using Csmith, found 202 LLVM bugs,
>> which represented 2% of all reported bugs at that time (PDF:
>> https://www.flux.utah.edu/download?uid=114): . However, after
>> this paper was published we are unaware of any further testing
>> using Csmith, and we would like to ask you if you are aware of
>> any such efforts or further results.
>
> Just speaking for myself here, I use Csmith as part of my
> pre-commit testing.
>
> vedant
>
>>
>> Best regards,
>> Radu Ometita,
>> Functional compilers engineer @IOHK
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181207/b4da2884/attachment.html>
Matt P. Dziubinski via llvm-dev
2018-Dec-12 16:54 UTC
[llvm-dev] Testing compiler reliability using Csmith
You may also be interested in the following resources on compilers correctness (articles, software, and talks -- from the general topics to the ones specifically focused on testing, validation, and verification): https://github.com/MattPD/cpplinks/blob/master/compilers.correctness.md Best regards, Matt P. Dziubinski On 12/7/2018 20:19, Philip Reames via llvm-dev wrote:> There's also a couple other active fuzzing efforts running concurrently: > > * OSSFuzz is continuously running a set of codegen and individual > pass fuzzers. A bit more information can be found here: > http://llvm.org/docs/FuzzingLLVM.html#continuously-running-and-finding-bugs > * We (Azul Systems) have a continuously running Java fuzzer which > exercises LLVM through our Falcon compiler, and regularly finds > regressions and occasional deep long lurking issues. We don't > have a public bug tracker for this, but a sizable portion of our > upstream bug fixing activity is driven by the output of this tool. > > Philip > > On 12/7/18 11:04 AM, Zhendong Su via llvm-dev wrote: >> Thanks, Vedant. Yes, we have done a lot of testing of Clang/LLVM >> (and GCC) in the past several years (more details at >> https://people.inf.ethz.ch/suz/emi/index.html): >> >> [GCC/LLVM bugs: *1,602* (total) / *1,007* (fixed)] >> [Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2 >> <http://tinyurl.com/oxlkmjc>, link3 <http://tinyurl.com/jd3jnl3>, >> link4 <http://tinyurl.com/zs2826l>, link5 >> <https://tinyurl.com/yb7okzqd>), LLVM (link1 >> <http://tinyurl.com/qj8x5ry>, link2 <http://tinyurl.com/nc9celr>, >> link3 <http://tinyurl.com/qy58smb>, link4 >> <http://tinyurl.com/z7y8rwo>, link5 <https://tinyurl.com/y7wdf5b4>)] >> >> Best, >> Zhendong >> >> -------- >> Zhendong Su >> Professor, Department of Computer Science, ETH Zurich >> Office: CAB E 64.1, Tel: +41 44 633 77 72 >> https://people.inf.ethz.ch/suz/ >> -------- >> >> On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com >> <mailto:vsk at apple.com>> wrote: >> >> Hello, >> >> + Zhendong, who’s done a lot of work on automated testing of llvm. >> >>> On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev >>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> Hello everyone! >>> >>> We are working on writing a paper about testing the reliability >>> of C compilers by using Csmith (a random C99 program generator). >>> >>> A previous testing effort, using Csmith, found 202 LLVM bugs, >>> which represented 2% of all reported bugs at that time (PDF: >>> https://www.flux.utah.edu/download?uid=114): . However, after >>> this paper was published we are unaware of any further testing >>> using Csmith, and we would like to ask you if you are aware of >>> any such efforts or further results. >> >> Just speaking for myself here, I use Csmith as part of my >> pre-commit testing. >> >> vedant >> >>> >>> Best regards, >>> Radu Ometita, >>> Functional compilers engineer @IOHK >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181212/52a14ac5/attachment.html>