Matt P. Dziubinski via llvm-dev
2018-Dec-12 16:54 UTC
[llvm-dev] Testing compiler reliability using Csmith
You may also be interested in the following resources on compilers correctness (articles, software, and talks -- from the general topics to the ones specifically focused on testing, validation, and verification): https://github.com/MattPD/cpplinks/blob/master/compilers.correctness.md Best regards, Matt P. Dziubinski On 12/7/2018 20:19, Philip Reames via llvm-dev wrote:> There's also a couple other active fuzzing efforts running concurrently: > > * OSSFuzz is continuously running a set of codegen and individual > pass fuzzers. A bit more information can be found here: > http://llvm.org/docs/FuzzingLLVM.html#continuously-running-and-finding-bugs > * We (Azul Systems) have a continuously running Java fuzzer which > exercises LLVM through our Falcon compiler, and regularly finds > regressions and occasional deep long lurking issues. We don't > have a public bug tracker for this, but a sizable portion of our > upstream bug fixing activity is driven by the output of this tool. > > Philip > > On 12/7/18 11:04 AM, Zhendong Su via llvm-dev wrote: >> Thanks, Vedant. Yes, we have done a lot of testing of Clang/LLVM >> (and GCC) in the past several years (more details at >> https://people.inf.ethz.ch/suz/emi/index.html): >> >> [GCC/LLVM bugs: *1,602* (total) / *1,007* (fixed)] >> [Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2 >> <http://tinyurl.com/oxlkmjc>, link3 <http://tinyurl.com/jd3jnl3>, >> link4 <http://tinyurl.com/zs2826l>, link5 >> <https://tinyurl.com/yb7okzqd>), LLVM (link1 >> <http://tinyurl.com/qj8x5ry>, link2 <http://tinyurl.com/nc9celr>, >> link3 <http://tinyurl.com/qy58smb>, link4 >> <http://tinyurl.com/z7y8rwo>, link5 <https://tinyurl.com/y7wdf5b4>)] >> >> Best, >> Zhendong >> >> -------- >> Zhendong Su >> Professor, Department of Computer Science, ETH Zurich >> Office: CAB E 64.1, Tel: +41 44 633 77 72 >> https://people.inf.ethz.ch/suz/ >> -------- >> >> On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com >> <mailto:vsk at apple.com>> wrote: >> >> Hello, >> >> + Zhendong, who’s done a lot of work on automated testing of llvm. >> >>> On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev >>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> Hello everyone! >>> >>> We are working on writing a paper about testing the reliability >>> of C compilers by using Csmith (a random C99 program generator). >>> >>> A previous testing effort, using Csmith, found 202 LLVM bugs, >>> which represented 2% of all reported bugs at that time (PDF: >>> https://www.flux.utah.edu/download?uid=114): . However, after >>> this paper was published we are unaware of any further testing >>> using Csmith, and we would like to ask you if you are aware of >>> any such efforts or further results. >> >> Just speaking for myself here, I use Csmith as part of my >> pre-commit testing. >> >> vedant >> >>> >>> Best regards, >>> Radu Ometita, >>> Functional compilers engineer @IOHK >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181212/52a14ac5/attachment.html>
Doerfert, Johannes Rudolf via llvm-dev
2018-Dec-24 13:10 UTC
[llvm-dev] Testing compiler reliability using Csmith
I just saw this related paper today on Hacker News: https://news.ycombinator.com/item?id=18748193 On 12/12, Matt P. Dziubinski via llvm-dev wrote:> You may also be interested in the following resources on compilers > correctness (articles, software, and talks -- from the general topics to the > ones specifically focused on testing, validation, and verification): > > https://github.com/MattPD/cpplinks/blob/master/compilers.correctness.md > > Best regards, > > Matt P. Dziubinski > > On 12/7/2018 20:19, Philip Reames via llvm-dev wrote: > > > There's also a couple other active fuzzing efforts running concurrently: > > > > * OSSFuzz is continuously running a set of codegen and individual > > pass fuzzers. A bit more information can be found here: > > http://llvm.org/docs/FuzzingLLVM.html#continuously-running-and-finding-bugs > > * We (Azul Systems) have a continuously running Java fuzzer which > > exercises LLVM through our Falcon compiler, and regularly finds > > regressions and occasional deep long lurking issues. We don't > > have a public bug tracker for this, but a sizable portion of our > > upstream bug fixing activity is driven by the output of this tool. > > > > Philip > > > > On 12/7/18 11:04 AM, Zhendong Su via llvm-dev wrote: > > > Thanks, Vedant. Yes, we have done a lot of testing of Clang/LLVM > > > (and GCC) in the past several years (more details at > > > https://people.inf.ethz.ch/suz/emi/index.html): > > > > > > [GCC/LLVM bugs: *1,602* (total) / *1,007* (fixed)] > > > [Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2 > > > <http://tinyurl.com/oxlkmjc>, link3 <http://tinyurl.com/jd3jnl3>, > > > link4 <http://tinyurl.com/zs2826l>, link5 > > > <https://tinyurl.com/yb7okzqd>), LLVM (link1 > > > <http://tinyurl.com/qj8x5ry>, link2 <http://tinyurl.com/nc9celr>, > > > link3 <http://tinyurl.com/qy58smb>, link4 > > > <http://tinyurl.com/z7y8rwo>, link5 <https://tinyurl.com/y7wdf5b4>)] > > > > > > Best, > > > Zhendong > > > > > > -------- > > > Zhendong Su > > > Professor, Department of Computer Science, ETH Zurich > > > Office: CAB E 64.1, Tel: +41 44 633 77 72 > > > https://people.inf.ethz.ch/suz/ > > > -------- > > > > > > On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com > > > <mailto:vsk at apple.com>> wrote: > > > > > > Hello, > > > > > > + Zhendong, who’s done a lot of work on automated testing of llvm. > > > > > > > On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev > > > > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: > > > > > > > > Hello everyone! > > > > > > > > We are working on writing a paper about testing the reliability > > > > of C compilers by using Csmith (a random C99 program generator). > > > > > > > > A previous testing effort, using Csmith, found 202 LLVM bugs, > > > > which represented 2% of all reported bugs at that time (PDF: > > > > https://www.flux.utah.edu/download?uid=114): . However, after > > > > this paper was published we are unaware of any further testing > > > > using Csmith, and we would like to ask you if you are aware of > > > > any such efforts or further results. > > > > > > Just speaking for myself here, I use Csmith as part of my > > > pre-commit testing. > > > > > > vedant > > > > > > > > > > > Best regards, > > > > Radu Ometita, > > > > Functional compilers engineer @IOHK > > > >> _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- Johannes Doerfert Researcher Argonne National Laboratory Lemont, IL 60439, USA jdoerfert at anl.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181224/46f4cb7f/attachment.sig>
Matt P. Dziubinski via llvm-dev
2018-Dec-25 15:26 UTC
[llvm-dev] Testing compiler reliability using Csmith
Hi Johannes! Updated; thank you! Best, Matt On 12/24/2018 14:10, Doerfert, Johannes Rudolf wrote:> I just saw this related paper today on Hacker News: > > https://news.ycombinator.com/item?id=18748193 > > > On 12/12, Matt P. Dziubinski via llvm-dev wrote: >> You may also be interested in the following resources on compilers >> correctness (articles, software, and talks -- from the general topics to the >> ones specifically focused on testing, validation, and verification): >> >> https://github.com/MattPD/cpplinks/blob/master/compilers.correctness.md >> >> Best regards, >> >> Matt P. Dziubinski >> >> On 12/7/2018 20:19, Philip Reames via llvm-dev wrote: >> >>> There's also a couple other active fuzzing efforts running concurrently: >>> >>> * OSSFuzz is continuously running a set of codegen and individual >>> pass fuzzers. A bit more information can be found here: >>> http://llvm.org/docs/FuzzingLLVM.html#continuously-running-and-finding-bugs >>> * We (Azul Systems) have a continuously running Java fuzzer which >>> exercises LLVM through our Falcon compiler, and regularly finds >>> regressions and occasional deep long lurking issues. We don't >>> have a public bug tracker for this, but a sizable portion of our >>> upstream bug fixing activity is driven by the output of this tool. >>> >>> Philip >>> >>> On 12/7/18 11:04 AM, Zhendong Su via llvm-dev wrote: >>>> Thanks, Vedant. Yes, we have done a lot of testing of Clang/LLVM >>>> (and GCC) in the past several years (more details at >>>> https://people.inf.ethz.ch/suz/emi/index.html): >>>> >>>> [GCC/LLVM bugs: *1,602* (total) / *1,007* (fixed)] >>>> [Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2 >>>> <http://tinyurl.com/oxlkmjc>, link3 <http://tinyurl.com/jd3jnl3>, >>>> link4 <http://tinyurl.com/zs2826l>, link5 >>>> <https://tinyurl.com/yb7okzqd>), LLVM (link1 >>>> <http://tinyurl.com/qj8x5ry>, link2 <http://tinyurl.com/nc9celr>, >>>> link3 <http://tinyurl.com/qy58smb>, link4 >>>> <http://tinyurl.com/z7y8rwo>, link5 <https://tinyurl.com/y7wdf5b4>)] >>>> >>>> Best, >>>> Zhendong >>>> >>>> -------- >>>> Zhendong Su >>>> Professor, Department of Computer Science, ETH Zurich >>>> Office: CAB E 64.1, Tel: +41 44 633 77 72 >>>> https://people.inf.ethz.ch/suz/ >>>> -------- >>>> >>>> On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com >>>> <mailto:vsk at apple.com>> wrote: >>>> >>>> Hello, >>>> >>>> + Zhendong, who’s done a lot of work on automated testing of llvm. >>>> >>>>> On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev >>>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>>>> >>>>> Hello everyone! >>>>> >>>>> We are working on writing a paper about testing the reliability >>>>> of C compilers by using Csmith (a random C99 program generator). >>>>> >>>>> A previous testing effort, using Csmith, found 202 LLVM bugs, >>>>> which represented 2% of all reported bugs at that time (PDF: >>>>> https://www.flux.utah.edu/download?uid=114): . However, after >>>>> this paper was published we are unaware of any further testing >>>>> using Csmith, and we would like to ask you if you are aware of >>>>> any such efforts or further results. >>>> Just speaking for myself here, I use Csmith as part of my >>>> pre-commit testing. >>>> >>>> vedant >>>> >>>>> Best regards, >>>>> Radu Ometita, >>>>> Functional compilers engineer @IOHK >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >