On Wed, Feb 1, 2017 at 8:45 AM, Mehdi Amini <mehdi.amini at apple.com> wrote:> > > On Feb 1, 2017, at 8:34 AM, Michael Kruse via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > > > Hi all, > > > > The blog entry [1] suggest that one of the buildbots constantly fuzzes > > clang and clang-format. However, the actual bot [2] only tests the > > fuzzer itself over a well-known set of bugs in standard software (eg. > > Heartbleed [3] seems to be among them). > > Isn’t it this stage? http://lab.llvm.org:8011/build > ers/sanitizer-x86_64-linux-fuzzer/builds/2755/steps/ > stage2%2Fasan%2Bassertions%20check-fuzzer/logs/stdio > > > Has there actually ever been a > > buildbot that fuzzes clang/LLVM itself? >Yes, I used to run clang-fuzzer and clang-format-fuzzer on this bot, but not any more. The reason is simple -- the bot was always red (well, orange) and the bugs were never fixed. Currently we run clang-fuzzer (but not clang-format-fuzzer) on our internal fuzzing infra and Richard has fixed at least one bug found this way. http://llvm.org/viewvc/llvm-project?view=revision&revision=291030 My llvm fuzzing bot was pretty naive and simple. If we want proper continuous fuzzing for parts of LLVM we either need to build a separate "real" continuous fuzzing process, or use an existing one. Luckily, there is one :) As a pilot I've recently added the cxa_demangler_fuzzer to OSS-Fuzz <https://github.com/google/oss-fuzz>: https://github.com/google/oss-fuzz/tree/master/projects/llvm_libcxxabi It even found one bug which Mehdi already fixed! http://llvm.org/viewvc/llvm-project?view=revision&revision=293330 The bug report itself will become public in ~4 days: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=370 If we want to run some more llvm fuzzers on OSS-Fuzz I'd be happy to (help) set them up.> > > > Another (obvious?) fuzzing candidate would be the LLVM's bitcode > > reader. I ran afl-fuzz on it and it found lots of failed assertions > > within seconds. Isn't fuzzing done on a regular basis as [1] suggests > > should be done? Should I report the crashes found by it? > > The bitcode reader is known to not be robust against malformed inputs. >Yes, I afraid the bitcode reader (as some other parts of LLVM) are not robust enough to withstand fuzzing. :( Note that if we want to use libFuzzer (which is an in-process fuzzer) the target should not assert/abort/exit on any input (if it's not a bug). --kcc> > — > Mehdi-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170201/33ce1dd8/attachment-0001.html>
2017-02-01 18:07 GMT+01:00 Kostya Serebryany <kcc at google.com>:> Yes, I used to run clang-fuzzer and clang-format-fuzzer on this bot, but not > any more. > The reason is simple -- the bot was always red (well, orange) and the bugs > were never fixed. > > Currently we run clang-fuzzer (but not clang-format-fuzzer) on our internal > fuzzing infra > and Richard has fixed at least one bug found this way. > http://llvm.org/viewvc/llvm-project?view=revision&revision=291030 > > My llvm fuzzing bot was pretty naive and simple. > If we want proper continuous fuzzing for parts of LLVM we either need to > build a separate "real" continuous fuzzing process, > or use an existing one. Luckily, there is one :) > As a pilot I've recently added the cxa_demangler_fuzzer to OSS-Fuzz: > https://github.com/google/oss-fuzz/tree/master/projects/llvm_libcxxabi > It even found one bug which Mehdi already fixed! > http://llvm.org/viewvc/llvm-project?view=revision&revision=293330 > The bug report itself will become public in ~4 days: > https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=370Thanks for the explanation.>> > Another (obvious?) fuzzing candidate would be the LLVM's bitcode >> > reader. I ran afl-fuzz on it and it found lots of failed assertions >> > within seconds. Isn't fuzzing done on a regular basis as [1] suggests >> > should be done? Should I report the crashes found by it? >> >> The bitcode reader is known to not be robust against malformed inputs. > > > Yes, I afraid the bitcode reader (as some other parts of LLVM) are not > robust enough to withstand fuzzing. :( > Note that if we want to use libFuzzer (which is an in-process fuzzer) the > target should not assert/abort/exit on any input (if it's not a bug).Is there any incentive to change that? A google Summer of Code project maybe? Michael
On Wed, Feb 1, 2017 at 9:19 AM, Michael Kruse <llvmdev at meinersbur.de> wrote:> 2017-02-01 18:07 GMT+01:00 Kostya Serebryany <kcc at google.com>: > > Yes, I used to run clang-fuzzer and clang-format-fuzzer on this bot, but > not > > any more. > > The reason is simple -- the bot was always red (well, orange) and the > bugs > > were never fixed. > > > > Currently we run clang-fuzzer (but not clang-format-fuzzer) on our > internal > > fuzzing infra > > and Richard has fixed at least one bug found this way. > > http://llvm.org/viewvc/llvm-project?view=revision&revision=291030 > > > > My llvm fuzzing bot was pretty naive and simple. > > If we want proper continuous fuzzing for parts of LLVM we either need to > > build a separate "real" continuous fuzzing process, > > or use an existing one. Luckily, there is one :) > > As a pilot I've recently added the cxa_demangler_fuzzer to OSS-Fuzz: > > https://github.com/google/oss-fuzz/tree/master/projects/llvm_libcxxabi > > It even found one bug which Mehdi already fixed! > > http://llvm.org/viewvc/llvm-project?view=revision&revision=293330 > > The bug report itself will become public in ~4 days: > > https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=370 > > Thanks for the explanation. > > > > >> > Another (obvious?) fuzzing candidate would be the LLVM's bitcode > >> > reader. I ran afl-fuzz on it and it found lots of failed assertions > >> > within seconds. Isn't fuzzing done on a regular basis as [1] suggests > >> > should be done? Should I report the crashes found by it? > >> > >> The bitcode reader is known to not be robust against malformed inputs. > > > > > > Yes, I afraid the bitcode reader (as some other parts of LLVM) are not > > robust enough to withstand fuzzing. :( > > Note that if we want to use libFuzzer (which is an in-process fuzzer) the > > target should not assert/abort/exit on any input (if it's not a bug). > > Is there any incentive to change that?Not that I know of.> A google Summer of Code project maybe? >Maybe. The bottleneck is not bug finding, but bug fixing, which sometimes may require large changes. And doing code review for such changes might be more work than just making them. --kcc> > Michael >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170201/cc8d21b7/attachment.html>