llvm dev - Feb 2017 - Fuzzing bitcode reader

Hi all,

The blog entry [1] suggest that one of the buildbots constantly fuzzes
clang and clang-format. However, the actual bot [2] only tests the
fuzzer itself over a well-known set of bugs in standard software (eg.
Heartbleed [3] seems to be among them). Has there actually ever been a
buildbot that fuzzes clang/LLVM itself?

Another (obvious?) fuzzing candidate would be the LLVM's bitcode
reader. I ran afl-fuzz on it and it found lots of failed assertions
within seconds. Isn't fuzzing done on a regular basis as [1] suggests
should be done? Should I report the crashes found by it?

Michael


[1] http://blog.llvm.org/2015/04/fuzz-all-clangs.html
[2] http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fuzzer/
[3] https://github.com/google/fuzzer-test-suite/tree/master/openssl-1.0.1f

> On Feb 1, 2017, at 8:34 AM, Michael Kruse via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi all,
> 
> The blog entry [1] suggest that one of the buildbots constantly fuzzes
> clang and clang-format. However, the actual bot [2] only tests the
> fuzzer itself over a well-known set of bugs in standard software (eg.
> Heartbleed [3] seems to be among them).
Isn’t it this stage?
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fuzzer/builds/2755/steps/stage2%2Fasan%2Bassertions%20check-fuzzer/logs/stdio
> Has there actually ever been a
> buildbot that fuzzes clang/LLVM itself?
> 
> Another (obvious?) fuzzing candidate would be the LLVM's bitcode
> reader. I ran afl-fuzz on it and it found lots of failed assertions
> within seconds. Isn't fuzzing done on a regular basis as [1] suggests
> should be done? Should I report the crashes found by it?
The bitcode reader is known to not be robust against malformed inputs.

— 
Mehdi

2017-02-01 17:45 GMT+01:00 Mehdi Amini <mehdi.amini at
apple.com>:
>
>> On Feb 1, 2017, at 8:34 AM, Michael Kruse via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>>
>> Hi all,
>>
>> The blog entry [1] suggest that one of the buildbots constantly fuzzes
>> clang and clang-format. However, the actual bot [2] only tests the
>> fuzzer itself over a well-known set of bugs in standard software (eg.
>> Heartbleed [3] seems to be among them).
>
> Isn’t it this stage?
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fuzzer/builds/2755/steps/stage2%2Fasan%2Bassertions%20check-fuzzer/logs/stdio
To me it looks like just the compilation and the unit+regression tests
("ninja check-fuzzer", not even depending on clang). It also completes
in only 10 minutes, which is not a lot for fuzzing.

Michael

On Wed, Feb 1, 2017 at 8:45 AM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
>
> > On Feb 1, 2017, at 8:34 AM, Michael Kruse via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > Hi all,
> >
> > The blog entry [1] suggest that one of the buildbots constantly fuzzes
> > clang and clang-format. However, the actual bot [2] only tests the
> > fuzzer itself over a well-known set of bugs in standard software (eg.
> > Heartbleed [3] seems to be among them).
>
> Isn’t it this stage? http://lab.llvm.org:8011/build
> ers/sanitizer-x86_64-linux-fuzzer/builds/2755/steps/
> stage2%2Fasan%2Bassertions%20check-fuzzer/logs/stdio
>
> > Has there actually ever been a
> > buildbot that fuzzes clang/LLVM itself?
>
Yes, I used to run clang-fuzzer and clang-format-fuzzer on this bot, but
not any more.
The reason is simple -- the bot was always red (well, orange) and the bugs
were never fixed.

Currently we run clang-fuzzer (but not clang-format-fuzzer) on our internal
fuzzing infra
and Richard has fixed at least one bug found this way.
http://llvm.org/viewvc/llvm-project?view=revision&revision=291030

My llvm fuzzing bot was pretty naive and simple.
If we want proper continuous fuzzing for parts of LLVM we either need to
build a separate "real" continuous fuzzing process,
or use an existing one. Luckily, there is one :)
As a pilot I've recently added the cxa_demangler_fuzzer to OSS-Fuzz
<https://github.com/google/oss-fuzz>:
https://github.com/google/oss-fuzz/tree/master/projects/llvm_libcxxabi
It even found one bug which Mehdi already fixed!
http://llvm.org/viewvc/llvm-project?view=revision&revision=293330
The bug report itself will become public in ~4 days:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=370

If we want to run some more llvm fuzzers on OSS-Fuzz I'd be happy to (help)
set them up.


> >
> > Another (obvious?) fuzzing candidate would be the LLVM's bitcode
> > reader. I ran afl-fuzz on it and it found lots of failed assertions
> > within seconds. Isn't fuzzing done on a regular basis as [1]
suggests
> > should be done? Should I report the crashes found by it?
>
> The bitcode reader is known to not be robust against malformed inputs.
>
Yes, I afraid the bitcode reader (as some other parts of LLVM) are not
robust enough to withstand fuzzing. :(
Note that if we want to use libFuzzer (which is an in-process fuzzer) the
target should not assert/abort/exit on any input (if it's not a bug).

--kcc


>
> —
> Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170201/33ce1dd8/attachment-0001.html>