llvm dev - Sep 2016 - -sanitizer-coverage-prune-blocks=true and LibFuzzer

Hello LLVM devs,

I'm running lots of experiments with LibFuzzer these days -- it's an
amazing tool!

I've noticed something weird while examining the effect of various coverage
options: for one of my benchmarks, the fuzzer was achieving a higher total
coverage before April 2016, when -sanitizer-coverage-prune-blocks became
true by default (commit
<https://github.com/llvm-mirror/llvm/commit/f593646d9ade6ab311cd6c55880bf3bdfc4b26a4>
).

Here are the numbers from running the fuzzer with different options for a
fixed amount of time.

#units  #blocks  #bits   options
492     447      1666    -fsanitize-coverage=edge,indirect-calls,8bit-counters
-mllvm -sanitizer-coverage-prune-blocks=false
135     447      754     -fsanitize-coverage=edge -mllvm
-sanitizer-coverage-prune-blocks=false
 58     103      185
-fsanitize-coverage=edge,indirect-calls,8bit-counters
135     447      754     -fsanitize-coverage=edge

Here, units is the number of test cases generated, blocks is the number of
basic blocks covered (as measured by replaying the corpus using the
-sanitizer-coverage-prune-blocks=false version), and bits is the number of
bits from the 8bit-counters that were seen.

The outlier is the case where I use 8bit-counters and set
-sanitizer-coverage-prune-blocks to true. There are two things I don't
understand: (1) why can the 8bit-counters perform worse than using edge
coverage only, and (2) why does -sanitizer-coverage-prune-blocks affect the
result so much?

Any ideas?

If needed, I can provide scripts to reproduce this (it's a bit of work to
clean them up, though). But maybe someone already has an idea why this
could happen.

Cheers,
Jonas
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160920/b91aba2a/attachment.html>

On Tue, Sep 20, 2016 at 8:00 AM, Jonas Wagner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hello LLVM devs,
>
> I'm running lots of experiments with LibFuzzer these days -- it's
an
> amazing tool!
>
Thank you!

>
> I've noticed something weird while examining the effect of various
> coverage options: for one of my benchmarks, the fuzzer was achieving a
> higher total coverage before April 2016, when
-sanitizer-coverage-prune-blocks
> became true by default (commit
>
<https://github.com/llvm-mirror/llvm/commit/f593646d9ade6ab311cd6c55880bf3bdfc4b26a4>
> ).
>
> Here are the numbers from running the fuzzer with different options for a
> fixed amount of time.
>
> #units  #blocks  #bits   options
> 492     447      1666   
-fsanitize-coverage=edge,indirect-calls,8bit-counters
> -mllvm -sanitizer-coverage-prune-blocks=false
> 135     447      754     -fsanitize-coverage=edge -mllvm
> -sanitizer-coverage-prune-blocks=false
>  58     103      185     -fsanitize-coverage=edge,
> indirect-calls,8bit-counters
> 135     447      754     -fsanitize-coverage=edge
>
> Here, units is the number of test cases generated, blocks is the number of
> basic blocks covered (as measured by replaying the corpus using the
> -sanitizer-coverage-prune-blocks=false version), and bits is the number
> of bits from the 8bit-counters that were seen.
>
> The outlier is the case where I use 8bit-counters and set
> -sanitizer-coverage-prune-blocks to true. There are two things I don't
> understand: (1) why can the 8bit-counters perform worse than using edge
> coverage only, and (2) why does -sanitizer-coverage-prune-blocks affect
> the result so much?
>
> Any ideas?
>
No idea out of the box, and we have not seen such effect.
But admittedly, our benchmarking is far from perfect.

Is this reproducible?
Fuzzing is a probabilistic business and one or even two runs don't prove
much.


> If needed, I can provide scripts to reproduce this (it's a bit of work
to
> clean them up, though).
>
If you can do that, please do!

Note that I am going to change all of these coverage options soon.
The new thing will be
http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards
It will replace regular (boolean) and 8-bit-counters coverage.

This will not affect sanitizer-coverage-prune-blocks

--kcc

> But maybe someone already has an idea why this could happen.
>
> Cheers,
> Jonas
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160920/74788371/attachment.html>

Hello,

Is this reproducible?
> Fuzzing is a probabilistic business and one or even two runs don't
prove
> much.
>
I've reproduced the behavior on two different machines. Attached is a
script to do so. To use the script,

- create an empty folder and copy both prune-blocks.sh and
ff-http-parser.sh in there
- ensure clang and clang++ are in your $PATH
- cd /path/to/prune-blocks.sh
- ./prune-blocks.sh

Let me know how it goes.


> Note that I am going to change all of these coverage options soon.
> The new thing will be
> http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards
> It will replace regular (boolean) and 8-bit-counters coverage.
>
Yay, sounds exciting! I've done a couple experiments to measure the
performance and effect of the different coverage options in the recent
past. If you're interested, I'd be happy to discuss off-list; simply
send
me an email.

Best,
Jonas
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/78453676/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ff-http-parser.c
Type: text/x-csrc
Size: 1836 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/78453676/attachment.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: prune-blocks.sh
Type: application/x-shellscript
Size: 5154 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/78453676/attachment.bin>