Jonas Wagner via llvm-dev
2016-Sep-21 13:00 UTC
[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer
Hello, Is this reproducible?> Fuzzing is a probabilistic business and one or even two runs don't prove > much. >I've reproduced the behavior on two different machines. Attached is a script to do so. To use the script, - create an empty folder and copy both prune-blocks.sh and ff-http-parser.sh in there - ensure clang and clang++ are in your $PATH - cd /path/to/prune-blocks.sh - ./prune-blocks.sh Let me know how it goes.> Note that I am going to change all of these coverage options soon. > The new thing will be > http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards > It will replace regular (boolean) and 8-bit-counters coverage. >Yay, sounds exciting! I've done a couple experiments to measure the performance and effect of the different coverage options in the recent past. If you're interested, I'd be happy to discuss off-list; simply send me an email. Best, Jonas -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/78453676/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: ff-http-parser.c Type: text/x-csrc Size: 1836 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/78453676/attachment.c> -------------- next part -------------- A non-text attachment was scrubbed... Name: prune-blocks.sh Type: application/x-shellscript Size: 5154 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/78453676/attachment.bin>
Kostya Serebryany via llvm-dev
2016-Sep-21 16:36 UTC
[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer
Exciting! (btw, I'd prefer libfuzzer at googlegroups.com for such discussions, please start new topics there) I can reproduce this too, but if i either increase FUZZER_TESTING_SECONDS to 600 or change seed=1 to seed=2 the problem is gone. Looks like one of the binaries got simply unlucky with a particular seed. You can observe it like this: for S in 1 2 3 4 5 6; do ./target-asan-8bit-prune-build/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE & done #10000000 DONE cov: 60 bits: 91 indir: 1 units: 59 exec/s: 625000 #10000000 DONE cov: 60 bits: 91 indir: 1 units: 57 exec/s: 588235 #10000000 DONE cov: 253 bits: 901 indir: 12 units: 467 exec/s: 526315 #10000000 DONE cov: 63 bits: 95 indir: 1 units: 64 exec/s: 476190 #10000000 DONE cov: 252 bits: 923 indir: 12 units: 491 exec/s: 454545 #10000000 DONE cov: 253 bits: 880 indir: 12 units: 471 exec/s: 384615 Similar things happen with other binaries: for S in 1 2 3 4 5 6; do ./target-asan-8bit-nopru-build/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE & done #10000000 DONE cov: 103 bits: 190 indir: 1 units: 62 exec/s: 526315 #10000000 DONE cov: 443 bits: 1730 indir: 12 units: 529 exec/s: 357142 #10000000 DONE cov: 443 bits: 1695 indir: 12 units: 509 exec/s: 344827 #10000000 DONE cov: 443 bits: 1682 indir: 12 units: 500 exec/s: 333333 #10000000 DONE cov: 444 bits: 1675 indir: 12 units: 501 exec/s: 277777 #10000000 DONE cov: 401 bits: 1443 indir: 12 units: 341 exec/s: 263157 I've also tried building with trace-pc-guard (the new thing) and results are similar. name cov bits execs execs_per_sec units actual_cov actual_bits asan-8bit-nopru 401 1443 19790806 324439 340 401 1441 asan-8bit-prune 256 897 26528866 434899 485 447 1651 asan-edge-nopru 447 0 35589496 583434 137 447 719 asan-edge-prune 256 0 37576436 616007 137 447 719 asan-trac-nopru 401 1443 12566606 206009 340 401 1441 asan-trac-prune 256 891 16295346 267136 480 447 1640 Conclusions: * testing a fuzzing engine is not trivial :( * testing it on a very short run with a single seed may be misleading BTW, I am also looking into more automation of libFuzzer testing. With trace-pc-guard we now have libFuzzer's flag -print_coverage=1 that will print all the covered lines. My hope is that this feature can be used for more detailed analysis of coverage differences. --kcc On Wed, Sep 21, 2016 at 6:00 AM, Jonas Wagner <jonas.wagner at epfl.ch> wrote:> Hello, > > Is this reproducible? >> Fuzzing is a probabilistic business and one or even two runs don't prove >> much. >> > > I've reproduced the behavior on two different machines. Attached is a > script to do so. To use the script, > > - create an empty folder and copy both prune-blocks.sh and > ff-http-parser.sh in there > - ensure clang and clang++ are in your $PATH > - cd /path/to/prune-blocks.sh > - ./prune-blocks.sh > > Let me know how it goes. > > > >> Note that I am going to change all of these coverage options soon. >> The new thing will be http://clang.llvm.org/docs/SanitizerCoverage.html# >> tracing-pcs-with-guards >> It will replace regular (boolean) and 8-bit-counters coverage. >> > > Yay, sounds exciting! I've done a couple experiments to measure the > performance and effect of the different coverage options in the recent > past. If you're interested, I'd be happy to discuss off-list; simply send > me an email. > > Best, > Jonas >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/acaffbd1/attachment.html>
Mehdi Amini via llvm-dev
2016-Sep-21 19:32 UTC
[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer
> On Sep 21, 2016, at 9:36 AM, Kostya Serebryany via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Exciting! > > (btw, I'd prefer libfuzzer at googlegroups.com <mailto:libfuzzer at googlegroups.com> for such discussions, please start new topics there)You mean a LLVM library has a separate mailing-list? Why? — Mehdi> > I can reproduce this too, but if i either increase FUZZER_TESTING_SECONDS to 600 or change seed=1 to seed=2 the problem is gone. > Looks like one of the binaries got simply unlucky with a particular seed. > You can observe it like this: > for S in 1 2 3 4 5 6; do ./target-asan-8bit-prune-build/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE & done > #10000000 DONE cov: 60 bits: 91 indir: 1 units: 59 exec/s: 625000 > #10000000 DONE cov: 60 bits: 91 indir: 1 units: 57 exec/s: 588235 > #10000000 DONE cov: 253 bits: 901 indir: 12 units: 467 exec/s: 526315 > #10000000 DONE cov: 63 bits: 95 indir: 1 units: 64 exec/s: 476190 > #10000000 DONE cov: 252 bits: 923 indir: 12 units: 491 exec/s: 454545 > #10000000 DONE cov: 253 bits: 880 indir: 12 units: 471 exec/s: 384615 > > Similar things happen with other binaries: > for S in 1 2 3 4 5 6; do ./target-asan-8bit-nopru-build/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE & done > #10000000 DONE cov: 103 bits: 190 indir: 1 units: 62 exec/s: 526315 > #10000000 DONE cov: 443 bits: 1730 indir: 12 units: 529 exec/s: 357142 > #10000000 DONE cov: 443 bits: 1695 indir: 12 units: 509 exec/s: 344827 > #10000000 DONE cov: 443 bits: 1682 indir: 12 units: 500 exec/s: 333333 > #10000000 DONE cov: 444 bits: 1675 indir: 12 units: 501 exec/s: 277777 > #10000000 DONE cov: 401 bits: 1443 indir: 12 units: 341 exec/s: 263157 > > I've also tried building with trace-pc-guard (the new thing) and results are similar. > name cov bits execs execs_per_sec units actual_cov actual_bits > asan-8bit-nopru 401 1443 19790806 324439 340 401 1441 > asan-8bit-prune 256 897 26528866 434899 485 447 1651 > asan-edge-nopru 447 0 35589496 583434 137 447 719 > asan-edge-prune 256 0 37576436 616007 137 447 719 > asan-trac-nopru 401 1443 12566606 206009 340 401 1441 > asan-trac-prune 256 891 16295346 267136 480 447 1640 > > Conclusions: > * testing a fuzzing engine is not trivial :( > * testing it on a very short run with a single seed may be misleading > > > BTW, I am also looking into more automation of libFuzzer testing. > With trace-pc-guard we now have libFuzzer's flag -print_coverage=1 that will print all the covered lines. > My hope is that this feature can be used for more detailed analysis of coverage differences. > > --kcc > > > On Wed, Sep 21, 2016 at 6:00 AM, Jonas Wagner <jonas.wagner at epfl.ch <mailto:jonas.wagner at epfl.ch>> wrote: > Hello, > > Is this reproducible? > Fuzzing is a probabilistic business and one or even two runs don't prove much. > > I've reproduced the behavior on two different machines. Attached is a script to do so. To use the script, > > - create an empty folder and copy both prune-blocks.sh and ff-http-parser.sh in there > - ensure clang and clang++ are in your $PATH > - cd /path/to/prune-blocks.sh > - ./prune-blocks.sh > > Let me know how it goes. > > > Note that I am going to change all of these coverage options soon. > The new thing will be http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards <http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards> > It will replace regular (boolean) and 8-bit-counters coverage. > > Yay, sounds exciting! I've done a couple experiments to measure the performance and effect of the different coverage options in the recent past. If you're interested, I'd be happy to discuss off-list; simply send me an email. > > Best, > Jonas > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/3a82d2aa/attachment.html>
Jonas Wagner via llvm-dev
2016-Sep-22 07:10 UTC
[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer
Hi, Thanks a lot for the tests and the info! Looks like one of the binaries got simply unlucky with a particular seed.>Ouch... I should have tried :/ Conclusions:> * testing a fuzzing engine is not trivial :( > * testing it on a very short run with a single seed may be misleading >Yup. I'll try to be more careful with this in the future. And if I'll have time, I'll try to analyze that coverage plateau a bit. It would be insightful to know which mutation takes the fuzzer beyond the 60 basic blocks, and which of the coverage counters picks up the change -- even if it's just one specific case for one specific benchmark. BTW, I am also looking into more automation of libFuzzer testing.> With trace-pc-guard we now have libFuzzer's flag -print_coverage=1 that > will print all the covered lines. > My hope is that this feature can be used for more detailed analysis of > coverage differences. >Cool. I'll give this a try too. I have some ideas about testing and analyzing coverage... will post them on libfuzzer@ once they are more mature. Cheers, Jonas On Wed, Sep 21, 2016 at 6:00 AM, Jonas Wagner <jonas.wagner at epfl.ch> wrote:> >> Hello, >> >> Is this reproducible? >>> Fuzzing is a probabilistic business and one or even two runs don't prove >>> much. >>> >> >> I've reproduced the behavior on two different machines. Attached is a >> script to do so. To use the script, >> >> - create an empty folder and copy both prune-blocks.sh and >> ff-http-parser.sh in there >> - ensure clang and clang++ are in your $PATH >> - cd /path/to/prune-blocks.sh >> - ./prune-blocks.sh >> >> Let me know how it goes. >> >> >> >>> Note that I am going to change all of these coverage options soon. >>> The new thing will be >>> http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards >>> It will replace regular (boolean) and 8-bit-counters coverage. >>> >> >> Yay, sounds exciting! I've done a couple experiments to measure the >> performance and effect of the different coverage options in the recent >> past. If you're interested, I'd be happy to discuss off-list; simply send >> me an email. >> >> Best, >> Jonas >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160922/bbb39cee/attachment.html>