Mehdi Amini via llvm-dev
2016-Sep-21 19:58 UTC
[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer
> On Sep 21, 2016, at 12:56 PM, Kostya Serebryany <kcc at google.com> wrote: > > > > On Wed, Sep 21, 2016 at 12:32 PM, Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote: > >> On Sep 21, 2016, at 9:36 AM, Kostya Serebryany via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> >> Exciting! >> >> (btw, I'd prefer libfuzzer at googlegroups.com <mailto:libfuzzer at googlegroups.com> for such discussions, please start new topics there) > > You mean a LLVM library has a separate mailing-list? Why? > > Because the topic is very separate.Can you clarify? I thought is was about the development/debug/evolution/usability of http://llvm.org/svn/llvm-project/llvm/trunk/lib/Fuzzer/ <http://llvm.org/svn/llvm-project/llvm/trunk/lib/Fuzzer/> — Mehid> > > — > Mehdi > > >> >> I can reproduce this too, but if i either increase FUZZER_TESTING_SECONDS to 600 or change seed=1 to seed=2 the problem is gone. >> Looks like one of the binaries got simply unlucky with a particular seed. >> You can observe it like this: >> for S in 1 2 3 4 5 6; do ./target-asan-8bit-prune-build/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE & done >> #10000000 DONE cov: 60 bits: 91 indir: 1 units: 59 exec/s: 625000 >> #10000000 DONE cov: 60 bits: 91 indir: 1 units: 57 exec/s: 588235 >> #10000000 DONE cov: 253 bits: 901 indir: 12 units: 467 exec/s: 526315 >> #10000000 DONE cov: 63 bits: 95 indir: 1 units: 64 exec/s: 476190 >> #10000000 DONE cov: 252 bits: 923 indir: 12 units: 491 exec/s: 454545 >> #10000000 DONE cov: 253 bits: 880 indir: 12 units: 471 exec/s: 384615 >> >> Similar things happen with other binaries: >> for S in 1 2 3 4 5 6; do ./target-asan-8bit-nopru-build/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE & done >> #10000000 DONE cov: 103 bits: 190 indir: 1 units: 62 exec/s: 526315 >> #10000000 DONE cov: 443 bits: 1730 indir: 12 units: 529 exec/s: 357142 >> #10000000 DONE cov: 443 bits: 1695 indir: 12 units: 509 exec/s: 344827 >> #10000000 DONE cov: 443 bits: 1682 indir: 12 units: 500 exec/s: 333333 >> #10000000 DONE cov: 444 bits: 1675 indir: 12 units: 501 exec/s: 277777 >> #10000000 DONE cov: 401 bits: 1443 indir: 12 units: 341 exec/s: 263157 >> >> I've also tried building with trace-pc-guard (the new thing) and results are similar. >> name cov bits execs execs_per_sec units actual_cov actual_bits >> asan-8bit-nopru 401 1443 19790806 324439 340 401 1441 >> asan-8bit-prune 256 897 26528866 434899 485 447 1651 >> asan-edge-nopru 447 0 35589496 583434 137 447 719 >> asan-edge-prune 256 0 37576436 616007 137 447 719 >> asan-trac-nopru 401 1443 12566606 206009 340 401 1441 >> asan-trac-prune 256 891 16295346 267136 480 447 1640 >> >> Conclusions: >> * testing a fuzzing engine is not trivial :( >> * testing it on a very short run with a single seed may be misleading >> >> >> BTW, I am also looking into more automation of libFuzzer testing. >> With trace-pc-guard we now have libFuzzer's flag -print_coverage=1 that will print all the covered lines. >> My hope is that this feature can be used for more detailed analysis of coverage differences. >> >> --kcc >> >> >> On Wed, Sep 21, 2016 at 6:00 AM, Jonas Wagner <jonas.wagner at epfl.ch <mailto:jonas.wagner at epfl.ch>> wrote: >> Hello, >> >> Is this reproducible? >> Fuzzing is a probabilistic business and one or even two runs don't prove much. >> >> I've reproduced the behavior on two different machines. Attached is a script to do so. To use the script, >> >> - create an empty folder and copy both prune-blocks.sh and ff-http-parser.sh in there >> - ensure clang and clang++ are in your $PATH >> - cd /path/to/prune-blocks.sh >> - ./prune-blocks.sh >> >> Let me know how it goes. >> >> >> Note that I am going to change all of these coverage options soon. >> The new thing will be http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards <http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards> >> It will replace regular (boolean) and 8-bit-counters coverage. >> >> Yay, sounds exciting! I've done a couple experiments to measure the performance and effect of the different coverage options in the recent past. If you're interested, I'd be happy to discuss off-list; simply send me an email. >> >> Best, >> Jonas >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/048889e9/attachment.html>
Kostya Serebryany via llvm-dev
2016-Sep-21 20:25 UTC
[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer
On Wed, Sep 21, 2016 at 12:58 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:> > On Sep 21, 2016, at 12:56 PM, Kostya Serebryany <kcc at google.com> wrote: > > > > On Wed, Sep 21, 2016 at 12:32 PM, Mehdi Amini <mehdi.amini at apple.com> > wrote: > >> >> On Sep 21, 2016, at 9:36 AM, Kostya Serebryany via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >> Exciting! >> >> (btw, I'd prefer libfuzzer at googlegroups.com for such discussions, please >> start new topics there) >> >> >> You mean a LLVM library has a separate mailing-list? Why? >> > > Because the topic is very separate. > > > Can you clarify? > > I thought is was about the development/debug/evolution/usability of > http://llvm.org/svn/llvm-project/llvm/trunk/lib/Fuzzer/ >Yes, and this topic is substantially different from most other topics discussed on llvm-dev. We also have separate maliing lists for asan/tsan/msan --kcc> > — > Mehid > > > > > > >> >> — >> Mehdi >> >> >> >> I can reproduce this too, but if i either increase FUZZER_TESTING_SECONDS >> to 600 or change seed=1 to seed=2 the problem is gone. >> Looks like one of the binaries got simply unlucky with a particular seed. >> You can observe it like this: >> >> for S in 1 2 3 4 5 6; do ./target-asan-8bit-prune-build/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE & done >> >> #10000000 DONE cov: 60 bits: 91 indir: 1 units: 59 exec/s: 625000 >> #10000000 DONE cov: 60 bits: 91 indir: 1 units: 57 exec/s: 588235 >> #10000000 DONE cov: 253 bits: 901 indir: 12 units: 467 exec/s: 526315 >> #10000000 DONE cov: 63 bits: 95 indir: 1 units: 64 exec/s: 476190 >> #10000000 DONE cov: 252 bits: 923 indir: 12 units: 491 exec/s: 454545 >> #10000000 DONE cov: 253 bits: 880 indir: 12 units: 471 exec/s: 384615 >> >> >> Similar things happen with other binaries: >> >> for S in 1 2 3 4 5 6; do ./target-asan-8bit-nopru-build/fuzzer -seed=$S -runs=10000000 2>&1 | grep DONE & done >> >> #10000000 DONE cov: 103 bits: 190 indir: 1 units: 62 exec/s: 526315 >> #10000000 DONE cov: 443 bits: 1730 indir: 12 units: 529 exec/s: 357142 >> #10000000 DONE cov: 443 bits: 1695 indir: 12 units: 509 exec/s: 344827 >> #10000000 DONE cov: 443 bits: 1682 indir: 12 units: 500 exec/s: 333333 >> #10000000 DONE cov: 444 bits: 1675 indir: 12 units: 501 exec/s: 277777 >> #10000000 DONE cov: 401 bits: 1443 indir: 12 units: 341 exec/s: 263157 >> >> >> I've also tried building with trace-pc-guard (the new thing) and results >> are similar. >> >> name cov bits execs execs_per_sec units actual_cov actual_bits >> asan-8bit-nopru 401 1443 19790806 324439 340 401 1441 >> asan-8bit-prune 256 897 26528866 434899 485 447 1651 >> asan-edge-nopru 447 0 35589496 583434 137 447 719 >> asan-edge-prune 256 0 37576436 616007 137 447 719 >> asan-trac-nopru 401 1443 12566606 206009 340 401 1441 >> asan-trac-prune 256 891 16295346 267136 480 447 1640 >> >> >> Conclusions: >> * testing a fuzzing engine is not trivial :( >> * testing it on a very short run with a single seed may be misleading >> >> >> BTW, I am also looking into more automation of libFuzzer testing. >> With trace-pc-guard we now have libFuzzer's flag -print_coverage=1 that >> will print all the covered lines. >> My hope is that this feature can be used for more detailed analysis of >> coverage differences. >> >> --kcc >> >> >> On Wed, Sep 21, 2016 at 6:00 AM, Jonas Wagner <jonas.wagner at epfl.ch> >> wrote: >> >>> Hello, >>> >>> Is this reproducible? >>>> Fuzzing is a probabilistic business and one or even two runs don't >>>> prove much. >>>> >>> >>> I've reproduced the behavior on two different machines. Attached is a >>> script to do so. To use the script, >>> >>> - create an empty folder and copy both prune-blocks.sh and >>> ff-http-parser.sh in there >>> - ensure clang and clang++ are in your $PATH >>> - cd /path/to/prune-blocks.sh >>> - ./prune-blocks.sh >>> >>> Let me know how it goes. >>> >>> >>> >>>> Note that I am going to change all of these coverage options soon. >>>> The new thing will be http://clang.llvm.org/docs/ >>>> SanitizerCoverage.html#tracing-pcs-with-guards >>>> It will replace regular (boolean) and 8-bit-counters coverage. >>>> >>> >>> Yay, sounds exciting! I've done a couple experiments to measure the >>> performance and effect of the different coverage options in the recent >>> past. If you're interested, I'd be happy to discuss off-list; simply send >>> me an email. >>> >>> Best, >>> Jonas >>> >> >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> >> >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/7ba952d0/attachment.html>
Mehdi Amini via llvm-dev
2016-Sep-21 20:37 UTC
[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer
> On Sep 21, 2016, at 1:25 PM, Kostya Serebryany <kcc at google.com> wrote: > > > > On Wed, Sep 21, 2016 at 12:58 PM, Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote: > >> On Sep 21, 2016, at 12:56 PM, Kostya Serebryany <kcc at google.com <mailto:kcc at google.com>> wrote: >> >> >> >> On Wed, Sep 21, 2016 at 12:32 PM, Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote: >> >>> On Sep 21, 2016, at 9:36 AM, Kostya Serebryany via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >>> >>> Exciting! >>> >>> (btw, I'd prefer libfuzzer at googlegroups.com <mailto:libfuzzer at googlegroups.com> for such discussions, please start new topics there) >> >> You mean a LLVM library has a separate mailing-list? Why? >> >> Because the topic is very separate. > > Can you clarify? > > I thought is was about the development/debug/evolution/usability of http://llvm.org/svn/llvm-project/llvm/trunk/lib/Fuzzer/ <http://llvm.org/svn/llvm-project/llvm/trunk/lib/Fuzzer/> > > Yes, and this topic is substantially different from most other topics discussed on llvm-dev.How so? If discussion on the development of a LLVM sub-library does not belong to llvm-dev, I wonder if the library belong to LLVM in the first place (add to this that libFuzzer does not use anything else in LLVM…).> We also have separate maliing lists for asan/tsan/msanIs it for more anything else than historical reasons? — Mehdi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/bdc723b9/attachment.html>