thr3ads.net - llvm dev - [llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer [Sep 2016]

If this information is useful, please help other people find it:
Share via:

Mehdi Amini via llvm-dev

2016-Sep-21 19:32 UTC

[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer

> On Sep 21, 2016, at 9:36 AM, Kostya Serebryany via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Exciting! 
> 
> (btw, I'd prefer libfuzzer at googlegroups.com <mailto:libfuzzer at
googlegroups.com> for such discussions, please start new topics there)
You mean a LLVM library has a separate mailing-list? Why?

— 
Mehdi

> 
> I can reproduce this too, but if i either increase FUZZER_TESTING_SECONDS
to 600 or change seed=1 to seed=2 the problem is gone.
> Looks like one of the binaries got simply unlucky with a particular seed. 
> You can observe it like this: 
> for S in 1 2 3 4 5 6; do ./target-asan-8bit-prune-build/fuzzer -seed=$S
-runs=10000000 2>&1 | grep DONE &  done
> #10000000       DONE   cov: 60 bits: 91 indir: 1 units: 59 exec/s: 625000
> #10000000       DONE   cov: 60 bits: 91 indir: 1 units: 57 exec/s: 588235
> #10000000       DONE   cov: 253 bits: 901 indir: 12 units: 467 exec/s:
526315
> #10000000       DONE   cov: 63 bits: 95 indir: 1 units: 64 exec/s: 476190
> #10000000       DONE   cov: 252 bits: 923 indir: 12 units: 491 exec/s:
454545
> #10000000       DONE   cov: 253 bits: 880 indir: 12 units: 471 exec/s:
384615
> 
> Similar things happen with other binaries: 
> for S in 1 2 3 4 5 6; do ./target-asan-8bit-nopru-build/fuzzer -seed=$S
-runs=10000000 2>&1 | grep DONE &  done
> #10000000       DONE   cov: 103 bits: 190 indir: 1 units: 62 exec/s: 526315
> #10000000       DONE   cov: 443 bits: 1730 indir: 12 units: 529 exec/s:
357142
> #10000000       DONE   cov: 443 bits: 1695 indir: 12 units: 509 exec/s:
344827
> #10000000       DONE   cov: 443 bits: 1682 indir: 12 units: 500 exec/s:
333333
> #10000000       DONE   cov: 444 bits: 1675 indir: 12 units: 501 exec/s:
277777
> #10000000       DONE   cov: 401 bits: 1443 indir: 12 units: 341 exec/s:
263157
> 
> I've also tried building with trace-pc-guard (the new thing) and
results are similar.
> name    cov     bits    execs   execs_per_sec   units   actual_cov     
actual_bits
> asan-8bit-nopru 401     1443    19790806        324439  340     401    
1441
> asan-8bit-prune 256     897     26528866        434899  485     447    
1651
> asan-edge-nopru 447     0       35589496        583434  137     447     719
> asan-edge-prune 256     0       37576436        616007  137     447     719
> asan-trac-nopru 401     1443    12566606        206009  340     401    
1441
> asan-trac-prune 256     891     16295346        267136  480     447    
1640
> 
> Conclusions: 
> * testing a fuzzing engine is not trivial :( 
> * testing it on a very short run with a single seed may be misleading
> 
> 
> BTW, I am also looking into more automation of libFuzzer testing.
> With trace-pc-guard we now have libFuzzer's flag -print_coverage=1 that
will print all the covered lines.
> My hope is that this feature can be used for more detailed analysis of
coverage differences.
> 
> --kcc 
> 
> 
> On Wed, Sep 21, 2016 at 6:00 AM, Jonas Wagner <jonas.wagner at epfl.ch
<mailto:jonas.wagner at epfl.ch>> wrote:
> Hello,
> 
> Is this reproducible? 
> Fuzzing is a probabilistic business and one or even two runs don't
prove much.
> 
> I've reproduced the behavior on two different machines. Attached is a
script to do so. To use the script,
> 
> - create an empty folder and copy both prune-blocks.sh and
ff-http-parser.sh in there
> - ensure clang and clang++ are in your $PATH
> - cd /path/to/prune-blocks.sh
> - ./prune-blocks.sh
> 
> Let me know how it goes.
> 
>  
> Note that I am going to change all of these coverage options soon. 
> The new thing will be
http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards
<http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards>
> It will replace regular (boolean) and 8-bit-counters coverage.  
> 
> Yay, sounds exciting! I've done a couple experiments to measure the
performance and effect of the different coverage options in the recent past. If
you're interested, I'd be happy to discuss off-list; simply send me an
email.
> 
> Best,
> Jonas
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/3a82d2aa/attachment.html>

Kostya Serebryany via llvm-dev

2016-Sep-21 19:56 UTC

head link

[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer

On Wed, Sep 21, 2016 at 12:32 PM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
>
> On Sep 21, 2016, at 9:36 AM, Kostya Serebryany via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Exciting!
>
> (btw, I'd prefer libfuzzer at googlegroups.com for such discussions,
please
> start new topics there)
>
>
> You mean a LLVM library has a separate mailing-list? Why?
>
Because the topic is very separate.

>
> —
> Mehdi
>
>
>
> I can reproduce this too, but if i either increase FUZZER_TESTING_SECONDS
> to 600 or change seed=1 to seed=2 the problem is gone.
> Looks like one of the binaries got simply unlucky with a particular seed.
> You can observe it like this:
>
> for S in 1 2 3 4 5 6; do ./target-asan-8bit-prune-build/fuzzer -seed=$S
-runs=10000000 2>&1 | grep DONE &  done
>
> #10000000       DONE   cov: 60 bits: 91 indir: 1 units: 59 exec/s: 625000
> #10000000       DONE   cov: 60 bits: 91 indir: 1 units: 57 exec/s: 588235
> #10000000       DONE   cov: 253 bits: 901 indir: 12 units: 467 exec/s:
526315
> #10000000       DONE   cov: 63 bits: 95 indir: 1 units: 64 exec/s: 476190
> #10000000       DONE   cov: 252 bits: 923 indir: 12 units: 491 exec/s:
454545
> #10000000       DONE   cov: 253 bits: 880 indir: 12 units: 471 exec/s:
384615
>
>
> Similar things happen with other binaries:
>
> for S in 1 2 3 4 5 6; do ./target-asan-8bit-nopru-build/fuzzer -seed=$S
-runs=10000000 2>&1 | grep DONE &  done
>
> #10000000       DONE   cov: 103 bits: 190 indir: 1 units: 62 exec/s: 526315
> #10000000       DONE   cov: 443 bits: 1730 indir: 12 units: 529 exec/s:
357142
> #10000000       DONE   cov: 443 bits: 1695 indir: 12 units: 509 exec/s:
344827
> #10000000       DONE   cov: 443 bits: 1682 indir: 12 units: 500 exec/s:
333333
> #10000000       DONE   cov: 444 bits: 1675 indir: 12 units: 501 exec/s:
277777
> #10000000       DONE   cov: 401 bits: 1443 indir: 12 units: 341 exec/s:
263157
>
>
> I've also tried building with trace-pc-guard (the new thing) and
results
> are similar.
>
> name    cov     bits    execs   execs_per_sec   units   actual_cov     
actual_bits
> asan-8bit-nopru 401     1443    19790806        324439  340     401    
1441
> asan-8bit-prune 256     897     26528866        434899  485     447    
1651
> asan-edge-nopru 447     0       35589496        583434  137     447     719
> asan-edge-prune 256     0       37576436        616007  137     447     719
> asan-trac-nopru 401     1443    12566606        206009  340     401    
1441
> asan-trac-prune 256     891     16295346        267136  480     447    
1640
>
>
> Conclusions:
> * testing a fuzzing engine is not trivial :(
> * testing it on a very short run with a single seed may be misleading
>
>
> BTW, I am also looking into more automation of libFuzzer testing.
> With trace-pc-guard we now have libFuzzer's flag -print_coverage=1 that
> will print all the covered lines.
> My hope is that this feature can be used for more detailed analysis of
> coverage differences.
>
> --kcc
>
>
> On Wed, Sep 21, 2016 at 6:00 AM, Jonas Wagner <jonas.wagner at
epfl.ch>
> wrote:
>
>> Hello,
>>
>> Is this reproducible?
>>> Fuzzing is a probabilistic business and one or even two runs
don't prove
>>> much.
>>>
>>
>> I've reproduced the behavior on two different machines. Attached is
a
>> script to do so. To use the script,
>>
>> - create an empty folder and copy both prune-blocks.sh and
>> ff-http-parser.sh in there
>> - ensure clang and clang++ are in your $PATH
>> - cd /path/to/prune-blocks.sh
>> - ./prune-blocks.sh
>>
>> Let me know how it goes.
>>
>>
>>
>>> Note that I am going to change all of these coverage options soon.
>>> The new thing will be http://clang.llvm.org/docs/
>>> SanitizerCoverage.html#tracing-pcs-with-guards
>>> It will replace regular (boolean) and 8-bit-counters coverage.
>>>
>>
>> Yay, sounds exciting! I've done a couple experiments to measure the
>> performance and effect of the different coverage options in the recent
>> past. If you're interested, I'd be happy to discuss off-list;
simply send
>> me an email.
>>
>> Best,
>> Jonas
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/26d834fb/attachment.html>

Mehdi Amini via llvm-dev

2016-Sep-21 19:58 UTC

head link

[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer

> On Sep 21, 2016, at 12:56 PM, Kostya Serebryany <kcc at google.com>
wrote:
> 
> 
> 
> On Wed, Sep 21, 2016 at 12:32 PM, Mehdi Amini <mehdi.amini at apple.com
<mailto:mehdi.amini at apple.com>> wrote:
> 
>> On Sep 21, 2016, at 9:36 AM, Kostya Serebryany via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>> 
>> Exciting! 
>> 
>> (btw, I'd prefer libfuzzer at googlegroups.com <mailto:libfuzzer
at googlegroups.com> for such discussions, please start new topics there)
> 
> You mean a LLVM library has a separate mailing-list? Why?
> 
> Because the topic is very separate. 
Can you clarify?

I thought is was about the development/debug/evolution/usability of
http://llvm.org/svn/llvm-project/llvm/trunk/lib/Fuzzer/
<http://llvm.org/svn/llvm-project/llvm/trunk/lib/Fuzzer/>

— 
Mehid



>  
> 
> — 
> Mehdi
> 
> 
>> 
>> I can reproduce this too, but if i either increase
FUZZER_TESTING_SECONDS to 600 or change seed=1 to seed=2 the problem is gone.
>> Looks like one of the binaries got simply unlucky with a particular
seed.
>> You can observe it like this: 
>> for S in 1 2 3 4 5 6; do ./target-asan-8bit-prune-build/fuzzer -seed=$S
-runs=10000000 2>&1 | grep DONE &  done
>> #10000000       DONE   cov: 60 bits: 91 indir: 1 units: 59 exec/s:
625000
>> #10000000       DONE   cov: 60 bits: 91 indir: 1 units: 57 exec/s:
588235
>> #10000000       DONE   cov: 253 bits: 901 indir: 12 units: 467 exec/s:
526315
>> #10000000       DONE   cov: 63 bits: 95 indir: 1 units: 64 exec/s:
476190
>> #10000000       DONE   cov: 252 bits: 923 indir: 12 units: 491 exec/s:
454545
>> #10000000       DONE   cov: 253 bits: 880 indir: 12 units: 471 exec/s:
384615
>> 
>> Similar things happen with other binaries: 
>> for S in 1 2 3 4 5 6; do ./target-asan-8bit-nopru-build/fuzzer -seed=$S
-runs=10000000 2>&1 | grep DONE &  done
>> #10000000       DONE   cov: 103 bits: 190 indir: 1 units: 62 exec/s:
526315
>> #10000000       DONE   cov: 443 bits: 1730 indir: 12 units: 529 exec/s:
357142
>> #10000000       DONE   cov: 443 bits: 1695 indir: 12 units: 509 exec/s:
344827
>> #10000000       DONE   cov: 443 bits: 1682 indir: 12 units: 500 exec/s:
333333
>> #10000000       DONE   cov: 444 bits: 1675 indir: 12 units: 501 exec/s:
277777
>> #10000000       DONE   cov: 401 bits: 1443 indir: 12 units: 341 exec/s:
263157
>> 
>> I've also tried building with trace-pc-guard (the new thing) and
results are similar.
>> name    cov     bits    execs   execs_per_sec   units   actual_cov     
actual_bits
>> asan-8bit-nopru 401     1443    19790806        324439  340     401    
1441
>> asan-8bit-prune 256     897     26528866        434899  485     447    
1651
>> asan-edge-nopru 447     0       35589496        583434  137     447    
719
>> asan-edge-prune 256     0       37576436        616007  137     447    
719
>> asan-trac-nopru 401     1443    12566606        206009  340     401    
1441
>> asan-trac-prune 256     891     16295346        267136  480     447    
1640
>> 
>> Conclusions: 
>> * testing a fuzzing engine is not trivial :( 
>> * testing it on a very short run with a single seed may be misleading
>> 
>> 
>> BTW, I am also looking into more automation of libFuzzer testing.
>> With trace-pc-guard we now have libFuzzer's flag -print_coverage=1
that will print all the covered lines.
>> My hope is that this feature can be used for more detailed analysis of
coverage differences.
>> 
>> --kcc 
>> 
>> 
>> On Wed, Sep 21, 2016 at 6:00 AM, Jonas Wagner <jonas.wagner at
epfl.ch <mailto:jonas.wagner at epfl.ch>> wrote:
>> Hello,
>> 
>> Is this reproducible? 
>> Fuzzing is a probabilistic business and one or even two runs don't
prove much.
>> 
>> I've reproduced the behavior on two different machines. Attached is
a script to do so. To use the script,
>> 
>> - create an empty folder and copy both prune-blocks.sh and
ff-http-parser.sh in there
>> - ensure clang and clang++ are in your $PATH
>> - cd /path/to/prune-blocks.sh
>> - ./prune-blocks.sh
>> 
>> Let me know how it goes.
>> 
>>  
>> Note that I am going to change all of these coverage options soon. 
>> The new thing will be
http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards
<http://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards>
>> It will replace regular (boolean) and 8-bit-counters coverage.  
>> 
>> Yay, sounds exciting! I've done a couple experiments to measure the
performance and effect of the different coverage options in the recent past. If
you're interested, I'd be happy to discuss off-list; simply send me an
email.
>> 
>> Best,
>> Jonas
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160921/048889e9/attachment.html>

Reasonably Related Threads

Search for more maybe matching threads

llvm dev - Sep 2016 - -sanitizer-coverage-prune-blocks=true and LibFuzzer

[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer

[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer

[llvm-dev] -sanitizer-coverage-prune-blocks=true and LibFuzzer

Reasonably Related Threads