Brian Cain via llvm-dev
2015-Dec-03 19:14 UTC
[llvm-dev] fuzzer crash (but not the good kind)
Ah, yes -- you need to clone with --recursive. I will try the workaround though. On Dec 3, 2015 1:12 PM, "Kostya Serebryany" <kcc at google.com> wrote:> > > On Wed, Dec 2, 2015 at 7:17 PM, Brian Cain <brian.cain at gmail.com> wrote: > >> Kostya, >> >> Here's the git repo: https://bitbucket.org/ebadf/fuzzpy >> >> I've only tested it on arm7 and x86_64 linux, I expect there's a good >> chance it may not work on other OSs. >> >> If you can build it successfully ("./build.sh", requires clang and >> clang++ in your path), then you should run the "testemail" case like so: >> >> > Does not build for me out of the box: > > ./build.sh: line 70: ./configure: No such file or directory > > I wonder if a smaller test possible here. > Meanwhile, here is a workaround for you. > Instead of > SANITIZE_COV_OPTS="-fsanitize-coverage=bb,indirect-calls,8bit-counters" > try using > SANITIZE_COV_OPTS="-fsanitize-coverage=edge,indirect-calls" > > > while true; do ITERS=1000 ./run.sh tests/build/testemail >> tests/testemail/inputs/; done >> >> Let me know if you have any challenges building or running the test case. >> >> >> On Tue, Dec 1, 2015 at 7:26 PM, Kostya Serebryany <kcc at google.com> wrote: >> >>> Hi Brian, >>> Yes, looks like a bug in sanitizer coverage, please send the reproducer. >>> >>> On Tue, Dec 1, 2015 at 5:22 PM, Brian Cain <brian.cain at gmail.com> wrote: >>> >>>> >>>> Kostya, >>>> >>>> I think I've found what looks like a reproducible bug in libFuzzer. >>>> The code under test is built with ASan and the first ASan CHECK failure >>>> shows fuzzer in the stack trace. (see below) >>>> >>>> One of the factors that may be unique in my testing is that each >>>> iteration can take a very long time to execute (tens or hundreds of >>>> seconds). >>>> >>>> Let me know if you need more info, I think it shouldn't take much test >>>> time to reproduce this. >>>> >>>> ================== Job 2 exited with exit code 256 ===========>>>> Flag: verbosity 3 >>>> Flag: use_traces 1 >>>> Flag: timeout 100 >>>> Flag: max_len 16384 >>>> Seed: 3259211893 >>>> PreferSmall: 0 >>>> #0 READ units: 4975 exec/s: 0 >>>> #1 pulse cov: 32410 bits: 30791 indir: 714 units: 4975 exec/s: 0 >>>> NEW0: 32410 L 13869 >>>> ==31301==AddressSanitizer CHECK failed: >>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc:467 >>>> "((n % 16)) == ((0))" (0x1, 0x0) >>>> #0 0x11d3b7 in __asan::AsanCheckFailed(char const*, int, char >>>> const*, unsigned long long, unsigned long long) >>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:67:3 >>>> #1 0x122f1f in __sanitizer::CheckFailed(char const*, int, char >>>> const*, unsigned long long, unsigned long long) >>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:159:5 >>>> #2 0x134317 in >>>> __sanitizer::CoverageData::Update8bitCounterBitsetAndClearCounters(unsigned >>>> char*) >>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc:467:5 >>>> #3 0x1b7b53 in fuzzer::Fuzzer::PrepareCoverageBeforeRun() >>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:264:5 >>>> #4 0x1b501b in fuzzer::Fuzzer::RunOne(std::vector<unsigned char, >>>> std::allocator<unsigned char> > const&) >>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:212:3 >>>> #5 0x1b6be3 in fuzzer::Fuzzer::ShuffleAndMinimize() >>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:195:11 >>>> #6 0x14477b in fuzzer::FuzzerDriver(std::vector<std::string, >>>> std::allocator<std::string> > const&, fuzzer::UserSuppliedFuzzer&) >>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:303:3 >>>> #7 0x14183f in fuzzer::FuzzerDriver(int, char**, >>>> fuzzer::UserSuppliedFuzzer&) >>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:201:10 >>>> #8 0x141427 in fuzzer::FuzzerDriver(int, char**, int (*)(unsigned >>>> char const*, unsigned int)) >>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:196:10 >>>> #9 0x1873e3 in main >>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerMain.cpp:19:10 >>>> #10 0xb6c86775 in __libc_start_main >>>> /build/buildd/glibc-2.21/csu/libc-start.c:289 >>>> >>>> DEATH: >>>> artifact_prefix='./'; Test unit written to >>>> ./crash-ec9fa023e9db127e2589d0ab4c506055e4174611 >>>> >>>> >>>> -- >>>> -Brian >>>> >>> >>> >> >> >> -- >> -Brian >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151203/4e99624f/attachment-0001.html>
Kostya Serebryany via llvm-dev
2015-Dec-03 20:15 UTC
[llvm-dev] fuzzer crash (but not the good kind)
I am able to build the fuzzer, but not to reproduce the crash. First: you seem to be using a not-the-most-recent libfuzzer source, please update. You will need to change LLVMFuzzerTestOneInput to return 0. Second, for some reason, the fuzzing is very slow, about 1 exec/s. You probably want to fix that too. Third: does this code have threads? dlopen? Running it for a short period does not show either, but probably the fuzzer discovers some such input that triggers dlopen in a non-main thread. I guess the sanitizer coverage has a race in this case. So, just drop 8bit-counters. (This flag is far from optimal, we may actually remove it completely at some point). hth, --kcc On Thu, Dec 3, 2015 at 11:14 AM, Brian Cain <brian.cain at gmail.com> wrote:> Ah, yes -- you need to clone with --recursive. > > I will try the workaround though. > On Dec 3, 2015 1:12 PM, "Kostya Serebryany" <kcc at google.com> wrote: > >> >> >> On Wed, Dec 2, 2015 at 7:17 PM, Brian Cain <brian.cain at gmail.com> wrote: >> >>> Kostya, >>> >>> Here's the git repo: https://bitbucket.org/ebadf/fuzzpy >>> >>> I've only tested it on arm7 and x86_64 linux, I expect there's a good >>> chance it may not work on other OSs. >>> >>> If you can build it successfully ("./build.sh", requires clang and >>> clang++ in your path), then you should run the "testemail" case like so: >>> >>> >> Does not build for me out of the box: >> >> ./build.sh: line 70: ./configure: No such file or directory >> >> I wonder if a smaller test possible here. >> Meanwhile, here is a workaround for you. >> Instead of >> SANITIZE_COV_OPTS="-fsanitize-coverage=bb,indirect-calls,8bit-counters" >> try using >> SANITIZE_COV_OPTS="-fsanitize-coverage=edge,indirect-calls" >> >> >> while true; do ITERS=1000 ./run.sh tests/build/testemail >>> tests/testemail/inputs/; done >>> >>> Let me know if you have any challenges building or running the test case. >>> >>> >>> On Tue, Dec 1, 2015 at 7:26 PM, Kostya Serebryany <kcc at google.com> >>> wrote: >>> >>>> Hi Brian, >>>> Yes, looks like a bug in sanitizer coverage, please send the >>>> reproducer. >>>> >>>> On Tue, Dec 1, 2015 at 5:22 PM, Brian Cain <brian.cain at gmail.com> >>>> wrote: >>>> >>>>> >>>>> Kostya, >>>>> >>>>> I think I've found what looks like a reproducible bug in libFuzzer. >>>>> The code under test is built with ASan and the first ASan CHECK failure >>>>> shows fuzzer in the stack trace. (see below) >>>>> >>>>> One of the factors that may be unique in my testing is that each >>>>> iteration can take a very long time to execute (tens or hundreds of >>>>> seconds). >>>>> >>>>> Let me know if you need more info, I think it shouldn't take much test >>>>> time to reproduce this. >>>>> >>>>> ================== Job 2 exited with exit code 256 ===========>>>>> Flag: verbosity 3 >>>>> Flag: use_traces 1 >>>>> Flag: timeout 100 >>>>> Flag: max_len 16384 >>>>> Seed: 3259211893 >>>>> PreferSmall: 0 >>>>> #0 READ units: 4975 exec/s: 0 >>>>> #1 pulse cov: 32410 bits: 30791 indir: 714 units: 4975 exec/s: 0 >>>>> NEW0: 32410 L 13869 >>>>> ==31301==AddressSanitizer CHECK failed: >>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc:467 >>>>> "((n % 16)) == ((0))" (0x1, 0x0) >>>>> #0 0x11d3b7 in __asan::AsanCheckFailed(char const*, int, char >>>>> const*, unsigned long long, unsigned long long) >>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:67:3 >>>>> #1 0x122f1f in __sanitizer::CheckFailed(char const*, int, char >>>>> const*, unsigned long long, unsigned long long) >>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:159:5 >>>>> #2 0x134317 in >>>>> __sanitizer::CoverageData::Update8bitCounterBitsetAndClearCounters(unsigned >>>>> char*) >>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc:467:5 >>>>> #3 0x1b7b53 in fuzzer::Fuzzer::PrepareCoverageBeforeRun() >>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:264:5 >>>>> #4 0x1b501b in fuzzer::Fuzzer::RunOne(std::vector<unsigned char, >>>>> std::allocator<unsigned char> > const&) >>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:212:3 >>>>> #5 0x1b6be3 in fuzzer::Fuzzer::ShuffleAndMinimize() >>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:195:11 >>>>> #6 0x14477b in fuzzer::FuzzerDriver(std::vector<std::string, >>>>> std::allocator<std::string> > const&, fuzzer::UserSuppliedFuzzer&) >>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:303:3 >>>>> #7 0x14183f in fuzzer::FuzzerDriver(int, char**, >>>>> fuzzer::UserSuppliedFuzzer&) >>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:201:10 >>>>> #8 0x141427 in fuzzer::FuzzerDriver(int, char**, int (*)(unsigned >>>>> char const*, unsigned int)) >>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:196:10 >>>>> #9 0x1873e3 in main >>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerMain.cpp:19:10 >>>>> #10 0xb6c86775 in __libc_start_main >>>>> /build/buildd/glibc-2.21/csu/libc-start.c:289 >>>>> >>>>> DEATH: >>>>> artifact_prefix='./'; Test unit written to >>>>> ./crash-ec9fa023e9db127e2589d0ab4c506055e4174611 >>>>> >>>>> >>>>> -- >>>>> -Brian >>>>> >>>> >>>> >>> >>> >>> -- >>> -Brian >>> >> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151203/c03fbed9/attachment.html>
Brian Cain via llvm-dev
2015-Dec-03 20:34 UTC
[llvm-dev] fuzzer crash (but not the good kind)
Oh I thought it was really recent (1-2 weeks behind ToT ). I must have done something wrong there -- I'll check. Yes, it's extraordinarily slow and it may do dlopen (), I'll find out. No, I don't *believe* it's using other threads but I will check. I would let it run for 24h before giving up on a crash though. On Dec 3, 2015 2:15 PM, "Kostya Serebryany" <kcc at google.com> wrote:> I am able to build the fuzzer, but not to reproduce the crash. > > First: you seem to be using a not-the-most-recent libfuzzer source, please > update. > You will need to change LLVMFuzzerTestOneInput to return 0. > > Second, for some reason, the fuzzing is very slow, about 1 exec/s. > You probably want to fix that too. > > Third: does this code have threads? dlopen? > Running it for a short period does not show either, but probably the > fuzzer discovers some such input that triggers dlopen in a non-main thread. > I guess the sanitizer coverage has a race in this case. > > So, just drop 8bit-counters. (This flag is far from optimal, we may > actually remove it completely at some point). > > hth, > --kcc > > > > > On Thu, Dec 3, 2015 at 11:14 AM, Brian Cain <brian.cain at gmail.com> wrote: > >> Ah, yes -- you need to clone with --recursive. >> >> I will try the workaround though. >> On Dec 3, 2015 1:12 PM, "Kostya Serebryany" <kcc at google.com> wrote: >> >>> >>> >>> On Wed, Dec 2, 2015 at 7:17 PM, Brian Cain <brian.cain at gmail.com> wrote: >>> >>>> Kostya, >>>> >>>> Here's the git repo: https://bitbucket.org/ebadf/fuzzpy >>>> >>>> I've only tested it on arm7 and x86_64 linux, I expect there's a good >>>> chance it may not work on other OSs. >>>> >>>> If you can build it successfully ("./build.sh", requires clang and >>>> clang++ in your path), then you should run the "testemail" case like so: >>>> >>>> >>> Does not build for me out of the box: >>> >>> ./build.sh: line 70: ./configure: No such file or directory >>> >>> I wonder if a smaller test possible here. >>> Meanwhile, here is a workaround for you. >>> Instead of >>> SANITIZE_COV_OPTS="-fsanitize-coverage=bb,indirect-calls,8bit-counters" >>> try using >>> SANITIZE_COV_OPTS="-fsanitize-coverage=edge,indirect-calls" >>> >>> >>> while true; do ITERS=1000 ./run.sh tests/build/testemail >>>> tests/testemail/inputs/; done >>>> >>>> Let me know if you have any challenges building or running the test >>>> case. >>>> >>>> >>>> On Tue, Dec 1, 2015 at 7:26 PM, Kostya Serebryany <kcc at google.com> >>>> wrote: >>>> >>>>> Hi Brian, >>>>> Yes, looks like a bug in sanitizer coverage, please send the >>>>> reproducer. >>>>> >>>>> On Tue, Dec 1, 2015 at 5:22 PM, Brian Cain <brian.cain at gmail.com> >>>>> wrote: >>>>> >>>>>> >>>>>> Kostya, >>>>>> >>>>>> I think I've found what looks like a reproducible bug in libFuzzer. >>>>>> The code under test is built with ASan and the first ASan CHECK failure >>>>>> shows fuzzer in the stack trace. (see below) >>>>>> >>>>>> One of the factors that may be unique in my testing is that each >>>>>> iteration can take a very long time to execute (tens or hundreds of >>>>>> seconds). >>>>>> >>>>>> Let me know if you need more info, I think it shouldn't take much >>>>>> test time to reproduce this. >>>>>> >>>>>> ================== Job 2 exited with exit code 256 ===========>>>>>> Flag: verbosity 3 >>>>>> Flag: use_traces 1 >>>>>> Flag: timeout 100 >>>>>> Flag: max_len 16384 >>>>>> Seed: 3259211893 >>>>>> PreferSmall: 0 >>>>>> #0 READ units: 4975 exec/s: 0 >>>>>> #1 pulse cov: 32410 bits: 30791 indir: 714 units: 4975 exec/s: 0 >>>>>> NEW0: 32410 L 13869 >>>>>> ==31301==AddressSanitizer CHECK failed: >>>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc:467 >>>>>> "((n % 16)) == ((0))" (0x1, 0x0) >>>>>> #0 0x11d3b7 in __asan::AsanCheckFailed(char const*, int, char >>>>>> const*, unsigned long long, unsigned long long) >>>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:67:3 >>>>>> #1 0x122f1f in __sanitizer::CheckFailed(char const*, int, char >>>>>> const*, unsigned long long, unsigned long long) >>>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:159:5 >>>>>> #2 0x134317 in >>>>>> __sanitizer::CoverageData::Update8bitCounterBitsetAndClearCounters(unsigned >>>>>> char*) >>>>>> /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc:467:5 >>>>>> #3 0x1b7b53 in fuzzer::Fuzzer::PrepareCoverageBeforeRun() >>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:264:5 >>>>>> #4 0x1b501b in fuzzer::Fuzzer::RunOne(std::vector<unsigned char, >>>>>> std::allocator<unsigned char> > const&) >>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:212:3 >>>>>> #5 0x1b6be3 in fuzzer::Fuzzer::ShuffleAndMinimize() >>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:195:11 >>>>>> #6 0x14477b in fuzzer::FuzzerDriver(std::vector<std::string, >>>>>> std::allocator<std::string> > const&, fuzzer::UserSuppliedFuzzer&) >>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:303:3 >>>>>> #7 0x14183f in fuzzer::FuzzerDriver(int, char**, >>>>>> fuzzer::UserSuppliedFuzzer&) >>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:201:10 >>>>>> #8 0x141427 in fuzzer::FuzzerDriver(int, char**, int (*)(unsigned >>>>>> char const*, unsigned int)) >>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:196:10 >>>>>> #9 0x1873e3 in main >>>>>> /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerMain.cpp:19:10 >>>>>> #10 0xb6c86775 in __libc_start_main >>>>>> /build/buildd/glibc-2.21/csu/libc-start.c:289 >>>>>> >>>>>> DEATH: >>>>>> artifact_prefix='./'; Test unit written to >>>>>> ./crash-ec9fa023e9db127e2589d0ab4c506055e4174611 >>>>>> >>>>>> >>>>>> -- >>>>>> -Brian >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> -Brian >>>> >>> >>> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151203/21135860/attachment.html>