llvm dev - Sep 2015 - Some feedback on Libfuzzer

Greg,
This is lots of useful feedback!
I'll reply to individual bullets when time permits (mostly after the
holidays).

If you find a bug in Postgres with libFuzzer, please let us know so that we
can add it to http://llvm.org/docs/LibFuzzer.html#trophies

On Sat, Sep 5, 2015 at 8:40 AM, Greg Stark via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Sat, Sep 5, 2015 at 1:50 PM, Greg Stark <stark at mit.edu> wrote:
> > 2) I've caught a bug which causes enormous stack growth. AFL
enforces
> > strict memory and time limits on the tests which is awfully
> > convenient. I can implement those myself in my fuzzer function (and in
> > fact in the Postgres context it would be best if I did) but some
> > simple AFL-style protection would be appreciated, as it is it takes a
> > *looong* time to fail and doesn't leave behind the crashing test.
It
> > would be nice if Libfuzzer took a page out of the sanitize code's
> > tricks and kept an mmapped file where it wrote the current test being
> > run. If the file is never synced then it shouldn't cause any
syscalls
> > or I/O until the program crashes and the file descriptor is closed.
> >
> > My thinking is I need to set an RLIMIT_STACK setting and then install
> > a SEGV handler which will longjmp back to the top level and return to
> > the fuzzer. That will be risky since it's in theory impossible to
> > restore any state the SEGV caused but in practice if it's always
> > caused by a stack overflow might be safe. I would also like to have an
> > ALRM handler but that requires calling alarm() on every call and
I'm
> > not sure if the setitimer in Libfuzzer can be disabled or if it'll
> > interfere with that. Maybe there's a better approach, I could call
> > setitimer and if I see more than n ALRMs during the execution decide
> > it's a fault. Again it would be nice if Libfuzzer provided that
> > itself.
>
>
> It occurs to me that this is silly as Libfuzzer is currently not
> capable of continuing once it finds one crash. There's no way for my
> FuzzOne() call to report that the test "failed" but that it's
still
> prepared to continue fuzzing more inputs. This seems like the
> fundamental problem I'm missing.
>
This is more like a limitation of asan, not libFuzzer.
By design, asan does not recover from the first crash.
This feature has been criticized quite a lot, but I am still convinced this
is a feature, not a bug.
IMHO, recovery mode will be misused/abused too often to be useful, besides
it adds complexity to the code.
(There is a patch under review right now to implement recovery mode for
asan,
but I am not sure if or when this patch will be committed)


>
> Also, one more thing, currently Libfuzzer does not catch SIGABRT and
> treat it as a fatal event. I've added a SIGABRT handler to my own code
> and moved StaticDeathCallback to public so I can call it from there.
>
> Again, this is asan, not libFuzzer.
You need ASAN_OPTIONS=handle_abort=1
I hope to make it the default soon-ish.

> --
> greg
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150905/08b23e1e/attachment.html>

On Sat, Sep 5, 2015 at 6:38 PM, Kostya Serebryany <kcc at google.com>
wrote:
>
> This is more like a limitation of asan, not libFuzzer.
> By design, asan does not recover from the first crash.
> This feature has been criticized quite a lot, but I am still convinced this
> is a feature, not a bug.
> IMHO, recovery mode will be misused/abused too often to be useful, besides
> it adds complexity to the code.
> (There is a patch under review right now to implement recovery mode for
> asan,
> but I am not sure if or when this patch will be committed)
Arguably a fuzzer changes the game somewhat. It's one thing to have a
crash during interactive testing but fuzzing is most useful if left to
run on a server unattended.

However I think this isn't the whole story here. There are many
different kinds of errors and not all are unrecoverable memory
failures. I may just get an internal error from my own code that I
know should never happen but doesn't represent major corruption. I
would like a way for the callback to return an error condition to the
fuzzer driver to have the test case set aside as a crash or perhaps as
a separate category (or perhaps to allow it to specify the name of the
category).
>> Also, one more thing, currently Libfuzzer does not catch SIGABRT and
>> treat it as a fatal event. I've added a SIGABRT handler to my own
code
>> and moved StaticDeathCallback to public so I can call it from there.
>>
> Again, this is asan, not libFuzzer.
> You need ASAN_OPTIONS=handle_abort=1
> I hope to make it the default soon-ish.
Ah, that would have made this work a bit simpler. Thanks.

I have yet to really experiment with the sanitizers so I don't know if
asan is really doing anything for me given Postgres's internal memory
management. I couldn't get msan running but it sounds more promising
to me. I'll have to try again.

-- 
greg

On Sat, Sep 5, 2015 at 6:38 PM, Kostya Serebryany <kcc at google.com>
wrote:
> This is lots of useful feedback!
> I'll reply to individual bullets when time permits (mostly after the
> holidays).
Oh. And enjoy your holidays!



-- 
greg

On Sat, Sep 5, 2015 at 11:50 AM, Greg Stark <stark at mit.edu> wrote:
> On Sat, Sep 5, 2015 at 6:38 PM, Kostya Serebryany <kcc at google.com>
wrote:
> >
> > This is more like a limitation of asan, not libFuzzer.
> > By design, asan does not recover from the first crash.
> > This feature has been criticized quite a lot, but I am still convinced
> this
> > is a feature, not a bug.
> > IMHO, recovery mode will be misused/abused too often to be useful,
> besides
> > it adds complexity to the code.
> > (There is a patch under review right now to implement recovery mode
for
> > asan,
> > but I am not sure if or when this patch will be committed)
>
> Arguably a fuzzer changes the game somewhat. It's one thing to have a
> crash during interactive testing but fuzzing is most useful if left to
> run on a server unattended.
>
> However I think this isn't the whole story here. There are many
> different kinds of errors and not all are unrecoverable memory
> failures. I may just get an internal error from my own code that I
> know should never happen but doesn't represent major corruption. I
> would like a way for the callback to return an error condition to the
> fuzzer driver to have the test case set aside as a crash or perhaps as
> a separate category (or perhaps to allow it to specify the name of the
> category).
>
For that you don't need libFuzzer support, right?
You can intercept your specific type of bug in the target function.


>
> >> Also, one more thing, currently Libfuzzer does not catch SIGABRT
and
> >> treat it as a fatal event. I've added a SIGABRT handler to my
own code
> >> and moved StaticDeathCallback to public so I can call it from
there.
> >>
> > Again, this is asan, not libFuzzer.
> > You need ASAN_OPTIONS=handle_abort=1
> > I hope to make it the default soon-ish.
>
> Ah, that would have made this work a bit simpler. Thanks.
>
> I have yet to really experiment with the sanitizers so I don't know if
> asan is really doing anything for me given Postgres's internal memory
> management.

That might be an interesting separate topic to discuss.

> I couldn't get msan running but it sounds more promising
> to me.

If you have custom memory management, msan will be as tricky to use as
asan.
Also, try ubsan for other kinds of bugs.

> I'll have to try again.
>
> --
> greg
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150908/a095941e/attachment.html>