llvm dev - Dec 2017 - ASan's FakeStack with a garbage-collected language

Hello ASan devs,

I am trying to improve ASan functionality with our D compiler, LDC. One of
the main issues is that D has a garbage collector, and so I have to modify
both the ASan instrumentation and the D runtime such that they work well
together.
My question is about the FakeStack instrumentation to detect
stack-use-after-return bugs.

The D garbage collector needs to scan the stack for pointers, in order to
make sure that an object is no longer referenced before it can be
collected. So when ASan is dynamically allocating a FakeStack, we have to
inform the GC such that it scans that region for pointers too.

I wrote a pass that adds calls to `gc_addRange` and `gc_removeRange` for
each encountered `__asan_stack_malloc` and `__asan_stack_free`,
respectively. However, for small stack frames, LLVM does not emit a call to
`__asan_stack_free` but instead emits directly inlined equivalent IR, and
so my pass misses a lot of stack frees and doesn't work.

How can I get things to work?

I see a few possibilities:
1. Add an LLVM option to disable the `__asan_stack_free` inlining, and use
my pass.
2. Add an LLVM option to emit calls to frontend-specified functions after
`__asan_stack_malloc` and `__asan_stack_free`, which replaces my pass.
3. Figure out how to use ASan's `ForEachExtraStackRange` (callback for each
fakestack) and use that in D's GC implementation.

Probably option 3 is the best (`gc_addRange` involves mutex locking), but
the hardest.

(I also noticed that I have to call __sanitizer_start_switch_fiber and
__sanitizer_finish_switch_fiber upon fiber switches, but after reading the
code, I think it's only needed when FakeStack is enabled)

Thanks for your advice,
Cheers,
  Johan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171229/b1f0ba03/attachment.html>