llvm dev - Apr 2019 - Implementing a SafeStack alternative using the frame pointer

Hello,


I'm attempting to implement a buffer overflow protection mechanism very
similar to safestack. The key difference of my approach is that the "unsafe
stack" should be accessed using the target dependent frame pointer (as for
now, I'm focusing on x86 architecture specifically).

However, I'm unsure how to go about this. Simply modifying the existing
safestack pass seems to be insufficient: from my understanding, safestack lets
the compiler choose a free GPR by design (see [0]) and I don't see a way to
force a specific register to be used at IR stage.

I thought about implementing an instruction similar to alloca (lets call it
allocaUS for the moment), which replaces all "unsafe" allocas and
works identically to alloca, but is eventually lowered in a way that it uses the
frame pointer instead of the stack pointer. I know that adding a new instruction
into the IR instruction set is discouraged, as stated at [1].

Another option would be to modify the the LLVM target-independent code generator
[2], so that "unsafe" alloca instructions will be treated differently
during the generation of the SelectionDAG or the instruction selection/register
allocation phases.

Which stage of code generation would be a suitable starting point for me to
implement such a mechanism? Did I miss an even better option?


Thank you

Oskar


[0]
https://github.com/llvm-mirror/llvm/blob/9736ffc02af86add87912db3427d088bf6bbd94d/lib/CodeGen/SafeStack.cpp#L796

[1] https://llvm.org/docs/ExtendingLLVM.html

[2] https://llvm.org/docs/CodeGenerator.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190408/defd6f09/attachment.html>

Hi Oskar,

You may want to look at how the ShadowCallStack is implemented:
https://clang.llvm.org/docs/ShadowCallStack.html
Note that the current LLVM trunk supports only AARch64,
but at some point we also had a x86_64 implementation (which wasn't very
useful and so we removed it).

Not sure if this is exactly what you are looking for, but will give you
something.

Also, maybe reach out to the authors of SCADS
<https://www.semanticscholar.org/paper/SCADS-Separated-Control-and-Data-Stacks-Kugler-M%C3%BCller/af9d39f8002ba2047841258e9d761d162fb5aea1>,
who did something very similar to your proposal.

--kcc



On Mon, Apr 8, 2019 at 6:00 AM Oskar Szakinnis via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hello,
>
>
> I'm attempting to implement a buffer overflow protection mechanism very
> similar to safestack. The key difference of my approach is that the
"unsafe
> stack" should be accessed using the target dependent frame pointer (as
for
> now, I'm focusing on x86 architecture specifically).
>
> However, I'm unsure how to go about this. Simply modifying the existing
> safestack pass seems to be insufficient: from my understanding, safestack
> lets the compiler choose a free GPR by design (see [0]) and I don't see
a
> way to force a specific register to be used at IR stage.
>
> I thought about implementing an instruction similar to alloca (lets call
> it allocaUS for the moment), which replaces all "unsafe" allocas
and works
> identically to alloca, but is eventually lowered in a way that it uses the
> frame pointer instead of the stack pointer. I know that adding a new
> instruction into the IR instruction set is discouraged, as stated at [1].
>
> Another option would be to modify the the LLVM target-independent code
> generator [2], so that "unsafe" alloca instructions will be
treated
> differently during the generation of the SelectionDAG or the instruction
> selection/register allocation phases.
>
> Which stage of code generation would be a suitable starting point for me
> to implement such a mechanism? Did I miss an even better option?
>
>
> Thank you
>
> Oskar
>
>
> [0]
>
https://github.com/llvm-mirror/llvm/blob/9736ffc02af86add87912db3427d088bf6bbd94d/lib/CodeGen/SafeStack.cpp#L796
>
> [1] https://llvm.org/docs/ExtendingLLVM.html
>
> [2] https://llvm.org/docs/CodeGenerator.html
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190422/fa56d7cf/attachment.html>