llvm dev - Nov 2017 - [RFC] ASan: patches to support 32-byte shadow granularity

+ more asan folks, please CC them to the code reviews.
Also please make sure llvm-commits is CC-ed (cfe-commits for clang changes)

On Tue, Oct 31, 2017 at 2:29 PM, Walter Lee <waltl at google.com> wrote:
> I've prepared a preliminary set of patches that makes ASan work with
> 32-byte shadow granularity, and I would like to get some feedback on
> those patches as well as my general plan.
>
> Some background: I am porting ASan to the Myriad platform.  I'm
> looking to break up that port into components that may be
> useful/relevant to other platforms -- the first of those pieces is the
> ability to use a coarser shadow granularity.  This is important for us
> because Myriad has a limited amount of physical memory and no virtual
> memory, so it is important to limit the amount of shadow memory
> required.
>
> My end-goal for this part is to be able to configure a build that
> overrides the default shadow granularity, that can cleanly run the
> clang/llvm test suite for 32-byte shadow granularity on i386/x86_64,
> so we can set up buildbot for that configuration.
>
> My basic plan:
> 1. Add build support to override of default shadow scale.
> 2. Fix various issues with 32-byte shadow granularity.
> 3. Propose improvements to 32-byte shadow granularity support.
> 4. Make test suite run cleanly for 32-byte shadow granularity.
> 5. Set up build bot for 32-byte shadow granularity on i386/x86_64.
>
> My initial set of patches adds the build support and makes some
> essential fixes to the compiler and run-time.  They are:
>
> https://reviews.llvm.org/D39469 [asan] Add cmake hook to override
> default shadow scale
> https://reviews.llvm.org/D39470 [asan] Fix size/alignment issues with
> non-default shadow scale
> https://reviews.llvm.org/D39471 [asan] Fix small X86_64 ShadowOffset
> for non-default shadow scale
> https://reviews.llvm.org/D39472 [asan] Ensure that the minimum redzone
> is at least SHADOW_GRANULARITY
> https://reviews.llvm.org/D39473 [sanitizers] Increase alignment of low
> level allocator
> https://reviews.llvm.org/D39474 [asan] Avoid assert failure for
> non-default shadow scale
> https://reviews.llvm.org/D39475 [asan] Improve stack error reports for
> large shadow granularity
>
> The following features don't work yet, but can be fixed:
>
> - i386/x86_64 assembly instrumentation.
>
That's fine. It doesn't really work in regular mode either.
>
> - Prelink support.  The memory map with MidMem for this configuration
>   is more complicated that that expected by compiler_rt.  The current
>   HighShadow would overlap with MidMem so would need to be adjusted.
>   It's not clear to me that whether is an important feature?
>
you may safely ignore this for 32-bit granularity, but please keep it
working in regular mode.

>
> - Intra object overflow.  This appears to be an experimental feature,
>   and I have not ported the padding insertion for different
>   granularity.
>
Yep, ignore it.
>
> Some features will not work as well with the larger shadow
> granularity:
>
> - Stack errors: it seems sensible not to insert 32-byte sentinels
>   between every object, but the result is that some stack overflow
>   gets reported as unknown or use-after-scope.  I have a patch that
>   improves on the default behavior, but there remains cases where the
>   error reports will not be as good.
>
Hmm. Not sure what's the problem here. It's totally fine to insert
32-byte
redzone around stack objects.
(in 32-byte granularity mode)

>
> - __asan_poison_memory_region is now more limited.  A typical case
>   that doesn't work is the poisoning of 8-byte or 16-byte that maps to
>   the middle of a shadow byte.
>
yep.

>
> For testing, I have a few questions:
>
> - Would it make sense to provide an internal compiler flag to set the
>   shadow granularity, so that there we can at least run the
>   instrumentation tests for 32-byte granularity in normal builds?
>
I'd prefer a proper flag, like -fsanitize-address-granularity=N (8,16,32)

>
> - Is there a reasonable subset of tests I can port to 32-byte
>   granularity to provide reasonable coverage, or should I aim to port
>   all tests?
>
Let's see what tests won't work out of the box and decide.
We can mark all failing tests as
   UNSUPPORTED: 32-bit-granularity
but ideally we shouldn't have to mark too many of those.

>
> For now, I've tested my changes against check-all (for default shadow
> granularity), and also set up a 32-byte shadow granularity build and
> manually inspected the failures to ensure that they are not
> unexpected.
>
> Thanks,
>
> Walter
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171031/fdfc6c5b/attachment.html>

I've finished my initial set of patches to make 32-byte shadow
granularity work on x86.  Here is a summary of the changes from last
week:

- As discussed, I added a full redzone after every stack variable.

- We discussed adding a -fsanitize-address-granularity=N flag, but I
  found the following existing flag has been sufficient for my
  purposes: -asan-mapping-scale N.  If anyone thinks I should add the
  flag anyways, possibly replacing the latter, please let me know.

- I've modified the build so that we always run the ASan
  instrumentation test suite for shadow scale values of 3 and 5.

- I've gone through the asan test suites to make them run cleanly for
  both shadow scale=3 and shadow scale=5.  Here are the tests I have
  disabled, grouped by categories:

  - Instrumentation/AddressSanitizer tests.  Most tests work out of
    the box, and I ported some basic tests, leaving the following:

    llvm/test/Instrumentation/AddressSanitizer/lifetime-throw.ll
    llvm/test/Instrumentation/AddressSanitizer/lifetime-uar-uas.ll
    llvm/test/Instrumentation/AddressSanitizer/lifetime.ll
   
llvm/test/Instrumentation/AddressSanitizer/stack-poisoning-and-lifetime-be.ll
    llvm/test/Instrumentation/AddressSanitizer/stack-poisoning-and-lifetime.ll
    llvm/test/Instrumentation/AddressSanitizer/stack-poisoning.ll
    llvm/test/Instrumentation/AddressSanitizer/stack_layout.ll

  - Asm instrumentation not supported:
    Instrumentation/AddressSanitizer/X86/*
    compiler-rt/lib/asan/tests/asan_asm_test.cc
    compiler-rt/test/asan/TestCases/Linux/asan-asm-stacktrace-test.cc

  - Prelinking not supported:
    compiler-rt/test/asan/TestCases/Linux/asan_prelink_test.cc

  - Intra-object padding not supported:
    compiler-rt/test/asan/TestCases/intra-object-overflow.cc

  - Calls __asan_poison_memory_region in middle of shadow byte:
    compiler-rt/lib/asan/tests/asan_interface_test.cc
      SimplePoisonMemoryRegionTest
      OverlappingPoisonMemoryRegionTest
      PoisoningStressTest
    compiler-rt/test/asan/TestCases/small_memcpy_test.cc
    compiler-rt/test/asan/TestCases/strtol_strict.c
    compiler-rt/test/asan/TestCases/strtoll_strict.c

  - Has hardwired memory map:
    compiler-rt/test/asan/TestCases/Linux/cuda_test.cc
    compiler-rt/test/asan/TestCases/Linux/kernel-area.cc
    compiler-rt/test/asan/TestCases/Linux/nohugepage_test.cc

  - Miscellaneous:
    compiler-rt/test/asan/TestCases/Linux/allocator_oom_test.cc
      This test assumes amount of available memory.
    compiler-rt/test/asan/TestCases/stack-buffer-overflow-with-position.cc
      This fails because we don't have enough redzones to disambiguate
      overflow of one stack object vs underflow of the next stack
      object.

- Here is the full list of revisions.  I'll add review requests
  shortly.

[asan] Add CMake hook to override default shadow scale
       https://reviews.llvm.org/D39469
[asan] Fix size/alignment issues with non-default shadow scale
       https://reviews.llvm.org/D39470
[asan] Fix small X86_64 ShadowOffset for non-default shadow scale
       https://reviews.llvm.org/D39471
[asan] Ensure that the minimum redzone is at least SHADOW_GRANULARITY
       https://reviews.llvm.org/D39472
[sanitizers] Increase alignment of low level allocator
       https://reviews.llvm.org/D39473
[asan] Avoid assert failure for non-default shadow scale
       https://reviews.llvm.org/D39474
[asan] Add full redzone after every stack variable
       https://reviews.llvm.org/D39475
[gtest] Increase stack size for child process in EXPECT_DEATH implementation
       https://reviews.llvm.org/D39771
[asan] Add lit feature for custom shadow scale
       https://reviews.llvm.org/D39772
[asan] Port tests to shadow scale of 5
       https://reviews.llvm.org/D39773
[asan] Disable unsupported tests for custom shadow scale
       https://reviews.llvm.org/D39774
[asan] Test ASan instrumentation for shadow scale value of 5
       https://reviews.llvm.org/D39775

Thanks,

Walter