llvm dev - Sep 2015 - Dynamic VMA in Sanitizers for AArch64

On 25 September 2015 at 20:11, Jakub Jelinek <jakub at redhat.com>
wrote:
> Note, in our distros we are shipping 42-bit VMA and are using patch on
> top of vanilla libsanitizer (with the 1UL << 36 shadow offset) and I
don't
> remember any bugs reported against this not working (and the testsuite
works
> too).  So, assuming 39-bit VMA works too, that would show that at least
> those two settings work, the question is if 48-bit VMA (or how many) works
> too, and if it does, the next thing is tweaking the library so that it can
> perhaps with some small but still acceptable performance hit decide between
> those at runtime (e.g. kAllocatorSpace/kAllocatorSize could be turned into
> non-const variables for aarch64, harder would be to add some allocator that
> at runtime picks if it uses 32-bit or 64-bit allocator.
Hi Jakub,

My assumption is based on what I understood from my talks with various
people in sanitizers, libraries, GCC, LLVM and the kernel. Please,
correct me if I'm wrong.

IIUC, using larger VMAs on kernels with smaller VMAs work, but
restrict the space that you can use as a shadow region, thus limiting
the size of programs you can run with the sanitizers. Also, the higher
you go, the higher is the levels of indirection you need to use
memory, so using higher VMAs may be more of a performance hit than it
should. This may not be such a big deal for 42 vs. 39 bits, but the
new kernels will come with 48 bits, and there's talks to push it up to
56 bits in the near future.

So, there are only three paths we can take:

1. Keep it constant, today at 42, and increase the constant slowly, as
kernels start popping in with higher VMAs. This could slow down for
large values of VMA and low values on kernel.

2. Create a compiler flag -mvma=NN. This would be as fast as native
when chosen correctly, but could break if lower than the machine's
VMA.

3. Make it dynamic, so that the VMA value doesn't matter.

I really have no idea on what the impact of dynamic VMA will have on
the sanitizers, nor I have on what it would be if we choose an
arbitrarily large VMA value (say, 56), and run on the lowest VMA (39).
We need to benchmark those things.

My email was just a summary of my discussions and a look forward. We
believe we can implement the dynamic VMA with very little impact, but
before going there, we need to understand what's the impact of using
higher than necessary VMA values. This is all part of the
investigation process that we'll start now.

All our changes until now will have no impact on the GCC port. If
anything, it'll make your life easier, since now you don't need a
patch to move it to 42-bits, it's just a compiler flag. That's why I
wanted to involve you in this discussion from now on, since we'll be
taking decisions that *will* affect you, and we need to be sure
they're the right decisions on both static and dynamic cases. But we
can't take any decision without hard data to go by, and that's why
we'll be investigating the performance compromises of each model.

cheers,
--renato

On Tue, Sep 29, 2015 at 02:38:18PM +0100, Renato Golin
wrote:
> My assumption is based on what I understood from my talks with various
> people in sanitizers, libraries, GCC, LLVM and the kernel. Please,
> correct me if I'm wrong.
> 
> IIUC, using larger VMAs on kernels with smaller VMAs work, but
> restrict the space that you can use as a shadow region, thus limiting
> the size of programs you can run with the sanitizers. Also, the higher
> you go, the higher is the levels of indirection you need to use
> memory, so using higher VMAs may be more of a performance hit than it
> should. This may not be such a big deal for 42 vs. 39 bits, but the
> new kernels will come with 48 bits, and there's talks to push it up to
> 56 bits in the near future.
> 
> So, there are only three paths we can take:
> 
> 1. Keep it constant, today at 42, and increase the constant slowly, as
> kernels start popping in with higher VMAs. This could slow down for
> large values of VMA and low values on kernel.
> 
> 2. Create a compiler flag -mvma=NN. This would be as fast as native
> when chosen correctly, but could break if lower than the machine's
> VMA.
> 
> 3. Make it dynamic, so that the VMA value doesn't matter.
You are mixing things up.  The size of virtual address space in bits is one
thing, and another one is the chosen ASAN shadow offset.
The way ASAN works is that for a normal memory address
corresponding shadow memory address is (addr >> 3) + shadow_offset
(the 3 hardwired at least into GCC, not sure about llvm).
Right now at least in GCC aarch64 shadow_offset is 1UL << 36, constant.
All you need to do is the math how the address space needs to look out
for the various VMA sizes, and whether it clashes with where the kernel
will normally try to allocate shared libraries or the stack.
If you have libsanitizer built for a particular VMA size and that
1UL << 36 shadow offset, you can also just ask the library to be verbose
through env var and dump you the layout (ASAN_OPTIONS=verbosity=1).
You'll get several regions of normal memory, several regions of shadow
memory and perhaps some gaps (shadow memory of shadow memory, which isn't
really useful).  What matters is whether the normal memory regions
in all those layouts cover normal location of stack, binaries, and where
kernel maps shared libraries.

	Jakub

On 29 September 2015 at 15:00, Jakub Jelinek <jakub at redhat.com>
wrote:
> You are mixing things up.  The size of virtual address space in bits is one
> thing, and another one is the chosen ASAN shadow offset.
Well, the shadow offset cannot overlap with the virtual address space,
and that's why the offset is different for different VMA values.

> All you need to do is the math how the address space needs to look out
> for the various VMA sizes, and whether it clashes with where the kernel
> will normally try to allocate shared libraries or the stack.
That's the part I'm not sure of. From my understanding, the kernel
*may* use memory beyond the stack, and that's largely based on the VMA
setting. Trying to find a value that works *well* for all alternatives
might be too restricting, or so I'm told.

I'd like the opinion of someone that understand the kernel better than
I do, however, to have a more informed decision. I'll be very happy to
be wrong, here, and start using 56-bits VMA for all AArch64 from now
on (as this may be a very likely future).

cheers,
--renato