Dan Liew via llvm-dev
2016-Feb-12 15:53 UTC
[llvm-dev] [cfe-dev] Buildling with/without AddressSanitizer causes divergent execution behaviour
On 11 February 2016 at 17:08, Reid Kleckner <rnk at google.com> wrote:> On Thu, Feb 11, 2016 at 5:53 AM, Dan Liew via cfe-dev > <cfe-dev at lists.llvm.org> wrote: >> >> > Can you somehow verify that this heap-use-after-free is happening? >> > E.g. print all the pointer values coming from memory::allocate, coming >> > into >> > memory::deallocate, and coming into sat::clause::operator[] >> > >> > If curious, check what size of quarantine is required to catch this bug >> > (ASAN_OPTIONS=quarantine_size_mb=N, default=256) >> > Valgrind may have smaller default quarantine and thus misses this bug. >> >> I was lazy and just told valgrind to execute the program (built by gcc >> without ASan) with the largest quarantine it supported. > > > There are some differences between Clang and GCC, such as left-to-right vs > right-to-left argument evaluation. This usually bites people who are moving > unique_ptr or vector and accessing it in the same expression. Given the > stack traces, I suspect that different orderings of refcount operations is > what's happening here. I would suggest building Z3 with Clang without ASan > and running valgrind on that.Yeah I tried building Z3 with Clang without ASan and running Valgrind. It didn't report any problems. I don't think this is a GCC vs Clang problem. When I build Z3 with GCC or Clang with ASan enabled both of them report the same heap-use-after-free issue. The original problem I reported (the divergent behaviour when building with and without ASan, i.e. the assert being hit/not hit respectively) should be disregarded. I was struggling to reproduce the issue just now and I gave up and then I accidently hit it again. Then when I tried to run valgrind on a binary (that wasn't supposed to be built with ASan) built by Z3's build system it complained about ASan. I think there's something with wrong with Z3's build system and reusing the same binary directory for non-asan / asan build is broken in some way. Lesson learnt, always use a fresh binary build directory! The heap-use-after-free issue is reproducible though. I'll report out what I found to the Z3 devs and leave it for now.>> >> > Did you try msan? >> >> I just have and it immediately reported a problem. I took a closer >> look and it looks like a false positive to me. > > ... > > This is because your STL was not compiled with MSan. The std::ofstream > constructor is provided by libstdc++, the writes are not instrumented, and > MSan never sees the initialization. Unfortunately, MSan is not very useful > unless you recompile your *entire* application minus glibc with msan. =/ > > It is possible to build an MSan-ified libc++ and use it if you want to keep > trying, but it's involved.Thanks for pointing out. Out of interest how is MSan able to work with a non MSan-ified glibc? Thanks, Dan.
Reid Kleckner via llvm-dev
2016-Feb-12 17:13 UTC
[llvm-dev] [cfe-dev] Buildling with/without AddressSanitizer causes divergent execution behaviour
On Fri, Feb 12, 2016 at 7:53 AM, Dan Liew <dan at su-root.co.uk> wrote:> > > This is because your STL was not compiled with MSan. The std::ofstream > > constructor is provided by libstdc++, the writes are not instrumented, > and > > MSan never sees the initialization. Unfortunately, MSan is not very > useful > > unless you recompile your *entire* application minus glibc with msan. =/ > > > > It is possible to build an MSan-ified libc++ and use it if you want to > keep > > trying, but it's involved. > > Thanks for pointing out. Out of interest how is MSan able to work with > a non MSan-ified glibc? >By intercepting almost the entire libc interface and annotating inputs and outputs: https://github.com/llvm-mirror/compiler-rt/blob/master/lib/sanitizer_common/sanitizer_common_interceptors.inc It's a C ABI boundary, so this is hard but feasible. Interposing all of libstdc++ is less feasible. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160212/cc59b051/attachment.html>
Dan Liew via llvm-dev
2016-Feb-17 18:48 UTC
[llvm-dev] [cfe-dev] Buildling with/without AddressSanitizer causes divergent execution behaviour
Hi, Well I dug into Z3's codebase a little more and figured out what the problem was. If you're curious see [1]. What worries me more is that prior to a heap-use-after-free being reported there an out of bounds write occurs but ASan doesn't catch it which seems like a bug to me. Note I'm using Clang 3.7.1 Here's how to reproduce (you need to build this on Linux on a x86_64 machine) ``` git clone -b asan_miss_out_of_bounds https://github.com/delcypher/z3-1.git src cd src CXX=clang++ CC=clang CXXFLAGS="-fno-omit-frame-pointer -fsanitize=address" LDFLAGS="-fsanitize=address" python scripts/mk_make.py --build build_clang_asan --noomp --debug cd build_clang_asan make -j4 make c_example LD_LIBRARY_PATH=`pwd` ./c_example ``` You should see output like ``` ... About to do out of bounds access! Did out of bounds access! If doing an ASan build do I get printed? ==================================================================12965==ERROR: AddressSanitizer: heap-use-after-free on address 0x60400005fe54 at pc 0x7f228dcad7a9 bp 0x7ffda15af250 sp 0x7ffda15af248 ... ``` In this version of Z3 I've added (see [2]) some printf statements that print a message just before doing an out of bounds write (``About to do out of bounds access!``) and then prints a message after doing it (``Did out of bounds access! If doing an ASan build do I get printed?``). If ASan were to catch the out of bounds access I would not expect the second message to be printed. However the second message gets printed and program continues running and later hits a heap-use-after-free. This seems like a bug to me. Thoughts? [1] https://github.com/Z3Prover/z3/issues/436#issuecomment-184713859 [2] https://github.com/delcypher/z3-1/commit/cc60cd483039af78604401abac703d9a903f74b6 Thanks, Dan.
Reid Kleckner via llvm-dev
2016-Feb-17 18:57 UTC
[llvm-dev] [cfe-dev] Buildling with/without AddressSanitizer causes divergent execution behaviour
On Wed, Feb 17, 2016 at 10:48 AM, Dan Liew <dan at su-root.co.uk> wrote:> Hi, > > Well I dug into Z3's codebase a little more and figured out what the > problem was. If you're curious see [1]. >Neat bug. :)> What worries me more is that prior to a heap-use-after-free being > reported there an out of bounds write occurs but ASan doesn't catch it > which seems like a bug to me. Note I'm using Clang 3.7.1 > ... > This seems like a bug to me. Thoughts? >m_segments is at the end of the clause_allocator object, which I'm assuming is allocated in another object sls here: clause_allocator m_alloc; // clause allocator clause_vector m_bin_clauses; // binary clauses The out-of-bounds access probably touches memory in m_bin_clauses. One of ASan's limitations is that it can't currently catch intra-object overflow: https://github.com/google/sanitizers/wiki/AddressSanitizerIntraObjectOverflow There's a prototype that adds padding to make it possible to catch this kind of bug, but I haven't seen anyone pushing it forward for a while now. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160217/9af254b0/attachment-0001.html>