llvm dev - Nov 2012 - [LLVMdev] radr://12777299, "potential pthread/eh bug exposed by libsanitizer"

Nick,
   Can you take a quick look at the asan_eh_bug.tar.bz testcase
I uploaded into the newly opened radr://12777299, "potential 
pthread/eh bug exposed by libsanitizer". The FSF gcc developers
have ported llvm.org's asan code into FSF gcc (and are keeping
it synced to the upstream llvm.org code). I have been helping
with the darwin build and testing -fsanitize=address against the
complete FSF gcc testsuite. This seems to have exposed a potential
bug in pthread or eh on darwin under libasan. Hundreds of test cases
in the g++ and libstdc++ testsuites fail under -fsanitize=address
in the following manner...

ASAN:SIGSEGV
==================================================================2738== ERROR:
AddressSanitizer: SEGV on unknown address 0x0000ffd27000 (pc 0x0000ffd27000 sp
0x7fff55e40828 bp 0x7fff55e408f0 T0)
AddressSanitizer can not provide additional info.
    #0 0xffd26fff (/Users/howarth/asan_eh_bug/./cond1_asan.exe+0xf5f67fff)
    #1 0x7fff8bd827e0 (/usr/lib/system/libdyld.dylib+0x27e0)
    #2 0x0
Stats: 0M malloced (0M for red zones) by 3 calls
Stats: 0M realloced by 0 calls
Stats: 0M freed by 0 calls
Stats: 0M really freed by 0 calls
Stats: 1M (384 full pages) mmaped in 3 calls
  mmaps   by size class: 7:4095; 8:2047; 9:1023; 
  mallocs by size class: 7:1; 8:1; 9:1; 
  frees   by size class: 
  rfrees  by size class: 
Stats: malloc large: 0 small slow: 3
==2738== ABORTING

The failure of...

FAIL: g++.dg/eh/cond1.C -std=c++98 execution test

was used as the test case for the radar report and compiled with...

g++-fsf-4.8 -static-libasan -fsanitize=address -std=c++98 cond1.C -g -O0 -o
cond1_asan.exe

to produce the above failure. When compiled without libasan as...

g++-fsf-4.8 -std=c++98 cond1.C -g -O0 -o cond1_no_asan.exe

the resulting executable runs fine. Debugging this in gdb seems to show that the
failure
is occuring in the final call to dyld_stub_pthread_once (). The same test case
compiles fine with -fsanitize=address under llvm 3.2 clang++ and produces no
runtime errors
but the code execution path is very different in that case (because of the
different
libstdc++).
    Can you take a quick peek at this and determine if this is a darwin pthread
or unwinder
bug or an issue with libasan that FSF gcc's compiler is exposing? Thanks in
advance for
any help on this.
         Jack

+glider
The compiler hardly matters here, I would expect the same failures with
clang.
Alex, could you please take a look?

--kcc

On Thu, Nov 29, 2012 at 9:55 PM, Jack Howarth <howarth at
bromo.med.uc.edu>wrote:
> Nick,
>    Can you take a quick look at the asan_eh_bug.tar.bz testcase
> I uploaded into the newly opened radr://12777299, "potential
> pthread/eh bug exposed by libsanitizer". The FSF gcc developers
> have ported llvm.org's asan code into FSF gcc (and are keeping
> it synced to the upstream llvm.org code). I have been helping
> with the darwin build and testing -fsanitize=address against the
> complete FSF gcc testsuite. This seems to have exposed a potential
> bug in pthread or eh on darwin under libasan. Hundreds of test cases
> in the g++ and libstdc++ testsuites fail under -fsanitize=address
> in the following manner...
>
> ASAN:SIGSEGV
> ================================================================>
==2738== ERROR: AddressSanitizer: SEGV on unknown address 0x0000ffd27000
> (pc 0x0000ffd27000 sp 0x7fff55e40828 bp 0x7fff55e408f0 T0)
> AddressSanitizer can not provide additional info.
>     #0 0xffd26fff (/Users/howarth/asan_eh_bug/./cond1_asan.exe+0xf5f67fff)
>     #1 0x7fff8bd827e0 (/usr/lib/system/libdyld.dylib+0x27e0)
>     #2 0x0
> Stats: 0M malloced (0M for red zones) by 3 calls
> Stats: 0M realloced by 0 calls
> Stats: 0M freed by 0 calls
> Stats: 0M really freed by 0 calls
> Stats: 1M (384 full pages) mmaped in 3 calls
>   mmaps   by size class: 7:4095; 8:2047; 9:1023;
>   mallocs by size class: 7:1; 8:1; 9:1;
>   frees   by size class:
>   rfrees  by size class:
> Stats: malloc large: 0 small slow: 3
> ==2738== ABORTING
>
> The failure of...
>
> FAIL: g++.dg/eh/cond1.C -std=c++98 execution test
>
> was used as the test case for the radar report and compiled with...
>
> g++-fsf-4.8 -static-libasan -fsanitize=address -std=c++98 cond1.C -g -O0
> -o cond1_asan.exe
>
> to produce the above failure. When compiled without libasan as...
>
> g++-fsf-4.8 -std=c++98 cond1.C -g -O0 -o cond1_no_asan.exe
>
> the resulting executable runs fine. Debugging this in gdb seems to show
> that the failure
> is occuring in the final call to dyld_stub_pthread_once (). The same test
> case
> compiles fine with -fsanitize=address under llvm 3.2 clang++ and produces
> no runtime errors
> but the code execution path is very different in that case (because of the
> different
> libstdc++).
>     Can you take a quick peek at this and determine if this is a darwin
> pthread or unwinder
> bug or an issue with libasan that FSF gcc's compiler is exposing?
Thanks
> in advance for
> any help on this.
>          Jack
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20121129/c68781c3/attachment.html>

Jack, can you please upload this test somewhere?

On Thu, Nov 29, 2012 at 10:09 AM, Kostya Serebryany <kcc at google.com>
wrote:
> +glider
> The compiler hardly matters here, I would expect the same failures with
> clang.
> Alex, could you please take a look?
>
> --kcc
>
>
> On Thu, Nov 29, 2012 at 9:55 PM, Jack Howarth <howarth at
bromo.med.uc.edu>
> wrote:
>>
>> Nick,
>>    Can you take a quick look at the asan_eh_bug.tar.bz testcase
>> I uploaded into the newly opened radr://12777299, "potential
>> pthread/eh bug exposed by libsanitizer". The FSF gcc developers
>> have ported llvm.org's asan code into FSF gcc (and are keeping
>> it synced to the upstream llvm.org code). I have been helping
>> with the darwin build and testing -fsanitize=address against the
>> complete FSF gcc testsuite. This seems to have exposed a potential
>> bug in pthread or eh on darwin under libasan. Hundreds of test cases
>> in the g++ and libstdc++ testsuites fail under -fsanitize=address
>> in the following manner...
>>
>> ASAN:SIGSEGV
>>
================================================================>>
==2738== ERROR: AddressSanitizer: SEGV on unknown address 0x0000ffd27000
>> (pc 0x0000ffd27000 sp 0x7fff55e40828 bp 0x7fff55e408f0 T0)
>> AddressSanitizer can not provide additional info.
>>     #0 0xffd26fff
(/Users/howarth/asan_eh_bug/./cond1_asan.exe+0xf5f67fff)
>>     #1 0x7fff8bd827e0 (/usr/lib/system/libdyld.dylib+0x27e0)
>>     #2 0x0
>> Stats: 0M malloced (0M for red zones) by 3 calls
>> Stats: 0M realloced by 0 calls
>> Stats: 0M freed by 0 calls
>> Stats: 0M really freed by 0 calls
>> Stats: 1M (384 full pages) mmaped in 3 calls
>>   mmaps   by size class: 7:4095; 8:2047; 9:1023;
>>   mallocs by size class: 7:1; 8:1; 9:1;
>>   frees   by size class:
>>   rfrees  by size class:
>> Stats: malloc large: 0 small slow: 3
>> ==2738== ABORTING
>>
>> The failure of...
>>
>> FAIL: g++.dg/eh/cond1.C -std=c++98 execution test
>>
>> was used as the test case for the radar report and compiled with...
>>
>> g++-fsf-4.8 -static-libasan -fsanitize=address -std=c++98 cond1.C -g
-O0
>> -o cond1_asan.exe
>>
>> to produce the above failure. When compiled without libasan as...
>>
>> g++-fsf-4.8 -std=c++98 cond1.C -g -O0 -o cond1_no_asan.exe
>>
>> the resulting executable runs fine. Debugging this in gdb seems to show
>> that the failure
>> is occuring in the final call to dyld_stub_pthread_once (). The same
test
>> case
>> compiles fine with -fsanitize=address under llvm 3.2 clang++ and
produces
>> no runtime errors
>> but the code execution path is very different in that case (because of
the
>> different
>> libstdc++).
>>     Can you take a quick peek at this and determine if this is a darwin
>> pthread or unwinder
>> bug or an issue with libasan that FSF gcc's compiler is exposing?
Thanks
>> in advance for
>> any help on this.
>>          Jack
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>


-- 
Alexander Potapenko
Software Engineer
Google Moscow

On Thu, Nov 29, 2012 at 10:09:36PM +0400, Kostya Serebryany
wrote:
> +glider
> The compiler hardly matters here, I would expect the same failures with
> clang.
> Alex, could you please take a look?
> 
> --kcc
Kostya,
    The problem is that clang uses a very different and older libstdc++ than FSF
gcc so
that the code exection path is very different. For example...

% /sw/opt/llvm-3.2/bin/clang++ -fsanitize=address -std=c++98 cond1.C -g -O0 -o
cond1_asan_clang.exe
% gdb ./cond1_asan_clang.exe
...
(gdb) break main
Breakpoint 1 at 0x100001bc4: file cond1.C, line 21.
(gdb) r
...
Breakpoint 1, main (argc=<value temporarily unavailable, due to
optimizations>, argv=0x7fff5fbfe5a0) at cond1.C:21
21	    (argc+1 ? has_destructor() : throw 0);
(gdb) s
has_destructor::~has_destructor (this=<value temporarily unavailable, due to
optimizations>) at cond1.C:10
10	    ~has_destructor() { }
(gdb) 
has_destructor::~has_destructor (this=<value temporarily unavailable, due to
optimizations>) at cond1.C:10
10	    ~has_destructor() { }
(gdb) 
0x00000001000084e8 in has_destructor::~has_destructor (this=<value
temporarily unavailable, due to optimizations>) at cond1.C:10
10	    ~has_destructor() { }
(gdb) 
main (argc=<value temporarily unavailable, due to optimizations>,
argv=<value temporarily unavailable, due to optimizations>) at cond1.C:22
22	    CI((argc+1 ? throw 0 : has_destructor()));
(gdb) 

Program exited normally.

whereas for FSF gcc...

% g++-fsf-4.8 -static-libasan -fsanitize=address -std=c++98 cond1.C -g -O0 -o
cond1_asan.exe
% gdb ./cond1_asan.exe
...
(gdb) break main
Breakpoint 1 at 0x100001b35: file cond1.C, line 21.
(gdb) r
Starting program: /Users/howarth/asan_eh_bug/cond1_asan.exe 
Reading symbols for shared libraries ++++................................. done

Breakpoint 1, main (argc=1, argv=0x7fff5fbff898) at cond1.C:21
21	    (argc+1 ? has_destructor() : throw 0);
(gdb) s
has_destructor::~has_destructor (this=0x7fff5fbff7cf) at cond1.C:10
10	    ~has_destructor() { }
(gdb) 
main (argc=1, argv=0x7fff5fbff898) at cond1.C:22
22	    CI((argc+1 ? throw 0 : has_destructor()));
(gdb) 
__cxa_allocate_exception (thrown_size=132) at
../../../../gcc-4.8-20121127/libstdc++-v3/libsupc++/eh_alloc.cc:102
102	  ret = malloc (thrown_size);
(gdb) 
0x00000001022b60da in dyld_stub_malloc () at
../../../../gcc-4.8-20121127/libstdc++-v3/libsupc++/eh_alloc.cc:204
204	__cxxabiv1::__cxa_free_dependent_exception
(gdb) 
0x00007fff8bd80878 in dyld_stub_binder ()
(gdb) 
Single stepping until exit from function dyld_stub_binder, 
which has no line number information.
0x00007fff8bd808a5 in misaligned_stack_error_entering_dyld_stub_binder ()
(gdb) 
Single stepping until exit from function
misaligned_stack_error_entering_dyld_stub_binder,
which has no line number information.
0x00007fff8bd808e1 in dyld_stub_binder_ ()
(gdb) 
Single stepping until exit from function dyld_stub_binder_, 
which has no line number information.
0x00007fff8bd81f61 in _dyld_fast_stub_entry ()
(gdb) 
Single stepping until exit from function _Z21_dyld_fast_stub_entryPvl, 
which has no line number information.
0x00007fff5fc03fbd in __dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm ()
(gdb) 
Single stepping until exit from function
__dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm,
which has no line number information.
0x00007fff8bd808ee in dyld_stub_binder_ ()
(gdb) 
Single stepping until exit from function dyld_stub_binder_, 
which has no line number information.
0x00007fff94c3bb7e in malloc ()
(gdb) 
Single stepping until exit from function malloc, 
which has no line number information.
__cxa_allocate_exception (thrown_size=132) at
../../../../gcc-4.8-20121127/libstdc++-v3/libsupc++/eh_alloc.cc:104
104	  if (! ret)
(gdb) 
102	  ret = malloc (thrown_size);
(gdb) 
104	  if (! ret)
(gdb) 
132	  __cxa_eh_globals *globals = __cxa_get_globals ();
(gdb) 
__cxa_get_globals () at
../../../../gcc-4.8-20121127/libstdc++-v3/libsupc++/eh_globals.cc:63
63	{ return get_global(); }
(gdb) 
0x00000001022b6254 in dyld_stub___emutls_get_address () at
../../../../gcc-4.8-20121127/libstdc++-v3/libsupc++/eh_alloc.cc:204
204	__cxxabiv1::__cxa_free_dependent_exception
(gdb) 
0x00007fff8bd80878 in dyld_stub_binder ()
(gdb) 
Single stepping until exit from function dyld_stub_binder, 
which has no line number information.
0x00007fff8bd808a5 in misaligned_stack_error_entering_dyld_stub_binder ()
(gdb) 
Single stepping until exit from function
misaligned_stack_error_entering_dyld_stub_binder,
which has no line number information.
0x00007fff8bd808e1 in dyld_stub_binder_ ()
(gdb) 
Single stepping until exit from function dyld_stub_binder_, 
which has no line number information.
0x00007fff8bd81f61 in _dyld_fast_stub_entry ()
(gdb) 
Single stepping until exit from function _Z21_dyld_fast_stub_entryPvl, 
which has no line number information.
0x00007fff5fc03fbd in __dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm ()
(gdb) 
Single stepping until exit from function
__dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm,
which has no line number information.
0x00007fff8bd808ee in dyld_stub_binder_ ()
(gdb) 
Single stepping until exit from function dyld_stub_binder_, 
which has no line number information.
__emutls_get_address (obj=0x1022f9520) at
../../../gcc-4.8-20121127/libgcc/emutls.c:128
128	{
(gdb) 
Current language:  auto; currently c
139	  pointer offset = obj->loc.offset;
(gdb) 
141	  if (__builtin_expect (offset == 0, 0))
(gdb) 
700	    return __gthrw_(pthread_once) (__once, __func);
(gdb) 
0x00000001023f842a in dyld_stub_pthread_once ()
(gdb) 
Single stepping until exit from function dyld_stub_pthread_once, 
which has no line number information.

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x00000000ffd27000
0x00000000ffd27000 in ?? ()
(gdb) 

I did try....

% /sw/opt/llvm-3.2/bin/clang++ -fsanitize=address -std=c++98 cond1.C -g -O0
-stdlib=libc++ -o cond1_asan_clang.exe

using the libc++ from Mountain Lion. It doesn't trigger the bug either and
executes without calling pthread...

% ./cond1_asan_clang.exe
% gdb ./cond1_asan_clang.exe
...
(gdb) break main
Breakpoint 1 at 0x100001bc4: file cond1.C, line 21.
(gdb) r
Starting program: /Users/howarth/asan_eh_bug/cond1_asan_clang.exe 
Reading symbols for shared libraries +++................................ done

Breakpoint 1, main (argc=<value temporarily unavailable, due to
optimizations>, argv=0x7fff5fbfe5a0) at cond1.C:21
21	    (argc+1 ? has_destructor() : throw 0);
(gdb) s
has_destructor::~has_destructor (this=<value temporarily unavailable, due to
optimizations>) at cond1.C:10
10	    ~has_destructor() { }
(gdb) 
has_destructor::~has_destructor (this=<value temporarily unavailable, due to
optimizations>) at cond1.C:10
10	    ~has_destructor() { }
(gdb) 
0x00000001000084e8 in has_destructor::~has_destructor (this=<value
temporarily unavailable, due to optimizations>) at cond1.C:10
10	    ~has_destructor() { }
(gdb) 
main (argc=<value temporarily unavailable, due to optimizations>,
argv=<value temporarily unavailable, due to optimizations>) at cond1.C:22
22	    CI((argc+1 ? throw 0 : has_destructor()));
(gdb) 

Program exited normally.

Note that we have run into pthread bugs in the past which Apple has slowly
fixed. Also, IMHO anything
that uses the unwinder is always suspect on FSF gcc because we are the only
target that uses a system
which has subsumed the unwinder and libgcc calls out of libgcc (ie we don't
run the unwinder and the
libgcc calls present in the libgcc_s.10.5 stub from the FSF gcc files but rather
those from libSystem).
               Jack
> 
> On Thu, Nov 29, 2012 at 9:55 PM, Jack Howarth <howarth at
bromo.med.uc.edu>wrote:
> 
> > Nick,
> >    Can you take a quick look at the asan_eh_bug.tar.bz testcase
> > I uploaded into the newly opened radr://12777299, "potential
> > pthread/eh bug exposed by libsanitizer". The FSF gcc developers
> > have ported llvm.org's asan code into FSF gcc (and are keeping
> > it synced to the upstream llvm.org code). I have been helping
> > with the darwin build and testing -fsanitize=address against the
> > complete FSF gcc testsuite. This seems to have exposed a potential
> > bug in pthread or eh on darwin under libasan. Hundreds of test cases
> > in the g++ and libstdc++ testsuites fail under -fsanitize=address
> > in the following manner...
> >
> > ASAN:SIGSEGV
> > ================================================================>
> ==2738== ERROR: AddressSanitizer: SEGV on unknown address 0x0000ffd27000
> > (pc 0x0000ffd27000 sp 0x7fff55e40828 bp 0x7fff55e408f0 T0)
> > AddressSanitizer can not provide additional info.
> >     #0 0xffd26fff
(/Users/howarth/asan_eh_bug/./cond1_asan.exe+0xf5f67fff)
> >     #1 0x7fff8bd827e0 (/usr/lib/system/libdyld.dylib+0x27e0)
> >     #2 0x0
> > Stats: 0M malloced (0M for red zones) by 3 calls
> > Stats: 0M realloced by 0 calls
> > Stats: 0M freed by 0 calls
> > Stats: 0M really freed by 0 calls
> > Stats: 1M (384 full pages) mmaped in 3 calls
> >   mmaps   by size class: 7:4095; 8:2047; 9:1023;
> >   mallocs by size class: 7:1; 8:1; 9:1;
> >   frees   by size class:
> >   rfrees  by size class:
> > Stats: malloc large: 0 small slow: 3
> > ==2738== ABORTING
> >
> > The failure of...
> >
> > FAIL: g++.dg/eh/cond1.C -std=c++98 execution test
> >
> > was used as the test case for the radar report and compiled with...
> >
> > g++-fsf-4.8 -static-libasan -fsanitize=address -std=c++98 cond1.C -g
-O0
> > -o cond1_asan.exe
> >
> > to produce the above failure. When compiled without libasan as...
> >
> > g++-fsf-4.8 -std=c++98 cond1.C -g -O0 -o cond1_no_asan.exe
> >
> > the resulting executable runs fine. Debugging this in gdb seems to
show
> > that the failure
> > is occuring in the final call to dyld_stub_pthread_once (). The same
test
> > case
> > compiles fine with -fsanitize=address under llvm 3.2 clang++ and
produces
> > no runtime errors
> > but the code execution path is very different in that case (because of
the
> > different
> > libstdc++).
> >     Can you take a quick peek at this and determine if this is a
darwin
> > pthread or unwinder
> > bug or an issue with libasan that FSF gcc's compiler is exposing?
Thanks
> > in advance for
> > any help on this.
> >          Jack
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> >