llvm dev - Oct 2012 - [LLVMdev] dyld: lazy symbol binding failed: fast lazy bind offset out of range

On Tue, Oct 23, 2012 at 01:05:04PM -0700, Nick Kledzik
wrote:
> On Oct 23, 2012, at 12:50 PM, Jack Howarth wrote:
> > On Mon, Oct 22, 2012 at 11:40:32AM -0700, Nick Kledzik wrote:
> >> 
> >> On Oct 22, 2012, at 11:34 AM, Jack Howarth wrote:
> >> 
> >>> Nick,
> >>>  I have uploaded the full walk with 'set env
DYLD_PRINT_INITIALIZERS'. It didn't seem very informative
> >>> as the dyld error occurs right after...
> >>> 
> >>> (gdb)
> >>> llvm::sys::DynamicLibrary::getPermanentLibrary
(filename=0x142903da8 "/sw/opt/llvm-3.2/lib/LLVMPolly.so",
errMsg=0x7fff5fbfe6e0) at
/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
> >>> 77        void *handle = dlopen(filename,
RTLD_LAZY|RTLD_GLOBAL);
> >>> (gdb)
> >>> Reading symbols for shared libraries ... done
> >>> dyld: calling initializer function 0x100eea5b0 in
/sw/opt/llvm-3.2/lib/LLVMPolly.so
> >> 
> >> You need to set a break point at 0x100eea5b0 and run again until
that break is hit, then start stepping.
> >> 
> >>> Are there any tricks for identifying the order that the
symbols are being resolved?
> >> (gdb) set env DYLD_PRINT_BINDINGS 1
> >> will show each use of a symbol as it is bound.  This can be *a
lot* of information.
> >> 
> >> -Nick
> > 
> > Nick,
> >   Can you clarify what I should be looking for here? I assumed you
expected that I should see some symbol
> > bindings occur immediately after the initializer function in
LLVMPolly.so is called but before dyld throws
> > the "fast lazy bind offset out of range" error message. In
the http://llvm.org/bugs/attachment.cgi?id=9397,
> > the log file for a walk with DYLD_PRINT_INITIALIZERS and
DYLD_PRINT_BINDINGS set on darwin12, I only see...
> > 
> > (gdb) 
> > llvm::sys::DynamicLibrary::getPermanentLibrary (filename=0x141328518
"/sw/opt/llvm-3.2/lib/LLVMPolly.so", errMsg=0x7fff5fbfe6d0) at
/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
> > 77        void *handle = dlopen(filename, RTLD_LAZY|RTLD_GLOBAL);
> > (gdb) 
> > Reading symbols for shared libraries ... done
> > dyld: bind: libisl.10.dylib:0x101292010 =
libsystem_c.dylib:__DefaultRuneLocale, *0x101292010 = 0x7FFF7A145890
> > dyld: bind: libisl.10.dylib:0x101292018 =
libsystem_c.dylib:___stack_chk_guard, *0x101292018 = 0x7FFF7A1474B0
> > ....
> > dyld: weak bind: LLVMPolly.so:0x100FB3128 = libc++.1.dylib:__ZdaPv,
*0x100FB3128 = 0x7FFF898BD1C6
> > dyld: weak bind: LLVMPolly.so:0x100FB3130 = libc++.1.dylib:__ZdlPv,
*0x100FB3130 = 0x7FFF898BD1AB
> > dyld: weak bind: LLVMPolly.so:0x100FB3138 = libc++.1.dylib:__Znam,
*0x100FB3138 = 0x7FFF898BD173
> > dyld: weak bind: LLVMPolly.so:0x100FB3140 = libc++.1.dylib:__Znwm,
*0x100FB3140 = 0x7FFF898BD0DF
> > dyld: calling initializer function 0x100ebb5b0 in
/sw/opt/llvm-3.2/lib/LLVMPolly.so
> > dyld: lazy symbol binding failed: fast lazy bind offset out of range
(53437, max=2928) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
> > dyld: fast lazy bind offset out of range (53437, max=2928) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
> > 
> > Program received signal SIGTRAP, Trace/breakpoint trap.
> > 0x00007fff5fc0109d in __dyld_dyld_fatal_error ()
> > 
> > where all of the bindings from LLVMPolly.so occur before the
initializer function is called. Was this behavior
> > expected?
> >   Also, as I noted on http://llvm.org/bugs/show_bug.cgi?id=14140#c13,
a static build of LLVMPolly.so on 10.6 exhibits the
> > same dyld error. Am I correct to assume that test proves that the
offending symbol can't be from libcloog-isl, libisl,
> > libgmp or libstdc++ because, when statically linked, those symbols
can't be obtained from cc1? Is it vaild to assume that the
> > offending symbol has to be both in the binding list seen when
LLVMPolly.so loads and the symbols (defined or undefined)
> > in cc1? Any suggestions on how to proceed in debugging this would be
appreciated.
> 
> As I said before, run once to get the address of the initializer being run
inside the dlopen call (0x100ebb5b0 in the case above).  Set a break point
there.  Run again.  When it stops at the break point (inside the initializer),
start stepping until it crashes.  You may need to step at the instruction level.
> 
> The point of the DYLD_PRINT_BINDINGS was so you can see what pointers dyld
was setting.  At some point in your trace you may be jumping through one of
those pointers.  You can check that the value makes sense and was not stomped by
something else.
> 
> -Nick
Nick,
   Can I do this without access to a debug version of dyld? Using the copy of
LLVMPolly.so with isl/cloog-isl/gmp statically linked,
I find that if I set the breakpoint to the address of the initializer...

dyld: calling initializer function 0x100ebb3a0 in
/sw/opt/llvm-3.2/lib/LLVMPolly.so
dyld: lazy symbol binding failed: fast lazy bind offset out of range (114808,
max=2928) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
dyld: fast lazy bind offset out of range (114808, max=2928) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

with...

(gdb) break *0x100ebb3a0
Breakpoint 2 at 0x100ebb3a0

this lands me at...

dyld: weak bind: LLVMPolly.so:0x1010F4BD0 = libc++.1.dylib:__Znwm, *0x1010F4BD0
= 0x7FFF898BD0DF

Breakpoint 2, 0x0000000100ebb3a0 in pch_address_space ()
(gdb) si
0x0000000100ebb3a1 in pch_address_space ()
(gdb) si
0x0000000100ebb3a4 in pch_address_space ()
(gdb) si
0x0000000100ebb380 in pch_address_space ()
(gdb) si
0x0000000100ebb381 in pch_address_space ()
...
and on in various dyld calls. Will I really be able to get anything useful from
this without a debug build of the system
dyld installed?
      Jack

> 
> 
>

On Oct 23, 2012, at 1:57 PM, Jack Howarth wrote:
> Nick,
>   Can I do this without access to a debug version of dyld? Using the copy
of LLVMPolly.so with isl/cloog-isl/gmp statically linked,
> I find that if I set the breakpoint to the address of the initializer...
> 
> dyld: calling initializer function 0x100ebb3a0 in
/sw/opt/llvm-3.2/lib/LLVMPolly.so
> dyld: lazy symbol binding failed: fast lazy bind offset out of range
(114808, max=2928) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
> dyld: fast lazy bind offset out of range (114808, max=2928) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
> 
> with...
> 
> (gdb) break *0x100ebb3a0
> Breakpoint 2 at 0x100ebb3a0
> 
> this lands me at...
> 
> dyld: weak bind: LLVMPolly.so:0x1010F4BD0 = libc++.1.dylib:__Znwm,
*0x1010F4BD0 = 0x7FFF898BD0DF
> 
> Breakpoint 2, 0x0000000100ebb3a0 in pch_address_space ()
> (gdb) si
> 0x0000000100ebb3a1 in pch_address_space ()
> (gdb) si
> 0x0000000100ebb3a4 in pch_address_space ()
> (gdb) si
> 0x0000000100ebb380 in pch_address_space ()
> (gdb) si
> 0x0000000100ebb381 in pch_address_space ()
> ...
> and on in various dyld calls. Will I really be able to get anything useful
from this without a debug build of the system
> dyld installed?
>   
You just need a build of LLVMPolly.so that does not have symbols stripped.  Or
get the .dSYM file for your build of LLVMPolly.so  and load that in gdb.

-Nick

On Tue, Oct 23, 2012 at 02:03:15PM -0700, Nick Kledzik
wrote:
> 
> On Oct 23, 2012, at 1:57 PM, Jack Howarth wrote:
> > Nick,
> >   Can I do this without access to a debug version of dyld? Using the
copy of LLVMPolly.so with isl/cloog-isl/gmp statically linked,
> > I find that if I set the breakpoint to the address of the
initializer...
> > 
> > dyld: calling initializer function 0x100ebb3a0 in
/sw/opt/llvm-3.2/lib/LLVMPolly.so
> > dyld: lazy symbol binding failed: fast lazy bind offset out of range
(114808, max=2928) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
> > dyld: fast lazy bind offset out of range (114808, max=2928) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
> > 
> > with...
> > 
> > (gdb) break *0x100ebb3a0
> > Breakpoint 2 at 0x100ebb3a0
> > 
> > this lands me at...
> > 
> > dyld: weak bind: LLVMPolly.so:0x1010F4BD0 = libc++.1.dylib:__Znwm,
*0x1010F4BD0 = 0x7FFF898BD0DF
> > 
> > Breakpoint 2, 0x0000000100ebb3a0 in pch_address_space ()
> > (gdb) si
> > 0x0000000100ebb3a1 in pch_address_space ()
> > (gdb) si
> > 0x0000000100ebb3a4 in pch_address_space ()
> > (gdb) si
> > 0x0000000100ebb380 in pch_address_space ()
> > (gdb) si
> > 0x0000000100ebb381 in pch_address_space ()
> > ...
> > and on in various dyld calls. Will I really be able to get anything
useful from this without a debug build of the system
> > dyld installed?
> >   
> You just need a build of LLVMPolly.so that does not have symbols stripped. 
Or get the .dSYM file for your build of LLVMPolly.so  and load that in gdb.
> 
> -Nick
> 
Nick,
   I have uploaded a bzip2 compressed log of the 'si' walk from the
failing initializer in LLVMPolly.so
to http://llvm.org/bugs/attachment.cgi?id=9408 in
http://llvm.org/bugs/show_bug.cgi?id=14140. The backtrace
from the final crash point shows...

(gdb) bt
#0  0x00007fff5fc0123f in
__dyld__ZN13dyldbootstrap5startEPK12macho_headeriPPKclS2_Pm ()
#1  0x00007fff5fc02138 in __dyld__ZN4dyld4haltEPKc ()
#2  0x00007fff5fc04048 in __dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm
()
#3  0x00007fff8bd808ee in dyld_stub_binder_ ()
#4  0x0000000100faf3e0 in Json::Value::maxUInt ()
#5  0x0000000100ebad65 in pch_address_space ()
#6  0x0000000100ebb5a0 in pch_address_space ()
#7  0x0000000100ebb5b9 in pch_address_space ()
#8  0x00007fff5fc13378 in
__dyld__ZN16ImageLoaderMachO18doModInitFunctionsERKN11ImageLoader11LinkContextE
()
#9  0x00007fff5fc13762 in
__dyld__ZN16ImageLoaderMachO16doInitializationERKN11ImageLoader11LinkContextE ()
#10 0x00007fff5fc1006e in
__dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEjRNS_21InitializerTimingListE
()
#11 0x00007fff5fc0feba in
__dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextERNS_21InitializerTimingListE
()
#12 0x00007fff5fc04e38 in __dyld__ZN4dyld15runInitializersEP11ImageLoader ()
#13 0x00007fff5fc0a87c in __dyld_dlopen ()
#14 0x00007fff8bd81dd8 in dlopen ()
#15 0x0000000142f9820f in llvm::sys::DynamicLibrary::getPermanentLibrary
(filename=0x141328d38
"/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so",
errMsg=0x7fff5fbfe6a0) at
/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
#16 0x0000000142f79ced in llvm::sys::DynamicLibrary::LoadLibraryPermanently
(Filename=0x141328d38
"/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so",
ErrMsg=0x7fff5fbfe6a0) at DynamicLibrary.h:77
#17 0x0000000142f79ab9 in llvm::PluginLoader::operator= (this=0x143497418,
Filename=@0x7fff5fbfe780) at
/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/PluginLoader.cpp:29
#18 0x00000001422137de in llvm::cl::opt_storage<llvm::PluginLoader, false,
true>::setValue<std::string> (this=0x143497418, V=@0x7fff5fbfe780,
initial=false) at CommandLine.h:1072
#19 0x0000000142213271 in llvm::cl::opt<llvm::PluginLoader, false,
llvm::cl::parser<std::string> >::handleOccurrence (this=0x1434973e0,
pos=2, ArgName={Data = 0x1413259f1
"load=/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so",
Length = 4, static npos = 18446744073709551615}, Arg={Data = 0x1413259f6
"/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so",
Length = 63, static npos = 18446744073709551615}) at CommandLine.h:1131
#20 0x0000000142f5e731 in llvm::cl::Option::addOccurrence (this=0x1434973e0,
pos=2, ArgName={Data = 0x1413259f1
"load=/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so",
Length = 4, static npos = 18446744073709551615}, Value={Data = 0x1413259f6
"/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so",
Length = 63, static npos = 18446744073709551615}, MultiArg=false) at
/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/CommandLine.cpp:883
#21 0x0000000142f635ab in CommaSeparateAndAddOccurence (Handler=0x1434973e0,
pos=2, ArgName={Data = 0x1413259f1
"load=/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so",
Length = 4, static npos = 18446744073709551615}, Value={Data = 0x1413259f6
"/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so",
Length = 63, static npos = 18446744073709551615}, MultiArg=false) at
/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/CommandLine.cpp:259
#22 0x0000000142f5ea31 in ProvideOption (Handler=0x1434973e0, ArgName={Data =
0x1413259f1
"load=/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so",
Length = 4, static npos = 18446744073709551615}, Value={Data = 0x1413259f6
"/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so",
Length = 63, static npos = 18446744073709551615}, argc=4, argv=0x141325a40,
i=@0x7fff5fbfefb0) at
/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/CommandLine.cpp:299
#23 0x0000000142f5c7d4 in llvm::cl::ParseCommandLineOptions (argc=4,
argv=0x141325a40, Overview=0x0) at
/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/CommandLine.cpp:724
#24 0x0000000142209df1 in ConfigureLLVM () at
/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/src/Backend.cpp:372
#25 0x0000000142208762 in InitializeBackend () at
/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/src/Backend.cpp:583
#26 0x00000001422079a8 in llvm_emit_globals () at
/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/src/Backend.cpp:1741
(gdb) 

The crash shows up as in the walk as...

(gdb) 
0x00007fff5fc24182 in __dyld_mach_init ()
(gdb) 
0x00007fff5fc01239 in
__dyld__ZN13dyldbootstrap5startEPK12macho_headeriPPKclS2_Pm ()
(gdb) 
0x00007fff5fc0123c in
__dyld__ZN13dyldbootstrap5startEPK12macho_headeriPPKclS2_Pm ()
(gdb) 
0x00007fff5fc0123f in
__dyld__ZN13dyldbootstrap5startEPK12macho_headeriPPKclS2_Pm ()
(gdb) 

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000008
0x00007fff5fc0123f in
__dyld__ZN13dyldbootstrap5startEPK12macho_headeriPPKclS2_Pm ()

The llvm build is built as a debug build as is the dragonegg build so both have
been
built with -g and are unstripped. I am still very unclear on how I am supposed
to
extract the offending symbol from this log. Since dyld seems to be from reading
pch,
I assume this is a symbol from libstdc++. This would seem odd since cc1
isn't built
with c++ on gcc 4.7.2 and isn't linked against libstdc++. The dragonegg
plugin that
dlopens the LLVMPolly plugin is built with c+ and linked against libstdc++ but I
verified that building it with the c compiler to avoid the libstdc++ linkage
doesn't
solve this bug. Thanks in advance for any analysis of the log.
           Jack