llvm dev - Oct 2012 - [LLVMdev] dyld: lazy symbol binding failed: fast lazy bind offset out of range

On Sun, Oct 21, 2012 at 06:00:36PM -0700, Nick Kledzik
wrote:
> Jack,
> 
> Some binary has an initializer which dyld is calling.  Somehow the
initializer gets to:
> #4  0x0000000100f3b2c0 in Json::Value::maxUInt ()
> which is calling a function in another dylib for the first time.  When you
call a function in another dylib, you actually jump through a (lazy) pointer. 
The pointer initially points to a helper which loads an index parameter
specifying which function to bind, then jumps into dyld.  Dyld is erroring
because the index is out of range.
> 
> Either:
> 1) The binary is corrupt. You can check that by running:
>   dyldinfo -lazy_bind to see the lazy pointers
Nick,
    I attached dragonegg_so_lazy_bind.txt and LLVMPolly_so_lazy_bind.txt to
http://llvm.org/bugs/show_bug.cgi?id=14140.
Is there anything significant in those lists of lazy pointers relative to the
failure we are seeing? FYI, I see the same
failure for both x86_64-apple-darwin11 and x86_64-apple-darwin12 with current
llvm/polly/dragonegg svn using Xcode 4.5.1's
linker.
> or
> 2) Something else is wrong with the initializer and it happens to be
jumping to the helper (that jumps into dyld), but the index parameter is
garbage.  You'll need to step though this in the debugger.
>From the backtrace quoted below, can you suggest a logical place to set the
breakpoint as the starting point to step through this code?
Thanks in advance.
       Jack
> 
> -Nick
> 
> On Oct 21, 2012, at 9:27 AM, Jack Howarth wrote
> > Nick,
> >   While enhancing the fink llvm32 packaging to support the polly tool,
I ran into
> > a dyld failure when the dragonegg plugin tries to load LLVMPolly.so
plugin...
> > 
> > dyld: lazy symbol binding failed: fast lazy bind offset out of range
(39257, max=7640) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin11.4.2/4.7.2/cc1
> > dyld: fast lazy bind offset out of range (39257, max=7640) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin11.4.2/4.7.2/cc1
> > Trace/BPT trap
> > 
> > which backtraces as...
> > 
> > (gdb) bt
> > #0  0x00007fff5fc0106d in __dyld_dyld_fatal_error ()
> > #1  0x00007fff5fc048c8 in __dyld__ZN4dyld4haltEPKc ()
> > #2  0x00007fff5fc04948 in
__dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm ()
> > #3  0x00007fff90af7716 in dyld_stub_binder_ ()
> > #4  0x0000000100f3b2c0 in Json::Value::maxUInt ()
> > #5  0x00007fff5fc0fda6 in
__dyld__ZN16ImageLoaderMachO18doModInitFunctionsERKN11ImageLoader11LinkContextE
()
> > #6  0x00007fff5fc0faf2 in
__dyld__ZN16ImageLoaderMachO16doInitializationERKN11ImageLoader11LinkContextE ()
> > #7  0x00007fff5fc0d2e4 in
__dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEjRNS_21InitializerTimingListE
()
> > #8  0x00007fff5fc0e0b7 in
__dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextERNS_21InitializerTimingListE
()
> > #9  0x00007fff5fc031b9 in
__dyld__ZN4dyld15runInitializersEP11ImageLoader ()
> > #10 0x00007fff5fc09657 in __dyld_dlopen ()
> > #11 0x00007fff90af795b in dlopen ()
> > #12 0x0000000142f3888e in
llvm::sys::DynamicLibrary::getPermanentLibrary ()
> > #13 0x0000000142f256bd in llvm::PluginLoader::operator= ()
> > #14 0x000000014280fbda in llvm::cl::opt<llvm::PluginLoader, false,
llvm::cl::parser<std::string> >::handleOccurrence ()
> > #15 0x0000000142f1730c in CommaSeparateAndAddOccurence ()
> > #16 0x0000000142f1395f in ProvideOption ()
> > #17 0x0000000142f1224b in llvm::cl::ParseCommandLineOptions ()
> > #18 0x000000014280d3ff in InitializeBackend ()
> > #19 0x000000014280c191 in llvm_emit_globals ()
> > 
> > This will be difficult to testcase since the FSF gcc 4.7.2 compiler is
involved but I was hoping that you might be able
> > to make some guesses about the origin of the problem from the current
polly and dragonegg source code. There doesn't
> > seem to be any reports of this error type (but then again few plugins,
dragonegg, indirectly call additional plugins,
> > LLVMPolly. Thanks in advance for any hints.
> >           Jack

On Mon, Oct 22, 2012 at 09:48:59AM -0400, Jack Howarth
wrote:
> On Sun, Oct 21, 2012 at 06:00:36PM -0700, Nick Kledzik wrote:
> > Jack,
> > 
> > Some binary has an initializer which dyld is calling.  Somehow the
initializer gets to:
> > #4  0x0000000100f3b2c0 in Json::Value::maxUInt ()
> > which is calling a function in another dylib for the first time.  When
you call a function in another dylib, you actually jump through a (lazy)
pointer.  The pointer initially points to a helper which loads an index
parameter specifying which function to bind, then jumps into dyld.  Dyld is
erroring because the index is out of range.
> > 
> > Either:
> > 1) The binary is corrupt. You can check that by running:
> >   dyldinfo -lazy_bind to see the lazy pointers
> 
> Nick,
>     I attached dragonegg_so_lazy_bind.txt and LLVMPolly_so_lazy_bind.txt to
http://llvm.org/bugs/show_bug.cgi?id=14140.
> Is there anything significant in those lists of lazy pointers relative to
the failure we are seeing? FYI, I see the same
> failure for both x86_64-apple-darwin11 and x86_64-apple-darwin12 with
current llvm/polly/dragonegg svn using Xcode 4.5.1's
> linker.
> 
> > or
> > 2) Something else is wrong with the initializer and it happens to be
jumping to the helper (that jumps into dyld), but the index parameter is
garbage.  You'll need to step though this in the debugger.
> 
> >From the backtrace quoted below, can you suggest a logical place to set
the breakpoint as the starting point to step through this code?
> Thanks in advance.
>        Jack
> 
Nick,
   I have attached a gdb walk log (LLVMPolly_load_walk.log) to
http://llvm.org/bugs/show_bug.cgi?id=14140 which
steps from llvm-3.2/lib/Support/CommandLine.cpp, line 259. The tail end of the
log is as follows...

(gdb) 
llvm::sys::DynamicLibrary::getPermanentLibrary (filename=0x144002188
"/sw/opt/llvm-3.2/lib/LLVMPolly.so", errMsg=0x7fff5fbfe700) at
/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
77        void *handle = dlopen(filename, RTLD_LAZY|RTLD_GLOBAL);
(gdb) 
Reading symbols for shared libraries ... done
dyld: lazy symbol binding failed: fast lazy bind offset out of range (53437,
max=7640) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
dyld: fast lazy bind offset out of range (53437, max=7640) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

Let me know if there is something else I can provide to help debug this nested
plugin loading issue on darwin.
           Jack
ps This is for an llvm/polly debug cmake build with assertions on as well as
dragonegg built with -g.
> > 
> > -Nick
> > 
> > On Oct 21, 2012, at 9:27 AM, Jack Howarth wrote
> > > Nick,
> > >   While enhancing the fink llvm32 packaging to support the polly
tool, I ran into
> > > a dyld failure when the dragonegg plugin tries to load
LLVMPolly.so plugin...
> > > 
> > > dyld: lazy symbol binding failed: fast lazy bind offset out of
range (39257, max=7640) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin11.4.2/4.7.2/cc1
> > > dyld: fast lazy bind offset out of range (39257, max=7640) in
image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin11.4.2/4.7.2/cc1
> > > Trace/BPT trap
> > > 
> > > which backtraces as...
> > > 
> > > (gdb) bt
> > > #0  0x00007fff5fc0106d in __dyld_dyld_fatal_error ()
> > > #1  0x00007fff5fc048c8 in __dyld__ZN4dyld4haltEPKc ()
> > > #2  0x00007fff5fc04948 in
__dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm ()
> > > #3  0x00007fff90af7716 in dyld_stub_binder_ ()
> > > #4  0x0000000100f3b2c0 in Json::Value::maxUInt ()
> > > #5  0x00007fff5fc0fda6 in
__dyld__ZN16ImageLoaderMachO18doModInitFunctionsERKN11ImageLoader11LinkContextE
()
> > > #6  0x00007fff5fc0faf2 in
__dyld__ZN16ImageLoaderMachO16doInitializationERKN11ImageLoader11LinkContextE ()
> > > #7  0x00007fff5fc0d2e4 in
__dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEjRNS_21InitializerTimingListE
()
> > > #8  0x00007fff5fc0e0b7 in
__dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextERNS_21InitializerTimingListE
()
> > > #9  0x00007fff5fc031b9 in
__dyld__ZN4dyld15runInitializersEP11ImageLoader ()
> > > #10 0x00007fff5fc09657 in __dyld_dlopen ()
> > > #11 0x00007fff90af795b in dlopen ()
> > > #12 0x0000000142f3888e in
llvm::sys::DynamicLibrary::getPermanentLibrary ()
> > > #13 0x0000000142f256bd in llvm::PluginLoader::operator= ()
> > > #14 0x000000014280fbda in llvm::cl::opt<llvm::PluginLoader,
false, llvm::cl::parser<std::string> >::handleOccurrence ()
> > > #15 0x0000000142f1730c in CommaSeparateAndAddOccurence ()
> > > #16 0x0000000142f1395f in ProvideOption ()
> > > #17 0x0000000142f1224b in llvm::cl::ParseCommandLineOptions ()
> > > #18 0x000000014280d3ff in InitializeBackend ()
> > > #19 0x000000014280c191 in llvm_emit_globals ()
> > > 
> > > This will be difficult to testcase since the FSF gcc 4.7.2
compiler is involved but I was hoping that you might be able
> > > to make some guesses about the origin of the problem from the
current polly and dragonegg source code. There doesn't
> > > seem to be any reports of this error type (but then again few
plugins, dragonegg, indirectly call additional plugins,
> > > LLVMPolly. Thanks in advance for any hints.
> > >           Jack
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

Jack,

I looks like the code is calling dlopen() on LLVMPolly.so and it or something it
links against has an initializer.  The initialer is run before dlopen() returns
and the crash is in the initializer.  The message:

dyld: fast lazy bind offset out of range (53437, max=7640) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

means the initializer called something which made its way into the cc1 binary,
probably it called some weak symbol which resolved in the main executable cc1.

In gdb you can set:
(gdb) set env DYLD_PRINT_INITIALIZERS
and dyld will print out the address of each initializer before calling it.  You
can then run again and set a break point at that address and step through it.

Also, the lazy bind info for LLVMPolly.so was a little odd in that all symbols
were marked "flat-namespace".  On MacOSX we normally record in which
dylib each symbol should be found.  The use of flat-namespace tells dyld to look
everywhere for those symbols.  This can cause the wrong symbol to be used at
runtime.  Was LLVMPolly.so build with -flat_namespace or -undefined
dynamic_lookup?

-Nick

On Oct 22, 2012, at 8:07 AM, Jack Howarth wrote:
> On Mon, Oct 22, 2012 at 09:48:59AM -0400, Jack Howarth wrote:
>> On Sun, Oct 21, 2012 at 06:00:36PM -0700, Nick Kledzik wrote:
>>> Jack,
>>> 
>>> Some binary has an initializer which dyld is calling.  Somehow the
initializer gets to:
>>> #4  0x0000000100f3b2c0 in Json::Value::maxUInt ()
>>> which is calling a function in another dylib for the first time. 
When you call a function in another dylib, you actually jump through a (lazy)
pointer.  The pointer initially points to a helper which loads an index
parameter specifying which function to bind, then jumps into dyld.  Dyld is
erroring because the index is out of range.
>>> 
>>> Either:
>>> 1) The binary is corrupt. You can check that by running:
>>>  dyldinfo -lazy_bind to see the lazy pointers
>> 
>> Nick,
>>    I attached dragonegg_so_lazy_bind.txt and LLVMPolly_so_lazy_bind.txt
to http://llvm.org/bugs/show_bug.cgi?id=14140.
>> Is there anything significant in those lists of lazy pointers relative
to the failure we are seeing? FYI, I see the same
>> failure for both x86_64-apple-darwin11 and x86_64-apple-darwin12 with
current llvm/polly/dragonegg svn using Xcode 4.5.1's
>> linker.
>> 
>>> or
>>> 2) Something else is wrong with the initializer and it happens to
be jumping to the helper (that jumps into dyld), but the index parameter is
garbage.  You'll need to step though this in the debugger.
>> 
>>> From the backtrace quoted below, can you suggest a logical place to
set the breakpoint as the starting point to step through this code?
>> Thanks in advance.
>>       Jack
>> 
> 
> Nick,
>   I have attached a gdb walk log (LLVMPolly_load_walk.log) to
http://llvm.org/bugs/show_bug.cgi?id=14140 which
> steps from llvm-3.2/lib/Support/CommandLine.cpp, line 259. The tail end of
the log is as follows...
> 
> (gdb) 
> llvm::sys::DynamicLibrary::getPermanentLibrary (filename=0x144002188
"/sw/opt/llvm-3.2/lib/LLVMPolly.so", errMsg=0x7fff5fbfe700) at
/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
> 77        void *handle = dlopen(filename, RTLD_LAZY|RTLD_GLOBAL);
> (gdb) 
> Reading symbols for shared libraries ... done
> dyld: lazy symbol binding failed: fast lazy bind offset out of range
(53437, max=7640) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
> dyld: fast lazy bind offset out of range (53437, max=7640) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
> 
> Let me know if there is something else I can provide to help debug this
nested plugin loading issue on darwin.
>           Jack
> ps This is for an llvm/polly debug cmake build with assertions on as well
as dragonegg built with -g.
> 
>>> 
>>> -Nick
>>> 
>>> On Oct 21, 2012, at 9:27 AM, Jack Howarth wrote
>>>> Nick,
>>>>  While enhancing the fink llvm32 packaging to support the polly
tool, I ran into
>>>> a dyld failure when the dragonegg plugin tries to load
LLVMPolly.so plugin...
>>>> 
>>>> dyld: lazy symbol binding failed: fast lazy bind offset out of
range (39257, max=7640) in image
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin11.4.2/4.7.2/cc1
>>>> dyld: fast lazy bind offset out of range (39257, max=7640) in
image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin11.4.2/4.7.2/cc1
>>>> Trace/BPT trap
>>>> 
>>>> which backtraces as...
>>>> 
>>>> (gdb) bt
>>>> #0  0x00007fff5fc0106d in __dyld_dyld_fatal_error ()
>>>> #1  0x00007fff5fc048c8 in __dyld__ZN4dyld4haltEPKc ()
>>>> #2  0x00007fff5fc04948 in
__dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm ()
>>>> #3  0x00007fff90af7716 in dyld_stub_binder_ ()
>>>> #4  0x0000000100f3b2c0 in Json::Value::maxUInt ()
>>>> #5  0x00007fff5fc0fda6 in
__dyld__ZN16ImageLoaderMachO18doModInitFunctionsERKN11ImageLoader11LinkContextE
()
>>>> #6  0x00007fff5fc0faf2 in
__dyld__ZN16ImageLoaderMachO16doInitializationERKN11ImageLoader11LinkContextE ()
>>>> #7  0x00007fff5fc0d2e4 in
__dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEjRNS_21InitializerTimingListE
()
>>>> #8  0x00007fff5fc0e0b7 in
__dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextERNS_21InitializerTimingListE
()
>>>> #9  0x00007fff5fc031b9 in
__dyld__ZN4dyld15runInitializersEP11ImageLoader ()
>>>> #10 0x00007fff5fc09657 in __dyld_dlopen ()
>>>> #11 0x00007fff90af795b in dlopen ()
>>>> #12 0x0000000142f3888e in
llvm::sys::DynamicLibrary::getPermanentLibrary ()
>>>> #13 0x0000000142f256bd in llvm::PluginLoader::operator= ()
>>>> #14 0x000000014280fbda in llvm::cl::opt<llvm::PluginLoader,
false, llvm::cl::parser<std::string> >::handleOccurrence ()
>>>> #15 0x0000000142f1730c in CommaSeparateAndAddOccurence ()
>>>> #16 0x0000000142f1395f in ProvideOption ()
>>>> #17 0x0000000142f1224b in llvm::cl::ParseCommandLineOptions ()
>>>> #18 0x000000014280d3ff in InitializeBackend ()
>>>> #19 0x000000014280c191 in llvm_emit_globals ()
>>>> 
>>>> This will be difficult to testcase since the FSF gcc 4.7.2
compiler is involved but I was hoping that you might be able
>>>> to make some guesses about the origin of the problem from the
current polly and dragonegg source code. There doesn't
>>>> seem to be any reports of this error type (but then again few
plugins, dragonegg, indirectly call additional plugins,
>>>> LLVMPolly. Thanks in advance for any hints.
>>>>          Jack
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev