[llvm-dev] Undefined symbols with inline functions using the ORC JIT on Linux

Martin Andersson via llvm-dev
Wed, 09 Jan 2019 16:12:59 -0000

Hi Lang,

On Tue, Jan 8, 2019 at 10:05 PM Lang Hames <lhames at gmail.com> wrote:

> Hi Martin,
>
> What is the behavior when your simple program fails on Windows?
>

When I run the simple program, "int main() { return 0; }", on Windows
with
lli (with orc-lazy) it simply prints the following and then exits:
JITDylib " " (ES: 0x000001b6e4ad3670):
Search order: [ (" ", all) ]
Symbol table:
    "__cxa_atexit": <not resolved> ( Lazy (MU=0x1b6e4ae3110),
[Data] )
    "main": <not resolved> ( Lazy (MU=0x1b6e4ae7f10), [Callable]
)
    "__dso_handle": <not resolved> ( Lazy (MU=0x1b6e4ae3110),
[Data] )


> Could you attach the IR that clang produces for 'clang -S -emit-llvm
> main.cpp'?
>

This is the IR:

; ModuleID = 'main.cpp'
source_filename = "main.cpp"
target datalayout = "e-m:w-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-pc-windows-msvc19.16.27024"

; Function Attrs: noinline norecurse nounwind optnone uwtable
define dso_local i32 @main() #0 {
  %1 = alloca i32, align 4
  store i32 0, i32* %1, align 4
  ret i32 0
}

attributes #0 = { noinline norecurse nounwind optnone uwtable
"correctly-rounded-divide-sqrt-fp-math"="false"
"disable-tail-calls"="false"
"less-precise-fpmad"="false"
"min-legal-vector-width"="0"
"no-frame-pointer-elim"="false"
"no-infs-fp-math"="false"
"no-jump-tables"="false"
"no-nans-fp-math"="false"
"no-signed-zeros-fp-math"="false"
"no-trapping-math"="false"
"stack-protector-buffer-size"="8"
"target-cpu"="x86-64"
"target-features"="+fxsr,+mmx,+sse,+sse2,+x87"
"unsafe-fp-math"="false"
"use-soft-float"="false" }

!llvm.module.flags = !{!0, !1}
!llvm.ident = !{!2}

!0 = !{i32 1, !"wchar_size", i32 2}
!1 = !{i32 7, !"PIC Level", i32 2}
!2 = !{!"clang version 8.0.0 (https://github.com/llvm-mirror/clang.git
f8ec7c38feebd5cccae31acc7a50182b5474bfa9) (
https://github.com/llvm-mirror/llvm.git
e7e4f1f171d0800bded962cc0181c6af5848d4d0)"}

I'll gladly try any eventual patches. I could also try to look into it
myself, my knowledge and time are both a bit limited though.

//Martin


>
> I suspect the problem has to do with the different ways that ORC and MCJIT
> (and ORC-MCJIT) treat C++ runtime calls. In MCJIT these fall through to the
> real runtime. As long as there are no global static destructors this works
> out fine. If static destructors are registered with the program runtime
> however, they will be executed *after* the JIT has been torn down which
> usually leads to crash when the runtime jumps into deallocated JIT'd
code.
> ORC tries to address this by intercepting certain runtime calls (in
> particular, __cxa_atexit) to register destructors with a simple c++ runtime
> managed by the JIT. That way the JIT can run static destructors before it
> deallocates code. I wrote and tested this on Darwin, and have tested it on
> Linux (though not recently), but I do not have access to a Windows machine
> to test with so it may well be broken there.
>
> Cheers,
> Lang.
>
>
> On Sat, Jan 5, 2019 at 3:45 AM Martin Andersson via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi Stefan,
>>
>> Thanks for your reply. I tried running my simple example on Linux using
>> lli and it does work fine. So I think the best long-term solution is to
>> migrate my code to the new lazy orc jit. Unfortunately, even the
simplest
>> example does not work on Windows:
>> int main() { return 0; }
>>
>> This is the output:
>> JITDylib " " (ES: 0x000001b6e4ad3670):
>> Search order: [ (" ", all) ]
>> Symbol table:
>>     "__cxa_atexit": <not resolved> ( Lazy
(MU=0x1b6e4ae3110), [Data] )
>>     "main": <not resolved> ( Lazy (MU=0x1b6e4ae7f10),
[Callable] )
>>     "__dso_handle": <not resolved> ( Lazy
(MU=0x1b6e4ae3110), [Data] )
>>
>> I run with these arguments, and I checked and -fno-use-cxa-atexit is
set
>> by clang:
>> clang -S -emit-llvm main.cpp
>> lli -jit-kind=orc-lazy main.ll
>>
>> Using mcjit and orc-mcjit as the jit-kind in lli works fine on Windows.
>>
>> I will keep investigating how to best proceed, thanks for pointing me
in
>> the right direction.
>>
>> //Martin
>>
>> On Fri, Jan 4, 2019 at 10:49 PM Stefan Gränitz <stefan.graenitz at
gmail.com>
>> wrote:
>>
>>> Hi Martin
>>>
>>> However, the inline functions are not in that set so they are not
>>> promoted to strong definitions. Shouldn't functions defined in
the jitted
>>> code be our responsibility?
>>> [...]
>>> This program does not work, clang-interpreter crashes because it
cannot
>>> find the symbol for the Test constructor function.
>>>
>>> class Test {
>>>     public: Test() {}
>>> };
>>>
>>> int main()
>>> {
>>>     Test test;
>>>     return 0;
>>> }
>>>
>>> You could compile your example to bitcode and run it with lli. This
will
>>> provide more information and the issue may be discussed easier on
bitcode
>>> level. (Not very familiar with clang-interpreter, but it looks more
like an
>>> example for illustration than a bulletproof tool.)
>>>
>>> Looking closer what happens with the ResponsibiltySet. When it is
>>> created it tries to to find symbols for all the names it knows
about.
>>> Eventually the look up request ends up in my application [...]
>>>
>>> Instead of using the legacy resolvers you might prefer a fallback
symbol
>>> generator here. The lli tool uses this approach to provide symbols
from the
>>> host process for the JITed code (see:
>>>
https://github.com/llvm-mirror/llvm/blob/8ffa038b3ef4448af8bf31f6c50281779939c774/tools/lli/lli.cpp#L807
>>> ).
>>>
>>> Hope it helps
>>> Stefan
>>>
>>> Am 04.01.19 um 18:49 schrieb Martin Andersson via llvm-dev:
>>>
>>> Hi,
>>>
>>> I am developing an application that uses the ORC api to JIT compile
C++
>>> code using Clang. So far I have done most of the work on Windows,
where it
>>> now mostly works as expected. However, when I tried to run my
application
>>> on Linux I ran into some problems.
>>>
>>> The problem I ran into is that symbols for jitted inline functions
>>> cannot be resolved. Both LLVM and Clang are checked out with latest
master
>>> branch.
>>>
>>> It is probably me that is doing something wrong but I cannot figure
out
>>> what it is. Here is what I found so far. In the file
RuntimeDyld.cpp in
>>> function loadObjectImpl, a check is made whether a particular
function is
>>> weak or not. Since inline functions are weak (as I understood it)
an
>>> attempt is made to promote this symbol to a strong definition. But
only if
>>> it is present in the "ResponsibilitySet", that check is
made on line 273.
>>> However, the inline functions are not in that set so they are not
promoted
>>> to strong definitions. Shouldn't functions defined in the
jitted code be
>>> our responsibility?
>>>
>>> Looking closer what happens with the ResponsibiltySet. When it is
>>> created it tries to to find symbols for all the names it knows
about.
>>> Eventually the look up request ends up in my application where I
use a
>>> LegacyIRCompileLayer to search for the symbol in the jitted module.
That
>>> function call eventually ends up in getSymbol in
RTDyldObjectLinkingLayer.h
>>> where the symbol is found but it does not have and address (Address
is zero
>>> and Flags is 50). So an instance of JITSymbol is returned to the
>>> LegacyRTDyldObjectLinkingLayer and the findSymbol function which
checks if
>>> an valid symbol was found (on line 406 in
RTDyldObjectLinkingLayer.h).
>>> Since the address is zero the JITSymbol is deemed to be not valid,
via the
>>> bool operator in JITSymbol.h and that is why that particular symbol
name
>>> does not end up in the ResponsibilitySet. That is as far as I got,
I don't
>>> know enough about LLVM to understand what the problem is (if any).
>>>
>>> This issue can be replicated with the clang-interpreter
application.
>>>
>>> This program does not work, clang-interpreter crashes because it
cannot
>>> find the symbol for the Test constructor function.
>>>
>>> class Test {
>>>     public: Test() {}
>>> };
>>>
>>> int main()
>>> {
>>>     Test test;
>>>     return 0;
>>> }
>>>
>>> This program works:
>>>
>>> class Test {
>>>     public: Test();
>>> };
>>>
>>> Test::Test() { }
>>>
>>> int main()
>>> {
>>>     Test test;
>>>     return 0;
>>> }
>>>
>>> The first version works on Windows since the inline constructor is
not
>>> marked as weak. Can anyone enlighten me on what is happening here?
Is this
>>> the expected behavior, and if it is, what am I doing wrong?
>>>
>>> Btw, I also tried various compiler flags (fno-inline and
>>> fno-inline-functions) but those do not help in this case.
>>>
>>> //Martin
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190109/54ed211f/attachment.html>