thr3ads.net - llvm dev - [llvm-dev] Undefined symbols with inline functions using the ORC JIT on Linux [Jan 2019]

If this information is useful, please help other people find it:
Share via:

Martin Andersson via llvm-dev

2019-Jan-05 11:44 UTC

[llvm-dev] Undefined symbols with inline functions using the ORC JIT on Linux

Hi Stefan,

Thanks for your reply. I tried running my simple example on Linux using lli
and it does work fine. So I think the best long-term solution is to migrate
my code to the new lazy orc jit. Unfortunately, even the simplest example
does not work on Windows:
int main() { return 0; }

This is the output:
JITDylib "<main>" (ES: 0x000001b6e4ad3670):
Search order: [ ("<main>", all) ]
Symbol table:
    "__cxa_atexit": <not resolved> ( Lazy (MU=0x1b6e4ae3110),
[Data] )
    "main": <not resolved> ( Lazy (MU=0x1b6e4ae7f10), [Callable]
)
    "__dso_handle": <not resolved> ( Lazy (MU=0x1b6e4ae3110),
[Data] )

I run with these arguments, and I checked and -fno-use-cxa-atexit is set by
clang:
clang -S -emit-llvm main.cpp
lli -jit-kind=orc-lazy main.ll

Using mcjit and orc-mcjit as the jit-kind in lli works fine on Windows.

I will keep investigating how to best proceed, thanks for pointing me in
the right direction.

//Martin

On Fri, Jan 4, 2019 at 10:49 PM Stefan Gränitz <stefan.graenitz at
gmail.com>
wrote:
> Hi Martin
>
> However, the inline functions are not in that set so they are not promoted
> to strong definitions. Shouldn't functions defined in the jitted code
be
> our responsibility?
> [...]
> This program does not work, clang-interpreter crashes because it cannot
> find the symbol for the Test constructor function.
>
> class Test {
>     public: Test() {}
> };
>
> int main()
> {
>     Test test;
>     return 0;
> }
>
> You could compile your example to bitcode and run it with lli. This will
> provide more information and the issue may be discussed easier on bitcode
> level. (Not very familiar with clang-interpreter, but it looks more like an
> example for illustration than a bulletproof tool.)
>
> Looking closer what happens with the ResponsibiltySet. When it is created
> it tries to to find symbols for all the names it knows about. Eventually
> the look up request ends up in my application [...]
>
> Instead of using the legacy resolvers you might prefer a fallback symbol
> generator here. The lli tool uses this approach to provide symbols from the
> host process for the JITed code (see:
>
https://github.com/llvm-mirror/llvm/blob/8ffa038b3ef4448af8bf31f6c50281779939c774/tools/lli/lli.cpp#L807
> ).
>
> Hope it helps
> Stefan
>
> Am 04.01.19 um 18:49 schrieb Martin Andersson via llvm-dev:
>
> Hi,
>
> I am developing an application that uses the ORC api to JIT compile C++
> code using Clang. So far I have done most of the work on Windows, where it
> now mostly works as expected. However, when I tried to run my application
> on Linux I ran into some problems.
>
> The problem I ran into is that symbols for jitted inline functions cannot
> be resolved. Both LLVM and Clang are checked out with latest master branch.
>
> It is probably me that is doing something wrong but I cannot figure out
> what it is. Here is what I found so far. In the file RuntimeDyld.cpp in
> function loadObjectImpl, a check is made whether a particular function is
> weak or not. Since inline functions are weak (as I understood it) an
> attempt is made to promote this symbol to a strong definition. But only if
> it is present in the "ResponsibilitySet", that check is made on
line 273.
> However, the inline functions are not in that set so they are not promoted
> to strong definitions. Shouldn't functions defined in the jitted code
be
> our responsibility?
>
> Looking closer what happens with the ResponsibiltySet. When it is created
> it tries to to find symbols for all the names it knows about. Eventually
> the look up request ends up in my application where I use a
> LegacyIRCompileLayer to search for the symbol in the jitted module. That
> function call eventually ends up in getSymbol in RTDyldObjectLinkingLayer.h
> where the symbol is found but it does not have and address (Address is zero
> and Flags is 50). So an instance of JITSymbol is returned to the
> LegacyRTDyldObjectLinkingLayer and the findSymbol function which checks if
> an valid symbol was found (on line 406 in RTDyldObjectLinkingLayer.h).
> Since the address is zero the JITSymbol is deemed to be not valid, via the
> bool operator in JITSymbol.h and that is why that particular symbol name
> does not end up in the ResponsibilitySet. That is as far as I got, I
don't
> know enough about LLVM to understand what the problem is (if any).
>
> This issue can be replicated with the clang-interpreter application.
>
> This program does not work, clang-interpreter crashes because it cannot
> find the symbol for the Test constructor function.
>
> class Test {
>     public: Test() {}
> };
>
> int main()
> {
>     Test test;
>     return 0;
> }
>
> This program works:
>
> class Test {
>     public: Test();
> };
>
> Test::Test() { }
>
> int main()
> {
>     Test test;
>     return 0;
> }
>
> The first version works on Windows since the inline constructor is not
> marked as weak. Can anyone enlighten me on what is happening here? Is this
> the expected behavior, and if it is, what am I doing wrong?
>
> Btw, I also tried various compiler flags (fno-inline and
> fno-inline-functions) but those do not help in this case.
>
> //Martin
>
>
> _______________________________________________
> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190105/5ed65281/attachment.html>

Lang Hames via llvm-dev

2019-Jan-08 21:05 UTC

head link

[llvm-dev] Undefined symbols with inline functions using the ORC JIT on Linux

Hi Martin,

What is the behavior when your simple program fails on Windows?

Could you attach the IR that clang produces for 'clang -S -emit-llvm
main.cpp'?

I suspect the problem has to do with the different ways that ORC and MCJIT
(and ORC-MCJIT) treat C++ runtime calls. In MCJIT these fall through to the
real runtime. As long as there are no global static destructors this works
out fine. If static destructors are registered with the program runtime
however, they will be executed *after* the JIT has been torn down which
usually leads to crash when the runtime jumps into deallocated JIT'd code.
ORC tries to address this by intercepting certain runtime calls (in
particular, __cxa_atexit) to register destructors with a simple c++ runtime
managed by the JIT. That way the JIT can run static destructors before it
deallocates code. I wrote and tested this on Darwin, and have tested it on
Linux (though not recently), but I do not have access to a Windows machine
to test with so it may well be broken there.

Cheers,
Lang.


On Sat, Jan 5, 2019 at 3:45 AM Martin Andersson via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi Stefan,
>
> Thanks for your reply. I tried running my simple example on Linux using
> lli and it does work fine. So I think the best long-term solution is to
> migrate my code to the new lazy orc jit. Unfortunately, even the simplest
> example does not work on Windows:
> int main() { return 0; }
>
> This is the output:
> JITDylib "<main>" (ES: 0x000001b6e4ad3670):
> Search order: [ ("<main>", all) ]
> Symbol table:
>     "__cxa_atexit": <not resolved> ( Lazy
(MU=0x1b6e4ae3110), [Data] )
>     "main": <not resolved> ( Lazy (MU=0x1b6e4ae7f10),
[Callable] )
>     "__dso_handle": <not resolved> ( Lazy
(MU=0x1b6e4ae3110), [Data] )
>
> I run with these arguments, and I checked and -fno-use-cxa-atexit is set
> by clang:
> clang -S -emit-llvm main.cpp
> lli -jit-kind=orc-lazy main.ll
>
> Using mcjit and orc-mcjit as the jit-kind in lli works fine on Windows.
>
> I will keep investigating how to best proceed, thanks for pointing me in
> the right direction.
>
> //Martin
>
> On Fri, Jan 4, 2019 at 10:49 PM Stefan Gränitz <stefan.graenitz at
gmail.com>
> wrote:
>
>> Hi Martin
>>
>> However, the inline functions are not in that set so they are not
>> promoted to strong definitions. Shouldn't functions defined in the
jitted
>> code be our responsibility?
>> [...]
>> This program does not work, clang-interpreter crashes because it cannot
>> find the symbol for the Test constructor function.
>>
>> class Test {
>>     public: Test() {}
>> };
>>
>> int main()
>> {
>>     Test test;
>>     return 0;
>> }
>>
>> You could compile your example to bitcode and run it with lli. This
will
>> provide more information and the issue may be discussed easier on
bitcode
>> level. (Not very familiar with clang-interpreter, but it looks more
like an
>> example for illustration than a bulletproof tool.)
>>
>> Looking closer what happens with the ResponsibiltySet. When it is
created
>> it tries to to find symbols for all the names it knows about.
Eventually
>> the look up request ends up in my application [...]
>>
>> Instead of using the legacy resolvers you might prefer a fallback
symbol
>> generator here. The lli tool uses this approach to provide symbols from
the
>> host process for the JITed code (see:
>>
https://github.com/llvm-mirror/llvm/blob/8ffa038b3ef4448af8bf31f6c50281779939c774/tools/lli/lli.cpp#L807
>> ).
>>
>> Hope it helps
>> Stefan
>>
>> Am 04.01.19 um 18:49 schrieb Martin Andersson via llvm-dev:
>>
>> Hi,
>>
>> I am developing an application that uses the ORC api to JIT compile C++
>> code using Clang. So far I have done most of the work on Windows, where
it
>> now mostly works as expected. However, when I tried to run my
application
>> on Linux I ran into some problems.
>>
>> The problem I ran into is that symbols for jitted inline functions
cannot
>> be resolved. Both LLVM and Clang are checked out with latest master
branch.
>>
>> It is probably me that is doing something wrong but I cannot figure out
>> what it is. Here is what I found so far. In the file RuntimeDyld.cpp in
>> function loadObjectImpl, a check is made whether a particular function
is
>> weak or not. Since inline functions are weak (as I understood it) an
>> attempt is made to promote this symbol to a strong definition. But only
if
>> it is present in the "ResponsibilitySet", that check is made
on line 273.
>> However, the inline functions are not in that set so they are not
promoted
>> to strong definitions. Shouldn't functions defined in the jitted
code be
>> our responsibility?
>>
>> Looking closer what happens with the ResponsibiltySet. When it is
created
>> it tries to to find symbols for all the names it knows about.
Eventually
>> the look up request ends up in my application where I use a
>> LegacyIRCompileLayer to search for the symbol in the jitted module.
That
>> function call eventually ends up in getSymbol in
RTDyldObjectLinkingLayer.h
>> where the symbol is found but it does not have and address (Address is
zero
>> and Flags is 50). So an instance of JITSymbol is returned to the
>> LegacyRTDyldObjectLinkingLayer and the findSymbol function which checks
if
>> an valid symbol was found (on line 406 in RTDyldObjectLinkingLayer.h).
>> Since the address is zero the JITSymbol is deemed to be not valid, via
the
>> bool operator in JITSymbol.h and that is why that particular symbol
name
>> does not end up in the ResponsibilitySet. That is as far as I got, I
don't
>> know enough about LLVM to understand what the problem is (if any).
>>
>> This issue can be replicated with the clang-interpreter application.
>>
>> This program does not work, clang-interpreter crashes because it cannot
>> find the symbol for the Test constructor function.
>>
>> class Test {
>>     public: Test() {}
>> };
>>
>> int main()
>> {
>>     Test test;
>>     return 0;
>> }
>>
>> This program works:
>>
>> class Test {
>>     public: Test();
>> };
>>
>> Test::Test() { }
>>
>> int main()
>> {
>>     Test test;
>>     return 0;
>> }
>>
>> The first version works on Windows since the inline constructor is not
>> marked as weak. Can anyone enlighten me on what is happening here? Is
this
>> the expected behavior, and if it is, what am I doing wrong?
>>
>> Btw, I also tried various compiler flags (fno-inline and
>> fno-inline-functions) but those do not help in this case.
>>
>> //Martin
>>
>>
>> _______________________________________________
>> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190108/c51df0e5/attachment.html>

Martin Andersson via llvm-dev

2019-Jan-09 16:12 UTC

head link

[llvm-dev] Undefined symbols with inline functions using the ORC JIT on Linux

Hi Lang,

On Tue, Jan 8, 2019 at 10:05 PM Lang Hames <lhames at gmail.com> wrote:
> Hi Martin,
>
> What is the behavior when your simple program fails on Windows?
>
When I run the simple program, "int main() { return 0; }", on Windows
with
lli (with orc-lazy) it simply prints the following and then exits:
JITDylib "<main>" (ES: 0x000001b6e4ad3670):
Search order: [ ("<main>", all) ]
Symbol table:
    "__cxa_atexit": <not resolved> ( Lazy (MU=0x1b6e4ae3110),
[Data] )
    "main": <not resolved> ( Lazy (MU=0x1b6e4ae7f10), [Callable]
)
    "__dso_handle": <not resolved> ( Lazy (MU=0x1b6e4ae3110),
[Data] )

> Could you attach the IR that clang produces for 'clang -S -emit-llvm
> main.cpp'?
>
This is the IR:

; ModuleID = 'main.cpp'
source_filename = "main.cpp"
target datalayout = "e-m:w-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-pc-windows-msvc19.16.27024"

; Function Attrs: noinline norecurse nounwind optnone uwtable
define dso_local i32 @main() #0 {
  %1 = alloca i32, align 4
  store i32 0, i32* %1, align 4
  ret i32 0
}

attributes #0 = { noinline norecurse nounwind optnone uwtable
"correctly-rounded-divide-sqrt-fp-math"="false"
"disable-tail-calls"="false"
"less-precise-fpmad"="false"
"min-legal-vector-width"="0"
"no-frame-pointer-elim"="false"
"no-infs-fp-math"="false"
"no-jump-tables"="false"
"no-nans-fp-math"="false"
"no-signed-zeros-fp-math"="false"
"no-trapping-math"="false"
"stack-protector-buffer-size"="8"
"target-cpu"="x86-64"
"target-features"="+fxsr,+mmx,+sse,+sse2,+x87"
"unsafe-fp-math"="false"
"use-soft-float"="false" }

!llvm.module.flags = !{!0, !1}
!llvm.ident = !{!2}

!0 = !{i32 1, !"wchar_size", i32 2}
!1 = !{i32 7, !"PIC Level", i32 2}
!2 = !{!"clang version 8.0.0 (https://github.com/llvm-mirror/clang.git
f8ec7c38feebd5cccae31acc7a50182b5474bfa9) (
https://github.com/llvm-mirror/llvm.git
e7e4f1f171d0800bded962cc0181c6af5848d4d0)"}

I'll gladly try any eventual patches. I could also try to look into it
myself, my knowledge and time are both a bit limited though.

//Martin

>
> I suspect the problem has to do with the different ways that ORC and MCJIT
> (and ORC-MCJIT) treat C++ runtime calls. In MCJIT these fall through to the
> real runtime. As long as there are no global static destructors this works
> out fine. If static destructors are registered with the program runtime
> however, they will be executed *after* the JIT has been torn down which
> usually leads to crash when the runtime jumps into deallocated JIT'd
code.
> ORC tries to address this by intercepting certain runtime calls (in
> particular, __cxa_atexit) to register destructors with a simple c++ runtime
> managed by the JIT. That way the JIT can run static destructors before it
> deallocates code. I wrote and tested this on Darwin, and have tested it on
> Linux (though not recently), but I do not have access to a Windows machine
> to test with so it may well be broken there.
>
> Cheers,
> Lang.
>
>
> On Sat, Jan 5, 2019 at 3:45 AM Martin Andersson via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi Stefan,
>>
>> Thanks for your reply. I tried running my simple example on Linux using
>> lli and it does work fine. So I think the best long-term solution is to
>> migrate my code to the new lazy orc jit. Unfortunately, even the
simplest
>> example does not work on Windows:
>> int main() { return 0; }
>>
>> This is the output:
>> JITDylib "<main>" (ES: 0x000001b6e4ad3670):
>> Search order: [ ("<main>", all) ]
>> Symbol table:
>>     "__cxa_atexit": <not resolved> ( Lazy
(MU=0x1b6e4ae3110), [Data] )
>>     "main": <not resolved> ( Lazy (MU=0x1b6e4ae7f10),
[Callable] )
>>     "__dso_handle": <not resolved> ( Lazy
(MU=0x1b6e4ae3110), [Data] )
>>
>> I run with these arguments, and I checked and -fno-use-cxa-atexit is
set
>> by clang:
>> clang -S -emit-llvm main.cpp
>> lli -jit-kind=orc-lazy main.ll
>>
>> Using mcjit and orc-mcjit as the jit-kind in lli works fine on Windows.
>>
>> I will keep investigating how to best proceed, thanks for pointing me
in
>> the right direction.
>>
>> //Martin
>>
>> On Fri, Jan 4, 2019 at 10:49 PM Stefan Gränitz <stefan.graenitz at
gmail.com>
>> wrote:
>>
>>> Hi Martin
>>>
>>> However, the inline functions are not in that set so they are not
>>> promoted to strong definitions. Shouldn't functions defined in
the jitted
>>> code be our responsibility?
>>> [...]
>>> This program does not work, clang-interpreter crashes because it
cannot
>>> find the symbol for the Test constructor function.
>>>
>>> class Test {
>>>     public: Test() {}
>>> };
>>>
>>> int main()
>>> {
>>>     Test test;
>>>     return 0;
>>> }
>>>
>>> You could compile your example to bitcode and run it with lli. This
will
>>> provide more information and the issue may be discussed easier on
bitcode
>>> level. (Not very familiar with clang-interpreter, but it looks more
like an
>>> example for illustration than a bulletproof tool.)
>>>
>>> Looking closer what happens with the ResponsibiltySet. When it is
>>> created it tries to to find symbols for all the names it knows
about.
>>> Eventually the look up request ends up in my application [...]
>>>
>>> Instead of using the legacy resolvers you might prefer a fallback
symbol
>>> generator here. The lli tool uses this approach to provide symbols
from the
>>> host process for the JITed code (see:
>>>
https://github.com/llvm-mirror/llvm/blob/8ffa038b3ef4448af8bf31f6c50281779939c774/tools/lli/lli.cpp#L807
>>> ).
>>>
>>> Hope it helps
>>> Stefan
>>>
>>> Am 04.01.19 um 18:49 schrieb Martin Andersson via llvm-dev:
>>>
>>> Hi,
>>>
>>> I am developing an application that uses the ORC api to JIT compile
C++
>>> code using Clang. So far I have done most of the work on Windows,
where it
>>> now mostly works as expected. However, when I tried to run my
application
>>> on Linux I ran into some problems.
>>>
>>> The problem I ran into is that symbols for jitted inline functions
>>> cannot be resolved. Both LLVM and Clang are checked out with latest
master
>>> branch.
>>>
>>> It is probably me that is doing something wrong but I cannot figure
out
>>> what it is. Here is what I found so far. In the file
RuntimeDyld.cpp in
>>> function loadObjectImpl, a check is made whether a particular
function is
>>> weak or not. Since inline functions are weak (as I understood it)
an
>>> attempt is made to promote this symbol to a strong definition. But
only if
>>> it is present in the "ResponsibilitySet", that check is
made on line 273.
>>> However, the inline functions are not in that set so they are not
promoted
>>> to strong definitions. Shouldn't functions defined in the
jitted code be
>>> our responsibility?
>>>
>>> Looking closer what happens with the ResponsibiltySet. When it is
>>> created it tries to to find symbols for all the names it knows
about.
>>> Eventually the look up request ends up in my application where I
use a
>>> LegacyIRCompileLayer to search for the symbol in the jitted module.
That
>>> function call eventually ends up in getSymbol in
RTDyldObjectLinkingLayer.h
>>> where the symbol is found but it does not have and address (Address
is zero
>>> and Flags is 50). So an instance of JITSymbol is returned to the
>>> LegacyRTDyldObjectLinkingLayer and the findSymbol function which
checks if
>>> an valid symbol was found (on line 406 in
RTDyldObjectLinkingLayer.h).
>>> Since the address is zero the JITSymbol is deemed to be not valid,
via the
>>> bool operator in JITSymbol.h and that is why that particular symbol
name
>>> does not end up in the ResponsibilitySet. That is as far as I got,
I don't
>>> know enough about LLVM to understand what the problem is (if any).
>>>
>>> This issue can be replicated with the clang-interpreter
application.
>>>
>>> This program does not work, clang-interpreter crashes because it
cannot
>>> find the symbol for the Test constructor function.
>>>
>>> class Test {
>>>     public: Test() {}
>>> };
>>>
>>> int main()
>>> {
>>>     Test test;
>>>     return 0;
>>> }
>>>
>>> This program works:
>>>
>>> class Test {
>>>     public: Test();
>>> };
>>>
>>> Test::Test() { }
>>>
>>> int main()
>>> {
>>>     Test test;
>>>     return 0;
>>> }
>>>
>>> The first version works on Windows since the inline constructor is
not
>>> marked as weak. Can anyone enlighten me on what is happening here?
Is this
>>> the expected behavior, and if it is, what am I doing wrong?
>>>
>>> Btw, I also tried various compiler flags (fno-inline and
>>> fno-inline-functions) but those do not help in this case.
>>>
>>> //Martin
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190109/54ed211f/attachment.html>

llvm dev - Jan 2019 - Undefined symbols with inline functions using the ORC JIT on Linux

[llvm-dev] Undefined symbols with inline functions using the ORC JIT on Linux

[llvm-dev] Undefined symbols with inline functions using the ORC JIT on Linux

[llvm-dev] Undefined symbols with inline functions using the ORC JIT on Linux