Andrew,
I've attached a small reproduction of the issue.
Reproduce by:
$ /usr/bin/g++ `llvm-config --cxxflags` -g -m32 -c mcjit_external_symbol.cpp
$ /usr/bin/g++ `llvm-config --ldflags` -g -m32 -o mcjit_external_symbol
mcjit_external_symbol.o `llvm-config --libs all`
$ ./mcjit_external_symbol
verifying...
LLVM ERROR: Program used external function '_external' which could not
be resolved!
Note that using JIT the code works fine, in case it might help.
Eran.
From: "Kaylor, Andrew" <andrew.kaylor at
intel.com<mailto:andrew.kaylor at intel.com>>
Date: Wed, 10 Apr 2013 20:50:14 -0400
To: Eran Weiss <eran.weiss at emc.com<mailto:eran.weiss at emc.com>>
Cc: "llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>"
<llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>, Jim
Grosbach <grosbach at apple.com<mailto:grosbach at apple.com>>,
Jiong Wang <jiwang at tilera.com<mailto:jiwang at tilera.com>>
Subject: RE: [LLVMdev] Migration from JIT to MCJIT
Eran,
Is there any chance you could boil this down to a small reproducer?
I’ve got a mid-to-long-term goal of getting rid of the ugliness in LLDB that Jim
mentioned, and fixing your problem would be a good first step.
Thanks,
Andy
From: Jim Grosbach [mailto:grosbach at apple.com]
Sent: Wednesday, April 10, 2013 4:19 PM
To: Kaylor, Andrew; Jiong Wang
Cc: Weiss, Eran; llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
Existing clients (LLDB) deal with externals by resolving them to constant
function pointers that are referenced in the IR. That’s obviously ugly as hell,
but it gets things done.
The old JIT was able to simplify things because it assumed the JITed code was
running in the same process as the JIT compiler. The MCJIT doesn’t assume that,
so it has to handle more possibilities. For example, that the invoked function
may be in a dylib which hasn’t yet been loaded into the target address space. I
suggest having a look at the ld64, lld and dyld implementations to see what
these relocation really imply. You won’t have to solve all of the potential
scenarios to get anything done, but you will want to make sure to carefully
check which use-case you’re in and error out when it’s one you haven’t handled
yet. The worst case is if the code makes overly broad assumptions about possible
inputs and resolves a relocation incorrectly in some subtle way.
-Jim
On Apr 10, 2013, at 11:49 AM, "Kaylor, Andrew" <andrew.kaylor at
intel.com<mailto:andrew.kaylor at intel.com>> wrote:
The MachO handling isn’t always straightforward in that it has a lot of bit
fields to handle while it’s also trying to accommodate the format abstractions
baked into the interfaces. The actual relocation type gets extracted in the
RuntimeDyldMachO::resolveRelocation function (RuntimeDyldMachO.cpp:32) before it
is passed on to the architecture-specific handlers. So the mask you mentioned
is probably OK.
The fact that this relocation type isn’t handled is, of course, the root of your
problem.
MCJIT has grown on an as-needed basis up to now, with only the relocation types
we were actually seeing being implemented. Unfortunately, I don’t know enough
about MachO to help with this one. Maybe Jim Grosbach can help?
-Andy
From: llvmdev-bounces at cs.uiuc.edu<mailto:llvmdev-bounces at
cs.uiuc.edu> [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Weiss, Eran
Sent: Tuesday, April 09, 2013 11:51 PM
To: Jiong Wang
Cc: llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
Thank you for the help.
The relocation type value is anded with 0xffffffffL. (RuntimeDyldMachO.cpp:214)
Maybe this mask should be different?
Anyway, it seems like this relocation isn't implemented.
(RuntimeDyldMachO.cpp:104)
From: Jiong Wang <jiwang at tilera.com<mailto:jiwang at tilera.com>>
Date: Tue, 9 Apr 2013 09:42:03 -0400
To: Eran Weiss <eran.weiss at emc.com<mailto:eran.weiss at emc.com>>
Cc: "llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>"
<llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
于 2013/4/9 21:08, Weiss, Eran 写道:
Hi,
I'm migrating my code (running on mac) from using JIT to MCJIT. My code
generates in memory, mostly using the llvm-c api, and then runs the generated
code.
When I try to use MCJIT I encounter a problem with relocations of external
symbols – functions compiled statically beforehand with gcc.
I get the following error:
Invalid relocation type!
UNREACHABLE executed at
/Users/weisse4/dev/llvm/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldMachO.cpp:89
While debugging, I see that the relocation type read in is 218103811, which
seems corrupt to me.
Hi Weiss,
I do not have any experience on Mach binary format, but the hex value of
218103811 is 0xd000003 (maybe the relocation type is
RIT_Generic_PreboundLazyPointer = 3), something looks like a relocation entry
composed of "symbol index" + "relocation type".
maybe something is wrong, that the relocation entry is not anded with a mask
to get the final relocation type.
---
Regards,
Jiong
Did someone encounter a similar error? Or can direct me to changes that I need
to do while migrating from JIT to MCJIT?
Thanks.
_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>
http://llvm.cs.uiuc.edu<http://llvm.cs.uiuc.edu/>http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130411/28d4700a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mcjit_external_symbol.cpp
Type: application/octet-stream
Size: 2680 bytes
Desc: mcjit_external_symbol.cpp
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130411/28d4700a/attachment.obj>
Thanks, Eran.
I’m not sure how soon I’ll have a solution for you, but it’s on my to-do list
now. I’ll also create a bugzilla record for this problem.
-Andy
From: Weiss, Eran [mailto:Eran.Weiss at emc.com]
Sent: Thursday, April 11, 2013 12:40 AM
To: Kaylor, Andrew
Cc: llvmdev at cs.uiuc.edu; Jim Grosbach; Jiong Wang
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
Andrew,
I've attached a small reproduction of the issue.
Reproduce by:
$ /usr/bin/g++ `llvm-config --cxxflags` -g -m32 -c mcjit_external_symbol.cpp
$ /usr/bin/g++ `llvm-config --ldflags` -g -m32 -o mcjit_external_symbol
mcjit_external_symbol.o `llvm-config --libs all`
$ ./mcjit_external_symbol
verifying...
LLVM ERROR: Program used external function '_external' which could not
be resolved!
Note that using JIT the code works fine, in case it might help.
Eran.
From: "Kaylor, Andrew" <andrew.kaylor at
intel.com<mailto:andrew.kaylor at intel.com>>
Date: Wed, 10 Apr 2013 20:50:14 -0400
To: Eran Weiss <eran.weiss at emc.com<mailto:eran.weiss at emc.com>>
Cc: "llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>"
<llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>, Jim
Grosbach <grosbach at apple.com<mailto:grosbach at apple.com>>,
Jiong Wang <jiwang at tilera.com<mailto:jiwang at tilera.com>>
Subject: RE: [LLVMdev] Migration from JIT to MCJIT
Eran,
Is there any chance you could boil this down to a small reproducer?
I’ve got a mid-to-long-term goal of getting rid of the ugliness in LLDB that Jim
mentioned, and fixing your problem would be a good first step.
Thanks,
Andy
From: Jim Grosbach [mailto:grosbach at apple.com]
Sent: Wednesday, April 10, 2013 4:19 PM
To: Kaylor, Andrew; Jiong Wang
Cc: Weiss, Eran; llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
Existing clients (LLDB) deal with externals by resolving them to constant
function pointers that are referenced in the IR. That’s obviously ugly as hell,
but it gets things done.
The old JIT was able to simplify things because it assumed the JITed code was
running in the same process as the JIT compiler. The MCJIT doesn’t assume that,
so it has to handle more possibilities. For example, that the invoked function
may be in a dylib which hasn’t yet been loaded into the target address space. I
suggest having a look at the ld64, lld and dyld implementations to see what
these relocation really imply. You won’t have to solve all of the potential
scenarios to get anything done, but you will want to make sure to carefully
check which use-case you’re in and error out when it’s one you haven’t handled
yet. The worst case is if the code makes overly broad assumptions about possible
inputs and resolves a relocation incorrectly in some subtle way.
-Jim
On Apr 10, 2013, at 11:49 AM, "Kaylor, Andrew" <andrew.kaylor at
intel.com<mailto:andrew.kaylor at intel.com>> wrote:
The MachO handling isn’t always straightforward in that it has a lot of bit
fields to handle while it’s also trying to accommodate the format abstractions
baked into the interfaces. The actual relocation type gets extracted in the
RuntimeDyldMachO::resolveRelocation function (RuntimeDyldMachO.cpp:32) before it
is passed on to the architecture-specific handlers. So the mask you mentioned
is probably OK.
The fact that this relocation type isn’t handled is, of course, the root of your
problem.
MCJIT has grown on an as-needed basis up to now, with only the relocation types
we were actually seeing being implemented. Unfortunately, I don’t know enough
about MachO to help with this one. Maybe Jim Grosbach can help?
-Andy
From: llvmdev-bounces at cs.uiuc.edu<mailto:llvmdev-bounces at
cs.uiuc.edu> [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Weiss, Eran
Sent: Tuesday, April 09, 2013 11:51 PM
To: Jiong Wang
Cc: llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
Thank you for the help.
The relocation type value is anded with 0xffffffffL. (RuntimeDyldMachO.cpp:214)
Maybe this mask should be different?
Anyway, it seems like this relocation isn't implemented.
(RuntimeDyldMachO.cpp:104)
From: Jiong Wang <jiwang at tilera.com<mailto:jiwang at tilera.com>>
Date: Tue, 9 Apr 2013 09:42:03 -0400
To: Eran Weiss <eran.weiss at emc.com<mailto:eran.weiss at emc.com>>
Cc: "llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>"
<llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
于 2013/4/9 21:08, Weiss, Eran 写道:
Hi,
I'm migrating my code (running on mac) from using JIT to MCJIT. My code
generates in memory, mostly using the llvm-c api, and then runs the generated
code.
When I try to use MCJIT I encounter a problem with relocations of external
symbols – functions compiled statically beforehand with gcc.
I get the following error:
Invalid relocation type!
UNREACHABLE executed at
/Users/weisse4/dev/llvm/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldMachO.cpp:89
While debugging, I see that the relocation type read in is 218103811, which
seems corrupt to me.
Hi Weiss,
I do not have any experience on Mach binary format, but the hex value of
218103811 is 0xd000003 (maybe the relocation type is
RIT_Generic_PreboundLazyPointer = 3), something looks like a relocation entry
composed of "symbol index" + "relocation type".
maybe something is wrong, that the relocation entry is not anded with a mask
to get the final relocation type.
---
Regards,
Jiong
Did someone encounter a similar error? Or can direct me to changes that I need
to do while migrating from JIT to MCJIT?
Thanks.
_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>
http://llvm.cs.uiuc.edu<http://llvm.cs.uiuc.edu/>http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130411/837760e9/attachment.html>
Submitted to bugzilla as PR 15729
From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On
Behalf Of Kaylor, Andrew
Sent: Thursday, April 11, 2013 1:13 PM
To: Weiss, Eran
Cc: llvmdev at cs.uiuc.edu
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
Thanks, Eran.
I’m not sure how soon I’ll have a solution for you, but it’s on my to-do list
now. I’ll also create a bugzilla record for this problem.
-Andy
From: Weiss, Eran [mailto:Eran.Weiss at emc.com]
Sent: Thursday, April 11, 2013 12:40 AM
To: Kaylor, Andrew
Cc: llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>; Jim Grosbach;
Jiong Wang
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
Andrew,
I've attached a small reproduction of the issue.
Reproduce by:
$ /usr/bin/g++ `llvm-config --cxxflags` -g -m32 -c mcjit_external_symbol.cpp
$ /usr/bin/g++ `llvm-config --ldflags` -g -m32 -o mcjit_external_symbol
mcjit_external_symbol.o `llvm-config --libs all`
$ ./mcjit_external_symbol
verifying...
LLVM ERROR: Program used external function '_external' which could not
be resolved!
Note that using JIT the code works fine, in case it might help.
Eran.
From: "Kaylor, Andrew" <andrew.kaylor at
intel.com<mailto:andrew.kaylor at intel.com>>
Date: Wed, 10 Apr 2013 20:50:14 -0400
To: Eran Weiss <eran.weiss at emc.com<mailto:eran.weiss at emc.com>>
Cc: "llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>"
<llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>, Jim
Grosbach <grosbach at apple.com<mailto:grosbach at apple.com>>,
Jiong Wang <jiwang at tilera.com<mailto:jiwang at tilera.com>>
Subject: RE: [LLVMdev] Migration from JIT to MCJIT
Eran,
Is there any chance you could boil this down to a small reproducer?
I’ve got a mid-to-long-term goal of getting rid of the ugliness in LLDB that Jim
mentioned, and fixing your problem would be a good first step.
Thanks,
Andy
From: Jim Grosbach [mailto:grosbach at apple.com]
Sent: Wednesday, April 10, 2013 4:19 PM
To: Kaylor, Andrew; Jiong Wang
Cc: Weiss, Eran; llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
Existing clients (LLDB) deal with externals by resolving them to constant
function pointers that are referenced in the IR. That’s obviously ugly as hell,
but it gets things done.
The old JIT was able to simplify things because it assumed the JITed code was
running in the same process as the JIT compiler. The MCJIT doesn’t assume that,
so it has to handle more possibilities. For example, that the invoked function
may be in a dylib which hasn’t yet been loaded into the target address space. I
suggest having a look at the ld64, lld and dyld implementations to see what
these relocation really imply. You won’t have to solve all of the potential
scenarios to get anything done, but you will want to make sure to carefully
check which use-case you’re in and error out when it’s one you haven’t handled
yet. The worst case is if the code makes overly broad assumptions about possible
inputs and resolves a relocation incorrectly in some subtle way.
-Jim
On Apr 10, 2013, at 11:49 AM, "Kaylor, Andrew" <andrew.kaylor at
intel.com<mailto:andrew.kaylor at intel.com>> wrote:
The MachO handling isn’t always straightforward in that it has a lot of bit
fields to handle while it’s also trying to accommodate the format abstractions
baked into the interfaces. The actual relocation type gets extracted in the
RuntimeDyldMachO::resolveRelocation function (RuntimeDyldMachO.cpp:32) before it
is passed on to the architecture-specific handlers. So the mask you mentioned
is probably OK.
The fact that this relocation type isn’t handled is, of course, the root of your
problem.
MCJIT has grown on an as-needed basis up to now, with only the relocation types
we were actually seeing being implemented. Unfortunately, I don’t know enough
about MachO to help with this one. Maybe Jim Grosbach can help?
-Andy
From: llvmdev-bounces at cs.uiuc.edu<mailto:llvmdev-bounces at
cs.uiuc.edu> [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Weiss, Eran
Sent: Tuesday, April 09, 2013 11:51 PM
To: Jiong Wang
Cc: llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
Thank you for the help.
The relocation type value is anded with 0xffffffffL. (RuntimeDyldMachO.cpp:214)
Maybe this mask should be different?
Anyway, it seems like this relocation isn't implemented.
(RuntimeDyldMachO.cpp:104)
From: Jiong Wang <jiwang at tilera.com<mailto:jiwang at tilera.com>>
Date: Tue, 9 Apr 2013 09:42:03 -0400
To: Eran Weiss <eran.weiss at emc.com<mailto:eran.weiss at emc.com>>
Cc: "llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>"
<llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>
Subject: Re: [LLVMdev] Migration from JIT to MCJIT
于 2013/4/9 21:08, Weiss, Eran 写道:
Hi,
I'm migrating my code (running on mac) from using JIT to MCJIT. My code
generates in memory, mostly using the llvm-c api, and then runs the generated
code.
When I try to use MCJIT I encounter a problem with relocations of external
symbols – functions compiled statically beforehand with gcc.
I get the following error:
Invalid relocation type!
UNREACHABLE executed at
/Users/weisse4/dev/llvm/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldMachO.cpp:89
While debugging, I see that the relocation type read in is 218103811, which
seems corrupt to me.
Hi Weiss,
I do not have any experience on Mach binary format, but the hex value of
218103811 is 0xd000003 (maybe the relocation type is
RIT_Generic_PreboundLazyPointer = 3), something looks like a relocation entry
composed of "symbol index" + "relocation type".
maybe something is wrong, that the relocation entry is not anded with a mask
to get the final relocation type.
---
Regards,
Jiong
Did someone encounter a similar error? Or can direct me to changes that I need
to do while migrating from JIT to MCJIT?
Thanks.
_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>
http://llvm.cs.uiuc.edu<http://llvm.cs.uiuc.edu/>http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130411/5762752c/attachment.html>