llvm dev - Nov 2016 - [llvm] To link or not to link

Hi,

I have migrated an LLVM front-end from LLVM 3.5 to 3.8 and now to 3.9 and ORC,
and there is a concept which I could not transfer. Consider:

extern "C" { void somefunc() {} }
…
auto llvmfunc = llvm::Function::Create(type, llvmFunction::PrivateLinkage,
"bla", module));
executionengine.addGlobalMapping(llvmfunc, &somefunc);
// now I have llvmfunc to work with and don't need to consider anything else

The GlobalMappingLayer class seems to provide the functionality I want. I
oriented myself on the example from
https://github.com/AndySomogyi/cayman/blob/aaa809c/src/llvm_orc_initial.cpp#L1172
(just oriented, because at least on 3.9, I can not stack the GlobalMappingLayer
on the ObjectLinkingLayer. I have to put it onto the IRCompileLayer. One
question I'd have: How does that order matter?)

The problem I have with that conceptually is: The name seems to matter. The
abstraction I'd like is that I get an llvm::Function*, and that is mapped to
&somefunc, the name "bla" I gave it should be irrelevant. (And
that's what I'd like that to be like for all functions and function
calls and whatnot: The name I give them is irrelevant, only the
llvm::Function*/llvm::Value* matters.) Am I using the wrong API and there is a
way to do this without the names, without any symbol resolution at all? (I
don't want to use symbols from the current executable or anywhere else,
either.)

The problem I have with that implementationally: It just doesn't work. I get
a LLVM ERROR: Undefined temporary symbol from the ELFObjectWriter (and the error
is actually about that symbol I'm trying to map), none of the llvm
assertions fail. I compiled a minimal example of what I'm doing and attached
it. The curious part is that the lambda resolver does not even get invoked… Can
anybody tell me what I am doing wrong?

Thank you
Julius


PS: Related thread:
http://lists.llvm.org/pipermail/llvm-dev/2015-August/thread.html#89230

PPS:
The whole thing I'm building is supposed to be a script engine for a game.
As the scripts aren't supposed to have any access outside the game, I'd
like as little linking magic as possible.
It's still in early stages.
https://git.openclonk.org/jcaesar/experimental.git/blob/7be141ce667f4689428943592bae0a7568134213:/src/script/C4AulCompiler.cpp
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mex.cpp
Type: text/x-c
Size: 5836 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161107/13c46e75/attachment.bin>
-------------- next part --------------
project(mex)

cmake_minimum_required (VERSION 3.0.2)
set(CMAKE_CXX_FLAGS "-std=c++14")
set(CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}")

# Done as per http://llvm.org/docs/CMake.html#embedding-llvm-in-your-project (?)
find_package(LLVM 3.9 REQUIRED)
message(STATUS "Found LLVM ${LLVM_PACKAGE_VERSION}")
message(STATUS "Using LLVMConfig.cmake in: ${LLVM_DIR}")
include_directories(${LLVM_INCLUDE_DIRS})
add_definitions(${LLVM_DEFINITIONS})
llvm_map_components_to_libnames(LLVM_LIBRARIES_OC Analysis Core ExecutionEngine
InstCombine Object RuntimeDyld ScalarOpts Support native)
include_directories(SYSTEM ${LLVM_INCLUDE_DIRS})
link_directories(${LLVM_LIBRARY_DIRS})

add_executable(mex mex.cpp)
target_link_libraries(mex ${LLVM_LIBRARIES_OC} ${LLVM_LDFLAGS})

Hi Julius, I had similar questions a while ago. First of all: the
GlobalMappingLayer is used for explicit mappings from strings to
function pointers, so that your generated code can access these
functions. It doesn't do much more.

I didn't inspect your attached example, but I think you can look at one
of my examples to find out how I wire it up. Maybe that helps:
https://github.com/weliveindetail/JitFromScratch/commit/70277db033d75599eff24ccd802cdf7fdfbcde99
> The problem I have with that conceptually is: The name seems to
> matter. The abstraction I'd like is that I get an llvm::Function*, and
> that is mapped to &somefunc, the name "bla" I gave it should
be
> irrelevant.
Just query the name of your llvm::Function and pass it also to the
GlobalMappingLayer (if you want to use it).
> (And that's what I'd like that to be like for all functions and
> function calls and whatnot: The name I give them is irrelevant, only
> the llvm::Function*/llvm::Value* matters.) Am I using the wrong API
> and there is a way to do this without the names, without any symbol
> resolution at all? (I don't want to use symbols from the current
> executable or anywhere else, either.) 
I think the ExecutionEngine just did the query and pass on implicitly.
> The problem I have with that implementationally: It just doesn't work.
> I get a LLVM ERROR: Undefined temporary symbol from the
> ELFObjectWriter (and the error is actually about that symbol I'm
> trying to map)
Query your mapping during symbol resolution like this:
https://github.com/weliveindetail/JitFromScratch/commit/70277db033d75599eff24ccd802cdf7fdfbcde99#diff-db46172f086b2c8918df9eea96b799c2R165

If you want your generated code to have implicit access to functions in
your binary, then have a look at these two lines:
https://github.com/weliveindetail/JitFromScratch/commit/860bec7e7fae1a1f31ef794bf78d191c44355c4c#diff-db46172f086b2c8918df9eea96b799c2R22
https://github.com/weliveindetail/JitFromScratch/commit/860bec7e7fae1a1f31ef794bf78d191c44355c4c#diff-db46172f086b2c8918df9eea96b799c2R64

It registers your binary as dynamic library and resolves symbols from
within your process. In order to avoid C++ mangling trouble, the
functions that get used may be extern "C" qualified. If you use a
frontend like Clang to generate your code, take care about compatible
settings, e.g. debug vs. release and ABI-related compile flags like
exceptions and RTTI.

Cheers,
Stefan

Am 07.11.16 um 11:47 schrieb Julius Michaelis via
llvm-dev:
> Hi,
>
> I have migrated an LLVM front-end from LLVM 3.5 to 3.8 and now to 3.9
> and ORC, and there is a concept which I could not transfer. Consider:
>
> extern "C" { void somefunc() {} }
> …
> auto llvmfunc = llvm::Function::Create(type,
> llvmFunction::PrivateLinkage, "bla", module));
> executionengine.addGlobalMapping(llvmfunc, &somefunc);
> // now I have llvmfunc to work with and don't need to consider
> anything else
>
> The GlobalMappingLayer class seems to provide the functionality I
> want. I oriented myself on the example from
>
https://github.com/AndySomogyi/cayman/blob/aaa809c/src/llvm_orc_initial.cpp#L1172
> (just oriented, because at least on 3.9, I can not stack the
> GlobalMappingLayer on the ObjectLinkingLayer. I have to put it onto
> the IRCompileLayer. One question I'd have: How does that order matter?)
>
> The problem I have with that conceptually is: The name seems to
> matter. The abstraction I'd like is that I get an llvm::Function*, and
> that is mapped to &somefunc, the name "bla" I gave it should
be
> irrelevant. (And that's what I'd like that to be like for all
> functions and function calls and whatnot: The name I give them is
> irrelevant, only the llvm::Function*/llvm::Value* matters.) Am I using
> the wrong API and there is a way to do this without the names, without
> any symbol resolution at all? (I don't want to use symbols from the
> current executable or anywhere else, either.)
>
> The problem I have with that implementationally: It just doesn't work.
> I get a LLVM ERROR: Undefined temporary symbol from the
> ELFObjectWriter (and the error is actually about that symbol I'm
> trying to map), none of the llvm assertions fail. I compiled a minimal
> example of what I'm doing and attached it. The curious part is that
> the lambda resolver does not even get invoked… Can anybody tell me
> what I am doing wrong?
>
> Thank you
> Julius
>
>
> PS: Related thread:
> http://lists.llvm.org/pipermail/llvm-dev/2015-August/thread.html#89230
>
> PPS:
> The whole thing I'm building is supposed to be a script engine for a
> game. As the scripts aren't supposed to have any access outside the
> game, I'd like as little linking magic as possible.
> It's still in early stages.
>
https://git.openclonk.org/jcaesar/experimental.git/blob/7be141ce667f4689428943592bae0a7568134213:/src/script/C4AulCompiler.cpp
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-- 
https://about.me/stefan.graenitz

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161108/c965867a/attachment.html>

Hi Stefan,

Okay, thanks to having your code as a working example, I figured out 
how to fix my problem. The reason was simply that my symbol name began 
with a ".".
>I think the ExecutionEngine just did the query and pass on 
>implicitly.
If it did, it did something to get around this… some day, I'll look 
into what that was.  For now, I'll just use different names.
>If you want your generated code to have implicit access to functions in
>your binary, then have a look at these two lines:
I precisely do not want that to happen. Anything implicit stinks of 
magic I have no control over, especially across 5 compilers on three 
platforms.

Greetings,
Julius


PS: Your example doesn't compile with gcc-5.4 (because of the 
DataLayout DataLayout-declaration), and the assertion that the mangled 
symbol name starts with an underscore does not hold, at least not on 
my system.

On  8.11 18:48, Stefan Gränitz wrote:
>Hi Julius, I had similar questions a while ago. First of all: the
>GlobalMappingLayer is used for explicit mappings from strings to
>function pointers, so that your generated code can access these
>functions. It doesn't do much more.
>
>I didn't inspect your attached example, but I think you can look at one
>of my examples to find out how I wire it up. Maybe that helps:
>https://github.com/weliveindetail/JitFromScratch/commit/70277db033d75599eff24ccd802cdf7fdfbcde99
>
>> The problem I have with that conceptually is: The name seems to
>> matter. The abstraction I'd like is that I get an llvm::Function*,
and
>> that is mapped to &somefunc, the name "bla" I gave it
should be
>> irrelevant.
>Just query the name of your llvm::Function and pass it also to the
>GlobalMappingLayer (if you want to use it).
>
>> (And that's what I'd like that to be like for all functions and
>> function calls and whatnot: The name I give them is irrelevant, only
>> the llvm::Function*/llvm::Value* matters.) Am I using the wrong API
>> and there is a way to do this without the names, without any symbol
>> resolution at all? (I don't want to use symbols from the current
>> executable or anywhere else, either.)
>I think the ExecutionEngine just did the query and pass on implicitly.
>
>> The problem I have with that implementationally: It just doesn't
work.
>> I get a LLVM ERROR: Undefined temporary symbol from the
>> ELFObjectWriter (and the error is actually about that symbol I'm
>> trying to map)
>Query your mapping during symbol resolution like this:
>https://github.com/weliveindetail/JitFromScratch/commit/70277db033d75599eff24ccd802cdf7fdfbcde99#diff-db46172f086b2c8918df9eea96b799c2R165
>
>If you want your generated code to have implicit access to functions in
>your binary, then have a look at these two lines:
>https://github.com/weliveindetail/JitFromScratch/commit/860bec7e7fae1a1f31ef794bf78d191c44355c4c#diff-db46172f086b2c8918df9eea96b799c2R22
>https://github.com/weliveindetail/JitFromScratch/commit/860bec7e7fae1a1f31ef794bf78d191c44355c4c#diff-db46172f086b2c8918df9eea96b799c2R64
>
>It registers your binary as dynamic library and resolves symbols from
>within your process. In order to avoid C++ mangling trouble, the
>functions that get used may be extern "C" qualified. If you use a
>frontend like Clang to generate your code, take care about compatible
>settings, e.g. debug vs. release and ABI-related compile flags like
>exceptions and RTTI.
>
>Cheers,
>Stefan
>
>Am 07.11.16 um 11:47 schrieb Julius Michaelis via llvm-dev:
>> Hi,
>>
>> I have migrated an LLVM front-end from LLVM 3.5 to 3.8 and now to 3.9
>> and ORC, and there is a concept which I could not transfer. Consider:
>>
>> extern "C" { void somefunc() {} }
>> …
>> auto llvmfunc = llvm::Function::Create(type,
>> llvmFunction::PrivateLinkage, "bla", module));
>> executionengine.addGlobalMapping(llvmfunc, &somefunc);
>> // now I have llvmfunc to work with and don't need to consider
>> anything else
>>
>> The GlobalMappingLayer class seems to provide the functionality I
>> want. I oriented myself on the example from
>>
https://github.com/AndySomogyi/cayman/blob/aaa809c/src/llvm_orc_initial.cpp#L1172
>> (just oriented, because at least on 3.9, I can not stack the
>> GlobalMappingLayer on the ObjectLinkingLayer. I have to put it onto
>> the IRCompileLayer. One question I'd have: How does that order
matter?)
>>
>> The problem I have with that conceptually is: The name seems to
>> matter. The abstraction I'd like is that I get an llvm::Function*,
and
>> that is mapped to &somefunc, the name "bla" I gave it
should be
>> irrelevant. (And that's what I'd like that to be like for all
>> functions and function calls and whatnot: The name I give them is
>> irrelevant, only the llvm::Function*/llvm::Value* matters.) Am I using
>> the wrong API and there is a way to do this without the names, without
>> any symbol resolution at all? (I don't want to use symbols from the
>> current executable or anywhere else, either.)
>>
>> The problem I have with that implementationally: It just doesn't
work.
>> I get a LLVM ERROR: Undefined temporary symbol from the
>> ELFObjectWriter (and the error is actually about that symbol I'm
>> trying to map), none of the llvm assertions fail. I compiled a minimal
>> example of what I'm doing and attached it. The curious part is that
>> the lambda resolver does not even get invoked… Can anybody tell me
>> what I am doing wrong?
>>
>> Thank you
>> Julius
>>
>>
>> PS: Related thread:
>> http://lists.llvm.org/pipermail/llvm-dev/2015-August/thread.html#89230
>>
>> PPS:
>> The whole thing I'm building is supposed to be a script engine for
a
>> game. As the scripts aren't supposed to have any access outside the
>> game, I'd like as little linking magic as possible.
>> It's still in early stages.
>>
https://git.openclonk.org/jcaesar/experimental.git/blob/7be141ce667f4689428943592bae0a7568134213:/src/script/C4AulCompiler.cpp
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>-- 
>https://about.me/stefan.graenitz
>