llvm dev - Nov 2012 - [LLVMdev] linking individual functions in execution module

Hi,


JIT does not allow functions to call others in different modules, so
all modules need to be statically linked in a big fat module. If a
module needs to be recompiled, all the others need to as well as
relinked.

there are two ways i intend to approach this problem:

1) forget about JITing, build each module into a .bc, call gcc to
generate .soname libraries, dynamically load with dlopen() those
libraries

2) keep the modules unlinked, and when execution is required, link the
required dependencies into an execution module. When one of the
dependencies is changed, most of the modules do not need to be rebuilt
(althought i probably will need to clone the module, since the Linker
seems to be destructive of the source), they just need to be relinked
in the execution module


first approach is my fallback approach, it is obviously rock-solid,
but i don't know how much overhead this actually represents, specially
if an alternative exists

i have questions regarding 2nd approach. each of the modules will have
the functions with generated IR, and among those generated IR will be
generated CallInst and InvokeInst to functions in the other modules.
The questions would be:

A) if i use the Linker class to link two or more modules, do i need to
update the CallInst/InvokeInst or it is part of its job to take care
of those?

B) Can i safely clone the modules before linking them into the
execution module? it seems the Linker is destructive on its sources,
but i don't know how much cycles would cloning the module save, as i'm
not clear if the module generation is more expensive than the linkage
itself


any high-level recommendations and suggestions are welcome

Hi Charles,

Are you using the old JIT engine or the newer MCJIT?

In either case, external functions are resolved through the
getPointerToNamedFunction method in the JIT Memory Manager.  If you provide your
own memory manager implementation, you should be able to link multiple modules
together.

If you re-JIT a module that you have previously linked to that will obviously
cause some problems, but you can probably work around that with a stub function.

-Andy

-----Original Message-----
From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On
Behalf Of charles quarra
Sent: Wednesday, November 21, 2012 12:43 PM
To: llvmdev at cs.uiuc.edu
Subject: [LLVMdev] linking individual functions in execution module

Hi,


JIT does not allow functions to call others in different modules, so all modules
need to be statically linked in a big fat module. If a module needs to be
recompiled, all the others need to as well as relinked.

there are two ways i intend to approach this problem:

1) forget about JITing, build each module into a .bc, call gcc to generate
.soname libraries, dynamically load with dlopen() those libraries

2) keep the modules unlinked, and when execution is required, link the required
dependencies into an execution module. When one of the dependencies is changed,
most of the modules do not need to be rebuilt (althought i probably will need to
clone the module, since the Linker seems to be destructive of the source), they
just need to be relinked in the execution module


first approach is my fallback approach, it is obviously rock-solid, but i
don't know how much overhead this actually represents, specially if an
alternative exists

i have questions regarding 2nd approach. each of the modules will have the
functions with generated IR, and among those generated IR will be generated
CallInst and InvokeInst to functions in the other modules.
The questions would be:

A) if i use the Linker class to link two or more modules, do i need to update
the CallInst/InvokeInst or it is part of its job to take care of those?

B) Can i safely clone the modules before linking them into the execution module?
it seems the Linker is destructive on its sources, but i don't know how much
cycles would cloning the module save, as i'm not clear if the module
generation is more expensive than the linkage itself


any high-level recommendations and suggestions are welcome
_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

2012/11/21 Kaylor, Andrew <andrew.kaylor at intel.com>:
> If you re-JIT a module that you have previously linked to that will
obviously cause some problems, but you can probably work around that with a stub
function.
could you elaborate a little bit on that ? i was thinking in something
different; linking a cloned copy of the module rather than the module
directly. Would that work better than this approach?


thanks