llvm dev - Jul 2019 - Eager compilation and relocatable dynamic linkable code-generation

Hi all,
I am following the LLVM JIT tutorial API based on:
https://llvm.org/docs/tutorial/BuildingAJIT1.html

I have 2 independent questions:
1. The tutorial mentions that "*will defer compilation of the module until
any of its definitions is looked up*".

Does that mean to force eager compilation, i have to keep track of all
functions added to the Module and do a lookup? I want the module to be
compiled fully after I do add module to compile_layer. What is the best way
to achieve this? Or should I not use ORC API at all?

2. I want to compile on one process and want to ship the machine code to
another process and load it for execution (with no lazy compilation where
an uncompiled function will trigger compilation to remote process which as
I understand is what the tutorial talks about). This means we need to
generate position-independent code and may need dynamic linking to resolve
any unknown function call based on symbol name. What is the best way to
achieve this? Is there some code pointers?

Thanks!

Rajesh S R
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190717/b5883e7f/attachment.html>

Hi Rajesh

On 1.: ORC defers compilation of the module and on first request,
compiles the entire module. This will trigger recursive lookups for
external symbols. Eager compilation means that the modules which provide
these symbols will now be compiled too, while lazy compilation should
instead insert stubs that will trigger compilation once they are reached
by execution.

On 2.: I am maybe not perfectly up-to-date, but what I would expect to
happen is this. First, your remote process allocates memory for the
executable code and sends the base address to the compiler process. Then
the compiler process compiles and links with this base address and sends
the (position-dependent) code back. You may also manage to go with
position-independent code, but then your remote needs a dynamic linker.
Not sure if there's an example for it.

Best
Stefan

On 18/07/2019 03:08, Rajesh S R via llvm-dev wrote:
> Hi all,
> I am following the LLVM JIT tutorial API based
> on: https://llvm.org/docs/tutorial/BuildingAJIT1.html
>
> I have 2 independent questions:
> 1. The tutorial mentions that "*will defer compilation of the module
> until any of its definitions is looked up*". 
>
> Does that mean to force eager compilation, i have to keep track of all
> functions added to the Module and do a lookup? I want the module to be
> compiled fully after I do add module to compile_layer. What is the
> best way to achieve this? Or should I not use ORC API at all?
>
> 2. I want to compile on one process and want to ship the machine code
> to another process and load it for execution (with no lazy compilation
> where an uncompiled function will trigger compilation to remote
> process which as I understand is what the tutorial talks about). This
> means we need to generate position-independent code and may need
> dynamic linking to resolve any unknown function call based on symbol
> name. What is the best way to achieve this? Is there some code pointers?
>
> Thanks!
>
> Rajesh S R
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-- 
https://flowcrypt.com/pub/stefan.graenitz at gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190718/fd8ac037/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190718/fd8ac037/attachment.sig>

Thanks Stefan!

On Thu, Jul 18, 2019 at 9:59 AM Stefan Gränitz <stefan.graenitz at
gmail.com>
wrote:
> Hi Rajesh
>
> On 1.: ORC defers compilation of the module and on first request, compiles
> the entire module. This will trigger recursive lookups for external
> symbols. Eager compilation means that the modules which provide these
> symbols will now be compiled too, while lazy compilation should instead
> insert stubs that will trigger compilation once they are reached by
> execution.
>
ha I see! So looking up any symbol in a module will trigger compilation of
a module. So one trick is to insert a dummy function that does nothing and
look it up, so that the entire module will be compiled?

Is there some API/mode to turn off lazy compilation entirely and just
compile each module on addModule call? My understanding is ORC is the new
API which supports both lazy and eager computation as the old APIs have
been made "legacy" APIs, so there must be some mode with ORC API to
default
to full eager compilation? Is that correct?

> On 2.: I am maybe not perfectly up-to-date, but what I would expect to
> happen is this. First, your remote process allocates memory for the
> executable code and sends the base address to the compiler process. Then
> the compiler process compiles and links with this base address and sends
> the (position-dependent) code back. You may also manage to go with
> position-independent code, but then your remote needs a dynamic linker. Not
> sure if there's an example for it.
>
hmm, interesting! Although for this work you need some lower bound on code
size generated. My case though needs position-independent code as i want to
just compile my module once and use it across multiple processes (same idea
as shared library) even across different machines.

Yeah a dynamic linker should be fine for this case. Just curious, the
tutorial uses this linker
<https://llvm.org/doxygen/classllvm_1_1orc_1_1RTDyldObjectLinkingLayer.html>.
Is this dynamic linking resolving any unknown symbol at first execution?
In this case the symbols in the generated code will fall into 2 category:
1. Symbols are defined in the host process. This is common case.
2. Symbols are defined in another module which is similarly compiled into
position-independent code. This is not likely and i don't see a use-case
currently, but may have some use-case in the future.

My understanding is that the symbol resolver does exactly that:

 Expected<JITEvaluatedSymbol> lookup(StringRef Name) {
    return ES.lookup({&ES.getMainJITDylib()}, Mangle(Name.str()));
  }


with symbols from hostporcess already exported to execution session.

In particular none of them needs to open a separate file and load the
symbol definition from it.
So, may be that "linker
<https://llvm.org/doxygen/classllvm_1_1orc_1_1RTDyldObjectLinkingLayer.html>"
is sufficient for my use-case?

So the main open question is: For compiling to position independent code,
wondering if there are some CompileLayer compiler which can be used to emit
position-independent code?

> Best
> Stefan
>
> On 18/07/2019 03:08, Rajesh S R via llvm-dev wrote:
>
> Hi all,
> I am following the LLVM JIT tutorial API based on:
> https://llvm.org/docs/tutorial/BuildingAJIT1.html
>
> I have 2 independent questions:
> 1. The tutorial mentions that "*will defer compilation of the module
> until any of its definitions is looked up*".
>
> Does that mean to force eager compilation, i have to keep track of all
> functions added to the Module and do a lookup? I want the module to be
> compiled fully after I do add module to compile_layer. What is the best way
> to achieve this? Or should I not use ORC API at all?
>
> 2. I want to compile on one process and want to ship the machine code to
> another process and load it for execution (with no lazy compilation where
> an uncompiled function will trigger compilation to remote process which as
> I understand is what the tutorial talks about). This means we need to
> generate position-independent code and may need dynamic linking to resolve
> any unknown function call based on symbol name. What is the best way to
> achieve this? Is there some code pointers?
>
> Thanks!
>
> Rajesh S R
>
> _______________________________________________
> LLVM Developers mailing listllvm-dev at
lists.llvm.orghttps://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> -- https://flowcrypt.com/pub/stefan.graenitz at gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190718/6854aca1/attachment.html>