llvm dev - Aug 2019 - [ORC] Removing / replacing JITDylibs

Hi,

I'm working on a runtime autotuner that utilizes ORCv2 JIT (I'm closely
tracking
tip-of-tree), so linking new object files and patching in the new function(s)
will happen frequently. 

One of the concerns my runtime system has is the ability to do one of the
following: (1) replacement of the contents of a JITDylib with a new object file
[to provide semi-space GC-style reclaiming], (2) the outright removal of a
JITDylib.

Right now, I have one ExecutionSession instance for my linker and am creating a
new JITDylib for each object file that I'd like to link in. There
doesn't appear
to be a corresponding ExecutionSession::removeJITDylib(..) method, so I'm
wondering: how do I reclaim the memory for code that I've linked in
previously
but no longer need?

When using MCJIT, I would reclaim this memory by destroying the ExecutionEngine
that was created for each "JITDylib". Should I do the same with
ExecutionSessions?

For reference, here's the short bit of code I'm playing with for linking
with
ORC: 

https://github.com/halo-project/llvm/blob/master/compiler-rt/lib/halomon/DynamicLinker.h#L55

Thanks,
Kavon
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190819/ae346237/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190819/ae346237/attachment.sig>

Hi Kavon,

Unfortunately we don't have a good way to do this at the moment, short of
maintaining multiple execution sessions (analogous to the way you
maintained multiple ExecutionEngines).

I am working on initializer/destructor support that will allow us to
perform the equivalent of dlopen/dlclose on JITDylibs. Once that support is
available I think it would be a good fit for your use case. Unfortunately I
think it is still a few months away.

Cheers,
Lang.

On Mon, Aug 19, 2019 at 3:38 PM Kavon Farvardin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi,
>
> I'm working on a runtime autotuner that utilizes ORCv2 JIT (I'm
closely
> tracking tip-of-tree), so linking new object files and patching in the new
> function(s) will happen frequently.
>
> One of the concerns my runtime system has is the ability to do one of the
> following: (1) replacement of the contents of a JITDylib with a new object
> file [to provide semi-space GC-style reclaiming], (2) the outright removal
> of a JITDylib.
>
> Right now, I have one ExecutionSession instance for my linker and am
> creating a new JITDylib for each object file that I'd like to link in.
> There doesn't appear to be a
> corresponding ExecutionSession::removeJITDylib(..) method, so I'm
> wondering: how do I reclaim the memory for code that I've linked in
> previously but no longer need?
>
> When using MCJIT, I would reclaim this memory by destroying the
> ExecutionEngine that was created for each "JITDylib". Should I do
the same
> with ExecutionSessions?
>
> For reference, here's the short bit of code I'm playing with for
linking
> with ORC:
>
>
https://github.com/halo-project/llvm/blob/master/compiler-rt/lib/halomon/DynamicLinker.h#L55
>
> Thanks,
> Kavon
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190822/e57a5981/attachment.html>

Thanks for the reply, Lang! I really appreciate the effort you've been
putting
into the JIT infrastructure and that you take the time to answer lots of
questions about it on the mailing list.
> I am working on initializer/destructor support that will allow us to
perform
> the equivalent of dlopen/dlclose on JITDylibs.
In my case, the process is already running with one version of the program code,
and another thread is hot-patching functions with dynamically recompiled
versions (I've extended XRay for this). Thus, I actually want to avoid
reinitializing or creating new versions of globals for the Dylib output by ORC
and instead link with the ones in-process.
My plan for achieving this is to externalize all internal globals in the
original bitcode before JIT compilation and hope that the ORC dynamic linker
handles the rest automatically.
Cheers,Kavon
On Thu, 2019-08-22 at 14:23 -0700, Lang Hames wrote:
> Hi Kavon,
> Unfortunately we don't have a good way to do this at the moment, short
of
> maintaining multiple execution sessions (analogous to the way you
maintained
> multiple ExecutionEngines).
> 
> I am working on initializer/destructor support that will allow us to
perform
> the equivalent of dlopen/dlclose on JITDylibs. Once that support is
available
> I think it would be a good fit for your use case. Unfortunately I think it
is
> still a few months away.
> 
> Cheers,
> Lang.
> On Mon, Aug 19, 2019 at 3:38 PM Kavon Farvardin via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> > Hi,
> > 
> > I'm working on a runtime autotuner that utilizes ORCv2 JIT
(I'm closely
> > tracking tip-of-tree), so linking new object files and patching in the
new
> > function(s) will happen frequently. 
> > 
> > One of the concerns my runtime system has is the ability to do one of
the
> > following: (1) replacement of the contents of a JITDylib with a new
object
> > file [to provide semi-space GC-style reclaiming], (2) the outright
removal
> > of a JITDylib.
> > 
> > Right now, I have one ExecutionSession instance for my linker and am
> > creating a new JITDylib for each object file that I'd like to link
in. There
> > doesn't appear to be a corresponding
ExecutionSession::removeJITDylib(..)
> > method, so I'm wondering: how do I reclaim the memory for code
that I've
> > linked in previously but no longer need?
> > 
> > When using MCJIT, I would reclaim this memory by destroying the
> > ExecutionEngine that was created for each "JITDylib". Should
I do the same
> > with ExecutionSessions?
> > 
> > For reference, here's the short bit of code I'm playing with
for linking
> > with ORC: 
> > 
https://github.com/halo-project/llvm/blob/master/compiler-rt/lib/halomon/DynamicLinker.h#L55
> > 
> > Thanks,
> > Kavon
> > _______________________________________________
> > 
> > LLVM Developers mailing list
> > 
> > llvm-dev at lists.llvm.org
> > 
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190823/0336c40c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190823/0336c40c/attachment-0001.sig>

Hi,

On 2019-08-22 14:23:48 -0700, Lang Hames via llvm-dev
wrote:
> Unfortunately we don't have a good way to do this at the moment, short
of
> maintaining multiple execution sessions (analogous to the way you
> maintained multiple ExecutionEngines).
FWIW, I/we use Orc v1, via the C stack, for compiling parts of SQL
queries. We currently rely quite heavily on being able to deallocate
"modules". I asume it'd be a problem for us to migrate to v2 due
to that
right now (ignoring the fact that there's no C API right now ...).

I have not yet benchmarked it, but it looks like it'd be fairly
noticable to create a separate ExecutionSession for every set of
functions I'd want to be able to deallocate inidividually.  And it's not
straightforward to just use ExecutionSessions in a "generational"
manner, as the lifetime of JITed expressions can vary widely (from
seconds to many days in the case of a longrunning query, which might
spawn many shorter queries internally).


Is that prototype available somewhere?


I'd be interested in replacing the current C stack with LLJIT (wrapping
it in C again, potentially even with a roughly compatible interface),
but especially if one desired to take advantage of the nicer features,
say parallel compilation, it looks infeasible to just create separate
LLJIT instances over and over.

> I am working on initializer/destructor support that will allow us to
> perform the equivalent of dlopen/dlclose on JITDylibs. Once that support is
> available I think it would be a good fit for your use case. Unfortunately I
> think it is still a few months away.
On 2019-09-02 12:26:48 -0700, Lang Hames via llvm-dev
wrote:
> Actually, if my current prototype pans out then there might be an even
> better solution for your use case: I'm hoping to provide fine-grained
> removal of modules from within a JITDylib (without removing the whole
> JITDylib). The advantage of this is that it's easier to reason about
> (functions can go in the conceptually "correct" JITDylib, even if
you want
> to remove them later) and less expensive (It's more expensive to
maintain
> one JITDylib per function than to maintain one JITDylib with many
> functions).
Interesting.  Are you thinking of providing both, wholesale JITDylib and
excision of individual functions? In postgres' case the modules we add
contain a bunch of exposed function, and then (depending on the cost of
the query) a lot of C functions and required globals (copied in the case
of static constant ones) that have been "inlined" from the main
binary,
to allow for proper IPO.  It seems like it might be hard to support that
case efficiently without replacing JITDylibs wholesale?


The new APIs look much cleaner to me. Thanks for all your work on that!


Greetings,

Andres Freund