llvm dev - Oct 2020 - OrcV1 removal

Hi,

On 2020-09-30 19:00:45 -0700, Lang Hames wrote:
> Ok -- I'll make getting the new lookup API working a priority then.
Cool.

> > I see a memory leak with OrcV2 that I didn't see with V1. I'm
not yet
> > sure where exactly they're coming from. Possible that I'm just
missing a
> > step somewhere. Or something around the removable code support
doesn't
> > yet fully work.
>
> I just checked and I've migrated ObjectLinkingLayer to support
> ResourceTracker, but not RTDyldObjectLinkingLayer yet. If you're
testing on
> Linux there's a good chance that's the source of your leak.
I'll get that
> updated tonight.
I indeed am testing on linux... Thanks.

Regards,

Andres

Hi Andres,
> I just checked and I've migrated ObjectLinkingLayer to support
> > ResourceTracker, but not RTDyldObjectLinkingLayer yet. If you're
testing
> on
> > Linux there's a good chance that's the source of your leak.
I'll get that
> > updated tonight.
> I indeed am testing on linux... Thanks.

I've taken a first shot at hooking RTDyldObjectLinkingLayer up to the
ResourceTracker API in 7436b2ab2428. Could you let me know whether that
fixes the leak you were seeing?

-- Lang.

On Wed, Sep 30, 2020 at 7:10 PM Andres Freund <andres at anarazel.de>
wrote:
> Hi,
>
> On 2020-09-30 19:00:45 -0700, Lang Hames wrote:
> > Ok -- I'll make getting the new lookup API working a priority
then.
>
> Cool.
>
>
> > > I see a memory leak with OrcV2 that I didn't see with V1.
I'm not yet
> > > sure where exactly they're coming from. Possible that I'm
just missing
> a
> > > step somewhere. Or something around the removable code support
doesn't
> > > yet fully work.
> >
> > I just checked and I've migrated ObjectLinkingLayer to support
> > ResourceTracker, but not RTDyldObjectLinkingLayer yet. If you're
testing
> on
> > Linux there's a good chance that's the source of your leak.
I'll get that
> > updated tonight.
>
> I indeed am testing on linux... Thanks.
>
> Regards,
>
> Andres
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200930/0cf00d44/attachment.html>

Hi,

On 2020-09-30 21:31:33 -0700, Lang Hames wrote:
> I've taken a first shot at hooking RTDyldObjectLinkingLayer up to the
> ResourceTracker API in 7436b2ab2428. Could you let me know whether that
> fixes the leak you were seeing?
It did improve the situation significantly, thanks!

There's still a smaller leak, unfortunately. The function comments for
modules say that:

/**
 * Create a ThreadSafeModule wrapper around the given LLVM module. This takes
 * ownership of the M argument which should not be disposed of or referenced
 * after this function returns.
 *
 * Ownership of the ThreadSafeModule is unique: If it is transferred to the JIT
 * (e.g. by LLVMOrcLLJITAddLLVMIRModule), in which case the client is no longer
 * responsible for it. If it is not transferred to the JIT then the client
 * should call LLVMOrcDisposeThreadSafeModule to dispose of it.
 */
LLVMOrcThreadSafeModuleRef
LLVMOrcCreateNewThreadSafeModule(LLVMModuleRef M,
                                 LLVMOrcThreadSafeContextRef TSCtx);

/**
 * Dispose of a ThreadSafeModule. This should only be called if ownership has
 * not been passed to LLJIT (e.g. because some error prevented the client from
 * adding this to the JIT).
 */
void LLVMOrcDisposeThreadSafeModule(LLVMOrcThreadSafeModuleRef TSM);

but when using -fsanitize-leaks I see:

Direct leak of 9192 byte(s) in 383 object(s) allocated from:
    #0 0x7fe249127670 in operator new(unsigned long)
(/lib/x86_64-linux-gnu/liblsan.so.0+0xf670)
    #1 0x7fe1bfc1cbc0 in LLVMOrcCreateNewThreadSafeModule
(/home/andres/build/llvm-project/master/debug/install/lib/libLLVMOrcJIT.so.12git+0x38bbc0)
    #2 0x7fe245f6ea6d in llvm_compile_module
/home/andres/src/postgresql/src/backend/jit/llvm/llvmjit.c:606
    #3 0x7fe245f6e0b0 in llvm_get_function
/home/andres/src/postgresql/src/backend/jit/llvm/llvmjit.c:275
    #4 0x7fe245f809f1 in ExecRunCompiledExpr
/home/andres/src/postgresql/src/backend/jit/llvm/llvmjit_expr.c:2410
    #5 0x5633fbb6c17f in ExecEvalExpr
/home/andres/src/postgresql/src/include/executor/executor.h:294

for modules which are passed to LLVMOrcLLJITAddLLVMIRModule:
		ts_module = LLVMOrcCreateNewThreadSafeModule(context->module,
llvm_ts_context);
		rt = LLVMOrcJITDylibCreateResourceTracker(jd);
		error = LLVMOrcLLJITAddLLVMIRModule(compile_orc, jd, ts_module, rt);

which seems to suggest that somehow there's a reference too many?

Greetings,

Andres Freund