Hi,
I'm seeing rather high memory usage from LLVM and I'd like to track down
what I'm doing to cause it. My application is a simple web application
server that compiles web pages with embedded script to bitcode and compiles
them with the JIT on demand. I've taken tools/lli.cpp as a starting point
and extended it to load additional modules.
However, if I load successive pages and watch top, my RSS size increases by
roughly 40 times the on disk size of the loaded bit code for each loaded
module, which I take to mean I have a massive leak. I'm probably just not
freeing stuff that I ought to but I don't know what - deleting the
MemoryBuffer or ModuleProvider results in a SEGV. I've read the doxygen docs
but I'm not clear what objects have allocated what memory for what purpose,
what can be freed once all functions in a module have been compiled or how
to do this.
I'd be grateful if anyone could tell me what I'm doing wrong or point me
at
docs or examples covering this topic? The gist of my code is:
Module *load_module(char *bitcode_name) {
MemoryBuffer *buffer = MemoryBuffer::getFile(bitcode_name,
&error_message);
ModuleProvider *mp = getBitcodeModuleProvider(buffer, getGlobalContext(),
&error_message);
if( first_time ) {
InitializeNativeTarget();
builder = new EngineBuilder(mp);
builder->setEngineKind(EngineKind::JIT);
CodeGenOpt::Level opt_level = CodeGenOpt::Default;
builder->setOptLevel(opt_level);
execution_engine = builder->create();
delete(builder); // lli doesn't do this - is it safe?
}
Module *module = mp->materializeModule(&error_message);
for (Module::iterator I = module->begin(), E = module->end(); I != E;
++I)
{
Function *f = &*I;
if(!f->isDeclaration()) {
execution_engine->getPointerToFunction(f);
}
}
// all functions are compiled so I'd like to dispose of the bitcode any
memory used in the compilation - how can I do this?
execution_engine->runStaticConstructorsDestructors(false);
return module;
}
-- James Williams
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20100211/085a9f28/attachment.html>
On Thu, Feb 11, 2010 at 6:53 PM, James Williams <junk at giantblob.com> wrote:> Hi, > > I'm seeing rather high memory usage from LLVM and I'd like to track down > what I'm doing to cause it. My application is a simple web application > server that compiles web pages with embedded script to bitcode and compiles > them with the JIT on demand. I've taken tools/lli.cpp as a starting point > and extended it to load additional modules. > > However, if I load successive pages and watch top, my RSS size increases by > roughly 40 times the on disk size of the loaded bit code for each loaded > module, which I take to mean I have a massive leak. I'm probably just not > freeing stuff that I ought to but I don't know what - deleting the > MemoryBuffer or ModuleProvider results in a SEGV. I've read the doxygen docs > but I'm not clear what objects have allocated what memory for what purpose, > what can be freed once all functions in a module have been compiled or how > to do this. > > I'd be grateful if anyone could tell me what I'm doing wrong or point me at > docs or examples covering this topic? The gist of my code is: > > Module *load_module(char *bitcode_name) { > MemoryBuffer *buffer = MemoryBuffer::getFile(bitcode_name, > &error_message); > ModuleProvider *mp = getBitcodeModuleProvider(buffer, getGlobalContext(), > &error_message); > > if( first_time ) { > InitializeNativeTarget(); > builder = new EngineBuilder(mp); > builder->setEngineKind(EngineKind::JIT); > CodeGenOpt::Level opt_level = CodeGenOpt::Default; > builder->setOptLevel(opt_level); > execution_engine = builder->create(); > delete(builder); // lli doesn't do this - is it safe?lli allocates the builder on the stack, so it is automatically destroyed on scope exit.> } > > Module *module = mp->materializeModule(&error_message); > for (Module::iterator I = module->begin(), E = module->end(); I != E; ++I) > { > Function *f = &*I; > if(!f->isDeclaration()) { > execution_engine->getPointerToFunction(f); > } > } > // all functions are compiled so I'd like to dispose of the bitcode any > memory used in the compilation - how can I do this?You can call (I think) deleteBody on each compiled function if you're done with it to delete the IR and leave the machine code. You can't delete the functions themselves, because their pointers are used as keys in a hashtables. Reid
On 12 February 2010 01:48, Reid Kleckner <rnk at mit.edu> wrote:> On Thu, Feb 11, 2010 at 6:53 PM, James Williams <junk at giantblob.com> > wrote: > > Hi, > > > > I'm seeing rather high memory usage from LLVM and I'd like to track down > > what I'm doing to cause it. My application is a simple web application > > server that compiles web pages with embedded script to bitcode and > compiles > > them with the JIT on demand. I've taken tools/lli.cpp as a starting point > > and extended it to load additional modules. > > > > However, if I load successive pages and watch top, my RSS size increases > by > > roughly 40 times the on disk size of the loaded bit code for each loaded > > module, which I take to mean I have a massive leak. I'm probably just not > > freeing stuff that I ought to but I don't know what - deleting the > > MemoryBuffer or ModuleProvider results in a SEGV. I've read the doxygen > docs > > but I'm not clear what objects have allocated what memory for what > purpose, > > what can be freed once all functions in a module have been compiled or > how > > to do this. > > > > I'd be grateful if anyone could tell me what I'm doing wrong or point me > at > > docs or examples covering this topic? The gist of my code is: > > > > Module *load_module(char *bitcode_name) { > > MemoryBuffer *buffer = MemoryBuffer::getFile(bitcode_name, > > &error_message); > > ModuleProvider *mp = getBitcodeModuleProvider(buffer, > getGlobalContext(), > > &error_message); > > > > if( first_time ) { > > InitializeNativeTarget(); > > builder = new EngineBuilder(mp); > > builder->setEngineKind(EngineKind::JIT); > > CodeGenOpt::Level opt_level = CodeGenOpt::Default; > > builder->setOptLevel(opt_level); > > execution_engine = builder->create(); > > delete(builder); // lli doesn't do this - is it safe? > > lli allocates the builder on the stack, so it is automatically > destroyed on scope exit. >OK. My C++ is very rusty!> > > } > > > > Module *module = mp->materializeModule(&error_message); > > for (Module::iterator I = module->begin(), E = module->end(); I != E; > ++I) > > { > > Function *f = &*I; > > if(!f->isDeclaration()) { > > execution_engine->getPointerToFunction(f); > > } > > } > > // all functions are compiled so I'd like to dispose of the bitcode any > > memory used in the compilation - how can I do this? > > You can call (I think) deleteBody on each compiled function if you're > done with it to delete the IR and leave the machine code. You can't > delete the functions themselves, because their pointers are used as > keys in a hashtables. >Thanks, I'll give this a try -- James> > Reid > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20100212/48730180/attachment.html>