llvm dev - Nov 2017 - Elves, Orcs, and the Memory Growth

Hi folks,

I have an issue with constant memory growth on Linux.
Here is what I do (real code is here[1]):

for (auto &module : modules) {
  auto owningObject = orcCompiler.compileModule(module);
  processObjectFile(owningObject.getBinary());
}

After each compilation of a module memory grows by N megabytes, which is fine
and expected.
The problem that the owningObject does not escape the scope, and I expect memory
to be freed after each loop iteration.
It works as expected on macOS: memory spikes when we enter the loop body but
gets back after we leave the body.
However, on Linux, it grows constantly and eventually exceeds all capacities.

The questions are: what can I do about it? What could hold the memory allocated
for each object file? Why is it different between macOS and Linux? Who is the
guilty here: the Elf or the Orc?

For your info: I use Orc APIs from LLVM 3.9.
Please let me know if I can provide more relevant information.

I would appreciate any hints and ideas on how to nail the problem down.

[1]
https://github.com/mull-project/mull/blob/disable_inmemory_cache/lib/Driver.cpp#L231

Cheers,
Alex.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 529 bytes
Desc: Message signed with OpenPGP
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171115/fb003cba/attachment.sig>

(+Lang, Orcmaster)

Any chance you could provide reproduction steps? ("download this code, run
this build command, run this test command", etc?) Preferably with a
smaller/isolated example, but that's not strictly needed, so long as
there's not lots of external dependencies to worry about, etc.

- Dave

On Wed, Nov 15, 2017 at 1:09 PM Alex Denisov via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi folks,
>
> I have an issue with constant memory growth on Linux.
> Here is what I do (real code is here[1]):
>
> for (auto &module : modules) {
>   auto owningObject = orcCompiler.compileModule(module);
>   processObjectFile(owningObject.getBinary());
> }
>
> After each compilation of a module memory grows by N megabytes, which is
> fine and expected.
> The problem that the owningObject does not escape the scope, and I expect
> memory to be freed after each loop iteration.
> It works as expected on macOS: memory spikes when we enter the loop body
> but gets back after we leave the body.
> However, on Linux, it grows constantly and eventually exceeds all
> capacities.
>
> The questions are: what can I do about it? What could hold the memory
> allocated for each object file? Why is it different between macOS and
> Linux? Who is the guilty here: the Elf or the Orc?
>
> For your info: I use Orc APIs from LLVM 3.9.
> Please let me know if I can provide more relevant information.
>
> I would appreciate any hints and ideas on how to nail the problem down.
>
> [1]
>
https://github.com/mull-project/mull/blob/disable_inmemory_cache/lib/Driver.cpp#L231
>
> Cheers,
> Alex.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20171116/0edc9286/attachment.html>