llvm dev - Oct 2020 - Memory mapping assumptions in RuntimeDyld

Hi!

Implementing the Memory::allocateMappedMemory() function on z/OS, I see a 
failure in the AArch64 COFF test case.
The test case has 3 sections. For each section, 
Memory::allocateMappedMemory() is called to reserve memory.
If the distance between the pointers gets too large, then the test case 
fails. It can be reliable produced with
a distance of 1MB between the pointers. An easy way to simulate this is to 
increase the number of memory bytes
requested by 1MB (e.g. adding NumBytes += (1<<20); at begin of the 
method). Then the check if a relocation is correctly
resolved fails.
My current understanding is that the kernel is more or less free to choose 
the address of the mapped memory, so this
situation can arise. Is this true, or do I miss something? Sorry if this 
is a trivial question!

Best regards,
Kai

Kai Nacke
IT Architect

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Sebastian Krause
Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert 
Janzen, Markus Koerner, Christian Noll, Nicole Reimer
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 14562 / WEEE-Reg.-Nr. DE 99369940

Hi Kai,

Is this true, or do I miss something? Sorry if this is a trivial question!


That's correct, and not at all. :)

RuntimeDyld does not support the small code model: It requires code to be
compiled with the large code model so that references can cover the entire
address space.

There is a newer JIT linker, JITLink, which removes this restriction. There
is a MachO implementation for AArch64, but no ELF or COFF implementation
yet. Once that is written we should be able to support small code model in
the JIT.

-- Lang.

On Fri, Oct 2, 2020 at 10:02 AM Kai Peter Nacke <kai.nacke at de.ibm.com>
wrote:
> Hi!
>
> Implementing the Memory::allocateMappedMemory() function on z/OS, I see a
> failure in the AArch64 COFF test case.
> The test case has 3 sections. For each section,
> Memory::allocateMappedMemory() is called to reserve memory.
> If the distance between the pointers gets too large, then the test case
> fails. It can be reliable produced with
> a distance of 1MB between the pointers. An easy way to simulate this is to
> increase the number of memory bytes
> requested by 1MB (e.g. adding NumBytes += (1<<20); at begin of the
> method). Then the check if a relocation is correctly
> resolved fails.
> My current understanding is that the kernel is more or less free to choose
> the address of the mapped memory, so this
> situation can arise. Is this true, or do I miss something? Sorry if this
> is a trivial question!
>
> Best regards,
> Kai
>
> Kai Nacke
> IT Architect
>
> IBM Deutschland GmbH
> Vorsitzender des Aufsichtsrats: Sebastian Krause
> Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert
> Janzen, Markus Koerner, Christian Noll, Nicole Reimer
> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
> HRB 14562 / WEEE-Reg.-Nr. DE 99369940
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20201002/dcc64625/attachment.html>

Thanks! That explains it.

Lang Hames <lhames at gmail.com> wrote on 02.10.2020 20:04:15:
> From: Lang Hames <lhames at gmail.com>
> To: Kai Peter Nacke <kai.nacke at de.ibm.com>
> Cc: LLVM Developers Mailing List <llvm-dev at lists.llvm.org>
> Date: 02.10.2020 20:04
> Subject: [EXTERNAL] Re: Memory mapping assumptions in RuntimeDyld
> 
> Hi Kai, Is this true, or do I miss something? Sorry if this is a 
> trivial question!...  
> 
> This Message Is From an External Sender
> 
> This message came from outside your organization.
> 
> Hi Kai,
> Is this true, or do I miss something? Sorry if this is a trivial 
question!
> 
> That's correct, and not at all. :)
> 
> RuntimeDyld does not support the small code model: It requires code 
> to be compiled with the large code model so that references can 
> cover the entire address space.
> 
> There is a newer JIT linker, JITLink, which removes this 
> restriction. There is a MachO implementation for AArch64, but no ELF
> or COFF implementation yet. Once that is written we should be able 
> to support small code model in the JIT.
> 
> -- Lang.
> 
> On Fri, Oct 2, 2020 at 10:02 AM Kai Peter Nacke <kai.nacke at
de.ibm.com>
wrote:
> Hi!
> 
> Implementing the Memory::allocateMappedMemory() function on z/OS, I see 
a 
> failure in the AArch64 COFF test case.
> The test case has 3 sections. For each section, 
> Memory::allocateMappedMemory() is called to reserve memory.
> If the distance between the pointers gets too large, then the test case 
> fails. It can be reliable produced with
> a distance of 1MB between the pointers. An easy way to simulate this is 
to 
> increase the number of memory bytes
> requested by 1MB (e.g. adding NumBytes += (1<<20); at begin of the 
> method). Then the check if a relocation is correctly
> resolved fails.
> My current understanding is that the kernel is more or less free to 
choose 
> the address of the mapped memory, so this
> situation can arise. Is this true, or do I miss something? Sorry if this 
> is a trivial question!
> 
> Best regards,
> Kai
> 
> Kai Nacke
> IT Architect
> 
> IBM Deutschland GmbH
> Vorsitzender des Aufsichtsrats: Sebastian Krause
> Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, 
Norbert 
> Janzen, Markus Koerner, Christian Noll, Nicole Reimer
> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht 
Stuttgart, 
> HRB 14562 / WEEE-Reg.-Nr. DE 99369940