llvm dev - Feb 2012 - [LLVMdev] Strange behaviour with x86-64 windows, bad call instruction address

Hi all,

Some background: I'm working on a project to replace a custom VM with
various components of llvm.  We have everything running just peachy keen with
one recent exception, one of our executables crashes when attempting run a
JIT'd function.  We have llvm building and running on 64 bit Windows and
Linux, using Visual Studio 2008 on Windows and gcc on Linux, and we have the
llvm static libs linked into one of a DLLs, which is then linked to several
different EXE's.  The DLL contains the code to compile to llvm IR, JIT and
run code written in our proprietary language.  Each EXE calls into this DLL the
same way.  The same chunk of IR, when JIT'd in 3 of the EXE's runs
perfectly, but in the last program, it dies in a call instruction out into an
invalid memory location.  All compiler and linker options are the same for all 4
exe's.  The one difference I've seen when debugging the assembly is that
the 3 that work all have JIT function pointer addresses less than a 32 bit value
but the one that is failing has a 64 bit address, as indicated in the snippet
below:

000007FFFFC511D7  pop         rbp
000007FFFFC511D8  ret
000007FFFFC511D9  sub         rsp,20h
000007FFFFC511DD  mov         rcx,qword ptr [rbp-70h]
000007FFFFC511E1  mov         edx,0FFFFFFFEh
000007FFFFC511E6  xor         r8d,r8d
000007FFFFC511E9  call        rsi
000007FFFFC511EB  add         rsp,20h
000007FFFFC511EF  test        al,1
000007FFFFC511F2  je          000007FFFFC511C3
000007FFFFC511F8  sub         rsp,20h
000007FFFFC511FC  mov         rax,7FFFFC30030h
000007FFFFC51206  mov         rcx,rdi
000007FFFFC51209  mov         edx,0FFFFFFFEh
000007FFFFC5120E  xor         r8d,r8d
000007FFFFC51211  call        rax
000007FFFFC51213  add         rsp,20h
000007FFFFC51217  test        al,1
000007FFFFC5121A  je          000007FFFFC511C3
000007FFFFC51220  mov         qword ptr [rbp-68h],rdi
000007FFFFC51224  mov         eax,10h
000007FFFFC51229  call        0000080077B3F1D0
000007FFFFC5122E  sub         rsp,rax
000007FFFFC51231  mov         rdx,rsp
000007FFFFC51234  mov         qword ptr [rbp-0F0h],rdx
000007FFFFC5123B  sub         rsp,20h

The call instruction at 000007FFFFC51229  is the one that jumps into invalid
memory at 80077B3F1D0.  I'm not sure why this particular EXE causes llvm to
use such large address values, but it looks like there might be some 32 bit vs
64 bit address calculation/offset problem when emitting the assembly.

The code that works looks like this:

0000000002931211  call        rax
0000000002931213  add         rsp,20h
0000000002931217  test        al,1
000000000293121A  je          00000000029311C3
0000000002931220  mov         qword ptr [rbp-68h],rdi
0000000002931224  mov         eax,10h
0000000002931229  call        0000000077B3F1D0
000000000293122E  sub         rsp,rax
0000000002931231  mov         rdx,rsp
0000000002931234  mov         qword ptr [rbp-0F0h],rdx
000000000293123B  sub         rsp,20h
000000000293123F  mov         r12,180071AD0h
0000000002931249  mov         ecx,0FFFFFFEEh
000000000293124E  xor         r8d,r8d
0000000002931251  mov         r9,29C02EAh
000000000293125B  call        r12
000000000293125E  add         rsp,20h
0000000002931262  mov         eax,10h
0000000002931267  call        0000000077B3F1D0
000000000293126C  sub         rsp,rax
000000000293126F  mov         rdx,rsp
0000000002931272  mov         qword ptr [rbp-58h],rdx
0000000002931276  sub         rsp,20h
000000000293127A  mov         ecx,0FFFFFFEEh
000000000293127F  xor         r8d,r8d
0000000002931282  mov         r9,29C02EAh
000000000293128C  call        r12
000000000293128F  add         rsp,20h
0000000002931293  mov         eax,10h
0000000002931298  call        0000000077B3F1D0
000000000293129D  sub         rsp,rax
00000000029312A0  mov         rax,rsp

And the code at 77B3F1D0 is this:

0000000077B3F1BE  nop
0000000077B3F1BF  nop
0000000077B3F1C0  int         3
0000000077B3F1C1  int         3
0000000077B3F1C2  int         3
0000000077B3F1C3  int         3
0000000077B3F1C4  int         3
0000000077B3F1C5  int         3
0000000077B3F1C6  nop         word ptr [rax+rax]
0000000077B3F1D0  sub         rsp,10h
0000000077B3F1D4  mov         qword ptr [rsp],r10
0000000077B3F1D8  mov         qword ptr [rsp+8],r11
0000000077B3F1DD  xor         r11,r11
0000000077B3F1E0  lea         r10,[rsp+18h]
0000000077B3F1E5  sub         r10,rax
0000000077B3F1E8  cmovb       r10,r11
0000000077B3F1EC  mov         r11,qword ptr gs:[10h]
0000000077B3F1F5  cmp         r10,r11
0000000077B3F1F8  jae         0000000077B3F210
0000000077B3F1FA  and         r10w,0F000h
0000000077B3F200  lea         r11,[r11-1000h]
0000000077B3F207  mov         byte ptr [r11],0
0000000077B3F20B  cmp         r10,r11
0000000077B3F20E  jne         0000000077B3F200
0000000077B3F210  mov         r10,qword ptr [rsp]
0000000077B3F214  mov         r11,qword ptr [rsp+8]
0000000077B3F219  add         rsp,10h
0000000077B3F21D  ret
0000000077B3F21E  nop
0000000077B3F21F  nop
0000000077B3F220  int         3
0000000077B3F221  int         3

  I searched the bug database for various topics but didn't see anything
specific other than one mention in bug 5201 to do with 32 bit address
truncating.  My dev system is a dual-core xeon with 16 gigs of ram.  I'm no
expert in how llvm works to output the asm, but I'm not afraid to delve into
it to see what's happening.  Has anyone else run into this?  Does anyone
have a suggestion of where I might start to debug in the X86 emitter code? 
I'm not even sure how to create a test case that would use a large starting
address for the JIT?  Any help is greatly appreciated.

Thanks in advance,

.r.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120214/06679ec8/attachment.html>