Hello everyone, I encountered a strange behaviour which I can't explain. I'm developing an application under Windows 7 64bit. This application is using the LLVM library to load and resolve BC files generated with clang. Then the program picks some functions from the module and executes them. That's the theory - but now it's getting strange. The BC files are compiled for 64bit windows but have the ELF abi (the application has PECOFF). When I now pick a normal function and call them, everything is fine. But then I decided to call a function which is virtual - the program and the ELF file are both sharing the same interface for this with pure virtual functions. When I now call a virtual function, then I will reach the correct function - but the return value will be faulty, which then corruptes the stack and it crashes. But the this pointer - when the function was called - is correct. What's more stranger: I casted the address of the virtual function "down" to a normal function, but with the same return value. When I call this construct, then the return value is correct! So wow! Isn't it possible to call virtual ELF functions from a "PECOFF code"? Kind regards fenir50 (Sorry for hiding my name)
David Chisnall via llvm-dev
2018-Feb-08 09:17 UTC
[llvm-dev] Calling virtual "ELF" functions - BC code
On 5 Feb 2018, at 21:19, via llvm-dev <llvm-dev at lists.llvm.org> wrote:> > Hello everyone, > > I encountered a strange behaviour which I can't explain. > > I'm developing an application under Windows 7 64bit. This application is using the LLVM library to load and resolve BC files generated with clang. Then the program picks some functions from the module and executes them. That's the theory - but now it's getting strange. > > The BC files are compiled for 64bit windows but have the ELF abi (the application has PECOFF). When I now pick a normal function and call them, everything is fine. But then I decided to call a function which is virtual - the program and the ELF file are both sharing the same interface for this with pure virtual functions. When I now call a virtual function, then I will reach the correct function - but the return value will be faulty, which then corruptes the stack and it crashes. But the this pointer - when the function was called - is correct. > What's more stranger: I casted the address of the virtual function "down" to a normal function, but with the same return value. When I call this construct, then the return value is correct!This doesn’t sound like an ELF issue. On Windows, virtual functions use the ‘thiscall’ calling convention, where the *callee* cleans up the stack. In the standard Windows x86 calling convention, ecx is the first integer or pointer argument. In the thiscall calling convention, it is the `this` pointer, so if you are creating a function with the normal calling convention and calling it with the thiscall convention then you will expect to see something like this: the argument is in the right place, but the caller expects the callee to clean up the stack and the callee expects the caller to do it. In your IR, are the functions that you intend to use as virtual functions declared to use the X86_ThisCall calling convention? David
<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div> <div>Hello David,</div> <div> </div> <div>I'm not a member of the LLVM-list so I send you the message direct (and to the list). I'm sorry when this is not the codex of the list.</div> <div> </div> <div>The IR code is generated via Clang-Cl with the --target=x86_64-pc-windows-elf flag set. The functions are declared as virtual member functions, so they must have the __thiscall calling convention - I hope.</div> <div>Also I said that I casted the functions down to normal functions and tried calling them. In the meanwhile I casted them to member functions and called them with a correct this pointer. This worked too!</div> <div> </div> <div>Kind regards</div> <div>fenir50 (and still sorry)</div> <div> <div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"> <div style="margin:0 0 10px 0;"><b>Gesendet:</b> Donnerstag, 08. Februar 2018 um 10:17 Uhr<br/> <b>Von:</b> "David Chisnall" <David.Chisnall@cl.cam.ac.uk><br/> <b>An:</b> Fenir50@web.de<br/> <b>Cc:</b> llvm-dev@lists.llvm.org<br/> <b>Betreff:</b> Re: [llvm-dev] Calling virtual "ELF" functions - BC code</div> <div name="quoted-content">On 5 Feb 2018, at 21:19, via llvm-dev <llvm-dev@lists.llvm.org> wrote:<br/>><br/> > Hello everyone,<br/> ><br/> > I encountered a strange behaviour which I can't explain.<br/> ><br/> > I'm developing an application under Windows 7 64bit. This application is using the LLVM library to load and resolve BC files generated with clang. Then the program picks some functions from the module and executes them. That's the theory - but now it's getting strange.<br/> ><br/> > The BC files are compiled for 64bit windows but have the ELF abi (the application has PECOFF). When I now pick a normal function and call them, everything is fine. But then I decided to call a function which is virtual - the program and the ELF file are both sharing the same interface for this with pure virtual functions. When I now call a virtual function, then I will reach the correct function - but the return value will be faulty, which then corruptes the stack and it crashes. But the this pointer - when the function was called - is correct.<br/> > What's more stranger: I casted the address of the virtual function "down" to a normal function, but with the same return value. When I call this construct, then the return value is correct!<br/><br/> This doesn’t sound like an ELF issue. On Windows, virtual functions use the ‘thiscall’ calling convention, where the *callee* cleans up the stack. In the standard Windows x86 calling convention, ecx is the first integer or pointer argument. In the thiscall calling convention, it is the `this` pointer, so if you are creating a function with the normal calling convention and calling it with the thiscall convention then you will expect to see something like this: the argument is in the right place, but the caller expects the callee to clean up the stack and the callee expects the caller to do it.<br/> <br/> In your IR, are the functions that you intend to use as virtual functions declared to use the X86_ThisCall calling convention?<br/> <br/> David<br/> </div> </div> </div> </div></div></body></html>