Gaier, Bjoern via llvm-dev
2018-Dec-13 06:59 UTC
[llvm-dev] Crashes when adding VisualStduio generated object files to the JIT process
Hello everyone, I'm using the LLVM for a JIT-Client under Windows 64bit. I tried various stuff with it and noticed, that object files or libraries which are generated by the Visual Studio Compiler 2015 or 2017 will break the jitted code, when they are added to the process. I opened a bug for this a while ago (https://bugs.llvm.org/show_bug.cgi?id=39447) - sadly this bug will become a stopper for me... So I wanted to ask for advice: Does anyone know anything about this bug or a similar one? Does anyone have an idea about this? Is this bug maybe a duplication with another bug? I'm still a beginner, so I hope I did everything right >o< Kind greetings Björn Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. Robert Plank, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima. Junichi Tajika -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181213/b22b8e9f/attachment.html>
Stefan Gränitz via llvm-dev
2018-Dec-15 17:44 UTC
[llvm-dev] Crashes when adding VisualStduio generated object files to the JIT process
Hi Björn I think it would be helpful to provide some more of the assembly you are ending up with. In the example below: Which source code are you compiling? What does it try to achieve? What happens to eax, ebx, rdi after the xor? +Andy,Zachary: Maybe you can correct, verify or explain in more detail my analysis in the end of this mail. I have no internal knowledge whatsoever, I just happened to reverse engineer this some time ago. From what I found in the old thread and in your bug report, I can make a guess.> I show you a short snippet of the assembly output. > > mov eax, DWORD PTR ?myInt@@3HA ; myInt > lea rdi, OFFSET FLAT:__ImageBase > xor ebx, ebx > > Then these offset is used to jump to some labels like "$LL4 at execute <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>:".The $LL4 looks like a symbol name in a jump table. There are a few examples (in the context of an unrelated bug report), that show how such tables look like: https://gist.github.com/rygorous/6790191 Here is an explanation of LEA: https://stackoverflow.com/questions/1658294 Am 22.03.18 um 18:43 schrieb Reid Kleckner via llvm-dev:> I wouldn't be surprised if JITing COFF files on Windows doesn't work > so well, since the object file format assumes most symbols are > dllimport or within the local 2GB module address range.Reid gave an important hint here already: dllimport symbols. MSVC apparently still assumes a small code model and uses 32 bit relocations. (Maybe you could change that nowadays with a compiler flag or so?) Have a look at this thread from 2015: http://lists.llvm.org/pipermail/llvm-dev/2015-November/092727.html Quoting Andy Ayers:> If at link time it turns out your export is from a DLL the linker will insert a jump stub / dllimport into the image for you which can handle larger distances.I guess this is the missing piece in your case. You have to do this patching step yourself, that would otherwise happen magically in the Microsoft linker (resolve relocation to a load-time jump stub) and loader (create/fill jump stub with the actual DLL symbol address), because the MSVC compiler assumes it when creating your obj file (creates the relocation for a base address, adds the offset and calls it). If this is the case, you should encounter __imp_foo symbol relocations. I don't remember the exact details, but this is what you want to watch out for. What needs to happen, conceptually, is something like this: __ImageBase is the address of a table that you create and OFFSET FLAT is a relative offset to the entry that you need to prepare. The entry must NOT store the target addresses (to be loaded and called), BUT a jump stub/trampoline like "jmp 0x00000000" (which forwards execution to the actual call target). When you resolve the relocation for the import symbol (__imp_foo) to the jump stub, you probably don't know the jmp target right away. So, you can leave it nulled and emit a new relocation to fill out the address (symbol foo). Depending on how your stub works, it may be a relative offset or a 64-bit absolute address. In the end your code calls the jump stub address as if it was the actual target function. IIRC you cannot simply resolve the original relocation to the target, even if you know it already, because the code emitted by MSVC has the base+offset calculation already built in. There are some pretty useful tools for all that in LLVM Orc: https://github.com/llvm-mirror/llvm/blob/master/lib/ExecutionEngine/Orc/IndirectionUtils.cpp Cheers Stefan Am 13.12.18 um 07:59 schrieb Gaier, Bjoern via llvm-dev:> > Hello everyone, > > > > I’m using the LLVM for a JIT-Client under Windows 64bit. I tried > various stuff with it and noticed, that object files or libraries > which are generated by the Visual Studio Compiler 2015 or 2017 will > break the jitted code, when they are added to the process. > > I opened a bug for this a while ago > (https://bugs.llvm.org/show_bug.cgi?id=39447) – sadly this bug will > become a stopper for me… So I wanted to ask for advice: > > > > Does anyone know anything about this bug or a similar one? Does anyone > have an idea about this? Is this bug maybe a duplication with another bug? > > > > I’m still a beginner, so I hope I did everything right >o< > > > > Kind greetings > > Björn > > Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, > USt.ID-Nr. DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. > Robert Plank, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi > Fukushima. Junichi Tajika > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- https://weliveindetail.github.io/blog/ https://cryptup.org/pub/stefan.graenitz at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181215/7f6af3fe/attachment.html>
Gaier, Bjoern via llvm-dev
2018-Dec-18 06:38 UTC
[llvm-dev] Crashes when adding VisualStduio generated object files to the JIT process
Hey Stefan, I provided the .cpp file, the .obj file and the .cod file in the attachments at the bug report (https://bugs.llvm.org/show_bug.cgi?id=39447) The .cod file is what you are looking for. I remember talking with you about that bug :'D But I'm just a beginner >o< Kind greetings Björn From: Stefan Gränitz <stefan.graenitz at gmail.com> Sent: Samstag, 15. Dezember 2018 18:45 To: Gaier, Bjoern <Bjoern.Gaier at horiba.com>; LLVM Developers Mailing List <llvm-dev at lists.llvm.org> Cc: andya at microsoft.com; Zachary Turner <zturner at google.com> Subject: Re: [llvm-dev] Crashes when adding VisualStduio generated object files to the JIT process Hi Björn I think it would be helpful to provide some more of the assembly you are ending up with. In the example below: Which source code are you compiling? What does it try to achieve? What happens to eax, ebx, rdi after the xor? +Andy,Zachary: Maybe you can correct, verify or explain in more detail my analysis in the end of this mail. I have no internal knowledge whatsoever, I just happened to reverse engineer this some time ago.>From what I found in the old thread and in your bug report, I can make a guess.I show you a short snippet of the assembly output. mov eax, DWORD PTR ?myInt@@3HA ; myInt lea rdi, OFFSET FLAT:__ImageBase xor ebx, ebx Then these offset is used to jump to some labels like "$LL4 at execute<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>:". The $LL4 looks like a symbol name in a jump table. There are a few examples (in the context of an unrelated bug report), that show how such tables look like: https://gist.github.com/rygorous/6790191 Here is an explanation of LEA: https://stackoverflow.com/questions/1658294 Am 22.03.18 um 18:43 schrieb Reid Kleckner via llvm-dev: I wouldn't be surprised if JITing COFF files on Windows doesn't work so well, since the object file format assumes most symbols are dllimport or within the local 2GB module address range. Reid gave an important hint here already: dllimport symbols. MSVC apparently still assumes a small code model and uses 32 bit relocations. (Maybe you could change that nowadays with a compiler flag or so?) Have a look at this thread from 2015: http://lists.llvm.org/pipermail/llvm-dev/2015-November/092727.html Quoting Andy Ayers: If at link time it turns out your export is from a DLL the linker will insert a jump stub / dllimport into the image for you which can handle larger distances. I guess this is the missing piece in your case. You have to do this patching step yourself, that would otherwise happen magically in the Microsoft linker (resolve relocation to a load-time jump stub) and loader (create/fill jump stub with the actual DLL symbol address), because the MSVC compiler assumes it when creating your obj file (creates the relocation for a base address, adds the offset and calls it). If this is the case, you should encounter __imp_foo symbol relocations. I don't remember the exact details, but this is what you want to watch out for. What needs to happen, conceptually, is something like this: __ImageBase is the address of a table that you create and OFFSET FLAT is a relative offset to the entry that you need to prepare. The entry must NOT store the target addresses (to be loaded and called), BUT a jump stub/trampoline like "jmp 0x00000000" (which forwards execution to the actual call target). When you resolve the relocation for the import symbol (__imp_foo) to the jump stub, you probably don't know the jmp target right away. So, you can leave it nulled and emit a new relocation to fill out the address (symbol foo). Depending on how your stub works, it may be a relative offset or a 64-bit absolute address. In the end your code calls the jump stub address as if it was the actual target function. IIRC you cannot simply resolve the original relocation to the target, even if you know it already, because the code emitted by MSVC has the base+offset calculation already built in. There are some pretty useful tools for all that in LLVM Orc: https://github.com/llvm-mirror/llvm/blob/master/lib/ExecutionEngine/Orc/IndirectionUtils.cpp Cheers Stefan Am 13.12.18 um 07:59 schrieb Gaier, Bjoern via llvm-dev: Hello everyone, I'm using the LLVM for a JIT-Client under Windows 64bit. I tried various stuff with it and noticed, that object files or libraries which are generated by the Visual Studio Compiler 2015 or 2017 will break the jitted code, when they are added to the process. I opened a bug for this a while ago (https://bugs.llvm.org/show_bug.cgi?id=39447) - sadly this bug will become a stopper for me... So I wanted to ask for advice: Does anyone know anything about this bug or a similar one? Does anyone have an idea about this? Is this bug maybe a duplication with another bug? I'm still a beginner, so I hope I did everything right >o< Kind greetings Björn Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. Robert Plank, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima. Junichi Tajika _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -- https://weliveindetail.github.io/blog/ https://cryptup.org/pub/stefan.graenitz at gmail.com Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. Robert Plank, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima. Junichi Tajika -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181218/769329cf/attachment.html>