llvm dev - Mar 2020 - [MCJIT] messy call stack debug on x64 code in VisualStudio

I've always just hacked support for this in to the various JITs (for
JuliaLang, in our debuginfo.cpp file), by setting the
no-frame-pointer-optim flag in the IR, then creating and populating a dummy
unwind description object in the .text section, and registering that
dynamically. Some day I hope to actually just register the .pdata/.xdata
sections with the unwinder.

PDBs are a bit different though, since the above steps work well for gdb,
but generally I find that WinDbg is less willing or able to be given
JIT-frame information from LLVM. (I assume somehow it can be done, for
dotNET. I just don't know how.)

On Sat, Feb 29, 2020 at 11:07 PM Reid Kleckner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Yes, I think https://bugs.llvm.org/show_bug.cgi?id=24233 needs to be
> implemented to fix this.
>
> The Windows x64 unwinder doesn't generally look at frame pointers. We
> would need to register unwind info to make this work. What you see is
> fairly typical of attempting to unwind the stack when unwind info is
> missing.
>
> PDBs shouldn't generally enter into the picture.
>
> On Sat, Feb 29, 2020 at 8:14 AM Vivien Millet via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi,
>>
>> I'm using IR and MCJIT to compile a script language. I debug it
with on
>> the fly generated .pdb files. During debugging, almost each time I step
>> into a function, I loose information about calling function inside the
>> visual studio callstack view or I have a bunch of pure addresses in the
>> callstack in between the current function and the calling function, for
>> example :
>>
>> MyJit.dll!MyCurrentFunction()
>> [0x1234567887654321]
>> [0x8765432112345678]
>> MyJit.dll!MyCallingFunction()
>> ...
>>
>> It looks like visual studio get lost while walking up stack.
>> Does anyone know where it could come from ?
>>
>> I have disabled all optimisations (among them is the
omit-frame-pointer).
>>
>> I have seen this bug here : https://bugs.llvm.org/show_bug.cgi?id=24233
which
>> is quite similar but it is quite old now, and since the proposed patch
has
>> been posted, the code in RuntimeDyldCOFFX86_64.h has changed and it is
>> difficult for me to know if it has really been fixed since or not.
>>
>> Could it be related to the way IR CreateAlloca are used to build local
>> variables ? Could it be related to missing informations inside the PDB
? (I
>> don't know if there is stack related information inside PDB files
to ensure
>> good stack walking).
>>
>> Thanks.
>>
>> Vivien
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200229/ab88b2c6/attachment.html>

Sorry for being late to reply, I was investigating what you advised to me.
And with all your informations and some (really) hard time I succeeded to
register the unwinding information (not the way you think) and have a clear
callstack ! it feels good !

@Reid Kleckner <rnk at google.com>
After looking deeper, I don't understand why this bug is open for such a
long time as everything is here to fix it : the unwinding process is
completely implemented in LLVM and is enabled by adding
llvm::Attribute::UWTable to llvm::Function instances.
The only thing missing is the calls to RtlAddFunctionTable. They should be
added to the default MCJIT or OrcJIT memory manager (maybe with an option
to enable/disable them), simply by tracking memory requested for allocation
of ".pdata" and ".xdata" sections and calling
RtlAddFunctionTable on
notifyObjectLoaded. I don't know who is responsible for this, but that
might be an easy win for a great feature completion (this is nice to avoid
the user having to understand all this machinery by themselves...)

In my case, I can't call RtlAddFunctionTable because I inject my code into
a fake .DLL (for PDB hotreload purpose) which already have its static
function table.
To explain my process (for other devs willing to suffer like I did) :
 If debugging is required by the user, I switch from MCJITMemoryManager to
a home-made DllMemoryManager which :
- loads a dummy .DLL consisting of empty .text .rdata (including .xdata)
and .pdata sections, without relocations.
- allocates memory inside the loaded .DLL address range (the dll has been
hacked for WRITE access)
- unload the .DLL
- generate a PDB
- *INTERESTING PART* : rewrites PDATA and XDATA sections with the one
emitted by LLVM (fixing virtual address inside the image).
- writes the .DLL back on disk
- make PDB file match .DLL file (GUID)
- reload the .DLL (it reloads at same position 100% of the time in my case,
I might be lucky but I'm ok with it) so that visual studio detects it and
load the matching PDB.

All of this process was painful but it works and I can build and rebuild my
language on-the-fly while debugging it alongside with native code inside
Visual Studio. You might wonder why not generate a real .dll and reload it
? Because I can't predict where the functions will be reloaded inside
memory and I need to keep "reflection" of the JIT symbols. (+ it's
slower
and requires a lot of work on the mangle/link/pdb dependency sides).

@Jameson : Thanks for your feedback, it helped me to identify .xdata and
.pdata sections for unwind stuff !
Are you sure that unwinding info is not enough for you to make it
debuggable ? I personally removed the "no-frame-pointer-elim" and it
keeps
working well, I keep seeing my full callstack (maybe is it only useful on
x86 ?), because Win64 does not use RBP to walk the stack at all, all is
done with unwind infos apparently.
PDB are not concerned at all with all of this, I thought it might but no...


Le dim. 1 mars 2020 à 05:54, Jameson Nash <vtjnash at gmail.com> a écrit :
> I've always just hacked support for this in to the various JITs (for
> JuliaLang, in our debuginfo.cpp file), by setting the
> no-frame-pointer-optim flag in the IR, then creating and populating a dummy
> unwind description object in the .text section, and registering that
> dynamically. Some day I hope to actually just register the .pdata/.xdata
> sections with the unwinder.
>
> PDBs are a bit different though, since the above steps work well for gdb,
> but generally I find that WinDbg is less willing or able to be given
> JIT-frame information from LLVM. (I assume somehow it can be done, for
> dotNET. I just don't know how.)
>
> On Sat, Feb 29, 2020 at 11:07 PM Reid Kleckner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Yes, I think https://bugs.llvm.org/show_bug.cgi?id=24233 needs to be
>> implemented to fix this.
>>
>> The Windows x64 unwinder doesn't generally look at frame pointers.
We
>> would need to register unwind info to make this work. What you see is
>> fairly typical of attempting to unwind the stack when unwind info is
>> missing.
>>
>> PDBs shouldn't generally enter into the picture.
>>
>> On Sat, Feb 29, 2020 at 8:14 AM Vivien Millet via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi,
>>>
>>> I'm using IR and MCJIT to compile a script language. I debug it
with on
>>> the fly generated .pdb files. During debugging, almost each time I
step
>>> into a function, I loose information about calling function inside
the
>>> visual studio callstack view or I have a bunch of pure addresses in
the
>>> callstack in between the current function and the calling function,
for
>>> example :
>>>
>>> MyJit.dll!MyCurrentFunction()
>>> [0x1234567887654321]
>>> [0x8765432112345678]
>>> MyJit.dll!MyCallingFunction()
>>> ...
>>>
>>> It looks like visual studio get lost while walking up stack.
>>> Does anyone know where it could come from ?
>>>
>>> I have disabled all optimisations (among them is the
omit-frame-pointer).
>>>
>>> I have seen this bug here :
https://bugs.llvm.org/show_bug.cgi?id=24233 which
>>> is quite similar but it is quite old now, and since the proposed
patch has
>>> been posted, the code in RuntimeDyldCOFFX86_64.h has changed and it
is
>>> difficult for me to know if it has really been fixed since or not.
>>>
>>> Could it be related to the way IR CreateAlloca are used to build
local
>>> variables ? Could it be related to missing informations inside the
PDB ? (I
>>> don't know if there is stack related information inside PDB
files to ensure
>>> good stack walking).
>>>
>>> Thanks.
>>>
>>> Vivien
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200304/7ce5d1f8/attachment.html>

I think it's not enough, for reasons related to the pain you went through.
Normally, the JIT doesn't have a backing DLL and so it doesn't support
the
relocation type required by the xdata and pdata sections. As you say, it
generally doesn't work very well in WinDBG anyways, since they removed the
use of RBP to walk the stack. At the time I wrote my hack, LLVM didn't even
have emission code for those unwind sections, and it hasn't been worth the
hard effort to change it for me. Is your code public somewhere? It seems
like it could be useful for all JIT users to have a drop-in option to
enable debugging.

On Tue, Mar 3, 2020 at 6:06 PM Vivien Millet <vivien.millet at gmail.com>
wrote:
> Sorry for being late to reply, I was investigating what you advised to me.
> And with all your informations and some (really) hard time I succeeded to
> register the unwinding information (not the way you think) and have a clear
> callstack ! it feels good !
>
> @Reid Kleckner <rnk at google.com>
> After looking deeper, I don't understand why this bug is open for such
a
> long time as everything is here to fix it : the unwinding process is
> completely implemented in LLVM and is enabled by adding
> llvm::Attribute::UWTable to llvm::Function instances.
> The only thing missing is the calls to RtlAddFunctionTable. They should be
> added to the default MCJIT or OrcJIT memory manager (maybe with an option
> to enable/disable them), simply by tracking memory requested for allocation
> of ".pdata" and ".xdata" sections and calling
RtlAddFunctionTable on
> notifyObjectLoaded. I don't know who is responsible for this, but that
> might be an easy win for a great feature completion (this is nice to avoid
> the user having to understand all this machinery by themselves...)
>
> In my case, I can't call RtlAddFunctionTable because I inject my code
into
> a fake .DLL (for PDB hotreload purpose) which already have its static
> function table.
> To explain my process (for other devs willing to suffer like I did) :
>  If debugging is required by the user, I switch from MCJITMemoryManager to
> a home-made DllMemoryManager which :
> - loads a dummy .DLL consisting of empty .text .rdata (including .xdata)
> and .pdata sections, without relocations.
> - allocates memory inside the loaded .DLL address range (the dll has been
> hacked for WRITE access)
> - unload the .DLL
> - generate a PDB
> - *INTERESTING PART* : rewrites PDATA and XDATA sections with the one
> emitted by LLVM (fixing virtual address inside the image).
> - writes the .DLL back on disk
> - make PDB file match .DLL file (GUID)
> - reload the .DLL (it reloads at same position 100% of the time in my
> case, I might be lucky but I'm ok with it) so that visual studio
detects it
> and load the matching PDB.
>
> All of this process was painful but it works and I can build and rebuild
> my language on-the-fly while debugging it alongside with native code inside
> Visual Studio. You might wonder why not generate a real .dll and reload it
> ? Because I can't predict where the functions will be reloaded inside
> memory and I need to keep "reflection" of the JIT symbols. (+
it's slower
> and requires a lot of work on the mangle/link/pdb dependency sides).
>
> @Jameson : Thanks for your feedback, it helped me to identify .xdata and
> .pdata sections for unwind stuff !
> Are you sure that unwinding info is not enough for you to make it
> debuggable ? I personally removed the "no-frame-pointer-elim" and
it keeps
> working well, I keep seeing my full callstack (maybe is it only useful on
> x86 ?), because Win64 does not use RBP to walk the stack at all, all is
> done with unwind infos apparently.
> PDB are not concerned at all with all of this, I thought it might but no...
>
>
> Le dim. 1 mars 2020 à 05:54, Jameson Nash <vtjnash at gmail.com> a
écrit :
>
>> I've always just hacked support for this in to the various JITs
(for
>> JuliaLang, in our debuginfo.cpp file), by setting the
>> no-frame-pointer-optim flag in the IR, then creating and populating a
dummy
>> unwind description object in the .text section, and registering that
>> dynamically. Some day I hope to actually just register the
.pdata/.xdata
>> sections with the unwinder.
>>
>> PDBs are a bit different though, since the above steps work well for
gdb,
>> but generally I find that WinDbg is less willing or able to be given
>> JIT-frame information from LLVM. (I assume somehow it can be done, for
>> dotNET. I just don't know how.)
>>
>> On Sat, Feb 29, 2020 at 11:07 PM Reid Kleckner via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Yes, I think https://bugs.llvm.org/show_bug.cgi?id=24233 needs to
be
>>> implemented to fix this.
>>>
>>> The Windows x64 unwinder doesn't generally look at frame
pointers. We
>>> would need to register unwind info to make this work. What you see
is
>>> fairly typical of attempting to unwind the stack when unwind info
is
>>> missing.
>>>
>>> PDBs shouldn't generally enter into the picture.
>>>
>>> On Sat, Feb 29, 2020 at 8:14 AM Vivien Millet via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm using IR and MCJIT to compile a script language. I
debug it with on
>>>> the fly generated .pdb files. During debugging, almost each
time I step
>>>> into a function, I loose information about calling function
inside the
>>>> visual studio callstack view or I have a bunch of pure
addresses in the
>>>> callstack in between the current function and the calling
function, for
>>>> example :
>>>>
>>>> MyJit.dll!MyCurrentFunction()
>>>> [0x1234567887654321]
>>>> [0x8765432112345678]
>>>> MyJit.dll!MyCallingFunction()
>>>> ...
>>>>
>>>> It looks like visual studio get lost while walking up stack.
>>>> Does anyone know where it could come from ?
>>>>
>>>> I have disabled all optimisations (among them is the
>>>> omit-frame-pointer).
>>>>
>>>> I have seen this bug here :
https://bugs.llvm.org/show_bug.cgi?id=24233 which
>>>> is quite similar but it is quite old now, and since the
proposed patch has
>>>> been posted, the code in RuntimeDyldCOFFX86_64.h has changed
and it is
>>>> difficult for me to know if it has really been fixed since or
not.
>>>>
>>>> Could it be related to the way IR CreateAlloca are used to
build local
>>>> variables ? Could it be related to missing informations inside
the PDB ? (I
>>>> don't know if there is stack related information inside PDB
files to ensure
>>>> good stack walking).
>>>>
>>>> Thanks.
>>>>
>>>> Vivien
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200304/d8f8266d/attachment.html>