We implemented a similar solution for handling crashes in production but one issue we came across with MCJIT was that the NotifyFunctionEmitted call from the old JIT was replaced with NotifyObjectEmitted. The ObjectImage used by NotifyObjectEmitted does have a way of iterating symbols but non-external functions used in the module didn't seem to appear in this list so we were left with some gaps in our symbol handling coverage when execution involved these. If there's a way around it that would be great, otherwise something to keep in mind. Cheers, Andrew On Wed, Nov 13, 2013 at 5:51 PM, Reid Kleckner <rnk at google.com> wrote:> Writing your own JITEventListener is probably the best way to capture > traces from production. Using the existing gdb/JIT interface support is > probably far too heavyweight for production. If you roll your own, you can > record the PC ranges efficiently and that should be good enough to capture > traces from production. > > > On Tue, Nov 12, 2013 at 2:56 PM, Priyendra Deshwal < > deshwal at thoughtspot.com> wrote: > >> [Resending this email after subscribing to the group. Got an error >> message the previous time around] >> >> Hey guys, >> >> In our project we have the following workflow: >> >> - Given a user request, we construct a C++ program to service the request >> - We create a compiler invocation to compile that program into an IR >> module >> - We use JIT (and optionally MCJIT) to convert the IR into executable >> code and run it >> >> Occasionally, we have a crash in the JIT code and we are looking to >> figure out best practices around debuggability. With the MCJIT option, we >> can debug in gdb so that is already a big step forward. >> >> However, that does not work for a production crash. We want to capture >> the stack trace from the crash and write it to stderr. Our binary has a >> signal handler that does stack unwinding and symbolization upon crash. >> Based on my reading so far, we need to implement a JITEventListener which >> would keep track of the dynamically generated symbol addresses and use the >> resulting maps for symbolization. >> >> Will that work? Is there a simpler way? >> >> Thanks, >> -- Priyendra >> >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >> >> > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131113/eea57069/attachment.html>
Thanks for the info guys. I will go ahead and implement the JITEventListener. Thanks for the MCJIT tip Andrew - will keep a look out for that. On Wed, Nov 13, 2013 at 9:23 AM, Andrew MacPherson <andrew.macp at gmail.com>wrote:> We implemented a similar solution for handling crashes in production but > one issue we came across with MCJIT was that the NotifyFunctionEmitted call > from the old JIT was replaced with NotifyObjectEmitted. The ObjectImage > used by NotifyObjectEmitted does have a way of iterating symbols but > non-external functions used in the module didn't seem to appear in this > list so we were left with some gaps in our symbol handling coverage when > execution involved these. If there's a way around it that would be great, > otherwise something to keep in mind. > > Cheers, > Andrew > > > On Wed, Nov 13, 2013 at 5:51 PM, Reid Kleckner <rnk at google.com> wrote: > >> Writing your own JITEventListener is probably the best way to capture >> traces from production. Using the existing gdb/JIT interface support is >> probably far too heavyweight for production. If you roll your own, you can >> record the PC ranges efficiently and that should be good enough to capture >> traces from production. >> >> >> On Tue, Nov 12, 2013 at 2:56 PM, Priyendra Deshwal < >> deshwal at thoughtspot.com> wrote: >> >>> [Resending this email after subscribing to the group. Got an error >>> message the previous time around] >>> >>> Hey guys, >>> >>> In our project we have the following workflow: >>> >>> - Given a user request, we construct a C++ program to service the request >>> - We create a compiler invocation to compile that program into an IR >>> module >>> - We use JIT (and optionally MCJIT) to convert the IR into executable >>> code and run it >>> >>> Occasionally, we have a crash in the JIT code and we are looking to >>> figure out best practices around debuggability. With the MCJIT option, we >>> can debug in gdb so that is already a big step forward. >>> >>> However, that does not work for a production crash. We want to capture >>> the stack trace from the crash and write it to stderr. Our binary has a >>> signal handler that does stack unwinding and symbolization upon crash. >>> Based on my reading so far, we need to implement a JITEventListener which >>> would keep track of the dynamically generated symbol addresses and use the >>> resulting maps for symbolization. >>> >>> Will that work? Is there a simpler way? >>> >>> Thanks, >>> -- Priyendra >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>> >>> >> >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131113/cb7ef86c/attachment.html>
>From NotifyObjectEmitted you should be able to get to any function that's visible via either the ELF headers or the DWARF information. Do you have functions that don't show up in either of those places?-Andy From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Andrew MacPherson Sent: Wednesday, November 13, 2013 9:23 AM To: Reid Kleckner Cc: llvmdev at cs.uiuc.edu Subject: Re: [LLVMdev] Stack traces from JIT code We implemented a similar solution for handling crashes in production but one issue we came across with MCJIT was that the NotifyFunctionEmitted call from the old JIT was replaced with NotifyObjectEmitted. The ObjectImage used by NotifyObjectEmitted does have a way of iterating symbols but non-external functions used in the module didn't seem to appear in this list so we were left with some gaps in our symbol handling coverage when execution involved these. If there's a way around it that would be great, otherwise something to keep in mind. Cheers, Andrew On Wed, Nov 13, 2013 at 5:51 PM, Reid Kleckner <rnk at google.com<mailto:rnk at google.com>> wrote: Writing your own JITEventListener is probably the best way to capture traces from production. Using the existing gdb/JIT interface support is probably far too heavyweight for production. If you roll your own, you can record the PC ranges efficiently and that should be good enough to capture traces from production. On Tue, Nov 12, 2013 at 2:56 PM, Priyendra Deshwal <deshwal at thoughtspot.com<mailto:deshwal at thoughtspot.com>> wrote: [Resending this email after subscribing to the group. Got an error message the previous time around] Hey guys, In our project we have the following workflow: - Given a user request, we construct a C++ program to service the request - We create a compiler invocation to compile that program into an IR module - We use JIT (and optionally MCJIT) to convert the IR into executable code and run it Occasionally, we have a crash in the JIT code and we are looking to figure out best practices around debuggability. With the MCJIT option, we can debug in gdb so that is already a big step forward. However, that does not work for a production crash. We want to capture the stack trace from the crash and write it to stderr. Our binary has a signal handler that does stack unwinding and symbolization upon crash. Based on my reading so far, we need to implement a JITEventListener which would keep track of the dynamically generated symbol addresses and use the resulting maps for symbolization. Will that work? Is there a simpler way? Thanks, -- Priyendra _______________________________________________ LLVM Developers mailing list LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu> http://llvm.cs.uiuc.edu http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev _______________________________________________ LLVM Developers mailing list LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu> http://llvm.cs.uiuc.edu http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131113/dcefaa9e/attachment.html>
Hi Andy, In the NotifyObjectEmitted method of our derived JITEventListener class we use the begin_symbol() iterator to walk the object's symbols looking for functions and only functions marked with ExternalLinkage seem to show up. I'm not sure how I would access the Dwarf info from within there, is there a way? Thanks, Andrew On Wed, Nov 13, 2013 at 8:23 PM, Kaylor, Andrew <andrew.kaylor at intel.com>wrote:> From NotifyObjectEmitted you should be able to get to any function > that’s visible via either the ELF headers or the DWARF information. Do you > have functions that don’t show up in either of those places? > > > > -Andy > > > > *From:* llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] *On > Behalf Of *Andrew MacPherson > *Sent:* Wednesday, November 13, 2013 9:23 AM > *To:* Reid Kleckner > *Cc:* llvmdev at cs.uiuc.edu > *Subject:* Re: [LLVMdev] Stack traces from JIT code > > > > We implemented a similar solution for handling crashes in production but > one issue we came across with MCJIT was that the NotifyFunctionEmitted call > from the old JIT was replaced with NotifyObjectEmitted. The ObjectImage > used by NotifyObjectEmitted does have a way of iterating symbols but > non-external functions used in the module didn't seem to appear in this > list so we were left with some gaps in our symbol handling coverage when > execution involved these. If there's a way around it that would be great, > otherwise something to keep in mind. > > Cheers, > Andrew > > > > On Wed, Nov 13, 2013 at 5:51 PM, Reid Kleckner <rnk at google.com> wrote: > > Writing your own JITEventListener is probably the best way to capture > traces from production. Using the existing gdb/JIT interface support is > probably far too heavyweight for production. If you roll your own, you can > record the PC ranges efficiently and that should be good enough to capture > traces from production. > > > > On Tue, Nov 12, 2013 at 2:56 PM, Priyendra Deshwal < > deshwal at thoughtspot.com> wrote: > > [Resending this email after subscribing to the group. Got an error > message the previous time around] > > > > Hey guys, > > > > In our project we have the following workflow: > > > > - Given a user request, we construct a C++ program to service the request > > - We create a compiler invocation to compile that program into an IR module > > - We use JIT (and optionally MCJIT) to convert the IR into executable code > and run it > > > > Occasionally, we have a crash in the JIT code and we are looking to figure > out best practices around debuggability. With the MCJIT option, we can > debug in gdb so that is already a big step forward. > > > > However, that does not work for a production crash. We want to capture the > stack trace from the crash and write it to stderr. Our binary has a signal > handler that does stack unwinding and symbolization upon crash. Based on my > reading so far, we need to implement a JITEventListener which would keep > track of the dynamically generated symbol addresses and use the resulting > maps for symbolization. > > > > Will that work? Is there a simpler way? > > > > Thanks, > > -- Priyendra > > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > > > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20131113/e023cf86/attachment.html>