I originally sent the message below to xen-tools, but received no response. I would really appreciate some help. I am writing a significant piece of Free Software (persistent memory for the Linux kernel) and would like to use gdbserver-xen to debug it (on x86_64). Although I have been very careful, gdbserver-xen does not quite work, and I have no idea why (see forwarded message below). I subsequently tried using xen 3.1.0_rc10, only to get similarly poor behaviour from gdb (maybe I should try 3.1 as well). I have spent quite some time fiddling with gdbserver-xen, and am rather disappointed to have nothing useful to show for it. Any suggestions would be appreciated. If anyone is interested, I can provide more detailed information. Tim _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I''m afraid the simple answer is that gdbserver-xen spends a lot of the time broken in one way or another because none of the core developers use it or test it. In this case it looks like guest memory can''t be accessed for some reason -- do you get similar messages if you try to use tools/xentrace/xenctx binary on this VM? It looks like it might not be too hard to fix this particular issue, since it''s rather un-subtle. :-) -- Keir On 22/6/07 18:51, "Tim Barbour" <trb@phasechangeit.com> wrote:> I originally sent the message below to xen-tools, but received no response. I > would really appreciate some help. > > I am writing a significant piece of Free Software (persistent memory for the > Linux kernel) and would like to use gdbserver-xen to debug it (on x86_64). > Although I have been very careful, gdbserver-xen does not quite work, and I > have > no idea why (see forwarded message below). I subsequently tried using xen > 3.1.0_rc10, only to get similarly poor behaviour from gdb (maybe I should try > 3.1 as well). > > I have spent quite some time fiddling with gdbserver-xen, and am rather > disappointed to have nothing useful to show for it. > > Any suggestions would be appreciated. If anyone is interested, I can provide > more detailed information. > > Tim > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser writes: > I''m afraid the simple answer is that gdbserver-xen spends a lot of the time > broken in one way or another because none of the core developers use it or > test it. In this case it looks like guest memory can''t be accessed for some > reason -- do you get similar messages if you try to use > tools/xentrace/xenctx binary on this VM? It looks like it might not be too > hard to fix this particular issue, since it''s rather un-subtle. :-) Thank you for the suggestion. I have not used tools/xentrace/xenctx before, but it looks as if it is working okay. I append its output below, followed by the corresponding output from xengdb (using gdbserver-xen). I tried xenctx on its own, then I attached gdbserver-xen to the VM, and used both together, so they should produce the same stack trace. What further diagnostic information would be useful ? Tim trb@elysium ~ OK sudo /usr/lib/xen/bin/xenctx --stack-trace -f -s /xen/boot/System.map-2.6.18-pmm 4 rip: ffffffff802063aa hypercall_page+0x3aa rsp: ffffffff806c9f10 rax: 00000000 rbx: 00000000 rcx: ffffffff802063aa rdx: deadbeef rsi: deadbeef rdi: deadbeef rbp: ffffffff806c9f48 r8: 00000011 r9: ffffffff8071fe58 r10: 100031311 r11: 00000246 r12: 00000000 r13: 00000000 r14: 00000000 r15: 00000000 cs: 0000e033 ds: 00000000 fs: 00000000 gs: 00000000 Stack: 0000000000000000 000121efe71d8a36 ffffffff8020e8c3 000121efe71d8a36 0000000000000000 ffffffff806c9f48 ffffffff806ff004 ffffffff806c9f58 ffffffff80209455 ffffffff806c9f78 ffffffff80208cba 0000000000000000 0000000000000800 ffffffff806c9f88 ffffffff802073b6 ffffffff806c9fa8 Code: 00 00 00 00 00 00 00 00 00 00 00 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 00 00 00 00 00 00 00 Stack Trace: * [<ffffffff802063aa>] hypercall_page+0x3aa <-- | 0000000000000000 | 000121efe71d8a36 | ffffffff8020e8c3 | 000121efe71d8a36 | 0000000000000000 | ffffffff806c9f48 | ffffffff806ff004 |-- ffffffff806c9f58 | [<ffffffff80209455>] xen_idle+0x75 |-- ffffffff806c9f78 | [<ffffffff80208cba>] cpu_idle+0x5a | 0000000000000000 | 0000000000000800 |-- ffffffff806c9f88 | [<ffffffff802073b6>] rest_init+0x26 |-- ffffffff806c9fa8 | [<ffffffff806d28d5>] start_kernel+0x275 | 0000000000000000 | ffffffff8070c180 |-- ffffffff806c9fe8 | [<ffffffff806d220d>] __init_begin+0x20d | ffff800000000000 | ffff804000000000 | 00000007ffffffff | 0000000000000000 | 0000000000000000 | 0000000000000000 |-- 0000000000000000 trb@elysium ~ OK trb@elysium ~ OK sudo gdbserver-xen 127.0.0.1:9999 --attach 4 Attached; pid = 4 Listening on port 9999 Remote debugging from host 127.0.0.1 trb@elysium /usr/local/src/linux-2.6.18-xen-pmm OK xengdb /xen/boot/vmlinux-syms-2.6.18-pmm GNU gdb 6.2.1 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1". (gdb) set architecture i386:x86-64:intel The target architecture is assumed to be i386:x86-64:intel (gdb) target remote 127.0.0.1:9999 Remote debugging using 127.0.0.1:9999 [New Thread 0] [Switching to Thread 0] 0xffffffff802063aa in hypercall_page () (gdb) bt #0 0xffffffff802063aa in hypercall_page () Cannot access memory at address 0xffffffff806c9f20 (gdb) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser writes: > I''m afraid the simple answer is that gdbserver-xen spends a lot of the time > broken in one way or another because none of the core developers use it or > test it. So what debugger do the core developers use or how do they debug things ? Tim _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim, Did you get any response? -Kaushik -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tim Barbour Sent: Friday, June 29, 2007 11:51 AM To: Keir Fraser Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] gdbserver-xen frustration on x86_64 Keir Fraser writes: > I''m afraid the simple answer is that gdbserver-xen spends a lot of the time > broken in one way or another because none of the core developers use it or > test it. So what debugger do the core developers use or how do they debug things ? Tim _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thank you for asking. I got no response at all. IIRC, your question is only the second response I have have ever received on this topic, despite several posts (starting with xen-tools). I am now using kgdb to debug my project. Please CC: me on any responses, as I unsubscribed from xen-devel yesterday. Tim Kaushik Barde writes: > Tim, > > Did you get any response? > > -Kaushik > > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tim Barbour > Sent: Friday, June 29, 2007 11:51 AM > To: Keir Fraser > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] gdbserver-xen frustration on x86_64 > > Keir Fraser writes: > > I''m afraid the simple answer is that gdbserver-xen spends a lot of > the time > > broken in one way or another because none of the core developers use > it or > > test it. > > So what debugger do the core developers use or how do they debug things > ? > > Tim > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
No problem, will do that. We may end-up doing the same. But can you debug xen (not dom0 code) with kgdb effectively? Best. -Kaushik -----Original Message----- From: Tim Barbour [mailto:trb@phasechangeit.com] Sent: Saturday, July 14, 2007 6:19 PM To: Kaushik Barde Cc: trb@sdrit.com; Keir Fraser; xen-devel@lists.xensource.com Subject: RE: [Xen-devel] gdbserver-xen frustration on x86_64 Thank you for asking. I got no response at all. IIRC, your question is only the second response I have have ever received on this topic, despite several posts (starting with xen-tools). I am now using kgdb to debug my project. Please CC: me on any responses, as I unsubscribed from xen-devel yesterday. Tim Kaushik Barde writes: > Tim, > > Did you get any response? > > -Kaushik > > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tim Barbour > Sent: Friday, June 29, 2007 11:51 AM > To: Keir Fraser > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] gdbserver-xen frustration on x86_64 > > Keir Fraser writes: > > I''m afraid the simple answer is that gdbserver-xen spends a lot of > the time > > broken in one way or another because none of the core developers use > it or > > test it. > > So what debugger do the core developers use or how do they debug things > ? > > Tim > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
If a bug crashes Xen then the printed backtrace is almost always sufficient. Kdump can also be useful if we want to post-mortem analyse the heap. For more subtle bugs, we almost always look to find a workload that reliably repros the issue, and then we add custom tracing to Xen to narrow down the possibilities. This latter kind of bug is not really amenable to debugging with a traditional debugger. -- Keir On 14/7/07 19:41, "Kaushik Barde" <Kaushik_Barde@Phoenix.com> wrote:> Tim, > > Did you get any response? > > -Kaushik > > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tim Barbour > Sent: Friday, June 29, 2007 11:51 AM > To: Keir Fraser > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] gdbserver-xen frustration on x86_64 > > Keir Fraser writes: >> I''m afraid the simple answer is that gdbserver-xen spends a lot of > the time >> broken in one way or another because none of the core developers use > it or >> test it. > > So what debugger do the core developers use or how do they debug things > ? > > Tim > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel