Bryan Donlan
2009-Mar-26 06:41 UTC
[Xen-devel] Bug: ptrace issues under x86_64 Xen kernel 2.6.29
Using 2.6.29 or 2.6.28 as a 64-bit Xen domU, a number of ptrace() users seem to have issues with unexpected breakpoints. ltrace and gdb both seem to be affected, under both 64-bit and 32-bit userspace. 32-bit kernels do not seem to be affected. Typical symptoms look like: x86 li63-205:/# ltrace /bin/true unexpected breakpoint at 0xf7e6d89f unexpected breakpoint at 0xf7e60a3f unexpected breakpoint at 0xf7e6464f unexpected breakpoint at 0x804933f unexpected breakpoint at 0xf7ea509f unexpected breakpoint at 0xf7e9b1ff unexpected breakpoint at 0xf7efef66 +++ exited (status 0) +++ x64 li63-205:~/linux-2.6# ltrace true unexpected breakpoint at 0x7f3379878f1f unexpected breakpoint at 0x7f337986ca3f unexpected breakpoint at 0x7f337986fd3f unexpected breakpoint at 0x402bdf unexpected breakpoint at 0x7f33798b07bf unexpected breakpoint at 0x7f33798a696f unexpected breakpoint at 0x7f3379905e3f +++ exited (status 0) +++ li63-205:~# gdb /bin/true GNU gdb 6.8-debian Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu"... (no debugging symbols found) (gdb) run Starting program: /bin/true (no debugging symbols found) (no debugging symbols found) Program received signal SIGTRAP, Trace/breakpoint trap. 0x00007f640680e4e7 in ?? () from /lib64/ld-linux-x86-64.so.2 (gdb) In the specific case of the above gdb session, here is /proc/pid/maps from the tracee: 00400000-00407000 r-xp 00000000 ca:00 9531 /bin/true 00607000-00608000 rw-p 00007000 ca:00 9531 /bin/true 7f64067f9000-7f6406816000 r-xp 00000000 ca:00 1678 /lib/ld-2.9.so 7f6406a12000-7f6406a15000 rw-p 7f6406a12000 00:00 0 7f6406a15000-7f6406a17000 rw-p 0001c000 ca:00 1678 /lib/ld-2.9.so 7fff0ea01000-7fff0ea16000 rw-p 7ffffffea000 00:00 0 [stack] 7fff0ebfe000-7fff0ebff000 r-xp 7fff0ebfe000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] strace seems(?) to be unaffected. I''ve attached my .config, dmesg, and strace output for running ltrace (32-bit) on /bin/true on the affected system. Unfortunately, I''m not enough of a ptrace guru to interpret what the cause might be. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Mar-26 19:15 UTC
[Xen-devel] Re: Bug: ptrace issues under x86_64 Xen kernel 2.6.29
Bryan Donlan wrote:> Using 2.6.29 or 2.6.28 as a 64-bit Xen domU, a number of ptrace() > users seem to have issues with unexpected breakpoints. ltrace and gdb > both seem to be affected, under both 64-bit and 32-bit userspace. > 32-bit kernels do not seem to be affected. Typical symptoms look like: >Thanks for the reminder. I''d noticed it in passing, but haven''t investigated it. J> x86 li63-205:/# ltrace /bin/true > unexpected breakpoint at 0xf7e6d89f > unexpected breakpoint at 0xf7e60a3f > unexpected breakpoint at 0xf7e6464f > unexpected breakpoint at 0x804933f > unexpected breakpoint at 0xf7ea509f > unexpected breakpoint at 0xf7e9b1ff > unexpected breakpoint at 0xf7efef66 > +++ exited (status 0) +++ > > x64 li63-205:~/linux-2.6# ltrace true > unexpected breakpoint at 0x7f3379878f1f > unexpected breakpoint at 0x7f337986ca3f > unexpected breakpoint at 0x7f337986fd3f > unexpected breakpoint at 0x402bdf > unexpected breakpoint at 0x7f33798b07bf > unexpected breakpoint at 0x7f33798a696f > unexpected breakpoint at 0x7f3379905e3f > +++ exited (status 0) +++ > > li63-205:~# gdb /bin/true > GNU gdb 6.8-debian > Copyright (C) 2008 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-linux-gnu"... > (no debugging symbols found) > (gdb) run > Starting program: /bin/true > (no debugging symbols found) > (no debugging symbols found) > > Program received signal SIGTRAP, Trace/breakpoint trap. > 0x00007f640680e4e7 in ?? () from /lib64/ld-linux-x86-64.so.2 > (gdb) > > In the specific case of the above gdb session, here is /proc/pid/maps > from the tracee: > 00400000-00407000 r-xp 00000000 ca:00 9531 > /bin/true > 00607000-00608000 rw-p 00007000 ca:00 9531 > /bin/true > 7f64067f9000-7f6406816000 r-xp 00000000 ca:00 1678 > /lib/ld-2.9.so > 7f6406a12000-7f6406a15000 rw-p 7f6406a12000 00:00 0 > 7f6406a15000-7f6406a17000 rw-p 0001c000 ca:00 1678 > /lib/ld-2.9.so > 7fff0ea01000-7fff0ea16000 rw-p 7ffffffea000 00:00 0 [stack] > 7fff0ebfe000-7fff0ebff000 r-xp 7fff0ebfe000 00:00 0 [vdso] > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 > [vsyscall] > > strace seems(?) to be unaffected. > > I''ve attached my .config, dmesg, and strace output for running ltrace > (32-bit) on /bin/true on the affected system. Unfortunately, I''m not > enough of a ptrace guru to interpret what the cause might be._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Mar-30 03:01 UTC
[Xen-devel] Re: Bug: ptrace issues under x86_64 Xen kernel 2.6.29
Bryan Donlan wrote:> Using 2.6.29 or 2.6.28 as a 64-bit Xen domU, a number of ptrace() > users seem to have issues with unexpected breakpoints. ltrace and gdb > both seem to be affected, under both 64-bit and 32-bit userspace. > 32-bit kernels do not seem to be affected. Typical symptoms look like: >It looks like this is because the kernel sets up int3 (breakpoint) and debug (watchpoints, etc) to be on a separate debug stack in the tss. Xen doesn''t do this (and doesn''t appear to have a mechanism to do so), so I guess the on-stack format isn''t what the kernel expects. Does the patch below work? J From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> Date: Sun, 29 Mar 2009 19:56:29 -0700 Subject: [PATCH] xen/x86-64: fix breakpoints and hardware watchpoints Native x86-64 uses the IST mechanism to run int3 and debug traps on an alternative stack. Xen does not do this, and so the frames were being misinterpreted by the ptrace code. This change special-cases these two exceptions by using Xen variants which run on the normal kernel stack properly. Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 0d53425..d696185 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -13,6 +13,8 @@ asmlinkage void divide_error(void); asmlinkage void debug(void); asmlinkage void nmi(void); asmlinkage void int3(void); +asmlinkage void xen_debug(void); +asmlinkage void xen_int3(void); asmlinkage void overflow(void); asmlinkage void bounds(void); asmlinkage void invalid_op(void); diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S index 3f129d9..6cde382 100644 --- a/arch/x86/kernel/entry_64.S +++ b/arch/x86/kernel/entry_64.S @@ -1383,6 +1383,10 @@ END(xen_failsafe_callback) paranoidzeroentry_ist debug do_debug DEBUG_STACK paranoidzeroentry_ist int3 do_int3 DEBUG_STACK +#ifdef CONFIG_XEN +zeroentry xen_debug do_debug +zeroentry xen_int3 do_int3 +#endif paranoiderrorentry stack_segment do_stack_segment errorentry general_protection do_general_protection errorentry page_fault do_page_fault diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c index 02b169e..f98124f 100644 --- a/arch/x86/xen/enlighten.c +++ b/arch/x86/xen/enlighten.c @@ -20,6 +20,7 @@ #include <linux/delay.h> #include <linux/start_kernel.h> #include <linux/sched.h> +#include <linux/kprobes.h> #include <linux/bootmem.h> #include <linux/module.h> #include <linux/mm.h> @@ -44,6 +45,7 @@ #include <asm/processor.h> #include <asm/proto.h> #include <asm/msr-index.h> +#include <asm/traps.h> #include <asm/setup.h> #include <asm/desc.h> #include <asm/pgtable.h> @@ -435,11 +437,24 @@ static void xen_write_ldt_entry(struct desc_struct *dt, int entrynum, static int cvt_gate_to_trap(int vector, const gate_desc *val, struct trap_info *info) { + unsigned long addr; + if (val->type != GATE_TRAP && val->type != GATE_INTERRUPT) return 0; info->vector = vector; - info->address = gate_offset(*val); + + addr = gate_offset(*val); +#ifdef CONFIG_X86_64 + if (addr == (unsigned long)debug) + addr = (unsigned long)xen_debug; + else if (addr == (unsigned long)int3) + addr = (unsigned long)xen_int3; + else + WARN_ON(val->ist != 0); +#endif /* CONFIG_X86_64 */ + info->address = addr; + info->cs = gate_segment(*val); info->flags = val->dpl; /* interrupt gates clear IF */ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mark Wielaard
2009-Apr-07 07:46 UTC
[Xen-devel] Re: Bug: ptrace issues under x86_64 Xen kernel 2.6.29
Jeremy Fitzhardinge <jeremy <at> goop.org> writes:> Bryan Donlan wrote: > > Using 2.6.29 or 2.6.28 as a 64-bit Xen domU, a number of ptrace() > > users seem to have issues with unexpected breakpoints. ltrace and gdb > > both seem to be affected, under both 64-bit and 32-bit userspace. > > 32-bit kernels do not seem to be affected. Typical symptoms look like: > > > It looks like this is because the kernel sets up int3 (breakpoint) and > debug (watchpoints, etc) to be on a separate debug stack in the tss. > Xen doesn''t do this (and doesn''t appear to have a mechanism to do so), > so I guess the on-stack format isn''t what the kernel expects. Does the > patch below work?It works for me. With the patch I can use ltrace, gdb and systemtap uprobes support on the guest. But during bootup I do see the newly introduced WARN_ON(val->ist != 0) being triggered: WARNING: at arch/x86/xen/enlighten.c:453 cvt_gate_to_trap+0x80/0xc0() (Not tainted) Modules linked in: Pid: 0, comm: swapper Not tainted 2.6.29.1-15.mjw.x86_64 #1 Call Trace: [<ffffffff81048e84>] warn_slowpath+0xdb/0xfa [<ffffffff8100e8a1>] ? __xen_spin_lock+0xae/0xc1 [<ffffffff8100e5d9>] ? xen_spin_unlock+0x11/0x2e [<ffffffff810993bc>] ? trace_hardirqs_off+0x9/0x20 [<ffffffff810993bc>] ? trace_hardirqs_off+0x9/0x20 [<ffffffff81365246>] ? _spin_unlock_irqrestore+0x27/0x3e [<ffffffff8104959d>] ? release_console_sem+0x1d4/0x1e0 [<ffffffff81049af9>] ? vprintk+0x313/0x326 [<ffffffff810993bc>] ? trace_hardirqs_off+0x9/0x20 [<ffffffff81365246>] ? _spin_unlock_irqrestore+0x27/0x3e [<ffffffff8100c4d5>] ? get_phys_to_machine+0x1a/0x31 [<ffffffff810993bc>] ? trace_hardirqs_off+0x9/0x20 [<ffffffff8102ac6b>] ? pvclock_clocksource_read+0x42/0x7b [<ffffffff8102ac6b>] ? pvclock_clocksource_read+0x42/0x7b [<ffffffff81365890>] ? nmi+0x0/0x51 [<ffffffff8100aa86>] cvt_gate_to_trap+0x80/0xc0 [<ffffffff8100ab14>] xen_convert_trap_info+0x4e/0x7e [<ffffffff8100b731>] xen_load_idt+0x47/0x71 [<ffffffff8135d8e3>] cpu_init+0xd6/0x331 [<ffffffff8100b4cc>] ? xen_write_idt_entry+0x41/0xa5 [<ffffffff8118679a>] ? generic_swap+0x0/0x1c [<ffffffff8117f418>] ? cmp_ex+0x0/0x15 [<ffffffff815d2a08>] trap_init+0x1b5/0x1b7 [<ffffffff815cbc06>] start_kernel+0x1f1/0x3c8 [<ffffffff815cb2c3>] x86_64_start_reservations+0xae/0xb2 [<ffffffff815d1c6c>] xen_start_kernel+0x584/0x593 WARNING: at arch/x86/xen/enlighten.c:453 cvt_gate_to_trap+0x80/0xc0() (Tainted: G W ) Modules linked in: Pid: 1, comm: swapper Tainted: G W 2.6.29.1-15.mjw.x86_64 #1 Call Trace: [<ffffffff81048e84>] warn_slowpath+0xdb/0xfa [<ffffffff8109f2b9>] ? __rmqueue+0x26/0x154 [<ffffffff810abca3>] ? __inc_zone_state+0x50/0x5c [<ffffffff8109ea7d>] ? page_count+0xd/0x1b [<ffffffff810a0b50>] ? get_page_from_freelist+0x500/0x66d [<ffffffff810a1417>] ? __alloc_pages_internal+0xa1/0x398 [<ffffffff810abca3>] ? __inc_zone_state+0x50/0x5c [<ffffffff810ac574>] ? inc_zone_page_state+0x65/0x6f [<ffffffff81365890>] ? nmi+0x0/0x51 [<ffffffff8100aa86>] cvt_gate_to_trap+0x80/0xc0 [<ffffffff8100ab14>] xen_convert_trap_info+0x4e/0x7e [<ffffffff8100ab67>] xen_copy_trap_info+0x23/0x25 [<ffffffff8135b754>] xen_cpu_up+0x167/0x2dd [<ffffffff81360eb8>] _cpu_up+0xa6/0x11c [<ffffffff813637de>] ? _cond_resched+0x9/0x38 [<ffffffff81360fb3>] cpu_up+0x5a/0x6a [<ffffffff815cb8ab>] kernel_init+0xbf/0x229 [<ffffffff8101264a>] child_rip+0xa/0x20 [<ffffffff81011f67>] ? restore_args+0x0/0x30 [<ffffffff8100c5cd>] ? xen_make_pmd+0x0/0xb [<ffffffff8100c5cd>] ? xen_make_pmd+0x0/0xb [<ffffffff81012640>] ? child_rip+0x0/0x20 WARNING: at arch/x86/xen/enlighten.c:453 cvt_gate_to_trap+0x80/0xc0() (Tainted: G W ) Modules linked in: Pid: 0, comm: swapper Tainted: G W 2.6.29.1-15.mjw.x86_64 #1 Call Trace: [<ffffffff81048e84>] warn_slowpath+0xdb/0xfa [<ffffffff8100e61c>] ? __spin_time_accum+0x1b/0x3a [<ffffffff8100e8a1>] ? __xen_spin_lock+0xae/0xc1 [<ffffffff8100e5d9>] ? xen_spin_unlock+0x11/0x2e [<ffffffff810993bc>] ? trace_hardirqs_off+0x9/0x20 [<ffffffff81365246>] ? _spin_unlock_irqrestore+0x27/0x3e [<ffffffff8100e5d9>] ? xen_spin_unlock+0x11/0x2e [<ffffffff81049af9>] ? vprintk+0x313/0x326 [<ffffffff810a1417>] ? __alloc_pages_internal+0xa1/0x398 [<ffffffff8100c4d5>] ? get_phys_to_machine+0x1a/0x31 [<ffffffff810993bc>] ? trace_hardirqs_off+0x9/0x20 [<ffffffff8102ac6b>] ? pvclock_clocksource_read+0x42/0x7b [<ffffffff8102ac6b>] ? pvclock_clocksource_read+0x42/0x7b [<ffffffff81365890>] ? nmi+0x0/0x51 [<ffffffff8100aa86>] cvt_gate_to_trap+0x80/0xc0 [<ffffffff8100ab14>] xen_convert_trap_info+0x4e/0x7e [<ffffffff8100b731>] xen_load_idt+0x47/0x71 [<ffffffff8135d8e2>] cpu_init+0xd5/0x331 [<ffffffff8135b52e>] cpu_bringup+0xe/0x99 [<ffffffff8135b5e6>] cpu_bringup_and_idle+0x9/0x10 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Bryan Donlan
2009-Apr-25 22:10 UTC
[Xen-devel] Re: Bug: ptrace issues under x86_64 Xen kernel 2.6.29
On Sun, Mar 29, 2009 at 11:01 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:> Bryan Donlan wrote: >> >> Using 2.6.29 or 2.6.28 as a 64-bit Xen domU, a number of ptrace() >> users seem to have issues with unexpected breakpoints. ltrace and gdb >> both seem to be affected, under both 64-bit and 32-bit userspace. >> 32-bit kernels do not seem to be affected. Typical symptoms look like: >> > > It looks like this is because the kernel sets up int3 (breakpoint) and debug > (watchpoints, etc) to be on a separate debug stack in the tss. Xen doesn''t > do this (and doesn''t appear to have a mechanism to do so), so I guess the > on-stack format isn''t what the kernel expects. Does the patch below work?Hi, Sorry for the late reply; this message never made it to my inbox for some reason... I did try the patch, and saw similar results to Mark; ptrace works, but lots of warnings: ------------[ cut here ]------------ WARNING: at arch/x86/xen/enlighten.c:447 cvt_gate_to_trap+0xe6/0xf0() Modules linked in: Pid: 0, comm: swapper Tainted: G W 2.6.30-rc3-ptracefix-00330-g6d03473 #4 Call Trace: [<ffffffff8075e9b0>] ? stack_segment+0x0/0x30 [<ffffffff8075e9b0>] ? stack_segment+0x0/0x30 [<ffffffff8023ec4a>] ? warn_slowpath+0xea/0x160 [<ffffffff8020dd79>] ? xen_force_evtchn_callback+0x9/0x10 [<ffffffff8020e512>] ? check_events+0x12/0x20 [<ffffffff8020dd79>] ? xen_force_evtchn_callback+0x9/0x10 [<ffffffff8020e512>] ? check_events+0x12/0x20 [<ffffffff8020e512>] ? check_events+0x12/0x20 [<ffffffff8020e4ff>] ? xen_restore_fl_direct_end+0x0/0x1 [<ffffffff8023f86f>] ? vprintk+0x1df/0x3f0 [<ffffffff8020bb49>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e [<ffffffff8020d928>] ? make_lowmem_page_readonly+0x28/0x40 [<ffffffff8075e9b0>] ? stack_segment+0x0/0x30 [<ffffffff8020a996>] ? cvt_gate_to_trap+0xe6/0xf0 [<ffffffff8020a9f9>] ? xen_convert_trap_info+0x59/0xa0 [<ffffffff8020b0cf>] ? xen_load_idt+0x3f/0x70 [<ffffffff809b3b3f>] ? cpu_init+0xf0/0x2da [<ffffffff809b19a6>] ? cpu_bringup_and_idle+0x6/0x71 ---[ end trace 4eaa2a86a8e2da36 ]--- Thanks, Bryan Donlan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Apr-25 23:19 UTC
[Xen-devel] Re: Bug: ptrace issues under x86_64 Xen kernel 2.6.29
Bryan Donlan wrote:> On Sun, Mar 29, 2009 at 11:01 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote: > >> Bryan Donlan wrote: >> >>> Using 2.6.29 or 2.6.28 as a 64-bit Xen domU, a number of ptrace() >>> users seem to have issues with unexpected breakpoints. ltrace and gdb >>> both seem to be affected, under both 64-bit and 32-bit userspace. >>> 32-bit kernels do not seem to be affected. Typical symptoms look like: >>> >>> >> It looks like this is because the kernel sets up int3 (breakpoint) and debug >> (watchpoints, etc) to be on a separate debug stack in the tss. Xen doesn''t >> do this (and doesn''t appear to have a mechanism to do so), so I guess the >> on-stack format isn''t what the kernel expects. Does the patch below work? >> > > Hi, > > Sorry for the late reply; this message never made it to my inbox for > some reason... > I did try the patch, and saw similar results to Mark; ptrace works, > but lots of warnings: >Thanks. Those warnings are harmless, but I have an updated version of the fix to suppress them in xen-tip/next. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel