Hi Jun, here is the x86-64 VT crash backtrace, as promised. I can trigger it within minutes by simply starting up a 64 bit VT domain on a 64 bit system. (XEN) ----[ Xen-3.0-unstable Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e010:[<ffff830000153cc8>] __cpus_subset+0x0/0x25 (XEN) RFLAGS: 0000000000010082 CONTEXT: hypervisor (XEN) rax: 0000000000000001 rbx: ffff830000ff7068 rcx: 0000000000000020 (XEN) rdx: 0000000000000020 rsi: ffff83000024db08 rdi: ffff830000ff7010 (XEN) rbp: ffff830000ff7018 rsp: ffff830000ff7000 r8: 0000000000000000 (XEN) r9: 0000010010bc7bc8 r10: 0000000000000004 r11: 00000000006bc6b0 (XEN) r12: ffffffff804bc220 r13: 00000100108e8230 r14: ffffffff804bc1a0 (XEN) r15: 00000004a0c7d6d9 cr0: 000000008005003b cr3: 0000000041137000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e010 (XEN) Xen stack trace from rsp=ffff830000ff7000: (XEN) ffff830000153c20 0000000000000000 0000000000000001 ffff830000ff7058 (XEN) ffff830000153ba8 000000fc00ff7068 0000000000000001 ffff830000ff7058 (XEN) 0000000000000001 0000002000ff7068 ffff830000ff7068 ffff830000ff7078 (XEN) ffff830000154489 0000000000000001 000000000001053d ffff830000ff70a8 (XEN) ffff83000011da63 0000000100ff70a8 0000000000000001 0000002000ff70c8 (XEN) 0000002000ff70c8 ffff830000ff70d8 ffff83000011d81e ffff830000268d80 (XEN) 0000000000ffee80 0000000000000001 ffff830000ffed80 ffff830000ff7108 (XEN) ffff83000011d516 000000125d845e14 ffff830000fee080 ffff830000268d80 (XEN) 000000000001053d ffff830000ff7138 ffff830000123e40 ffff830000fee138 (XEN) ffff830000fee080 0000000000000000 0000000000000086 ffff830000ff7158 (XEN) ffff83000010c60b ffff830000fee138 ffff830000fee080 ffff830000ff7188 (XEN) ffff83000010c595 ffff83000020b000 ffff830000fee080 0000001000000000 (XEN) 0000000000000000 ffff830000ff71a8 ffff83000010c566 0000000000000000 (XEN) ffff830000fee080 ffff830000ff71d8 ffff83000010c471 0000000f00000000 (XEN) ffff830000fee080 ffff830000268080 ffff83000020b000 ffff830000ff7208 (XEN) ffff83000010c85e 0000001500268408 ffff830000268080 0000000f00208c88 (XEN) ffff8300002031e8 ffff830000ff7258 ffff830000141399 ffff830000ff7238 (XEN) 000000c00012d383 0000001500ff7258 ffff83000023cf80 ffff830000208c80 (XEN) ffff830000268080 000000000a4011c4 0000000100000000 ffff830000ff7298 (XEN) ffff830000140cbf fdb3830000f77020 ffff830000ff72a8 0000000000000004 (XEN) Xen call trace: (XEN) [<ffff830000153cc8>] __cpus_subset+0x0/0x25 (XEN) [<ffff830000153ba8>] send_IPI_mask_flat+0x20/0x77 (XEN) [<ffff830000154489>] smp_send_event_check_mask+0x51/0x58 (XEN) [<ffff83000011da63>] cpumask_raise_softirq+0x74/0x76 (XEN) [<ffff83000011d81e>] __runq_tickle+0x192/0x194 (XEN) [<ffff83000011d516>] csched_vcpu_wake+0xf5/0xf7 (XEN) [<ffff830000123e40>] vcpu_wake+0x68/0xf9 (XEN) [<ffff83000010c60b>] vcpu_unblock+0x2e/0x30 (XEN) [<ffff83000010c595>] vcpu_kick+0x2d/0x51 (XEN) [<ffff83000010c566>] vcpu_mark_events_pending+0x2e/0x30 (XEN) [<ffff83000010c471>] evtchn_set_pending+0x87/0x107 (XEN) [<ffff83000010c85e>] send_guest_pirq+0xce/0xd0 (XEN) [<ffff830000141399>] __do_IRQ_guest+0x2f6/0x30b (XEN) [<ffff830000140cbf>] do_IRQ+0x65/0x1bf (XEN) [<ffff83000013c196>] common_interrupt+0x26/0x30 (XEN) [<ffff830000117cd6>] free_heap_pages+0x330/0x3b0 (XEN) [<ffff8300001198e3>] free_domheap_pages+0x54f/0x57b (XEN) [<ffff83000016e9bb>] free_shadow_page+0x3d3/0x3da (XEN) [<ffff83000016e23f>] put_shadow_ref+0x16f/0x171 (XEN) [<ffff83000016e0b2>] shadow_free_snapshot+0x2e3/0x2e5 (XEN) [<ffff83000016e31a>] release_out_of_sync_entry+0xd9/0xdb (XEN) [<ffff83000016e4fc>] remove_out_of_sync_entries+0xac/0xf0 (XEN) [<ffff83000016f41e>] shadow_demote+0x120/0x122 (XEN) [<ffff83000016e80d>] free_shadow_page+0x225/0x3da (XEN) [<ffff83000016e23f>] put_shadow_ref+0x16f/0x171 (XEN) [<ffff83000016f7b6>] free_shadow_tables+0x1c3/0x294 (XEN) [<ffff83000016e85e>] free_shadow_page+0x276/0x3da (XEN) [<ffff83000016e23f>] put_shadow_ref+0x16f/0x171 (XEN) [<ffff83000016f7b6>] free_shadow_tables+0x1c3/0x294 (XEN) [<ffff83000016e85e>] free_shadow_page+0x276/0x3da (XEN) [<ffff830000167346>] put_shadow_ref+0x16f/0x171 (XEN) [<ffff830000167466>] validate_entry_change+0x11e/0x130 (XEN) [<ffff8300001683b0>] resync_all+0xa56/0xc47 (XEN) [<ffff830000168839>] resync_all_levels_guest_page+0x9a/0xa2 (XEN) [<ffff830000168a1a>] sync_all+0x1d9/0x218 (XEN) [<ffff83000016d8d1>] __shadow_sync_all+0x21/0x23 (XEN) [<ffff8300001b276b>] shadow_sync_all+0xa3/0x1b5 (XEN) [<ffff8300001b406e>] mov_to_cr+0x4e7/0xb70 (XEN) [<ffff8300001b4c1b>] vmx_cr_access+0x114/0x4d5 (XEN) [<ffff8300001b5ea3>] vmx_vmexit_handler+0xbd3/0x1101 (XEN) (XEN) ************************************ (XEN) CPU1 FATAL TRAP 8 (double fault), ERROR_CODE 0000, IN INTERRUPT CONTEXT. (XEN) System shutting down -- need manual reset. (XEN) ************************************ -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Rik van Riel wrote:> here is the x86-64 VT crash backtrace, as promised. I can trigger it > within minutes by simply starting up a 64 bit VT domain on a 64 bit system.And another, more creative one. It seems to have bitmap_subset in common with the first backtrace, and the assertion looks like it could be a useful hint: (XEN) (file=extable.c, line=77) Pre-exception: ffff830000153d1d -> 0000000000000000 (XEN) ----[ Xen-3.0-unstable Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e010:[<ffff830000153d1d>] bitmap_subset+0x30/0x9d (XEN) RFLAGS: 0000000000010046 CONTEXT: hypervisor (XEN) rax: fffffffffffffffc rbx: ffff8300002230b8 rcx: 0000000000000000 (XEN) rdx: 0000000000000002 rsi: ffff83000024db08 rdi: ffff830000223060 (XEN) rbp: ffff830000223018 rsp: ffff830000222fe8 r8: 0000000000000000 (XEN) r9: 70202020200a0a3b r10: 542e2220746e6972 r11: 6d202020200a3b29 (XEN) r12: ffffffff804bc220 r13: 000001000d3f4e20 r14: ffffffff804bc1a0 (XEN) r15: 000000629838fcd8 cr0: 000000008005003b cr3: 0000000009137000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e010 (XEN) Xen stack trace from rsp=ffff830000222fe8: (XEN) (file=extable.c, line=77) Pre-exception: ffff83000015773e -> 0000000000000000 (XEN) Assertion ''diff < STACK_SIZE'' failed, line 38, file traps.c (XEN) BUG at traps.c:38 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bca48 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bc820 -> 0000000000000000 (XEN) Assertion ''diff < STACK_SIZE'' failed, line 38, file traps.c (XEN) BUG at traps.c:38 (XEN) (file=extable.c, line=77) Pre-exception: ffff8300001bca48 -> 0000000000000000 (XEN) ----[ Xen-3.0-unstable Not tainted ]---- (XEN) CPU: 2196608 (XEN) RIP: e010:[<ffff8300001bca48>] show_registers+0x264/0x752 (XEN) RFLAGS: 0000000000010086 CONTEXT: hypervisor (XEN) rax: ffff8300001e70e4 rbx: ffff8300002230b8 rcx: 00000000000032dc (XEN) rdx: 0000000000000000 rsi: 000000000000000a rdi: ffff8300001e70e4 (XEN) rbp: ffff830000218098 rsp: ffff830000217f08 r8: 00000000ffffffff (XEN) r9: 00000000ffffffff r10: ffff83000022ed3f r11: ffff83000022e94f (XEN) r12: ffffffff804bc220 r13: 000001000d3f4e20 r14: ffffffff804bc1a0 (XEN) r15: 000000629838fcd8 cr0: 000000008005003b cr3: 0000000009137000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e010 (XEN) Xen stack trace from rsp=ffff830000217f08: (XEN) 0000000a00000000 ffff8300001e71a0 0000000000000000 ffff830000218108 (XEN) ffff830000217f58 ffff83000012cd48 0000000a00000000 ffff8300001e71a0 (XEN) ffff830000217fe8 ffff8300001e7214 ffff830000217f98 0000000000000082 (XEN) ffff8300001d0a1a 000000080000004d 0000000a00000009 ffff8300001e71a0 (XEN) 000000629838fcd8 ffffffff804bc1a0 000001000d3f4e20 ffffffff804bc220 (XEN) ffff830000218348 ffff8300002230b8 ffff83000022e956 ffff83000022e97d (XEN) 00000000ffffffff 0000000000000010 0000000a00000009 0000000000000000 (XEN) 00000000000000c8 ffff830000218480 ffff830000218300 (XEN) Xen call trace: (XEN) [<ffff8300001bca48>] show_registers+0x264/0x752 (XEN) (XEN) ************************************ (XEN) CPU2196608 FATAL TRAP 6 (invalid opcode), ERROR_CODE 0000, IN INTERRUPT CONTEXT. (XEN) System shutting down -- need manual reset. (XEN) ************************************ -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 25 Jul 2006, at 17:02, Rik van Riel wrote:> Hi Jun, > > here is the x86-64 VT crash backtrace, as promised. I can trigger it > within minutes by simply starting up a 64 bit VT domain on a 64 bit > system.The hideously long (but apparently valid) call sequence has overflowed the 4kB Xen stack. Given that stacks are only per-cpu on x86 Xen, I should probably just make them bigger (perhaps 8kB on debug builds, 16kB on non-debug builds). 64-bit non-debug builds already have (almost) 8kB stacks, so shouldn''t see this crash. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> > On 25 Jul 2006, at 17:02, Rik van Riel wrote: > >> Hi Jun, >> >> here is the x86-64 VT crash backtrace, as promised. I can trigger it >> within minutes by simply starting up a 64 bit VT domain on a 64 bit >> system. > > The hideously long (but apparently valid) call sequence has overflowed > the 4kB Xen stack. Given that stacks are only per-cpu on x86 Xen, I > should probably just make them bigger (perhaps 8kB on debug builds, 16kB > on non-debug builds). 64-bit non-debug builds already have (almost) 8kB > stacks, so shouldn''t see this crash.These crashes are on a 64 bit system. So yes, even with 8kB stacks it crashes. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 25 Jul 2006, at 17:55, Rik van Riel wrote:>> here is the x86-64 VT crash backtrace, as promised. I can trigger it >> within minutes by simply starting up a 64 bit VT domain on a 64 bit >> system. > > And another, more creative one. It seems to have bitmap_subset > in common with the first backtrace, and the assertion looks like > it could be a useful hint:Looks like another stack overflow. I think maybe the double-fault handling is broken and that caused the assertion. What xen-unstable changeset are you running on? -- Keir> (XEN) (file=extable.c, line=77) Pre-exception: ffff830000153d1d -> > 0000000000000000 > (XEN) ----[ Xen-3.0-unstable Not tainted ]---- > (XEN) CPU: 0 > (XEN) RIP: e010:[<ffff830000153d1d>] bitmap_subset+0x30/0x9d > (XEN) RFLAGS: 0000000000010046 CONTEXT: hypervisor > (XEN) rax: fffffffffffffffc rbx: ffff8300002230b8 rcx: > 0000000000000000 > (XEN) rdx: 0000000000000002 rsi: ffff83000024db08 rdi: > ffff830000223060 > (XEN) rbp: ffff830000223018 rsp: ffff830000222fe8 r8: > 0000000000000000_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 25 Jul 2006, at 18:09, Rik van Riel wrote:>> The hideously long (but apparently valid) call sequence has >> overflowed the 4kB Xen stack. Given that stacks are only per-cpu on >> x86 Xen, I should probably just make them bigger (perhaps 8kB on >> debug builds, 16kB on non-debug builds). 64-bit non-debug builds >> already have (almost) 8kB stacks, so shouldn''t see this crash. > > These crashes are on a 64 bit system. > > So yes, even with 8kB stacks it crashes.But you''re running a debug build, so you have a 4kB stack. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> But you''re running a debug build, so you have a 4kB stack.That seems wrong. The debug build enables extra code, so if anything it would need more stack space than the standard build. Simply running the extra code in the same stack space should be enough... Does the attached patch look remotely sane? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel