Hi, While playing with my xen server (which is running xen-unstable/linux pvops), it suddenly crashed with the following messages on the serial port. This is a recent version of xen-unstable, but i am a few updates behind. I''ve seen this only once, so perhaps it is hard to reproduce. I hope this info is still of use to someone. Regards, Mark. (XEN) tmem: all pools frozen for all domains (XEN) tmem: all pools frozen for all domains (XEN) tmem: all pools thawed for all domains (XEN) tmem: all pools thawed for all domains (XEN) paging.c:170: paging_free_log_dirty_bitmap: used 19 pages for domain 3 dirty logging (XEN) ----[ Xen-4.0.0-rc3-pre x86_64 debug=y Tainted: C ]---- (XEN) CPU: 2 (XEN) RIP: e008:[<ffff82c4801150c5>] free_heap_pages+0x53a/0x555 (XEN) RFLAGS: 0000000000010282 CONTEXT: hypervisor (XEN) rax: ffff82c4803004c0 rbx: ffff82f600ae4b40 rcx: ffff8315ffffffe0 (XEN) rdx: 00000000ffffffff rsi: ffff8315ffffffe0 rdi: ffff82f600000000 (XEN) rbp: ffff83013ff27bc8 rsp: ffff83013ff27b68 r8: 0000000000000000 (XEN) r9: 0200000000000000 r10: 0000000000000001 r11: 0080000000000000 (XEN) r12: ffff82f600ae4b60 r13: 0000000000000000 r14: 00007d0a00000000 (XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: 00000000000026f0 (XEN) cr3: 0000000101001000 cr2: ffff8315ffffffe4 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen stack trace from rsp=ffff83013ff27b68: (XEN) c2c2c2c2c2c2c2c2 0000000000000064 0000000000000000 0000000000000012 (XEN) 0000000000000297 000000000000017a ffff82c48011e1e3 0000000000000000 (XEN) ffff83010fc50000 ffff82f600ae4b60 0000000000069f65 ffff82f600ae4b80 (XEN) ffff83013ff27c18 ffff82c4801153ee 0000000000000001 0000000000000001 (XEN) ffff82f600ae49c8 ffff82f600ae4b60 0000000000800727 ffff83013fef0000 (XEN) ffff82f600ae4b60 ffff83010fc50000 ffff83013ff27c38 ffff82c48015d4d0 (XEN) 000000000000e010 800000005725b727 ffff83013ff27c78 ffff82c48015f8d8 (XEN) 80000000571bf727 ffff8300aae3ac60 ffff83013fef0000 ffff8300aae3b000 (XEN) ffff83013ff27f28 0000000000000000 ffff83013ff27cd8 ffff82c48015eaf4 (XEN) ffff83013ff27d08 ffff82c48015fe3d ffff83013ff27cf8 ffff82c48015d4fe (XEN) ffff83013ff27cc8 1400000000000001 ffff82f60155c740 ffff82f60155c740 (XEN) ffff83013ff27f28 007fffffffffffff ffff83013ff27d28 ffff82c48015f11c (XEN) 000000003fef0000 ffff82f60155c750 ffff83013ff27d38 ffff83013fef0000 (XEN) 0000000000000000 ffffc9000000c2b0 00000000000aae3a ffff83013ff27f28 (XEN) ffff83013ff27d38 ffff82c48015f2f8 ffff83013ff27e38 ffff82c480163a4f (XEN) ffff83013fef0018 00007ff03fef0000 0000000000000000 ffff82c480264db0 (XEN) ffff82c480264db8 ffff83013ff27f28 ffff83013ff27f28 ffff83013fef0218 (XEN) ffff8300bf524000 ffff83013fef0000 ffff8300bf524000 ffff83013fef0000 (XEN) ffff83013fff3da8 0000000100000002 ffff830100000000 ffff82f60155c740 (XEN) 800000008eadf063 ffff880000000001 ffff83013ff27de8 000000003fff3d90 (XEN) Xen call trace: (XEN) [<ffff82c4801150c5>] free_heap_pages+0x53a/0x555 (XEN) [<ffff82c4801153ee>] free_domheap_pages+0x30e/0x3cc (XEN) [<ffff82c48015d4d0>] put_page+0x6c/0x73 (XEN) [<ffff82c48015f8d8>] put_page_from_l1e+0x19f/0x1b5 (XEN) [<ffff82c48015eaf4>] free_page_type+0x25c/0x7b0 (XEN) [<ffff82c48015f11c>] __put_page_type+0xd4/0x292 (XEN) [<ffff82c48015f2f8>] put_page_type+0xe/0x23 (XEN) [<ffff82c480163a4f>] do_mmuext_op+0x6ff/0x14b8 (XEN) [<ffff82c480114235>] do_multicall+0x285/0x410 (XEN) [<ffff82c4801f01bf>] syscall_enter+0xef/0x149 (XEN) (XEN) Pagetable walk from ffff8315ffffffe4: (XEN) L4[0x106] = 00000000bf4f5027 5555555555555555 (XEN) L3[0x057] = 0000000000000000 ffffffffffffffff (XEN) (XEN) **************************************** (XEN) Panic on CPU 2: (XEN) FATAL PAGE FAULT (XEN) [error_code=0002] (XEN) Faulting linear address: ffff8315ffffffe4 (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I''ll have to decode the backtrace a bit, but I would guess most likely is some memory got corrupted along the way, which would be rather nasty. I already need to follow up on a report of apparent memory corruption in a domU userspace (testing with the ''memtester'' utility), so with a bit of luck they could be maifestations of the same bug. -- Keir On 06/02/2010 22:56, "Mark Hurenkamp" <mark.hurenkamp@xs4all.nl> wrote:> Hi, > > > While playing with my xen server (which is running xen-unstable/linux pvops), > it suddenly crashed with the following messages on the serial port. > This is a recent version of xen-unstable, but i am a few updates behind. > I''ve seen this only once, so perhaps it is hard to reproduce. I hope this > info is still of use to someone. > > > Regards, > Mark. > > > (XEN) tmem: all pools frozen for all domains > (XEN) tmem: all pools frozen for all domains > (XEN) tmem: all pools thawed for all domains > (XEN) tmem: all pools thawed for all domains > (XEN) paging.c:170: paging_free_log_dirty_bitmap: used 19 pages for domain 3 > dirty logging > (XEN) ----[ Xen-4.0.0-rc3-pre x86_64 debug=y Tainted: C ]---- > (XEN) CPU: 2 > (XEN) RIP: e008:[<ffff82c4801150c5>] free_heap_pages+0x53a/0x555 > (XEN) RFLAGS: 0000000000010282 CONTEXT: hypervisor > (XEN) rax: ffff82c4803004c0 rbx: ffff82f600ae4b40 rcx: ffff8315ffffffe0 > (XEN) rdx: 00000000ffffffff rsi: ffff8315ffffffe0 rdi: ffff82f600000000 > (XEN) rbp: ffff83013ff27bc8 rsp: ffff83013ff27b68 r8: 0000000000000000 > (XEN) r9: 0200000000000000 r10: 0000000000000001 r11: 0080000000000000 > (XEN) r12: ffff82f600ae4b60 r13: 0000000000000000 r14: 00007d0a00000000 > (XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: 00000000000026f0 > (XEN) cr3: 0000000101001000 cr2: ffff8315ffffffe4 > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 > (XEN) Xen stack trace from rsp=ffff83013ff27b68: > (XEN) c2c2c2c2c2c2c2c2 0000000000000064 0000000000000000 0000000000000012 > (XEN) 0000000000000297 000000000000017a ffff82c48011e1e3 0000000000000000 > (XEN) ffff83010fc50000 ffff82f600ae4b60 0000000000069f65 ffff82f600ae4b80 > (XEN) ffff83013ff27c18 ffff82c4801153ee 0000000000000001 0000000000000001 > (XEN) ffff82f600ae49c8 ffff82f600ae4b60 0000000000800727 ffff83013fef0000 > (XEN) ffff82f600ae4b60 ffff83010fc50000 ffff83013ff27c38 ffff82c48015d4d0 > (XEN) 000000000000e010 800000005725b727 ffff83013ff27c78 ffff82c48015f8d8 > (XEN) 80000000571bf727 ffff8300aae3ac60 ffff83013fef0000 ffff8300aae3b000 > (XEN) ffff83013ff27f28 0000000000000000 ffff83013ff27cd8 ffff82c48015eaf4 > (XEN) ffff83013ff27d08 ffff82c48015fe3d ffff83013ff27cf8 ffff82c48015d4fe > (XEN) ffff83013ff27cc8 1400000000000001 ffff82f60155c740 ffff82f60155c740 > (XEN) ffff83013ff27f28 007fffffffffffff ffff83013ff27d28 ffff82c48015f11c > (XEN) 000000003fef0000 ffff82f60155c750 ffff83013ff27d38 ffff83013fef0000 > (XEN) 0000000000000000 ffffc9000000c2b0 00000000000aae3a ffff83013ff27f28 > (XEN) ffff83013ff27d38 ffff82c48015f2f8 ffff83013ff27e38 ffff82c480163a4f > (XEN) ffff83013fef0018 00007ff03fef0000 0000000000000000 ffff82c480264db0 > (XEN) ffff82c480264db8 ffff83013ff27f28 ffff83013ff27f28 ffff83013fef0218 > (XEN) ffff8300bf524000 ffff83013fef0000 ffff8300bf524000 ffff83013fef0000 > (XEN) ffff83013fff3da8 0000000100000002 ffff830100000000 ffff82f60155c740 > (XEN) 800000008eadf063 ffff880000000001 ffff83013ff27de8 000000003fff3d90 > (XEN) Xen call trace: > (XEN) [<ffff82c4801150c5>] free_heap_pages+0x53a/0x555 > (XEN) [<ffff82c4801153ee>] free_domheap_pages+0x30e/0x3cc > (XEN) [<ffff82c48015d4d0>] put_page+0x6c/0x73 > (XEN) [<ffff82c48015f8d8>] put_page_from_l1e+0x19f/0x1b5 > (XEN) [<ffff82c48015eaf4>] free_page_type+0x25c/0x7b0 > (XEN) [<ffff82c48015f11c>] __put_page_type+0xd4/0x292 > (XEN) [<ffff82c48015f2f8>] put_page_type+0xe/0x23 > (XEN) [<ffff82c480163a4f>] do_mmuext_op+0x6ff/0x14b8 > (XEN) [<ffff82c480114235>] do_multicall+0x285/0x410 > (XEN) [<ffff82c4801f01bf>] syscall_enter+0xef/0x149 > (XEN) > (XEN) Pagetable walk from ffff8315ffffffe4: > (XEN) L4[0x106] = 00000000bf4f5027 5555555555555555 > (XEN) L3[0x057] = 0000000000000000 ffffffffffffffff > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 2: > (XEN) FATAL PAGE FAULT > (XEN) [error_code=0002] > (XEN) Faulting linear address: ffff8315ffffffe4 > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, Feb 07, 2010 at 11:56:26AM +0000, Keir Fraser wrote:> I''ll have to decode the backtrace a bit, but I would guess most likely is > some memory got corrupted along the way, which would be rather nasty. I > already need to follow up on a report of apparent memory corruption in a > domU userspace (testing with the ''memtester'' utility), so with a bit of luck > they could be maifestations of the same bug.I saw a similar crash with 4.0.0 release. The server survived memtest86 for several time and works (mostly) with Xen 3.4. (XEN) ----[ Xen-4.0.0 x86_64 debug=n Tainted: C ]---- (XEN) CPU: 8 (XEN) RIP: e008:[<ffff82c480114e26>] free_heap_pages+0x366/0x4b0 (XEN) RFLAGS: 0000000000010286 CONTEXT: hypervisor (XEN) rax: ffff82c480375070 rbx: ffff82f6143cc240 rcx: ffff8315ffffffe0 (XEN) rdx: ffff8315ffffffe0 rsi: 00000000ffffffff rdi: 0000000000a1e613 (XEN) rbp: 0000000000000000 rsp: ffff830a2fee7cf8 r8: 0000000000000020 (XEN) r9: 0000000000000001 r10: 0200000000000000 r11: 0080000000000000 (XEN) r12: ffff82f6143cc260 r13: 00007d0a00000000 r14: 0180000000000000 (XEN) r15: 00000000000001b5 cr0: 000000008005003b cr4: 00000000000006f0 (XEN) cr3: 0000000a1ff80000 cr2: ffff8315ffffffe4 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen stack trace from rsp=ffff830a2fee7cf8: (XEN) 0000000000000000 ffff82c4803764c0 00000000000001b5 00000000143cc240 (XEN) ffff830840060000 ffff830840060000 ffff82f6143cc240 0000000000000000 (XEN) ffff830840060014 00000000000ffea5 0000000000000000 ffff82c4801150b8 (XEN) ffff830840060000 ffff82f6143cc240 ffff8300cfe2c000 0000000000000000 (XEN) ffff8300cfe2c000 0000000000000001 0010000a1e612067 ffff830840060000 (XEN) ffff83092f69c9c8 ffff82c48016371f ffff830a2fee7f28 ffff82c48015b9f8 (XEN) 0080000000000000 ffff8800ffa55db0 ffff830a2fee7ed0 ffff830840060000 (XEN) 000000000092f69c 000000000092f69c 000000000092f69c 0000000000000000 (XEN) 00000000125ed380 ffff830840060000 ffff82f6125ed380 0000000000000000 (XEN) ffff8300cfe2c000 ffff880002939000 ffff830840060000 ffff82c480166b1c (XEN) 0000000000000001 0000000000000000 ffff83092f69c9c8 ffff830a2fee7f28 (XEN) 0000000000000001 ffff830a2fee7f28 ffff830840060000 0000000000000001 (XEN) ffff8800ffa55dd8 000000000092f69c 0000000000000000 ffff82c480266008 (XEN) ffff8800038c7060 0000000000000000 ffff82c480266000 0000000000000000 (XEN) ffff830a2fee7f28 ffff82c480114129 0000000100000006 ffff830a2fee7f28 (XEN) ffff830a2fee7f28 ffff82c480266010 0000000000000000 ffff8300cfe2c000 (XEN) 0000000000000000 ffff8800010060a0 ffff88000293a000 0000000000000260 (XEN) 0000000000000000 ffff82c4801e7169 0000000000000000 0000000000000260 (XEN) ffff88000293a000 ffff8800010060a0 0000000000000000 ffff8800038c7060 (XEN) 0000000000000212 0000000000000000 ffff8800ffc03080 0000000000000000 (XEN) Xen call trace: (XEN) [<ffff82c480114e26>] free_heap_pages+0x366/0x4b0 (XEN) [<ffff82c4801150b8>] free_domheap_pages+0x148/0x380 (XEN) [<ffff82c48016371f>] mod_l1_entry+0x16f/0x740 (XEN) [<ffff82c48015b9f8>] get_page+0x28/0xf0 (XEN) [<ffff82c480166b1c>] __do_update_va_mapping+0x64c/0x6e0 (XEN) [<ffff82c480114129>] do_multicall+0x189/0x320 (XEN) [<ffff82c4801e7169>] syscall_enter+0xa9/0xae (XEN) (XEN) Pagetable walk from ffff8315ffffffe4: (XEN) L4[0x106] = 00000000cfd69027 5555555555555555 (XEN) L3[0x057] = 0000000000000000 ffffffffffffffff (XEN) (XEN) **************************************** (XEN) Panic on CPU 8: (XEN) FATAL PAGE FAULT (XEN) [error_code=0002] (XEN) Faulting linear address: ffff8315ffffffe4 (XEN) ****************************************> > (XEN) Pagetable walk from ffff8315ffffffe4: > > (XEN) L4[0x106] = 00000000bf4f5027 5555555555555555 > > (XEN) L3[0x057] = 0000000000000000 ffffffffffffffff > > (XEN) > > (XEN) **************************************** > > (XEN) Panic on CPU 2: > > (XEN) FATAL PAGE FAULT > > (XEN) [error_code=0002] > > (XEN) Faulting linear address: ffff8315ffffffe4 > > (XEN) ****************************************The addresses are even identical. Bastian -- I have never understood the female capacity to avoid a direct answer to any question. -- Spock, "This Side of Paradise", stardate 3417.3 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel