My Debian 5 Lenny Dom0 has been running great for a few weeks with two PV DomU''s running Ubuntu 8.04 on an 8-core Intel E405 12Gb machine. Xen 3.2 from Debian''s packages. Both DomU''s are webservers, and one runs qmail as well. The DomU that crashes is not under heavy load. However, twice since moving to this new system, one of my DomU''s has crashed bad and needed to be taken down and restarted. This last time, I captured some of the info in the console. There''s lots more if anyone wants to see it. Does this look familiar to anyone? I''m wondering if the FS (ext3) is implicated, or just an innocent bystander. I''m using LVM storage on a RAID1 done with Linux kernel software RAID. Thanks in advance for any advice. [140027.089210] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 [140027.089216] CR0: 8005003b CR2: b697f090 CR3: 2864e000 CR4: 00002620 [140027.089220] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [140027.089225] DR6: ffff0ff0 DR7: 00000400 [140027.089227] [<c01a6d3d>] try_to_free_buffers+0x2d/0x90 [140027.089235] [<c0167c15>] shrink_page_list+0x4c5/0x600 [140027.089242] [<c0166daf>] isolate_lru_pages+0x5f/0x1c0 [140027.089248] [<c0167e6f>] shrink_inactive_list+0x11f/0x3b0 [140027.089255] [<c016819c>] shrink_zone+0x9c/0x100 [140027.089260] [<c016883c>] kswapd+0x44c/0x490 [140027.089266] [<c013bac0>] autoremove_wake_function+0x0/0x40 [140027.089274] [<c011e260>] complete+0x40/0x60 [140027.089280] [<c01683f0>] kswapd+0x0/0x490 [140027.089285] [<c013b802>] kthread+0x42/0x70 [140027.089289] [<c013b7c0>] kthread+0x0/0x70 [140027.089294] [<c0105bb7>] kernel_thread_helper+0x7/0x10 [140027.089299] ======================[140027.190913] BUG: soft lockup - CPU#0 stuck for 11s! [multilog:4725] [140027.190921] [140027.190925] Pid: 4725, comm: multilog Tainted: G D (2.6.24-23-xen #1) [140027.190928] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 0 [140027.190936] EIP is at _spin_lock+0x7/0x10 [140027.190939] EAX: ed1a96d8 EBX: 00000000 ECX: 00000000 EDX: f578e000 [140027.190943] ESI: 00098002 EDI: 00000000 EBP: 00000000 ESP: eb8d1c78 [140027.190946] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 [140027.190954] CR0: 8005003b CR2: 080521c0 CR3: 2c7a1000 CR4: 00002620 [140027.190959] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [140027.190962] DR6: ffff0ff0 DR7: 00000400 [140027.190966] [<c01a720b>] __find_get_block_slow+0x5b/0x150 [140027.190977] [<ee094cde>] ext3_mark_iloc_dirty+0x1ce/0x350 [ext3] [140027.190991] [<c01a75f6>] __find_get_block+0xd6/0x200 [140027.190998] [<c01a7743>] __getblk+0x23/0x2b0 [140027.191005] [<ee094f50>] __ext3_get_inode_loc+0xf0/0x360 [ext3] [140027.191016] [<c020a5e2>] __next_cpu+0x12/0x20 [140027.191022] [<ee05ba6f>] start_this_handle+0x8f/0x360 [jbd] [140027.191032] [<c011d2d7>] find_busiest_group+0x177/0x550 [140027.191039] [<ee095207>] ext3_reserve_inode_write+0x27/0x80 [ext3] [140027.191050] [<ee095283>] ext3_mark_inode_dirty+0x23/0x50 [ext3] [140027.191060] [<ee05bdc2>] journal_start+0x82/0xb0 [jbd] [140027.191069] [<ee0983e9>] ext3_dirty_inode+0x79/0x90 [ext3] [140027.191081] [<c01a35c0>] __mark_inode_dirty+0x30/0x180 [140027.191087] [<c015ef46>] __generic_file_aio_write_nolock+0x266/0x540 [140027.191095] [<c0119532>] kmap_atomic_prot+0x152/0x190 [140027.191102] [<c015f285>] generic_file_aio_write+0x65/0xe0 [140027.191109] [<ee093690>] ext3_file_write+0x30/0xc0 [ext3] [140027.191119] [<c0185385>] do_sync_write+0xd5/0x120 [140027.191126] [<c013bac0>] autoremove_wake_function+0x0/0x40 [140027.191133] [<c012c002>] __do_softirq+0x92/0x130 [140027.191139] [<c01852b0>] do_sync_write+0x0/0x120 [140027.191144] [<c0185c89>] vfs_write+0xb9/0x170 [140027.191149] [<c01863c1>] sys_write+0x41/0x70 [140027.191154] [<c0105832>] syscall_call+0x7/0xb [140027.191160] ======================[140031.736833] BUG: soft lockup - CPU#3 stuck for 11s! [apache2:10423] [140031.736840] [140031.736842] Pid: 10423, comm: apache2 Tainted: G D (2.6.24-23-xen #1) [140031.736846] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 3 [140031.736851] EIP is at _spin_lock+0x7/0x10 [140031.736853] EAX: ed1a96d8 EBX: ed1a9694 ECX: c1c9cc40 EDX: 00000000 [140031.736856] ESI: c1c9cc40 EDI: 00000000 EBP: e8787dd0 ESP: e8787c34 [140031.736860] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 [140031.736865] CR0: 80050033 CR2: 087b3010 CR3: 14a2e000 CR4: 00002620 [140031.736869] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [140031.736873] DR6: ffff0ff0 DR7: 00000400 [140031.736875] [<c01a6d3d>] try_to_free_buffers+0x2d/0x90 [140031.736884] [<c0167c15>] shrink_page_list+0x4c5/0x600 [140031.736891] [<c0166daf>] isolate_lru_pages+0x5f/0x1c0 [140031.736898] [<c0167e6f>] shrink_inactive_list+0x11f/0x3b0 [140031.736906] [<c016819c>] shrink_zone+0x9c/0x100 [140031.736910] [<c0168ce2>] try_to_free_pages+0x152/0x250 [140031.736916] [<c0162f2b>] __alloc_pages+0x14b/0x390 [140031.736922] [<c017130c>] handle_mm_fault+0x89c/0x1350 [140031.736928] [<c0109030>] timer_interrupt+0x3a0/0x770 [140031.736934] [<c013eaea>] hrtimer_run_queues+0xda/0x1e0 [140031.736939] [<c0173d2b>] vma_adjust+0x10b/0x440 [140031.736944] [<c03298c6>] do_page_fault+0x366/0xe90 [140031.736949] [<c01744d4>] vma_merge+0x144/0x1d0 [140031.736954] [<c0174a75>] do_brk+0x195/0x240 [140031.736961] [<c0175026>] sys_brk+0xb6/0xf0 [140031.736966] [<c0329560>] do_page_fault+0x0/0xe90 [140031.736972] [<c0328205>] error_code+0x35/0x40 [140031.736977] [<c0320000>] vcc_ioctl+0x1e0/0x2d0 [140031.736982] ======================[140035.315737] BUG: soft lockup - CPU#2 stuck for 11s! [apache2:7138] [140035.315745] [140035.315748] Pid: 7138, comm: apache2 Tainted: G D (2.6.24-23-xen #1) [140035.315752] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 2 [140035.315758] EIP is at _spin_lock+0x7/0x10 [140035.315761] EAX: ed1a96d8 EBX: 00000000 ECX: 00000000 EDX: f578e000 [140035.315764] ESI: 000a98d2 EDI: 00000000 EBP: 00000000 ESP: e4f05c80 [140035.315768] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 [140035.315774] CR0: 80050033 CR2: b6eb2000 CR3: 2b8d9000 CR4: 00002620 [140035.315779] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [140035.315783] DR6: ffff0ff0 DR7: 00000400 [140035.315786] [<c01a720b>] __find_get_block_slow+0x5b/0x150 [140035.315795] [<ee0957ee>] ext3_get_blocks_handle+0x9e/0x9e0 [ext3] [140035.315809] [<c01a75f6>] __find_get_block+0xd6/0x200 [140035.315815] [<c01a7743>] __getblk+0x23/0x2b0 [140035.315821] [<c0130370>] run_timer_softirq+0x30/0x220 [140035.315827] [<ee096204>] ext3_getblk+0xd4/0x1d0 [ext3] [140035.315840] [<ee09a994>] ext3_find_entry+0x254/0x650 [ext3] [140035.315853] [<c018cc57>] permission+0x97/0x120 [140035.315858] [<c01dba7c>] security_inode_permission+0x1c/0x20 [140035.315864] [<c018e979>] __link_path_walk+0x79/0xe10 [140035.315870] [<c018187e>] cache_alloc_refill+0x6e/0x570 [140035.315876] [<c020a677>] _atomic_dec_and_lock+0x47/0x70 [140035.315884] [<ee09c51c>] ext3_lookup+0x3c/0x120 [ext3] [140035.315896] [<c0197b04>] d_alloc+0x114/0x1a0 [140035.315902] [<c018d1b5>] __lookup_hash+0xd5/0x120 [140035.315907] [<c01908c4>] open_namei+0x114/0x640 [140035.315912] [<c0107ec5>] local_clock+0x55/0xa0 [140035.315918] [<c0183ab3>] do_filp_open+0x33/0x60 [140035.315923] [<c020a677>] _atomic_dec_and_lock+0x47/0x70 [140035.315930] [<c0183772>] get_unused_fd_flags+0x52/0xd0 [140035.315936] [<c0183b2c>] do_sys_open+0x4c/0xe0 [140035.315942] [<c0183bfc>] sys_open+0x1c/0x20 [140035.315947] [<c0105832>] syscall_call+0x7/0xb [140035.315951] [<c0320000>] vcc_ioctl+0x1e0/0x2d0 [140035.315957] ======================[140038.904832] BUG: soft lockup - CPU#1 stuck for 11s! [kswapd0:130] [140038.904839] [140038.904842] Pid: 130, comm: kswapd0 Tainted: G D (2.6.24-23-xen #1) [140038.904846] EIP: 0061:[<c0327d25>] EFLAGS: 00000286 CPU: 1 [140038.904851] EIP is at _spin_lock+0x5/0x10 [140038.904854] EAX: ed1a96d8 EBX: ed1a9694 ECX: c18566e0 EDX: 00000000 [140038.904857] ESI: c18566e0 EDI: 00000000 EBP: ed7c1f7c ESP: ed7c1dcc [140038.904860] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 [140038.904866] CR0: 8005003b CR2: b697f090 CR3: 2864e000 CR4: 00002620 [140038.904871] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [140038.904875] DR6: ffff0ff0 DR7: 00000400 [140038.904878] [<c01a6d3d>] try_to_free_buffers+0x2d/0x90 [140038.904886] [<c0167c15>] shrink_page_list+0x4c5/0x600 [140038.904893] [<c0166daf>] isolate_lru_pages+0x5f/0x1c0 [140038.904899] [<c0167e6f>] shrink_inactive_list+0x11f/0x3b0 [140038.904906] [<c016819c>] shrink_zone+0x9c/0x100 [140038.904911] [<c016883c>] kswapd+0x44c/0x490 [140038.904917] [<c013bac0>] autoremove_wake_function+0x0/0x40 [140038.904924] [<c011e260>] complete+0x40/0x60 [140038.904929] [<c01683f0>] kswapd+0x0/0x490 [140038.904933] [<c013b802>] kthread+0x42/0x70 [140038.904938] [<c013b7c0>] kthread+0x0/0x70 [140038.904943] [<c0105bb7>] kernel_thread_helper+0x7/0x10 [140038.904948] ======================[140039.006416] BUG: soft lockup - CPU#0 stuck for 11s! [multilog:4725] [140039.006424] [140039.006427] Pid: 4725, comm: multilog Tainted: G D (2.6.24-23-xen #1) [140039.006431] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 0 [140039.006436] EIP is at _spin_lock+0x7/0x10 [140039.006439] EAX: ed1a96d8 EBX: 00000000 ECX: 00000000 EDX: f578e000 [140039.006443] ESI: 00098002 EDI: 00000000 EBP: 00000000 ESP: eb8d1c78 [140039.006446] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 [140039.006451] CR0: 8005003b CR2: 080521c0 CR3: 2c7a1000 CR4: 00002620 [140039.006456] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [140039.006460] DR6: ffff0ff0 DR7: 00000400 [140039.006462] [<c01a720b>] __find_get_block_slow+0x5b/0x150 [140039.006470] [<ee094cde>] ext3_mark_iloc_dirty+0x1ce/0x350 [ext3] [140039.006484] [<c01a75f6>] __find_get_block+0xd6/0x200 [140039.006491] [<c01a7743>] __getblk+0x23/0x2b0 [140039.006498] [<ee094f50>] __ext3_get_inode_loc+0xf0/0x360 [ext3] [140039.006510] [<c020a5e2>] __next_cpu+0x12/0x20 [140039.006516] [<ee05ba6f>] start_this_handle+0x8f/0x360 [jbd] [140039.006526] [<c011d2d7>] find_busiest_group+0x177/0x550 [140039.006532] [<ee095207>] ext3_reserve_inode_write+0x27/0x80 [ext3] [140039.006543] [<ee095283>] ext3_mark_inode_dirty+0x23/0x50 [ext3] [140039.006554] [<ee05bdc2>] journal_start+0x82/0xb0 [jbd] [140039.006563] [<ee0983e9>] ext3_dirty_inode+0x79/0x90 [ext3] [140039.006574] [<c01a35c0>] __mark_inode_dirty+0x30/0x180 [140039.006580] [<c015ef46>] __generic_file_aio_write_nolock+0x266/0x540 [140039.006588] [<c0119532>] kmap_atomic_prot+0x152/0x190 [140039.006596] [<c015f285>] generic_file_aio_write+0x65/0xe0 [140039.006602] [<ee093690>] ext3_file_write+0x30/0xc0 [ext3] [140039.006613] [<c0185385>] do_sync_write+0xd5/0x120 [140039.006620] [<c013bac0>] autoremove_wake_function+0x0/0x40 [140039.006626] [<c012c002>] __do_softirq+0x92/0x130 [140039.006632] [<c01852b0>] do_sync_write+0x0/0x120 [140039.006638] [<c0185c89>] vfs_write+0xb9/0x170 [140039.006642] [<c01863c1>] sys_write+0x41/0x70 [140039.006647] [<c0105832>] syscall_call+0x7/0xb [140039.006653] ======================[140043.552310] BUG: soft lockup - CPU#3 stuck for 11s! [apache2:10423] [140043.552316] [140043.552319] Pid: 10423, comm: apache2 Tainted: G D (2.6.24-23-xen #1) [140043.552322] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 3 [140043.552326] EIP is at _spin_lock+0x7/0x10 [140043.552329] EAX: ed1a96d8 EBX: ed1a9694 ECX: c1c9cc40 EDX: 00000000 [140043.552332] ESI: c1c9cc40 EDI: 00000000 EBP: e8787dd0 ESP: e8787c34 [140043.552335] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 [140043.552341] CR0: 80050033 CR2: 087b3010 CR3: 14a2e000 CR4: 00002620 [140043.552346] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [140043.552349] DR6: ffff0ff0 DR7: 00000400 [140043.552352] [<c01a6d3d>] try_to_free_buffers+0x2d/0x90 [140043.552359] [<c0167c15>] shrink_page_list+0x4c5/0x600 [140043.552366] [<c0166daf>] isolate_lru_pages+0x5f/0x1c0 [140043.552372] [<c0167e6f>] shrink_inactive_list+0x11f/0x3b0 [140043.552380] [<c016819c>] shrink_zone+0x9c/0x100 [140043.552385] [<c0168ce2>] try_to_free_pages+0x152/0x250 [140043.552391] [<c0162f2b>] __alloc_pages+0x14b/0x390 [140043.552398] [<c017130c>] handle_mm_fault+0x89c/0x1350 [140043.552403] [<c0109030>] timer_interrupt+0x3a0/0x770 [140043.552410] [<c013eaea>] hrtimer_run_queues+0xda/0x1e0 [140043.552415] [<c0173d2b>] vma_adjust+0x10b/0x440 [140043.552420] [<c03298c6>] do_page_fault+0x366/0xe90 [140043.552425] [<c01744d4>] vma_merge+0x144/0x1d0 [140043.552430] [<c0174a75>] do_brk+0x195/0x240 [140043.552436] [<c0175026>] sys_brk+0xb6/0xf0 [140043.552440] [<c0329560>] do_page_fault+0x0/0xe90 [140043.552445] [<c0328205>] error_code+0x35/0x40 [140043.552451] [<c0320000>] vcc_ioctl+0x1e0/0x2d0 [140043.552456] ======================[140047.127138] BUG: soft lockup - CPU#2 stuck for 11s! [apache2:7138] [140047.127157] [140047.127162] Pid: 7138, comm: apache2 Tainted: G D (2.6.24-23-xen #1) [140047.127166] EIP: 0061:[<c0327d27>] EFLAGS: 00000286 CPU: 2 [140047.127171] EIP is at _spin_lock+0x7/0x10 [140047.127174] EAX: ed1a96d8 EBX: 00000000 ECX: 00000000 EDX: f578e000 [140047.127177] ESI: 000a98d2 EDI: 00000000 EBP: 00000000 ESP: e4f05c80 [140047.127180] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 [140047.127186] CR0: 80050033 CR2: b6eb2000 CR3: 2b8d9000 CR4: 00002620 [140047.127190] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [140047.127194] DR6: ffff0ff0 DR7: 00000400 [140047.127197] [<c01a720b>] __find_get_block_slow+0x5b/0x150 [140047.127205] [<ee0957ee>] ext3_get_blocks_handle+0x9e/0x9e0 [ext3] [140047.127218] [<c01a75f6>] __find_get_block+0xd6/0x200 [140047.127225] [<c01a7743>] __getblk+0x23/0x2b0 [140047.127230] [<c0130370>] run_timer_softirq+0x30/0x220 [140047.127236] [<ee096204>] ext3_getblk+0xd4/0x1d0 [ext3] [140047.127249] [<ee09a994>] ext3_find_entry+0x254/0x650 [ext3] [140047.127261] [<c018cc57>] permission+0x97/0x120 [140047.127266] [<c01dba7c>] security_inode_permission+0x1c/0x20 [140047.127272] [<c018e979>] __link_path_walk+0x79/0xe10 [140047.127279] [<c018187e>] cache_alloc_refill+0x6e/0x570 [140047.127285] [<c020a677>] _atomic_dec_and_lock+0x47/0x70 [140047.127292] [<ee09c51c>] ext3_lookup+0x3c/0x120 [ext3] [140047.127303] [<c0197b04>] d_alloc+0x114/0x1a0 [140047.127309] [<c018d1b5>] __lookup_hash+0xd5/0x120 [140047.127314] [<c01908c4>] open_namei+0x114/0x640 [140047.127320] [<c0107ec5>] local_clock+0x55/0xa0 [140047.127325] [<c0183ab3>] do_filp_open+0x33/0x60 [140047.127331] [<c020a677>] _atomic_dec_and_lock+0x47/0x70 [140047.127337] [<c0183772>] get_unused_fd_flags+0x52/0xd0 [140047.127343] [<c0183b2c>] do_sys_open+0x4c/0xe0 [140047.127348] [<c0183bfc>] sys_open+0x1c/0x20 [140047.127353] [<c0105832>] syscall_call+0x7/0xb [140047.127358] [<c0320000>] vcc_ioctl+0x1e0/0x2d0 [140047.127363] ======================[140050.719988] BUG: soft lockup - CPU#1 stuck for 11s! [kswapd0:130] [140050.719997] [140050.720001] Pid: 130, comm: kswapd0 Tainted: G D (2.6.24-23-xen #1) [140050.720005] EIP: 0061:[<c0327d2a>] EFLAGS: 00000286 CPU: 1 [140050.720011] EIP is at _spin_lock+0xa/0x10 [140050.720014] EAX: ed1a96d8 EBX: ed1a9694 ECX: c18566e0 EDX: 00000000 [140050.720017] ESI: c18566e0 EDI: 00000000 EBP: ed7c1f7c ESP: ed7c1dcc [140050.720021] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 [140050.720029] CR0: 8005003b CR2: b697f090 CR3: 2864e000 CR4: 00002620 [140050.720033] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [140050.720037] DR6: ffff0ff0 DR7: 00000400 [140050.720041] [<c01a6d3d>] try_to_free_buffers+0x2d/0x90 [140050.720051] [<c0167c15>] shrink_page_list+0x4c5/0x600 [140050.720060] [<c0166daf>] isolate_lru_pages+0x5f/0x1c0 [140050.720065] [<c0167e6f>] shrink_inactive_list+0x11f/0x3b0 [140050.720073] [<c016819c>] shrink_zone+0x9c/0x100 [140050.720077] [<c016883c>] kswapd+0x44c/0x490 [140050.720084] [<c013bac0>] autoremove_wake_function+0x0/0x40 [140050.720090] [<c011e260>] complete+0x40/0x60 [140050.720096] [<c01683f0>] kswapd+0x0/0x490 [140050.720101] [<c013b802>] kthread+0x42/0x70 [140050.720106] [<c013b7c0>] kthread+0x0/0x70 [140050.720112] [<c0105bb7>] kernel_thread_helper+0x7/0x10 [140050.720118] ======================[140050.818054] BUG: soft lockup - CPU#0 stuck for 11s! [multilog:4725] [140050.818060] -- Chris ''Xenon'' Hanson, omo sanza lettere Xenon AlphaPixel.com PixelSense Landsat processing now available! http://www.alphapixel.com/demos/ "There is no Truth. There is only Perception. To Perceive is to Exist." - Xen _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, May 12, 2009 at 7:25 PM, Chris ''Xenon'' Hanson <xenon@alphapixel.com> wrote:> My Debian 5 Lenny Dom0 has been running great for a few weeks with two PV DomU''s running > Ubuntu 8.04 on an 8-core Intel E405 12Gb machine. Xen 3.2 from Debian''s packages. Both > DomU''s are webservers, and one runs qmail as well. The DomU that crashes is not under > heavy load. > > However, twice since moving to this new system, one of my DomU''s has crashed bad and > needed to be taken down and restarted. This last time, I captured some of the info in the > console. There''s lots more if anyone wants to see it.Is your domU also debian? If yes, you should find out if debian has a fix for this. If that failed, you might have better luck using the well-tested 2.6.18 kernel from xen.org. Recently I tried using RHEL5.3 kernel (also 2.6.18), converted using alien on Ubuntu. It seems to run nicely so far. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Chris ''Xenon'' Hanson:> My Debian 5 Lenny Dom0 has been running great for a few weeks with two PV DomU''s running > Ubuntu 8.04 on an 8-core Intel E405 12Gb machine. Xen 3.2 from Debian''s packages. Both > DomU''s are webservers, and one runs qmail as well. The DomU that crashes is not under > heavy load. > > However, twice since moving to this new system, one of my DomU''s has crashed bad and > needed to be taken down and restarted. This last time, I captured some of the info in the > console. There''s lots more if anyone wants to see it. > > Does this look familiar to anyone?Yes, I have seen those "BUG: soft lockup - CPU#0 stuck for 11s!" hangs a couple of times, too.> I''m wondering if the FS (ext3) is implicated, or just > an innocent bystander. I''m using LVM storage on a RAID1 done with Linux kernel software RAID.Same on the machine I had problems with. So, this might indeed be a problematic combination - or simply pure chance.> Thanks in advance for any advice.I have neither found the root of the problem nor a solution. But replacing the domU''s Ubuntu kernel by a self-compiled recent vanilla kernel (2.6.28.8) with domU support is a workaround that lets the domU run stable since 54 days now - previously one week was the most I could get for uptime. -- Regards mks _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, May 12, 2009 at 07:25:14AM -0500, Chris ''Xenon'' Hanson wrote: ...> Does this look familiar to anyone? I''m wondering if the FS (ext3) is implicated, or just > an innocent bystander. I''m using LVM storage on a RAID1 done with Linux kernel software RAID. > > Thanks in advance for any advice.> [140027.190913] BUG: soft lockup - CPU#0 stuck for 11s! [multilog:4725]I saw this same thing after I was messing around with the vcpu-set and -pin settings for my domU''s. Somehow I really managed to muck things up, and on one particular domU whenever I tried to compile software it would choke. I rebooted the dom0 to just simply get things back the way they were, and it worked fine. Aaron -- Aaron L. Meehan - System Administrator Central Oregon Internet, Bend, Oregon Central Oregon PBX http://copbx.com/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Markus Schönhaber wrote:> Yes, I have seen those "BUG: soft lockup - CPU#0 stuck for 11s!" hangs a > couple of times, too. > Same on the machine I had problems with. So, this might indeed be a > problematic combination - or simply pure chance. > I have neither found the root of the problem nor a solution. But > replacing the domU''s Ubuntu kernel by a self-compiled recent vanilla > kernel (2.6.28.8) with domU support is a workaround that lets the domU > run stable since 54 days now - previously one week was the most I could > get for uptime.Hmm. My other DomU (same kernel) hasn''t glitched and it''s under heavier load. Odd. I''ve been reluctant to compile a new kernel because then I''m running something different than the rest of the world and it''s harder to compare configurations. But maybe I''ll look into it. Thanks for the advice. -- Chris ''Xenon'' Hanson, omo sanza lettere Xenon AlphaPixel.com PixelSense Landsat processing now available! http://www.alphapixel.com/demos/ "There is no Truth. There is only Perception. To Perceive is to Exist." - Xen _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2009/5/13 Markus Schönhaber <xen-users@list-post.mks-mail.de>> Chris ''Xenon'' Hanson: > > > My Debian 5 Lenny Dom0 has been running great for a few weeks with two > PV DomU''s running > > Ubuntu 8.04 on an 8-core Intel E405 12Gb machine. Xen 3.2 from Debian''s > packages. Both > > DomU''s are webservers, and one runs qmail as well. The DomU that crashes > is not under > > heavy load. > > > > However, twice since moving to this new system, one of my DomU''s has > crashed bad and > > needed to be taken down and restarted. This last time, I captured some of > the info in the > > console. There''s lots more if anyone wants to see it. > > > > Does this look familiar to anyone? > > Yes, I have seen those "BUG: soft lockup - CPU#0 stuck for 11s!" hangs a > couple of times, too. > > > I''m wondering if the FS (ext3) is implicated, or just > > an innocent bystander. I''m using LVM storage on a RAID1 done with Linux > kernel software RAID. > > Same on the machine I had problems with. So, this might indeed be a > problematic combination - or simply pure chance. > > > Thanks in advance for any advice. > > I have neither found the root of the problem nor a solution. But > replacing the domU''s Ubuntu kernel by a self-compiled recent vanilla > kernel (2.6.28.8) with domU support is a workaround that lets the domU > run stable since 54 days now - previously one week was the most I could > get for uptime. > > Hi Markus,Do you mind posting brief instructions as to how you built the Ubuntu 2.6.28.8 domU kernel? I''m also experiencing similar crashes with the stock Ubuntu 8.04 domU kernel and would like to try your solution :-) Regards _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thiago Camargo Martins Cordeiro
2009-May-14 02:35 UTC
Re: [Xen-users] Ubuntu 8.04 DomU crash
The easy/fast way that I know to stabilize Ubuntu under Xen is: echo "deb http://ftp.br.debian.org/debian lenny main" >> /etc/apt/sources.list aptitude update aptitude install linux-image-2.6.26-2-xen-686 And configure the Debian Linux kernel within your domU... OpenSUSE kernel its great too, but it isn''t a debianified package. Just my two cents... - Thiago 2009/5/13 rcha108 rcha108 <rcha108@aucklanduni.ac.nz>> 2009/5/13 Markus Schönhaber <xen-users@list-post.mks-mail.de> > >> Chris ''Xenon'' Hanson: >> >> > My Debian 5 Lenny Dom0 has been running great for a few weeks with two >> PV DomU''s running >> > Ubuntu 8.04 on an 8-core Intel E405 12Gb machine. Xen 3.2 from Debian''s >> packages. Both >> > DomU''s are webservers, and one runs qmail as well. The DomU that crashes >> is not under >> > heavy load. >> > >> > However, twice since moving to this new system, one of my DomU''s has >> crashed bad and >> > needed to be taken down and restarted. This last time, I captured some >> of the info in the >> > console. There''s lots more if anyone wants to see it. >> > >> > Does this look familiar to anyone? >> >> Yes, I have seen those "BUG: soft lockup - CPU#0 stuck for 11s!" hangs a >> couple of times, too. >> >> > I''m wondering if the FS (ext3) is implicated, or just >> > an innocent bystander. I''m using LVM storage on a RAID1 done with Linux >> kernel software RAID. >> >> Same on the machine I had problems with. So, this might indeed be a >> problematic combination - or simply pure chance. >> >> > Thanks in advance for any advice. >> >> I have neither found the root of the problem nor a solution. But >> replacing the domU''s Ubuntu kernel by a self-compiled recent vanilla >> kernel (2.6.28.8) with domU support is a workaround that lets the domU >> run stable since 54 days now - previously one week was the most I could >> get for uptime. >> >> Hi Markus, > > Do you mind posting brief instructions as to how you built the Ubuntu > 2.6.28.8 domU kernel? I''m also experiencing similar crashes with the stock > Ubuntu 8.04 domU kernel and would like to try your solution :-) > > Regards > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thiago Camargo Martins Cordeiro wrote:> The easy/fast way that I know to stabilize Ubuntu under Xen is: > echo "deb http://ftp.br.debian.org/debian lenny main" >> > /etc/apt/sources.list > aptitude update > aptitude install linux-image-2.6.26-2-xen-686 > And configure the Debian Linux kernel within your domU... > OpenSUSE kernel its great too, but it isn''t a debianified package.I just wanted to follow up and document what I ended up doing, for anyone else who runs into it. I tried the suggestion above, but I became worried when it wanted to update libc and warned about a bunch of other packages it thought should be updated but wasn''t going to update. having been burned by libc mismatches in the past, I decided not to perform the update from within the DomU, and instead pursue manually updating the kernel that Xen uses to launch the DomU (but which is stored within the Dom0). I manually downloaded the kernel and modules packages within the Dom0, and took them apart with ar and tar. I placed the kernel where it would be accessible to xm create in the Dom0, then copied the kernel AND the modules folder over to their proper locations within the DomU. I updated the initramfs with update-initramfs -c, and copied the resulting ramdisk filesystem back to the Dom0 (where xen launches it from). Then, I changed my DomU config file to point to the new kernel and initrd files, and stopped and restarted the DomU. It seems to be working well, and I haven''t had a crash in a few days. As far as why I''m not using pygrub or pvgrub to launch the DomU, I had heard (perhaps erroneously) from other users that this was a security risk -- that storing the kernel where it is potentially writable from within the DomU was a bit of a violation of the virtualization principle, since it is code that the Dom0 sort of runs. I know some Xen VPS ISPs won''t allow you to supply your own kernel for security reasons. This could be a whole different discussion if anyone wanted to reflect on it. Anyway, thanks to all for the assistance, and I hope this info helps others. I wish Ubuntu 8.04 had a Xen-stable kernel right out of the box, as it would save a lot of trouble. Actually, first-class Xen support from later Ubuntu releases would be even better. -- Chris ''Xenon'' Hanson, omo sanza lettere Xenon AlphaPixel.com PixelSense Landsat processing now available! http://www.alphapixel.com/demos/ "There is no Truth. There is only Perception. To Perceive is to Exist." - Xen _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Jun 3, 2009 at 4:44 AM, Chris ''Xenon'' Hanson <xenon@alphapixel.com> wrote:> Thiago Camargo Martins Cordeiro wrote: >> The easy/fast way that I know to stabilize Ubuntu under Xen is: >> echo "deb http://ftp.br.debian.org/debian lenny main" >> >> /etc/apt/sources.list >> aptitude update >> aptitude install linux-image-2.6.26-2-xen-686 >> And configure the Debian Linux kernel within your domU... >> OpenSUSE kernel its great too, but it isn''t a debianified package. > > I just wanted to follow up and document what I ended up doing, for anyone else who runs > into it. > > I tried the suggestion above, but I became worried when it wanted to update libc and > warned about a bunch of other packages it thought should be updated but wasn''t going to > update.Lenny''s kernel fit nicely on Ubuntu 9.04. For older ubuntu version, Etch''s kernel might be a better choice for seamless upgrade.> As far as why I''m not using pygrub or pvgrub to launch the DomU, I had heard (perhaps > erroneously) from other users that this was a security risk -- that storing the kernel > where it is potentially writable from within the DomU was a bit of a violation of the > virtualization principle, since it is code that the Dom0 sort of runs. I know some Xen VPS > ISPs won''t allow you to supply your own kernel for security reasons. This could be a whole > different discussion if anyone wanted to reflect on it.Fair enough, IMHO It''s a mix between security and easy of use. In my case since I control all domUs running on my server, I can trust pygrub enough to use it.> > Anyway, thanks to all for the assistance, and I hope this info helps others. I wish > Ubuntu 8.04 had a Xen-stable kernel right out of the box, as it would save a lot of > trouble. Actually, first-class Xen support from later Ubuntu releases would be even better.Same wish here. Not holding my breath though. Support (including for Xen) is the primary reason I use RHEL5. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha wrote:> Lenny''s kernel fit nicely on Ubuntu 9.04. > For older ubuntu version, Etch''s kernel might be a better choice for > seamless upgrade.I was gun-shy of taking the Dom0''s Lenny kernel because it was a x64 and the DomU was an x86-32, and I didn''t want to open that can of worms.> Fair enough, > IMHO It''s a mix between security and easy of use. In my case since I > control all domUs running on my server, I can trust pygrub enough to > use it.I can see both sides of the issue. I control the DomU too, at least, as long as there isn''t a root exploit within it. ;) It''s a webserver, so it''s obviously more exposed than the Dom0.>> Anyway, thanks to all for the assistance, and I hope this info helps others. I wish >> Ubuntu 8.04 had a Xen-stable kernel right out of the box, as it would save a lot of >> trouble. Actually, first-class Xen support from later Ubuntu releases would be even better. > Same wish here. Not holding my breath though. > Support (including for Xen) is the primary reason I use RHEL5.Sigh. -- Chris ''Xenon'' Hanson, omo sanza lettere Xenon AlphaPixel.com PixelSense Landsat processing now available! http://www.alphapixel.com/demos/ "There is no Truth. There is only Perception. To Perceive is to Exist." - Xen _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users