Christopher S. Aker
2009-Jan-07 16:03 UTC
[Xen-devel] 2.6.28 domU panics in blk_invoke_request_fn()
Below are two similar panics we''ve captured thus far with some light testing of 2.6.28 domUs across our Xen cluster. The kernel binary and associated files can be found here: http://theshore.net/~caker/xen/BUGS/2.6.28/ ------------[ cut here ]------------ Kernel BUG at c03eed00 [verbose debug info unavailable] invalid opcode: 0000 [#1] SMP last sysfs file: Modules linked in: Pid: 0, comm: swapper Not tainted (2.6.28-linode15 #1) EIP: 0061:[<c03eed00>] EFLAGS: 00010046 CPU: 0 EIP is at do_blkif_request+0x2e0/0x360 EAX: 00000001 EBX: 00000000 ECX: d51650c0 EDX: d529c3f0 ESI: d5984288 EDI: d59842c8 EBP: 000003b8 ESP: c06cde20 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 Process swapper (pid: 0, ti=c06cc000 task=c0671340 task.ti=c06cc000) Stack: 00000005 d5984288 00000288 d5874ae0 d5940000 d53bf3c4 00000000 00000001 d5940000 00000002 00000006 d5984000 00000000 d51650c0 d529c3cc ffffffff d5874ae0 d5940000 00000003 00000001 c03a7165 d5940000 c03eed96 00000000 Call Trace: [<c03a7165>] blk_invoke_request_fn+0x95/0x100 [<c03eed96>] kick_pending_request_queues+0x16/0x30 [<c03eef3d>] blkif_interrupt+0x18d/0x1d0 [<c0159500>] handle_IRQ_event+0x30/0x60 [<c015b418>] handle_level_irq+0x78/0xf0 [<c010ab17>] do_IRQ+0x77/0x90 [<c0105d0a>] check_events+0x8/0xe [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150 [<c01091c7>] xen_do_upcall+0x7/0xc [<c01013a7>] _stext+0x3a7/0x1000 [<c010547f>] xen_safe_halt+0xf/0x20 [<c0103c50>] xen_idle+0x20/0x40 [<c0106d2f>] cpu_idle+0x5f/0xb0 Code: 2c 8d 54 03 40 8d 44 0e 54 b9 6c 00 00 00 e8 68 a5 fc ff 8b 44 24 3c e8 cf 92 fd ff 83 44 24 18 01 e9 40 fd ff ff 0f 0b eb fe 90 <0f> 0b eb fe 8b 44 24 20 ba 10 ea 3e c0 8b 4c 24 20 c7 04 24 0b EIP: [<c03eed00>] do_blkif_request+0x2e0/0x360 SS:ESP 0069:c06cde20 Kernel panic - not syncing: Fatal exception in interrupt ------------[ cut here ]------------ WARNING: at kernel/smp.c:333 smp_call_function_mask+0x1cb/0x1d0() Modules linked in: Pid: 0, comm: swapper Tainted: G D 2.6.28-linode15 #1 Call Trace: [<c0128aaf>] warn_on_slowpath+0x5f/0x90 [<c03b92c6>] memmove+0x36/0x40 [<c03dd10a>] scrup+0x7a/0xe0 [<c0140967>] atomic_notifier_call_chain+0x17/0x20 [<c03dd18f>] notify_update+0x1f/0x30 [<c03dd41a>] vt_console_print+0x20a/0x2d0 [<c0105427>] xen_force_evtchn_callback+0x17/0x30 [<c0105d0a>] check_events+0x8/0xe [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1 [<c0105427>] xen_force_evtchn_callback+0x17/0x30 [<c0105d0a>] check_events+0x8/0xe [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1 [<c01295b0>] vprintk+0x170/0x350 [<c014a45b>] smp_call_function_mask+0x1cb/0x1d0 [<c0106000>] stop_self+0x0/0x30 [<c0105427>] xen_force_evtchn_callback+0x17/0x30 [<c0105d0a>] check_events+0x8/0xe [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 [<c03df146>] do_unblank_screen+0x16/0x130 [<c014a474>] smp_call_function+0x14/0x20 [<c0128b3e>] panic+0x4e/0x100 [<c010ac6c>] oops_end+0x8c/0xa0 [<c0109b80>] do_invalid_op+0x0/0xa0 [<c0109bff>] do_invalid_op+0x7f/0xa0 [<c03eed00>] do_blkif_request+0x2e0/0x360 [<c011689e>] pvclock_clocksource_read+0x4e/0xe0 [<c01059b3>] get_abs_timeout+0x13/0x30 [<c0105427>] xen_force_evtchn_callback+0x17/0x30 [<c0105d0a>] check_events+0x8/0xe [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 [<c056245a>] error_code+0x72/0x78 [<c03eed00>] do_blkif_request+0x2e0/0x360 [<c03a7165>] blk_invoke_request_fn+0x95/0x100 [<c03eed96>] kick_pending_request_queues+0x16/0x30 [<c03eef3d>] blkif_interrupt+0x18d/0x1d0 [<c0159500>] handle_IRQ_event+0x30/0x60 [<c015b418>] handle_level_irq+0x78/0xf0 [<c010ab17>] do_IRQ+0x77/0x90 [<c0105d0a>] check_events+0x8/0xe [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150 [<c01091c7>] xen_do_upcall+0x7/0xc [<c01013a7>] _stext+0x3a7/0x1000 [<c010547f>] xen_safe_halt+0xf/0x20 [<c0103c50>] xen_idle+0x20/0x40 [<c0106d2f>] cpu_idle+0x5f/0xb0 ---[ end trace d5f11e988eae6396 ]--- And this one is from another user on a different host: ------------[ cut here ]------------ Kernel BUG at c03eed00 [verbose debug info unavailable] invalid opcode: 0000 [#1] SMP last sysfs file: /sys/class/net/tun1/type Modules linked in: dahdi_dummy dahdi Pid: 157, comm: kswapd0 Not tainted (2.6.28-linode15 #1) EIP: 0061:[<c03eed00>] EFLAGS: 00010046 CPU: 0 EIP is at do_blkif_request+0x2e0/0x360 EAX: 00000001 EBX: 00000000 ECX: c1bc7240 EDX: d6132d20 ESI: d5966000 EDI: d5966040 EBP: 00000165 ESP: d61e1b90 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 Process kswapd0 (pid: 157, ti=d61e0000 task=d6106ef0 task.ti=d61e0000) Stack: 00000005 d5966000 00000000 d61da9e0 d596e000 d5488d0c 00000000 00000018 d596e000 00000002 00000000 d5966000 00000000 c1bc7240 d613294c ffffffff d61da9e0 d596e000 00000006 00000018 c03a7165 d596e000 c03eed96 00000000 Call Trace: [<c03a7165>] blk_invoke_request_fn+0x95/0x100 [<c03eed96>] kick_pending_request_queues+0x16/0x30 [<c03eef3d>] blkif_interrupt+0x18d/0x1d0 [<c03a514d>] elv_next_request+0x1d/0x170 [<c0159500>] handle_IRQ_event+0x30/0x60 [<c015b418>] handle_level_irq+0x78/0xf0 [<c010ab17>] do_IRQ+0x77/0x90 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150 [<c01091c7>] xen_do_upcall+0x7/0xc [<c0101227>] _stext+0x227/0x1000 [<c0105427>] xen_force_evtchn_callback+0x17/0x30 [<c0182687>] kmem_cache_alloc+0x57/0xb0 [<c016140d>] mempool_alloc+0x2d/0xe0 [<c016140d>] mempool_alloc+0x2d/0xe0 [<c01774dc>] try_to_unmap_one+0x8c/0x240 [<c01a737b>] bvec_alloc_bs+0x7b/0x140 [<c01a7641>] bio_alloc_bioset+0x51/0xe0 [<c01a773b>] bio_alloc+0xb/0x20 [<c017ae20>] get_swap_bio+0x20/0xa0 [<c017afdb>] swap_writepage+0x4b/0xc0 [<c017b050>] end_swap_bio_write+0x0/0x80 [<c01692cd>] shrink_page_list+0x51d/0x650 [<c016550f>] determine_dirtyable_memory+0x1f/0x90 [<c0165598>] get_dirty_limits+0x18/0x2e0 [<c0169837>] shrink_zone+0x437/0x860 [<c0142117>] getnstimeofday+0x37/0xe0 [<c0562025>] _spin_lock+0x5/0x10 [<c0280215>] nfs_access_cache_shrinker+0xa5/0x1f0 [<c01b9e59>] mb_cache_shrink_fn+0x59/0x100 [<c016a7f8>] kswapd+0x4e8/0x500 [<c0168220>] isolate_pages_global+0x0/0x220 [<c013c3c0>] autoremove_wake_function+0x0/0x40 [<c011ec00>] complete+0x40/0x60 [<c016a310>] kswapd+0x0/0x500 [<c013c0a2>] kthread+0x42/0x70 [<c013c060>] kthread+0x0/0x70 [<c0109177>] kernel_thread_helper+0x7/0x10 Code: 2c 8d 54 03 40 8d 44 0e 54 b9 6c 00 00 00 e8 68 a5 fc ff 8b 44 24 3c e8 cf 92 fd ff 83 44 24 18 01 e9 40 fd ff ff 0f 0b eb fe 90 <0f> 0b eb fe 8b 44 24 20 ba 10 ea 3e c0 8b 4c 24 20 c7 04 24 0b EIP: [<c03eed00>] do_blkif_request+0x2e0/0x360 SS:ESP 0069:d61e1b90 Kernel panic - not syncing: Fatal exception in interrupt ------------[ cut here ]------------ WARNING: at kernel/smp.c:333 smp_call_function_mask+0x1cb/0x1d0() Modules linked in: dahdi_dummy dahdi Pid: 157, comm: kswapd0 Tainted: G D 2.6.28-linode15 #1 Call Trace: [<c0128aaf>] warn_on_slowpath+0x5f/0x90 [<c03b92c6>] memmove+0x36/0x40 [<c03dd10a>] scrup+0x7a/0xe0 [<c0140967>] atomic_notifier_call_chain+0x17/0x20 [<c03dd18f>] notify_update+0x1f/0x30 [<c03dd41a>] vt_console_print+0x20a/0x2d0 [<c0562183>] _spin_lock_irqsave+0x33/0x50 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 [<c01291dc>] release_console_sem+0x19c/0x1e0 [<c01295b0>] vprintk+0x170/0x350 [<c014a45b>] smp_call_function_mask+0x1cb/0x1d0 [<c0106000>] stop_self+0x0/0x30 [<c0562183>] _spin_lock_irqsave+0x33/0x50 [<c01324a7>] lock_timer_base+0x27/0x60 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 [<c03df146>] do_unblank_screen+0x16/0x130 [<c014a474>] smp_call_function+0x14/0x20 [<c0128b3e>] panic+0x4e/0x100 [<c010ac6c>] oops_end+0x8c/0xa0 [<c0109b80>] do_invalid_op+0x0/0xa0 [<c0109bff>] do_invalid_op+0x7f/0xa0 [<c03eed00>] do_blkif_request+0x2e0/0x360 [<c01055d5>] get_runstate_snapshot+0x75/0x90 [<c0105a2f>] xen_sched_clock+0x1f/0x80 [<c014135b>] __update_sched_clock+0x2b/0x140 [<c011689e>] pvclock_clocksource_read+0x4e/0xe0 [<c0562183>] _spin_lock_irqsave+0x33/0x50 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 [<c056245a>] error_code+0x72/0x78 [<c03eed00>] do_blkif_request+0x2e0/0x360 [<c03a7165>] blk_invoke_request_fn+0x95/0x100 [<c03eed96>] kick_pending_request_queues+0x16/0x30 [<c03eef3d>] blkif_interrupt+0x18d/0x1d0 [<c03a514d>] elv_next_request+0x1d/0x170 [<c0159500>] handle_IRQ_event+0x30/0x60 [<c015b418>] handle_level_irq+0x78/0xf0 [<c010ab17>] do_IRQ+0x77/0x90 [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150 [<c01091c7>] xen_do_upcall+0x7/0xc [<c0101227>] _stext+0x227/0x1000 [<c0105427>] xen_force_evtchn_callback+0x17/0x30 [<c0182687>] kmem_cache_alloc+0x57/0xb0 [<c016140d>] mempool_alloc+0x2d/0xe0 [<c016140d>] mempool_alloc+0x2d/0xe0 [<c01774dc>] try_to_unmap_one+0x8c/0x240 [<c01a737b>] bvec_alloc_bs+0x7b/0x140 [<c01a7641>] bio_alloc_bioset+0x51/0xe0 [<c01a773b>] bio_alloc+0xb/0x20 [<c017ae20>] get_swap_bio+0x20/0xa0 [<c017afdb>] swap_writepage+0x4b/0xc0 [<c017b050>] end_swap_bio_write+0x0/0x80 [<c01692cd>] shrink_page_list+0x51d/0x650 [<c016550f>] determine_dirtyable_memory+0x1f/0x90 [<c0165598>] get_dirty_limits+0x18/0x2e0 [<c0169837>] shrink_zone+0x437/0x860 [<c0142117>] getnstimeofday+0x37/0xe0 [<c0562025>] _spin_lock+0x5/0x10 [<c0280215>] nfs_access_cache_shrinker+0xa5/0x1f0 [<c01b9e59>] mb_cache_shrink_fn+0x59/0x100 [<c016a7f8>] kswapd+0x4e8/0x500 [<c0168220>] isolate_pages_global+0x0/0x220 [<c013c3c0>] autoremove_wake_function+0x0/0x40 [<c011ec00>] complete+0x40/0x60 [<c016a310>] kswapd+0x0/0x500 [<c013c0a2>] kthread+0x42/0x70 [<c013c060>] kthread+0x0/0x70 [<c0109177>] kernel_thread_helper+0x7/0x10 ---[ end trace 505b292c57e00b05 ]--- -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Jan-13 23:43 UTC
[Xen-devel] Re: 2.6.28 domU panics in blk_invoke_request_fn()
Christopher S. Aker wrote:> Below are two similar panics we''ve captured thus far with some light > testing of 2.6.28 domUs across our Xen cluster. The kernel binary and > associated files can be found here:How frequently does this occur? Did it just started appearing on 2.6.28, but .27.x is OK? Does it happen often enough that you could practically bisect? Could you compile the kernel with CONFIG_BUG_VERBOSE enabled so that it at prints some source/line info? The location of the BUG isn''t obvious. What''s your dom0? J> > http://theshore.net/~caker/xen/BUGS/2.6.28/ > > ------------[ cut here ]------------ > Kernel BUG at c03eed00 [verbose debug info unavailable] > invalid opcode: 0000 [#1] SMP > last sysfs file: > Modules linked in: > > Pid: 0, comm: swapper Not tainted (2.6.28-linode15 #1) > EIP: 0061:[<c03eed00>] EFLAGS: 00010046 CPU: 0 > EIP is at do_blkif_request+0x2e0/0x360 > EAX: 00000001 EBX: 00000000 ECX: d51650c0 EDX: d529c3f0 > ESI: d5984288 EDI: d59842c8 EBP: 000003b8 ESP: c06cde20 > DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 > Process swapper (pid: 0, ti=c06cc000 task=c0671340 task.ti=c06cc000) > Stack: > 00000005 d5984288 00000288 d5874ae0 d5940000 d53bf3c4 00000000 00000001 > d5940000 00000002 00000006 d5984000 00000000 d51650c0 d529c3cc ffffffff > d5874ae0 d5940000 00000003 00000001 c03a7165 d5940000 c03eed96 00000000 > Call Trace: > [<c03a7165>] blk_invoke_request_fn+0x95/0x100 > [<c03eed96>] kick_pending_request_queues+0x16/0x30 > [<c03eef3d>] blkif_interrupt+0x18d/0x1d0 > [<c0159500>] handle_IRQ_event+0x30/0x60 > [<c015b418>] handle_level_irq+0x78/0xf0 > [<c010ab17>] do_IRQ+0x77/0x90 > [<c0105d0a>] check_events+0x8/0xe > [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150 > [<c01091c7>] xen_do_upcall+0x7/0xc > [<c01013a7>] _stext+0x3a7/0x1000 > [<c010547f>] xen_safe_halt+0xf/0x20 > [<c0103c50>] xen_idle+0x20/0x40 > [<c0106d2f>] cpu_idle+0x5f/0xb0 > Code: 2c 8d 54 03 40 8d 44 0e 54 b9 6c 00 00 00 e8 68 a5 fc ff 8b 44 > 24 3c e8 cf 92 fd ff 83 44 24 18 01 e9 40 fd ff ff 0f 0b eb fe 90 <0f> > 0b eb fe 8b 44 24 20 ba 10 ea 3e c0 8b 4c 24 20 > c7 04 24 0b > EIP: [<c03eed00>] do_blkif_request+0x2e0/0x360 SS:ESP 0069:c06cde20 > Kernel panic - not syncing: Fatal exception in interrupt > ------------[ cut here ]------------ > WARNING: at kernel/smp.c:333 smp_call_function_mask+0x1cb/0x1d0() > Modules linked in: > Pid: 0, comm: swapper Tainted: G D 2.6.28-linode15 #1 > Call Trace: > [<c0128aaf>] warn_on_slowpath+0x5f/0x90 > [<c03b92c6>] memmove+0x36/0x40 > [<c03dd10a>] scrup+0x7a/0xe0 > [<c0140967>] atomic_notifier_call_chain+0x17/0x20 > [<c03dd18f>] notify_update+0x1f/0x30 > [<c03dd41a>] vt_console_print+0x20a/0x2d0 > [<c0105427>] xen_force_evtchn_callback+0x17/0x30 > [<c0105d0a>] check_events+0x8/0xe > [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1 > [<c0105427>] xen_force_evtchn_callback+0x17/0x30 > [<c0105d0a>] check_events+0x8/0xe > [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1 > [<c01295b0>] vprintk+0x170/0x350 > [<c014a45b>] smp_call_function_mask+0x1cb/0x1d0 > [<c0106000>] stop_self+0x0/0x30 > [<c0105427>] xen_force_evtchn_callback+0x17/0x30 > [<c0105d0a>] check_events+0x8/0xe > [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1 > [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 > [<c03df146>] do_unblank_screen+0x16/0x130 > [<c014a474>] smp_call_function+0x14/0x20 > [<c0128b3e>] panic+0x4e/0x100 > [<c010ac6c>] oops_end+0x8c/0xa0 > [<c0109b80>] do_invalid_op+0x0/0xa0 > [<c0109bff>] do_invalid_op+0x7f/0xa0 > [<c03eed00>] do_blkif_request+0x2e0/0x360 > [<c011689e>] pvclock_clocksource_read+0x4e/0xe0 > [<c01059b3>] get_abs_timeout+0x13/0x30 > [<c0105427>] xen_force_evtchn_callback+0x17/0x30 > [<c0105d0a>] check_events+0x8/0xe > [<c0105c73>] xen_restore_fl_direct_end+0x0/0x1 > [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 > [<c056245a>] error_code+0x72/0x78 > [<c03eed00>] do_blkif_request+0x2e0/0x360 > [<c03a7165>] blk_invoke_request_fn+0x95/0x100 > [<c03eed96>] kick_pending_request_queues+0x16/0x30 > [<c03eef3d>] blkif_interrupt+0x18d/0x1d0 > [<c0159500>] handle_IRQ_event+0x30/0x60 > [<c015b418>] handle_level_irq+0x78/0xf0 > [<c010ab17>] do_IRQ+0x77/0x90 > [<c0105d0a>] check_events+0x8/0xe > [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150 > [<c01091c7>] xen_do_upcall+0x7/0xc > [<c01013a7>] _stext+0x3a7/0x1000 > [<c010547f>] xen_safe_halt+0xf/0x20 > [<c0103c50>] xen_idle+0x20/0x40 > [<c0106d2f>] cpu_idle+0x5f/0xb0 > ---[ end trace d5f11e988eae6396 ]--- > > > > And this one is from another user on a different host: > > ------------[ cut here ]------------ > Kernel BUG at c03eed00 [verbose debug info unavailable] > invalid opcode: 0000 [#1] SMP > last sysfs file: /sys/class/net/tun1/type > Modules linked in: dahdi_dummy dahdi > > Pid: 157, comm: kswapd0 Not tainted (2.6.28-linode15 #1) > EIP: 0061:[<c03eed00>] EFLAGS: 00010046 CPU: 0 > EIP is at do_blkif_request+0x2e0/0x360 > EAX: 00000001 EBX: 00000000 ECX: c1bc7240 EDX: d6132d20 > ESI: d5966000 EDI: d5966040 EBP: 00000165 ESP: d61e1b90 > DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 > Process kswapd0 (pid: 157, ti=d61e0000 task=d6106ef0 task.ti=d61e0000) > Stack: > 00000005 d5966000 00000000 d61da9e0 d596e000 d5488d0c 00000000 00000018 > d596e000 00000002 00000000 d5966000 00000000 c1bc7240 d613294c ffffffff > d61da9e0 d596e000 00000006 00000018 c03a7165 d596e000 c03eed96 00000000 > Call Trace: > [<c03a7165>] blk_invoke_request_fn+0x95/0x100 > [<c03eed96>] kick_pending_request_queues+0x16/0x30 > [<c03eef3d>] blkif_interrupt+0x18d/0x1d0 > [<c03a514d>] elv_next_request+0x1d/0x170 > [<c0159500>] handle_IRQ_event+0x30/0x60 > [<c015b418>] handle_level_irq+0x78/0xf0 > [<c010ab17>] do_IRQ+0x77/0x90 > [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 > [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150 > [<c01091c7>] xen_do_upcall+0x7/0xc > [<c0101227>] _stext+0x227/0x1000 > [<c0105427>] xen_force_evtchn_callback+0x17/0x30 > [<c0182687>] kmem_cache_alloc+0x57/0xb0 > [<c016140d>] mempool_alloc+0x2d/0xe0 > [<c016140d>] mempool_alloc+0x2d/0xe0 > [<c01774dc>] try_to_unmap_one+0x8c/0x240 > [<c01a737b>] bvec_alloc_bs+0x7b/0x140 > [<c01a7641>] bio_alloc_bioset+0x51/0xe0 > [<c01a773b>] bio_alloc+0xb/0x20 > [<c017ae20>] get_swap_bio+0x20/0xa0 > [<c017afdb>] swap_writepage+0x4b/0xc0 > [<c017b050>] end_swap_bio_write+0x0/0x80 > [<c01692cd>] shrink_page_list+0x51d/0x650 > [<c016550f>] determine_dirtyable_memory+0x1f/0x90 > [<c0165598>] get_dirty_limits+0x18/0x2e0 > [<c0169837>] shrink_zone+0x437/0x860 > [<c0142117>] getnstimeofday+0x37/0xe0 > [<c0562025>] _spin_lock+0x5/0x10 > [<c0280215>] nfs_access_cache_shrinker+0xa5/0x1f0 > [<c01b9e59>] mb_cache_shrink_fn+0x59/0x100 > [<c016a7f8>] kswapd+0x4e8/0x500 > [<c0168220>] isolate_pages_global+0x0/0x220 > [<c013c3c0>] autoremove_wake_function+0x0/0x40 > [<c011ec00>] complete+0x40/0x60 > [<c016a310>] kswapd+0x0/0x500 > [<c013c0a2>] kthread+0x42/0x70 > [<c013c060>] kthread+0x0/0x70 > [<c0109177>] kernel_thread_helper+0x7/0x10 > Code: 2c 8d 54 03 40 8d 44 0e 54 b9 6c 00 00 00 e8 68 a5 fc ff 8b 44 > 24 3c e8 cf 92 fd ff 83 44 24 18 01 e9 40 fd ff ff 0f 0b eb fe 90 <0f> > 0b eb fe 8b 44 24 20 ba 10 ea 3e c0 8b 4c 24 20 c7 04 24 0b > EIP: [<c03eed00>] do_blkif_request+0x2e0/0x360 SS:ESP 0069:d61e1b90 > Kernel panic - not syncing: Fatal exception in interrupt > ------------[ cut here ]------------ > WARNING: at kernel/smp.c:333 smp_call_function_mask+0x1cb/0x1d0() > Modules linked in: dahdi_dummy dahdi > Pid: 157, comm: kswapd0 Tainted: G D 2.6.28-linode15 #1 > Call Trace: > [<c0128aaf>] warn_on_slowpath+0x5f/0x90 > [<c03b92c6>] memmove+0x36/0x40 > [<c03dd10a>] scrup+0x7a/0xe0 > [<c0140967>] atomic_notifier_call_chain+0x17/0x20 > [<c03dd18f>] notify_update+0x1f/0x30 > [<c03dd41a>] vt_console_print+0x20a/0x2d0 > [<c0562183>] _spin_lock_irqsave+0x33/0x50 > [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 > [<c01291dc>] release_console_sem+0x19c/0x1e0 > [<c01295b0>] vprintk+0x170/0x350 > [<c014a45b>] smp_call_function_mask+0x1cb/0x1d0 > [<c0106000>] stop_self+0x0/0x30 > [<c0562183>] _spin_lock_irqsave+0x33/0x50 > [<c01324a7>] lock_timer_base+0x27/0x60 > [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 > [<c03df146>] do_unblank_screen+0x16/0x130 > [<c014a474>] smp_call_function+0x14/0x20 > [<c0128b3e>] panic+0x4e/0x100 > [<c010ac6c>] oops_end+0x8c/0xa0 > [<c0109b80>] do_invalid_op+0x0/0xa0 > [<c0109bff>] do_invalid_op+0x7f/0xa0 > [<c03eed00>] do_blkif_request+0x2e0/0x360 > [<c01055d5>] get_runstate_snapshot+0x75/0x90 > [<c0105a2f>] xen_sched_clock+0x1f/0x80 > [<c014135b>] __update_sched_clock+0x2b/0x140 > [<c011689e>] pvclock_clocksource_read+0x4e/0xe0 > [<c0562183>] _spin_lock_irqsave+0x33/0x50 > [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 > [<c056245a>] error_code+0x72/0x78 > [<c03eed00>] do_blkif_request+0x2e0/0x360 > [<c03a7165>] blk_invoke_request_fn+0x95/0x100 > [<c03eed96>] kick_pending_request_queues+0x16/0x30 > [<c03eef3d>] blkif_interrupt+0x18d/0x1d0 > [<c03a514d>] elv_next_request+0x1d/0x170 > [<c0159500>] handle_IRQ_event+0x30/0x60 > [<c015b418>] handle_level_irq+0x78/0xf0 > [<c010ab17>] do_IRQ+0x77/0x90 > [<c05621b3>] _spin_unlock_irqrestore+0x13/0x20 > [<c03c8e08>] xen_evtchn_do_upcall+0xe8/0x150 > [<c01091c7>] xen_do_upcall+0x7/0xc > [<c0101227>] _stext+0x227/0x1000 > [<c0105427>] xen_force_evtchn_callback+0x17/0x30 > [<c0182687>] kmem_cache_alloc+0x57/0xb0 > [<c016140d>] mempool_alloc+0x2d/0xe0 > [<c016140d>] mempool_alloc+0x2d/0xe0 > [<c01774dc>] try_to_unmap_one+0x8c/0x240 > [<c01a737b>] bvec_alloc_bs+0x7b/0x140 > [<c01a7641>] bio_alloc_bioset+0x51/0xe0 > [<c01a773b>] bio_alloc+0xb/0x20 > [<c017ae20>] get_swap_bio+0x20/0xa0 > [<c017afdb>] swap_writepage+0x4b/0xc0 > [<c017b050>] end_swap_bio_write+0x0/0x80 > [<c01692cd>] shrink_page_list+0x51d/0x650 > [<c016550f>] determine_dirtyable_memory+0x1f/0x90 > [<c0165598>] get_dirty_limits+0x18/0x2e0 > [<c0169837>] shrink_zone+0x437/0x860 > [<c0142117>] getnstimeofday+0x37/0xe0 > [<c0562025>] _spin_lock+0x5/0x10 > [<c0280215>] nfs_access_cache_shrinker+0xa5/0x1f0 > [<c01b9e59>] mb_cache_shrink_fn+0x59/0x100 > [<c016a7f8>] kswapd+0x4e8/0x500 > [<c0168220>] isolate_pages_global+0x0/0x220 > [<c013c3c0>] autoremove_wake_function+0x0/0x40 > [<c011ec00>] complete+0x40/0x60 > [<c016a310>] kswapd+0x0/0x500 > [<c013c0a2>] kthread+0x42/0x70 > [<c013c060>] kthread+0x0/0x70 > [<c0109177>] kernel_thread_helper+0x7/0x10 > ---[ end trace 505b292c57e00b05 ]--- > > -Chris_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher S. Aker
2009-Jan-14 16:03 UTC
[Xen-devel] Re: 2.6.28 domU panics in blk_invoke_request_fn()
Jeremy Fitzhardinge wrote:> Christopher S. Aker wrote: >> Below are two similar panics we''ve captured thus far with some light >> testing of 2.6.28 domUs across our Xen cluster. The kernel binary and >> associated files can be found here: > > How frequently does this occur? > > Did it just started appearing on 2.6.28, but .27.x is OK? Does it > happen often enough that you could practically bisect?This is new in 2.6.28. So far it''s occurring randomly so bisecting isn''t an option.> Could you compile the kernel with CONFIG_BUG_VERBOSE enabled so that it > at prints some source/line info? The location of the BUG isn''t obvious.I''ve compiled in CONFIG_DEBUG_BUGVERBOSE and redeployed the kernels. I''ll post to our community forums asking for wider testing.> What''s your dom0?3.2.1 (+/- a few changesets), 64 bit hypervisor, with dom0 2.6.18.8 PAE from xenbits at changeset number 622 and the other host''s dom0 at 524. -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel