This is on a Debian 2.6.32 lenny-backports amd64 domU - SysRq+W (show
blocked tasks) says:
[...]
[78137.427846] apache2 D 0000000000000000 0 9566 24433 0x00000000
[78137.427846] ffff8800bfbcd4c0 0000000000000286 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff88001889ffd8
0000000000015780
[78137.427846] 0000000000015780 ffff880001a662e0 ffff880001a665d8
0000000200000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff81241f5e>] ? sys_accept4+0x168/0x1d7
[78137.427846] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff8800034f4780 0 9568 24433 0x00000000
[78137.427846] ffff8800bddf7810 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880035f2dfd8
0000000000015780
[78137.427846] 0000000000015780 ffff880001a67100 ffff880001a673f8
0000000500000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff81097d1c>] ? __call_rcu+0x110/0x118
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff81152a9f>] ? cap_file_free_security+0x0/0x1
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff8800034b8780 0 9570 24433 0x00000000
[78137.427846] ffff880002a50710 0000000000000286 ffff880053f9db38
ffff880053f9db34
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880053f9dfd8
0000000000015780
[78137.427846] 0000000000015780 ffff880001a60000 ffff880001a602f8
0000000300000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff812fb8da>] ? _spin_lock_irq+0x7/0x22
[78137.427846] [<ffffffff8105b8ea>] ? do_sigaction+0x155/0x16a
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff88000345e780 0 9572 24433 0x00000000
[78137.427846] ffff880083928710 0000000000000286 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff8800a7819fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880002abf810 ffff880002abfb08
0000000000000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff81097d1c>] ? __call_rcu+0x110/0x118
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff81152a9f>] ? cap_file_free_security+0x0/0x1
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff880003512780 0 9574 24433 0x00000000
[78137.427846] ffff8800865fa350 0000000000000286 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff8800094d5fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880002abc6a0 ffff880002abc998
0000000600000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff811099c9>] ? splice_from_pipe+0x58/0x63
[78137.427846] [<ffffffff810ff546>] ? touch_atime+0x7c/0x127
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff88000347c780 0 9576 24433 0x00000000
[78137.427846] ffff880092e862e0 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880051e4ffd8
0000000000015780
[78137.427846] 0000000000015780 ffff880002ab8710 ffff880002ab8a08
0000000100000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff88000345e780 0 9577 24433 0x00000000
[78137.427846] ffff8800018aa350 0000000000000286 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880021df1fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880002aba350 ffff880002aba648
0000000000000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff8100e5d4>] ? xen_smp_send_reschedule+0x0/0x7
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff812fb8da>] ? _spin_lock_irq+0x7/0x22
[78137.427846] [<ffffffff8105b8ea>] ? do_sigaction+0x155/0x16a
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff88000345e780 0 9588 24433 0x00000000
[78137.427846] ffff880086d82350 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff88003407dfd8
0000000000015780
[78137.427846] 0000000000015780 ffff8800699f0000 ffff8800699f02f8
0000000000000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff81097d1c>] ? __call_rcu+0x110/0x118
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff81152a9f>] ? cap_file_free_security+0x0/0x1
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff8800034d6780 0 9592 24433 0x00000000
[78137.427846] ffff880001addbd0 0000000000000282 000000009b082000
0000000000000002
[78137.427846] 0000000000000040 000000000000f9e0 ffff88004e16ffd8
0000000000015780
[78137.427846] 0000000000015780 ffff8800699f2a60 ffff8800699f2d58
00000004810414a4
[78137.427846] Call Trace:
[78137.427846] [<ffffffff81243032>] ? release_sock+0x13/0xa0
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff812fae21>] ? __mutex_lock_common+0x122/0x192
[78137.427846] [<ffffffff812faf49>] ? mutex_lock+0x1a/0x31
[78137.427846] [<ffffffff810f5ffc>] ? do_lookup+0x80/0x15d
[78137.427846] [<ffffffff810f6a7c>] ? __link_path_walk+0x5a5/0x6f5
[78137.427846] [<ffffffff810f6dfa>] ? path_walk+0x66/0xc9
[78137.427846] [<ffffffff810f8264>] ? do_path_lookup+0x20/0x77
[78137.427846] [<ffffffff810f9746>] ? user_path_at+0x48/0x79
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff810f1c22>] ? vfs_fstatat+0x2c/0x57
[78137.427846] [<ffffffff810f1d15>] ? sys_newstat+0x11/0x30
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[78137.427846] apache2 D ffff88000345e780 0 9593 24433 0x00000000
[78137.427846] ffff88005fff4db0 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880028e37fd8
0000000000015780
[78137.427846] 0000000000015780 ffff8800699f54c0 ffff8800699f57b8
0000000000000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff810ee6e1>] ? do_sync_read+0xce/0x113
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff8800034d6780 0 9594 24433 0x00000000
[78137.427846] ffff880086d84db0 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff8800a2a19fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880001af3f90 ffff880001af4288
0000000400000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff812fb8da>] ? _spin_lock_irq+0x7/0x22
[78137.427846] [<ffffffff8105b8ea>] ? do_sigaction+0x155/0x16a
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff88000345e780 0 9599 24433 0x00000000
[78137.427846] ffff880001920710 0000000000000282 ffff880092bbbb38
ffff880092bbbb34
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880092bbbfd8
0000000000015780
[78137.427846] 0000000000015780 ffff880001af1530 ffff880001af1828
0000000000000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff810fcde3>] ? d_kill+0x58/0x61
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff88000347c780 0 9600 24433 0x00000000
[78137.427846] ffff88009ad7e2e0 0000000000000286 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff8800bbf79fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880001af2350 ffff880001af2648
0000000100000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff810ee6e1>] ? do_sync_read+0xce/0x113
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff88000347c780 0 9601 24433 0x00000008
[78137.427846] ffff880092e83880 0000000000000282 ffff88007664bb38
ffff88007664bb34
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff88007664bfd8
0000000000015780
[78137.427846] 0000000000015780 ffff880001af0e20 ffff880001af1118
0000000100000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8105363b>] ? _local_bh_enable_ip+0x22/0x8f
[78137.427846] [<ffffffff8127b537>] ? inet_csk_accept+0x1bf/0x1e9
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff81241f5e>] ? sys_accept4+0x168/0x1d7
[78137.427846] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff8800034f4780 0 9604 24433 0x00000000
[78137.427846] ffff8800018ae9f0 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880072835fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880001af7100 ffff880001af73f8
0000000500000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff812fb8da>] ? _spin_lock_irq+0x7/0x22
[78137.427846] [<ffffffff8105b8ea>] ? do_sigaction+0x155/0x16a
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff88000347c780 0 9606 24433 0x00000000
[78137.427846] ffff880001878710 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff8800a6875fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880001af3170 ffff880001af3468
0000000100000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff810f9750>] ? user_path_at+0x52/0x79
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff88000345e780 0 9608 24433 0x00000000
[78137.427846] ffff88008392bf90 0000000000000286 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff88001c965fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880001af1c40 ffff880001af1f38
0000000000000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff8800034d6780 0 9610 24433 0x00000000
[78137.427846] ffff88005fcdc6a0 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff8800066e3fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880092e87100 ffff880092e873f8
0000000400000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff81289e23>] ? tcp_write_xmit+0x93e/0x96c
[78137.427846] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846] [<ffffffff8106c1df>] ? ktime_get_ts+0x68/0xb2
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D 0000000000000001 0 9611 24433 0x00000000
[78137.427846] ffff8800bfbcd4c0 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880057f57fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880092e84db0 ffff880092e850a8
0000000000000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff810fcde3>] ? d_kill+0x58/0x61
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D 0000000000000000 0 9612 24433 0x00000000
[78137.427846] ffff8800bfa58e20 0000000000000286 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880087639fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880092e83f90 ffff880092e84288
0000000100000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff81289e23>] ? tcp_write_xmit+0x93e/0x96c
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D 0000000000000000 0 9614 24433 0x00000000
[78137.427846] ffff8800bfbcd4c0 0000000000000286 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880024ffdfd8
0000000000015780
[78137.427846] 0000000000015780 ffff880092e82350 ffff880092e82648
0000000400000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff810cc61f>] ? handle_mm_fault+0x27a/0x80f
[78137.427846] [<ffffffff81241f5e>] ? sys_accept4+0x168/0x1d7
[78137.427846] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff8800034b8780 0 9616 24433 0x00000000
[78137.427846] ffff880065e1f100 0000000000000286 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880067231fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880092e80710 ffff880092e80a08
0000000300000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff8100e5d4>] ? xen_smp_send_reschedule+0x0/0x7
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff81152ab5>] ? cap_dentry_open+0x0/0x3
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff88000347c780 0 9620 24433 0x00000000
[78137.427846] ffff880002abc6a0 0000000000000286 0000000000000002
ffffffff8103f604
[78137.427846] 0000000000000001 000000000000f9e0 ffff88008c655fd8
0000000000015780
[78137.427846] 0000000000015780 ffff88006c7a7810 ffff88006c7a7b08
00000001810414a4
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8103f604>] ? find_busiest_queue+0xba/0xdb
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff812fae21>] ? __mutex_lock_common+0x122/0x192
[78137.427846] [<ffffffff812faf49>] ? mutex_lock+0x1a/0x31
[78137.427846] [<ffffffff810f5ffc>] ? do_lookup+0x80/0x15d
[78137.427846] [<ffffffff810f6a7c>] ? __link_path_walk+0x5a5/0x6f5
[78137.427846] [<ffffffff810f6dfa>] ? path_walk+0x66/0xc9
[78137.427846] [<ffffffff810f8264>] ? do_path_lookup+0x20/0x77
[78137.427846] [<ffffffff810f9746>] ? user_path_at+0x48/0x79
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff810f1c22>] ? vfs_fstatat+0x2c/0x57
[78137.427846] [<ffffffff810f1d15>] ? sys_newstat+0x11/0x30
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[78137.427846] apache2 D ffff88000349a780 0 9625 24433 0x00000000
[78137.427846] ffff880072080000 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880052299fd8
0000000000015780
[78137.427846] 0000000000015780 ffff880072082350 ffff880072082648
0000000200000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff810cc61f>] ? handle_mm_fault+0x27a/0x80f
[78137.427846] [<ffffffff81241f5e>] ? sys_accept4+0x168/0x1d7
[78137.427846] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff8800034f4780 0 9627 24433 0x00000000
[78137.427846] ffff8800bd16e2e0 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff880066639fd8
0000000000015780
[78137.427846] 0000000000015780 ffff88000186a350 ffff88000186a648
0000000500000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff81243032>] ? release_sock+0x13/0xa0
[78137.427846] [<ffffffff8105363b>] ? _local_bh_enable_ip+0x22/0x8f
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff88000345e780 0 9629 24433 0x00000000
[78137.427846] ffff880001923170 0000000000000282 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff8800a66e5fd8
0000000000015780
[78137.427846] 0000000000015780 ffff88009ad7aa60 ffff88009ad7ad58
0000000000000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff810ee6e1>] ? do_sync_read+0xce/0x113
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff812fbf5a>] ? error_exit+0x2a/0x60
[78137.427846] [<ffffffff8101151d>] ? retint_restore_args+0x5/0x6
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff8800034b8780 0 9630 24433 0x00000000
[78137.427846] ffff880086d82a60 0000000000000286 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff8800913e9fd8
0000000000015780
[78137.427846] 0000000000015780 ffff88009ad7b170 ffff88009ad7b468
0000000300000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff81289e23>] ? tcp_write_xmit+0x93e/0x96c
[78137.427846] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[78137.427846] [<ffffffff8106c1df>] ? ktime_get_ts+0x68/0xb2
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] apache2 D ffff880003512780 0 9651 24433 0x00000000
[78137.427846] ffff880002abc6a0 0000000000000286 0000000000000000
000000010128ed1c
[78137.427846] ffffffff8100dc35 000000000000f9e0 ffff88005dfeffd8
0000000000015780
[78137.427846] 0000000000015780 ffff88009ad7a350 ffff88009ad7a648
0000000600000200
[78137.427846] Call Trace:
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78137.427846] [<ffffffff8105abfa>] ? __mod_timer+0x141/0x153
[78137.427846] [<ffffffff810c0121>] ? try_to_free_pages+0x379/0x38e
[78137.427846] [<ffffffff812faa3a>] ? schedule_timeout+0xa5/0xdd
[78137.427846] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78137.427846] [<ffffffff810ba126>] ? __alloc_pages_nodemask+0x48b/0x5f5
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fa700>] ? io_schedule+0x93/0xb7
[78137.427846] [<ffffffff810bb6e5>] ?
__do_page_cache_readahead+0x9b/0x1b4
[78137.427846] [<ffffffff81064d04>] ? wake_bit_function+0x0/0x23
[78137.427846] [<ffffffff810bb81a>] ? ra_submit+0x1c/0x20
[78137.427846] [<ffffffff810b44f6>] ? filemap_fault+0x17d/0x2f6
[78137.427846] [<ffffffff810ca43e>] ? __do_fault+0x54/0x3c3
[78137.427846] [<ffffffff8100c241>] ?
__raw_callee_save_xen_pud_val+0x11/0x1e
[78137.427846] [<ffffffff810cc75d>] ? handle_mm_fault+0x3b8/0x80f
[78137.427846] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78137.427846] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78137.427846] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78137.427846] [<ffffffff812fde86>] ? do_page_fault+0x2e0/0x2fc
[78137.427846] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78137.427846] Sched Debug Version: v0.09, 2.6.32-bpo.5-amd64 #1
[78137.427846] now at 78148825.414302 msecs
[78137.427846] .jiffies : 4314426651
[78137.427846] .sysctl_sched_latency : 20.000000
[78137.427846] .sysctl_sched_min_granularity : 4.000000
[78137.427846] .sysctl_sched_wakeup_granularity : 4.000000
[78137.427846] .sysctl_sched_child_runs_first : 0.000000
[78137.427846] .sysctl_sched_features : 15834235
[78137.427846]
[78137.427846] cpu#0, 2493.750 MHz
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_switches : 47902047
[78137.427846] .nr_load_updates : 15938294
[78137.427846] .nr_uninterruptible : 75
[78137.427846] .next_balance : 4314.426652
[78137.427846] .curr->pid : 0
[78137.427846] .clock : 78137422.159104
[78137.427846] .cpu_load[0] : 0
[78137.427846] .cpu_load[1] : 0
[78137.427846] .cpu_load[2] : 0
[78137.427846] .cpu_load[3] : 0
[78137.427846] .cpu_load[4] : 0
[78137.427846]
[78137.427846] cfs_rq[0]:/
[78137.427846] .exec_clock : 0.000000
[78137.427846] .MIN_vruntime : 0.000001
[78137.427846] .min_vruntime : 5680735.179660
[78137.427846] .max_vruntime : 0.000001
[78137.427846] .spread : 0.000000
[78137.427846] .spread0 : 0.000000
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_spread_over : 0
[78137.427846] .shares : 0
[78137.427846]
[78137.427846] rt_rq[0]:
[78137.427846] .rt_nr_running : 0
[78137.427846] .rt_throttled : 0
[78137.427846] .rt_time : 0.000000
[78137.427846] .rt_runtime : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[78137.427846]
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#1, 2493.750 MHz
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_switches : 8912456
[78137.427846] .nr_load_updates : 10191035
[78137.427846] .nr_uninterruptible : 30
[78137.427846] .next_balance : 4314.426652
[78137.427846] .curr->pid : 0
[78137.427846] .clock : 78148824.260676
[78137.427846] .cpu_load[0] : 0
[78137.427846] .cpu_load[1] : 0
[78137.427846] .cpu_load[2] : 0
[78137.427846] .cpu_load[3] : 0
[78137.427846] .cpu_load[4] : 0
[78137.427846]
[78137.427846] cfs_rq[1]:/
[78137.427846] .exec_clock : 0.000000
[78137.427846] .MIN_vruntime : 0.000001
[78137.427846] .min_vruntime : 3652776.005874
[78137.427846] .max_vruntime : 0.000001
[78137.427846] .spread : 0.000000
[78137.427846] .spread0 : -2027959.173786
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_spread_over : 0
[78137.427846] .shares : 0
[78137.427846]
[78137.427846] rt_rq[1]:
[78137.427846] .rt_nr_running : 0
[78137.427846] .rt_throttled : 0
[78137.427846] .rt_time : 0.000000
[78137.427846] .rt_runtime : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[78137.427846]
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#2, 2493.750 MHz
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_switches : 6929803
[78137.427846] .nr_load_updates : 6509315
[78137.427846] .nr_uninterruptible : 20
[78137.427846] .next_balance : 4314.426652
[78137.427846] .curr->pid : 0
[78137.427846] .clock : 78148824.506917
[78137.427846] .cpu_load[0] : 0
[78137.427846] .cpu_load[1] : 0
[78137.427846] .cpu_load[2] : 0
[78137.427846] .cpu_load[3] : 0
[78137.427846] .cpu_load[4] : 0
[78137.427846]
[78137.427846] cfs_rq[2]:/
[78137.427846] .exec_clock : 0.000000
[78137.427846] .MIN_vruntime : 0.000001
[78137.427846] .min_vruntime : 3012547.394539
[78137.427846] .max_vruntime : 0.000001
[78137.427846] .spread : 0.000000
[78137.427846] .spread0 : -2668187.785121
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_spread_over : 0
[78137.427846] .shares : 0
[78137.427846]
[78137.427846] rt_rq[2]:
[78137.427846] .rt_nr_running : 0
[78137.427846] .rt_throttled : 0
[78137.427846] .rt_time : 0.000000
[78137.427846] .rt_runtime : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[78137.427846]
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#3, 2493.750 MHz
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_switches : 4929509
[78137.427846] .nr_load_updates : 4016716
[78137.427846] .nr_uninterruptible : 13
[78137.427846] .next_balance : 4314.426652
[78137.427846] .curr->pid : 0
[78137.427846] .clock : 78149024.755812
[78137.427846] .cpu_load[0] : 0
[78137.427846] .cpu_load[1] : 0
[78137.427846] .cpu_load[2] : 0
[78137.427846] .cpu_load[3] : 0
[78137.427846] .cpu_load[4] : 0
[78137.427846]
[78137.427846] cfs_rq[3]:/
[78137.427846] .exec_clock : 0.000000
[78137.427846] .MIN_vruntime : 0.000001
[78137.427846] .min_vruntime : 2379542.626815
[78137.427846] .max_vruntime : 0.000001
[78137.427846] .spread : 0.000000
[78137.427846] .spread0 : -3301192.552845
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_spread_over : 0
[78137.427846] .shares : 0
[78137.427846]
[78137.427846] rt_rq[3]:
[78137.427846] .rt_nr_running : 0
[78137.427846] .rt_throttled : 0
[78137.427846] .rt_time : 0.000000
[78137.427846] .rt_runtime : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[78137.427846]
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#4, 2493.750 MHz
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_switches : 3274386
[78137.427846] .nr_load_updates : 2348628
[78137.427846] .nr_uninterruptible : 33
[78137.427846] .next_balance : 4314.426714
[78137.427846] .curr->pid : 0
[78137.427846] .clock : 78149025.003801
[78137.427846] .cpu_load[0] : 0
[78137.427846] .cpu_load[1] : 0
[78137.427846] .cpu_load[2] : 0
[78137.427846] .cpu_load[3] : 0
[78137.427846] .cpu_load[4] : 0
[78137.427846]
[78137.427846] cfs_rq[4]:/
[78137.427846] .exec_clock : 0.000000
[78137.427846] .MIN_vruntime : 0.000001
[78137.427846] .min_vruntime : 1864943.780970
[78137.427846] .max_vruntime : 0.000001
[78137.427846] .spread : 0.000000
[78137.427846] .spread0 : -3815791.398690
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_spread_over : 0
[78137.427846] .shares : 0
[78137.427846]
[78137.427846] rt_rq[4]:
[78137.427846] .rt_nr_running : 0
[78137.427846] .rt_throttled : 0
[78137.427846] .rt_time : 0.000000
[78137.427846] .rt_runtime : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[78137.427846]
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#5, 2493.750 MHz
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_switches : 3594802
[78137.427846] .nr_load_updates : 1453667
[78137.427846] .nr_uninterruptible : 39
[78137.427846] .next_balance : 4314.426652
[78137.427846] .curr->pid : 0
[78137.427846] .clock : 78149025.261256
[78137.427846] .cpu_load[0] : 0
[78137.427846] .cpu_load[1] : 0
[78137.427846] .cpu_load[2] : 0
[78137.427846] .cpu_load[3] : 0
[78137.427846] .cpu_load[4] : 0
[78137.427846]
[78137.427846] cfs_rq[5]:/
[78137.427846] .exec_clock : 0.000000
[78137.427846] .MIN_vruntime : 0.000001
[78137.427846] .min_vruntime : 1282845.273539
[78137.427846] .max_vruntime : 0.000001
[78137.427846] .spread : 0.000000
[78137.427846] .spread0 : -4397889.906121
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_spread_over : 0
[78137.427846] .shares : 0
[78137.427846]
[78137.427846] rt_rq[5]:
[78137.427846] .rt_nr_running : 0
[78137.427846] .rt_throttled : 0
[78137.427846] .rt_time : 0.000000
[78137.427846] .rt_runtime : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[78137.427846]
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#6, 2493.750 MHz
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_switches : 3841965
[78137.427846] .nr_load_updates : 1239272
[78137.427846] .nr_uninterruptible : 26
[78137.427846] .next_balance : 4314.426652
[78137.427846] .curr->pid : 0
[78137.427846] .clock : 78149025.516745
[78137.427846] .cpu_load[0] : 0
[78137.427846] .cpu_load[1] : 0
[78137.427846] .cpu_load[2] : 0
[78137.427846] .cpu_load[3] : 0
[78137.427846] .cpu_load[4] : 0
[78137.427846]
[78137.427846] cfs_rq[6]:/
[78137.427846] .exec_clock : 0.000000
[78137.427846] .MIN_vruntime : 0.000001
[78137.427846] .min_vruntime : 1124955.770063
[78137.427846] .max_vruntime : 0.000001
[78137.427846] .spread : 0.000000
[78137.427846] .spread0 : -4555779.409597
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_spread_over : 0
[78137.427846] .shares : 0
[78137.427846]
[78137.427846] rt_rq[6]:
[78137.427846] .rt_nr_running : 0
[78137.427846] .rt_throttled : 0
[78137.427846] .rt_time : 0.000000
[78137.427846] .rt_runtime : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[78137.427846]
----------------------------------------------------------------------------------------------------------
[78137.427846]
[78137.427846] cpu#7, 2493.750 MHz
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_switches : 3495511
[78137.427846] .nr_load_updates : 999351
[78137.427846] .nr_uninterruptible : 15
[78137.427846] .next_balance : 4314.426652
[78137.427846] .curr->pid : 0
[78137.427846] .clock : 78149221.757059
[78137.427846] .cpu_load[0] : 0
[78137.427846] .cpu_load[1] : 0
[78137.427846] .cpu_load[2] : 0
[78137.427846] .cpu_load[3] : 0
[78137.427846] .cpu_load[4] : 0
[78137.427846]
[78137.427846] cfs_rq[7]:/
[78137.427846] .exec_clock : 0.000000
[78137.427846] .MIN_vruntime : 0.000001
[78137.427846] .min_vruntime : 899194.224398
[78137.427846] .max_vruntime : 0.000001
[78137.427846] .spread : 0.000000
[78137.427846] .spread0 : -4781540.955262
[78137.427846] .nr_running : 0
[78137.427846] .load : 0
[78137.427846] .nr_spread_over : 0
[78137.427846] .shares : 0
[78137.427846]
[78137.427846] rt_rq[7]:
[78137.427846] .rt_nr_running : 0
[78137.427846] .rt_throttled : 0
[78137.427846] .rt_time : 0.000000
[78137.427846] .rt_runtime : 950.000000
[78137.427846]
[78137.427846] runnable tasks:
[78137.427846] task PID tree-key switches prio
exec-runtime sum-exec sum-sleep
[78137.427846]
----------------------------------------------------------------------------------------------------------
[78137.427846]
And then I did:
[78270.883837] SysRq : Show backtrace of all active CPUs
[78270.883858] sending NMI to all CPUs:
[78270.883878] BUG: unable to handle kernel paging request at ffffffffff5fc310
[78270.883889] IP: [<ffffffff81027e23>] _flat_send_IPI_mask+0x4b/0x78
[78270.883905] PGD 1003067 PUD 1004067 PMD 159c067 PTE 0
[78270.883917] Thread overran stack, or stack corrupted
[78270.883929] Oops: 0002 [#1] SMP
[78270.883937] last sysfs file:
/sys/devices/vbd-51713/block/xvda1/queue/scheduler
[78270.883947] CPU 0
[78270.883956] Modules linked in: ip6t_LOG ip6t_REJECT nf_conntrack_ipv6
ip6table_filter ip6_tables xt_owner xt_recent ipt_REJECT ipt_LOG xt_state
xt_tcpudp iptable_filter ip_tables x_tables nf_nat_ftp nf_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_conntrack_ftp nf_conntrack snd_pcm snd_timer snd soundcore
snd_page_alloc pcspkr xen_netfront evdev ext3 jbd mbcache dm_mirror
dm_region_hash dm_log dm_snapshot dm_mod xen_blkfront thermal_sys
[78270.884063] Pid: 0, comm: swapper Not tainted 2.6.32-bpo.5-amd64 #1
[78270.884069] RIP: e030:[<ffffffff81027e23>] [<ffffffff81027e23>]
_flat_send_IPI_mask+0x4b/0x78
[78270.884079] RSP: e02b:ffff88000344cd88 EFLAGS: 00010086
[78270.884085] RAX: 00000000ff000000 RBX: 0000000000000800 RCX: 000000000000fe7d
[78270.884091] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 00000000000000ff
[78270.884097] RBP: 0000000000000002 R08: 000000000013f3fa R09: 000000000000000a
[78270.884103] R10: 0000000000000000 R11: 00000000000186a0 R12: ffffffff814d0040
[78270.884109] R13: 00000000000000ff R14: 0000000000000007 R15: 000000000000006c
[78270.884124] FS: 00007ffc218d0750(0000) GS:ffff880003449000(0000)
knlGS:0000000000000000
[78270.884131] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[78270.884137] CR2: ffffffffff5fc310 CR3: 0000000092198000 CR4: 0000000000002620
[78270.884144] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[78270.884150] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[78270.884157] Process swapper (pid: 0, threadinfo ffffffff81428000, task
ffffffff8145d1f0)
[78270.884164] Stack:
[78270.884168] ffffffff8100e22f 0000000000000000 000000000000006c
0000000000000001
[78270.884178] <0> ffff880002b09800 ffffffff8102506e 00000000000000f0
ffffffff812fb8b4
[78270.884190] <0> ffffffff81493620 ffffffff8120c26d 0000000000000001
ffff8800bd355200
[78270.884205] Call Trace:
[78270.884210] <IRQ>
[78270.884217] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78270.884225] [<ffffffff8102506e>] ?
arch_trigger_all_cpu_backtrace+0x3c/0x69
[78270.884235] [<ffffffff812fb8b4>] ? _spin_lock_irqsave+0x15/0x34
[78270.884244] [<ffffffff8120c26d>] ? __handle_sysrq+0xaf/0x14b
[78270.884251] [<ffffffff8120cd3f>] ? hvc_poll+0x143/0x22c
[78270.884262] [<ffffffff8120cf9d>] ? hvc_handle_interrupt+0xc/0x1f
[78270.884272] [<ffffffff810945f0>] ? handle_IRQ_event+0x58/0x126
[78270.884281] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78270.884288] [<ffffffff81095e34>] ? handle_edge_irq+0xc0/0x101
[78270.884296] [<ffffffff8101327f>] ? handle_irq+0x17/0x1d
[78270.884305] [<ffffffff811eeb8a>] ? __xen_evtchn_do_upcall+0xd5/0x15e
[78270.884313] [<ffffffff811eec4f>] ? xen_evtchn_do_upcall+0x2e/0x41
[78270.884320] [<ffffffff81011cfe>] ?
xen_do_hypervisor_callback+0x1e/0x30
[78270.884326] <EOI>
[78270.884332] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78270.884340] [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001
[78270.884348] [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001
[78270.884355] [<ffffffff8100dcb3>] ? xen_safe_halt+0xc/0x15
[78270.884362] [<ffffffff8100be63>] ? xen_idle+0x37/0x40
[78270.884372] [<ffffffff8100feb1>] ? cpu_idle+0xa2/0xda
[78270.884383] [<ffffffff814efcd1>] ? start_kernel+0x3dc/0x3e8
[78270.884389] Code: 48 8b 05 a1 a3 4a 00 83 fe 02 8b 58 34 75 0a ff 90 58 01 00
00 eb 0e f3 90 8b 04 25 00 c3 5f ff f6 c4 10 75 f2 44 89 e8 c1 e0 18 <89>
04 25 10 c3 5f ff 89 e8 09 d8 80 cf 04 83 fd 02 0f 44 c3 89
[78270.884482] RIP [<ffffffff81027e23>] _flat_send_IPI_mask+0x4b/0x78
[78270.884491] RSP <ffff88000344cd88>
[78270.884495] CR2: ffffffffff5fc310
[78270.884503] ---[ end trace 198fbc07a27eaf2a ]---
[78270.884509] Kernel panic - not syncing: Fatal exception in interrupt
[78270.884516] Pid: 0, comm: swapper Tainted: G D 2.6.32-bpo.5-amd64 #1
[78270.884522] Call Trace:
[78270.884526] <IRQ> [<ffffffff812f9adc>] ? panic+0x86/0x143
[78270.884536] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78270.884546] [<ffffffff811b703c>] ? dummycon_dummy+0x0/0x3
[78270.884552] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[78270.884560] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78270.884566] [<ffffffff811b703c>] ? dummycon_dummy+0x0/0x3
[78270.884573] [<ffffffff812fb9a2>] ? _spin_unlock_irqrestore+0xd/0xe
[78270.884580] [<ffffffff812fc845>] ? oops_end+0xa7/0xb4
[78270.884588] [<ffffffff810323e4>] ? no_context+0x1e9/0x1f8
[78270.884595] [<ffffffff8100e242>] ? check_events+0x12/0x20
[78270.884602] [<ffffffff81032599>] ? __bad_area_nosemaphore+0x1a6/0x1ca
[78270.884609] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78270.884622] [<ffffffff8104e8ef>] ? vprintk+0x321/0x364
[78270.884629] [<ffffffff8100c205>] ?
__raw_callee_save_xen_pmd_val+0x11/0x1e
[78270.884641] [<ffffffff81031d07>] ? spurious_fault+0x12c/0x1c4
[78270.884649] [<ffffffff812fdc0f>] ? do_page_fault+0x69/0x2fc
[78270.884656] [<ffffffff812fbd25>] ? page_fault+0x25/0x30
[78270.884663] [<ffffffff81027e23>] ? _flat_send_IPI_mask+0x4b/0x78
[78270.884670] [<ffffffff81027e0d>] ? _flat_send_IPI_mask+0x35/0x78
[78270.884677] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[78270.884685] [<ffffffff8102506e>] ?
arch_trigger_all_cpu_backtrace+0x3c/0x69
[78270.884693] [<ffffffff812fb8b4>] ? _spin_lock_irqsave+0x15/0x34
[78270.884706] [<ffffffff8120c26d>] ? __handle_sysrq+0xaf/0x14b
[78270.884713] [<ffffffff8120cd3f>] ? hvc_poll+0x143/0x22c
[78270.884720] [<ffffffff8120cf9d>] ? hvc_handle_interrupt+0xc/0x1f
[78270.884731] [<ffffffff810945f0>] ? handle_IRQ_event+0x58/0x126
[78270.884743] [<ffffffff8105a798>] ? process_timeout+0x0/0x5
[78270.884750] [<ffffffff81095e34>] ? handle_edge_irq+0xc0/0x101
[78270.884761] [<ffffffff8101327f>] ? handle_irq+0x17/0x1d
[78270.884773] [<ffffffff811eeb8a>] ? __xen_evtchn_do_upcall+0xd5/0x15e
[78270.884780] [<ffffffff811eec4f>] ? xen_evtchn_do_upcall+0x2e/0x41
[78270.884788] [<ffffffff81011cfe>] ?
xen_do_hypervisor_callback+0x1e/0x30
[78270.884793] <EOI> [<ffffffff8100e22f>] ?
xen_restore_fl_direct_end+0x0/0x1
[78270.884804] [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001
[78270.884811] [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1001
[78270.884818] [<ffffffff8100dcb3>] ? xen_safe_halt+0xc/0x15
[78270.884825] [<ffffffff8100be63>] ? xen_idle+0x37/0x40
[78270.884831] [<ffffffff8100feb1>] ? cpu_idle+0xa2/0xda
[78270.887828] [<ffffffff814efcd1>] ? start_kernel+0x3dc/0x3e8
Whoopsy daisy.
--
2. That which causes joy or happiness.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel