Lu Wang
2009-Aug-31 01:35 UTC
[Lustre-discuss] Unable to handle kernel paging request at virtual address
Dear list, According to discussion thread http://groups.google.com/group/lustre-discuss-list/browse_thread/thread/a4517a537beb89f3?hl=en I reduce max_cached_mb=1024, and the clients crashed less frequently. However, we found 3 clients dead yesterday, with log: At the end of this log, from " _spin_unlock" to "kprobe_exceptions_notify" repeated several times, and then the node died. Is this caused by the same reason? Aug 30 15:17:55 bws0060 kernel: do_IRQ: stack overflow: 460 Aug 30 15:17:55 bws0060 kernel: [<c010795b>] do_IRQ+0x49/0x1ae Aug 30 15:17:55 bws0060 kernel: [<c02d6c60>] common_interrupt+0x18/0x20 Aug 30 15:17:55 bws0060 kernel: [<c01c3228>] number+0x148/0x25d Aug 30 15:17:55 bws0060 kernel: [<c011cd20>] recalc_task_prio+0x106/0x133 Aug 30 15:17:55 bws0060 kernel: [<c01c3785>] vsnprintf+0x448/0x488 Aug 30 15:17:55 bws0060 kernel: [<c01c37f3>] snprintf+0x17/0x1a Aug 30 15:17:55 bws0060 kernel: [<f94a08dc>] libcfs_ip_addr2str+0x3c/0x40 [libcfs] Aug 30 15:17:55 bws0060 kernel: [<f94a0d0b>] libcfs_nid2str+0x7b/0x140 [libcfs] Aug 30 15:17:55 bws0060 kernel: [<f94a105b>] libcfs_id2str+0x2b/0xb0 [libcfs] Aug 30 15:17:55 bws0060 kernel: [<f9832434>] ksocknal_queue_tx_locked+0x404/0x630 [ksocklnd] Aug 30 15:17:55 bws0060 kernel: [<f982727d>] ksocknal_find_peer_locked+0x14d/0x1b0 [ksocklnd] Aug 30 15:17:55 bws0060 kernel: [<f9832889>] ksocknal_launch_packet+0x139/0x5b0 [ksocklnd] Aug 30 15:17:55 bws0060 kernel: [<f9832e7d>] ksocknal_send+0x17d/0x430 [ksocklnd] Aug 30 15:17:55 bws0060 kernel: Unable to handle kernel paging request at virtual address 343ce120 Aug 30 15:17:55 bws0060 kernel: printing eip: Aug 30 15:17:55 bws0060 kernel: c011974d Aug 30 15:17:55 bws0060 kernel: *pde = 1cce7001 Aug 30 15:17:55 bws0060 kernel: Oops: 0000 [#1] Aug 30 15:17:55 bws0060 kernel: SMP Aug 30 15:17:55 bws0060 kernel: Modules linked in: mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) nfs lockd nfs_acl blcr(U) blcr_imports(U) libafs(U) md5 ipv6 autofs4 i2c_dev i2c_core sunrpc loop dm_mirror button battery ac uhci_hcd ehci_hcd hw_random bnx2 ext3 jbd dm_mod ata_piix libata mptscsih mptsas mptspi mptscsi mptbase sd_mod scsi_mod Aug 30 15:17:55 bws0060 kernel: CPU: 6 Aug 30 15:17:55 bws0060 kernel: EIP: 0060:[<c011974d>] Tainted: PF VLI Aug 30 15:17:55 bws0060 kernel: EFLAGS: 00010086 (2.6.9-55.EL.cernsmp) Aug 30 15:17:55 bws0060 kernel: EIP is at kprobe_exceptions_notify+0x126/0x1fc Aug 30 15:17:55 bws0060 kernel: eax: c03c21a0 ebx: c032ae3c ecx: d703b068 edx: 5d000000 Aug 30 15:17:55 bws0060 kernel: esi: d703b068 edi: d703b124 ebp: 80625fde esp: d703b030 Aug 30 15:17:55 bws0060 kernel: ds: 007b es: 007b ss: 0068 Aug 30 15:17:55 bws0060 kernel: Process ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ!^F?^E& (pid: 15158 Aug 30 15:17:55 bws0060 kernel: 0810, threadinfo=d703a000 t Aug 30 15:17:55 bws0060 kernel: Stack: 00000000 c032ae3c d703b068 0000000d 80625fde c012de7e c0453c20 00000000 Aug 30 15:17:55 bws0060 kernel: c011ae6d c011aebf c02f25d9 343ce120 c0122900 c02f25d9 d703b124 c02e766c Aug 30 15:17:55 bws0060 kernel: 00000000 0000000e 0000000b 0000017d f984530c 636f736b 6c616e6b 6e65735f Aug 30 15:17:55 bws0060 kernel: Call Trace: Aug 30 15:17:55 bws0060 kernel: [<c012de7e>] notifier_call_chain+0x17/0x2e Aug 30 15:17:55 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:55 bws0060 kernel: [<c011aebf>] do_page_fault+0x52/0x5c6 Aug 30 15:17:55 bws0060 kernel: [<c0122900>] printk+0xe/0x11 Aug 30 15:17:55 bws0060 kernel: [<c011cd42>] recalc_task_prio+0x128/0x133 Aug 30 15:17:55 bws0060 kernel: [<c011cd42>] recalc_task_prio+0x128/0x133 Aug 30 15:17:55 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:55 bws0060 kernel: [<c011cd42>] recalc_task_prio+0x128/0x133 Aug 30 15:17:55 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:55 bws0060 kernel: [<c02d6d7f>] error_code+0x2f/0x38 Aug 30 15:17:56 bws0060 kernel: [<c020007b>] send_break+0x37/0x5f Aug 30 15:17:56 bws0060 kernel: [<c0129dde>] __mod_timer+0x4a/0x10b Aug 30 15:17:56 bws0060 kernel: [<c020d9a4>] poke_blanked_console+0x8f/0x9a Aug 30 15:17:56 bws0060 kernel: [<c020cd45>] vt_console_print+0x294/0x2a5 Aug 30 15:17:56 bws0060 kernel: [<c020cab1>] vt_console_print+0x0/0x2a5 Aug 30 15:17:56 bws0060 kernel: [<c01226e3>] __call_console_drivers+0x36/0x40 Aug 30 15:17:56 bws0060 kernel: [<c01227fb>] call_console_drivers+0xb6/0xd8 Aug 30 15:17:56 bws0060 kernel: [<c0122aef>] release_console_sem+0x43/0xa9 Aug 30 15:17:56 bws0060 kernel: [<c0122a39>] vprintk+0x136/0x14a Aug 30 15:17:56 bws0060 kernel: [<f9832e7d>] ksocknal_send+0x17d/0x430 [ksocklnd] Aug 30 15:17:56 bws0060 kernel: [<c0122900>] printk+0xe/0x11 Aug 30 15:17:56 bws0060 kernel: [<c0105d03>] show_trace+0x44/0x6b Aug 30 15:17:56 bws0060 kernel: [<c0105db4>] dump_stack+0x11/0x13 Aug 30 15:17:56 bws0060 kernel: [<c010795b>] do_IRQ+0x49/0x1ae Aug 30 15:17:56 bws0060 kernel: [<c02d6c60>] common_interrupt+0x18/0x20 Aug 30 15:17:56 bws0060 kernel: [<c01c3228>] number+0x148/0x25d Aug 30 15:17:56 bws0060 kernel: [<c011cd20>] recalc_task_prio+0x106/0x133 Aug 30 15:17:56 bws0060 kernel: [<c01c3785>] vsnprintf+0x448/0x488 Aug 30 15:17:56 bws0060 kernel: [<c01c37f3>] snprintf+0x17/0x1a Aug 30 15:17:56 bws0060 kernel: [<f94a08dc>] libcfs_ip_addr2str+0x3c/0x40 [libcfs] Aug 30 15:17:56 bws0060 kernel: [<f94a0d0b>] libcfs_nid2str+0x7b/0x140 [libcfs] Aug 30 15:17:56 bws0060 kernel: [<f94a105b>] libcfs_id2str+0x2b/0xb0 [libcfs] Aug 30 15:17:56 bws0060 kernel: [<f9832434>] ksocknal_queue_tx_locked+0x404/0x630 [ksocklnd] Aug 30 15:17:56 bws0060 kernel: [<f982727d>] ksocknal_find_peer_locked+0x14d/0x1b0 [ksocklnd] Aug 30 15:17:56 bws0060 kernel: [<f9832889>] ksocknal_launch_packet+0x139/0x5b0 [ksocklnd] Aug 30 15:17:56 bws0060 kernel: [<f9832e7d>] ksocknal_send+0x17d/0x430 [ksocklnd] Aug 30 15:17:56 bws0060 kernel: [<f97194e1>] lnet_ni_send+0x41/0xd0 [lnet] Aug 30 15:17:56 bws0060 kernel: [<f971a4c1>] lnet_send+0x231/0xd20 [lnet] Aug 30 15:17:56 bws0060 kernel: [<f971f42d>] LNetPut+0x3fd/0xce0 [lnet] Aug 30 15:17:56 bws0060 kernel: [<f9661f30>] ptl_send_buf+0x2a0/0xa80 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<f9665801>] ptl_send_rpc+0x591/0x1790 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<c011e851>] __wake_up+0x29/0x3c Aug 30 15:17:56 bws0060 kernel: [<f965a22f>] ptlrpc_queue_wait+0x18f/0x2720 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<c011e851>] __wake_up+0x29/0x3c Aug 30 15:17:56 bws0060 kernel: [<f965a22f>] ptlrpc_queue_wait+0x18f/0x2720 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<f966db2e>] lustre_msg_add_version+0xbe/0x130 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<f949a359>] cfs_alloc+0x29/0x70 [libcfs] Aug 30 15:17:56 bws0060 kernel: [<f96679a3>] lustre_pack_request_v2+0x83/0x3c0 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<f960ad90>] ldlm_resource_putref+0xa0/0x680 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<f966fb2e>] lustre_msg_set_opc+0x2e/0x120 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<f949a359>] cfs_alloc+0x29/0x70 [libcfs] Aug 30 15:17:56 bws0060 kernel: [<f965d7bc>] ptlrpc_next_xid+0x3c/0x50 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<f967021e>] lustre_msg_set_timeout+0x2e/0x100 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<f966d726>] lustre_msg_get_type+0xd6/0x210 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<f97da4cb>] mdc_close+0x22b/0xdf0 [mdc] Aug 30 15:17:56 bws0060 kernel: [<f98490d3>] ll_release+0xd3/0x600 [lustre] Aug 30 15:17:56 bws0060 kernel: [<f985ada2>] ll_close_inode_openhandle+0x152/0xb80 [lustre] Aug 30 15:17:56 bws0060 kernel: [<f985b8fb>] ll_mdc_real_close+0x12b/0x520 [lustre] Aug 30 15:17:56 bws0060 kernel: [<f98b0504>] ll_mdc_blocking_ast+0x224/0x950 [lustre] Aug 30 15:17:56 bws0060 kernel: [<f9647ee5>] ldlm_pool_del+0x75/0x2f0 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<f95faac7>] ldlm_lock_destroy_nolock+0x87/0x1f0 [ptlrpc] Aug 30 15:17:56 bws0060 kernel: [<f966e0ae>] lustre_msg_get_last_committed+0xde/0x220 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f95f9118>] unlock_res_and_lock+0x58/0xe0 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f95f9118>] unlock_res_and_lock+0x58/0xe0 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f960261b>] ldlm_cancel_callback+0x10b/0x160 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f95f9045>] lock_res_and_lock+0x45/0xc0 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f96296b4>] ldlm_cli_cancel_local+0xa4/0x6f0 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f962bb77>] ldlm_cancel_list+0x137/0x360 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f9623f70>] ldlm_completion_ast+0x0/0xdf0 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f96242ab>] ldlm_completion_ast+0x33b/0xdf0 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f962bf32>] ldlm_cancel_lrur_policy+0x92/0x150 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f962c206>] ldlm_cancel_lru_local+0x126/0x480 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f95f9118>] unlock_res_and_lock+0x58/0xe0 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f960befb>] ldlm_resource_unlink_lock+0x4b/0xb0 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f962bea0>] ldlm_cancel_lrur_policy+0x0/0x150 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f96276f0>] ldlm_prep_elc_req+0x2d0/0x550 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f95fea07>] ldlm_lock_match+0x317/0x1010 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f96279ac>] ldlm_prep_enqueue_req+0x3c/0x50 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f97ea13f>] mdc_intent_lookup_pack+0xcf/0x160 [mdc] Aug 30 15:17:57 bws0060 kernel: [<f96279ac>] ldlm_prep_enqueue_req+0x3c/0x50 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f97ea13f>] mdc_intent_lookup_pack+0xcf/0x160 [mdc] Aug 30 15:17:57 bws0060 kernel: [<f9627e8c>] ldlm_cli_enqueue+0x4cc/0xc70 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f97eb65f>] mdc_enqueue+0x76f/0xd90 [mdc] Aug 30 15:17:57 bws0060 kernel: [<f955a01a>] class_handle2object+0x10a/0x2b0 [obdclass] Aug 30 15:17:57 bws0060 kernel: [<c0170bc8>] d_rehash+0x53/0x77 Aug 30 15:17:57 bws0060 kernel: [<f955a01a>] class_handle2object+0x10a/0x2b0 [obdclass] Aug 30 15:17:57 bws0060 kernel: [<f98b116b>] ll_d_add+0x7b/0x360 [lustre] Aug 30 15:17:57 bws0060 kernel: [<f98afa70>] ll_test_inode+0x0/0x430 [lustre] Aug 30 15:17:57 bws0060 kernel: [<f955a01a>] class_handle2object+0x10a/0x2b0 [obdclass] Aug 30 15:17:57 bws0060 kernel: [<f97ecd05>] mdc_intent_lock+0x1e5/0x690 [mdc] Aug 30 15:17:57 bws0060 kernel: [<f96575ad>] __ptlrpc_free_req+0x1ad/0xbc0 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f98b1ac8>] lookup_it_finish+0x1c8/0x720 [lustre] Aug 30 15:17:57 bws0060 kernel: [<f98b02e0>] ll_mdc_blocking_ast+0x0/0x950 [lustre] Aug 30 15:17:57 bws0060 kernel: [<f9623f70>] ldlm_completion_ast+0x0/0xdf0 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<c012fb15>] in_group_p+0x31/0x58 Aug 30 15:17:57 bws0060 kernel: [<f98b02e0>] ll_mdc_blocking_ast+0x0/0x950 [lustre] Aug 30 15:17:57 bws0060 kernel: [<f9623f70>] ldlm_completion_ast+0x0/0xdf0 [ptlrpc] Aug 30 15:17:57 bws0060 kernel: [<f98b1031>] ll_prepare_mdc_op_data+0x51/0x110 [lustre] Aug 30 15:17:57 bws0060 kernel: [<f98b24c7>] ll_lookup_it+0x4a7/0xc10 [lustre] Aug 30 15:17:57 bws0060 kernel: [<f98b02e0>] ll_mdc_blocking_ast+0x0/0x950 [lustre] Aug 30 15:17:57 bws0060 kernel: [<f98b02e0>] ll_mdc_blocking_ast+0x0/0x950 [lustre] Aug 30 15:17:57 bws0060 kernel: [<f94dd12b>] nfs_commit_write+0x69/0x72 [nfs] Aug 30 15:17:57 bws0060 kernel: [<c02d47e3>] __cond_resched+0x14/0x39 Aug 30 15:17:57 bws0060 kernel: [<f98b2c89>] ll_convert_intent+0x59/0x230 [lustre] Aug 30 15:17:57 bws0060 kernel: [<f98b2f54>] ll_lookup_nd+0xf4/0x510 [lustre] Aug 30 15:17:57 bws0060 kernel: [<c02d47e3>] __cond_resched+0x14/0x39 Aug 30 15:17:57 bws0060 kernel: [<c017066c>] d_alloc+0x175/0x17d Aug 30 15:17:57 bws0060 kernel: [<c0166d43>] real_lookup+0x6e/0xec Aug 30 15:17:57 bws0060 kernel: [<c0166f81>] do_lookup+0x5d/0xb1 Aug 30 15:17:57 bws0060 kernel: [<c0167819>] __link_path_walk+0x844/0xc25 Aug 30 15:17:57 bws0060 kernel: [<c0167c3d>] link_path_walk+0x43/0xbe Aug 30 15:17:57 bws0060 kernel: [<c01c402c>] atomic_dec_and_lock+0x20/0x40 Aug 30 15:17:57 bws0060 kernel: [<c010b052>] do_gettimeofday+0x1a/0x9c Aug 30 15:17:57 bws0060 kernel: [<c0167fd2>] path_lookup+0x14b/0x17f Aug 30 15:17:58 bws0060 kernel: [<c016811a>] __user_walk+0x21/0x51 Aug 30 15:17:57 bws0060 kernel: [<c0167fd2>] path_lookup+0x14b/0x17f Aug 30 15:17:58 bws0060 kernel: [<c016811a>] __user_walk+0x21/0x51 Aug 30 15:17:58 bws0060 kernel: [<c015a3e7>] sys_access+0x8f/0x134 Aug 30 15:17:58 bws0060 kernel: [<c01c402c>] atomic_dec_and_lock+0x20/0x40 Aug 30 15:17:58 bws0060 kernel: [<c010b052>] do_gettimeofday+0x1a/0x9c Aug 30 15:17:58 bws0060 kernel: [<c02d6287>] syscall_call+0x7/0xb Aug 30 15:17:58 bws0060 kernel: [<c02d007b>] unix_stream_sendmsg+0x33/0x33a Aug 30 15:17:58 bws0060 kernel: ======================Aug 30 15:17:58 bws0060 kernel: Unable to handle kernel paging request at virtual address 80625fde Aug 30 15:17:58 bws0060 kernel: printing eip: Aug 30 15:17:58 bws0060 kernel: c0105cd0 Aug 30 15:17:58 bws0060 kernel: *pde = 00000000 Aug 30 15:17:58 bws0060 kernel: Oops: 0000 [#2] Aug 30 15:17:58 bws0060 kernel: SMP Aug 30 15:17:58 bws0060 kernel: Modules linked in: mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) nfs lockd nfs_a Aug 30 15:17:58 bws0060 kernel: [<c02d6d7f>] error_code+0x2f/0x38 Aug 30 15:17:58 bws0060 kernel: [<c02d007b>] unix_stream_sendmsg+0x33/0x33a Aug 30 15:17:58 bws0060 kernel: [<c02d4e4c>] _spin_unlock+0x1c/0x27 Aug 30 15:17:58 bws0060 kernel: [<c02d392b>] schedule+0x7f/0x8ec Aug 30 15:17:58 bws0060 kernel: [<c020cab1>] vt_console_print+0x0/0x2a5 Aug 30 15:17:58 bws0060 kernel: [<c0129e95>] __mod_timer+0x101/0x10b Aug 30 15:17:58 bws0060 kernel: [<c02d4a06>] schedule_timeout+0x139/0x154 Aug 30 15:17:58 bws0060 kernel: [<c012a73a>] process_timeout+0x0/0x5 Aug 30 15:17:58 bws0060 kernel: [<c0122900>] printk+0xe/0x11 Aug 30 15:17:58 bws0060 kernel: [<c01060c2>] die+0x15a/0x16b Aug 30 15:17:58 bws0060 kernel: [<c0122b21>] release_console_sem+0x75/0xa9 Aug 30 15:17:58 bws0060 kernel: [<c0122a39>] vprintk+0x136/0x14a Aug 30 15:17:58 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:58 bws0060 kernel: [<c011b25d>] do_page_fault+0x3f0/0x5c6 Aug 30 15:17:58 bws0060 kernel: [<c0105cd0>] show_trace+0x11/0x6b Aug 30 15:17:58 bws0060 kernel: [<c020cab1>] vt_console_print+0x0/0x2a5 Aug 30 15:17:58 bws0060 kernel: [<c01226e3>] __call_console_drivers+0x36/0x40 Aug 30 15:17:58 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:58 bws0060 kernel: [<c01226e3>] __call_console_drivers+0x36/0x40 Aug 30 15:17:58 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:58 bws0060 kernel: [<c02d6d7f>] error_code+0x2f/0x38 Aug 30 15:17:58 bws0060 kernel: [<c0105cd0>] show_trace+0x11/0x6b Aug 30 15:17:58 bws0060 kernel: [<c0105d9d>] show_stack+0x73/0x79 Aug 30 15:17:58 bws0060 kernel: [<c0105e9c>] show_registers+0xe6/0x14d Aug 30 15:17:58 bws0060 kernel: [<c0106043>] die+0xdb/0x16b Aug 30 15:17:58 bws0060 kernel: [<c0122a39>] vprintk+0x136/0x14a Aug 30 15:17:58 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:58 bws0060 kernel: [<c011b25d>] do_page_fault+0x3f0/0x5c6 Aug 30 15:17:58 bws0060 kernel: [<c0105cd0>] show_trace+0x11/0x6b Aug 30 15:17:58 bws0060 kernel: [<c020cab1>] vt_console_print+0x0/0x2a5 Aug 30 15:17:58 bws0060 kernel: [<c01226e3>] __call_console_drivers+0x36/0x40 Aug 30 15:17:58 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:58 bws0060 kernel: [<c02d6d7f>] error_code+0x2f/0x38 Aug 30 15:17:58 bws0060 kernel: [<c0105cd0>] show_trace+0x11/0x6b Aug 30 15:17:58 bws0060 kernel: [<c0105d9d>] show_stack+0x73/0x79 Aug 30 15:17:58 bws0060 kernel: [<c0105e9c>] show_registers+0xe6/0x14d Aug 30 15:17:58 bws0060 kernel: [<c0106043>] die+0xdb/0x16b Aug 30 15:17:59 bws0060 kernel: [<c0106425>] do_invalid_op+0xcf/0xf2 Aug 30 15:17:59 bws0060 kernel: [<c02d4e4c>] _spin_unlock+0x1c/0x27 Aug 30 15:17:59 bws0060 kernel: [<c0106356>] do_invalid_op+0x0/0xf2 Aug 30 15:17:59 bws0060 kernel: [<c02d6d7f>] error_code+0x2f/0x38 Aug 30 15:17:59 bws0060 kernel: [<c02d007b>] unix_stream_sendmsg+0x33/0x33a Aug 30 15:17:59 bws0060 kernel: [<c02d4e4c>] _spin_unlock+0x1c/0x27 Aug 30 15:17:59 bws0060 kernel: [<c02d392b>] schedule+0x7f/0x8ec Aug 30 15:17:59 bws0060 kernel: [<c020cab1>] vt_console_print+0x0/0x2a5 Aug 30 15:17:59 bws0060 kernel: [<c0129e95>] __mod_timer+0x101/0x10b Aug 30 15:17:59 bws0060 kernel: [<c02d4a06>] schedule_timeout+0x139/0x154 Aug 30 15:17:59 bws0060 kernel: [<c012a73a>] process_timeout+0x0/0x5 Aug 30 15:17:59 bws0060 kernel: [<c0122900>] printk+0xe/0x11 Aug 30 15:17:59 bws0060 kernel: [<c01060c2>] die+0x15a/0x16b Aug 30 15:17:59 bws0060 kernel: [<c0122b21>] release_console_sem+0x75/0xa9 Aug 30 15:17:59 bws0060 kernel: [<c0122a39>] vprintk+0x136/0x14a Aug 30 15:17:59 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:59 bws0060 kernel: [<c011b25d>] do_page_fault+0x3f0/0x5c6 Aug 30 15:17:59 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:59 bws0060 kernel: [<c011b25d>] do_page_fault+0x3f0/0x5c6 Aug 30 15:17:59 bws0060 kernel: [<c0105cd0>] show_trace+0x11/0x6b Aug 30 15:17:59 bws0060 kernel: [<c020cab1>] vt_console_print+0x0/0x2a5 Aug 30 15:17:59 bws0060 kernel: [<c01226e3>] __call_console_drivers+0x36/0x40 Aug 30 15:17:59 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:59 bws0060 kernel: [<c02d6d7f>] error_code+0x2f/0x38 Aug 30 15:17:59 bws0060 hm[4031]: Server went down, finding new server. Aug 30 15:17:59 bws0060 kernel: [<c0105cd0>] show_trace+0x11/0x6b Aug 30 15:17:59 bws0060 kernel: [<c0105d9d>] show_stack+0x73/0x79 Aug 30 15:17:59 bws0060 kernel: [<c0105e9c>] show_registers+0xe6/0x14d Aug 30 15:17:59 bws0060 kernel: [<c0106043>] die+0xdb/0x16b Aug 30 15:17:59 bws0060 kernel: [<c0122a39>] vprintk+0x136/0x14a Aug 30 15:17:59 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:59 bws0060 kernel: [<c011b25d>] do_page_fault+0x3f0/0x5c6 Aug 30 15:17:59 bws0060 kernel: [<c0105cd0>] show_trace+0x11/0x6b Aug 30 15:17:59 bws0060 kernel: [<c020cab1>] vt_console_print+0x0/0x2a5 Aug 30 15:17:59 bws0060 kernel: [<c01226e3>] __call_console_drivers+0x36/0x40 Aug 30 15:17:59 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:59 bws0060 kernel: [<c02d6d7f>] error_code+0x2f/0x38 Aug 30 15:17:59 bws0060 kernel: [<c0105cd0>] show_trace+0x11/0x6b Aug 30 15:17:59 bws0060 kernel: [<c0105d9d>] show_stack+0x73/0x79 Aug 30 15:17:59 bws0060 kernel: [<c0105e9c>] show_registers+0xe6/0x14d Aug 30 15:17:59 bws0060 kernel: [<c0106043>] die+0xdb/0x16b Aug 30 15:17:59 bws0060 kernel: [<c0122a39>] vprintk+0x136/0x14a Aug 30 15:17:59 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:59 bws0060 kernel: [<c011b25d>] do_page_fault+0x3f0/0x5c6 Aug 30 15:17:59 bws0060 kernel: [<c011974d>] kprobe_exceptions_notify+0x126/0x1fc Aug 30 15:17:59 bws0060 kernel: [<c011cd42>] recalc_task_prio+0x128/0x133 Aug 30 15:17:59 bws0060 kernel: [<c011d2f4>] try_to_wake_up+0x28e/0x299 Aug 30 15:17:59 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:17:59 bws0060 kernel: [<c02d6d7f>] error_code+0x2f/0x38 Aug 30 15:17:59 bws0060 kernel: ======================Aug 30 15:17:59 bws0060 kernel: [<c02d4e4c>] _spin_unlock+0x1c/0x27 Aug 30 15:18:00 bws0060 kernel: [<c0106356>] do_invalid_op+0x0/0xf2 Aug 30 15:18:00 bws0060 kernel: [<c02d6d7f>] error_code+0x2f/0x38 Aug 30 15:18:00 bws0060 kernel: [<c02d007b>] unix_stream_sendmsg+0x33/0x33a Aug 30 15:18:00 bws0060 kernel: [<c02d4e4c>] _spin_unlock+0x1c/0x27 Aug 30 15:18:00 bws0060 kernel: [<c02d392b>] schedule+0x7f/0x8ec Aug 30 15:18:00 bws0060 kernel: [<c020cab1>] vt_console_print+0x0/0x2a5 Aug 30 15:18:00 bws0060 kernel: [<c0129e95>] __mod_timer+0x101/0x10b Aug 30 15:18:00 bws0060 kernel: [<c02d4a06>] schedule_timeout+0x139/0x154 Aug 30 15:18:00 bws0060 kernel: [<c012a73a>] process_timeout+0x0/0x5 Aug 30 15:18:00 bws0060 kernel: [<c0122900>] printk+0xe/0x11 Aug 30 15:18:00 bws0060 kernel: [<c01060c2>] die+0x15a/0x16b Aug 30 15:18:00 bws0060 kernel: [<c0122b21>] release_console_sem+0x75/0xa9 Aug 30 15:18:00 bws0060 kernel: [<c0122a39>] vprintk+0x136/0x14a Aug 30 15:18:00 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:18:00 bws0060 kernel: [<c011b25d>] do_page_fault+0x3f0/0x5c6 Aug 30 15:18:00 bws0060 kernel: [<c0105cd0>] show_trace+0x11/0x6b Aug 30 15:18:00 bws0060 kernel: [<c020cab1>] vt_console_print+0x0/0x2a5 Aug 30 15:18:00 bws0060 kernel: [<c01226e3>] __call_console_drivers+0x show_registers+0xe6/0x14d Aug 30 15:18:00 bws0060 kernel: [<c0106043>] die+0xdb/0x16b Aug 30 15:18:00 bws0060 kernel: [<c0122a39>] vprintk+0x136/0x14a Aug 30 15:18:00 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:18:00 bws0060 kernel: [<c011b25d>] do_page_fault+0x3f0/0x5c6 Aug 30 15:18:00 bws0060 kernel: [<c0105cd0>] show_trace+0x11/0x6b Aug 30 15:18:00 bws0060 kernel: [<c020cab1>] vt_console_print+0x0/0x2a5 Aug 30 15:18:00 bws0060 kernel: [<c01226e3>] __call_console_drivers+0x36/0x40 Aug 30 15:18:00 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:18:00 bws0060 kernel: [<c02d6d7f>] error_code+0x2f/0x38 Aug 30 15:18:00 bws0060 kernel: [<c0105cd0>] show_trace+0x11/0x6b Aug 30 15:18:00 bws0060 kernel: [<c0105d9d>] show_stack+0x73/0x79 Aug 30 15:18:00 bws0060 kernel: [<c0105e9c>] show_registers+0xe6/0x14d Aug 30 15:18:00 bws0060 kernel: [<c0106043>] die+0xdb/0x16b Aug 30 15:18:00 bws0060 kernel: [<c0122a39>] vprintk+0x136/0x14a Aug 30 15:18:00 bws0060 kernel: [<c011ae6d>] do_page_fault+0x0/0x5c6 Aug 30 15:18:00 bws0060 kernel: [<c011b25d>] do_page_fault+0x3f0/0x5c6 Aug 30 15:18:00 bws0060 kernel: [<c011974d>] kprobe_exceptions_notify+0x126/0x1fc ------------------ Best Regards Lu Wang -------------------------------------------------------------- Computing Center IHEP Office: Computing Center,123 19B Yuquan Road Tel: (+86) 10 88236012-607 P.O. Box 918-7 Fax: (+86) 10 8823 6839 Beijing 100049,China Email: Lu.Wang at ihep.ac.cn --------------------------------------------------------------