th@llnl.gov
2006-Dec-29 21:12 UTC
[Lustre-devel] [Bug 11491] open_req->rq_type != LI_POISON asserts
Please don''t reply to lustre-devel. Instead, comment in Bugzilla by using the following link: https://bugzilla.lustre.org/show_bug.cgi?id=11491 Two alc/production client nodes (1.4.6.95_17.2llnl) hit this ASSERTION today. The ep0 messages might indicate an underlying elan problem. alc952: 2006-12-29 18:37:18 LustreError: 4656:0:(mdc_request.c:600:mdc_commit_close()) ASSERTION(open_req->rq_type != LI_POISON) failed 2006-12-29 18:37:18 LustreError: 4656:0:(linux-debug.c:130:lbug_with_loc()) LBUG 2006-12-29 18:37:18 Lustre: 4656:0:(linux-debug.c:155:libcfs_debug_dumpstack()) showing stack for process 4656 2006-12-29 18:37:18 run R running 4896 4656 4650 4657 4655 (NOTLB) 2006-12-29 18:37:18 fab8c06a de4f7d34 de4f7d48 c0106390 c02b31e3 c02b31e3 de4f7d20 00000190 2006-12-29 18:37:18 fab8a852 fab8c06a de4f7d5c fab81aff f6710880 e8d59c00 fd2780e5 de4f7d64 2006-12-29 18:37:18 fab87c31 de4f7d84 fd26c90b 00000258 e55e0a00 f65fc640 e224b200 e55e0a00 2006-12-29 18:37:18 Call Trace: 2006-12-29 18:37:18 [<c01063a8>] show_stack+0x76/0x7e 2006-12-29 18:37:18 [<fab81aff>] lbug_with_loc+0x8b/0xb2 [libcfs] 2006-12-29 18:37:18 [<fab87c31>] collect_pages_on_cpu+0x0/0x98 [libcfs] 2006-12-29 18:37:18 [<fd26c90b>] mdc_commit_close+0x28b/0x4f8 [mdc] 2006-12-29 18:37:18 [<fcf78e04>] ptlrpc_free_committed+0xbb8/0xc74 [ptlrpc] 2006-12-29 18:37:18 [<fcf74358>] after_reply+0x7b9/0x85c [ptlrpc] 2006-12-29 18:37:18 [<fcf7bc7f>] ptlrpc_queue_wait+0x21c6/0x2a78 [ptlrpc] 2006-12-29 18:37:18 [<fd26d249>] mdc_close+0x6d1/0xbfe [mdc] 2006-12-29 18:37:18 [<fd1ab55b>] ll_close_inode_openhandle+0x55b/0x7ea [llite] 2006-12-29 18:37:18 [<fd1ab9f2>] ll_mdc_real_close+0x208/0x35a [llite] 2006-12-29 18:37:18 [<fd1abdec>] ll_mdc_close+0x2a8/0x3e0 [llite] 2006-12-29 18:37:18 [<fd1ac19f>] ll_file_release+0x27b/0x30e [llite] 2006-12-29 18:37:18 [<c0156cd8>] __fput+0x56/0x104 2006-12-29 18:37:18 [<c01558f0>] filp_close+0x5b/0x65 2006-12-29 18:37:18 [<c02a8c4f>] syscall_call+0x7/0xb 2006-12-29 18:37:18 LDec 29 18:37:18 alc952 LustreError: 4656:0:(mdc_request.c:600:mdc_commit_close()) ASSERTION(openr_req->rq_type !LI_POISON) failed 2006-12-29 18:37:18 Dec 29 18:37:18 alc952 LustreError: 4656:0:(linux-debug.c:130:lbug_with_loc()u) LBUG 2006-12-29 18:37:18 mping log to /var/tmp/lustre-log.1167446238.4656 2006-12-29 18:37:19 Lustre: 4656:0:(linux-debug.c:96:libcfs_run_upcall()) Invoked LNET upcall /usr/lib/lustre/lnet_upcall LBUG,/tmp/root.10816/rpm/BUILD/lustre-1.4.6.95_17.2llnl/lnet/lib cfs/tracefile.c,libcfs_assertion_failed,400 2006-12-29 18:39:04 ep0[952]: manager thread stuck - scheduled 2006-12-29 18:39:04 ep0[952]: REJOINING at level 0 because of manager thread 2006-12-29 18:39:06 ep0[952]: Withdraw at Level 0 2006-12-29 18:39:06 ep0[952]: Withdraw at Level 1 2006-12-29 18:39:06 ep0[952]: Withdraw at Level 2 2006-12-29 18:39:06 ep0[952]: Withdraw at Level 3 2006-12-29 18:39:06 ep0[952]: Withdraw at Level 4 2006-12-29 18:39:06 ep0[952]: Withdraw from [953-955] 2006-12-29 18:39:06 ep0[952]: Withdraw from [944-951][956-959] 2006-12-29 18:39:06 ep0[952]: Withdraw from [864-943] 2006-12-29 18:39:06 ep0[952]: Withdraw from [768-863][960-1151] 2006-12-29 18:39:06 ep0[952]: Withdraw from [0-767][1152-1535] alc956: 2006-12-29 18:37:18 LustreError: 30851:0:(mdc_request.c:600:mdc_commit_close()) ASSERTION(open_req->rq_type != LI_POISON) failed 2006-12-29 18:37:18 LustreError: 30851:0:(linux-debug.c:130:lbug_with_loc()) LBUG 2006-12-29 18:37:18 Lustre: 30851:0:(linux-debug.c:155:libcfs_debug_dumpstack()) showing stack for process 30851 2006-12-29 18:37:18 run R running 4896 30851 30845 30852 30850 (NOTLB) 2006-12-29 18:37:18 fab8c06a d1625d34 d1625d48 c0106390 c02b31e3 c02b31e3 d1625d20 00000190 2006-12-29 18:37:18 fab8a852 fab8c06a d1625d5c fab81aff f0515b80 ca75f000 fd25a0e5 d1625d64 2006-12-29 18:37:18 fab87c31 d1625d84 fd24e90b 00000258 cc27e600 d5f7afc0 dd09ce00 cc27e600 2006-12-29 18:37:18 Call Trace: 2006-12-29 18:37:18 [<c01063a8>] show_stack+0x76/0x7e 2006-12-29 18:37:18 [<fab81aff>] lbug_with_loc+0x8b/0xb2 [libcfs] 2006-12-29 18:37:18 [<fab87c31>] collect_pages_on_cpu+0x0/0x98 [libcfs] 2006-12-29 18:37:18 [<fd24e90b>] mdc_commit_close+0x28b/0x4f8 [mdc] 2006-12-29 18:37:18 [<fcf78e04>] ptlrpc_free_committed+0xbb8/0xc74 [ptlrpc] 2006-12-29 18:37:18 [<fcf74358>] after_reply+0x7b9/0x85c [ptlrpc] 2006-12-29 18:37:18 [<fcf7bc7f>] ptlrpc_queue_wait+0x21c6/0x2a78 [ptlrpc] 2006-12-29 18:37:18 [<fd24f249>] mdc_close+0x6d1/0xbfe [mdc] 2006-12-29 18:37:18 [<fd18d55b>] ll_close_inode_openhandle+0x55b/0x7ea [llite] 2006-12-29 18:37:18 [<fd18d9f2>] ll_mdc_real_close+0x208/0x35a [llite] 2006-12-29 18:37:18 [<fd18ddec>] ll_mdc_close+0x2a8/0x3e0 [llite] 2006-12-29 18:37:18 [<fd18e19f>] ll_file_release+0x27b/0x30e [llite] 2006-12-29 18:37:18 [<c0156cd8>] __fput+0x56/0x104 2006-12-29 18:37:18 [<c01558f0>] filp_close+0x5b/0x65 2006-12-29 18:37:18 [<c02a8c4f>] syscall_call+0x7/0xb 2006-12-29 18:37:18 LustreErDec 29 18:37:18 ralc956 LustreError: 30851:0:(mdc_request.c:600:mdc_commit_close()) ASSERTION(open_req->rq_type !LI_POISON) failed 2006-12-29 18:37:18 Dec 29 18:37:18 alc956 LustreError: 30851:0:(linux-debug.c:130:lbug_with_locl()) LBUG 2006-12-29 18:37:18 og to /var/tmp/lustre-log.1167446238.30851 2006-12-29 18:37:21 Lustre: 30851:0:(linux-debug.c:96:libcfs_run_upcall()) Invoked LNET upcall /usr/lib/lustre/lnet_upcall LBUG,/tmp/root.10816/rpm/BUILD/lustre-1.4.6.95_17.2llnl/lnet/li bcfs/tracefile.c,libcfs_assertion_failed,400 2006-12-29 18:37:21 Lustre: 30851:0:(linux-debug.c:96:libcfs_run_upcall()) Skipped 16 previous similar messages 2006-12-29 18:38:34 ep0[956]: manager thread stuck - scheduled 2006-12-29 18:38:34 ep0[956]: REJOINING at level 0 because of manager thread 2006-12-29 18:38:36 ep0[956]: Withdraw at Level 0 2006-12-29 18:38:36 ep0[956]: Withdraw at Level 1 2006-12-29 18:38:36 ep0[956]: Withdraw at Level 2 2006-12-29 18:38:36 ep0[956]: Withdraw at Level 3 2006-12-29 18:38:36 ep0[956]: Withdraw at Level 4 2006-12-29 18:38:36 ep0[956]: Withdraw from [957-959] 2006-12-29 18:38:36 ep0[956]: Withdraw from [944-955] 2006-12-29 18:38:36 ep0[956]: Withdraw from [864-943] 2006-12-29 18:38:36 ep0[956]: Withdraw from [768-863][960-1151] 2006-12-29 18:38:37 ep0[956]: Withdraw from [0-767][1152-1535]