All Hopefully someone here can help me with figuring out this problem to start Linux nas-0-0.local 2.6.18-164.11.1.el5_lustre.1.8.2 #1 SMP Fri Jan 22 19:11:17 MST 2010 x86_64 x86_64 x86_64 GNU/Linux we are using a HW mirrored 10k sas drive for the MGS/MDS we have 1 DDN LUN assigned as an ost when we mount the mgs and ost we see allot of Apr 8 15:41:23 nas-0-0 kernel: LustreError: dumping log to /tmp/lustre-log.1270766483.6273 i cat the root at nas-0-0 home-MDT0000]# cat recovery_status /proc/fs/lustre/mds/home-MDT0000 status: COMPLETE recovery_start: 1270757997 recovery_duration: 12 delayed_clients: 0/119 completed_clients: 119/119 replayed_requests: 3492 last_transno: 64424512932 [root at nas-0-0 home-MDT0000]# but i cannot mount the file system, it hangs and or gives me an endpoint error we have run e2fsck and we are considering lfsck but according to the documentation you need to be able to mount the file system and we cant even do that here are some of the log file messages we are getting Apr 8 13:31:50 nas-0-0 kernel: LustreError: 6514:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 1 previous similar message Apr 8 13:34:03 nas-0-0 kernel: LustreError: 6474:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16) req at ffff810645d49400 x1331863949761197/t0 o38->71d90a65-46cf-1c5f-12b9-030426af8970 at NET_0x500000a640108_UUID:0/0 lens 368/264 e 0 to 0 dl 1270758943 ref 1 fl Interpret:/0/0 rc -16/0 Apr 8 13:34:03 nas-0-0 kernel: LustreError: 6474:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 18 previous similar messages Apr 8 13:38:23 nas-0-0 kernel: LustreError: 6518:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16) req at ffff81062ccae800 x1331863949761384/t0 o38->71d90a65-46cf-1c5f-12b9-030426af8970 at NET_0x500000a640108_UUID:0/0 lens 368/264 e 0 to 0 dl 1270759203 ref 1 fl Interpret:/0/0 rc -16/0 Apr 8 13:38:23 nas-0-0 kernel: LustreError: 6518:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 36 previous similar messages Apr 8 13:43:55 nas-0-0 kernel: LustreError: dumping log to /tmp/lustre-log.1270759435.6272 Apr 8 13:46:57 nas-0-0 kernel: LustreError: 6562:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16) req at ffff81065fba3400 x1331950603635300/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270759717 ref 1 fl Interpret:/0/0 rc -16/0 Apr 8 13:46:57 nas-0-0 kernel: LustreError: 6562:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 8497 previous similar messages Apr 8 13:55:58 nas-0-0 kernel: LustreError: 6273:0:(mgs_handler.c:660:mgs_handle()) MGS handle cmd=250 rc=-16 Apr 8 13:56:59 nas-0-0 kernel: LustreError: 6514:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16) req at ffff810632ef1400 x1331950603635761/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270760319 ref 1 fl Interpret:/0/0 rc -16/0 Apr 8 13:56:59 nas-0-0 kernel: LustreError: 6514:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10148 previous similar messages Apr 8 14:05:46 nas-0-0 kernel: LustreError: dumping log to /tmp/lustre-log.1270760746.6360 Apr 8 14:07:01 nas-0-0 kernel: LustreError: 6514:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16) req at ffff81065480b400 x1331950603636207/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270760921 ref 1 fl Interpret:/0/0 rc -16/0 Apr 8 14:07:01 nas-0-0 kernel: LustreError: 6514:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10147 previous similar messages Apr 8 14:17:03 nas-0-0 kernel: LustreError: 6299:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16) req at ffff810634515000 x1331950603636653/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270761523 ref 1 fl Interpret:/0/0 rc -16/0 Apr 8 14:17:03 nas-0-0 kernel: LustreError: 6299:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10147 previous similar messages Apr 8 14:27:05 nas-0-0 kernel: LustreError: 6316:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16) req at ffff81064ee71800 x1331950603637099/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270762125 ref 1 fl Interpret:/0/0 rc -16/0 Apr 8 14:27:05 nas-0-0 kernel: LustreError: 6316:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10147 previous similar messages Apr 8 14:37:07 nas-0-0 kernel: LustreError: 6306:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16) req at ffff81063e665c00 x1331950603637545/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270762727 ref 1 fl Interpret:/0/0 rc -16/0 Apr 8 14:37:07 nas-0-0 kernel: LustreError: 6306:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10147 previous similar messages Apr 8 14:47:09 nas-0-0 kernel: LustreError: 6306:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16) req at ffff810617725000 x1331950603637991/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270763329 ref 1 fl Interpret:/0/0 rc -16/0 Apr 8 14:47:09 nas-0-0 kernel: LustreError: 6306:0:(ldlm_lib.c:1848:target_send_reply_msg()) Skipped 10146 previous similar messages Apr 8 14:57:11 nas-0-0 kernel: LustreError: 6563:0:(ldlm_lib.c:1848:target_send_reply_msg()) @@@ processing error (-16) req at ffff81064cd85400 x1331950603638437/t0 o38->cff7f730-90bc-fc37-2daf-537dd35cd46d at NET_0x200000a01ff18_UUID:0/0 lens 368/264 e 0 to 0 dl 1270763931 ref 1 fl Interpret:/0/0 rc -16/0 Apr 8 14:57:1 Eric Adint ehadint at nps.edu HPC specialist Research Computing Naval Postgraduate School 833 Dyer Road Bldg 232 Room 139a Monterey Ca 93943 831-402-5996