Hi Andreas,
I re-created the crash situation i mentioned in the last
email. The volume already has an rsync running.
I ran bonnie++ from another node -
./bonnie++
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...
done
Reading intelligently...done
start ''em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...
Can''t create file 7AArbw1J0012855
Cleaning up test directory after error.
In the "Create files in random order" stage, i do a few
ls
on the directory where bonnie is creating the small files and the
following happens.
Actually this happens eventually, even without trying ls on the
directory. The nfs re-exports hang until bonnie++ is terminated. The
iowait is already in the 50% range due to rsync and gets to 90% at times
when bonnie creates files in the random order. Is this caused by too
much torture to the file system or a genuine problem i would face later?
Thanks a lot.
Regards
Balagopal
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
3663:0:(lustre_fsfilt.h:205:fsfilt_start_log()) scratch-OST0000: slow
journal start 44s
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
6364:0:(lustre_fsfilt.h:205:fsfilt_start_log()) scratch-OST0000: slow
journal start 43s
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
5110:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) scratch-OST0000:
slow journal start 35s
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
5110:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) Skipped 4 previous
similar messages
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
5110:0:(filter_io_26.c:680:filter_commitrw_write()) scratch-OST0000:
slow brw_start 35s
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
5110:0:(filter_io_26.c:680:filter_commitrw_write()) Skipped 4 previous
similar messages
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
4857:0:(filter.c:1139:filter_parent_lock()) scratch-OST0000: slow parent
lock 43s
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
4857:0:(filter.c:1139:filter_parent_lock()) Skipped 6 previous similar
messages
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
4859:0:(filter_io.c:546:filter_preprw_write()) scratch-OST0000: slow
preprw_write setup 35s
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
4859:0:(filter_io.c:649:filter_preprw_write()) scratch-OST0000: slow
start_page_write 35s
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
5110:0:(filter_io_26.c:732:filter_commitrw_write()) scratch-OST0000:
slow direct_io 35s
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
5110:0:(filter_io_26.c:732:filter_commitrw_write()) Skipped 4 previous
similar messages
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
3664:0:(filter_io_26.c:745:filter_commitrw_write()) scratch-OST0000:
slow commitrw commit 35s
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
3664:0:(filter_io_26.c:745:filter_commitrw_write()) Skipped 3 previous
similar messages
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
3689:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) scratch-OST0000:
slow journal start 35s
Jun 21 16:37:32 lustre-3ware kernel: Lustre:
3689:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) Skipped 1 previous
similar message
Jun 21 16:37:33 lustre-3ware kernel: Lustre:
3689:0:(filter_io_26.c:680:filter_commitrw_write()) scratch-OST0000:
slow brw_start 35s
Jun 21 16:37:33 lustre-3ware kernel: Lustre:
3689:0:(filter_io_26.c:680:filter_commitrw_write()) Skipped 1 previous
similar message
Jun 21 16:37:33 lustre-3ware kernel: Lustre:
3689:0:(filter_io_26.c:732:filter_commitrw_write()) scratch-OST0000:
slow direct_io 35s
Jun 21 16:37:33 lustre-3ware kernel: Lustre:
3689:0:(filter_io_26.c:732:filter_commitrw_write()) Skipped 1 previous
similar message
Jun 21 16:37:33 lustre-3ware kernel: Lustre:
3689:0:(filter_io_26.c:745:filter_commitrw_write()) scratch-OST0000:
slow commitrw commit 35s
Jun 21 16:37:33 lustre-3ware kernel: Lustre:
3689:0:(filter_io_26.c:745:filter_commitrw_write()) Skipped 1 previous
similar message
Jun 21 16:37:33 lustre-3ware kernel: Lustre:
4908:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) scratch-OST0000:
slow journal start 35s
Jun 21 16:37:33 lustre-3ware kernel: Lustre:
4908:0:(filter_io_26.c:680:filter_commitrw_write()) scratch-OST0000:
slow brw_start 35s
Jun 21 16:37:33 lustre-3ware kernel: Lustre:
4908:0:(filter_io_26.c:732:filter_commitrw_write()) scratch-OST0000:
slow direct_io 35s
Jun 21 16:37:33 lustre-3ware kernel: Lustre:
3684:0:(filter_io_26.c:745:filter_commitrw_write()) scratch-OST0000:
slow commitrw commit 35s
Jun 21 16:37:33 lustre-3ware kernel: Lustre:
3684:0:(filter_io_26.c:745:filter_commitrw_write()) Skipped 1 previous
similar message
Jun 21 16:40:08 lustre-3ware kernel: LustreError:
3396:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent at
1182454758, 50s ago) req@000001002b92d600 x13158823/t0
o5->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 336/336 ref 1 fl
Rpc:/0/0 rc 0/-22
Jun 21 16:40:08 lustre-3ware kernel: Lustre: scratch-OST0000-osc:
Connection to service scratch-OST0000 via nid 0@lo was lost; in progress
operations using this service will wait for recovery to complete.
Jun 21 16:40:08 lustre-3ware kernel: Lustre:
3647:0:(ldlm_lib.c:497:target_handle_reconnect()) scratch-OST0000:
scratch-mdtlov_UUID reconnecting
Jun 21 16:40:08 lustre-3ware kernel: Lustre:
3647:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 8 previous
similar messages
Jun 21 16:40:08 lustre-3ware kernel: Lustre:
3647:0:(ldlm_lib.c:709:target_handle_connect()) scratch-OST0000: refuse
reconnection from scratch-mdtlov_UUID@0@lo to 0x00000100716f3000/2
Jun 21 16:40:08 lustre-3ware kernel: Lustre:
3647:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 2 previous
similar messages
Jun 21 16:40:08 lustre-3ware kernel: LustreError:
3647:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error
(-16) req@0000010051619000 x13158900/t0
o8->scratch-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl
Interpret:/0/0 rc -16/0
Jun 21 16:40:08 lustre-3ware kernel: LustreError:
3647:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 10291 previous
similar messages
Jun 21 16:40:08 lustre-3ware kernel: LustreError:
3397:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR,
err == -16 req@000001001f138600 x13158900/t0
o8->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl
Rpc:R/0/0 rc 0/-16
Jun 21 16:40:33 lustre-3ware kernel: Lustre:
6422:0:(ldlm_lib.c:709:target_handle_connect()) scratch-OST0000: refuse
reconnection from scratch-mdtlov_UUID@0@lo to 0x00000100716f3000/2
Jun 21 16:40:33 lustre-3ware kernel: LustreError:
3397:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR,
err == -16 req@000001005fee0a00 x13158903/t0
o8->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl
Rpc:R/0/0 rc 0/-16
Jun 21 16:40:38 lustre-3ware kernel: LustreError:
3671:0:(lustre_fsfilt.h:205:fsfilt_start_log()) scratch-OST0000: slow
journal start 80s
Jun 21 16:40:38 lustre-3ware kernel: LustreError:
4871:0:(filter.c:1139:filter_parent_lock()) scratch-OST0000: slow parent
lock 80s
Jun 21 16:40:38 lustre-3ware kernel: LustreError:
4871:0:(filter.c:1139:filter_parent_lock()) Skipped 11 previous similar
messages
Jun 21 16:40:38 lustre-3ware kernel: Lustre:
3663:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-0@lo
portal 4 match 13158823 offset 0 length 336: 2
Jun 21 16:40:38 lustre-3ware kernel: LustreError:
3680:0:(filter.c:1139:filter_parent_lock()) scratch-OST0000: slow parent
lock 81s
Jun 21 16:40:38 lustre-3ware kernel: LustreError:
3680:0:(filter.c:1139:filter_parent_lock()) Skipped 21 previous similar
messages
Jun 21 16:40:38 lustre-3ware kernel: LustreError:
4866:0:(filter.c:1139:filter_parent_lock()) scratch-OST0000: slow parent
lock 81s
Jun 21 16:40:38 lustre-3ware kernel: LustreError:
4866:0:(filter.c:1139:filter_parent_lock()) Skipped 13 previous similar
messages
Jun 21 16:40:46 lustre-3ware kernel: Lustre:
6368:0:(ldlm_lib.c:497:target_handle_reconnect()) scratch-OST0000:
e98c0aac-519a-287c-4ea9-f19b3ff2fce3 reconnecting
Jun 21 16:40:46 lustre-3ware kernel: Lustre:
6368:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 2 previous
similar messages
Jun 21 16:40:46 lustre-3ware kernel: LustreError:
6368:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error
(-16) req@0000010003fd9e00 x58933061/t0
o8->e98c0aac-519a-287c-4ea9-f19b3ff2fce3@NET_0x20000c0a80014_UUID:-1
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
Jun 21 16:40:46 lustre-3ware kernel: LustreError:
6368:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 2 previous
similar messages
Jun 21 16:40:58 lustre-3ware kernel: Lustre: scratch-OST0000-osc:
Connection restored to service scratch-OST0000 using nid 0@lo.
Jun 21 16:40:58 lustre-3ware kernel: Lustre: scratch-OST0000: received
MDS connection from 0@lo
Jun 21 16:40:58 lustre-3ware kernel: Lustre:
6361:0:(recov_thread.c:565:llog_repl_connect()) llcd
000001007b0d8000:000001007453e180 not empty
Jun 21 16:40:58 lustre-3ware kernel: Lustre: MDS scratch-MDT0000:
scratch-OST0000_UUID now active, resetting orphans
Jun 21 16:40:58 lustre-3ware kernel: Lustre:
6341:0:(filter.c:2600:filter_destroy_precreated()) scratch-OST0000:
deleting orphan objects from 5730490 to 5730521
Jun 21 16:41:25 lustre-3ware kernel: Lustre:
3637:0:(ldlm_lib.c:709:target_handle_connect()) scratch-OST0000: refuse
reconnection from f1855140-85d9-2aee-5238-5b3af18f9a3a@192.168.0.22@tcp
to 0x0000010071af4000/125
Jun 21 16:41:25 lustre-3ware kernel: Lustre:
3637:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 3 previous
similar messages
Jun 21 16:41:29 lustre-3ware kernel: LustreError:
0:0:(ldlm_lockd.c:213:waiting_locks_callback()) ### lock callback timer
expired: evicting client
f1855140-85d9-2aee-5238-5b3af18f9a3a@NET_0x20000c0a80016_UUID nid
192.168.0.22@tcp ns: filter-scratch-OST0000_UUID lock:
0000010043563680/0x24c4cddb1c3cd11c lrc: 1/0,0 mode: PR/PR res:
5724232/0 rrc: 2 type: EXT [0->18446744073709551615] (req
0->18446744073709551615) flags: 10020 remote: 0xbc86ffbdff7b731b expref:
650 pid: 3659
Jun 21 16:41:29 lustre-3ware kernel: LustreError:
0:0:(ldlm_lockd.c:213:waiting_locks_callback()) Skipped 42 previous
similar messages
Jun 21 16:41:35 lustre-3ware kernel: Lustre:
3663:0:(lustre_fsfilt.h:205:fsfilt_start_log()) scratch-OST0000: slow
journal start 32s
Jun 21 16:41:35 lustre-3ware kernel: LustreError:
4861:0:(service.c:616:ptlrpc_server_handle_request()) Dropping timed-out
opc 6 request from 12345-192.168.0.22@tcp: 137 seconds old
Jun 21 16:41:35 lustre-3ware kernel: LustreError:
4861:0:(service.c:616:ptlrpc_server_handle_request()) Skipped 239
previous similar messages
Jun 21 16:41:35 lustre-3ware kernel: LustreError:
4861:0:(service.c:590:ptlrpc_server_handle_request()) @@@ DROPPING req
from old connection 9 < 11 req@0000010049d20850 x58933057/t0
o4->e98c0aac-519a-287c-4ea9-f19b3ff2fce3@NET_0x20000c0a80014_UUID:-1
lens 384/0 ref 0 fl New:/0/0 rc 0/0
Jun 21 16:41:35 lustre-3ware kernel: LustreError:
4861:0:(service.c:590:ptlrpc_server_handle_request()) @@@ DROPPING req
from old connection 9 < 11 req@000001000aa58a00 x58933058/t0
o4->e98c0aac-519a-287c-4ea9-f19b3ff2fce3@NET_0x20000c0a80014_UUID:-1
lens 384/0 ref 0 fl New:/0/0 rc 0/0
Jun 21 16:43:07 lustre-3ware kernel: LustreError:
3396:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent at
1182454937, 50s ago) req@000001004f4cf400 x13159834/t0
o5->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 336/336 ref 1 fl
Rpc:/0/0 rc 0/-22
Jun 21 16:43:07 lustre-3ware kernel: Lustre: scratch-OST0000-osc:
Connection to service scratch-OST0000 via nid 0@lo was lost; in progress
operations using this service will wait for recovery to complete.
Jun 21 16:43:07 lustre-3ware kernel: Lustre:
3641:0:(ldlm_lib.c:497:target_handle_reconnect()) scratch-OST0000:
scratch-mdtlov_UUID reconnecting
Jun 21 16:43:07 lustre-3ware kernel: Lustre:
3641:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 4 previous
similar messages
Jun 21 16:43:07 lustre-3ware kernel: Lustre:
3641:0:(ldlm_lib.c:709:target_handle_connect()) scratch-OST0000: refuse
reconnection from scratch-mdtlov_UUID@0@lo to 0x00000100716f3000/2
Jun 21 16:43:07 lustre-3ware kernel: LustreError:
3641:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error
(-16) req@00000100137ea400 x13159855/t0
o8->scratch-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl
Interpret:/0/0 rc -16/0
Jun 21 16:43:07 lustre-3ware kernel: LustreError:
3641:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 2 previous
similar messages
Jun 21 16:43:07 lustre-3ware kernel: LustreError:
3397:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR,
err == -16 req@000001002e188e00 x13159855/t0
o8->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl
Rpc:R/0/0 rc 0/-16
Jun 21 16:43:32 lustre-3ware kernel: LustreError:
3397:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR,
err == -16 req@000001000c8ec000 x13159858/t0
o8->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl
Rpc:R/0/0 rc 0/-16
Jun 21 16:43:57 lustre-3ware kernel: Lustre:
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 3663: it was
inactive for 100s
Jun 21 16:43:57 lustre-3ware kernel: Lustre:
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for
process 3663
Jun 21 16:43:57 lustre-3ware kernel: ll_ost_creat_ D
00000100714038f8 0 3663 1 3664 3662 (L-TLB)
Jun 21 16:43:57 lustre-3ware kernel: 00000100743dd788 0000000000000046
000001007413c030 ffffffff00000073
Jun 21 16:43:57 lustre-3ware kernel: 00000100743dd708
00000000743dd708 0000010001021aa0 0000000000000000
Jun 21 16:43:57 lustre-3ware kernel: 000001007413c030
000000000000016d
Jun 21 16:43:57 lustre-3ware kernel: Call
Trace:<ffffffff80308223>{__down+147}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:43:57 lustre-3ware kernel:
<ffffffff80309cbb>{__down_failed+53}
<ffffffffa0063155>{:jbd:.text.lock.checkpoint+5}
Jun 21 16:43:57 lustre-3ware kernel:
<ffffffffa005e3c3>{:jbd:start_this_handle+963}
<ffffffffa03dffeb>{:ldiskfs:iopen_connect_dentry+679}
Jun 21 16:43:57 lustre-3ware kernel:
<ffffffff80134b62>{autoremove_wake_function+0} ll_mdt_10 S
0000000000000000 0 3556 1 3557 3555 (L-TLB)
Jun 21 16:43:57 lustre-3ware kernel: 00000100748dd238 0000000000000046
0000003000000030 0000010000000073
Jun 21 16:43:59 lustre-3ware kernel: 00000100748dd1c8
0000000000000246 0000010001021aa0 0000000000000008
Jun 21 16:44:05 lustre-3ware kernel: 00000100774ca030
0000000000001f3a
Jun 21 16:44:14 lustre-3ware kernel: Call
Trace:<ffffffffa049e528>{:osc:oscc_has_objects+259}<ffffffff80134b62>{autoremove_wake_function+0}
Jun 21 16:44:25 lustre-3ware kernel:
<ffffffffa005e51e>{:jbd:journal_start+223}
<ffffffffa04293e2>{:fsfilt_ldiskfs:fsfilt_ldiskfs_start+682}
Jun 21 16:44:34 lustre-3ware kernel:
<ffffffffa04cd24b>{:obdfilter:filter_precreate+2066}
Jun 21 16:44:44 lustre-3ware kernel:
<ffffffffa04cfbb7>{:obdfilter:filter_create+3455}
<ffffffffa01e206b>{:lnet:LNetPut+1425}
Jun 21 16:44:54 lustre-3ware kernel:
<ffffffffa0297383>{:ptlrpc:ptl_send_buf+727}
<ffffffffa029a5a8>{:ptlrpc:lustre_msg_add_version+67}
Jun 21 16:44:56 lustre-3ware kernel:
<ffffffffa029adc5>{:ptlrpc:lustre_pack_reply+1928}
Jun 21 16:45:05 lustre-3ware kernel:
<ffffffffa04b3609>{:ost:ost_handle+3156}
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:45:16 lustre-3ware kernel:
<ffffffffa049e9a6>{:osc:oscc_precreate+580}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa049f86b>{:osc:osc_create+3297}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0364a76>{:lov:qos_prep_create+3728}
<ffffffffa035bbe4>{:lov:lov_prep_create_set+541}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0348d59>{:lov:lov_create+6516}
<ffffffffa03dd4af>{:ldiskfs:ldiskfs_xattr_ibody_get+403}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0429f3d>{:fsfilt_ldiskfs:fsfilt_ldiskfs_get_md+101}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04415c1>{:mds:mds_get_md+105}
<ffffffffa046155b>{:mds:mds_create_objects+3818}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04421c2>{:mds:mds_pack_md+409}
<ffffffffa04631aa>{:mds:mds_finish_open+704}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0465625>{:mds:mds_open+6486}
<ffffffffa0459c91>{:mds:mds_reint_rec+373}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0443e26>{:mds:mds_reint+637}
<ffffffffa044c22a>{:mds:mds_intent_policy+890}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0278eee>{:ptlrpc:ldlm_resource_putref+356}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0274ca2>{:ptlrpc:ldlm_lock_create+1375}
<ffffffffa0275d85>{:ptlrpc:ldlm_lock_enqueue+208}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02887d7>{:ptlrpc:ldlm_handle_enqueue+2524}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02871ba>{:ptlrpc:ldlm_server_blocking_ast+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa028772d>{:ptlrpc:ldlm_server_completion_ast+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04483a9>{:mds:mds_handle+15254}
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801e9ce7>{vsnprintf+1406}
<ffffffff801e9ce7>{vsnprintf+1406}
<ffffffff801e9dca>{snprintf+131}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a1e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f100>{__mod_timer+293}
<ffffffffa02a3d1b>{:ptlrpc:ptlrpc_main+2018}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
<ffffffff80110e23>{child_rip+8}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a3539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jun 21 16:45:17 lustre-3ware kernel:
Jun 21 16:45:17 lustre-3ware kernel: ll_mdt_20 S
0000000000000000 0 3566 1 <ffffffff801e9dca>{snprintf+131}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a1e49>{:ptlrpc:ptlrpc_server_handle_request+2514}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a1e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jun 21 16:45:17 lustre-3ware kernel: 3567 3565 (L-TLB)
Jun 21 16:45:17 lustre-3ware kernel: 0000010074ed7238 0000000000000046
0000003000000030 0000010074ed72e0
Jun 21 16:45:17 lustre-3ware kernel: 0000010074ed71c8
00000000801e9dca 0000010000000076 0000000100000008
Jun 21 16:45:17 lustre-3ware kernel: 0000010078e2e030
000000000000216a
Jun 21 16:45:17 lustre-3ware kernel: Call
Trace:<ffffffffa049e528>{:osc:oscc_has_objects+259}
<ffffffffa049e9a6>{:osc:oscc_precreate+580}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f100>{__mod_timer+293}
<ffffffffa02a3d1b>{:ptlrpc:ptlrpc_main+2018}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa049f86b>{:osc:osc_create+3297}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0364a76>{:lov:qos_prep_create+3728}
<ffffffffa035bbe4>{:lov:lov_prep_create_set+541}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0348d59>{:lov:lov_create+6516}
<ffffffffa03dd4af>{:ldiskfs:ldiskfs_xattr_ibody_get+403}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0429f3d>{:fsfilt_ldiskfs:fsfilt_ldiskfs_get_md+101}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04415c1>{:mds:mds_get_md+105}
<ffffffffa046155b>{:mds:mds_create_objects+3818}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
<ffffffffa04421c2>{:mds:mds_pack_md+409}<ffffffff80110e23>{child_rip+8}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a3539>{:ptlrpc:ptlrpc_main+0}
<ffffffffa04631aa>{:mds:mds_finish_open+704}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80110e1b>{child_rip+0}
Jun 21 16:45:17 lustre-3ware kernel:
Jun 21 16:45:17 lustre-3ware kernel: ll_ost_io_12 D
000000000627d866 0 3675 1 3676 3674 (L-TLB)
Jun 21 16:45:17 lustre-3ware kernel: 0000010073097638 0000000000000046
000001004b099048 0000010000000073
Jun 21 16:45:17 lustre-3ware kernel: 0000000000000000
0000000000000001 0000010001021aa0 000000007af136c0
Jun 21 16:45:17 lustre-3ware kernel: 000001007375c030
00000000000000ad
Jun 21 16:45:17 lustre-3ware kernel: Call
Trace:<ffffffffa0465625>{:mds:mds_open+6486}
<ffffffffa0459c91>{:mds:mds_reint_rec+373}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0443e26>{:mds:mds_reint+637}
<ffffffffa044c22a>{:mds:mds_intent_policy+890}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0278eee>{:ptlrpc:ldlm_resource_putref+356}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80308f91>{thread_return+88}
<ffffffff80131551>{recalc_task_prio+337}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa0274ca2>{:ptlrpc:ldlm_lock_create+1375}
<ffffffffa0275d85>{:ptlrpc:ldlm_lock_enqueue+208}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02887d7>{:ptlrpc:ldlm_handle_enqueue+2524}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02871ba>{:ptlrpc:ldlm_server_blocking_ast+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa028772d>{:ptlrpc:ldlm_server_completion_ast+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04483a9>{:mds:mds_handle+15254}
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801e9ce7>{vsnprintf+1406} <ffffffff80308223>{__down+147}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801e9dca>{snprintf+131}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a1e49>{:ptlrpc:ptlrpc_server_handle_request+2514}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a1e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f100>{__mod_timer+293}
<ffffffffa02a3d1b>{:ptlrpc:ptlrpc_main+2018}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
<ffffffff80110e23>{child_rip+8}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a3539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jun 21 16:45:17 lustre-3ware kernel:
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80309cbb>{__down_failed+53}
<ffffffffa0063155>{:jbd:.text.lock.checkpoint+5}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa005e3c3>{:jbd:start_this_handle+963}
<ffffffffa022325d>{:obdclass:lprocfs_oh_tally+30}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80134b62>{autoremove_wake_function+0}
<ffffffff80134b62>{autoremove_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa005e51e>{:jbd:journal_start+223}
<ffffffffa04296f2>{:fsfilt_ldiskfs:fsfilt_ldiskfs_brw_start+649}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04d8f9f>{:obdfilter:filter_commitrw_write+1480}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80308f91>{thread_return+88}
<ffffffffa04d3756>{:obdfilter:filter_commitrw+84}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f23f>{del_timer+107}
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80309a63>{schedule_timeout+375}
<ffffffffa04b0db1>{:ost:ost_brw_write+5253}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa029a5a8>{:ptlrpc:lustre_msg_add_version+67}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04ad513>{:ost:ost_bulk_timeout+0}
<ffffffffa029b471>{:ptlrpc:lustre_msg_get_version+64}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04b4500>{:ost:ost_handle+6987}
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801e9ce7>{vsnprintf+1406} <ffffffff801e9dca>{snprintf+131}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a1e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f100>{__mod_timer+293}
<ffffffffa02a3d1b>{:ptlrpc:ptlrpc_main+2018}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
<ffffffff80110e23>{child_rip+8}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a3539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jun 21 16:45:17 lustre-3ware kernel:
Jun 21 16:45:17 lustre-3ware kernel: ll_ost_io_78 D
00000100714038f8 0 4902 1 4903 4901 (L-TLB)
Jun 21 16:45:17 lustre-3ware kernel: 00000100053a9638 0000000000000046
00000100245bbca8 0000010000000073
Jun 21 16:45:17 lustre-3ware kernel: 0000000000000000
0000000000000206 0000010001021aa0 000000007af136c0
Jun 21 16:45:17 lustre-3ware kernel: 000001000abe5800
00000000000000a7
Jun 21 16:45:17 lustre-3ware kernel: Call
Trace:<ffffffff80308f91>{thread_return+88}
<ffffffff80131551>{recalc_task_prio+337}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80308223>{__down+147}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80309cbb>{__down_failed+53}
<ffffffffa0063155>{:jbd:.text.lock.checkpoint+5}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa005e3c3>{:jbd:start_this_handle+963}
<ffffffffa022325d>{:obdclass:lprocfs_oh_tally+30}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80134b62>{autoremove_wake_function+0}
<ffffffff80134b62>{autoremove_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa005e51e>{:jbd:journal_start+223}
<ffffffffa04296f2>{:fsfilt_ldiskfs:fsfilt_ldiskfs_brw_start+649}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04d8f9f>{:obdfilter:filter_commitrw_write+1480}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04d3756>{:obdfilter:filter_commitrw+84}
<ffffffff8013f23f>{del_timer+107}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
<ffffffff80309a63>{schedule_timeout+375}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04b0db1>{:ost:ost_brw_write+5253}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04ad513>{:ost:ost_bulk_timeout+0}
<ffffffffa029b471>{:ptlrpc:lustre_msg_get_version+64}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04b4500>{:ost:ost_handle+6987}
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801e9ce7>{vsnprintf+1406} <ffffffff801e9dca>{snprintf+131}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a1e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f100>{__mod_timer+293}
<ffffffffa02a3d1b>{:ptlrpc:ptlrpc_main+2018}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
<ffffffff80110e23>{child_rip+8}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a3539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jun 21 16:45:17 lustre-3ware kernel:
Jun 21 16:45:17 lustre-3ware kernel: ll_ost_io_20 D
00000100714038f8 0 3683 1 3684 3682 (L-TLB)
Jun 21 16:45:17 lustre-3ware kernel: 00000100728f1638 0000000000000046
0000010004b99db0 0000010000000073
Jun 21 16:45:17 lustre-3ware kernel: 0000000000000000
0000000071403800 0000010001021aa0 000000007af136c0
Jun 21 16:45:17 lustre-3ware kernel: 0000010072fb5030
00000000000000a2
Jun 21 16:45:17 lustre-3ware kernel: Call
Trace:<ffffffff80308f91>{thread_return+88}
<ffffffff80131551>{recalc_task_prio+337}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80308223>{__down+147}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel: <1>LustreError: dumping log
to /tmp/lustre-log.1182455037.4902
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80309cbb>{__down_failed+53}
<ffffffffa0063155>{:jbd:.text.lock.checkpoint+5}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa005e3c3>{:jbd:start_this_handle+963}
<ffffffffa022325d>{:obdclass:lprocfs_oh_tally+30}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80134b62>{autoremove_wake_function+0}
<ffffffff80134b62>{autoremove_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa005e51e>{:jbd:journal_start+223}
<ffffffffa04296f2>{:fsfilt_ldiskfs:fsfilt_ldiskfs_brw_start+649}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04d8f9f>{:obdfilter:filter_commitrw_write+1480}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04d3756>{:obdfilter:filter_commitrw+84}
<ffffffff8013f23f>{del_timer+107}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
<ffffffff80309a63>{schedule_timeout+375}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04b0db1>{:ost:ost_brw_write+5253}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa029a5a8>{:ptlrpc:lustre_msg_add_version+67}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04ad513>{:ost:ost_bulk_timeout+0}
<ffffffffa029b471>{:ptlrpc:lustre_msg_get_version+64}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04b4500>{:ost:ost_handle+6987}
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801e9ce7>{vsnprintf+1406} <ffffffff801e9dca>{snprintf+131}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a1e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f100>{__mod_timer+293}
<ffffffffa02a3d1b>{:ptlrpc:ptlrpc_main+2018}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
<ffffffff80110e23>{child_rip+8}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a3539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jun 21 16:45:17 lustre-3ware kernel:
Jun 21 16:45:17 lustre-3ware kernel: ll_ost_io_19 D
00000100714038f8 0 3682 1 3683 3681 (L-TLB)
Jun 21 16:45:17 lustre-3ware kernel: 0000010072fe5638 0000000000000046
000001007d1c5b40 0000000000000073
Jun 21 16:45:17 lustre-3ware kernel: 0000000000000000
0000000000000001 0000010001021aa0 000000007af136c0
Jun 21 16:45:17 lustre-3ware kernel: 0000010072fb5800
000000000000009c
Jun 21 16:45:17 lustre-3ware kernel: Call
Trace:<ffffffff80308f91>{thread_return+88}
<ffffffff80131551>{recalc_task_prio+337}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80308223>{__down+147}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80309cbb>{__down_failed+53}
<ffffffffa0063155>{:jbd:.text.lock.checkpoint+5}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa005e3c3>{:jbd:start_this_handle+963}
<ffffffffa022325d>{:obdclass:lprocfs_oh_tally+30}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80134b62>{autoremove_wake_function+0}
<ffffffff80134b62>{autoremove_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa005e51e>{:jbd:journal_start+223}
<ffffffffa04296f2>{:fsfilt_ldiskfs:fsfilt_ldiskfs_brw_start+649}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04d8f9f>{:obdfilter:filter_commitrw_write+1480}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04d3756>{:obdfilter:filter_commitrw+84}
<ffffffff8013f23f>{del_timer+107}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
<ffffffff80309a63>{schedule_timeout+375}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04b0db1>{:ost:ost_brw_write+5253}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04ad513>{:ost:ost_bulk_timeout+0}
<ffffffffa029b471>{:ptlrpc:lustre_msg_get_version+64}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04b4500>{:ost:ost_handle+6987}
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801e9ce7>{vsnprintf+1406} <ffffffff801e9dca>{snprintf+131}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a1e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f100>{__mod_timer+293}
<ffffffffa02a3d1b>{:ptlrpc:ptlrpc_main+2018}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
<ffffffff80110e23>{child_rip+8}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa02a3539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jun 21 16:45:17 lustre-3ware kernel:
Jun 21 16:45:17 lustre-3ware kernel: ll_ost_io_105 D
00000100714038f8 0 4929 1 4930 4928 (L-TLB)
Jun 21 16:45:17 lustre-3ware kernel: 0000010026951638 0000000000000046
0000010054d47570 0000010000000073
Jun 21 16:45:17 lustre-3ware kernel: 0000000000000000
0000000071403800 0000010001021aa0 000000007af136c0
Jun 21 16:45:17 lustre-3ware kernel: 0000010054b17800
0000000000000092
Jun 21 16:45:17 lustre-3ware kernel: Call
Trace:<ffffffff80308f91>{thread_return+88}
<ffffffff80131551>{recalc_task_prio+337}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80308223>{__down+147}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80309cbb>{__down_failed+53}
<ffffffffa0063155>{:jbd:.text.lock.checkpoint+5}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa005e3c3>{:jbd:start_this_handle+963}
<ffffffffa022325d>{:obdclass:lprocfs_oh_tally+30}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80134b62>{autoremove_wake_function+0}
<ffffffff80134b62>{autoremove_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa005e51e>{:jbd:journal_start+223}
<ffffffffa04296f2>{:fsfilt_ldiskfs:fsfilt_ldiskfs_brw_start+649}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04d8f9f>{:obdfilter:filter_commitrw_write+1480}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04d3756>{:obdfilter:filter_commitrw+84}
<ffffffff8013f23f>{del_timer+107}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
<ffffffff80309a63>{schedule_timeout+375}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04b0db1>{:ost:ost_brw_write+5253}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04ad513>{:ost:ost_bulk_timeout+0}
<ffffffffa029b471>{:ptlrpc:lustre_msg_get_version+64}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffffa04b4500>{:ost:ost_handle+6987}
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:45:17 lustre-3ware kernel:
<ffffffff801e9ce7>{vsnprintf+1406} <ffffffff801e9dca>{snprintf+131}
Jun 21 16:45:18 lustre-3ware kernel:
<ffffffffa02a1e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jun 21 16:45:18 lustre-3ware kernel:
<ffffffff8013f100>{__mod_timer+293}
<ffffffffa02a3d1b>{:ptlrpc:ptlrpc_main+2018}
Jun 21 16:45:18 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jun 21 16:45:18 lustre-3ware kernel:
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
<ffffffff80110e23>{child_rip+8}
Jun 21 16:45:18 lustre-3ware kernel:
<ffffffffa02a3539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jun 21 16:45:18 lustre-3ware kernel:
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3397:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR,
err == -16 req@000001007dd8b600 x13159861/t0
o8->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl
Rpc:R/0/0 rc 0/-16
Jun 21 16:45:18 lustre-3ware kernel: LustreError: dumping log to
/tmp/lustre-log.1182455051.4929
Jun 21 16:45:18 lustre-3ware kernel: LustreError: dumping log to
/tmp/lustre-log.1182455060.3682
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3397:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR,
err == -16 req@000001007afbe600 x13159864/t0
o8->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl
Rpc:R/0/0 rc 0/-16
Jun 21 16:45:18 lustre-3ware kernel: LustreError: dumping log to
/tmp/lustre-log.1182455070.3683
Jun 21 16:45:18 lustre-3ware kernel: LustreError: dumping log to
/tmp/lustre-log.1182455080.3675
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3663:0:(lustre_fsfilt.h:205:fsfilt_start_log()) scratch-OST0000: slow
journal start 147s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(lustre_fsfilt.h:205:fsfilt_start_log()) scratch-OST0000: slow
journal start 147s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(lustre_fsfilt.h:205:fsfilt_start_log()) Skipped 20 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
3663:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-0@lo
portal 4 match 13159834 offset 0 length 336: 2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3663:0:(service.c:668:ptlrpc_server_handle_request()) request 13159834
opc 5 from 12345-0@lo processed in 147s trans 0 rc 0/0
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
3663:0:(watchdog.c:312:lcw_update_time()) Expired watchdog for pid 3663
disabled after 147.4653s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
4929:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) scratch-OST0000:
slow journal start 147s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
4929:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) Skipped 2 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
4929:0:(filter_io_26.c:680:filter_commitrw_write()) scratch-OST0000:
slow brw_start 147s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
4929:0:(filter_io_26.c:680:filter_commitrw_write()) Skipped 2 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3682:0:(filter_io_26.c:732:filter_commitrw_write()) scratch-OST0000:
slow direct_io 149s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3682:0:(filter_io_26.c:732:filter_commitrw_write()) Skipped 8 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3675:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) scratch-OST0000:
slow journal start 149s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3675:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) Skipped 1 previous
similar message
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3675:0:(filter_io_26.c:680:filter_commitrw_write()) scratch-OST0000:
slow brw_start 149s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3675:0:(filter_io_26.c:680:filter_commitrw_write()) Skipped 1 previous
similar message
Jun 21 16:45:18 lustre-3ware kernel: Lustre: scratch-OST0000-osc:
Connection restored to service scratch-OST0000 using nid 0@lo.
Jun 21 16:45:18 lustre-3ware kernel: LustreError: dumping log to
/tmp/lustre-log.1182455090.3566
Jun 21 16:45:18 lustre-3ware kernel: Lustre: scratch-OST0000: received
MDS connection from 0@lo
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
6411:0:(recov_thread.c:565:llog_repl_connect()) llcd
000001000e9ed000:000001007453e180 not empty
Jun 21 16:45:18 lustre-3ware kernel: Lustre: MDS scratch-MDT0000:
scratch-OST0000_UUID now active, resetting orphans
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
3663:0:(filter.c:2791:filter_precreate()) scratch-OST0000: precreate
aborted by destroy
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3396:0:(osc_create.c:82:osc_interpret_create()) @@@ unknown rc -11 from
async create: failing oscc req@000001004f4cf400 x13159834/t0
o5->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 336/336 ref 1 fl
Complete:R/2/0 rc -11/-11
Jun 21 16:45:18 lustre-3ware kernel: Lustre: scratch-OST0000-osc:
Connection to service scratch-OST0000 via nid 0@lo was lost; in progress
operations using this service will wait for recovery to complete.
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
6425:0:(filter.c:2600:filter_destroy_precreated()) scratch-OST0000:
deleting orphan objects from 5737276 to 5737294
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3397:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR,
err == -16 req@000001004f4cf400 x13159878/t0
o8->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl
Rpc:R/0/0 rc 0/-16
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3682:0:(filter_io_26.c:745:filter_commitrw_write()) scratch-OST0000:
slow commitrw commit 158s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3675:0:(filter_io_26.c:732:filter_commitrw_write()) scratch-OST0000:
slow direct_io 158s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
4929:0:(service.c:668:ptlrpc_server_handle_request()) request 58935176
opc 4 from 12345-192.168.0.20@tcp processed in 158s trans 19977643 rc 0/0
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
4929:0:(watchdog.c:312:lcw_update_time()) Expired watchdog for pid 4929
disabled after 158.5931s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3682:0:(filter_io_26.c:745:filter_commitrw_write()) Skipped 4 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5587578 ino 39977056 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5587578: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5589650 ino 61521936 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5589650: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3683:0:(filter_io_26.c:745:filter_commitrw_write()) scratch-OST0000:
slow commitrw commit 163s
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
4902:0:(service.c:668:ptlrpc_server_handle_request()) request 58935173
opc 4 from 12345-192.168.0.20@tcp processed in 163s trans 19977645 rc 0/0
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
4902:0:(service.c:668:ptlrpc_server_handle_request()) Skipped 1 previous
similar message
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
4902:0:(watchdog.c:312:lcw_update_time()) Expired watchdog for pid 4902
disabled after 163.1802s
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
4902:0:(watchdog.c:312:lcw_update_time()) Skipped 1 previous similar message
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3683:0:(filter_io_26.c:745:filter_commitrw_write()) Skipped 2 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5592130 ino 85680368 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) Skipped 2 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5592130: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) Skipped 2 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError: dumping log to
/tmp/lustre-log.1182455101.3556
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5588025 ino 44482608 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5588025: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5588387 ino 47816884 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5588387: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5590990 ino 74309824 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5590990: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5575264 ino 273055808 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5575264: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5588108 ino 45326575 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) Skipped 2 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5588108: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) Skipped 2 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3479:0:(llog_server.c:435:llog_origin_handle_cancel()) cancel 124
llog-records failed: -22
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3611:0:(recov_thread.c:396:log_commit_thread()) commit
000001002eccf000:000001007453e180 drop 124 cookies: rc -22
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5579833 ino 324575296 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1305:filter_destroy_internal()) Skipped 6 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5579833: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1311:filter_destroy_internal()) Skipped 6 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5577616 ino 300097759 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1305:filter_destroy_internal()) Skipped 7 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5577616: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6976:0:(filter.c:1311:filter_destroy_internal()) Skipped 7 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5582326 ino 350011599 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1305:filter_destroy_internal()) Skipped 9 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5582326: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1311:filter_destroy_internal()) Skipped 9 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
4857:0:(llog_server.c:435:llog_origin_handle_cancel()) cancel 124
llog-records failed: -22
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3705:0:(recov_thread.c:396:log_commit_thread()) commit
00000100179c6000:000001007453e180 drop 124 cookies: rc -22
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
6395:0:(ldlm_lib.c:497:target_handle_reconnect()) scratch-OST0000:
scratch-mdtlov_UUID reconnecting
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
6395:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 17 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: Lustre: scratch-OST0000-osc:
Connection restored to service scratch-OST0000 using nid 0@lo.
Jun 21 16:45:18 lustre-3ware kernel: Lustre: scratch-OST0000: received
MDS connection from 0@lo
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
6366:0:(recov_thread.c:565:llog_repl_connect()) llcd
000001004ac41000:000001007453e180 not empty
Jun 21 16:45:18 lustre-3ware kernel: Lustre: MDS scratch-MDT0000:
scratch-OST0000_UUID now active, resetting orphans
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3488:0:(llog_server.c:435:llog_origin_handle_cancel()) cancel 118
llog-records failed: -22
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3705:0:(recov_thread.c:396:log_commit_thread()) commit
000001004ac41000:000001007453e180 drop 118 cookies: rc -22
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5575760 ino 278782095 nlink 0 count 1
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1305:filter_destroy_internal()) Skipped 21 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5575760: rc -2
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1311:filter_destroy_internal()) Skipped 21 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3566:0:(service.c:668:ptlrpc_server_handle_request()) request 58935185
opc 101 from 12345-192.168.0.20@tcp processed in 181s trans 39633343 rc
301/301
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3566:0:(service.c:668:ptlrpc_server_handle_request()) Skipped 2 previous
similar messages
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
3566:0:(watchdog.c:312:lcw_update_time()) Expired watchdog for pid 3566
disabled after 181.6070s
Jun 21 16:45:18 lustre-3ware kernel: Lustre:
3566:0:(watchdog.c:312:lcw_update_time()) Skipped 2 previous similar
messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3487:0:(llog_server.c:435:llog_origin_handle_cancel()) cancel 124
llog-records failed: -22
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3487:0:(llog_server.c:435:llog_origin_handle_cancel()) Skipped 2
previous similar messages
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3705:0:(recov_thread.c:396:log_commit_thread()) commit
000001000761d000:000001007453e180 drop 124 cookies: rc -22
Jun 21 16:45:18 lustre-3ware kernel: LustreError:
3705:0:(recov_thread.c:396:log_commit_thread()) Skipped 2 previous
similar messages
Jun 21 16:45:19 lustre-3ware kernel: LustreError:
3485:0:(llog_server.c:435:llog_origin_handle_cancel()) cancel 124
llog-records failed: -22
Jun 21 16:45:19 lustre-3ware kernel: LustreError:
3485:0:(llog_server.c:435:llog_origin_handle_cancel()) Skipped 5
previous similar messages
Jun 21 16:45:19 lustre-3ware kernel: LustreError:
4425:0:(recov_thread.c:396:log_commit_thread()) commit
00000100499ca000:000001007453e180 drop 124 cookies: rc -22
Jun 21 16:45:19 lustre-3ware kernel: LustreError:
4425:0:(recov_thread.c:396:log_commit_thread()) Skipped 5 previous
similar messages
Jun 21 16:45:19 lustre-3ware kernel: LustreError:
6979:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5594033 ino 107847792 nlink 0 count 1
Jun 21 16:45:19 lustre-3ware kernel: LustreError:
6979:0:(filter.c:1305:filter_destroy_internal()) Skipped 165 previous
similar messages
Jun 21 16:45:19 lustre-3ware kernel: LustreError:
6979:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5594033: rc -2
Jun 21 16:45:19 lustre-3ware kernel: LustreError:
6979:0:(filter.c:1311:filter_destroy_internal()) Skipped 165 previous
similar messages
Jun 21 16:45:20 lustre-3ware kernel: LustreError:
3485:0:(llog_server.c:435:llog_origin_handle_cancel()) cancel 124
llog-records failed: -22
Jun 21 16:45:20 lustre-3ware kernel: LustreError:
3485:0:(llog_server.c:435:llog_origin_handle_cancel()) Skipped 9
previous similar messages
Jun 21 16:45:20 lustre-3ware kernel: LustreError:
3705:0:(recov_thread.c:396:log_commit_thread()) commit
00000100499ca000:000001007453e180 drop 124 cookies: rc -22
Jun 21 16:45:20 lustre-3ware kernel: LustreError:
3705:0:(recov_thread.c:396:log_commit_thread()) Skipped 9 previous
similar messages
Jun 21 16:45:21 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5582339 ino 350117936 nlink 0 count 1
Jun 21 16:45:21 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1305:filter_destroy_internal()) Skipped 442 previous
similar messages
Jun 21 16:45:21 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5582339: rc -2
Jun 21 16:45:21 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1311:filter_destroy_internal()) Skipped 442 previous
similar messages
Jun 21 16:45:22 lustre-3ware kernel: LustreError:
3482:0:(llog_server.c:435:llog_origin_handle_cancel()) cancel 124
llog-records failed: -22
Jun 21 16:45:22 lustre-3ware kernel: LustreError:
3482:0:(llog_server.c:435:llog_origin_handle_cancel()) Skipped 18
previous similar messages
Jun 21 16:45:22 lustre-3ware kernel: LustreError:
3631:0:(recov_thread.c:396:log_commit_thread()) commit
0000010013a9b000:000001007453e180 drop 124 cookies: rc -22
Jun 21 16:45:22 lustre-3ware kernel: LustreError:
3631:0:(recov_thread.c:396:log_commit_thread()) Skipped 18 previous
similar messages
Jun 21 16:45:25 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5594796 ino 118235151 nlink 0 count 1
Jun 21 16:45:25 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1305:filter_destroy_internal()) Skipped 820 previous
similar messages
Jun 21 16:45:25 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5594796: rc -2
Jun 21 16:45:25 lustre-3ware kernel: LustreError:
6970:0:(filter.c:1311:filter_destroy_internal()) Skipped 820 previous
similar messages
Jun 21 16:45:26 lustre-3ware kernel: LustreError:
3485:0:(llog_server.c:435:llog_origin_handle_cancel()) cancel 124
llog-records failed: -22
Jun 21 16:45:26 lustre-3ware kernel: LustreError:
3485:0:(llog_server.c:435:llog_origin_handle_cancel()) Skipped 65
previous similar messages
Jun 21 16:45:26 lustre-3ware kernel: LustreError:
3499:0:(recov_thread.c:396:log_commit_thread()) commit
0000010071598000:000001007453e180 drop 124 cookies: rc -22
Jun 21 16:45:26 lustre-3ware kernel: LustreError:
3499:0:(recov_thread.c:396:log_commit_thread()) Skipped 65 previous
similar messages
Jun 21 16:45:33 lustre-3ware kernel: LustreError:
6979:0:(filter.c:1305:filter_destroy_internal()) destroying objid
5579465 ino 321093744 nlink 0 count 1
Jun 21 16:45:33 lustre-3ware kernel: LustreError:
6979:0:(filter.c:1305:filter_destroy_internal()) Skipped 1602 previous
similar messages
Jun 21 16:45:33 lustre-3ware kernel: LustreError:
6979:0:(filter.c:1311:filter_destroy_internal()) error unlinking objid
5579465: rc -2
Jun 21 16:45:33 lustre-3ware kernel: LustreError:
6979:0:(filter.c:1311:filter_destroy_internal()) Skipped 1602 previous
similar messages
Jun 21 16:45:35 lustre-3ware kernel: LustreError:
3487:0:(llog_server.c:435:llog_origin_handle_cancel()) cancel 124
llog-records failed: -22
Jun 21 16:45:35 lustre-3ware kernel: LustreError:
3487:0:(llog_server.c:435:llog_origin_handle_cancel()) Skipped 119
previous similar messages
Jun 21 16:45:35 lustre-3ware kernel: LustreError:
4425:0:(recov_thread.c:396:log_commit_thread()) commit
000001002088c000:000001007453e180 drop 124 cookies: rc -22
Jun 21 16:45:35 lustre-3ware kernel: LustreError:
4425:0:(recov_thread.c:396:log_commit_thread()) Skipped 119 previous
similar messages
Jun 21 16:45:35 lustre-3ware kernel: LustreError:
5304:0:(llog_lvfs.c:597:llog_lvfs_create()) error looking up logfile
0x1c106ba:0x61036960: rc -2
Jun 21 16:45:35 lustre-3ware kernel: LustreError:
5304:0:(llog_cat.c:156:llog_cat_id2handle()) error opening log id
0x1c106ba:61036960: rc -2
Jun 21 16:45:35 lustre-3ware kernel: LustreError:
5304:0:(llog_cat.c:307:llog_cat_cancel_records()) Cannot find log 0x1c106ba
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
4851:0:(llog_lvfs.c:597:llog_lvfs_create()) error looking up logfile
0x1c106ba:0x61036960: rc -2
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
4851:0:(llog_lvfs.c:597:llog_lvfs_create()) Skipped 31 previous similar
messages
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
4851:0:(llog_cat.c:156:llog_cat_id2handle()) error opening log id
0x1c106ba:61036960: rc -2
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
4851:0:(llog_cat.c:156:llog_cat_id2handle()) Skipped 31 previous similar
messages
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
4851:0:(llog_cat.c:307:llog_cat_cancel_records()) Cannot find log 0x1c106ba
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
4851:0:(llog_cat.c:307:llog_cat_cancel_records()) Skipped 31 previous
similar messages
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
3486:0:(llog_lvfs.c:597:llog_lvfs_create()) error looking up logfile
0x1c106ba:0x61036960: rc -2
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
3486:0:(llog_lvfs.c:597:llog_lvfs_create()) Skipped 247 previous similar
messages
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
3486:0:(llog_cat.c:156:llog_cat_id2handle()) error opening log id
0x1c106ba:61036960: rc -2
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
3486:0:(llog_cat.c:156:llog_cat_id2handle()) Skipped 247 previous
similar messages
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
3486:0:(llog_cat.c:307:llog_cat_cancel_records()) Cannot find log 0x1c106ba
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
3486:0:(llog_cat.c:307:llog_cat_cancel_records()) Skipped 247 previous
similar messages
Jun 21 16:45:46 lustre-3ware kernel: LustreError:
3488:0:(llog_lvfs.c:597:llog_lvfs_create()) error looking up logfile
0x1c106ba:0x61036960: rc -2
Jun 21 16:45:51 lustre-3ware kernel: LustreError:
3488:0:(llog_lvfs.c:597:llog_lvfs_create()) Skipped 351 previous similar
messages
Jun 21 16:45:55 lustre-3ware kernel: LustreError:
3488:0:(llog_cat.c:156:llog_cat_id2handle()) error opening log id
0x1c106ba:61036960: rc -2
Jun 21 16:46:00 lustre-3ware kernel: LustreError:
3488:0:(llog_cat.c:156:llog_cat_id2handle()) Skipped 351 previous
similar messages
Jun 21 16:46:10 lustre-3ware kernel: LustreError:
3488:0:(llog_cat.c:307:llog_cat_cancel_records()) Cannot find log 0x1c106ba
Jun 21 16:46:16 lustre-3ware kernel: LustreError:
3488:0:(llog_cat.c:307:llog_cat_cancel_records()) Skipped 351 previous
similar messages
Jun 21 16:46:16 lustre-3ware kernel: Lustre:
3663:0:(lustre_fsfilt.h:205:fsfilt_start_log()) scratch-OST0000: slow
journal start 37s
Jun 21 16:46:16 lustre-3ware kernel: Lustre:
3663:0:(lustre_fsfilt.h:205:fsfilt_start_log()) Skipped 2 previous
similar messages
Jun 21 16:46:16 lustre-3ware sshd(pam_unix)[6982]: session opened for
user root by root(uid=0)
Jun 21 16:46:22 lustre-3ware kernel: LustreError:
3481:0:(llog_lvfs.c:597:llog_lvfs_create()) error looking up logfile
0x1c106ba:0x61036960: rc -2
Jun 21 16:46:22 lustre-3ware kernel: LustreError:
3481:0:(llog_lvfs.c:597:llog_lvfs_create()) Skipped 37 previous similar
messages
Jun 21 16:46:22 lustre-3ware kernel: LustreError:
3481:0:(llog_cat.c:156:llog_cat_id2handle()) error opening log id
0x1c106ba:61036960: rc -2
Jun 21 16:46:22 lustre-3ware kernel: LustreError:
3481:0:(llog_cat.c:156:llog_cat_id2handle()) Skipped 37 previous similar
messages
Jun 21 16:46:22 lustre-3ware kernel: LustreError:
3481:0:(llog_cat.c:307:llog_cat_cancel_records()) Cannot find log 0x1c106ba
Jun 21 16:46:22 lustre-3ware kernel: LustreError:
3481:0:(llog_cat.c:307:llog_cat_cancel_records()) Skipped 37 previous
similar messages
Jun 21 16:47:14 lustre-3ware kernel: LustreError:
3396:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent at
1182455184, 50s ago) req@000001002cafb800 x13164484/t0
o5->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 336/336 ref 1 fl
Rpc:/0/0 rc 0/-22
Jun 21 16:47:14 lustre-3ware kernel: Lustre: scratch-OST0000-osc:
Connection to service scratch-OST0000 via nid 0@lo was lost; in progress
operations using this service will wait for recovery to complete.
Jun 21 16:47:14 lustre-3ware kernel: Lustre:
6384:0:(ldlm_lib.c:709:target_handle_connect()) scratch-OST0000: refuse
reconnection from scratch-mdtlov_UUID@0@lo to 0x00000100716f3000/2
Jun 21 16:47:14 lustre-3ware kernel: Lustre:
6384:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 15 previous
similar messages
Jun 21 16:47:14 lustre-3ware kernel: LustreError:
6384:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error
(-16) req@000001007dcfea00 x13164491/t0
o8->scratch-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl
Interpret:/0/0 rc -16/0
Jun 21 16:47:14 lustre-3ware kernel: LustreError:
6384:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 15 previous
similar messages
Jun 21 16:47:14 lustre-3ware kernel: LustreError:
3397:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR,
err == -16 req@0000010076972e00 x13164491/t0
o8->scratch-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl
Rpc:R/0/0 rc 0/-16
Jun 21 16:48:04 lustre-3ware kernel: Lustre:
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 6427: it was
inactive for 100s
Jun 21 16:48:04 lustre-3ware kernel: Lustre:
0:0:(watchdog.c:130:lcw_cb()) Skipped 7 previous similar messages
Jun 21 16:48:04 lustre-3ware kernel: Lustre:
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for
process 6427
Jun 21 16:48:04 lustre-3ware kernel: Lustre:
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 7 previous
similar messages
Jun 21 16:48:04 lustre-3ware kernel: ll_ost_122 D
00000100327453e8 0 6427 1 6428 6426 (L-TLB)
Jun 21 16:48:04 lustre-3ware kernel: 0000010032745388 0000000000000046
ffffffff80134b62 0000010032745300
Jun 21 16:48:04 lustre-3ware kernel: 0000010032745300
0000010058a22030 0000000000000000 0000000158a22030
Jun 21 16:48:04 lustre-3ware kernel: 0000010058a22030
0000000000005118
Jun 21 16:48:04 lustre-3ware kernel: Call
Trace:<ffffffff80134b62>{autoremove_wake_function+0}
<ffffffff803098af>{io_schedule+38}
Jun 21 16:48:04 lustre-3ware kernel:
<ffffffff8024d028>{get_request_wait+156}
<ffffffff80134b62>{autoremove_wake_function+0}
Jun 21 16:48:04 lustre-3ware kernel:
<ffffffff80134b62>{autoremove_wake_function+0}
<ffffffff8024dcef>{__make_request+837}
Jun 21 16:48:04 lustre-3ware kernel:
<ffffffff8024e0f7>{generic_make_request+400}
<ffffffff80134b62>{autoremove_wake_function+0}
Jun 21 16:48:04 lustre-3ware kernel:
<ffffffff8024e203>{submit_bio+247} <ffffffff8017b914>{bio_alloc+288}
Jun 21 16:48:12 lustre-3ware kernel:
<ffffffff80179816>{submit_bh+255} <ffffffff80179896>{ll_rw_block+96}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa0062470>{:jbd:__flush_batch+44}
<ffffffffa0062da3>{:jbd:log_do_checkpoint+417}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa0063124>{:jbd:__log_wait_for_space+216}
<ffffffffa005e3c3>{:jbd:start_this_handle+963}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff801e9ce7>{vsnprintf+1406}
<ffffffffa005e51e>{:jbd:journal_start+223}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa04293e2>{:fsfilt_ldiskfs:fsfilt_ldiskfs_start+682}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa04cae23>{:obdfilter:filter_setattr_internal+1605}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff801cd460>{dummy_inode_permission+0}
<ffffffffa04c27f2>{:obdfilter:filter_fmd_get+624}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa04cbda4>{:obdfilter:filter_setattr+422}
<ffffffffa01e206b>{:lnet:LNetPut+1425}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa029a5a8>{:ptlrpc:lustre_msg_add_version+67}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa04ad465>{:ost:ost_setattr+1043}
<ffffffffa029b51b>{:ptlrpc:lustre_msg_check_version+49}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa04b42bc>{:ost:ost_handle+6407}
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff801e9ce7>{vsnprintf+1406} <ffffffff801e9dca>{snprintf+131}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02a1e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff8013f100>{__mod_timer+293}
<ffffffffa02a3d1b>{:ptlrpc:ptlrpc_main+2018}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
<ffffffff80110e23>{child_rip+8}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02a3539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jun 21 16:48:15 lustre-3ware kernel:
Jun 21 16:48:15 lustre-3ware kernel: LustreError: dumping log to
/tmp/lustre-log.1182455284.6427
Jun 21 16:48:15 lustre-3ware kernel: ll_ost_creat_ D
ffffffff80308644 0 3663 1 3664 3662 (L-TLB)
Jun 21 16:48:15 lustre-3ware kernel: 00000100743dd788 0000000000000046
00000100774ca800 0000010000000073
Jun 21 16:48:15 lustre-3ware kernel: 000001007109ab98
00000000a03d2c04 0000010001021aa0 0000000000000002
Jun 21 16:48:15 lustre-3ware kernel: 000001007413c030
000000000000012e
Jun 21 16:48:15 lustre-3ware kernel: Call
Trace:<ffffffffa03d4aed>{:ldiskfs:dx_release+36}
<ffffffffa03d526e>{:ldiskfs:ldiskfs_find_entry+714}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff80308223>{__down+147}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff80309cbb>{__down_failed+53}
<ffffffffa0063155>{:jbd:.text.lock.checkpoint+5}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa005e3c3>{:jbd:start_this_handle+963}
<ffffffffa03dffeb>{:ldiskfs:iopen_connect_dentry+679}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa03d5734>{:ldiskfs:ldiskfs_lookup+396}
<ffffffff80185132>{__lookup_hash+227}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa005e51e>{:jbd:journal_start+223}
<ffffffffa04293e2>{:fsfilt_ldiskfs:fsfilt_ldiskfs_start+682}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa04cd24b>{:obdfilter:filter_precreate+2066}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa04cfbb7>{:obdfilter:filter_create+3455}
<ffffffffa01e206b>{:lnet:LNetPut+1425}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa0297383>{:ptlrpc:ptl_send_buf+727}
<ffffffffa029a5a8>{:ptlrpc:lustre_msg_add_version+67}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa029adc5>{:ptlrpc:lustre_pack_reply+1928}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa04b3609>{:ost:ost_handle+3156}
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff801e9ce7>{vsnprintf+1406} <ffffffff801e9dca>{snprintf+131}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02a1e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff8013f100>{__mod_timer+293}
<ffffffffa02a3d1b>{:ptlrpc:ptlrpc_main+2018}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
<ffffffff80110e23>{child_rip+8}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02a3539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jun 21 16:48:15 lustre-3ware kernel:
Jun 21 16:48:15 lustre-3ware kernel: ll_mdt_13 S
0000000000000000 0 3559 1 3560 3558 (L-TLB)
Jun 21 16:48:15 lustre-3ware kernel: 00000100748eb238 0000000000000046
0000003000000030 00000100748eb2e0
Jun 21 16:48:15 lustre-3ware kernel: 00000100748eb1c8
ffffffff801331f6 0000000300000076 0000000000000008
Jun 21 16:48:15 lustre-3ware kernel: 0000010077ff8030
0000000000002013
Jun 21 16:48:15 lustre-3ware kernel: Call
Trace:<ffffffff801331f6>{__wake_up_common+67}
<ffffffffa049e528>{:osc:oscc_has_objects+259}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa049e9a6>{:osc:oscc_precreate+580}
<ffffffff801331a5>{default_wake_function+0}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa049f86b>{:osc:osc_create+3297}
<ffffffffa0364a76>{:lov:qos_prep_create+3728}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa035bbe4>{:lov:lov_prep_create_set+541}
<ffffffffa0348d59>{:lov:lov_create+6516}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa03dd4af>{:ldiskfs:ldiskfs_xattr_ibody_get+403}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa0429f3d>{:fsfilt_ldiskfs:fsfilt_ldiskfs_get_md+101}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa04415c1>{:mds:mds_get_md+105}
<ffffffffa046155b>{:mds:mds_create_objects+3818}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa04421c2>{:mds:mds_pack_md+409}
<ffffffffa04631aa>{:mds:mds_finish_open+704}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa0465625>{:mds:mds_open+6486}
<ffffffffa0459c91>{:mds:mds_reint_rec+373}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa0443e26>{:mds:mds_reint+637}
<ffffffffa044c22a>{:mds:mds_intent_policy+890}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa0278eee>{:ptlrpc:ldlm_resource_putref+356}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa0274ca2>{:ptlrpc:ldlm_lock_create+1375}
<ffffffffa0275d85>{:ptlrpc:ldlm_lock_enqueue+208}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02887d7>{:ptlrpc:ldlm_handle_enqueue+2524}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02871ba>{:ptlrpc:ldlm_server_blocking_ast+0}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa028772d>{:ptlrpc:ldlm_server_completion_ast+0}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa04483a9>{:mds:mds_handle+15254}
<ffffffffa01b3f69>{:libcfs:libcfs_debug_vmsg2+1713}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff801e9ce7>{vsnprintf+1406} <ffffffff801e9dca>{snprintf+131}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02a1e49>{:ptlrpc:ptlrpc_server_handle_request+2514}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02a1e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff8013f100>{__mod_timer+293}
<ffffffffa02a3d1b>{:ptlrpc:ptlrpc_main+2018}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffff801331a5>{default_wake_function+0}
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02a297a>{:ptlrpc:ptlrpc_retry_rqbds+0}
<ffffffff80110e23>{child_rip+8}
Jun 21 16:48:15 lustre-3ware kernel:
<ffffffffa02a3539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jun 21 16:48:15 lustre-3ware kernel:
Jun 21 16:48:15 lustre-3ware kernel: LustreError: dumping log to
/tmp/lustre-log.1182455292.3559
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
6427:0:(lustre_fsfilt.h:205:fsfilt_start_log()) scratch-OST0000: slow
journal start 111s
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
6427:0:(service.c:668:ptlrpc_server_handle_request()) request 58947777
opc 2 from 12345-192.168.0.20@tcp processed in 111s trans 19979014 rc 0/0
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
6427:0:(service.c:668:ptlrpc_server_handle_request()) Skipped 1 previous
similar message
Jun 21 16:48:15 lustre-3ware kernel: Lustre:
6427:0:(watchdog.c:312:lcw_update_time()) Expired watchdog for pid 6427
disabled after 111.1903s
Jun 21 16:48:15 lustre-3ware kernel: Lustre:
6427:0:(watchdog.c:312:lcw_update_time()) Skipped 1 previous similar message
Jun 21 16:48:15 lustre-3ware kernel: Lustre:
3663:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-0@lo
portal 4 match 13164484 offset 0 length 336: 2
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
4907:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) scratch-OST0000:
slow journal start 87s
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
4867:0:(filter_io_26.c:680:filter_commitrw_write()) scratch-OST0000:
slow brw_start 87s
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
4867:0:(filter_io_26.c:680:filter_commitrw_write()) Skipped 2 previous
similar messages
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
3484:0:(filter.c:1139:filter_parent_lock()) scratch-OST0000: slow parent
lock 109s
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
3484:0:(filter.c:1139:filter_parent_lock()) Skipped 64 previous similar
messages
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
4907:0:(lustre_fsfilt.h:240:fsfilt_brw_start_log()) Skipped 4 previous
similar messages
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
4907:0:(filter_io_26.c:732:filter_commitrw_write()) scratch-OST0000:
slow direct_io 87s
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
4907:0:(filter_io_26.c:732:filter_commitrw_write()) Skipped 2 previous
similar messages
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
4874:0:(filter_io_26.c:745:filter_commitrw_write()) scratch-OST0000:
slow commitrw commit 87s
Jun 21 16:48:15 lustre-3ware kernel: LustreError:
4874:0:(filter_io_26.c:745:filter_commitrw_write()) Skipped 2 previous
similar messages
Jun 21 16:48:29 lustre-3ware kernel: LustreError:
3481:0:(ldlm_lockd.c:1099:ldlm_handle_cancel()) received cancel for
unknown lock cookie 0x24c4cddb1c4a2e7f from client
e98c0aac-519a-287c-4ea9-f19b3ff2fce3 id 12345-192.168.0.20@tcp
Jun 21 16:48:29 lustre-3ware kernel: Lustre: scratch-OST0000-osc:
Connection restored to service scratch-OST0000 using nid 0@lo.
Jun 21 16:48:29 lustre-3ware kernel: Lustre: scratch-OST0000: received
MDS connection from 0@lo
Jun 21 16:48:29 lustre-3ware kernel: Lustre:
6421:0:(recov_thread.c:565:llog_repl_connect()) llcd
0000010066c72000:000001007453e180 not empty
Jun 21 16:48:29 lustre-3ware kernel: Lustre: MDS scratch-MDT0000:
scratch-OST0000_UUID now active, resetting orphans
Jun 21 16:48:29 lustre-3ware kernel: Lustre:
6396:0:(filter.c:2600:filter_destroy_precreated()) scratch-OST0000:
deleting orphan objects from 5743933 to 5743964
Jun 21 16:48:29 lustre-3ware kernel: LustreError:
3559:0:(service.c:668:ptlrpc_server_handle_request()) request 3939462
opc 101 from 12345-192.168.0.22@tcp processed in 125s trans 39649478 rc
301/301
Jun 21 16:48:29 lustre-3ware kernel: LustreError:
3559:0:(service.c:668:ptlrpc_server_handle_request()) Skipped 2 previous
similar messages
Jun 21 16:48:29 lustre-3ware kernel: Lustre:
3559:0:(watchdog.c:312:lcw_update_time()) Expired watchdog for pid 3559
disabled after 125.8117s
Jun 21 16:48:29 lustre-3ware kernel: Lustre:
3559:0:(watchdog.c:312:lcw_update_time()) Skipped 1 previous similar message
Jun 21 16:50:35 lustre-3ware kernel: Lustre:
6405:0:(lustre_fsfilt.h:205:fsfilt_start_log()) scratch-OST0000: slow
journal start 31s
Andreas Dilger wrote:> On Jun 21, 2007 10:20 -0300, Balagopal Pillai wrote:
>
>> I am using Lustre 1.6.0.1 with one OST and 20 clients in an
>> HPC cluster.
>> The OST/MDT/MGS has a 16 channel 3ware 9650 using raid6.
>>
>> *Now the problem - Lustre performance dips a lot when it comes to small
>> files. Please see the following fileop -f 5 test comparing NFS and
Lustre -
>>
>
> We have had reports of read performance problems with 3ware in the past.
>
>
>> Lustre -
>>
>> Fileop: File size is 1, Output is in Ops/sec. (A=Avg, B=Best,
W=Worst)
>> . mkdir rmdir create read write close stat access chmod
>> readdir link unlink delete Total_files
>> A 5 1654 691 132 14228 719 4874 1987 32737 1718
>> 2506 1262 1340 1608 125
>>
>>
>> NFS -
>> Fileop: File size is 1, Output is in Ops/sec. (A=Avg, B=Best,
W=Worst)
>> . mkdir rmdir create read write close stat access chmod
>> readdir link unlink delete Total_files
>> A 5 177 594 459 380747 137392 2282 1219 444312 502
>> 1274 306 513 464 125
>>
>
>
>> Could you please recommend any tunables to get a bit more
>> performance out of Lustre with lots of small files? Lots of small files
>> was bad in GFS too, but it was better than NFS though.
>>
>
> Try setting the debug on the clients to zero "sysctl -w
lnet.debug=0".
>
>
>> *Also the read performance of Lustre seems to be a little behind NFS. I
>> had /opt which has all the software for users moved to Lustre in the
new
>> setup. But
>> software like Matlab, Splus etc takes almost a minute to come up. The
>> second time is very fast though, maybe due to caching. So i am thinking
>> of putting /opt
>> back to NFS. Is it possible to boost the read performance of Lustre a
bit?
>>
>
> Turn off debug is the easiest.
>
>
>> *One last question. In the older Lustre setup (version 1.4.5), i have 5
>> scsi drives one each as an OST for a single volume. The volume became
>> full. But df still reported that there is 27GB free. There
doesn''t seem
>> to be an lfs df option in that version of Lustre. So i
couldn''t see the
>> individual utilization of each of the 5 OST. Is this a striping
problem?
>>
>
> In the absence of "lfs df" use /proc to find per-OST usage on the
client:
>
> grep ''[0-9]'' /proc/fs/lustre/osc/*/kbytes*
>
> Lustre 1.6 should be better at balancing the OST usage than 1.4.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Principal Software Engineer
> Cluster File Systems, Inc.
>