thr3ads.net - Lustre discuss - [Lustre-discuss] lustre crash [Jul 2007]

If this information is useful, please help other people find it:
Share via:
Balagopal Pillai
2007-Jul-30 08:41 UTC
[Lustre-discuss] lustre crash

Hi,

        One of the Lustre filesystems hung today morning, while the 
other big lustre volume was working fine. The one
that hung was a quota enforced volume. The MGS/MDT/OST server needed a 
power recycle along with all the clients.
Now all volumes are working fine. But i am curious about what could have 
caused this crash. The log is pasted below.
I posted a previous message about disk quota showing the wrong number of 
blocks. Surprisingly it shows the right number after the
restart of the lustre server. Is this crash a quota related issue? Any 
help and comments are much appreciated.  Thanks.


Regards
Balagopal



Jul 29 04:02:04 lustre-3ware syslogd 1.4.1: restart.
Jul 30 10:21:50 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 3658: it was 
inactive for 18s
Jul 30 10:21:50 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Skipped 4 previous similar messages
Jul 30 10:21:50 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 3658
Jul 30 10:21:50 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 4 previous 
similar messages
Jul 30 10:21:50 lustre-3ware kernel: ldlm_cb_05    D 
0000000000000001     0  3658      1          3659  3657 (L-TLB)
Jul 30 10:21:50 lustre-3ware kernel: 0000010073961c48 0000000000000046 
0000010077d98e80 00000100746f8030
Jul 30 10:21:50 lustre-3ware kernel:        ffffffff80134b62 
0000010073961ba0 0000010073961ba0 00000000a01f481a
Jul 30 10:21:50 lustre-3ware kernel:        00000100746f8030 
00000000000006f4
Jul 30 10:21:50 lustre-3ware kernel: Call 
Trace:<ffffffff80134b62>{autoremove_wake_function+0} 
<ffffffff80308223>{__down+147}
Jul 30 10:21:50 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffff80309cbb>{__down_failed+53}
Jul 30 10:21:50 lustre-3ware kernel:        
<ffffffffa034cccc>{:lquota:.text.lock.quota_master+15}
Jul 30 10:21:50 lustre-3ware kernel:        
<ffffffffa02964b6>{:ptlrpc:target_handle_dqacq_callback+953}
Jul 30 10:21:50 lustre-3ware kernel:        
<ffffffffa029fec8>{:ptlrpc:ldlm_callback_handler+1486}
Jul 30 10:21:51 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:21:51 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:21:51 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:21:51 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:21:51 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:21:51 lustre-3ware kernel:       
Jul 30 10:21:51 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185801710.3658
Jul 30 10:23:12 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 13615: it was 
inactive for 100s
Jul 30 10:23:12 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 13615
Jul 30 10:23:12 lustre-3ware kernel: ll_ost_io_34  D 
00000100726d0800     0 13615      1         13616 13614 (L-TLB)
Jul 30 10:23:12 lustre-3ware kernel: 00000100206876f8 0000000000000046 
0000010020687630 0000000000000202
Jul 30 10:23:12 lustre-3ware kernel:        000001006e056780 
0000000000000000 000001004d4e2458 0000000000000202
Jul 30 10:23:12 lustre-3ware kernel:        0000010019d8f800 
000000000000011b
Jul 30 10:23:12 lustre-3ware kernel: Call 
Trace:<ffffffff80308223>{__down+147} 
<ffffffff801331a5>{default_wake_function+0}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff80134b62>{autoremove_wake_function+0} 
<ffffffff80309cbb>{__down_failed+53}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff801a6f46>{.text.lock.dquot+335} 
<ffffffffa0441e21>{:fsfilt_ldiskfs:fsfilt_ldiskfs_quotactl+2032}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa03464b6>{:lquota:filter_quota_getflag+732}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa04ef88a>{:obdfilter:filter_commitrw_write+3763}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff80308f39>{thread_return+0} ll_mdt_01     S 000001007295d998     
0  3682      1          3683  3673 (L-TLB)
Jul 30 10:23:12 lustre-3ware kernel: 000001007295d8d8 0000000000000046 
00000000043bfe8b ffffffff00000073
Jul 30 10:23:12 lustre-3ware kernel:        00000000043bfe8b 
0000000000000000 0000010001021aa0 000000007b673b10
Jul 30 10:23:12 lustre-3ware kernel:        0000010077733030 
0000000000000889
Jul 30 10:23:12 lustre-3ware kernel: Call 
Trace:<ffffffff8013f100>{__mod_timer+293} 
<ffffffff80308f91>{thread_return+88}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa01f5bfe>{:lnet:lnet_send+2251} 
<ffffffffa04e9756>{:obdfilter:filter_commitrw+84}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff8013f23f>{del_timer+107} 
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff80309a5b>{schedule_timeout+367}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff8013fb2a>{process_timeout+0} 
<ffffffffa02aa7cb>{:ptlrpc:ptlrpc_queue_wait+2772}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02a63a5>{:ptlrpc:ptlrpc_prep_req_pool+1493}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa02a86cc>{:ptlrpc:expired_request+0} 
<ffffffffa02a8770>{:ptlrpc:interrupted_request+0}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa0344eb0>{:lquota:client_quota_ctl+447} 
<ffffffffa0345469>{:lquota:lov_quota_ctl+1159}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa034badb>{:lquota:mds_get_dqblk+1616} 
<ffffffffa02b05a8>{:ptlrpc:lustre_msg_add_version+67}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa02b0dc5>{:ptlrpc:lustre_pack_reply+1928}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa0344758>{:lquota:mds_quota_ctl+248} 
<ffffffffa045a755>{:mds:mds_handle_quotactl+994}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa045e19f>{:mds:mds_handle+14732} 
<ffffffff801315df>{activate_task+124}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff80131b1a>{try_to_wake_up+876} 
<ffffffff80309a63>{schedule_timeout+375} 
<ffffffffa04c6db1>{:ost:ost_brw_write+5253}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff80134b6b>{autoremove_wake_function+9}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa04c3513>{:ost:ost_bulk_timeout+0}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa02b1471>{:ptlrpc:lustre_msg_get_version+64}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa04ca500>{:ost:ost_handle+6987} 
<ffffffff801331f6>{__wake_up_common+67} 
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff80131b1a>{try_to_wake_up+876}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffff80131551>{recalc_task_prio+337} 
<ffffffff80134b6b>{autoremove_wake_function+9} 
<ffffffff801331f6>{__wake_up_common+67}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:23:12 lustre-3ware kernel:        
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:23:13 lustre-3ware kernel:       
Jul 30 10:23:13 lustre-3ware kernel: <ffffffff80308f39>{thread_return+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80308f91>{thread_return+88} 
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:23:13 lustre-3ware kernel:       
Jul 30 10:23:13 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185801792.3682
Jul 30 10:23:13 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185801792.13615
Jul 30 10:23:13 lustre-3ware kernel: ll_ost_io_97  D 
00000000b3665227     0 23087      1         23088 23086 (L-TLB)
Jul 30 10:23:13 lustre-3ware kernel: 000001005b659748 0000000000000046 
0000000000000000 ffffffff80308f39
Jul 30 10:23:13 lustre-3ware kernel:        000001005b659768 
ffffffff80308f91 ffffffff80366500 000000008024e0f7
Jul 30 10:23:13 lustre-3ware kernel:        000001006ca16800 
0000000000000138
Jul 30 10:23:13 lustre-3ware kernel: Call 
Trace:<ffffffff80308f39>{thread_return+0} 
<ffffffff80308f91>{thread_return+88}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80131551>{recalc_task_prio+337} 
<ffffffffa005e381>{:jbd:start_this_handle+897}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80131b1a>{try_to_wake_up+876} 
<ffffffff80134b62>{autoremove_wake_function+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80134b62>{autoremove_wake_function+0} 
<ffffffff80134b6b>{autoremove_wake_function+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa005e51e>{:jbd:journal_start+223} 
<ffffffffa03f174c>{:ldiskfs:ldiskfs_dquot_initialize+27}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04eed9d>{:obdfilter:filter_commitrw_write+966}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f5bfe>{:lnet:lnet_send+2251} 
<ffffffffa04e9756>{:obdfilter:filter_commitrw+84}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f23f>{del_timer+107} 
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80309a63>{schedule_timeout+375} 
<ffffffffa04c6db1>{:ost:ost_brw_write+5253}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa04c3513>{:ost:ost_bulk_timeout+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b1471>{:ptlrpc:lustre_msg_get_version+64}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04ca500>{:ost:ost_handle+6987}
<ffffffff801206e1>{dma_map_sg+642}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa002b75f>{:3w_9xxx:twa_post_command_packet+78}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa002bf72>{:3w_9xxx:twa_scsiop_execute_scsi+1312}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa0002874>{:scsi_mod:scsi_done+0} 
<ffffffffa002dc26>{:3w_9xxx:twa_scsi_queue+150}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80132452>{move_tasks+406} 
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:23:13 lustre-3ware kernel:       
Jul 30 10:23:13 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 13625: it was 
inactive for 100s
Jul 30 10:23:13 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Skipped 2 previous similar messages
Jul 30 10:23:13 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 13625
Jul 30 10:23:13 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 2 previous 
similar messages
Jul 30 10:23:13 lustre-3ware kernel: ll_ost_io_44  S 
0000000000000259     0 13625      1         13626 13624 (L-TLB)
Jul 30 10:23:13 lustre-3ware kernel: 00000100219fd5b8 0000000000000046 
0000010056c58a80 ffffffff801792ac
Jul 30 10:23:13 lustre-3ware kernel:        0000000000000246 
0000000000000212 0000010056c58a80 0000000032533710
Jul 30 10:23:13 lustre-3ware kernel:        0000010063d07030 
0000000000000530
Jul 30 10:23:13 lustre-3ware kernel: Call 
Trace:<ffffffff801792ac>{__find_get_block+396} 
<ffffffffa034286a>{:lquota:schedule_dqacq+2775}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa03f6515>{:ldiskfs:ldiskfs_ext_find_extent+500}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0}
<ffffffff801a48af>{dqput+136}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa0342b49>{:lquota:split_before_schedule_dqacq+248}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa034395b>{:lquota:qctxt_adjust_qunit+333}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa0440dd4>{:fsfilt_ldiskfs:fsfilt_ldiskfs_map_ext_inode_pages+457}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa0346667>{:lquota:filter_quota_acquire+122}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04ee325>{:obdfilter:filter_direct_io+1281}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa043f6f2>{:fsfilt_ldiskfs:fsfilt_ldiskfs_brw_start+649}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04ef77d>{:obdfilter:filter_commitrw_write+3494}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f5bfe>{:lnet:lnet_send+2251} 
<ffffffffa04e9756>{:obdfilter:filter_commitrw+84}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f23f>{del_timer+107} 
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80309a63>{schedule_timeout+375} 
<ffffffffa04c6db1>{:ost:ost_brw_write+5253}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa04c3513>{:ost:ost_bulk_timeout+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b1471>{:ptlrpc:lustre_msg_get_version+64}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04ca500>{:ost:ost_handle+6987} 
<ffffffff80131b1a>{try_to_wake_up+876}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80134b6b>{autoremove_wake_function+9} 
<ffffffff801331f6>{__wake_up_common+67}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80132452>{move_tasks+406} 
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:23:13 lustre-3ware kernel:       
Jul 30 10:23:13 lustre-3ware kernel: ll_ost_07     D 
00000000b366520f     0  3811      1          3812  3810 (L-TLB)
Jul 30 10:23:13 lustre-3ware kernel: 0000010071e89908 0000000000000046 
0000010077dbc800 000001007dc6c200
Jul 30 10:23:13 lustre-3ware kernel:        0000010077d98e80 
ffffffffa01f9256 000001007dc6c200 0000000000000080
Jul 30 10:23:13 lustre-3ware kernel:        00000100721a1030 
000000000000042c
Jul 30 10:23:13 lustre-3ware kernel: Call 
Trace:<ffffffffa01f9256>{:lnet:lolnd_recv+159} 
<ffffffffa005e381>{:jbd:start_this_handle+897}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa031253e>{:ksocklnd:ksocknal_queue_tx_locked+527}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80134b62>{autoremove_wake_function+0} 
<ffffffff80134b62>{autoremove_wake_function+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa005e51e>{:jbd:journal_start+223} 
<ffffffffa03f1850>{:ldiskfs:ldiskfs_acquire_dquot+46}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801a4efa>{dqget+710} <ffffffff801a695f>{vfs_get_dqblk+75}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa0441e21>{:fsfilt_ldiskfs:fsfilt_ldiskfs_quotactl+2032}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa0344a0d>{:lquota:filter_quota_ctl+346} 
<ffffffffa02b0dc5>{:ptlrpc:lustre_pack_reply+1928}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04c8083>{:ost:ost_handle_quotactl+983} 
<ffffffffa04cbdda>{:ost:ost_handle+13349}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff802a97a7>{alloc_skb+92} 
<ffffffffa00ba491>{:e1000:e1000_alloc_rx_buffers+641}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa00b5385>{:e1000:e1000_unmap_and_free_tx_resource+213}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:23:13 lustre-3ware kernel:       
Jul 30 10:23:13 lustre-3ware kernel: ll_ost_io_123 D 
00000000b366524b     0 28814      1         28815 28813 (L-TLB)
Jul 30 10:23:13 lustre-3ware kernel: 000001004c187748 0000000000000046 
000001006fa9e1f0 0000000000000000
Jul 30 10:23:13 lustre-3ware kernel:        000aac911268a0cd 
000001007c54d030 00000100010287e0 0000000000000001
Jul 30 10:23:13 lustre-3ware kernel:        0000010029ae1800 
0000000000000151
Jul 30 10:23:13 lustre-3ware kernel: Call 
Trace:<ffffffff80131551>{recalc_task_prio+337} 
<ffffffffa005e381>{:jbd:start_this_handle+897}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80131b1a>{try_to_wake_up+876} 
<ffffffff80134b62>{autoremove_wake_function+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80134b62>{autoremove_wake_function+0} 
<ffffffff80134b6b>{autoremove_wake_function+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa005e51e>{:jbd:journal_start+223} 
<ffffffffa03f174c>{:ldiskfs:ldiskfs_dquot_initialize+27}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04eed9d>{:obdfilter:filter_commitrw_write+966}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f5bfe>{:lnet:lnet_send+2251} 
<ffffffffa04e9756>{:obdfilter:filter_commitrw+84}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f23f>{del_timer+107} 
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80309a63>{schedule_timeout+375} 
<ffffffffa04c6db1>{:ost:ost_brw_write+5253}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa04c3513>{:ost:ost_bulk_timeout+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b1471>{:ptlrpc:lustre_msg_get_version+64}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04ca500>{:ost:ost_handle+6987} 
<ffffffff80134b6b>{autoremove_wake_function+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331f6>{__wake_up_common+67} 
<ffffffffa00b5385>{:e1000:e1000_unmap_and_free_tx_resource+213}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80132384>{move_tasks+200} 
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:23:13 lustre-3ware kernel:       
Jul 30 10:23:13 lustre-3ware kernel: ll_ost_io_62  D 
00000000b366524f     0 13646      1         13647 13642 (L-TLB)
Jul 30 10:23:13 lustre-3ware kernel: 000001003219b748 0000000000000046 
000001006988bdc0 0000000000000009
Jul 30 10:23:13 lustre-3ware kernel:        0000000000002706 
0000000000000001 0000010064604800 0000000000000000
Jul 30 10:23:13 lustre-3ware kernel:        0000010042d09800 
000000000000013f
Jul 30 10:23:13 lustre-3ware kernel: Call 
Trace:<ffffffff80131551>{recalc_task_prio+337} 
<ffffffffa005e381>{:jbd:start_this_handle+897}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80131b1a>{try_to_wake_up+876} <3>LustreError: 
3682:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent at 
1185801692, 101s ago) req@000001007dd86c00 x71040651/t0 
o19->home-OST0000_UUID@192.168.0.24@tcp:28 lens 240/240 ref 1 fl 
Rpc:/0/0 rc 0/-22
Jul 30 10:23:13 lustre-3ware kernel: Lustre: home-OST0000-osc: 
Connection to service home-OST0000 via nid 0@lo was lost; in progress 
operations using this service will wait for recovery to complete.
Jul 30 10:23:13 lustre-3ware kernel: 
<ffffffff80134b62>{autoremove_wake_function+0}
Jul 30 10:23:13 lustre-3ware kernel:        <1>LustreError: dumping log 
to /tmp/lustre-log.1185801793.28814
Jul 30 10:23:13 lustre-3ware kernel: Lustre: 
3813:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:23:13 lustre-3ware kernel: Lustre: 
3813:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:23:13 lustre-3ware kernel: LustreError: 
3813:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@000001005b4f2c00 x71040676/t0 
o8->home-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl 
Interpret:/0/0 rc -16/0
Jul 30 10:23:13 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001007e124400 x71040676/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:23:13 lustre-3ware kernel: 
<ffffffff80134b62>{autoremove_wake_function+0} 
<ffffffff80134b6b>{autoremove_wake_function+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa005e51e>{:jbd:journal_start+223} 
<ffffffffa03f174c>{:ldiskfs:ldiskfs_dquot_initialize+27}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04eed9d>{:obdfilter:filter_commitrw_write+966}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f5bfe>{:lnet:lnet_send+2251} 
<ffffffffa04e9756>{:obdfilter:filter_commitrw+84}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f23f>{del_timer+107} 
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80309a63>{schedule_timeout+375} 
<ffffffffa04c6db1>{:ost:ost_brw_write+5253}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa04c3513>{:ost:ost_bulk_timeout+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b1471>{:ptlrpc:lustre_msg_get_version+64}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04ca500>{:ost:ost_handle+6987} 
<ffffffff80131b1a>{try_to_wake_up+876}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80134b6b>{autoremove_wake_function+9} 
<ffffffff801331f6>{__wake_up_common+67}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:23:13 lustre-3ware kernel:       
Jul 30 10:23:13 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185801793.13646
Jul 30 10:23:13 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185801793.3811
Jul 30 10:23:13 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185801793.13625
Jul 30 10:23:13 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185801793.23087
Jul 30 10:23:13 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 23095: it was 
inactive for 100s
Jul 30 10:23:13 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Skipped 3 previous similar messages
Jul 30 10:23:13 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 23095
Jul 30 10:23:13 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 3 previous 
similar messages
Jul 30 10:23:13 lustre-3ware kernel: ll_ost_io_105 D 
00000000b36653e4     0 23095      1         23096 23094 (L-TLB)
Jul 30 10:23:13 lustre-3ware kernel: 00000100522ed748 0000000000000046 
0000000000000000 ffffffff80308f39
Jul 30 10:23:13 lustre-3ware kernel:        00000100522ed768 
ffffffff80308f91 ffffffff80366500 000000008024e0f7
Jul 30 10:23:13 lustre-3ware kernel:        0000010048744800 
000000000000011f
Jul 30 10:23:13 lustre-3ware kernel: Call 
Trace:<ffffffff80308f39>{thread_return+0} 
<ffffffff80308f91>{thread_return+88}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80131551>{recalc_task_prio+337} 
<ffffffffa005e381>{:jbd:start_this_handle+897}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80131b1a>{try_to_wake_up+876} 
<ffffffff80134b62>{autoremove_wake_function+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80134b62>{autoremove_wake_function+0} 
<ffffffff80134b6b>{autoremove_wake_function+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa005e51e>{:jbd:journal_start+223} 
<ffffffffa03f174c>{:ldiskfs:ldiskfs_dquot_initialize+27}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04eed9d>{:obdfilter:filter_commitrw_write+966}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f5bfe>{:lnet:lnet_send+2251} 
<ffffffffa04e9756>{:obdfilter:filter_commitrw+84}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f23f>{del_timer+107} 
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80309a63>{schedule_timeout+375} 
<ffffffffa04c6db1>{:ost:ost_brw_write+5253}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa04c3513>{:ost:ost_bulk_timeout+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b1471>{:ptlrpc:lustre_msg_get_version+64}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04ca500>{:ost:ost_handle+6987} 
<ffffffff80131b1a>{try_to_wake_up+876}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80134b6b>{autoremove_wake_function+9} 
<ffffffff801331f6>{__wake_up_common+67}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80132075>{__might_sleep+173} 
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80132384>{move_tasks+200} 
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:23:13 lustre-3ware kernel:       
Jul 30 10:23:13 lustre-3ware kernel: ll_ost_io_25  <1>LustreError: 
dumping log to /tmp/lustre-log.1185801793.23095
Jul 30 10:23:13 lustre-3ware kernel: D 00000000b36653e8     0  3861      
1          3862  3860 (L-TLB)
Jul 30 10:23:13 lustre-3ware kernel: 000001007032f748 0000000000000046 
0000000000000000 ffffffff80308f39
Jul 30 10:23:13 lustre-3ware kernel:        000001007032f768 
ffffffff80308f91 ffffffff80366500 000000008024e0f7
Jul 30 10:23:13 lustre-3ware kernel:        00000100701f6030 
0000000000000155
Jul 30 10:23:13 lustre-3ware kernel: Call 
Trace:<ffffffff80308f39>{thread_return+0} 
<ffffffff80308f91>{thread_return+88}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80131551>{recalc_task_prio+337} 
<ffffffffa005e381>{:jbd:start_this_handle+897}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80131b1a>{try_to_wake_up+876} 
<ffffffff80134b62>{autoremove_wake_function+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80134b62>{autoremove_wake_function+0} 
<ffffffff80134b6b>{autoremove_wake_function+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa005e51e>{:jbd:journal_start+223} 
<ffffffffa03f174c>{:ldiskfs:ldiskfs_dquot_initialize+27}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04eed9d>{:obdfilter:filter_commitrw_write+966}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f5bfe>{:lnet:lnet_send+2251} 
<ffffffffa04e9756>{:obdfilter:filter_commitrw+84}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f23f>{del_timer+107} 
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80309a63>{schedule_timeout+375} 
<ffffffffa04c6db1>{:ost:ost_brw_write+5253}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa04c3513>{:ost:ost_bulk_timeout+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b1471>{:ptlrpc:lustre_msg_get_version+64}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04ca500>{:ost:ost_handle+6987} 
<ffffffff80131b1a>{try_to_wake_up+876}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff802a97a7>{alloc_skb+92} 
<ffffffffa01f645c>{:lnet:lnet_match_blocked_msg+712}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:23:13 lustre-3ware kernel:       
Jul 30 10:23:13 lustre-3ware kernel: ll_ost_io_02  D 
00000000b36653ec     0  3838      1          3839  3837 (L-TLB)
Jul 30 10:23:13 lustre-3ware kernel: 0000010071b2d748 0000000000000046 
000001006fa9e1f0 0000000000000000
Jul 30 10:23:13 lustre-3ware kernel:        000aac9114c03b06 
000001007c54d030 00000100010287e0 0000000000000001
Jul 30 10:23:13 lustre-3ware kernel:        0000010071afa800 
000000000000013f
Jul 30 10:23:13 lustre-3ware kernel: Call 
Trace:<ffffffff80131551>{recalc_task_prio+337} 
<ffffffffa005e381>{:jbd:start_this_handle+897}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80131b1a>{try_to_wake_up+876} 
<ffffffff80134b62>{autoremove_wake_function+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80134b62>{autoremove_wake_function+0} 
<ffffffff80134b6b>{autoremove_wake_function+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa005e51e>{:jbd:journal_start+223} 
<ffffffffa03f174c>{:ldiskfs:ldiskfs_dquot_initialize+27}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04eed9d>{:obdfilter:filter_commitrw_write+966}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f5bfe>{:lnet:lnet_send+2251} 
<ffffffffa04e9756>{:obdfilter:filter_commitrw+84}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f23f>{del_timer+107} 
<ffffffff8013f2fc>{del_singleshot_timer_sync+9}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80309a63>{schedule_timeout+375} 
<ffffffffa04c6db1>{:ost:ost_brw_write+5253}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa04c3513>{:ost:ost_bulk_timeout+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b1471>{:ptlrpc:lustre_msg_get_version+64}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa04ca500>{:ost:ost_handle+6987} 
<ffffffff80131b1a>{try_to_wake_up+876}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80134b6b>{autoremove_wake_function+9} 
<ffffffff801331f6>{__wake_up_common+67}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff80132384>{move_tasks+200} 
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:23:13 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:23:13 lustre-3ware kernel:       
Jul 30 10:23:13 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185801793.3838
Jul 30 10:23:13 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185801793.3861
Jul 30 10:24:03 lustre-3ware kernel: Lustre: 
3824:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:24:03 lustre-3ware kernel: Lustre: 
3824:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:24:03 lustre-3ware kernel: LustreError: 
3824:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@00000100013bec00 x71040688/t0 
o8->home-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl 
Interpret:/0/0 rc -16/0
Jul 30 10:24:03 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@0000010074209e00 x71040688/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:24:52 lustre-3ware kernel: LustreError: 
3537:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent at 
1185801692, 200s ago) req@0000010077da6800 x71040650/t0 
o601->@192.168.0.24@tcp:15 lens 144/144 ref 1 fl Rpc:/0/0 rc 0/-22
Jul 30 10:24:52 lustre-3ware kernel: Lustre: 
3708:0:(ldlm_lib.c:497:target_handle_reconnect()) home-MDT0000: 
3579ecd4-03f1-8760-6b01-8e3313ff730d reconnecting
Jul 30 10:24:52 lustre-3ware kernel: Lustre: 
3708:0:(ldlm_lib.c:709:target_handle_connect()) home-MDT0000: refuse 
reconnection from 
3579ecd4-03f1-8760-6b01-8e3313ff730d@129.173.118.68@tcp to 
0x000001001d367000/3
Jul 30 10:24:52 lustre-3ware kernel: LustreError: 
3708:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@0000010072e11050 x158273013/t0 
o38->3579ecd4-03f1-8760-6b01-8e3313ff730d@NET_0x2000081ad7644_UUID:-1 
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
Jul 30 10:24:53 lustre-3ware kernel: Lustre: 
3810:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:24:53 lustre-3ware kernel: Lustre: 
3810:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:24:53 lustre-3ware kernel: LustreError: 
3810:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@000001007df6f400 x71040700/t0 
o8->home-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl 
Interpret:/0/0 rc -16/0
Jul 30 10:24:53 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001000139d200 x71040700/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:24:53 lustre-3ware kernel: Lustre: 
3812:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
3579ecd4-03f1-8760-6b01-8e3313ff730d reconnecting
Jul 30 10:24:53 lustre-3ware kernel: Lustre: 
3812:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from 
3579ecd4-03f1-8760-6b01-8e3313ff730d@129.173.118.68@tcp to 
0x000001004d4e2000/9
Jul 30 10:24:53 lustre-3ware kernel: LustreError: 
3812:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@000001007e122a00 x158273014/t0 
o8->3579ecd4-03f1-8760-6b01-8e3313ff730d@NET_0x2000081ad7644_UUID:-1 
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
Jul 30 10:25:10 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 3656: it was 
inactive for 18s
Jul 30 10:25:10 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Skipped 2 previous similar messages
Jul 30 10:25:10 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 3656
Jul 30 10:25:10 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 2 previous 
similar messages
Jul 30 10:25:10 lustre-3ware kernel: ldlm_cb_03    D 
0000000000000001     0  3656      1          3657  3655 (L-TLB)
Jul 30 10:25:10 lustre-3ware kernel: 000001007395dc48 0000000000000046 
0000010077d98e80 0000000000000000
Jul 30 10:25:10 lustre-3ware kernel:        0000010074b9de00 
0009000000000000 00000000043bfe83 00000001a01f481a
Jul 30 10:25:10 lustre-3ware kernel:        000001007404c030 
000000000000041c
Jul 30 10:25:10 lustre-3ware kernel: Call 
Trace:<ffffffff80308223>{__down+147} 
<ffffffff801331a5>{default_wake_function+0}
Jul 30 10:25:10 lustre-3ware kernel:        
<ffffffff80309cbb>{__down_failed+53} 
<ffffffffa034cccc>{:lquota:.text.lock.quota_master+15}
Jul 30 10:25:10 lustre-3ware kernel:        
<ffffffffa02964b6>{:ptlrpc:target_handle_dqacq_callback+953}
Jul 30 10:25:10 lustre-3ware kernel:        
<ffffffffa029fec8>{:ptlrpc:ldlm_callback_handler+1486}
Jul 30 10:25:10 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:25:10 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:25:10 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:25:10 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:25:10 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:25:10 lustre-3ware kernel:       
Jul 30 10:25:10 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185801910.3656
Jul 30 10:25:43 lustre-3ware kernel: Lustre: 
3830:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:25:43 lustre-3ware kernel: Lustre: 
3830:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:25:43 lustre-3ware kernel: LustreError: 
3830:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@00000100723cf850 x71040709/t0 
o8->home-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl 
Interpret:/0/0 rc -16/0
Jul 30 10:25:43 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@0000010077dbbe00 x71040709/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:25:43 lustre-3ware kernel: Lustre: 
3806:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
3579ecd4-03f1-8760-6b01-8e3313ff730d reconnecting
Jul 30 10:25:43 lustre-3ware kernel: Lustre: 
3697:0:(ldlm_lib.c:709:target_handle_connect()) home-MDT0000: refuse 
reconnection from 
3579ecd4-03f1-8760-6b01-8e3313ff730d@129.173.118.68@tcp to 
0x000001001d367000/3
Jul 30 10:25:43 lustre-3ware kernel: LustreError: 
3697:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@0000010069db9c00 x158273019/t0 
o38->3579ecd4-03f1-8760-6b01-8e3313ff730d@NET_0x2000081ad7644_UUID:-1 
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
Jul 30 10:25:43 lustre-3ware kernel: Lustre: 
3806:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 1 previous 
similar message
Jul 30 10:26:31 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 3707: it was 
inactive for 100s
Jul 30 10:26:31 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 3707
Jul 30 10:26:31 lustre-3ware kernel: ll_mdt_26     D 
00000100724ea4e0     0  3707      1          3708  3706 (L-TLB)
Jul 30 10:26:31 lustre-3ware kernel: 0000010072e3b9d8 0000000000000046 
0000010077d98d80 0000000000000246
Jul 30 10:26:31 lustre-3ware kernel:        0000000000000246 
ffffffffa0312a93 00020000c0a8000c 0000000100003039
Jul 30 10:26:31 lustre-3ware kernel:        00000100729a9030 
000000000000031c
Jul 30 10:26:31 lustre-3ware kernel: Call 
Trace:<ffffffffa0312a93>{:ksocklnd:ksocknal_launch_packet+443}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffffa0312e56>{:ksocklnd:ksocknal_send+632} 
<ffffffff80308223>{__down+147}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffff80309cbb>{__down_failed+53}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffffa034cdda>{:lquota:.text.lock.quota_master+285}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffffa02b05a8>{:ptlrpc:lustre_msg_add_version+67}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffffa02b0dc5>{:ptlrpc:lustre_pack_reply+1928}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffffa0344758>{:lquota:mds_quota_ctl+248} 
<ffffffffa045a755>{:mds:mds_handle_quotactl+994}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffffa045e19f>{:mds:mds_handle+14732} 
<ffffffff802e1e2e>{tcp_v4_rcv+1761}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffffa00ba491>{:e1000:e1000_alloc_rx_buffers+641}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffff802af3b8>{netif_receive_skb+791} 
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffff80131551>{recalc_task_prio+337} 
<ffffffff80308f39>{thread_return+0}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffff80308f91>{thread_return+88} 
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:26:31 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:26:31 lustre-3ware kernel:       
Jul 30 10:26:31 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185801991.3707
Jul 30 10:26:33 lustre-3ware kernel: Lustre: 
3828:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:26:33 lustre-3ware kernel: Lustre: 
3828:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:26:33 lustre-3ware kernel: Lustre: 
3828:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 1 previous 
similar message
Jul 30 10:26:33 lustre-3ware kernel: LustreError: 
3828:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@0000010071c52050 x71040722/t0 
o8->home-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl 
Interpret:/0/0 rc -16/0
Jul 30 10:26:33 lustre-3ware kernel: LustreError: 
3828:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 1 previous 
similar message
Jul 30 10:26:33 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@00000100013bfe00 x71040722/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:26:33 lustre-3ware kernel: Lustre: 
3701:0:(ldlm_lib.c:497:target_handle_reconnect()) home-MDT0000: 
3579ecd4-03f1-8760-6b01-8e3313ff730d reconnecting
Jul 30 10:26:33 lustre-3ware kernel: Lustre: 
3701:0:(ldlm_lib.c:709:target_handle_connect()) home-MDT0000: refuse 
reconnection from 
3579ecd4-03f1-8760-6b01-8e3313ff730d@129.173.118.68@tcp to 
0x000001001d367000/3
Jul 30 10:27:23 lustre-3ware kernel: Lustre: 
3818:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:27:23 lustre-3ware kernel: Lustre: 
3818:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 1 previous 
similar message
Jul 30 10:27:23 lustre-3ware kernel: Lustre: 
3818:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:27:23 lustre-3ware kernel: Lustre: 
3818:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 1 previous 
similar message
Jul 30 10:27:23 lustre-3ware kernel: LustreError: 
3818:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@000001007238a050 x71040727/t0 
o8->home-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl 
Interpret:/0/0 rc -16/0
Jul 30 10:27:23 lustre-3ware kernel: LustreError: 
3818:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 2 previous 
similar messages
Jul 30 10:27:23 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001004f9d0c00 x71040727/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:28:12 lustre-3ware kernel: LustreError: 
3537:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent at 
1185801892, 200s ago) req@0000010077da6800 x71040650/t0 
o601->@192.168.0.24@tcp:15 lens 144/144 ref 1 fl Rpc:/0/0 rc 0/-22
Jul 30 10:28:13 lustre-3ware kernel: Lustre: 
3831:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:28:13 lustre-3ware kernel: Lustre: 
3831:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 2 previous 
similar messages
Jul 30 10:28:13 lustre-3ware kernel: Lustre: 
3831:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:28:13 lustre-3ware kernel: Lustre: 
3831:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 2 previous 
similar messages
Jul 30 10:28:13 lustre-3ware kernel: LustreError: 
3831:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@0000010072a67450 x71040730/t0 
o8->home-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl 
Interpret:/0/0 rc -16/0
Jul 30 10:28:13 lustre-3ware kernel: LustreError: 
3831:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 2 previous 
similar messages
Jul 30 10:28:13 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001000139d400 x71040730/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:28:30 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 3655: it was 
inactive for 18s
Jul 30 10:28:30 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 3655
Jul 30 10:28:30 lustre-3ware kernel: ldlm_cb_02    D 
0000000000000001     0  3655      1          3656  3654 (L-TLB)
Jul 30 10:28:30 lustre-3ware kernel: 0000010073959c48 0000000000000046 
0000010077d98e80 0000000000000000
Jul 30 10:28:30 lustre-3ware kernel:        000001007c585400 
0009000000000000 00000000043bfe84 00000000a01f481a
Jul 30 10:28:30 lustre-3ware kernel:        000001007404c800 
00000000000004af
Jul 30 10:28:30 lustre-3ware kernel: Call 
Trace:<ffffffff80308223>{__down+147} 
<ffffffff801331a5>{default_wake_function+0}
Jul 30 10:28:30 lustre-3ware kernel:        
<ffffffff80309cbb>{__down_failed+53} 
<ffffffffa034cccc>{:lquota:.text.lock.quota_master+15}
Jul 30 10:28:30 lustre-3ware kernel:        
<ffffffffa02964b6>{:ptlrpc:target_handle_dqacq_callback+953}
Jul 30 10:28:30 lustre-3ware kernel:        
<ffffffffa029fec8>{:ptlrpc:ldlm_callback_handler+1486}
Jul 30 10:28:30 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:28:30 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:28:30 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:28:30 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:28:30 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:28:30 lustre-3ware kernel:       
Jul 30 10:28:30 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185802110.3655
Jul 30 10:28:31 lustre-3ware kernel: ll_mdt_30     S 
0000000000000000     0  3711      1          3712  3710 (L-TLB)
Jul 30 10:28:31 lustre-3ware kernel: 00000100725d7298 0000000000000046 
0000010001390030 0000010000003039
Jul 30 10:28:31 lustre-3ware kernel:        00000000000000e0 
000001007b5a9c80 00000000000000e0 0000000137e84a00
Jul 30 10:28:31 lustre-3ware kernel:        000001007299b030 
00000000000001c1
Jul 30 10:28:31 lustre-3ware kernel: Call 
Trace:<ffffffff8013f100>{__mod_timer+293} 
<ffffffff80309a5b>{schedule_timeout+367}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffff8013fb2a>{process_timeout+0} 
<ffffffffa02a9ae0>{:ptlrpc:ptlrpc_set_wait+755}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02a838c>{:ptlrpc:ptlrpc_expired_set+0}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa02a6ac8>{:ptlrpc:ptlrpc_interrupted_set+0}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa02a838c>{:ptlrpc:ptlrpc_expired_set+0} 
<ffffffffa02a6ac8>{:ptlrpc:ptlrpc_interrupted_set+0}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa035e9dd>{:lov:lov_create+5624} 
<ffffffffa03f34af>{:ldiskfs:ldiskfs_xattr_ibody_get+403}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffff801e8f7d>{__up_read+16} 
<ffffffffa03f461d>{:ldiskfs:ldiskfs_xattr_get+120}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa043ff3d>{:fsfilt_ldiskfs:fsfilt_ldiskfs_get_md+101}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa04575c1>{:mds:mds_get_md+105} 
<ffffffffa047755b>{:mds:mds_create_objects+3818}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa04581c2>{:mds:mds_pack_md+409} 
<ffffffffa04791aa>{:mds:mds_finish_open+704}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa047b625>{:mds:mds_open+6486} 
<ffffffffa046fc91>{:mds:mds_reint_rec+373}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa0459e26>{:mds:mds_reint+637} 
<ffffffffa046222a>{:mds:mds_intent_policy+890}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa028eeee>{:ptlrpc:ldlm_resource_putref+356}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa028aca2>{:ptlrpc:ldlm_lock_create+1375} 
<ffffffffa028bd85>{:ptlrpc:ldlm_lock_enqueue+208}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa029e7d7>{:ptlrpc:ldlm_handle_enqueue+2524}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa029d1ba>{:ptlrpc:ldlm_server_blocking_ast+0}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa029d72d>{:ptlrpc:ldlm_server_completion_ast+0}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa045e3a9>{:mds:mds_handle+15254} 
<ffffffff802e1e2e>{tcp_v4_rcv+1761}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffff80131b1a>{try_to_wake_up+876}
<ffffffff802a97a7>{alloc_skb+92}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffff802af3b8>{netif_receive_skb+791} 
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffff80131551>{recalc_task_prio+337} 
<ffffffff80308f39>{thread_return+0}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffff80308f91>{thread_return+88} 
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:28:31 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:28:31 lustre-3ware kernel:       
Jul 30 10:28:31 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185802111.3711
Jul 30 10:29:03 lustre-3ware kernel: Lustre: 
3833:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:29:03 lustre-3ware kernel: Lustre: 
3833:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 2 previous 
similar messages
Jul 30 10:29:03 lustre-3ware kernel: Lustre: 
3833:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:29:03 lustre-3ware kernel: Lustre: 
3833:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 2 previous 
similar messages
Jul 30 10:29:03 lustre-3ware kernel: LustreError: 
3833:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@0000010072378050 x71040733/t0 
o8->home-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl 
Interpret:/0/0 rc -16/0
Jul 30 10:29:03 lustre-3ware kernel: LustreError: 
3833:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 2 previous 
similar messages
Jul 30 10:29:03 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@0000010013aea400 x71040733/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:29:53 lustre-3ware kernel: Lustre: 
3809:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:29:53 lustre-3ware kernel: Lustre: 
3809:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 2 previous 
similar messages
Jul 30 10:29:53 lustre-3ware kernel: Lustre: 
3809:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:29:53 lustre-3ware kernel: Lustre: 
3809:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 2 previous 
similar messages
Jul 30 10:29:53 lustre-3ware kernel: LustreError: 
3809:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@0000010071e1b850 x71040736/t0 
o8->home-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl 
Interpret:/0/0 rc -16/0
Jul 30 10:29:53 lustre-3ware kernel: LustreError: 
3809:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 2 previous 
similar messages
Jul 30 10:29:53 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001007dcf1400 x71040736/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:30:11 lustre-3ware kernel: Lustre: 
3704:0:(ldlm_lib.c:497:target_handle_reconnect()) home-MDT0000: 
38868c2b-58bf-f3c5-4950-33079393b6b0 reconnecting
Jul 30 10:30:11 lustre-3ware kernel: Lustre: 
3704:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 2 previous 
similar messages
Jul 30 10:30:11 lustre-3ware kernel: Lustre: 
3704:0:(ldlm_lib.c:709:target_handle_connect()) home-MDT0000: refuse 
reconnection from 38868c2b-58bf-f3c5-4950-33079393b6b0@192.168.0.12@tcp 
to 0x000001006a922000/2
Jul 30 10:30:11 lustre-3ware kernel: Lustre: 
3704:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 2 previous 
similar messages
Jul 30 10:30:11 lustre-3ware kernel: LustreError: 
3704:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@0000010011048c00 x1101183/t0 
o38->38868c2b-58bf-f3c5-4950-33079393b6b0@NET_0x20000c0a8000c_UUID:-1 
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
Jul 30 10:30:11 lustre-3ware kernel: LustreError: 
3704:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 2 previous 
similar messages
Jul 30 10:30:43 lustre-3ware kernel: Lustre: 
3829:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:30:43 lustre-3ware kernel: Lustre: 
3829:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:30:43 lustre-3ware kernel: LustreError: 
3829:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@000001007dc9a600 x71040739/t0 
o8->home-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl 
Interpret:/0/0 rc -16/0
Jul 30 10:30:43 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001007dc95400 x71040739/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:31:01 lustre-3ware kernel: Lustre: 
3704:0:(ldlm_lib.c:497:target_handle_reconnect()) home-MDT0000: 
38868c2b-58bf-f3c5-4950-33079393b6b0 reconnecting
Jul 30 10:31:01 lustre-3ware kernel: Lustre: 
3704:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 2 previous 
similar messages
Jul 30 10:31:01 lustre-3ware kernel: Lustre: 
3704:0:(ldlm_lib.c:709:target_handle_connect()) home-MDT0000: refuse 
reconnection from 38868c2b-58bf-f3c5-4950-33079393b6b0@192.168.0.12@tcp 
to 0x000001006a922000/2
Jul 30 10:31:01 lustre-3ware kernel: Lustre: 
3704:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 2 previous 
similar messages
Jul 30 10:31:01 lustre-3ware kernel: LustreError: 
3704:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@000001003c5dd200 x1101187/t0 
o38->38868c2b-58bf-f3c5-4950-33079393b6b0@NET_0x20000c0a8000c_UUID:-1 
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
Jul 30 10:31:01 lustre-3ware kernel: LustreError: 
3704:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 2 previous 
similar messages
Jul 30 10:31:32 lustre-3ware kernel: LustreError: 
3537:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent at 
1185802092, 200s ago) req@0000010077da6800 x71040650/t0 
o601->@192.168.0.24@tcp:15 lens 144/144 ref 1 fl Rpc:/0/0 rc 0/-22
Jul 30 10:31:33 lustre-3ware kernel: Lustre: 
3829:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:31:33 lustre-3ware kernel: Lustre: 
3829:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:31:33 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@0000010013aea200 x71040742/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:31:50 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 3661: it was 
inactive for 18s
Jul 30 10:31:50 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Skipped 1 previous similar message
Jul 30 10:31:50 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 3661
Jul 30 10:31:50 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 1 previous 
similar message
Jul 30 10:31:50 lustre-3ware kernel: ldlm_cb_08    D 
0000000000000001     0  3661      1          3662  3660 (L-TLB)
Jul 30 10:31:50 lustre-3ware kernel: 0000010073969c48 0000000000000046 
0000010077d98e80 0000000000000074
Jul 30 10:31:50 lustre-3ware kernel:        000001005b4f2400 
0000000000000000 0000010001029aa0 00000001a01f481a
Jul 30 10:31:50 lustre-3ware kernel:        000001007444d800 
0000000000000435
Jul 30 10:31:50 lustre-3ware kernel: Call 
Trace:<ffffffff80308223>{__down+147} 
<ffffffff801331a5>{default_wake_function+0}
Jul 30 10:31:50 lustre-3ware kernel:        
<ffffffff80309cbb>{__down_failed+53} 
<ffffffffa034cccc>{:lquota:.text.lock.quota_master+15}
Jul 30 10:31:50 lustre-3ware kernel:        
<ffffffffa02964b6>{:ptlrpc:target_handle_dqacq_callback+953}
Jul 30 10:31:50 lustre-3ware kernel:        
<ffffffffa029fec8>{:ptlrpc:ldlm_callback_handler+1486}
Jul 30 10:31:50 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:31:50 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:31:50 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:31:50 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:31:50 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:31:50 lustre-3ware kernel:       
Jul 30 10:31:50 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185802310.3661
Jul 30 10:31:51 lustre-3ware kernel: LustreError: 
3683:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@0000010078a95c00 x1101192/t0 
o38->38868c2b-58bf-f3c5-4950-33079393b6b0@NET_0x20000c0a8000c_UUID:-1 
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
Jul 30 10:31:51 lustre-3ware kernel: LustreError: 
3683:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 3 previous 
similar messages
Jul 30 10:32:23 lustre-3ware kernel: Lustre: 
3823:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:32:23 lustre-3ware kernel: Lustre: 
3823:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 3 previous 
similar messages
Jul 30 10:32:23 lustre-3ware kernel: Lustre: 
3823:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:32:23 lustre-3ware kernel: Lustre: 
3823:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 3 previous 
similar messages
Jul 30 10:32:23 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001007dcf2200 x71040745/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:33:13 lustre-3ware kernel: LustreError: 
3823:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@000001007dcd3600 x158273075/t0 
o8->3579ecd4-03f1-8760-6b01-8e3313ff730d@NET_0x2000081ad7644_UUID:-1 
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
Jul 30 10:33:13 lustre-3ware kernel: LustreError: 
3823:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 5 previous 
similar messages
Jul 30 10:33:13 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001007dec0800 x71040748/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:33:31 lustre-3ware kernel: Lustre: 
3686:0:(ldlm_lib.c:497:target_handle_reconnect()) home-MDT0000: 
38868c2b-58bf-f3c5-4950-33079393b6b0 reconnecting
Jul 30 10:33:31 lustre-3ware kernel: Lustre: 
3686:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 6 previous 
similar messages
Jul 30 10:33:31 lustre-3ware kernel: Lustre: 
3686:0:(ldlm_lib.c:709:target_handle_connect()) home-MDT0000: refuse 
reconnection from 38868c2b-58bf-f3c5-4950-33079393b6b0@192.168.0.12@tcp 
to 0x000001006a922000/2
Jul 30 10:33:31 lustre-3ware kernel: Lustre: 
3686:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 6 previous 
similar messages
Jul 30 10:34:03 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001007dc9a600 x71040751/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:34:52 lustre-3ware kernel: LustreError: 
3537:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent at 
1185802292, 200s ago) req@0000010077da6800 x71040650/t0 
o601->@192.168.0.24@tcp:15 lens 144/144 ref 1 fl Rpc:/0/0 rc 0/-22
Jul 30 10:34:53 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001007dffe600 x71040754/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:35:10 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 3657: it was 
inactive for 18s
Jul 30 10:35:10 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 3657
Jul 30 10:35:10 lustre-3ware kernel: ldlm_cb_04    D 
0000000000000001     0  3657      1          3658  3656 (L-TLB)
Jul 30 10:35:10 lustre-3ware kernel: 000001007395fc48 0000000000000046 
0000010077d98e80 0000000000000074
Jul 30 10:35:10 lustre-3ware kernel:        000001005b4f2400 
0000000000000000 0000010001029aa0 00000001a01f481a
Jul 30 10:35:10 lustre-3ware kernel:        00000100746f8800 
000000000000042f
Jul 30 10:35:10 lustre-3ware kernel: Call 
Trace:<ffffffff80308223>{__down+147} 
<ffffffff801331a5>{default_wake_function+0}
Jul 30 10:35:10 lustre-3ware kernel:        
<ffffffff80309cbb>{__down_failed+53} 
<ffffffffa034cccc>{:lquota:.text.lock.quota_master+15}
Jul 30 10:35:10 lustre-3ware kernel:        
<ffffffffa02964b6>{:ptlrpc:target_handle_dqacq_callback+953}
Jul 30 10:35:10 lustre-3ware kernel:        
<ffffffffa029fec8>{:ptlrpc:ldlm_callback_handler+1486}
Jul 30 10:35:10 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:35:10 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:35:10 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:35:10 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:35:10 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:35:10 lustre-3ware kernel:       
Jul 30 10:35:10 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185802510.3657
Jul 30 10:35:43 lustre-3ware kernel: Lustre: 
3684:0:(ldlm_lib.c:497:target_handle_reconnect()) home-MDT0000: 
3579ecd4-03f1-8760-6b01-8e3313ff730d reconnecting
Jul 30 10:35:43 lustre-3ware kernel: Lustre: 
3826:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from 
3579ecd4-03f1-8760-6b01-8e3313ff730d@129.173.118.68@tcp to 
0x000001004d4e2000/9
Jul 30 10:35:43 lustre-3ware kernel: Lustre: 
3826:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 8 previous 
similar messages
Jul 30 10:35:43 lustre-3ware kernel: LustreError: 
3826:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@0000010013aeae00 x158273091/t0 
o8->3579ecd4-03f1-8760-6b01-8e3313ff730d@NET_0x2000081ad7644_UUID:-1 
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
Jul 30 10:35:43 lustre-3ware kernel: LustreError: 
3826:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 10 previous 
similar messages
Jul 30 10:35:43 lustre-3ware kernel: Lustre: 
3684:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 9 previous 
similar messages
Jul 30 10:35:43 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001003c5ddc00 x71040757/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:36:33 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001007e121200 x71040760/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:37:23 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001000139be00 x71040763/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:38:12 lustre-3ware kernel: LustreError: 
3537:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent at 
1185802492, 200s ago) req@0000010077da6800 x71040650/t0 
o601->@192.168.0.24@tcp:15 lens 144/144 ref 1 fl Rpc:/0/0 rc 0/-22
Jul 30 10:38:30 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 3660: it was 
inactive for 18s
Jul 30 10:38:30 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 3660
Jul 30 10:38:30 lustre-3ware kernel: ldlm_cb_07    D 
0000000000000001     0  3660      1          3661  3659 (L-TLB)
Jul 30 10:38:30 lustre-3ware kernel: 0000010073967c48 0000000000000046 
0000010077d98e80 0000000000000074
Jul 30 10:38:30 lustre-3ware kernel:        000001005b4f2400 
0000000000000000 0000010001029aa0 00000001a01f481a
Jul 30 10:38:30 lustre-3ware kernel:        000001007404a030 
0000000000000464
Jul 30 10:38:30 lustre-3ware kernel: Call 
Trace:<ffffffff80308223>{__down+147} 
<ffffffff801331a5>{default_wake_function+0}
Jul 30 10:38:30 lustre-3ware kernel:        
<ffffffff80309cbb>{__down_failed+53} 
<ffffffffa034cccc>{:lquota:.text.lock.quota_master+15}
Jul 30 10:38:30 lustre-3ware kernel:        
<ffffffffa02964b6>{:ptlrpc:target_handle_dqacq_callback+953}
Jul 30 10:38:30 lustre-3ware kernel:        
<ffffffffa029fec8>{:ptlrpc:ldlm_callback_handler+1486}
Jul 30 10:38:30 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:38:30 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:38:30 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:38:30 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:38:30 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:38:30 lustre-3ware kernel:       
Jul 30 10:38:30 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185802710.3660
Jul 30 10:39:03 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@0000010037e84c00 x71040769/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:39:03 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) Skipped 1 previous similar 
message
Jul 30 10:40:11 lustre-3ware kernel: Lustre: 
3698:0:(ldlm_lib.c:497:target_handle_reconnect()) home-MDT0000: 
38868c2b-58bf-f3c5-4950-33079393b6b0 reconnecting
Jul 30 10:40:11 lustre-3ware kernel: Lustre: 
3698:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 21 previous 
similar messages
Jul 30 10:40:11 lustre-3ware kernel: Lustre: 
3698:0:(ldlm_lib.c:709:target_handle_connect()) home-MDT0000: refuse 
reconnection from 38868c2b-58bf-f3c5-4950-33079393b6b0@192.168.0.12@tcp 
to 0x000001006a922000/2
Jul 30 10:40:11 lustre-3ware kernel: Lustre: 
3698:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 22 previous 
similar messages
Jul 30 10:40:11 lustre-3ware kernel: LustreError: 
3698:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@0000010077dbb800 x1101242/t0 
o38->38868c2b-58bf-f3c5-4950-33079393b6b0@NET_0x20000c0a8000c_UUID:-1 
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
Jul 30 10:40:11 lustre-3ware kernel: LustreError: 
3698:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 22 previous 
similar messages
Jul 30 10:41:32 lustre-3ware kernel: LustreError: 
3537:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent at 
1185802692, 200s ago) req@0000010077da6800 x71040650/t0 
o601->@192.168.0.24@tcp:15 lens 144/144 ref 1 fl Rpc:/0/0 rc 0/-22
Jul 30 10:41:33 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001005b4f2000 x71040778/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:41:33 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) Skipped 2 previous similar 
messages
Jul 30 10:41:50 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 3659: it was 
inactive for 18s
Jul 30 10:41:50 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 3659
Jul 30 10:41:50 lustre-3ware kernel: ldlm_cb_06    D 
0000000000000001     0  3659      1          3660  3658 (L-TLB)
Jul 30 10:41:50 lustre-3ware kernel: 0000010073965c48 0000000000000046 
0000010077d98e80 0000000000000000
Jul 30 10:41:50 lustre-3ware kernel:        0000010077da6800 
0009000000000000 00000000043bfe88 00000000a01f481a
Jul 30 10:41:50 lustre-3ware kernel:        000001007404a800 
00000000000004e2
Jul 30 10:41:50 lustre-3ware kernel: Call 
Trace:<ffffffff80308223>{__down+147} 
<ffffffff801331a5>{default_wake_function+0}
Jul 30 10:41:50 lustre-3ware kernel:        
<ffffffff80309cbb>{__down_failed+53} 
<ffffffffa034cccc>{:lquota:.text.lock.quota_master+15}
Jul 30 10:41:50 lustre-3ware kernel:        
<ffffffffa02964b6>{:ptlrpc:target_handle_dqacq_callback+953}
Jul 30 10:41:50 lustre-3ware kernel:        
<ffffffffa029fec8>{:ptlrpc:ldlm_callback_handler+1486}
Jul 30 10:41:50 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:41:50 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:41:50 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:41:50 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:41:50 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:41:50 lustre-3ware kernel:       
Jul 30 10:41:50 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185802910.3659
Jul 30 10:45:10 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 413: it was 
inactive for 18s
Jul 30 10:45:10 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 413
Jul 30 10:45:10 lustre-3ware kernel: ldlm_cb_09    D 
0000000000000001     0   413      1               31690 (L-TLB)
Jul 30 10:45:10 lustre-3ware kernel: 000001003b285c48 0000000000000046 
0000000000000000 0000000000000073
Jul 30 10:45:10 lustre-3ware kernel:        00000039d0e2e2b0 
0000000000000000 0000010001021aa0 0000000000000000
Jul 30 10:45:10 lustre-3ware kernel:        000001001fa85030 
0000000000004333
Jul 30 10:45:10 lustre-3ware kernel: Call 
Trace:<ffffffff80308223>{__down+147} 
<ffffffff801331a5>{default_wake_function+0}
Jul 30 10:45:10 lustre-3ware kernel:        
<ffffffff80309cbb>{__down_failed+53} 
<ffffffffa034cccc>{:lquota:.text.lock.quota_master+15}
Jul 30 10:45:10 lustre-3ware kernel:        
<ffffffffa02964b6>{:ptlrpc:target_handle_dqacq_callback+953}
Jul 30 10:45:10 lustre-3ware kernel:        
<ffffffffa029fec8>{:ptlrpc:ldlm_callback_handler+1486}
Jul 30 10:45:10 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:45:10 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:45:10 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:45:10 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:45:10 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:45:10 lustre-3ware kernel:       
Jul 30 10:45:10 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185803110.413
Jul 30 10:46:33 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@000001004f9d0a00 x71040796/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:46:33 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) Skipped 5 previous similar 
messages
Jul 30 10:48:12 lustre-3ware kernel: LustreError: 
3537:0:(client.c:950:ptlrpc_expire_one_request()) @@@ timeout (sent at 
1185803092, 200s ago) req@0000010077da6800 x71040650/t0 
o601->@192.168.0.24@tcp:15 lens 144/144 ref 1 fl Rpc:/0/0 rc 0/-22
Jul 30 10:48:12 lustre-3ware kernel: LustreError: 
3537:0:(client.c:950:ptlrpc_expire_one_request()) Skipped 1 previous 
similar message
Jul 30 10:48:30 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 414: it was 
inactive for 18s
Jul 30 10:48:30 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 414
Jul 30 10:48:30 lustre-3ware kernel: ldlm_cb_10    D 
0000000000000001     0   414      1                 413 (L-TLB)
Jul 30 10:48:30 lustre-3ware kernel: 00000100468bfc48 0000000000000046 
0000000000000046 0000000000000073
Jul 30 10:48:30 lustre-3ware kernel:        0000000000000002 
0000000000000002 0000010001021aa0 000000006e0d6870
Jul 30 10:48:30 lustre-3ware kernel:        000001003636a800 
0000000000003ece
Jul 30 10:48:30 lustre-3ware kernel: Call 
Trace:<ffffffff80308223>{__down+147} 
<ffffffff801331a5>{default_wake_function+0}
Jul 30 10:48:30 lustre-3ware kernel:        
<ffffffff80309cbb>{__down_failed+53} 
<ffffffffa034cccc>{:lquota:.text.lock.quota_master+15}
Jul 30 10:48:30 lustre-3ware kernel:        
<ffffffffa02964b6>{:ptlrpc:target_handle_dqacq_callback+953}
Jul 30 10:48:30 lustre-3ware kernel:        
<ffffffffa029fec8>{:ptlrpc:ldlm_callback_handler+1486}
Jul 30 10:48:30 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:48:30 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:48:30 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80176ae8>{filp_close+103}
Jul 30 10:48:30 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:48:30 lustre-3ware kernel:        
<ffffffff80110e23>{child_rip+8}
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
Jul 30 10:48:30 lustre-3ware kernel:       
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:48:30 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185803310.414
Jul 30 10:48:46 lustre-3ware sshd(pam_unix)[419]: session opened for 
user root by root(uid=0)
Jul 30 10:49:02 lustre-3ware kernel: Lustre: 
3689:0:(ldlm_lib.c:497:target_handle_reconnect()) home-MDT0000: 
3579ecd4-03f1-8760-6b01-8e3313ff730d reconnecting
Jul 30 10:49:02 lustre-3ware kernel: Lustre: 
3835:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from 
3579ecd4-03f1-8760-6b01-8e3313ff730d@129.173.118.68@tcp to 
0x000001004d4e2000/9
Jul 30 10:49:02 lustre-3ware kernel: Lustre: 
3835:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 40 previous 
similar messages
Jul 30 10:49:02 lustre-3ware kernel: LustreError: 
3835:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@00000100723cf450 x158273181/t0 
o8->3579ecd4-03f1-8760-6b01-8e3313ff730d@NET_0x2000081ad7644_UUID:-1 
lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0
Jul 30 10:49:02 lustre-3ware kernel: LustreError: 
3835:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 40 previous 
similar messages
Jul 30 10:49:02 lustre-3ware kernel: Lustre: 
3689:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 41 previous 
similar messages
Jul 30 10:49:52 lustre-3ware kernel: ll_mdt_24     D 
00000100724ea4e0     0  3705      1          3706  3704 (L-TLB)
Jul 30 10:49:52 lustre-3ware kernel: 0000010072e2f9d8 0000000000000046 
0000010077d98d80 0000000000000246
Jul 30 10:49:52 lustre-3ware kernel:        0000000000000246 
ffffffffa0312a93 0002000081ad7644 0000000000003039
Jul 30 10:49:52 lustre-3ware kernel:        00000100729a8030 
000000000000036e
Jul 30 10:49:52 lustre-3ware kernel: Call 
Trace:<ffffffffa0312a93>{:ksocklnd:ksocknal_launch_packet+443}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffffa0312e56>{:ksocklnd:ksocknal_send+632} 
<ffffffff80308223>{__down+147}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffff80309cbb>{__down_failed+53}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffffa034cdda>{:lquota:.text.lock.quota_master+285}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffffa02b05a8>{:ptlrpc:lustre_msg_add_version+67}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffffa02b0dc5>{:ptlrpc:lustre_pack_reply+1928}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffffa0344758>{:lquota:mds_quota_ctl+248} 
<ffffffffa045a755>{:mds:mds_handle_quotactl+994}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffffa045e19f>{:mds:mds_handle+14732} 
<ffffffff801315df>{activate_task+124}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffff80131b1a>{try_to_wake_up+876} 
<ffffffff80134b6b>{autoremove_wake_function+9}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffff801331f6>{__wake_up_common+67} 
<ffffffffa01f640e>{:lnet:lnet_match_blocked_msg+634}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffff80132384>{move_tasks+200}
<ffffffff80131551>{recalc_task_prio+337}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffff80308f39>{thread_return+0}
<ffffffff80308f91>{thread_return+88}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:49:52 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:49:52 lustre-3ware kernel:       
Jul 30 10:49:52 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185803392.3705
Jul 30 10:51:50 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 468: it was 
inactive for 18s
Jul 30 10:51:50 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Skipped 1 previous similar message
Jul 30 10:51:50 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 468
Jul 30 10:51:50 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 1 previous 
similar message
Jul 30 10:51:50 lustre-3ware kernel: ldlm_cb_11    D 
0000000000000001     0   468      1                 414 (L-TLB)
Jul 30 10:51:50 lustre-3ware kernel: 000001004e0b9c48 0000000000000046 
0000000000000012 0000000000000073
Jul 30 10:51:50 lustre-3ware kernel:        000001002e6fba18 
0000000000000000 0000010001021aa0 0000000000000000
Jul 30 10:51:50 lustre-3ware kernel:        0000010034ec7800 
0000000000003de1
Jul 30 10:51:50 lustre-3ware kernel: Call 
Trace:<ffffffff8013324c>{__wake_up+54}
<ffffffff80308223>{__down+147}
Jul 30 10:51:50 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffff80309cbb>{__down_failed+53}
Jul 30 10:51:50 lustre-3ware kernel:        
<ffffffffa034cccc>{:lquota:.text.lock.quota_master+15}
Jul 30 10:51:50 lustre-3ware kernel:        
<ffffffffa02964b6>{:ptlrpc:target_handle_dqacq_callback+953}
Jul 30 10:51:50 lustre-3ware kernel:        
<ffffffffa029fec8>{:ptlrpc:ldlm_callback_handler+1486}
Jul 30 10:51:50 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:51:50 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:51:50 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:51:50 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:51:50 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:51:50 lustre-3ware kernel:       
Jul 30 10:51:50 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185803510.468
Jul 30 10:52:23 lustre-3ware sshd(pam_unix)[472]: session opened for 
user root by root(uid=0)
Jul 30 10:55:10 lustre-3ware kernel: ldlm_cb_12    D 
0000000000000001     0   520      1                 469 (L-TLB)
Jul 30 10:55:10 lustre-3ware kernel: 000001003d84fc48 0000000000000046 
0000000046adeda7 0000000000000073
Jul 30 10:55:10 lustre-3ware kernel:        000001007e2c2400 
00000000801a909b 0000010001021aa0 000000008018f037
Jul 30 10:55:10 lustre-3ware kernel:        000001003636a030 
000000000000463c
Jul 30 10:55:10 lustre-3ware kernel: Call 
Trace:<ffffffff8018e56d>{__d_rehash+115}
<ffffffff80308223>{__down+147}
Jul 30 10:55:10 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffff80309cbb>{__down_failed+53}
Jul 30 10:55:10 lustre-3ware kernel:        
<ffffffffa034cccc>{:lquota:.text.lock.quota_master+15}
Jul 30 10:55:10 lustre-3ware kernel:        
<ffffffffa02964b6>{:ptlrpc:target_handle_dqacq_callback+953}
Jul 30 10:55:10 lustre-3ware kernel:        
<ffffffffa029fec8>{:ptlrpc:ldlm_callback_handler+1486}
Jul 30 10:55:10 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:55:10 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:55:10 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:55:10 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:55:10 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:55:10 lustre-3ware kernel:       
Jul 30 10:55:10 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185803710.520
Jul 30 10:55:43 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) @@@ type == PTL_RPC_MSG_ERR, 
err == -16 req@00000100744bec00 x71040829/t0 
o8->home-OST0000_UUID@192.168.0.24@tcp:28 lens 304/328 ref 1 fl 
Rpc:R/0/0 rc 0/-16
Jul 30 10:55:43 lustre-3ware kernel: LustreError: 
3538:0:(client.c:574:ptlrpc_check_status()) Skipped 10 previous similar 
messages
Jul 30 10:57:26 lustre-3ware sshd(pam_unix)[528]: session opened for 
user root by root(uid=0)
Jul 30 10:57:44 lustre-3ware sshd(pam_unix)[564]: session opened for 
user root by root(uid=0)
Jul 30 10:57:55 lustre-3ware kernel: Lustre: 
20917:0:(ldlm_lib.c:663:target_handle_connect()) home-OST0000: exp 
000001006e7f5000 already connecting
Jul 30 10:58:08 lustre-3ware sshd(pam_unix)[600]: session opened for 
user root by root(uid=0)
Jul 30 10:58:45 lustre-3ware kernel: Lustre: 
23100:0:(ldlm_lib.c:663:target_handle_connect()) home-OST0000: exp 
000001006e7f5000 already connecting
Jul 30 10:58:45 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 3865: it was 
inactive for 100s
Jul 30 10:58:45 lustre-3ware kernel: Lustre: 
0:0:(watchdog.c:130:lcw_cb()) Skipped 1 previous similar message
Jul 30 10:58:45 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for 
process 3865
Jul 30 10:58:45 lustre-3ware kernel: Lustre: 
0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 1 previous 
similar message
Jul 30 10:58:45 lustre-3ware kernel: ll_ost_io_29  D 
00000100669043a8     0  3865      1          3866  3864 (L-TLB)
Jul 30 10:58:45 lustre-3ware kernel: 000001006ff5d8e8 0000000000000046 
0000000000000078 0000010072f54000
Jul 30 10:58:45 lustre-3ware kernel:        0000000000000000 
0000000000000011 00000000000008b9 00000000000008b8
Jul 30 10:58:45 lustre-3ware kernel:        000001006fe24030 
000000000000b441
Jul 30 10:58:45 lustre-3ware kernel: Call 
Trace:<ffffffffa03e408d>{:ldiskfs:ldiskfs_count_free_blocks+46}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffffa03e6411>{:ldiskfs:ldiskfs_count_free_inodes+37}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffff80308223>{__down+147}
<ffffffff801331a5>{default_wake_function+0}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffff80309cbb>{__down_failed+53} 
<ffffffffa02396b5>{:obdclass:.text.lock.lprocfs_status+95}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffffa04d74f4>{:obdfilter:filter_export_stats_init+205}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffffa04ded8e>{:obdfilter:filter_connect+422} 
<ffffffffa02937e1>{:ptlrpc:target_handle_connect+4437}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffffa04c89b5>{:ost:ost_handle+0} 
<ffffffffa04c70e4>{:ost:ost_brw_write+6072}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffffa02b1471>{:ptlrpc:lustre_msg_get_version+64}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffffa04c8ed6>{:ost:ost_handle+1313} 
<ffffffff80131b1a>{try_to_wake_up+876}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffff80134b6b>{autoremove_wake_function+9} 
<ffffffff801331f6>{__wake_up_common+67}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffffa01f64b5>{:lnet:lnet_match_blocked_msg+801}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffff80132452>{move_tasks+406} 
<ffffffffa0239a51>{:obdclass:class_handle2object+207}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffffa02b7e57>{:ptlrpc:ptlrpc_server_handle_request+2528}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffff8013f100>{__mod_timer+293} 
<ffffffffa02b9d1b>{:ptlrpc:ptlrpc_main+2018}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffff801331a5>{default_wake_function+0} 
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffffa02b897a>{:ptlrpc:ptlrpc_retry_rqbds+0} 
<ffffffff80110e23>{child_rip+8}
Jul 30 10:58:45 lustre-3ware kernel:        
<ffffffffa02b9539>{:ptlrpc:ptlrpc_main+0}
<ffffffff80110e1b>{child_rip+0}
Jul 30 10:58:45 lustre-3ware kernel:       
Jul 30 10:58:45 lustre-3ware kernel: LustreError: dumping log to 
/tmp/lustre-log.1185803925.3865
Jul 30 10:59:03 lustre-3ware kernel: Lustre: 
3806:0:(ldlm_lib.c:497:target_handle_reconnect()) home-OST0000: 
home-mdtlov_UUID reconnecting
Jul 30 10:59:03 lustre-3ware kernel: Lustre: 
3806:0:(ldlm_lib.c:497:target_handle_reconnect()) Skipped 54 previous 
similar messages
Jul 30 10:59:03 lustre-3ware kernel: Lustre: 
3806:0:(ldlm_lib.c:709:target_handle_connect()) home-OST0000: refuse 
reconnection from home-mdtlov_UUID@0@lo to 0x0000010077c4f000/2
Jul 30 10:59:03 lustre-3ware kernel: Lustre: 
3806:0:(ldlm_lib.c:709:target_handle_connect()) Skipped 55 previous 
similar messages
Jul 30 10:59:03 lustre-3ware kernel: LustreError: 
3806:0:(ldlm_lib.c:1363:target_send_reply_msg()) @@@ processing error 
(-16) req@000001007230ec50 x71040842/t0 
o8->home-mdtlov_UUID@192.168.0.24@tcp:-1 lens 304/200 ref 0 fl 
Interpret:/0/0 rc -16/0
Jul 30 10:59:03 lustre-3ware kernel: LustreError: 
3806:0:(ldlm_lib.c:1363:target_send_reply_msg()) Skipped 57 previous 
similar messages
Jul 30 10:59:03 lustre-3ware sshd(pam_unix)[640]: session opened for 
user root by root(uid=0)
Jul 30 10:59:15 lustre-3ware kernel: Lustre: Failing over home-MDT0000
Jul 30 10:59:15 lustre-3ware kernel: LustreError: 
673:0:(mds_lov.c:558:mds_iocontrol()) *** setting device 
unknown-block(8,6) read-only ***
Jul 30 10:59:15 lustre-3ware kernel: Turning device sda (0x800006) read-only
Jul 30 10:59:15 lustre-3ware kernel: LustreError: 
3682:0:(service.c:668:ptlrpc_server_handle_request()) request 158272955 
opc 48 from 12345-129.173.118.68@tcp processed in 2262s trans 0 rc -19/-19
Jul 30 10:59:15 lustre-3ware kernel: LustreError: 
3658:0:(service.c:668:ptlrpc_server_handle_request()) request 71040650 
opc 601 from 12345-0@lo processed in 2262s trans 0 rc -122/-122
Jul 30 10:59:15 lustre-3ware kernel: Lustre: 
3658:0:(watchdog.c:312:lcw_update_time()) Expired watchdog for pid 3658 
disabled after 2262.5638s
Jul 30 10:59:15 lustre-3ware kernel: Lustre: 
3658:0:(watchdog.c:312:lcw_update_time()) Skipped 4 previous similar 
messages
Jul 30 10:59:15 lustre-3ware kernel: LustreError: 
3711:0:(mds_open.c:420:mds_create_objects()) error creating objects for 
inode 10092381: rc = -5
Jul 30 10:59:15 lustre-3ware kernel: LustreError: 
3711:0:(mds_open.c:711:mds_finish_open()) mds_create_objects: rc = -5
Jul 30 10:59:15 lustre-3ware kernel: LustreError: 
3711:0:(mds_reint.c:137:mds_finish_transno()) fsfilt_start: -30
Jul 30 10:59:15 lustre-3ware kernel: LustreError: 
3707:0:(quota_ctl.c:247:lov_quota_ctl()) ost 0 is inactive
Jul 30 10:59:15 lustre-3ware kernel: LustreError: 
3707:0:(service.c:668:ptlrpc_server_handle_request()) request 158273012 
opc 48 from 12345-129.173.118.68@tcp processed in 2063s trans 0 rc -19/-19
Jul 30 10:59:15 lustre-3ware kernel: Lustre: 
3656:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from 12345-0@lo 
portal 16 match 71040650 offset 0 length 144: 2
Jul 30 10:59:15 lustre-3ware kernel: LustreError: 
3707:0:(service.c:668:ptlrpc_server_handle_request()) Skipped 3 previous 
similar messages
Jul 30 10:59:17 lustre-3ware kernel: LustreError: 
3706:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.0.8@tcp
Jul 30 10:59:17 lustre-3ware kernel: LustreError: 
3683:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010078a95600 x336994/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:21 lustre-3ware kernel: LustreError: 
3708:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@000001007dcc2c00 x1101357/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:21 lustre-3ware kernel: LustreError: 
3697:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.0.5@tcp
Jul 30 10:59:21 lustre-3ware kernel: LustreError: 
3700:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010074b9d400 x302136/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:24 lustre-3ware kernel: LustreError: 
3699:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.1.20@tcp
Jul 30 10:59:24 lustre-3ware kernel: LustreError: 
3691:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010011048000 x717102/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:27 lustre-3ware sshd(pam_unix)[676]: session opened for 
user root by root(uid=0)
Jul 30 10:59:35 lustre-3ware kernel: LustreError: 
3686:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.0.2@tcp
Jul 30 10:59:35 lustre-3ware kernel: Lustre: 
20921:0:(ldlm_lib.c:663:target_handle_connect()) home-OST0000: exp 
000001006e7f5000 already connecting
Jul 30 10:59:35 lustre-3ware kernel: LustreError: 
3703:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010078a94200 x701022/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:36 lustre-3ware kernel: LustreError: 
3689:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.0.4@tcp
Jul 30 10:59:36 lustre-3ware kernel: LustreError: 
3704:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010074209c00 x597117/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:39 lustre-3ware kernel: LustreError: 
3694:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.0.16@tcp
Jul 30 10:59:39 lustre-3ware kernel: LustreError: 
3684:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010069db9000 x318136/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:43 lustre-3ware kernel: LustreError: 
3712:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.0.19@tcp
Jul 30 10:59:43 lustre-3ware kernel: LustreError: 
3687:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@00000100185aec00 x619901/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:44 lustre-3ware kernel: LustreError: 
3702:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.1.22@tcp
Jul 30 10:59:44 lustre-3ware kernel: LustreError: 
3709:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@000001007dcd3e00 x1149062/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:50 lustre-3ware kernel: LustreError: 
3701:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.0.15@tcp
Jul 30 10:59:50 lustre-3ware kernel: LustreError: 
3685:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010078a92e00 x681258/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:51 lustre-3ware kernel: LustreError: 
3690:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.0.6@tcp
Jul 30 10:59:51 lustre-3ware kernel: LustreError: 
3693:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010069db9a00 x216681/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:52 lustre-3ware kernel: LustreError: 
3688:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.1.23@tcp
Jul 30 10:59:52 lustre-3ware kernel: LustreError: 
3682:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010072e0f450 x583058/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:54 lustre-3ware kernel: LustreError: 
3706:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.0.7@tcp
Jul 30 10:59:54 lustre-3ware kernel: LustreError: 
3706:0:(handler.c:1489:mds_handle()) Skipped 1 previous similar message
Jul 30 10:59:54 lustre-3ware kernel: LustreError: 
3683:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@000001003c5dda00 x465396/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 10:59:54 lustre-3ware kernel: LustreError: 
3683:0:(ldlm_lib.c:576:target_handle_connect()) Skipped 2 previous 
similar messages
Jul 30 10:59:59 lustre-3ware kernel: LustreError: 
3708:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.1.21@tcp
Jul 30 10:59:59 lustre-3ware kernel: LustreError: 
3697:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010078a93c00 x13192713/t0 o38-><?>@<?>:-1 lens 304/0 ref
0 fl
Interpret:/0/0 rc 0/0
Jul 30 11:00:01 lustre-3ware kernel: LustreError: 
3704:0:(handler.c:1489:mds_handle()) operation 400 on unconnected MDS 
from 12345-192.168.0.14@tcp
Jul 30 11:00:01 lustre-3ware kernel: LustreError: 
3704:0:(handler.c:1489:mds_handle()) Skipped 3 previous similar messages
Jul 30 11:00:03 lustre-3ware kernel: LustreError: 
3710:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010077dbc800 x658671/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 11:00:03 lustre-3ware kernel: LustreError: 
3710:0:(ldlm_lib.c:576:target_handle_connect()) Skipped 4 previous 
similar messages
Jul 30 11:00:14 lustre-3ware kernel: LustreError: 
3701:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010001395400 x717106/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 11:00:14 lustre-3ware kernel: LustreError: 
3701:0:(ldlm_lib.c:576:target_handle_connect()) Skipped 4 previous 
similar messages
Jul 30 11:00:28 lustre-3ware sshd(pam_unix)[724]: session opened for 
user root by root(uid=0)
Jul 30 11:00:33 lustre-3ware kernel: LustreError: 
3682:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@000001007dc62200 x619905/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 11:00:33 lustre-3ware kernel: LustreError: 
3682:0:(ldlm_lib.c:576:target_handle_connect()) Skipped 6 previous 
similar messages
Jul 30 11:01:15 lustre-3ware kernel: LustreError: 
3709:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010074209a00 x701031/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Jul 30 11:01:15 lustre-3ware kernel: LustreError: 
3709:0:(ldlm_lib.c:576:target_handle_connect()) Skipped 37 previous 
similar messages
Jul 30 11:02:20 lustre-3ware kernel: Removing read-only on sda (0x800006)
Jul 30 11:02:20 lustre-3ware kernel: LustreError: 
778:0:(obd_mount.c:92:server_register_mount()) Already registered 
home-MDT0000
Jul 30 11:02:20 lustre-3ware kernel: LustreError: 
778:0:(obd_mount.c:1557:server_fill_super()) Unable to start targets: -17
Jul 30 11:02:20 lustre-3ware kernel: LustreError: 
778:0:(obd_mount.c:1356:server_put_super()) no obd home-MDT0000
Jul 30 11:02:20 lustre-3ware kernel: Lustre: 
778:0:(obd_mount.c:1391:server_put_super()) Cleaning orphaned obd 
home-mdtlov
Jul 30 11:02:22 lustre-3ware kernel: LustreError: 
3688:0:(ldlm_lib.c:576:target_handle_connect()) @@@ UUID 
''home-MDT0000_UUID'' is not available  for connect (stopping) 
req@0000010040260200 x583073/t0 o38-><?>@<?>:-1 lens 304/0 ref 0
fl
Interpret:/0/0 rc 0/0
Lustre discuss - Jul 2007 - lustre crash

[Lustre-discuss] lustre crash
