thr3ads.net - Lustre discuss - [Lustre-discuss] Lustre Issues [Jan 2011]

If this information is useful, please help other people find it:
Share via:
Nauman Yousuf
2011-Jan-26 14:36 UTC
[Lustre-discuss] Lustre Issues

Dear All

I am getting these errors in my dmesg file. My lustre client which is my
mail storage is generating lot of system load due to which all operations
went slow and server average load is above 60% which delayed my whole email
server logs are below. can any one guide me what the prb is

primary MDS

bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex
bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex
bonding: bond0: link status up for interface eth0, enabling it in 1000 ms.
bonding: bond0: link status up for interface eth1, enabling it in 1000 ms.
bonding: bond0: link status definitely up for interface eth0.
bonding: bond0: link status definitely up for interface eth1.
bonding: bond0: making interface eth0 the new active one.
drbd: initialised. Version: 0.7.23 (api:79/proto:74)
drbd: SVN Revision: 2686 build by root at satyrs, 2007-04-24 14:43:41
drbd: registered as block device major 147
drbd0: Adjusting my ra_pages to backing device''s (32 -> 1024)
drbd0: resync bitmap: bits=158692071 words=4959128
drbd0: size = 605 GB (634768281 KB)
drbd0: 605 GB marked out-of-sync by on disk bit-map.
drbd0: Found 6 transactions (324 active extents) in activity log.
drbd0: drbdsetup [1125]: cstate Unconfigured --> StandAlone
drbd0: drbdsetup [1138]: cstate StandAlone --> Unconnected
drbd0: drbd0_receiver [1139]: cstate Unconnected --> WFConnection
eth0: no IPv6 routers present
bond0: no IPv6 routers present
eth1: no IPv6 routers present
drbd0: Secondary/Unknown --> Primary/Unknown
Lustre: 1502:0:(module.c:382:init_libcfs_module()) maximum lustre stack 8192
Lustre: OBD class driver Build Version:
1.4.9.1-19691231170000-PRISTINE-.testsuite.tmp.lbuild-boulder.lbuild-v1_4_9_1-2.6-rhel4-i686.lbuild.BUILD.lustre-kerne
l-2.6.9.lustre.linux-2.6.9-42.0.8.EL_lustre.1.4.9.1smp, info at clusterfs.com
Lustre: Added LNI 10.65.200.6 at tcp [8/256]
Lustre: Accept secure, port 988
kjournald starting.  Commit interval 5 seconds
LDISKFS-fs warning: maximal mount count reached, running e2fsck is
recommended
LDISKFS FS on drbd0, internal journal
LDISKFS-fs: mounted filesystem with ordered data mode.
Lustre: 1679:0:(llog_obd.c:160:cat_cancel_cb()) processing log
0x8ab8005:4732d8ff at index 356 of catalog 0x8ab8002
Lustre: 1679:0:(llog_obd.c:160:cat_cancel_cb()) processing log
0x8ab8004:472cceab at index 345 of catalog 0x8ab8003
Lustre: 1679:0:(llog_obd.c:160:cat_cancel_cb()) processing log
0x8ab8006:472ed7dc at index 346 of catalog 0x8ab8003
Lustre: 1679:0:(llog_obd.c:160:cat_cancel_cb()) processing log
0x8ab8008:47468c57 at index 358 of catalog 0x8ab8003
Lustre: MDT mds01 now serving /dev/drbd0
(f74ccccb-80c7-443c-be33-1c2ec53f4b0c) with recovery enabled
Lustre: MDS mds01: All OSCs now active, resetting orphans
drbd0: drbd0_receiver [1139]: cstate WFConnection --> WFReportParams
drbd0: Handshake successful: DRBD Network Protocol version 74
drbd0: Connection established.
drbd0: I am(P): 1:00000004:00000001:00000041:00000002:10
drbd0: Peer(S): 0:00000003:00000001:00000040:00000002:00
drbd0: drbd0_receiver [1139]: cstate WFReportParams --> WFBitMapS
drbd0: Primary/Unknown --> Primary/Secondary
drbd0: drbd0_receiver [1139]: cstate WFBitMapS --> SyncSource
drbd0: Resync started as SyncSource (need to sync 634747844 KB [158686961
bits set]).
drbd0: Resync done (total 97313 sec; paused 0 sec; 6520 K/sec)
drbd0: drbd0_worker [1126]: cstate SyncSource --> Connected
Lustre: 1588:0:(lustre_fsfilt.h:283:fsfilt_setattr()) mds01: slow setattr
31s
Lustre: 1595:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 33s
Lustre: 1720:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 32s
Lustre: 1602:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 38s
Lustre: 1602:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous
similar messages
Lustre: 1656:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 34s
Lustre: 1741:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 34s
Lustre: 1601:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s
Lustre: 1667:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s

Lustre: 1601:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1737:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 37s
Lustre: 1737:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1604:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 34s
Lustre: 1604:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous
similar messages
Lustre: 1719:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 31s
Lustre: 1719:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 5 previous
similar messages
Lustre: 1716:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 31s
Lustre: 1716:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous
similar messages
Lustre: 1592:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 31s
Lustre: 1592:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous
similar messages
Lustre: 1643:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 41s
Lustre: 1643:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1741:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 40s
Lustre: 1741:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous
similar messages
Lustre: 1736:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 32s
Lustre: 1736:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 6 previous
similar messages
Lustre: 1593:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 32s
Lustre: 1593:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous
similar messages
Lustre: 1651:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 31s
Lustre: 1588:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s
Lustre: 1740:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s
Lustre: 1740:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous
similar messages
Lustre: 1671:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 39s
Lustre: 1607:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 39s
Lustre: 1607:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous
similar messages
Lustre: 1743:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 36s
Lustre: 1743:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 6 previous
similar messages
Lustre: 1581:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 32s
Lustre: 1670:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 32s
Lustre: 1735:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 32s
LustreError: 1593:0:(client.c:940:ptlrpc_expire_one_request()) @@@ timeout
(sent at 1276502817, 6s ago)  req at c229d200 x1425932/t0
o104->@NET_0x200000a41c82d_U
UID:15 lens 176/64 ref 1 fl Rpc:/0/0 rc 0/0
LustreError: A client on nid 10.65.200.45 at tcp was evicted from service
mds01.
LustreError: 1593:0:(ldlm_lockd.c:427:ldlm_failed_ast()) ### blocking AST
failed (-110): evicting client
b59fa_MNT_client_5e85c4f00a at NET_0x200000a41c82d_UUID
NID 10.65.200.45 at tcp (10.65.200.45 at tcp) ns: mds-mds01_UUID lock:
e24e6440/0x7bfc427d0b5b000e lrc: 2/0,0 mode: CR/CR res: 99848313/2526868437
bits 0x3 rrc: 4 t
ype: IBT flags: 20 remote: 0x54fc0cf21abf3df8 expref: 4296 pid 1586
Lustre: 7:0:(linux-debug.c:98:libcfs_run_upcall()) Invoked LNET upcall
/usr/lib/lustre/lnet_upcall ROUTER_NOTIFY,10.65.200.45 at tcp,down,1276502808
drbd0: sock was shut down by peer
drbd0: drbd0_receiver [1139]: cstate Connected --> BrokenPipe
drbd0: short read expecting header on sock: r=0
drbd0: meta connection shut down by peer.
drbd0: asender terminated
drbd0: worker terminated
drbd0: drbd0_receiver [1139]: cstate BrokenPipe --> Unconnected
drbd0: Connection lost.

drbd0: drbd0_receiver [1139]: cstate Unconnected --> WFConnection
drbd0: drbd0_receiver [1139]: cstate WFConnection --> WFReportParams
drbd0: Handshake successful: DRBD Network Protocol version 74
drbd0: Connection established.
drbd0: I am(P): 1:00000004:00000001:00000042:00000002:10
drbd0: Peer(S): 1:00000004:00000001:00000041:00000002:00
drbd0: drbd0_receiver [1139]: cstate WFReportParams --> WFBitMapS
drbd0: Primary/Unknown --> Primary/Secondary
drbd0: drbd0_receiver [1139]: cstate WFBitMapS --> SyncSource
drbd0: Resync started as SyncSource (need to sync 33752 KB [8438 bits set]).
drbd0: Resync done (total 9 sec; paused 0 sec; 3748 K/sec)
drbd0: drbd0_worker [15901]: cstate SyncSource --> Connected
bnx2: eth0 NIC Link is Down
bonding: bond0: link status down for idle  interface eth0, disabling it in
1000 ms.
bonding: bond0: link status definitely down for interface eth0, disabling it
bonding: bond0: making interface eth1 the new active one.
drbd0: PingAck did not arrive in time.
drbd0: drbd0_asender [15902]: cstate Connected --> NetworkFailure
drbd0: asender terminated
drbd0: drbd0_receiver [1139]: cstate NetworkFailure --> BrokenPipe
drbd0: short read expecting header on sock: r=-512
drbd0: worker terminated
drbd0: drbd0_receiver [1139]: cstate BrokenPipe --> Unconnected
drbd0: Connection lost.
drbd0: drbd0_receiver [1139]: cstate Unconnected --> WFConnection
bnx2: eth1 NIC Link is Down
bonding: bond0: link status down for idle  interface eth1, disabling it in
1000 ms.
bonding: bond0: link status definitely down for interface eth1, disabling it
bonding: bond0: now running without any active interface !
LustreError: OSC_satyrs_ost1_mds01: Connection to service ost1 via nid
10.65.200.21 at tcp was lost; in progress operations using this service will
wait for reco
very to complete.
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message

Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c22afe00 x8555234/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at f7cd3800 x8555238/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c2259c00 x8555242/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at e2a7ca00 x8555246/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 1 previous similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 1 previous similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at f7ccda00 x8555250/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 3 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at e1f40800 x8555254/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 3 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at d150d800 x8555258/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c22ae200 x8555262/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp

Lustre: Skipped 7 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at e18ad200 x8555266/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c2264400 x8555270/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c22b2600 x8555274/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c228b200 x8555278/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 15 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at e18aca00 x8555282/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c22b2200 x8555286/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c2289600 x8555290/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at e18a7400 x8555294/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c229c200 x8555298/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at f7ce4600 x8555302/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at e1f4ea00 x8555306/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 1 previous
similar message
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 27 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c22b2800 x8555314/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 3 previous
similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c22a1c00 x8555330/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 7 previous
similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp

Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c22b2e00 x8555358/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 13 previous
similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c228ee00 x8555394/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous
similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c228e400 x8555430/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous
similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at c228ee00 x8555466/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous
similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 31 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at f7cce600 x8555498/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 15 previous
similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 31 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at e18bf800 x8555534/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous
similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at f7ce4600 x8555570/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous
similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to cyclops_UUID/
10.65.200.30 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at e18ada00 x8555606/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
pc:/0/0 rc 0/0
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 17 previous
similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 33 previous similar messages
LustreError: 1780:0:(events.c:53:request_out_callback()) @@@ type 4, status
-5  req at e182ae00 x8555638/t0 o8->ost1_UUID at cyclops_UUID:28 lens 240/272
ref
2 fl R
LustreError: 1780:0:(events.c:53:request_out_callback()) Skipped 15 previous
similar messages
Lustre: Changing connection for OSC_satyrs_ost1_mds01 to typhoon_UUID/
10.65.200.21 at tcp
Lustre: Skipped 31 previous similar messages
bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex
bonding: bond0: link status up for interface eth1, enabling it in 1000 ms.
bonding: bond0: link status definitely up for interface eth1.
bonding: bond0: making interface eth1 the new active one.
bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex
bonding: bond0: link status up for interface eth0, enabling it in 1000 ms.
bonding: bond0: link status definitely up for interface eth0.
drbd0: drbd0_receiver [1139]: cstate WFConnection --> WFReportParams
drbd0: Handshake successful: DRBD Network Protocol version 74
drbd0: Connection established.
drbd0: I am(P): 1:00000004:00000001:00000043:00000002:10
drbd0: Peer(S): 1:00000004:00000001:00000042:00000002:01
drbd0: drbd0_receiver [1139]: cstate WFReportParams --> WFBitMapS
drbd0: Primary/Unknown --> Primary/Secondary
drbd0: drbd0_receiver [1139]: cstate WFBitMapS --> SyncSource
drbd0: Resync started as SyncSource (need to sync 12 KB [3 bits set]).
drbd0: Resync done (total 1 sec; paused 0 sec; 12 K/sec)
drbd0: drbd0_worker [28899]: cstate SyncSource --> Connected
Lustre: 1606:0:(ldlm_lib.c:486:target_handle_reconnect()) mds01:
92e82_MNT_client_8029478b2f reconnecting
Lustre: OSC_satyrs_ost1_mds01: Connection restored to service ost1 using nid
10.65.200.21 at tcp.
Lustre: MDS mds01: ost1_UUID now active, resetting orphans
Lustre: mds01: haven''t heard from 10.65.200.37 at tcp in 8538 seconds.
Last
request was at 1280642838. I think it''s dead, and I am evicting it.
LustreError: 1643:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99713500: cookie 0x7bfc427e3595552c  req at e1f4e400 x456756486/t0
o35->252fb_
MNT_client_3715c9a62f at NET_0x200000a41c825_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1643:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at e1f4e400 x456756486/t0
o35->252fb_MNT_client_3715c9a62f at NET_0x2
00000a41c825_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 30s
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 7 previous
similar messages
Lustre: 1650:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 30s
Lustre: 1650:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 9 previous
similar messages
Lustre: 1734:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 35s
Lustre: 1652:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 37s
Lustre: 1652:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1665:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 37s
Lustre: 1665:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1655:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 33s
Lustre: 1730:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 32s
Lustre: 1730:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 4 previous
similar messages
Lustre: 1590:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s
Lustre: 1590:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous
similar messages
Lustre: 1645:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s
Lustre: 1733:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s

LustreError: 1725:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 66s
LustreError: 1648:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 64s
LustreError: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 58s
LustreError: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2
previous similar messages
Lustre: 1724:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 40s
Lustre: 1724:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 9 previous
similar messages
LustreError: 1658:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 66s
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 36s
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1653:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 36s
Lustre: 1593:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 37s
Lustre: 1593:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous
similar messages
Lustre: 1662:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 30s
Lustre: 1662:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1657:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 30s
Lustre: 1743:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 30s
Lustre: 1743:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1657:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1598:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s
Lustre: 1598:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1671:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s
Lustre: 1744:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s
Lustre: 1648:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 30s
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 30s
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous
similar messages
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 47s
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1718:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 47s
Lustre: 1718:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2 previous
similar messages
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 40s
Lustre: 1610:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous
similar messages
Lustre: 1729:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 38s
Lustre: 1729:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
LustreError: 1665:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 68s
LustreError: 1665:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1
previous similar message
LustreError: 1731:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 63s
LustreError: 1731:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 2
previous similar messages
LustreError: 1738:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 52s
Lustre: 1724:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s
LustreError: 1727:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 70s
LustreError: 1595:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 70s
LustreError: 1595:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1
previous similar message
Lustre: 1591:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 45s
Lustre: 1591:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous
similar messages
Lustre: 1718:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 31s
Lustre: 1718:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous
similar messages

LustreError: 1578:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 59s
LustreError: 1578:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1
previous similar message
LustreError: 1724:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 58s
LustreError: 1724:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1
previous similar message
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 48s
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 4 previous
similar messages
LustreError: 1606:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 58s
LustreError: 1606:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3
previous similar messages
LustreError: 1670:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 58s
LustreError: 1742:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 58s
LustreError: 1742:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 5
previous similar messages
Lustre: 1727:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 44s
Lustre: 1661:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 30s
Lustre: 1661:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous
similar messages
Lustre: 1594:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s
Lustre: 1594:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1659:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 46s
Lustre: 1717:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 42s
Lustre: 1717:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1602:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 49s
Lustre: 1602:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
LustreError: 1671:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow
journal start 65s
LustreError: 1671:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1
previous similar message
Lustre: 1727:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 49s
Lustre: 1727:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 4 previous
similar messages
Lustre: 1659:0:(lustre_fsfilt.h:283:fsfilt_setattr()) mds01: slow setattr
31s
LustreError: 1735:0:(llog_server.c:433:llog_origin_handle_cancel()) cancel
125 llog-records failed: -22
Lustre: 1582:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 33s
Lustre: 1582:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 4 previous
similar messages
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 32s
Lustre: 1714:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 8 previous
similar messages
Lustre: 1658:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 45s
Lustre: 1658:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 1 previous
similar message
Lustre: 1729:0:(lustre_fsfilt.h:182:fsfilt_start_log()) mds01: slow journal
start 43s
Lustre: 1729:0:(lustre_fsfilt.h:182:fsfilt_start_log()) Skipped 3 previous
similar messages
LustreError: 1595:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x5f4b399 sub-object on OST idx 0/1: rc = -28
LustreError: 1582:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8747f6d sub-object on OST idx 0/1: rc = -28
LustreError: 1609:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x3fa8002 sub-object on OST idx 0/1: rc = -28
LustreError: 1609:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87504c3 sub-object on OST idx 0/1: rc = -28
LustreError: 1589:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8750510 sub-object on OST idx 0/1: rc = -28
LustreError: 1610:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x5f7cf48 sub-object on OST idx 0/1: rc = -28
LustreError: 1588:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8750c4d sub-object on OST idx 0/1: rc = -28
LustreError: 1589:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x5f38fae sub-object on OST idx 0/1: rc = -28
LustreError: 1589:0:(lov_request.c:666:lov_update_create_set()) Skipped 1
previous similar message
LustreError: 1591:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x5f38fae sub-object on OST idx 0/1: rc = -28
LustreError: 1607:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87516a7 sub-object on OST idx 0/1: rc = -28
LustreError: 1595:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x5230001 sub-object on OST idx 0/1: rc = -28
LustreError: 1580:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x875276c sub-object on OST idx 0/1: rc = -28
LustreError: 1578:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8752eb5 sub-object on OST idx 0/1: rc = -28
LustreError: 1607:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8752ef5 sub-object on OST idx 0/1: rc = -28
LustreError: 1592:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x5f2b2fd sub-object on OST idx 0/1: rc = -28
LustreError: 1578:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8753772 sub-object on OST idx 0/1: rc = -28
LustreError: 1595:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8753844 sub-object on OST idx 0/1: rc = -28
LustreError: 1588:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x875389c sub-object on OST idx 0/1: rc = -28
LustreError: 1587:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8753cca sub-object on OST idx 0/1: rc = -28
LustreError: 1583:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87548ab sub-object on OST idx 0/1: rc = -28
LustreError: 1600:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x875499b sub-object on OST idx 0/1: rc = -28
LustreError: 1582:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8754aba sub-object on OST idx 0/1: rc = -28
LustreError: 1591:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8754b5a sub-object on OST idx 0/1: rc = -28
LustreError: 1603:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8754bd9 sub-object on OST idx 0/1: rc = -28
LustreError: 1604:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8755bd1 sub-object on OST idx 0/1: rc = -28
LustreError: 1600:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8755d06 sub-object on OST idx 0/1: rc = -28
LustreError: 1580:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8755e91 sub-object on OST idx 0/1: rc = -28
LustreError: 1580:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8756425 sub-object on OST idx 0/1: rc = -28
LustreError: 1608:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8756995 sub-object on OST idx 0/1: rc = -28
LustreError: 1590:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8756a0a sub-object on OST idx 0/1: rc = -28
LustreError: 1599:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8756c09 sub-object on OST idx 0/1: rc = -28
LustreError: 1578:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8756c49 sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8756c69 sub-object on OST idx 0/1: rc = -28
LustreError: 1600:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87571ab sub-object on OST idx 0/1: rc = -28
LustreError: 1583:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87700ab sub-object on OST idx 0/1: rc = -28
LustreError: 1583:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87702b5 sub-object on OST idx 0/1: rc = -28
LustreError: 1609:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8770330 sub-object on OST idx 0/1: rc = -28
LustreError: 1578:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x5f186b9 sub-object on OST idx 0/1: rc = -28
LustreError: 1608:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x877034c sub-object on OST idx 0/1: rc = -28
LustreError: 1585:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87703c5 sub-object on OST idx 0/1: rc = -28
LustreError: 1585:0:(lov_request.c:666:lov_update_create_set()) Skipped 1
previous similar message
LustreError: 1610:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8770d69 sub-object on OST idx 0/1: rc = -28
LustreError: 1599:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x5f99c50 sub-object on OST idx 0/1: rc = -28
LustreError: 1609:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8771c20 sub-object on OST idx 0/1: rc = -28
LustreError: 1606:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8771ded sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8742c64 sub-object on OST idx 0/1: rc = -28
LustreError: 1592:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8754719 sub-object on OST idx 0/1: rc = -28
LustreError: 1588:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87566ef sub-object on OST idx 0/1: rc = -28
LustreError: 1583:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8773897 sub-object on OST idx 0/1: rc = -28
LustreError: 1578:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8773d33 sub-object on OST idx 0/1: rc = -28
LustreError: 1593:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87740ca sub-object on OST idx 0/1: rc = -28
LustreError: 1590:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8774d26 sub-object on OST idx 0/1: rc = -28
LustreError: 1588:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87754e0 sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x877559f sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x877575d sub-object on OST idx 0/1: rc = -28
LustreError: 1609:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87757b0 sub-object on OST idx 0/1: rc = -28

LustreError: 1594:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x877580e sub-object on OST idx 0/1: rc = -28
LustreError: 1587:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x877631e sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8776b90 sub-object on OST idx 0/1: rc = -28
LustreError: 1582:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8776e7a sub-object on OST idx 0/1: rc = -28
LustreError: 1583:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8777bfe sub-object on OST idx 0/1: rc = -28
LustreError: 1596:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x8777cbc sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87b0396 sub-object on OST idx 0/1: rc = -28
LustreError: 1590:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87b0473 sub-object on OST idx 0/1: rc = -28
LustreError: 1592:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87b0820 sub-object on OST idx 0/1: rc = -28
LustreError: 1586:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87b095e sub-object on OST idx 0/1: rc = -28
LustreError: 1604:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87b1326 sub-object on OST idx 0/1: rc = -28
LustreError: 1596:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87b1346 sub-object on OST idx 0/1: rc = -28
LustreError: 1610:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87b216a sub-object on OST idx 0/1: rc = -28
LustreError: 1589:0:(lov_request.c:666:lov_update_create_set()) error
creating fid 0x87b2362 sub-object on OST idx 0/1: rc = -28
bnx2: eth0 NIC Link is Down
bonding: bond0: link status down for idle  interface eth0, disabling it in
1000 ms.
bonding: bond0: link status definitely down for interface eth0, disabling it
drbd0: PingAck did not arrive in time.
drbd0: drbd0_asender [28919]: cstate Connected --> NetworkFailure
drbd0: asender terminated
drbd0: drbd0_receiver [1139]: cstate NetworkFailure --> BrokenPipe
drbd0: short read expecting header on sock: r=-512
drbd0: worker terminated
drbd0: drbd0_receiver [1139]: cstate BrokenPipe --> Unconnected
drbd0: Connection lost.
drbd0: drbd0_receiver [1139]: cstate Unconnected --> WFConnection
bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex
bonding: bond0: link status up for interface eth0, enabling it in 1000 ms.
bonding: bond0: link status definitely up for interface eth0.
drbd0: drbd0_receiver [1139]: cstate WFConnection --> WFReportParams
drbd0: Handshake successful: DRBD Network Protocol version 74
drbd0: Connection established.
drbd0: I am(P): 1:00000004:00000001:00000044:00000002:10
drbd0: Peer(S): 1:00000004:00000001:00000043:00000002:01
drbd0: drbd0_receiver [1139]: cstate WFReportParams --> WFBitMapS
drbd0: Primary/Unknown --> Primary/Secondary
drbd0: drbd0_receiver [1139]: cstate WFBitMapS --> SyncSource
drbd0: Resync started as SyncSource (need to sync 936 KB [234 bits set]).
drbd0: Resync done (total 1 sec; paused 0 sec; 936 K/sec)
drbd0: drbd0_worker [6997]: cstate SyncSource --> Connected
Lustre: mds01: haven''t heard from 10.65.200.45 at tcp in 253 seconds.
Last
request was at 1294236586. I think it''s dead, and I am evicting it.
LustreError: 1665:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 100242438: cookie 0x7bfc4281d33d729e  req at e18acc00 x936095341/t0
o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1665:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar
message
LustreError: 1665:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at e18acc00 x936095341/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1665:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
LustreError: 1650:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99550273: cookie 0x7bfc4281d33d778a  req at c2289e00 x936095393/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1650:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar
message
LustreError: 1650:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at c2289e00 x936095393/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1650:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
LustreError: 1658:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99714850: cookie 0x7bfc4281d33d7b57  req at c221842c x936095417/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1658:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar
message
LustreError: 1658:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at c221842c x936095417/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1658:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
LustreError: 1661:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 100139209: cookie 0x7bfc4281d33d7bff  req at e1f4ec00 x936095444/t0
o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1661:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar
message
LustreError: 1661:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at e1f4ec00 x936095444/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1661:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
LustreError: 1646:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 100143301: cookie 0x7bfc4281d33d7c14  req at e1832200 x936095446/t0
o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1646:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar
message
LustreError: 1646:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at e1832200 x936095446/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1646:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
LustreError: 1646:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 100238688: cookie 0x7bfc4281d33d7e05  req at f7cd5200 x936095461/t0
o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1646:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at f7cd5200 x936095461/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1671:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 100336631: cookie 0x7bfc4281d33d6e29  req at f7cc8400 x936095478/t0
o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1671:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar
message
LustreError: 1671:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at f7cc8400 x936095478/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1671:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
LustreError: 1663:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99781569: cookie 0x7bfc4281d33d7cb5  req at e18bb800 x936095491/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1663:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar
message
LustreError: 1663:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at e18bb800 x936095491/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1663:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
LustreError: 1646:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99793390: cookie 0x7bfc4281d33d7cca  req at e18b5400 x936095499/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1646:0:(mds_open.c:1455:mds_close()) Skipped 3 previous similar
messages
LustreError: 1646:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at e18b5400 x936095499/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0

LustreError: 1657:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 100336520: cookie 0x7bfc4281d33d7a07  req at e1832200 x936095531/t0
o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1657:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at e1832200 x936095531/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1662:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99784193: cookie 0x7bfc4281d33d7584  req at c2296e00 x936095549/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1662:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar
message
LustreError: 1662:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at c2296e00 x936095549/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1662:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
LustreError: 1662:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99946358: cookie 0x7bfc4281d33d6f79  req at e18bb200 x936095566/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1662:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar
message
LustreError: 1662:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at e18bb200 x936095566/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1662:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
LustreError: 1649:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99782610: cookie 0x7bfc4281d33d7036  req at c22af600 x936095603/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1649:0:(mds_open.c:1455:mds_close()) Skipped 3 previous similar
messages
LustreError: 1649:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at c22af600 x936095603/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1649:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 3
previous similar messages
LustreError: 1674:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99586302: cookie 0x7bfc4281d33d7426  req at c2289e00 x936095620/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1674:0:(mds_open.c:1455:mds_close()) Skipped 1 previous similar
message
LustreError: 1674:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at c2289e00 x936095620/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1674:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
LustreError: 1643:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99883919: cookie 0x7bfc4281d33d8552  req at c2264c00 x936095638/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1643:0:(mds_open.c:1455:mds_close()) Skipped 2 previous similar
messages
LustreError: 1643:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at c2264c00 x936095638/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1643:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 2
previous similar messages
LustreError: 1669:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99850165: cookie 0x7bfc4281d33d6bdd  req at c2264e00 x936095798/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1669:0:(mds_open.c:1455:mds_close()) Skipped 4 previous similar
messages
LustreError: 1669:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at c2264e00 x936095798/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1669:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 4
previous similar messages
LustreError: 1664:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 100208138: cookie 0x7bfc4281d16f1663  req at e18a7800 x936098100/t0
o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1664:0:(mds_open.c:1455:mds_close()) Skipped 3 previous similar
messages
LustreError: 1664:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at e18a7800 x936098100/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1664:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 3
previous similar messages

LustreError: 1645:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 100340007: cookie 0x7bfc4281d17d91ed  req at c2259200 x936107868/t0
o35->92e82
_MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1645:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at c2259200 x936107868/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1674:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99550113: cookie 0x7bfc4281d16a1a87  req at e1f4ee00 x936108706/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1674:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at e1f4ee00 x936108706/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1538:0:(acceptor.c:422:lnet_acceptor()) Refusing connection
from 127.0.0.1: insecure port 37757
LustreError: 1657:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99794102: cookie 0x7bfc4281d0b4b154  req at f7d42200 x941376814/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1657:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at f7d42200 x941376814/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0
LustreError: 1658:0:(mds_open.c:1455:mds_close()) @@@ no handle for file
close ino 99713568: cookie 0x7bfc4281d15a5658  req at f7ce4c00 x943537900/t0
o35->92e82_
MNT_client_8029478b2f at NET_0x200000a41c82d_UUID:-1 lens 240/392 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1658:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-116)  req at f7ce4c00 x943537900/t0
o35->92e82_MNT_client_8029478b2f at NET_0x2
00000a41c82d_UUID:-1 lens 240/392 ref 0 fl Interpret:/0/0 rc -116/0




Primary OSS

LustreError: dumping log to
/tmp/lustre-log-typhoon.exampe.com.1296015719.1704
Lustre: 0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 1702: it
was inactive for 100000ms
Lustre: 0:0:(watchdog.c:130:lcw_cb()) Skipped 1 previous similar message
Lustre: 0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for
process 1702
Lustre: 0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 1 previous
similar message
ll_ost_io_30  S 00000001  4156  1702      1          1703  1701 (L-TLB)
dbbf5c10 00000046 00000000 00000001 ffffffff ffffffff 00000000 00000000
       00000002 00000000 d42bf7c0 c201ade0 00000000 00000000 d5e3f580
0056cb5d
       c0331a80 dbbd6cf0 dbbd6e5c 00000000 00000246 b02ca346 b02ca346
ffffffff
Call Trace:
 [<c02dfaa9>] schedule_timeout+0x139/0x154
 [<c01296a4>] process_timeout+0x0/0x5
 [<c011f2f9>]ll_ost_io_03  S 00000001  4028  1675      1          1676 
1674
(L-TLB)
 add_wait_queue+0x12/0x30
 [<f930fb61>] ldlm_completion_ast+0x4bd/0x993 [ptlrpc]
 [<f930f045>] ldlm_process_extent_lock+0x4df/0x63b [ptlrpc]
 [<c011d6f8>]de2cdc10 00000046 00000000 00000001 ffffffff ffffffff
00000000
default_wake_function+0x0/0xc
 [<f92f815f>] l_unlock+0xab/0xc4 [ptlrpc]
 [<f930f42d>] ldlm_expired_completion_wait+0x0/0x277 [ptlrpc]
 [<f930f42c>] interrupted_completion_wait+0x0/0x1 [ptlrpc]
 [<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
 [<f931091c>] ldlm_cli_enqueue_local+0x4b4/0x5ce [ptlrpc]
 [<f9c9fcb6>] filter_prepare_destroy+0x11a/0x1c7 [obdfilter]
 [<f9310037>] ldlm_blocking_ast+0x0/0x42b [ptlrpc]
 [<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
 [<f9cac988>] filter_destroy+0x36d/0x184a [obdfilter]
 [<f8c2d62f>] lnet_ni_send+0x70/0x88 [lnet]
 [<f8c31c93>] LNetPut+0x838/0x8ea [lnet]
 [<f93a0704>] obd_destroy+0x3ef/0x484 [ost]
 [<f93a0280>] ost_destroy+0x236/0x2cb [ost]
 [<f93accaf>] ost_handle+0xfeb/0x383f [ost]
 [<f9339841>] ptlrpc_update_export_timer+0x233/0x454 [ptlrpc]
 [<f933a4d4>] ptlrpc_server_handle_request+0xa72/0x1204 [ptlrpc]
 [<f933bb8a>] ptlrpc_main+0x827/0x9e9 [ptlrpc]
 [<c011d6f8>] default_wake_function+0x0/0xc
 [<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
 [<c02e123e>]00000000
       00000002 00000000 ee7deac0 c2022de0 00000001 00000000 d5f337c0
0056cb5d
       c22710b0 de2c36b0 de2c381c 00000000 00000246 b02ca347 b02ca347
ffffffff
Call Trace:
 [<c02dfaa9>] ret_from_fork+0x6/0x14
 [<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
 [<f933b363>] ptlrpc_main+0x0/0x9e9 [ptlrpc]
 [<c01041f5>] kernel_thread_helper+0x5/0xb
LustreError: 1702:0:(ldlm_request.c:59:ldlm_expired_completion_wait()) ###
lock timed out (enqueued at 1296015621, 100s ago); not entering recovery in
server code, just going back to sleep ns: filter-ost1_UUID lock:
d42bf7c0/0x64a0224008967602 lrc: 3/0,1 mode: --/PW res: 28303634/0 rrc: 2
type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags:
80004000 remote: 0x0 expref: -99 pid: 1702
LustreError: 1702:0:(ldlm_request.c:59:ldlm_expired_completion_wait())
Skipped 1 previous similar message
 schedule_timeout+0x139/0x154
 [<c01296a4>] process_timeout+0x0/0x5
 [<c011f2f9>] add_wait_queue+0x12/0x30
 [<f930fb61>] ldlm_completion_ast+0x4bd/0x993 [ptlrpc]
 [<f930f045>] ldlm_process_extent_lock+0x4df/0x63b [ptlrpc]
 [<c011d6f8>] default_wake_function+0x0/0xc
 [<f92f815f>] l_unlock+0xab/0xc4 [ptlrpc]
 [<f930f42d>] ldlm_expired_completion_wait+0x0/0x277 [ptlrpc]
 [<f930f42c>] interrupted_completion_wait+0x0/0x1 [ptlrpc]
 [<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
 [<f931091c>] ldlm_cli_enqueue_local+0x4b4/0x5ce [ptlrpc]
 [<f9c9fcb6>] filter_prepare_destroy+0x11a/0x1c7 [obdfilter]
 [<f9310037>] ldlm_blocking_ast+0x0/0x42b [ptlrpc]
 [<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
 [<f9cac988>] filter_destroy+0x36d/0x184a [obdfilter]
 [<f8c2d62f>] lnet_ni_send+0x70/0x88 [lnet]
 [<f8c31c93>] LNetPut+0x838/0x8ea [lnet]
 [<f93a0704>] obd_destroy+0x3ef/0x484 [ost]
 [<f93a0280>] ost_destroy+0x236/0x2cb [ost]
 [<f93accaf>] ost_handle+0xfeb/0x383f [ost]
 [<f9339841>] ptlrpc_update_export_timer+0x233/0x454 [ptlrpc]
 [<f933a4d4>] ptlrpc_server_handle_request+0xa72/0x1204 [ptlrpc]
 [<f933bb8a>] ptlrpc_main+0x827/0x9e9 [ptlrpc]
 [<c011d6f8>] default_wake_function+0x0/0xc
 [<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
 [<c02e123e>]<1>LustreError: dumping log to
/tmp/lustre-log-typhoon.exampe.com.1296015721.1702
 ret_from_fork+0x6/0x14
 [<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
ll_ost_io_11  S 00000001  4196  1683      1          1684  1682 (L-TLB)
de317c10 00000046 00000000 00000001 ffffffff ffffffff 00000000 00000000
       00000002 00000000 dead5d40 c201ade0 00000000 00000000 d64ec540
0056cb5d
       c0331a80 de2f6670 de2f67dc 00000000 00000246 b02ca34d b02ca34d
ffffffff
Call Trace:
 [<c02dfaa9>] [<f933b363>] ptlrpc_main+0x0/0x9e9 [ptlrpc]
 [<c01041f5>] kernel_thread_helper+0x5/0xb
 schedule_timeout+0x139/0x154
 [<c01296a4>] process_timeout+0x0/0x5
 [<c011f2f9>] add_wait_queue+0x12/0x30
 [<f930fb61>] ldlm_completion_ast+0x4bd/0x993 [ptlrpc]
 [<f930f045>] ldlm_process_extent_lock+0x4df/0x63b [ptlrpc]
 [<c011d6f8>] default_wake_function+0x0/0xc
 [<f92f815f>] l_unlock+0xab/0xc4 [ptlrpc]
 [<f930f42d>] ldlm_expired_completion_wait+0x0/0x277 [ptlrpc]
 [<f930f42c>] interrupted_completion_wait+0x0/0x1 [ptlrpc]
 [<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
 [<f931091c>] ldlm_cli_enqueue_local+0x4b4/0x5ce [ptlrpc]
 [<f9c9fcb6>] filter_prepare_destroy+0x11a/0x1c7 [obdfilter]
 [<f9310037>] ldlm_blocking_ast+0x0/0x42b [ptlrpc]
 [<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
 [<f9cac988>] filter_destroy+0x36d/0x184a [obdfilter]
 [<f8c2d62f>] lnet_ni_send+0x70/0x88 [lnet]
 [<f8c31c93>] LNetPut+0x838/0x8ea [lnet]
 [<f93a0704>] obd_destroy+0x3ef/0x484 [ost]
 [<f93a0280>] ost_destroy+0x236/0x2cb [ost]
 [<f93accaf>] ost_handle+0xfeb/0x383f [ost]
 [<f9339841>] ptlrpc_update_export_timer+0x233/0x454 [ptlrpc]
 [<f933a4d4>] ptlrpc_server_handle_request+0xa72/0x1204 [ptlrpc]
 [<f933bb8a>] ptlrpc_main+0x827/0x9e9 [ptlrpc]
 [<c011d6f8>] default_wake_function+0x0/0xc
 [<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
 [<c02e123e>] ret_from_fork+0x6/0x14
 [<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
 [<f933b363>] ptlrpc_main+0x0/0x9e9 [ptlrpc]
 [<c01041f5>] kernel_thread_helper+0x5/0xb
LustreError: dumping log to
/tmp/lustre-log-typhoon.exampe.com.1296015721.1683
LustreError: dumping log to
/tmp/lustre-log-typhoon.exampe.com.1296015721.1675
LustreError: 1655:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-16)  req at e9cb2600 x226166541/t0
o8->c2914_lov1_153086c94b at NET_0x200000a41c825_UUID:-1 lens 240/144 ref 0
fl
Interpret:/0/0 rc -16/0
LustreError: 1655:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
LustreError: 1659:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID
''ost2_UUID'' is not available  for connect (no target)
req at f182e600x226166544/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 0:0:(ldlm_lockd.c:205:waiting_locks_callback()) ### lock
callback timer expired: evicting client
c2914_lov1_153086c94b at NET_0x200000a41c825_UUID nid 10.65.200.37 at tcp  ns:
filter-ost1_UUID lock: e9a1c180/0x64a0224008966edf lrc: 1/0,0 mode: PW/PW
res: 28303634/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095)
flags: 10020 remote: 0xfbc2b603d4a9afd2 expref: 1976 pid: 1625
LustreError: 1704:0:(service.c:654:ptlrpc_server_handle_request()) request
226164089 opc 6 from 12345-10.65.200.37 at tcp processed in 106s trans 49298371
rc 0/0
Lustre: 1704:0:(watchdog.c:311:lcw_update_time()) Expired watchdog for pid
1704 disabled after 106098642.0001s
Lustre: 1704:0:(watchdog.c:311:lcw_update_time()) Skipped 1 previous similar
message
LustreError: 1556:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) received cancel
for unknown lock cookie 0x64a0224008966f72 from client c2914_lov1_153086c94b
id 12345-10.65.200.37 at tcp
LustreError: 1556:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) Skipped 1
previous similar message
LustreError: 1722:0:(filter_io.c:532:filter_preprw_write()) ost1: trying to
BRW to non-existent file 28303634
LustreError: 1538:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) received cancel
for unknown lock cookie 0x64a0224008966afd from client c2914_lov1_153086c94b
id 12345-10.65.200.37 at tcp
LustreError: 1546:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) received cancel
for unknown lock cookie 0x64a0224008966edf from client c2914_lov1_153086c94b
id 12345-10.65.200.37 at tcp
LustreError: 1546:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) Skipped 1
previous similar message
LustreError: 1609:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID
''ost2_UUID'' is not available  for connect (no target)
req at de92ba00x226305667/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1609:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-19)  req at de92ba00 x226305667/t0 o8-><?>@<?>:-1 lens
240/0 ref 0 fl
Interpret:/0/0 rc -19/0
LustreError: 1609:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 1
previous similar message
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from
12345-10.65.200.37 at tcp portal 16 match 10625151 offset 0 length 64: 2
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from
12345-10.65.200.37 at tcp portal 16 match 10625153 offset 0 length 64: 2
LustreError: 1636:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID
''ost2_UUID'' is not available  for connect (no target)
req at c5ef1200x226306267/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1636:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-19)  req at c5ef1200 x226306267/t0 o8-><?>@<?>:-1 lens
240/0 ref 0 fl
Interpret:/0/0 rc -19/0
LustreError: 1563:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ###
enqueue wait took 162499891us from 1296018348 ns: filter-ost1_UUID lock:
c4248040/0x64a0224008993a36 lrc: 2/0,0 mode: PW/PW res: 28132994/0 rrc: 2
type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0
remote: 0xfbc2b603d54bef71 expref: 2511 pid: 1640
LustreError: 1566:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ###
enqueue wait took 117534968us from 1296025387 ns: filter-ost1_UUID lock:
e5233bc0/0x64a02240089dd1f0 lrc: 2/0,0 mode: PR/PR res: 28145386/0 rrc: 8
type: EXT [0->18446744073709551615] (req 0->8191) flags: 0 remote:
0x943ed9a0643337b0 expref: 2553 pid: 1626
LustreError: 1566:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ###
enqueue wait took 118528501us from 1296025386 ns: filter-ost1_UUID lock:
ceb2a640/0x64a02240089dd1db lrc: 2/0,0 mode: PR/PR res: 28145386/0 rrc: 8
type: EXT [0->18446744073709551615] (req 4096->8191) flags: 0 remote:
0x943ed9a064330123 expref: 2553 pid: 1614
LustreError: 1566:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ###
enqueue wait took 161988593us from 1296025342 ns: filter-ost1_UUID lock:
c46ba3c0/0x64a02240089dcb4b lrc: 2/0,0 mode: PR/PR res: 28145386/0 rrc: 8
type: EXT [0->18446744073709551615] (req 4096->8191) flags: 0 remote:
0xc4d09fb494319a1e expref: 2549 pid: 1654
LustreError: 1566:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) Skipped
3 previous similar messages
LustreError: 1551:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ###
enqueue wait took 103243702us from 1296025522 ns: filter-ost1_UUID lock:
ef5628c0/0x64a02240089dede9 lrc: 2/0,0 mode: PW/PW res: 28327121/0 rrc: 2
type: EXT [0->18446744073709551615] (req 0->8191) flags: 0 remote:
0x943ed9a0643d689c expref: 2549 pid: 1609
LustreError: 1550:0:(ldlm_lockd.c:584:ldlm_server_completion_ast()) ###
enqueue wait took 111169666us from 1296025557 ns: filter-ost1_UUID lock:
e2926b40/0x64a02240089df376 lrc: 2/0,0 mode: PR/PR res: 28145386/0 rrc: 11
type: EXT [0->18446744073709551615] (req 0->8191) flags: 0 remote:
0xc4d09fb4943eb436 expref: 2538 pid: 1626
Lustre: 0:0:(watchdog.c:130:lcw_cb()) Watchdog triggered for pid 1691: it
was inactive for 100000ms
Lustre: 0:0:(watchdog.c:130:lcw_cb()) Skipped 2 previous similar messages
Lustre: 0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) showing stack for
process 1691
Lustre: 0:0:(linux-debug.c:166:libcfs_debug_dumpstack()) Skipped 2 previous
similar messages
ll_ost_io_19  S 00000001  4180  1691      1          1692  1690 (L-TLB)
f73ebc10 00000046 00000000 00000001 ffffffff ffffffff 00000000 00000000
       00000002 00000000 eaa9f480 c2022de0 00000001 00000000 a504d000
0056d473
       c22710b0 f73df1f0 f73df35c 00000000 00000246 b0c50fe0 b0c50fe0
ffffffff
Call Trace:
 [<c02dfaa9>] schedule_timeout+0x139/0x154
 [<c01296a4>] process_timeout+0x0/0x5
 [<c011f2f9>] add_wait_queue+0x12/0x30
 [<f930fb61>] ldlm_completion_ast+0x4bd/0x993 [ptlrpc]
 [<f930f045>] ldlm_process_extent_lock+0x4df/0x63b [ptlrpc]
 [<c011d6f8>] default_wake_function+0x0/0xc
 [<f92f815f>] l_unlock+0xab/0xc4 [ptlrpc]
 [<f930f42d>] ldlm_expired_completion_wait+0x0/0x277 [ptlrpc]
 [<f930f42c>] interrupted_completion_wait+0x0/0x1 [ptlrpc]
 [<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
 [<f931091c>] ldlm_cli_enqueue_local+0x4b4/0x5ce [ptlrpc]
 [<f9c9fcb6>] filter_prepare_destroy+0x11a/0x1c7 [obdfilter]
 [<f9310037>] ldlm_blocking_ast+0x0/0x42b [ptlrpc]
 [<f930f6a4>] ldlm_completion_ast+0x0/0x993 [ptlrpc]
 [<f9cac988>] filter_destroy+0x36d/0x184a [obdfilter]
 [<f93306ec>] ptlrpc_send_reply+0x38b/0x392 [ptlrpc]
 [<f93a8cbd>] ost_brw_write+0x224a/0x2418 [ost]
 [<f93a0704>] obd_destroy+0x3ef/0x484 [ost]
 [<f93a0280>] ost_destroy+0x236/0x2cb [ost]
 [<f93accaf>] ost_handle+0xfeb/0x383f [ost]
 [<f9339841>] ptlrpc_update_export_timer+0x233/0x454 [ptlrpc]
 [<f933a4d4>] ptlrpc_server_handle_request+0xa72/0x1204 [ptlrpc]
 [<f933bb8a>] ptlrpc_main+0x827/0x9e9 [ptlrpc]
 [<c011d6f8>] default_wake_function+0x0/0xc
 [<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
 [<c02e123e>] ret_from_fork+0x6/0x14
 [<f933b356>] ptlrpc_retry_rqbds+0x0/0xd [ptlrpc]
 [<f933b363>] ptlrpc_main+0x0/0x9e9 [ptlrpc]
 [<c01041f5>] kernel_thread_helper+0x5/0xb
LustreError: dumping log to
/tmp/lustre-log-typhoon.exampe.com.1296025710.1691
LustreError: 1691:0:(ldlm_request.c:59:ldlm_expired_completion_wait()) ###
lock timed out (enqueued at 1296025610, 100s ago); not entering recovery in
server code, just going back to sleep ns: filter-ost1_UUID lock:
eaa9f480/0x64a02240089dfe97 lrc: 3/0,1 mode: --/PW res: 28327587/0 rrc: 2
type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags:
80004000 remote: 0x0 expref: -99 pid: 1691
LustreError: 1691:0:(ldlm_request.c:59:ldlm_expired_completion_wait())
Skipped 2 previous similar messages
LustreError: 1654:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID
''ost2_UUID'' is not available  for connect (no target)
req at e22b0e00x904040/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1654:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-19)  req at e22b0e00 x904040/t0 o8-><?>@<?>:-1 lens 240/0
ref 0 fl
Interpret:/0/0 rc -19/0
LustreError: 1644:0:(client.c:940:ptlrpc_expire_one_request()) @@@ timeout
(sent at 1296025713, 20s ago)  req at cee31800 x10419493/t0
o104->@NET_0x200000a41c825_UUID:15 lens 176/64 ref 1 fl Rpc:/0/0 rc 0/0
LustreError: A client on nid 10.65.200.37 at tcp was evicted from service ost1.
LustreError: 1644:0:(ldlm_lockd.c:427:ldlm_failed_ast()) ### blocking AST
failed (-110): evicting client
7c2f0_lov1_34edb7f8f7 at NET_0x200000a41c825_UUID NID
10.65.200.37 at tcp(10.65.200.37 at tcp)
ns: filter-ost1_UUID lock: ed9131c0/0x64a02240089dfbd4 lrc: 2/0,0 mode:
PW/PW res: 28177721/0 rrc: 2 type: EXT [0->18446744073709551615] (req
0->18446744073709551615) flags: 20 remote: 0x943ed9a06440dfce expref: 2519
pid: 1662
LustreError: 1691:0:(service.c:654:ptlrpc_server_handle_request()) request
876972 opc 6 from 12345-10.65.200.37 at tcp processed in 122s trans 49349204 rc
0/0
LustreError: 1691:0:(service.c:654:ptlrpc_server_handle_request()) Skipped 6
previous similar messages
Lustre: 1691:0:(watchdog.c:311:lcw_update_time()) Expired watchdog for pid
1691 disabled after 122643196.0001s
Lustre: 1691:0:(watchdog.c:311:lcw_update_time()) Skipped 5 previous similar
messages
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from
12345-10.65.200.37 at tcp portal 16 match 10663839 offset 0 length 112: 2
LustreError: 1638:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID
''ost2_UUID'' is not available  for connect (no target)
req at d9df1400x904744/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1638:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-19)  req at d9df1400 x904744/t0 o8-><?>@<?>:-1 lens 240/0
ref 0 fl
Interpret:/0/0 rc -19/0
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from
12345-10.65.200.37 at tcp portal 16 match 10663856 offset 0 length 112: 2
LustreError: 1543:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) received cancel
for unknown lock cookie 0x64a02240089dfbd4 from client 7c2f0_lov1_34edb7f8f7
id 12345-10.65.200.37 at tcp
LustreError: 1543:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) Skipped 1
previous similar message
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from
12345-10.65.200.37 at tcp portal 16 match 10663860 offset 0 length 64: 2
Lustre: 1470:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from
12345-10.65.200.37 at tcp portal 16 match 10663863 offset 0 length 112: 2
LustreError: 1730:0:(filter_io.c:532:filter_preprw_write()) ost1: trying to
BRW to non-existent file 28331257
LustreError: 1730:0:(filter_io.c:532:filter_preprw_write()) Skipped 1
previous similar message
LustreError: 0:0:(ldlm_lockd.c:205:waiting_locks_callback()) ### lock
callback timer expired: evicting client
7c2f0_lov1_34edb7f8f7 at NET_0x200000a41c825_UUID nid 10.65.200.37 at tcp  ns:
filter-ost1_UUID lock: d089b080/0x64a02240089fdf2f lrc: 1/0,0 mode: PW/PW
res: 28049109/0 rrc: 3 type: EXT [0->18446744073709551615] (req
0->18446744073709551615) flags: 20 remote: 0x943ed9a064e092a6 expref: 2547
pid: 1650
LustreError: 1708:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-107)  req at d1b25200 x1184444/t0 o4-><?>@<?>:-1 lens
328/0 ref 0 fl
Interpret:/0/0 rc -107/0
LustreError: 1542:0:(ldlm_lockd.c:1056:ldlm_handle_cancel()) received cancel
for unknown lock cookie 0x64a02240089f6002 from client 7c2f0_lov1_34edb7f8f7
id 12345-10.65.200.37 at tcp
LustreError: 1638:0:(ldlm_lib.c:557:target_handle_connect()) @@@ UUID
''ost2_UUID'' is not available  for connect (no target)
req at d0c7da00x1209963/t0 o8-><?>@<?>:-1 lens 240/0 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 1638:0:(ldlm_lib.c:1318:target_send_reply_msg()) @@@ processing
error (-19)  req at d0c7da00 x1209963/t0 o8-><?>@<?>:-1 lens
240/0 ref 0 fl
Interpret:/0/0 rc -19/0
LustreError: 1638:0:(ldlm_lib.c:1318:target_send_reply_msg()) Skipped 2
previous similar messages

Clinet loogs

LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 27
previous similar messages
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) obd destroy objid
0xbc84e6 error -5
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) Skipped 3 previous
similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c1b33ea0 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 10
previous similar messages
LustreError: 6561:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup;
forcing cleanup.
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) obd destroy objid
0xbc8654 error -5
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) Skipped 11 previous
similar messages
LustreError: 6383:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc
-5, returning -EIO
LustreError: 6383:0:(file.c:1003:ll_glimpse_size()) Skipped 3 previous
similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c1225440 failed: -5
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@
IMP_INVALID  req at f7d8d400 x906093/t0 o4->ost1_UUID at typhoon_UUID:28 lens
328/288 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 505
previous similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c12f00e0 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 236
previous similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c124bb80 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 430
previous similar messages
LustreError: 6419:0:(namei.c:941:ll_objects_destroy()) obd destroy objid
0xbc8776 error -5
LustreError: 6419:0:(namei.c:941:ll_objects_destroy()) Skipped 14 previous
similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c14d3bc0 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 269
previous similar messages
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) obd destroy objid
0xbc8732 error -5
LustreError: 6370:0:(namei.c:941:ll_objects_destroy()) Skipped 42 previous
similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c179c260 failed: -5
LustreError: 6561:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup;
forcing cleanup.
LustreError: 6561:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost1_MNT_client resource refcount 2 after lock cleanup;
forcing cleanup.
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@
IMP_INVALID  req at f7d8d800 x907615/t0 o4->ost1_UUID at typhoon_UUID:28 lens
328/288 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 997
previous similar messages
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c1094f20 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 712
previous similar messages
LustreError: 5723:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc
-5, returning -EIO
LustreError: 5723:0:(file.c:1003:ll_glimpse_size()) Skipped 4 previous
similar messages
LustreError: 6564:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async
fails: rc=-5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c13206e0 failed: -5
LustreError: 6561:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 2057
previous similar messages
Lustre: OSC_hades.exampe.com_ost1_MNT_client: Connection restored to service
ost1 using nid 10.65.200.21 at tcp.
LustreError: 3036:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue
objid 0x5f7a008 subobj 0x1ad4682 on OST idx 0: rc -5
LustreError: 2182:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server
(nid 10.65.200.21 at tcp) out of sync -- not fatal, flags 322c90
LustreError: 2182:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel
failed: 116
Lustre: 2118:0:(lib-move.c:1644:lnet_parse_put()) Dropping PUT from
12345-10.65.200.30 at tcp portal 4 match 877070 offset 0 length 280: 2
Lustre: Changing connection for OSC_hades.exampe.com_ost2_MNT_client to
cyclops_UUID/10.65.200.30 at tcp
LustreError: This client was evicted by ost2; in progress operations using
this service will fail.
LustreError: 2183:0:(ldlm_request.c:752:ldlm_cli_cancel()) Got rc -5 from
cancel RPC: canceling anyway
LustreError: 6445:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue
objid 0x7670003 subobj 0x1ab6cc4 on OST idx 1: rc -5
LustreError: 6445:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 1
previous similar message
LustreError: 6445:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc
-5, returning -EIO
LustreError: 6445:0:(file.c:1003:ll_glimpse_size()) Skipped 1 previous
similar message
LustreError: 5590:0:(rw.c:966:ll_issue_page_read()) page c12300e0 map
eab7c3f0 index 0 flags 20001023 count 4 priv e9ee5dc0: read queue failed: rc
-5
LustreError: 2183:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel
failed: -5
LustreError: 2183:0:(file.c:754:ll_extent_lock_callback()) Skipped 1
previous similar message
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@
IMP_INVALID  req at c229d200 x911586/t0 o4->ost2_UUID at cyclops_UUID:28 lens
328/288 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 1141
previous similar messages
LustreError: 3252:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async
fails: rc=-5
LustreError: 6332:0:(rw.c:966:ll_issue_page_read()) page c1793ba0 map
e746f630 index 0 flags 40001123 count 5 priv d5c0a980: read queue failed: rc
-5
LustreError: 6435:0:(rw.c:966:ll_issue_page_read()) page c1793ba0 map
e746f630 index 0 flags 40001123 count 4 priv d5c0a980: read queue failed: rc
-5
LustreError: 6435:0:(rw.c:966:ll_issue_page_read()) page c1793ba0 map
e746f630 index 0 flags 40001123 count 4 priv d5c0a980: read queue failed: rc
-5
LustreError: 6275:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue
objid 0x6728007 subobj 0x1ab6cc8 on OST idx 1: rc -5
LustreError: 6275:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 7
previous similar messages
LustreError: 2191:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c1363380 failed: -5
LustreError: 2191:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 3493
previous similar messages
LustreError: 5672:0:(rw.c:966:ll_issue_page_read()) page c1402980 map
d706c870 index 1 flags 20001023 count 7 priv e92c1380: read queue failed: rc
-5
LustreError: 5747:0:(rw.c:966:ll_issue_page_read()) page c1402980 map
d706c870 index 1 flags 20001023 count 6 priv e92c1380: read queue failed: rc
-5
LustreError: 5747:0:(rw.c:966:ll_issue_page_read()) page c1402980 map
d706c870 index 1 flags 20001023 count 8 priv e92c1380: read queue failed: rc
-5
LustreError: 5834:0:(rw.c:966:ll_issue_page_read()) page c1402980 map
d706c870 index 1 flags 20001023 count 7 priv e92c1380: read queue failed: rc
-5
LustreError: 5834:0:(rw.c:966:ll_issue_page_read()) page c1402980 map
d706c870 index 1 flags 20001023 count 7 priv e92c1380: read queue failed: rc
-5
LustreError: 6579:0:(rw.c:966:ll_issue_page_read()) page c1ad5900 map
e3420ab0 index 0 flags 40001123 count 4 priv e3910100: read queue failed: rc
-5
LustreError: 6579:0:(rw.c:966:ll_issue_page_read()) page c1ad5900 map
e3420ab0 index 0 flags 40001123 count 4 priv e3910100: read queue failed: rc
-5
LustreError: 6538:0:(rw.c:966:ll_issue_page_read()) page c15db2a0 map
f5273630 index 0 flags 20001023 count 5 priv da791f00: read queue failed: rc
-5
LustreError: 2918:0:(rw.c:966:ll_issue_page_read()) page c15db2a0 map
f5273630 index 0 flags 20001023 count 4 priv da791f00: read queue failed: rc
-5
LustreError: 6579:0:(rw.c:966:ll_issue_page_read()) page c1ad5900 map
e3420ab0 index 0 flags 40001123 count 4 priv e3910100: read queue failed: rc
-5
LustreError: 6517:0:(rw.c:966:ll_issue_page_read()) page c1ad5900 map
e3420ab0 index 0 flags 40001123 count 4 priv e3910100: read queue failed: rc
-5
LustreError: 6517:0:(rw.c:966:ll_issue_page_read()) page c1ad5900 map
e3420ab0 index 0 flags 40001123 count 4 priv e3910100: read queue failed: rc
-5
LustreError: 5726:0:(rw.c:966:ll_issue_page_read()) page c1402980 map
d706c870 index 1 flags 20001023 count 6 priv e92c1380: read queue failed: rc
-5
LustreError: 5993:0:(rw.c:966:ll_issue_page_read()) page c1402980 map
d706c870 index 1 flags 20001023 count 5 priv e92c1380: read queue failed: rc
-5
LustreError: 5672:0:(rw.c:966:ll_issue_page_read()) page c1402980 map
d706c870 index 1 flags 20001023 count 4 priv e92c1380: read queue failed: rc
-5
LustreError: 3050:0:(rw.c:966:ll_issue_page_read()) page c1257680 map
e5c73870 index 0 flags 20001023 count 4 priv d8ef9180: read queue failed: rc
-5
LustreError: 5993:0:(rw.c:966:ll_issue_page_read()) page c1402980 map
d706c870 index 1 flags 20001023 count 4 priv e92c1380: read queue failed: rc
-5
LustreError: 6594:0:(rw.c:966:ll_issue_page_read()) page c1638a20 map
f50903f0 index 0 flags 20001023 count 4 priv f4c60100: read queue failed: rc
-5
LustreError: 6538:0:(rw.c:966:ll_issue_page_read()) page c15db2a0 map
f5273630 index 0 flags 20001023 count 4 priv da791f00: read queue failed: rc
-5
LustreError: 6596:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async
fails: rc=-5
LustreError: 6596:0:(llite_lib.c:1334:ll_setattr_raw()) Skipped 3 previous
similar messages
LustreError: 6412:0:(namei.c:941:ll_objects_destroy()) obd destroy objid
0x74005ef error -5
LustreError: 6412:0:(namei.c:941:ll_objects_destroy()) Skipped 134 previous
similar messages
LustreError: 6412:0:(file.c:106:ll_close_inode_openhandle()) inode 99614226
ll_objects destroy: rc = -5
LustreError: 6597:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost2_MNT_client resource refcount 2 after lock cleanup;
forcing cleanup.
LustreError: 6601:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc
-5, returning -EIO
LustreError: 6601:0:(file.c:1003:ll_glimpse_size()) Skipped 13 previous
similar messages
Lustre: OSC_hades.exampe.com_ost2_MNT_client: Connection restored to service
ost2 using nid 10.65.200.30 at tcp.
LustreError: 6536:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue
objid 0x5efcea5 subobj 0x1a88749 on OST idx 1: rc -5
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type
=PTL_RPC_MSG_ERR, err == -2  req at f7d8f400 x1081227/t0
o4->ost1_UUID at typhoon_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 1 previous
similar message
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type
=PTL_RPC_MSG_ERR, err == -107  req at f7d8dc00 x1184444/t0
o4->ost1_UUID at typhoon_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-107
LustreError: OSC_hades.exampe.com_ost1_MNT_client: Connection to service
ost1 via nid 10.65.200.21 at tcp was lost; in progress operations using this
service will wait for recovery to complete.
Lustre: Changing connection for OSC_hades.exampe.com_ost1_MNT_client to
cyclops_UUID/10.65.200.30 at tcp
Lustre: Changing connection for OSC_hades.exampe.com_ost1_MNT_client to
typhoon_UUID/10.65.200.21 at tcp
LustreError: This client was evicted by ost1; in progress operations using
this service will fail.
LustreError: 7673:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue
objid 0x6c08150 subobj 0x1b055ab on OST idx 0: rc -5
LustreError: 2944:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc
-5, returning -EIO
LustreError: 7673:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 2
previous similar messages
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@
IMP_INVALID  req at f6878a00 x1186234/t0 o3->ost1_UUID at typhoon_UUID:28
lens
328/280 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 657
previous similar messages
LustreError: 6363:0:(rw.c:1446:ll_readpage()) page c1a9d160 map f48753f0
index 0 flags 40001123 count 5 priv e33a2d00: lock match failed: rc -5
LustreError: 6227:0:(rw.c:1446:ll_readpage()) page c1a9d160 map f48753f0
index 0 flags 40001123 count 4 priv e33a2d00: lock match failed: rc -5
LustreError: 7722:0:(rw.c:1446:ll_readpage()) page c13a5240 map f23e6630
index 1 flags 20001023 count 3 priv f4198e80: lock match failed: rc -5
LustreError: 7736:0:(rw.c:1446:ll_readpage()) page c1a9d160 map f48753f0
index 0 flags 40001123 count 3 priv e33a2d00: lock match failed: rc -5
LustreError: 6949:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0
index 1 flags 20001023 count 14 priv dc8db080: lock match failed: rc -5
LustreError: 6637:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0
index 1 flags 20001023 count 12 priv dc8db080: lock match failed: rc -5
LustreError: 5723:0:(rw.c:1446:ll_readpage()) page c1abd4c0 map f6a4ef30
index 0 flags 40001123 count 3 priv d33c85c0: lock match failed: rc -5
LustreError: 7615:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async
fails: rc=-5
LustreError: 6790:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0
index 1 flags 20001023 count 11 priv dc8db080: lock match failed: rc -5
LustreError: 7467:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0
index 1 flags 20001023 count 10 priv dc8db080: lock match failed: rc -5
LustreError: 6689:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0
index 1 flags 20001023 count 9 priv dc8db080: lock match failed: rc -5
LustreError: 6547:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0
index 1 flags 20001023 count 8 priv dc8db080: lock match failed: rc -5
LustreError: 7149:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0
index 1 flags 20001023 count 7 priv dc8db080: lock match failed: rc -5
LustreError: 6997:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0
index 1 flags 20001023 count 6 priv dc8db080: lock match failed: rc -5
LustreError: 6734:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0
index 1 flags 20001023 count 5 priv dc8db080: lock match failed: rc -5
LustreError: 7134:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0
index 1 flags 20001023 count 4 priv dc8db080: lock match failed: rc -5
LustreError: 7597:0:(rw.c:1446:ll_readpage()) page c1add3a0 map e3a961b0
index 1 flags 40001123 count 4 priv f60943c0: lock match failed: rc -5
LustreError: 7513:0:(rw.c:1446:ll_readpage()) page c15ee600 map d79073f0
index 1 flags 20001023 count 3 priv dc8db080: lock match failed: rc -5
LustreError: 7744:0:(rw.c:1446:ll_readpage()) page c12cf300 map f3e5ecf0
index 0 flags 20001023 count 3 priv e4c4aa00: lock match failed: rc -5
LustreError: 7741:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async
fails: rc=-5
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c1bd2380 failed: -5
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 7533
previous similar messages
LustreError: 7741:0:(namei.c:941:ll_objects_destroy()) obd destroy objid
0xbcac42 error -5
LustreError: 7741:0:(namei.c:941:ll_objects_destroy()) Skipped 493 previous
similar messages
LustreError: 6689:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async
fails: rc=-5
LustreError: 6689:0:(llite_lib.c:1334:ll_setattr_raw()) Skipped 1 previous
similar message
LustreError: 7452:0:(file.c:106:ll_close_inode_openhandle()) inode 100175813
ll_objects destroy: rc = -5
LustreError: 7650:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc
-5, returning -EIO
LustreError: 7650:0:(file.c:1003:ll_glimpse_size()) Skipped 4 previous
similar messages
LustreError: 7650:0:(llite_lib.c:1334:ll_setattr_raw()) obd_setattr_async
fails: rc=-5
LustreError: 7650:0:(llite_lib.c:1334:ll_setattr_raw()) Skipped 6 previous
similar messages
LustreError: 7650:0:(namei.c:941:ll_objects_destroy()) obd destroy objid
0x429000a error -5
LustreError: 7650:0:(namei.c:941:ll_objects_destroy()) Skipped 5 previous
similar messages
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@
IMP_INVALID  req at f7d8d600 x1187047/t0 o4->ost1_UUID at typhoon_UUID:28
lens
328/288 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 620
previous similar messages
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c12edce0 failed: -5
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 1711
previous similar messages
LustreError: 7778:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup;
forcing cleanup.
LustreError: 7778:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup;
forcing cleanup.
LustreError: 6689:0:(namei.c:941:ll_objects_destroy()) obd destroy objid
0xbacaf0 error -5
LustreError: 6689:0:(namei.c:941:ll_objects_destroy()) Skipped 79 previous
similar messages
LustreError: 7778:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup;
forcing cleanup.
LustreError: 7513:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc
-5, returning -EIO
LustreError: 7513:0:(file.c:1003:ll_glimpse_size()) Skipped 15 previous
similar messages
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c1a28b40 failed: -5
LustreError: 7778:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 5271
previous similar messages
LustreError: 7778:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup;
forcing cleanup.
LustreError: 7778:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost1_MNT_client resource refcount 1 after lock cleanup;
forcing cleanup.
Lustre: OSC_hades.exampe.com_ost1_MNT_client: Connection restored to service
ost1 using nid 10.65.200.21 at tcp.
LustreError: 5978:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue
objid 0x5f9b8ed subobj 0x1adf539 on OST idx 0: rc -5
LustreError: 5978:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 5
previous similar messages
LustreError: 2206:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server
(nid 10.65.200.21 at tcp) out of sync -- not fatal, flags 322c90
LustreError: 2206:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel
failed: 116
LustreError: 2206:0:(file.c:754:ll_extent_lock_callback()) Skipped 2
previous similar messages
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type
=PTL_RPC_MSG_ERR, err == -107  req at c229d400 x1209948/t0
o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-107
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 3 previous
similar messages
LustreError: OSC_hades.exampe.com_ost2_MNT_client: Connection to service
ost2 via nid 10.65.200.30 at tcp was lost; in progress operations using this
service will wait for recovery to complete.
Lustre: Changing connection for OSC_hades.exampe.com_ost2_MNT_client to
typhoon_UUID/10.65.200.21 at tcp
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type
=PTL_RPC_MSG_ERR, err == -107  req at f7e0ca00 x1209953/t0
o4->ost2_UUID at typhoon_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-107
LustreError: 2173:0:(ldlm_request.c:752:ldlm_cli_cancel()) Got rc -107 from
cancel RPC: canceling anyway
LustreError: 2173:0:(ldlm_request.c:752:ldlm_cli_cancel()) Skipped 3
previous similar messages
LustreError: 2173:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel
failed: -107
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 5 previous
similar messages
LustreError: 2209:0:(ldlm_request.c:752:ldlm_cli_cancel()) Got rc -107 from
cancel RPC: canceling anyway
Lustre: Changing connection for OSC_hades.exampe.com_ost2_MNT_client to
cyclops_UUID/10.65.200.30 at tcp
LustreError: This client was evicted by ost2; in progress operations using
this service will fail.
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) @@@
IMP_INVALID  req at c22a3200 x1219312/t0 o4->ost2_UUID at cyclops_UUID:28
lens
328/288 ref 2 fl Rpc:/0/0 rc 0/0
LustreError: 2169:0:(client.c:511:ptlrpc_import_delay_req()) Skipped 514
previous similar messages
LustreError: 7819:0:(file.c:1003:ll_glimpse_size()) obd_enqueue returned rc
-5, returning -EIO
LustreError: 7819:0:(file.c:1003:ll_glimpse_size()) Skipped 4 previous
similar messages
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost2_MNT_client resource refcount 1 after lock cleanup;
forcing cleanup.
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Skipped 1
previous similar message
LustreError: 2176:0:(file.c:703:ll_pgcache_remove_extent()) writepage of
page c17af000 failed: -5
LustreError: 2176:0:(file.c:703:ll_pgcache_remove_extent()) Skipped 1080
previous similar messages
LustreError: 7786:0:(rw.c:1446:ll_readpage()) page c1245840 map f1cb7870
index 1 flags 20001023 count 3 priv d8334b80: lock match failed: rc -5
LustreError: 7819:0:(namei.c:941:ll_objects_destroy()) obd destroy objid
0x5f90f20 error -5
LustreError: 7819:0:(namei.c:941:ll_objects_destroy()) Skipped 218 previous
similar messages
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost2_MNT_client resource refcount 1 after lock cleanup;
forcing cleanup.
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Skipped 1
previous similar message
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Namespace
OSC_hades.exampe.com_ost2_MNT_client resource refcount 1 after lock cleanup;
forcing cleanup.
LustreError: 7832:0:(ldlm_resource.c:365:ldlm_namespace_cleanup()) Skipped
17 previous similar messages
Lustre: OSC_hades.exampe.com_ost2_MNT_client: Connection restored to service
ost2 using nid 10.65.200.30 at tcp.
LustreError: 2220:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server
(nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2220:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel
failed: 116
LustreError: 2220:0:(file.c:754:ll_extent_lock_callback()) Skipped 6
previous similar messages
LustreError: 2203:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server
(nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2203:0:(ldlm_request.c:746:ldlm_cli_cancel()) Skipped 4
previous similar messages
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type
=PTL_RPC_MSG_ERR, err == -2  req at c229d400 x1219419/t0
o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 2 previous
similar messages
LustreError: 7781:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue
objid 0x5ef90fd subobj 0x1a5e593 on OST idx 1: rc -5
LustreError: 7781:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 1
previous similar message
LustreError: 2227:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel
failed: 116
LustreError: 2227:0:(file.c:754:ll_extent_lock_callback()) Skipped 12
previous similar messages
LustreError: 2195:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server
(nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2195:0:(ldlm_request.c:746:ldlm_cli_cancel()) Skipped 10
previous similar messages
LustreError: 7813:0:(lov_request.c:181:lov_update_enqueue_set()) enqueue
objid 0x7108001 subobj 0x1a76e0d on OST idx 1: rc -5
LustreError: 7813:0:(lov_request.c:181:lov_update_enqueue_set()) Skipped 1
previous similar message
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type
=PTL_RPC_MSG_ERR, err == -2  req at c22a3a00 x1219453/t0
o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 4 previous
similar messages
LustreError: 2199:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel
failed: 116
LustreError: 2199:0:(file.c:754:ll_extent_lock_callback()) Skipped 3
previous similar messages
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type
=PTL_RPC_MSG_ERR, err == -2  req at c229d200 x1219484/t0
o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 17 previous
similar messages
LustreError: 2208:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server
(nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2208:0:(ldlm_request.c:746:ldlm_cli_cancel()) Skipped 1
previous similar message
LustreError: 2208:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel
failed: 116
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type
=PTL_RPC_MSG_ERR, err == -2  req at c229bc00 x1219552/t0
o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 40 previous
similar messages
LustreError: 2188:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server
(nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2188:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel
failed: 116
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) @@@ type
=PTL_RPC_MSG_ERR, err == -2  req at c22a3a00 x1219666/t0
o4->ost2_UUID at cyclops_UUID:28 lens 328/288 ref 2 fl Rpc:R/0/0 rc 0/-2
LustreError: 2169:0:(client.c:576:ptlrpc_check_status()) Skipped 88 previous
similar messages
LustreError: 2231:0:(ldlm_request.c:746:ldlm_cli_cancel()) client/server
(nid 10.65.200.30 at tcp) out of sync -- not fatal, flags 332c90
LustreError: 2231:0:(ldlm_request.c:746:ldlm_cli_cancel()) Skipped 2
previous similar messages
LustreError: 2231:0:(file.c:754:ll_extent_lock_callback()) ldlm_cli_cancel
failed: 116
LustreError: 2231:0:(file.c:754:ll_extent_lock_callback()) Skipped 2
previous similar messages





-- 
Regards

Nauman Yousuf
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20110126/e42c90b6/attachment-0001.html
Lustre discuss - Jan 2011 - Lustre Issues

[Lustre-discuss] Lustre Issues
