Song
2013-Jun-14 09:28 UTC
[Gluster-users] bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) ... timeout = 1800
Hi, everyone: We deploy GlusterFS3.2.7 with 24 servers and each server has 12 disks using for brick server. The volume type is DHT + AFR(replica=3) and transport type is socket. Use native client and each server has 8 mount points. The gluster client will log some error once in a while, following is an example. We google these error, but not find any method to resolve it. Can give me some help, thanks! [2013-06-14 15:58:41.913210] W [client-handshake.c:264:client_ping_cbk] 0-gfs1-client-174: timer must have expired [2013-06-14 15:58:41.913439] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde]))) 0-gfs1-client-184: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-06-14 15:57:36.516579 [2013-06-14 15:58:41.913456] W [client-handshake.c:264:client_ping_cbk] 0-gfs1-client-184: timer must have expired [2013-06-14 15:58:41.913526] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde]))) 0-gfs1-client-187: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-06-14 15:57:36.516591 [2013-06-14 15:58:41.913542] W [client-handshake.c:264:client_ping_cbk] 0-gfs1-client-187: timer must have expired [2013-06-14 15:58:41.913662] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde]))) 0-gfs1-client-170: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-06-14 15:57:36.516554 [2013-06-14 15:58:41.913678] W [client-handshake.c:264:client_ping_cbk] 0-gfs1-client-170: timer must have expired [2013-06-14 15:58:41.914089] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x32f320e6e8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x32f320de7d] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x32f320ddde]))) 0-gfs1-client-180: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-06-14 15:57:36.516543 [2013-06-14 15:58:41.914115] W [client-handshake.c:264:client_ping_cbk] 0-gfs1-client-180: timer must have expired [2013-06-14 16:16:36.467651] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11081156x sent = 2013-06-14 15:46:27.573575. timeout = 1800 [2013-06-14 16:16:36.467723] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote operation failed: Transport endpoint is not connected [2013-06-14 16:32:07.91967] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-48: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11115092x sent = 2013-06-14 16:02:02.712824. timeout = 1800 [2013-06-14 16:32:07.92014] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-48: remote operation failed: Transport endpoint is not connected [2013-06-14 16:35:42.221471] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-48: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11115696x sent = 2013-06-14 16:05:41.199990. timeout = 1800 [2013-06-14 16:35:42.221586] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-48: remote operation failed: Transport endpoint is not connected [2013-06-14 16:36:12.766153] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-48: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11115817x sent = 2013-06-14 16:06:02.592259. timeout = 1800 [2013-06-14 16:36:12.766201] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-48: remote operation failed: Transport endpoint is not connected [2013-06-14 16:47:32.203692] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11095024x sent = 2013-06-14 16:17:31.966654. timeout = 1800 [2013-06-14 16:47:32.203766] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote operation failed: Transport endpoint is not connected [2013-06-14 17:17:42.983248] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11112188x sent = 2013-06-14 16:47:32.210391. timeout = 1800 [2013-06-14 17:17:42.983337] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote operation failed: Transport endpoint is not connected [2013-06-14 17:17:42.983413] E [rpc-clnt.c:197:call_bail] 0-gfs1-client-51: bailing out frame type(GlusterFS 3.1) op(FINODELK(30)) xid = 0x11112186x sent = 2013-06-14 16:47:32.206834. timeout = 1800 [2013-06-14 17:17:42.983449] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gfs1-client-51: remote operation failed: Transport endpoint is not connected [2013-06-14 17:19:15.201853] W [dict.c:418:dict_unref] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x32f320e4e5] (-->/usr/lib/glusterfs/3.2.7/xlator/protocol/client.so(client3_1_fstat_cbk+0 x343) [0x7fd4d4c4a373] (-->/usr/lib/glusterfs/3.2.7/xlator/cluster/replicate.so(afr_sh_data_fstat_c bk+0x1eb) [0x7fd4d49ed90b]))) 0-dict: dict is NULL Volume info as below: [root at bj-nx-cip-w83 ~]# gluster volume info Volume Name: gfs1 Type: Distributed-Replicate Status: Started Number of Bricks: 94 x 3 = 282 Transport-type: tcp Bricks: Brick1: 10.0.11.81:/xmail/disk1/gfs1 Brick2: 10.0.12.71:/xmail/disk1/gfs1 Brick3: 10.0.13.91:/xmail/disk1/gfs1 Brick4: 10.0.11.82:/xmail/disk1/gfs1 Brick5: 10.0.12.72:/xmail/disk1/gfs1 Brick6: 10.0.13.92:/xmail/disk1/gfs1 ?? Brick280: 10.0.13.88:/xmail/disk12/gfs1 Brick281: 10.0.11.78:/xmail/disk12/gfs1 Brick282: 10.0.12.68:/xmail/disk12/gfs1 Options Reconfigured: performance.read-ahead: on performance.io-thread-count: 16 diagnostics.client-log-level: WARNING diagnostics.brick-log-level: WARNING nfs.disable: on performance.io-cache: off performance.write-behind: off -------------------------- Cailiang song -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130614/02b5df9c/attachment.html>