Morning everyone, Hoping someone can help me out with this. I've been running GlusterFS for awhile now and everything was great. Now for about the last month I'm lucky if it runs for a few days without crashing and bringing all the servers down. Here's what I can see in the logs when a failure occurs. I see this across all three hosts in the cluster. [2015-05-19 04:12:33.761831] C [rpc-clnt-ping.c:109:rpc_clnt_ping_timer_expired] 0-www-client-0: server x.x.x.x:49157 has not responded in the last 42 seconds, disconnecting. [2015-05-19 04:12:33.762269] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff0ae43c550] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff0ae211787] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff0ae2118 9e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7ff0ae211951] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff 0ae211f1f] ))))) 0-www-client-0: forced unwinding frame type(GlusterFS 3.3) op(OPENDIR(20)) called at 2015-05-19 04:11:51.000813 (xid=0x4a67) [2015-05-19 04:12:33.762302] E [client-rpc-fops.c:2686:client3_3_opendir_cbk] 0-www-client-0: remote operation failed: Transport endpoint is n ot connected. Path: <gfid:a1fb01c7-bc8e-4854-9760-8da8d62519bc> (a1fb01c7-bc8e-4854-9760-8da8d62519bc) [2015-05-19 04:12:33.762436] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff0ae43c550] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff0ae211787] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff0ae2118 9e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7ff0ae211951] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff 0ae211f1f] ))))) 0-www-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2015-05-19 04:11:51.000832 (xid=0x4a68) [2015-05-19 04:12:33.762455] W [rpc-clnt-ping.c:154:rpc_clnt_ping_cbk] 0-www-client-0: socket disconnected [2015-05-19 04:16:45.804515] C [rpc-clnt-ping.c:109:rpc_clnt_ping_timer_expired] 0-www-conf-client-0: server x.x.x.x:49156 has not responde d in the last 42 seconds, disconnecting. [2015-05-19 04:16:45.804884] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff0ae43c550] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff0ae211787] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff0ae2118 9e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7ff0ae211951] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff 0ae211f1f] ))))) 0-www-conf-client-0: forced unwinding frame type(GlusterFS 3.3) op(OPENDIR(20)) called at 2015-05-19 04:16:03.000774 (xid=0x4 a83) Here's info about the version I'm running: glusterfs 3.6.3 built on Apr 23 2015 16:12:23 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. Any insight would be appreciated, -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150519/962f2d9f/attachment.html>