web client http://fpaste.org/105784/14014066/ gluster storage server http://fpaste.org/105786/40140677/ Please find the above fpaste a situation I have experienced twice where my 8-node gluster 2x4 distributed-replication was doing the connect/disconnect game. My servers are rhel 6.4 on glusterfs3.4.2 . Volume Name: prodstatic Type: Distributed-Replicate Volume ID: 187c241d-0eeb-4405-80f2-c704ea44bc36 Status: Started Number of Bricks: 2 x 4 = 8 Transport-type: tcp Bricks: Brick1: omhq1140:/export/content/static Brick2: omdx1c5d:/export/content/static Brick3: omhq11ad:/export/content/static Brick4: omdx1781:/export/content/static Brick5: omhq1c56:/export/content/static Brick6: omdx1c58:/export/content/static Brick7: omhq1c57:/export/content/static Brick8: omdx1c59:/export/content/static Options Reconfigured: server.statedump-path: /debug features.quota: on server.allow-insecure: on network.ping-timeout: 10 My question is why the frequent disconnect/connects ? It was very sporadic in my environment across multiple clients/storage nodes. The chaos lasted about 10:20-12:00. My network admins are looking at the network while i'm trying to make sense of my storage logs. 30 mins prior, i made a config change 'gluster volume profile prodstatic stop'. I couldn't imagine that issuing that command could drop my storage/clients like flies, can it ? From the client logs it appears to establish an rpc connection, but immediately drops it? Are the logs lying? Is my understanding of network.ping-timeout too low? I wanted the clients to act if the storage is not responding drop him, and move on to the next available replica. If that is not the intention then a higher network.ping-timeout would mean a longer stall which is not my desire. All my storage disks are fine, I never once got an alert for an offline server/client. During this duration the mounted fuse filesystems were crawling! No rebalance was done. Any bugzilla's or threads is much appreciated. Khoi ** This email and any attachments may contain information that is confidential and/or privileged for the sole use of the intended recipient. Any use, review, disclosure, copying, distribution or reliance by others, and any forwarding of this email or its contents, without the express permission of the sender is strictly prohibited by law. If you are not the intended recipient, please contact the sender immediately, delete the e-mail and destroy all copies. ** -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140529/8706cff3/attachment.html>