Havriliuc Andrei
2018-Jul-13 09:44 UTC
[Gluster-users] Problem with SSL/TLS encryption on Gluster 4.0 & 4.1
Hello David, Yeah. I also disabled SSL/TLS for management and communication for Gluster. The problem is that I will have traffic over public WAN for a geo replication instance and I cannot go forward until this is solved. Was there a bug filed about this? I've looked at the issues on the Glusterfs github page but I didn't find anything related to this: https://github.com/gluster/glusterfs/issues Regards, Andrei On 7/12/18 4:03 PM, David Spisla wrote:> Hello Andrei, > > I am also using Gluster 4.1 on CentOS and I have the same problem. I > tested it with a volume which had no network encryption and one with > network encryption. You are not the only one. > It seems to be a bug. At the moment there is no other choice to > disable client.ssl and sever.ssl on a volume to have stable I/O. > > Regards > David > > > 2018-07-12 10:45 GMT+02:00 Havriliuc Andrei <andrei at hostvision.ro > <mailto:andrei at hostvision.ro>>: > > Hello, > > I am doing some tests with GlusterFS 4.0 and I can't seem to solve > some SSL/TLS issues. I am trying to set up a 2 node replicated > gluster volume with SSL/TLS. For this setup, I use 3 KVM VMs (2 > storage nodes + 1 client node). For the networking part, I use a > dedicated private LAN for the KVM VMs. Each VM is able to ping the > other, so there's no problem with the connectivity. > > To try to make the procedure I used as clear as possible, I will > put all commands in chronological order: > > > > ====================================================> > 1. First, I update the systems, install ntp and then reboot: > > yum update > > yum install ntp > > systemctl status ntpd > systemctl start ntpd > systemctl enable ntpd > systemctl status ntpd > > > ====================================================> > > 2. I use a separate partition within the two VM storage nodes. > Each of the two nodes see the separate partition as /dev/sdb. > After creating a thinly provisioned LV on each of the two nodes, I > create an XFS filesystem on them: > > > pvcreate /dev/sdb1 > > vgcreate -s 32M vg_glusterfs /dev/sdb1 > > lvcreate -L 20G --thinpool glusterfs_thin_pool vg_glusterfs > > lvcreate -V 15G --thin -n glusterfs_thin_vol1 > vg_glusterfs/glusterfs_thin_pool > > mkfs.xfs -i size=512 /dev/vg_glusterfs/glusterfs_thin_vol1 > > > ====================================================> > > 3. On each node, I configured the brick data partition and added > the following to /etc/fstab: > > mkdir /data > > echo "/dev/vg_glusterfs/glusterfs_thin_vol1 /data xfs defaults 1 > 2" >> /etc/fstab > > mount -a > > > ====================================================> > 4. After mounting the volume, I see the following in df -Th, which > is correct: > > [root at gluster1 brick1]# df -Th > Filesystem Type Size Used > Avail Use% Mounted on > /dev/sda1 ext4 46G 1.4G > 42G 4% / > devtmpfs devtmpfs 3.9G 0 > 3.9G 0% /dev > tmpfs tmpfs 3.9G 0 > 3.9G 0% /dev/shm > tmpfs tmpfs 3.9G 8.6M > 3.9G 1% /run > tmpfs tmpfs 3.9G 0 > 3.9G 0% /sys/fs/cgroup > tmpfs tmpfs 783M 0 > 783M 0% /run/user/0 > /dev/mapper/vg_glusterfs-glusterfs_thin_vol1 xfs 15G 34M 15G > 1% /data > > > ====================================================> > 5. Create specific volume dirs on all storage nodes: > > > mkdir -pv /data/glusterfs/${HOSTNAME%%.*}/vol01 > > > ====================================================> > 6. Add entries in /etc/hosts: > > vim /etc/hosts > > 192.168.10.233 gluster1 > 192.168.10.234 gluster2 > 192.168.10.237 gluster-client > > ====================================================> > 7. Install gluster from the CentOS SIG: > > > yum search centos-release-gluster > yum install centos-release-gluster40 > > > yum install glusterfs-server > > ====================================================> > 8. Set up TLS/SSL encryption on all nodes and clients (gluster1, > gluster2, gluster-client): > > openssl genrsa -out /etc/ssl/glusterfs.key 2048 > > In gluster1 node: > openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj > "/CN=gluster1" -out /etc/ssl/glusterfs.pem > In gluster2 node: > openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj > "/CN=gluster2" -out /etc/ssl/glusterfs.pem > In gluster-client node: > openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj > "/CN=gluster-client" -out /etc/ssl/glusterfs.pem > > > ====================================================> > 9. On another box, I concatenate all of the .pem certificates into > a .ca file. > > Bring all .pem files locally: > > scp gluster1:/etc/ssl/glusterfs.pe <http://glusterfs.pe>m > gluster01.pem > scp gluster2:/etc/ssl/glusterfs.pe <http://glusterfs.pe>m > gluster02.pem > scp gluster-client:/etc/ssl/glusterfs.pem gluster-client.pem > > For storage nodes, I concatenate all .pem certificates (including > the client's .pem): > > cat gluster01.pem gluster02.pem gluster-client.pem > > glusterfs-nodes.ca <http://glusterfs-nodes.ca> > > For server clients: > > cat gluster01.pem gluster02.pem > glusterfs-client.ca > <http://glusterfs-client.ca> > > > ====================================================> > > 10. Put glusterfs-nodes.ca <http://glusterfs-nodes.ca> file on all > the storage nodes (this includes storage nodes .pem + client's .pem): > > scp glusterfs-nodes.ca <http://glusterfs-nodes.ca> > gluster1:/etc/ssl/glusterfs.ca <http://glusterfs.ca> > scp glusterfs-nodes.ca <http://glusterfs-nodes.ca> > gluster2:/etc/ssl/glusterfs.ca <http://glusterfs.ca> > > Put glusterfs-client.ca <http://glusterfs-client.ca> file on all > the storage nodes (this includes only storage nodes .pem): > > scp glusterfs-client.ca <http://glusterfs-client.ca> > gluster-client:/etc/ssl/glusterfs.ca <http://glusterfs.ca> > > > ====================================================> > > 11. Enable management encryption on each node (command ran on > gluster1, gluster2, gluster-client): > > > touch /var/lib/glusterd/secure-access > > > ====================================================> > > 12. Start, enable and check status of glusterd on gluster1 and > gluster2: > > systemctl start glusterd > systemctl enable glusterd > systemctl status glusterd > > ====================================================> > 13. Configure trusted storage pool (TSP): > > >From gluster1: > > > gluster peer status > > > gluster peer probe gluster2 > > ====================================================> > 14. Create the replicated gluster volume but don't start it: > > > gluster volume create vol01 replica 2 transport tcp > gluster1:/data/glusterfs/gluster1/vol01/brick1 > gluster2:/data/glusterfs/gluster2/vol01/brick1 > > ====================================================> > > 15. Setup SSL/TLS access to the volume: > > gluster volume set vol01 auth.ssl-allow > 'gluster01,gluster02,gluster-client' > > > gluster volume set vol01 client.ssl on > gluster volume set vol01 server.ssl on > > gluster volume set vol01 network.ping-timeout "5" > > gluster volume start vol01 > > > ====================================================> > > 16. Mount the volume on gluster-client: > > mount -t glusterfs gluster1:/vol01 /mnt > > > After mounting the volume, df -h has the correct output: > > [root at gluster-client mnt]# df -h > Filesystem Size Used Avail Use% Mounted on > /dev/sda1 48G 36G 9.3G 80% / > devtmpfs 1.9G 0 1.9G 0% /dev > tmpfs 1.9G 0 1.9G 0% /dev/shm > tmpfs 1.9G 8.5M 1.9G 1% /run > tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup > tmpfs 379M 0 379M 0% /run/user/0 > gluster1:/vol01 15G 187M 15G 2% /mnt > > > ====================================================> > 17. If I try to copy an archive, it throws an error after > transferring part of the file and the /mnt folder becomes > inaccessible for a while: > > [root at gluster-client mnt]# cp /root/mybee.tar.gz /mnt/ > cp: error writing ?/mnt/mybee.tar.gz?: Transport endpoint is not > connected > cp: failed to extend ?/mnt/mybee.tar.gz?: Transport endpoint is > not connected > cp: failed to close ?/mnt/mybee.tar.gz?: Transport endpoint is not > connected > [root at gluster-client mnt]# ll /mnt > ls: cannot open directory .: Transport endpoint is not connected > > > ====================================================> > This is the log from gluster-client for the mnt mountpoint (mnt.log): > > > [2018-06-29 07:52:03.869382] I [MSGID: 100030] > [glusterfsd.c:2625:main] 0-/usr/sbin/glusterfs: Started running > /usr/sbin/glusterfs version 4.0.2 (args: /usr/sbin/glusterfs > --process-name fuse --volfile-server=gluster1 --volfile-id=/vol01 > /mnt) > [2018-06-29 07:52:03.878709] I [socket.c:4470:socket_init] > 0-glusterfs: SSL support on the I/O path is ENABLED > [2018-06-29 07:52:03.878782] I [socket.c:4473:socket_init] > 0-glusterfs: SSL support for glusterd is ENABLED > [2018-06-29 07:52:03.878795] I [socket.c:4490:socket_init] > 0-glusterfs: using private polling thread > [2018-06-29 07:52:03.879108] E [socket.c:4541:socket_init] > 0-glusterfs: failed to open /etc/ssl/dhparam.pem, DH ciphers are > disabled > [2018-06-29 07:52:03.885718] I [MSGID: 101190] > [event-epoll.c:609:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 1 > [2018-06-29 07:52:03.901983] I [MSGID: 101190] > [event-epoll.c:609:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 2 > [2018-06-29 07:52:03.902092] I [socket.c:4470:socket_init] > 0-vol01-client-1: SSL support on the I/O path is ENABLED > [2018-06-29 07:52:03.902117] I [socket.c:4473:socket_init] > 0-vol01-client-1: SSL support for glusterd is ENABLED > [2018-06-29 07:52:03.902142] I [socket.c:4490:socket_init] > 0-vol01-client-1: using private polling thread > [2018-06-29 07:52:03.902266] E [socket.c:4541:socket_init] > 0-vol01-client-1: failed to open /etc/ssl/dhparam.pem, DH ciphers > are disabled > [2018-06-29 07:52:03.902977] I [socket.c:4470:socket_init] > 0-vol01-client-0: SSL support on the I/O path is ENABLED > [2018-06-29 07:52:03.903018] I [socket.c:4473:socket_init] > 0-vol01-client-0: SSL support for glusterd is ENABLED > [2018-06-29 07:52:03.903028] I [socket.c:4490:socket_init] > 0-vol01-client-0: using private polling thread > [2018-06-29 07:52:03.903161] E [socket.c:4541:socket_init] > 0-vol01-client-0: failed to open /etc/ssl/dhparam.pem, DH ciphers > are disabled > [2018-06-29 07:52:03.903756] I [MSGID: 114020] > [client.c:2300:notify] 0-vol01-client-0: parent translators are > ready, attempting connect on transport > [2018-06-29 07:52:03.908432] I [MSGID: 114020] > [client.c:2300:notify] 0-vol01-client-1: parent translators are > ready, attempting connect on transport > Final graph: > +------------------------------------------------------------------------------+ > 1: volume vol01-client-0 > 2: type protocol/client > 3: option ping-timeout 5 > 4: option remote-host gluster1 > 5: option remote-subvolume /data/glusterfs/gluster1/vol01/brick1 > 6: option transport-type socket > 7: option transport.address-family inet > 8: option transport.socket.ssl-enabled on > 9: option transport.tcp-user-timeout 0 > 10: option transport.socket.keepalive-time 20 > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-count 9 > 13: option send-gids true > 14: end-volume > 15: > 16: volume vol01-client-1 > 17: type protocol/client > 18: option ping-timeout 5 > 19: option remote-host gluster2 > 20: option remote-subvolume /data/glusterfs/gluster2/vol01/brick1 > 21: option transport-type socket > 22: option transport.address-family inet > 23: option transport.socket.ssl-enabled on > 24: option transport.tcp-user-timeout 0 > 25: option transport.socket.keepalive-time 20 > 26: option transport.socket.keepalive-interval 2 > 27: option transport.socket.keepalive-count 9 > 28: option send-gids true > 29: end-volume > 30: > 31: volume vol01-replicate-0 > 32: type cluster/replicate > 33: option afr-pending-xattr vol01-client-0,vol01-client-1 > 34: option use-compound-fops off > 35: subvolumes vol01-client-0 vol01-client-1 > 36: end-volume > 37: > 38: volume vol01-dht > 39: type cluster/distribute > 40: option lock-migration off > 41: option force-migration off > 42: subvolumes vol01-replicate-0 > 43: end-volume > 44: > 45: volume vol01-write-behind > 46: type performance/write-behind > 47: subvolumes vol01-dht > 48: end-volume > 49: > 50: volume vol01-read-ahead > 51: type performance/read-ahead > 52: subvolumes vol01-write-behind > 53: end-volume > 54: > 55: volume vol01-readdir-ahead > 56: type performance/readdir-ahead > 57: option parallel-readdir off > 58: option rda-request-size 131072 > 59: option rda-cache-limit 10MB > 60: subvolumes vol01-read-ahead > 61: end-volume > 62: > 63: volume vol01-io-cache > 64: type performance/io-cache > 65: subvolumes vol01-readdir-ahead > 66: end-volume > 67: > 68: volume vol01-quick-read > 69: type performance/quick-read > 70: subvolumes vol01-io-cache > 71: end-volume > 72: > 73: volume vol01-open-behind > 74: type performance/open-behind > 75: subvolumes vol01-quick-read > 76: end-volume > 77: > 78: volume vol01-md-cache > 79: type performance/md-cache > 80: subvolumes vol01-open-behind > 81: end-volume > 82: > 83: volume vol01 > 84: type debug/io-stats > 85: option log-level INFO > 86: option latency-measurement off > 87: option count-fop-hits off > 88: subvolumes vol01-md-cache > 89: end-volume > 90: > 91: volume meta-autoload > 92: type meta > 93: subvolumes vol01 > 94: end-volume > 95: > +------------------------------------------------------------------------------+ > [2018-06-29 07:52:03.917518] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:03.918459] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:03.919289] I [rpc-clnt.c:2071:rpc_clnt_reconfig] > 0-vol01-client-0: changing port to 49152 (from 0) > [2018-06-29 07:52:03.922903] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:03.923922] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:03.925115] I [rpc-clnt.c:2071:rpc_clnt_reconfig] > 0-vol01-client-1: changing port to 49152 (from 0) > [2018-06-29 07:52:03.930023] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:03.930756] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:03.931859] I [MSGID: 114046] > [client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-0: > Connected to vol01-client-0, attached to remote volume > '/data/glusterfs/gluster1/vol01/brick1'. > [2018-06-29 07:52:03.931894] I [MSGID: 108005] > [afr-common.c:5081:__afr_handle_child_up_event] > 0-vol01-replicate-0: Subvolume 'vol01-client-0' came back up; > going online. > [2018-06-29 07:52:03.938903] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:03.939803] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:03.941300] I [MSGID: 114046] > [client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-1: > Connected to vol01-client-1, attached to remote volume > '/data/glusterfs/gluster2/vol01/brick1'. > [2018-06-29 07:52:03.942884] I [fuse-bridge.c:4234:fuse_init] > 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs > 7.24 kernel 7.22 > [2018-06-29 07:52:03.942907] I > [fuse-bridge.c:4864:fuse_graph_sync] 0-fuse: switched to graph 0 > [2018-06-29 07:52:03.946767] W [socket.c:592:__socket_rwv] > 0-vol01-client-0: readv on 192.168.10.233:49152 > <http://192.168.10.233:49152> failed (Input/output error) > [2018-06-29 07:52:03.946791] E [socket.c:2785:socket_poller] > 0-vol01-client-0: socket_poller 192.168.10.233:49152 > <http://192.168.10.233:49152> failed (Input/output error) > [2018-06-29 07:52:03.946837] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:03.947264] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) > op(LOOKUP(27)) called at 2018-06-29 07:52:03.945958 (xid=0xb) > [2018-06-29 07:52:03.947297] W [MSGID: 114031] > [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-vol01-client-0: > remote operation failed. Path: / > (00000000-0000-0000-0000-000000000001) [Transport endpoint is not > connected] > [2018-06-29 07:52:03.947455] I [MSGID: 109005] > [dht-selfheal.c:2328:dht_selfheal_directory] 0-vol01-dht: > Directory selfheal failed: Unable to form layout for directory / > [2018-06-29 07:52:07.885389] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:07.886457] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:07.887452] I [rpc-clnt.c:2071:rpc_clnt_reconfig] > 0-vol01-client-0: changing port to 49152 (from 0) > [2018-06-29 07:52:07.900627] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:07.901594] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:07.903220] I [MSGID: 114046] > [client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-0: > Connected to vol01-client-0, attached to remote volume > '/data/glusterfs/gluster1/vol01/brick1'. > [2018-06-29 07:52:13.062404] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:13.062780] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) > op(WRITE(13)) called at 2018-06-29 07:52:13.048761 (xid=0x22) > [2018-06-29 07:52:13.062826] W [MSGID: 114031] > [client-rpc-fops_v2.c:658:client4_0_writev_cbk] 0-vol01-client-0: > remote operation failed [Transport endpoint is not connected] > [2018-06-29 07:52:13.063017] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) > op(WRITE(13)) called at 2018-06-29 07:52:13.049162 (xid=0x23) > [2018-06-29 07:52:13.063175] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) > op(WRITE(13)) called at 2018-06-29 07:52:13.049865 (xid=0x24) > [2018-06-29 07:52:13.063319] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) > op(WRITE(13)) called at 2018-06-29 07:52:13.050199 (xid=0x25) > [2018-06-29 07:52:13.063473] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) > op(WRITE(13)) called at 2018-06-29 07:52:13.050489 (xid=0x26) > [2018-06-29 07:52:13.063600] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) > op(WRITE(13)) called at 2018-06-29 07:52:13.050757 (xid=0x27) > [2018-06-29 07:52:13.063745] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) > op(WRITE(13)) called at 2018-06-29 07:52:13.051031 (xid=0x28) > [2018-06-29 07:52:13.063883] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) > op(WRITE(13)) called at 2018-06-29 07:52:13.053035 (xid=0x29) > The message "W [MSGID: 114031] > [client-rpc-fops_v2.c:658:client4_0_writev_cbk] 0-vol01-client-0: > remote operation failed [Transport endpoint is not connected]" > repeated 7 times between [2018-06-29 07:52:13.062826] and > [2018-06-29 07:52:13.063903] > [2018-06-29 07:52:13.104580] W [MSGID: 114031] > [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] > 0-vol01-client-0: remote operation failed > [2018-06-29 07:52:13.109230] W [socket.c:592:__socket_rwv] > 0-vol01-client-1: writev on 192.168.10.234:49152 > <http://192.168.10.234:49152> failed (No data available) > [2018-06-29 07:52:13.109282] E [socket.c:2777:socket_poller] > 0-vol01-client-1: poll error on socket > [2018-06-29 07:52:13.109340] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected > from vol01-client-1. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:13.109357] E [MSGID: 108006] > [afr-common.c:5158:__afr_handle_child_down_event] > 0-vol01-replicate-0: All subvolumes are down. Going offline until > atleast one of them comes back up. > [2018-06-29 07:52:13.109461] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:13.109679] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) > op(FSTAT(25)) called at 2018-06-29 07:52:13.104513 (xid=0x42) > [2018-06-29 07:52:13.109700] W [MSGID: 114031] > [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] 0-vol01-client-1: > remote operation failed [Transport endpoint is not connected] > [2018-06-29 07:52:13.109854] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) > op(FXATTROP(34)) called at 2018-06-29 07:52:13.104729 (xid=0x43) > [2018-06-29 07:52:13.109877] W [MSGID: 114031] > [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] > 0-vol01-client-1: remote operation failed > [2018-06-29 07:52:13.109897] E [MSGID: 114031] > [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] > 0-vol01-client-0: remote operation failed [Transport endpoint is > not connected] > [2018-06-29 07:52:13.110036] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) > op(WRITE(13)) called at 2018-06-29 07:52:13.105623 (xid=0x44) > [2018-06-29 07:52:13.110053] W [MSGID: 114031] > [client-rpc-fops_v2.c:658:client4_0_writev_cbk] 0-vol01-client-1: > remote operation failed [Transport endpoint is not connected] > [2018-06-29 07:52:13.110232] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) > op(FSTAT(25)) called at 2018-06-29 07:52:13.105658 (xid=0x45) > [2018-06-29 07:52:13.110250] W [MSGID: 114031] > [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] 0-vol01-client-1: > remote operation failed [Transport endpoint is not connected] > [2018-06-29 07:52:13.110386] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) > op(FSTAT(25)) called at 2018-06-29 07:52:13.106862 (xid=0x46) > [2018-06-29 07:52:13.110530] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) > op(FSTAT(25)) called at 2018-06-29 07:52:13.108103 (xid=0x47) > [2018-06-29 07:52:13.110660] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) > op(FSTAT(25)) called at 2018-06-29 07:52:13.108969 (xid=0x48) > [2018-06-29 07:52:13.110793] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) > op(FXATTROP(34)) called at 2018-06-29 07:52:13.109256 (xid=0x49) > The message "W [MSGID: 114031] > [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] 0-vol01-client-1: > remote operation failed [Transport endpoint is not connected]" > repeated 3 times between [2018-06-29 07:52:13.110250] and > [2018-06-29 07:52:13.110675] > [2018-06-29 07:52:13.110808] W [MSGID: 114031] > [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] > 0-vol01-client-1: remote operation failed > [2018-06-29 07:52:13.110933] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) > op(FXATTROP(34)) called at 2018-06-29 07:52:13.109284 (xid=0x4a) > The message "W [MSGID: 114031] > [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] > 0-vol01-client-1: remote operation failed" repeated 3 times > between [2018-06-29 07:52:13.110808] and [2018-06-29 07:52:14.878874] > [2018-06-29 07:52:14.878895] E [MSGID: 114031] > [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] > 0-vol01-client-1: remote operation failed [Transport endpoint is > not connected] > [2018-06-29 07:52:14.879095] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:14.879217] W > [fuse-bridge.c:2427:fuse_writev_cbk] 0-glusterfs-fuse: 76: WRITE > => -1 gfid=a8848be6-1c05-4a72-88a7-5209eeef7c2c fd=0x7f7c50023a68 > (Transport endpoint is not connected) > [2018-06-29 07:52:14.880509] W [fuse-bridge.c:1406:fuse_err_cbk] > 0-glusterfs-fuse: 77: FLUSH() ERR => -1 (Transport endpoint is not > connected) > The message "I [MSGID: 108006] [afr-common.c:5444:afr_local_init] > 0-vol01-replicate-0: no subvolumes up" repeated 11 times between > [2018-06-29 07:52:14.879095] and [2018-06-29 07:52:17.908056] > [2018-06-29 07:52:17.908066] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null > [2018-06-29 07:52:17.908175] W [fuse-bridge.c:896:fuse_attr_cbk] > 0-glusterfs-fuse: 79: LOOKUP() / => -1 (Transport endpoint is not > connected) > [2018-06-29 07:52:18.758975] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:18.759155] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:18.759165] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null > [2018-06-29 07:52:18.759214] W [fuse-bridge.c:896:fuse_attr_cbk] > 0-glusterfs-fuse: 80: LOOKUP() / => -1 (Transport endpoint is not > connected) > [2018-06-29 07:52:21.662626] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:21.662756] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:21.662765] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null > [2018-06-29 07:52:21.662811] W [fuse-bridge.c:896:fuse_attr_cbk] > 0-glusterfs-fuse: 81: LOOKUP() / => -1 (Transport endpoint is not > connected) > [2018-06-29 07:52:21.663567] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:21.663630] W > [fuse-bridge.c:3205:fuse_statfs_cbk] 0-glusterfs-fuse: 82: ERR => > -1 (Transport endpoint is not connected) > [2018-06-29 07:52:23.896362] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:23.896687] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:23.897321] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:23.896427 (xid=0x2a) > [2018-06-29 07:52:23.897353] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-0: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:23.903241] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:23.903566] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected > from vol01-client-1. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:23.904166] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:23.903281 (xid=0x4b) > [2018-06-29 07:52:23.904199] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-1: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:26.909819] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:26.910175] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:26.910858] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:26.909871 (xid=0x2b) > [2018-06-29 07:52:26.910888] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-0: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:26.913921] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:26.914163] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected > from vol01-client-1. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:26.914417] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:26.913961 (xid=0x4c) > [2018-06-29 07:52:26.914443] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-1: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:29.921628] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:29.921867] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:29.922215] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:29.921674 (xid=0x2c) > [2018-06-29 07:52:29.922257] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-0: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:29.926462] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:29.926774] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected > from vol01-client-1. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:29.927415] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:29.926499 (xid=0x4d) > [2018-06-29 07:52:29.927447] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-1: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:32.935218] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:32.935553] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:32.936196] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:32.935271 (xid=0x2d) > [2018-06-29 07:52:32.936230] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-0: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:32.938940] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:32.939232] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected > from vol01-client-1. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:32.939497] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:32.938982 (xid=0x4e) > [2018-06-29 07:52:32.939526] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-1: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:35.948789] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:35.949172] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:35.949860] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:35.948839 (xid=0x2e) > [2018-06-29 07:52:35.949901] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-0: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:35.952154] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:35.952408] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected > from vol01-client-1. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:35.952670] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:35.952199 (xid=0x4f) > [2018-06-29 07:52:35.952699] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-1: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:38.959369] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:38.959733] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:38.960536] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:38.959426 (xid=0x2f) > [2018-06-29 07:52:38.960573] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-0: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:38.963381] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:38.963618] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected > from vol01-client-1. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:38.963877] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:38.963428 (xid=0x50) > [2018-06-29 07:52:38.963906] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-1: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:41.969152] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:41.969397] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:41.969731] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:41.969204 (xid=0x30) > [2018-06-29 07:52:41.969769] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-0: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:41.974130] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:41.974381] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected > from vol01-client-1. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:41.974669] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:41.974198 (xid=0x51) > [2018-06-29 07:52:41.974698] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-1: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:44.984465] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:44.984781] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:44.985444] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:44.984514 (xid=0x31) > [2018-06-29 07:52:44.985477] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-0: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:44.990039] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:44.990352] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected > from vol01-client-1. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:44.990729] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:44.990094 (xid=0x52) > [2018-06-29 07:52:44.990771] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-1: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:47.994448] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:47.994684] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:47.994947] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:47.994499 (xid=0x32) > [2018-06-29 07:52:47.994984] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-0: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:48.002544] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:48.002809] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected > from vol01-client-1. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:48.003388] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:48.002577 (xid=0x53) > [2018-06-29 07:52:48.003416] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-1: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:49.440101] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:49.440257] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:49.440266] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null > [2018-06-29 07:52:49.440310] W [fuse-bridge.c:896:fuse_attr_cbk] > 0-glusterfs-fuse: 83: LOOKUP() / => -1 (Transport endpoint is not > connected) > [2018-06-29 07:52:50.182948] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:50.183108] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:50.183116] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null > [2018-06-29 07:52:50.197572] W [fuse-bridge.c:896:fuse_attr_cbk] > 0-glusterfs-fuse: 84: LOOKUP() / => -1 (Transport endpoint is not > connected) > [2018-06-29 07:52:50.575887] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:50.576030] I [MSGID: 108006] > [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no > subvolumes up > [2018-06-29 07:52:50.576039] E [MSGID: 101046] > [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null > [2018-06-29 07:52:50.576087] W [fuse-bridge.c:896:fuse_attr_cbk] > 0-glusterfs-fuse: 85: LOOKUP() / => -1 (Transport endpoint is not > connected) > [2018-06-29 07:52:51.012626] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-0: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:51.012924] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected > from vol01-client-0. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:51.013615] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:51.012667 (xid=0x33) > [2018-06-29 07:52:51.013647] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-0: received RPC status error [Transport endpoint is > not connected] > [2018-06-29 07:52:51.018855] W [rpc-clnt.c:1739:rpc_clnt_submit] > 0-vol01-client-1: error returned while attempting to connect to > host:(null), port:0 > [2018-06-29 07:52:51.019093] I [MSGID: 114018] > [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected > from vol01-client-1. Client process will keep trying to connect to > glusterd until brick's port is available > [2018-06-29 07:52:51.019378] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] > (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> > /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] > (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) > 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) > called at 2018-06-29 07:52:51.018893 (xid=0x54) > [2018-06-29 07:52:51.019408] W [MSGID: 114032] > [client-handshake.c:1620:client_dump_version_cbk] > 0-vol01-client-1: received RPC status error [Transport endpoint is > not connected] > > > > ====================================================> > After trying to copy the archive, I end up with different sized > files on the two bricks: > > [root at gluster1 brick1]# pwd > /data/glusterfs/gluster1/vol01/brick1 > [root at gluster1 brick1]# ll > total 384 > -rw------- 2 root root 393216 Jun 29 10:52 mybee.tar.gz > > > > [root at gluster2 brick1]# pwd > /data/glusterfs/gluster2/vol01/brick1 > [root at gluster2 brick1]# ll > total 1280 > -rw------- 2 root root 1310720 Jun 29 10:52 mybee.tar.gz > > > ====================================================> > These are the informations regarding the gluster volume: > > [root at gluster1 glusterfs]# gluster volume info vol01 > > Volume Name: vol01 > Type: Replicate > Volume ID: d3b39520-3fbb-4cfc-b736-338d7726b5a1 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: gluster1:/data/glusterfs/gluster1/vol01/brick1 > Brick2: gluster2:/data/glusterfs/gluster2/vol01/brick1 > Options Reconfigured: > network.ping-timeout: 5 > server.ssl: on > client.ssl: on > auth.ssl-allow: gluster01,gluster02,gluster-client > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: off > > ====================================================> > Here is the peers information: > > [root at gluster1 brick1]# gluster peer status > Number of Peers: 1 > > Hostname: gluster2 > Uuid: 9ccac8e3-71c1-43e4-b723-fb7374be8a1d > State: Peer in Cluster (Connected) > > > ====================================================> > Here is the volume status: > > [root at gluster1 brick1]# gluster volume status vol01 > Status of volume: vol01 > Gluster process TCP Port RDMA Port > Online Pid > ------------------------------------------------------------------------------ > Brick gluster1:/data/glusterfs/gluster1/vol > 01/brick1 49152 0 Y 11108 > Brick gluster2:/data/glusterfs/gluster2/vol > 01/brick1 49152 0 Y 10942 > Self-heal Daemon on localhost N/A N/A Y 11133 > Self-heal Daemon on gluster2 N/A N/A Y 10968 > > Task Status of Volume vol01 > ------------------------------------------------------------------------------ > There are no active volume tasks > > > ====================================================> > > These are the installed packages on gluster-client: > > [root at gluster-client glusterfs]# rpm -qa | grep "gluster\|fuse" > centos-release-gluster40-1.0-2.el7.centos.x86_64 > glusterfs-client-xlators-4.0.2-1.el7.x86_64 > fuse-2.9.2-10.el7.x86_64 > glusterfs-4.0.2-1.el7.x86_64 > glusterfs-fuse-4.0.2-1.el7.x86_64 > fuse-libs-2.9.2-10.el7.x86_64 > glusterfs-libs-4.0.2-1.el7.x86_64 > > > > And these are the installed packages on gluster1 and gluster2 > storage nodes: > > > [root at gluster1 glusterfs]# rpm -qa | grep "gluster\|fuse" > glusterfs-api-4.0.2-1.el7.x86_64 > centos-release-gluster40-1.0-2.el7.centos.x86_64 > glusterfs-libs-4.0.2-1.el7.x86_64 > glusterfs-client-xlators-4.0.2-1.el7.x86_64 > glusterfs-cli-4.0.2-1.el7.x86_64 > glusterfs-fuse-4.0.2-1.el7.x86_64 > glusterfs-server-4.0.2-1.el7.x86_64 > glusterfs-4.0.2-1.el7.x86_64 > > > > > Can anyone point me in the right direction? What am I doing wrong? > The mount works fine until I try to copy an archive, multiple > smaller files or a bigger file on it (meaning it shows correctly > in df -Th and I can create several files with "touch file1 > file2..."). Basically, after any data transfer, I get these errors. > > > I followed the indications from the redhat page: > > https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/chap-network_encryption > <https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/chap-network_encryption> > > UPDATE 1: > I tried doing the exact same steps in Gluster 3.12 and had no > problem. The steps worked and SSL/TLS was enabled. There was no > transport error or anything and I also checked if SSL/TLS was > enabled. Afterwards, I also tried with the new release 4.1 and the > problem persists (same error with "Transport endpoint is not > connected"). > Let me know if you need any other info. Any help is much appreciated. > > Thank you. > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > https://lists.gluster.org/mailman/listinfo/gluster-users > <https://lists.gluster.org/mailman/listinfo/gluster-users> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180713/0e686085/attachment.html>
David Spisla
2018-Jul-13 10:14 UTC
[Gluster-users] Problem with SSL/TLS encryption on Gluster 4.0 & 4.1
Hello Andrei, I did some research. But I didn't find a bug report for this issue. If you want you can create such a report here: https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS It is the recommended way in GlusterFS to report a bug, github issues are for new features Regards David 2018-07-13 11:44 GMT+02:00 Havriliuc Andrei <andrei at hostvision.ro>:> Hello David, > > > Yeah. I also disabled SSL/TLS for management and communication for > Gluster. The problem is that I will have traffic over public WAN for a geo > replication instance and I cannot go forward until this is solved. > > > Was there a bug filed about this? I've looked at the issues on the > Glusterfs github page but I didn't find anything related to this: > > > https://github.com/gluster/glusterfs/issues > > > Regards, > > Andrei > > On 7/12/18 4:03 PM, David Spisla wrote: > > Hello Andrei, > > I am also using Gluster 4.1 on CentOS and I have the same problem. I > tested it with a volume which had no network encryption and one with > network encryption. You are not the only one. > It seems to be a bug. At the moment there is no other choice to disable > client.ssl and sever.ssl on a volume to have stable I/O. > > Regards > David > > > 2018-07-12 10:45 GMT+02:00 Havriliuc Andrei <andrei at hostvision.ro>: > >> Hello, >> >> I am doing some tests with GlusterFS 4.0 and I can't seem to solve some >> SSL/TLS issues. I am trying to set up a 2 node replicated gluster volume >> with SSL/TLS. For this setup, I use 3 KVM VMs (2 storage nodes + 1 client >> node). For the networking part, I use a dedicated private LAN for the KVM >> VMs. Each VM is able to ping the other, so there's no problem with the >> connectivity. >> >> To try to make the procedure I used as clear as possible, I will put all >> commands in chronological order: >> >> >> >> ====================================================>> >> 1. First, I update the systems, install ntp and then reboot: >> >> yum update >> >> yum install ntp >> >> systemctl status ntpd >> systemctl start ntpd >> systemctl enable ntpd >> systemctl status ntpd >> >> >> ====================================================>> >> >> 2. I use a separate partition within the two VM storage nodes. Each of >> the two nodes see the separate partition as /dev/sdb. After creating a >> thinly provisioned LV on each of the two nodes, I create an XFS filesystem >> on them: >> >> >> pvcreate /dev/sdb1 >> >> vgcreate -s 32M vg_glusterfs /dev/sdb1 >> >> lvcreate -L 20G --thinpool glusterfs_thin_pool vg_glusterfs >> >> lvcreate -V 15G --thin -n glusterfs_thin_vol1 >> vg_glusterfs/glusterfs_thin_pool >> >> mkfs.xfs -i size=512 /dev/vg_glusterfs/glusterfs_thin_vol1 >> >> >> ====================================================>> >> >> 3. On each node, I configured the brick data partition and added the >> following to /etc/fstab: >> >> mkdir /data >> >> echo "/dev/vg_glusterfs/glusterfs_thin_vol1 /data xfs defaults 1 2" >> >> /etc/fstab >> >> mount -a >> >> >> ====================================================>> >> 4. After mounting the volume, I see the following in df -Th, which is >> correct: >> >> [root at gluster1 brick1]# df -Th >> Filesystem Type Size Used Avail >> Use% Mounted on >> /dev/sda1 ext4 46G 1.4G 42G >> 4% / >> devtmpfs devtmpfs 3.9G 0 3.9G >> 0% /dev >> tmpfs tmpfs 3.9G 0 3.9G >> 0% /dev/shm >> tmpfs tmpfs 3.9G 8.6M 3.9G >> 1% /run >> tmpfs tmpfs 3.9G 0 3.9G >> 0% /sys/fs/cgroup >> tmpfs tmpfs 783M 0 783M >> 0% /run/user/0 >> /dev/mapper/vg_glusterfs-glusterfs_thin_vol1 xfs 15G 34M 15G >> 1% /data >> >> >> ====================================================>> >> 5. Create specific volume dirs on all storage nodes: >> >> >> mkdir -pv /data/glusterfs/${HOSTNAME%%.*}/vol01 >> >> >> ====================================================>> >> 6. Add entries in /etc/hosts: >> >> vim /etc/hosts >> >> 192.168.10.233 gluster1 >> 192.168.10.234 gluster2 >> 192.168.10.237 gluster-client >> >> ====================================================>> >> 7. Install gluster from the CentOS SIG: >> >> >> yum search centos-release-gluster >> yum install centos-release-gluster40 >> >> >> yum install glusterfs-server >> >> ====================================================>> >> 8. Set up TLS/SSL encryption on all nodes and clients (gluster1, >> gluster2, gluster-client): >> >> openssl genrsa -out /etc/ssl/glusterfs.key 2048 >> >> In gluster1 node: >> openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=gluster1" >> -out /etc/ssl/glusterfs.pem >> In gluster2 node: >> openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=gluster2" >> -out /etc/ssl/glusterfs.pem >> In gluster-client node: >> openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj >> "/CN=gluster-client" -out /etc/ssl/glusterfs.pem >> >> >> ====================================================>> >> 9. On another box, I concatenate all of the .pem certificates into a .ca >> file. >> >> Bring all .pem files locally: >> >> scp gluster1:/etc/ssl/glusterfs.pem gluster01.pem >> scp gluster2:/etc/ssl/glusterfs.pem gluster02.pem >> scp gluster-client:/etc/ssl/glusterfs.pem gluster-client.pem >> >> For storage nodes, I concatenate all .pem certificates (including the >> client's .pem): >> >> cat gluster01.pem gluster02.pem gluster-client.pem > glusterfs-nodes.ca >> >> For server clients: >> >> cat gluster01.pem gluster02.pem > glusterfs-client.ca >> >> >> ====================================================>> >> >> 10. Put glusterfs-nodes.ca file on all the storage nodes (this includes >> storage nodes .pem + client's .pem): >> >> scp glusterfs-nodes.ca gluster1:/etc/ssl/glusterfs.ca >> scp glusterfs-nodes.ca gluster2:/etc/ssl/glusterfs.ca >> >> Put glusterfs-client.ca file on all the storage nodes (this includes >> only storage nodes .pem): >> >> scp glusterfs-client.ca gluster-client:/etc/ssl/glusterfs.ca >> >> >> ====================================================>> >> >> 11. Enable management encryption on each node (command ran on gluster1, >> gluster2, gluster-client): >> >> >> touch /var/lib/glusterd/secure-access >> >> >> ====================================================>> >> >> 12. Start, enable and check status of glusterd on gluster1 and gluster2: >> >> systemctl start glusterd >> systemctl enable glusterd >> systemctl status glusterd >> >> ====================================================>> >> 13. Configure trusted storage pool (TSP): >> >> >From gluster1: >> >> >> gluster peer status >> >> >> gluster peer probe gluster2 >> >> ====================================================>> >> 14. Create the replicated gluster volume but don't start it: >> >> >> gluster volume create vol01 replica 2 transport tcp >> gluster1:/data/glusterfs/gluster1/vol01/brick1 >> gluster2:/data/glusterfs/gluster2/vol01/brick1 >> >> ====================================================>> >> >> 15. Setup SSL/TLS access to the volume: >> >> gluster volume set vol01 auth.ssl-allow 'gluster01,gluster02,gluster-c >> lient' >> >> >> gluster volume set vol01 client.ssl on >> gluster volume set vol01 server.ssl on >> >> gluster volume set vol01 network.ping-timeout "5" >> >> gluster volume start vol01 >> >> >> ====================================================>> >> >> 16. Mount the volume on gluster-client: >> >> mount -t glusterfs gluster1:/vol01 /mnt >> >> >> After mounting the volume, df -h has the correct output: >> >> [root at gluster-client mnt]# df -h >> Filesystem Size Used Avail Use% Mounted on >> /dev/sda1 48G 36G 9.3G 80% / >> devtmpfs 1.9G 0 1.9G 0% /dev >> tmpfs 1.9G 0 1.9G 0% /dev/shm >> tmpfs 1.9G 8.5M 1.9G 1% /run >> tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup >> tmpfs 379M 0 379M 0% /run/user/0 >> gluster1:/vol01 15G 187M 15G 2% /mnt >> >> >> ====================================================>> >> 17. If I try to copy an archive, it throws an error after transferring >> part of the file and the /mnt folder becomes inaccessible for a while: >> >> [root at gluster-client mnt]# cp /root/mybee.tar.gz /mnt/ >> cp: error writing ?/mnt/mybee.tar.gz?: Transport endpoint is not connected >> cp: failed to extend ?/mnt/mybee.tar.gz?: Transport endpoint is not >> connected >> cp: failed to close ?/mnt/mybee.tar.gz?: Transport endpoint is not >> connected >> [root at gluster-client mnt]# ll /mnt >> ls: cannot open directory .: Transport endpoint is not connected >> >> >> ====================================================>> >> This is the log from gluster-client for the mnt mountpoint (mnt.log): >> >> >> [2018-06-29 07:52:03.869382] I [MSGID: 100030] [glusterfsd.c:2625:main] >> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.0.2 >> (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=gluster1 >> --volfile-id=/vol01 /mnt) >> [2018-06-29 07:52:03.878709] I [socket.c:4470:socket_init] 0-glusterfs: >> SSL support on the I/O path is ENABLED >> [2018-06-29 07:52:03.878782] I [socket.c:4473:socket_init] 0-glusterfs: >> SSL support for glusterd is ENABLED >> [2018-06-29 07:52:03.878795] I [socket.c:4490:socket_init] 0-glusterfs: >> using private polling thread >> [2018-06-29 07:52:03.879108] E [socket.c:4541:socket_init] 0-glusterfs: >> failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled >> [2018-06-29 07:52:03.885718] I [MSGID: 101190] >> [event-epoll.c:609:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 1 >> [2018-06-29 07:52:03.901983] I [MSGID: 101190] >> [event-epoll.c:609:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 2 >> [2018-06-29 07:52:03.902092] I [socket.c:4470:socket_init] >> 0-vol01-client-1: SSL support on the I/O path is ENABLED >> [2018-06-29 07:52:03.902117] I [socket.c:4473:socket_init] >> 0-vol01-client-1: SSL support for glusterd is ENABLED >> [2018-06-29 07:52:03.902142] I [socket.c:4490:socket_init] >> 0-vol01-client-1: using private polling thread >> [2018-06-29 07:52:03.902266] E [socket.c:4541:socket_init] >> 0-vol01-client-1: failed to open /etc/ssl/dhparam.pem, DH ciphers are >> disabled >> [2018-06-29 07:52:03.902977] I [socket.c:4470:socket_init] >> 0-vol01-client-0: SSL support on the I/O path is ENABLED >> [2018-06-29 07:52:03.903018] I [socket.c:4473:socket_init] >> 0-vol01-client-0: SSL support for glusterd is ENABLED >> [2018-06-29 07:52:03.903028] I [socket.c:4490:socket_init] >> 0-vol01-client-0: using private polling thread >> [2018-06-29 07:52:03.903161] E [socket.c:4541:socket_init] >> 0-vol01-client-0: failed to open /etc/ssl/dhparam.pem, DH ciphers are >> disabled >> [2018-06-29 07:52:03.903756] I [MSGID: 114020] [client.c:2300:notify] >> 0-vol01-client-0: parent translators are ready, attempting connect on >> transport >> [2018-06-29 07:52:03.908432] I [MSGID: 114020] [client.c:2300:notify] >> 0-vol01-client-1: parent translators are ready, attempting connect on >> transport >> Final graph: >> +----------------------------------------------------------- >> -------------------+ >> 1: volume vol01-client-0 >> 2: type protocol/client >> 3: option ping-timeout 5 >> 4: option remote-host gluster1 >> 5: option remote-subvolume /data/glusterfs/gluster1/vol01/brick1 >> 6: option transport-type socket >> 7: option transport.address-family inet >> 8: option transport.socket.ssl-enabled on >> 9: option transport.tcp-user-timeout 0 >> 10: option transport.socket.keepalive-time 20 >> 11: option transport.socket.keepalive-interval 2 >> 12: option transport.socket.keepalive-count 9 >> 13: option send-gids true >> 14: end-volume >> 15: >> 16: volume vol01-client-1 >> 17: type protocol/client >> 18: option ping-timeout 5 >> 19: option remote-host gluster2 >> 20: option remote-subvolume /data/glusterfs/gluster2/vol01/brick1 >> 21: option transport-type socket >> 22: option transport.address-family inet >> 23: option transport.socket.ssl-enabled on >> 24: option transport.tcp-user-timeout 0 >> 25: option transport.socket.keepalive-time 20 >> 26: option transport.socket.keepalive-interval 2 >> 27: option transport.socket.keepalive-count 9 >> 28: option send-gids true >> 29: end-volume >> 30: >> 31: volume vol01-replicate-0 >> 32: type cluster/replicate >> 33: option afr-pending-xattr vol01-client-0,vol01-client-1 >> 34: option use-compound-fops off >> 35: subvolumes vol01-client-0 vol01-client-1 >> 36: end-volume >> 37: >> 38: volume vol01-dht >> 39: type cluster/distribute >> 40: option lock-migration off >> 41: option force-migration off >> 42: subvolumes vol01-replicate-0 >> 43: end-volume >> 44: >> 45: volume vol01-write-behind >> 46: type performance/write-behind >> 47: subvolumes vol01-dht >> 48: end-volume >> 49: >> 50: volume vol01-read-ahead >> 51: type performance/read-ahead >> 52: subvolumes vol01-write-behind >> 53: end-volume >> 54: >> 55: volume vol01-readdir-ahead >> 56: type performance/readdir-ahead >> 57: option parallel-readdir off >> 58: option rda-request-size 131072 >> 59: option rda-cache-limit 10MB >> 60: subvolumes vol01-read-ahead >> 61: end-volume >> 62: >> 63: volume vol01-io-cache >> 64: type performance/io-cache >> 65: subvolumes vol01-readdir-ahead >> 66: end-volume >> 67: >> 68: volume vol01-quick-read >> 69: type performance/quick-read >> 70: subvolumes vol01-io-cache >> 71: end-volume >> 72: >> 73: volume vol01-open-behind >> 74: type performance/open-behind >> 75: subvolumes vol01-quick-read >> 76: end-volume >> 77: >> 78: volume vol01-md-cache >> 79: type performance/md-cache >> 80: subvolumes vol01-open-behind >> 81: end-volume >> 82: >> 83: volume vol01 >> 84: type debug/io-stats >> 85: option log-level INFO >> 86: option latency-measurement off >> 87: option count-fop-hits off >> 88: subvolumes vol01-md-cache >> 89: end-volume >> 90: >> 91: volume meta-autoload >> 92: type meta >> 93: subvolumes vol01 >> 94: end-volume >> 95: >> +----------------------------------------------------------- >> -------------------+ >> [2018-06-29 07:52:03.917518] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:03.918459] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:03.919289] I [rpc-clnt.c:2071:rpc_clnt_reconfig] >> 0-vol01-client-0: changing port to 49152 (from 0) >> [2018-06-29 07:52:03.922903] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:03.923922] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:03.925115] I [rpc-clnt.c:2071:rpc_clnt_reconfig] >> 0-vol01-client-1: changing port to 49152 (from 0) >> [2018-06-29 07:52:03.930023] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:03.930756] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:03.931859] I [MSGID: 114046] >> [client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-0: >> Connected to vol01-client-0, attached to remote volume >> '/data/glusterfs/gluster1/vol01/brick1'. >> [2018-06-29 07:52:03.931894] I [MSGID: 108005] >> [afr-common.c:5081:__afr_handle_child_up_event] 0-vol01-replicate-0: >> Subvolume 'vol01-client-0' came back up; going online. >> [2018-06-29 07:52:03.938903] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:03.939803] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:03.941300] I [MSGID: 114046] >> [client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-1: >> Connected to vol01-client-1, attached to remote volume >> '/data/glusterfs/gluster2/vol01/brick1'. >> [2018-06-29 07:52:03.942884] I [fuse-bridge.c:4234:fuse_init] >> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel >> 7.22 >> [2018-06-29 07:52:03.942907] I [fuse-bridge.c:4864:fuse_graph_sync] >> 0-fuse: switched to graph 0 >> [2018-06-29 07:52:03.946767] W [socket.c:592:__socket_rwv] >> 0-vol01-client-0: readv on 192.168.10.233:49152 failed (Input/output >> error) >> [2018-06-29 07:52:03.946791] E [socket.c:2785:socket_poller] >> 0-vol01-client-0: socket_poller 192.168.10.233:49152 failed >> (Input/output error) >> [2018-06-29 07:52:03.946837] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:03.947264] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) >> op(LOOKUP(27)) called at 2018-06-29 07:52:03.945958 (xid=0xb) >> [2018-06-29 07:52:03.947297] W [MSGID: 114031] >> [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-vol01-client-0: >> remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) >> [Transport endpoint is not connected] >> [2018-06-29 07:52:03.947455] I [MSGID: 109005] >> [dht-selfheal.c:2328:dht_selfheal_directory] 0-vol01-dht: Directory >> selfheal failed: Unable to form layout for directory / >> [2018-06-29 07:52:07.885389] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:07.886457] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:07.887452] I [rpc-clnt.c:2071:rpc_clnt_reconfig] >> 0-vol01-client-0: changing port to 49152 (from 0) >> [2018-06-29 07:52:07.900627] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:07.901594] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:07.903220] I [MSGID: 114046] >> [client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-0: >> Connected to vol01-client-0, attached to remote volume >> '/data/glusterfs/gluster1/vol01/brick1'. >> [2018-06-29 07:52:13.062404] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:13.062780] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) >> op(WRITE(13)) called at 2018-06-29 07:52:13.048761 (xid=0x22) >> [2018-06-29 07:52:13.062826] W [MSGID: 114031] >> [client-rpc-fops_v2.c:658:client4_0_writev_cbk] 0-vol01-client-0: remote >> operation failed [Transport endpoint is not connected] >> [2018-06-29 07:52:13.063017] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) >> op(WRITE(13)) called at 2018-06-29 07:52:13.049162 (xid=0x23) >> [2018-06-29 07:52:13.063175] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) >> op(WRITE(13)) called at 2018-06-29 07:52:13.049865 (xid=0x24) >> [2018-06-29 07:52:13.063319] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) >> op(WRITE(13)) called at 2018-06-29 07:52:13.050199 (xid=0x25) >> [2018-06-29 07:52:13.063473] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) >> op(WRITE(13)) called at 2018-06-29 07:52:13.050489 (xid=0x26) >> [2018-06-29 07:52:13.063600] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) >> op(WRITE(13)) called at 2018-06-29 07:52:13.050757 (xid=0x27) >> [2018-06-29 07:52:13.063745] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) >> op(WRITE(13)) called at 2018-06-29 07:52:13.051031 (xid=0x28) >> [2018-06-29 07:52:13.063883] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) >> op(WRITE(13)) called at 2018-06-29 07:52:13.053035 (xid=0x29) >> The message "W [MSGID: 114031] [client-rpc-fops_v2.c:658:client4_0_writev_cbk] >> 0-vol01-client-0: remote operation failed [Transport endpoint is not >> connected]" repeated 7 times between [2018-06-29 07:52:13.062826] and >> [2018-06-29 07:52:13.063903] >> [2018-06-29 07:52:13.104580] W [MSGID: 114031] >> [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] 0-vol01-client-0: >> remote operation failed >> [2018-06-29 07:52:13.109230] W [socket.c:592:__socket_rwv] >> 0-vol01-client-1: writev on 192.168.10.234:49152 failed (No data >> available) >> [2018-06-29 07:52:13.109282] E [socket.c:2777:socket_poller] >> 0-vol01-client-1: poll error on socket >> [2018-06-29 07:52:13.109340] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from >> vol01-client-1. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:13.109357] E [MSGID: 108006] >> [afr-common.c:5158:__afr_handle_child_down_event] 0-vol01-replicate-0: >> All subvolumes are down. Going offline until atleast one of them comes back >> up. >> [2018-06-29 07:52:13.109461] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:13.109679] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) >> op(FSTAT(25)) called at 2018-06-29 07:52:13.104513 (xid=0x42) >> [2018-06-29 07:52:13.109700] W [MSGID: 114031] >> [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] 0-vol01-client-1: remote >> operation failed [Transport endpoint is not connected] >> [2018-06-29 07:52:13.109854] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) >> op(FXATTROP(34)) called at 2018-06-29 07:52:13.104729 (xid=0x43) >> [2018-06-29 07:52:13.109877] W [MSGID: 114031] >> [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] 0-vol01-client-1: >> remote operation failed >> [2018-06-29 07:52:13.109897] E [MSGID: 114031] >> [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 0-vol01-client-0: >> remote operation failed [Transport endpoint is not connected] >> [2018-06-29 07:52:13.110036] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) >> op(WRITE(13)) called at 2018-06-29 07:52:13.105623 (xid=0x44) >> [2018-06-29 07:52:13.110053] W [MSGID: 114031] >> [client-rpc-fops_v2.c:658:client4_0_writev_cbk] 0-vol01-client-1: remote >> operation failed [Transport endpoint is not connected] >> [2018-06-29 07:52:13.110232] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) >> op(FSTAT(25)) called at 2018-06-29 07:52:13.105658 (xid=0x45) >> [2018-06-29 07:52:13.110250] W [MSGID: 114031] >> [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] 0-vol01-client-1: remote >> operation failed [Transport endpoint is not connected] >> [2018-06-29 07:52:13.110386] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) >> op(FSTAT(25)) called at 2018-06-29 07:52:13.106862 (xid=0x46) >> [2018-06-29 07:52:13.110530] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) >> op(FSTAT(25)) called at 2018-06-29 07:52:13.108103 (xid=0x47) >> [2018-06-29 07:52:13.110660] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) >> op(FSTAT(25)) called at 2018-06-29 07:52:13.108969 (xid=0x48) >> [2018-06-29 07:52:13.110793] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) >> op(FXATTROP(34)) called at 2018-06-29 07:52:13.109256 (xid=0x49) >> The message "W [MSGID: 114031] [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] >> 0-vol01-client-1: remote operation failed [Transport endpoint is not >> connected]" repeated 3 times between [2018-06-29 07:52:13.110250] and >> [2018-06-29 07:52:13.110675] >> [2018-06-29 07:52:13.110808] W [MSGID: 114031] >> [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] 0-vol01-client-1: >> remote operation failed >> [2018-06-29 07:52:13.110933] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) >> op(FXATTROP(34)) called at 2018-06-29 07:52:13.109284 (xid=0x4a) >> The message "W [MSGID: 114031] [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] >> 0-vol01-client-1: remote operation failed" repeated 3 times between >> [2018-06-29 07:52:13.110808] and [2018-06-29 07:52:14.878874] >> [2018-06-29 07:52:14.878895] E [MSGID: 114031] >> [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 0-vol01-client-1: >> remote operation failed [Transport endpoint is not connected] >> [2018-06-29 07:52:14.879095] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:14.879217] W [fuse-bridge.c:2427:fuse_writev_cbk] >> 0-glusterfs-fuse: 76: WRITE => -1 gfid=a8848be6-1c05-4a72-88a7-5209eeef7c2c >> fd=0x7f7c50023a68 (Transport endpoint is not connected) >> [2018-06-29 07:52:14.880509] W [fuse-bridge.c:1406:fuse_err_cbk] >> 0-glusterfs-fuse: 77: FLUSH() ERR => -1 (Transport endpoint is not >> connected) >> The message "I [MSGID: 108006] [afr-common.c:5444:afr_local_init] >> 0-vol01-replicate-0: no subvolumes up" repeated 11 times between >> [2018-06-29 07:52:14.879095] and [2018-06-29 07:52:17.908056] >> [2018-06-29 07:52:17.908066] E [MSGID: 101046] >> [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null >> [2018-06-29 07:52:17.908175] W [fuse-bridge.c:896:fuse_attr_cbk] >> 0-glusterfs-fuse: 79: LOOKUP() / => -1 (Transport endpoint is not connected) >> [2018-06-29 07:52:18.758975] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:18.759155] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:18.759165] E [MSGID: 101046] >> [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null >> [2018-06-29 07:52:18.759214] W [fuse-bridge.c:896:fuse_attr_cbk] >> 0-glusterfs-fuse: 80: LOOKUP() / => -1 (Transport endpoint is not connected) >> [2018-06-29 07:52:21.662626] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:21.662756] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:21.662765] E [MSGID: 101046] >> [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null >> [2018-06-29 07:52:21.662811] W [fuse-bridge.c:896:fuse_attr_cbk] >> 0-glusterfs-fuse: 81: LOOKUP() / => -1 (Transport endpoint is not connected) >> [2018-06-29 07:52:21.663567] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:21.663630] W [fuse-bridge.c:3205:fuse_statfs_cbk] >> 0-glusterfs-fuse: 82: ERR => -1 (Transport endpoint is not connected) >> [2018-06-29 07:52:23.896362] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:23.896687] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:23.897321] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:23.896427 (xid=0x2a) >> [2018-06-29 07:52:23.897353] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:23.903241] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:23.903566] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from >> vol01-client-1. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:23.904166] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:23.903281 (xid=0x4b) >> [2018-06-29 07:52:23.904199] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:26.909819] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:26.910175] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:26.910858] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:26.909871 (xid=0x2b) >> [2018-06-29 07:52:26.910888] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:26.913921] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:26.914163] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from >> vol01-client-1. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:26.914417] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:26.913961 (xid=0x4c) >> [2018-06-29 07:52:26.914443] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:29.921628] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:29.921867] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:29.922215] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:29.921674 (xid=0x2c) >> [2018-06-29 07:52:29.922257] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:29.926462] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:29.926774] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from >> vol01-client-1. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:29.927415] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:29.926499 (xid=0x4d) >> [2018-06-29 07:52:29.927447] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:32.935218] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:32.935553] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:32.936196] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:32.935271 (xid=0x2d) >> [2018-06-29 07:52:32.936230] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:32.938940] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:32.939232] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from >> vol01-client-1. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:32.939497] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:32.938982 (xid=0x4e) >> [2018-06-29 07:52:32.939526] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:35.948789] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:35.949172] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:35.949860] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:35.948839 (xid=0x2e) >> [2018-06-29 07:52:35.949901] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:35.952154] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:35.952408] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from >> vol01-client-1. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:35.952670] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:35.952199 (xid=0x4f) >> [2018-06-29 07:52:35.952699] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:38.959369] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:38.959733] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:38.960536] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:38.959426 (xid=0x2f) >> [2018-06-29 07:52:38.960573] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:38.963381] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:38.963618] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from >> vol01-client-1. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:38.963877] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:38.963428 (xid=0x50) >> [2018-06-29 07:52:38.963906] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:41.969152] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:41.969397] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:41.969731] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:41.969204 (xid=0x30) >> [2018-06-29 07:52:41.969769] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:41.974130] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:41.974381] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from >> vol01-client-1. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:41.974669] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:41.974198 (xid=0x51) >> [2018-06-29 07:52:41.974698] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:44.984465] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:44.984781] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:44.985444] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:44.984514 (xid=0x31) >> [2018-06-29 07:52:44.985477] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:44.990039] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:44.990352] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from >> vol01-client-1. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:44.990729] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:44.990094 (xid=0x52) >> [2018-06-29 07:52:44.990771] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:47.994448] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:47.994684] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:47.994947] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:47.994499 (xid=0x32) >> [2018-06-29 07:52:47.994984] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:48.002544] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:48.002809] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from >> vol01-client-1. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:48.003388] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:48.002577 (xid=0x53) >> [2018-06-29 07:52:48.003416] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:49.440101] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:49.440257] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:49.440266] E [MSGID: 101046] >> [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null >> [2018-06-29 07:52:49.440310] W [fuse-bridge.c:896:fuse_attr_cbk] >> 0-glusterfs-fuse: 83: LOOKUP() / => -1 (Transport endpoint is not connected) >> [2018-06-29 07:52:50.182948] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:50.183108] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:50.183116] E [MSGID: 101046] >> [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null >> [2018-06-29 07:52:50.197572] W [fuse-bridge.c:896:fuse_attr_cbk] >> 0-glusterfs-fuse: 84: LOOKUP() / => -1 (Transport endpoint is not connected) >> [2018-06-29 07:52:50.575887] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:50.576030] I [MSGID: 108006] >> [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up >> [2018-06-29 07:52:50.576039] E [MSGID: 101046] >> [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null >> [2018-06-29 07:52:50.576087] W [fuse-bridge.c:896:fuse_attr_cbk] >> 0-glusterfs-fuse: 85: LOOKUP() / => -1 (Transport endpoint is not connected) >> [2018-06-29 07:52:51.012626] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-0: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:51.012924] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from >> vol01-client-0. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:51.013615] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:51.012667 (xid=0x33) >> [2018-06-29 07:52:51.013647] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: >> received RPC status error [Transport endpoint is not connected] >> [2018-06-29 07:52:51.018855] W [rpc-clnt.c:1739:rpc_clnt_submit] >> 0-vol01-client-1: error returned while attempting to connect to >> host:(null), port:0 >> [2018-06-29 07:52:51.019093] I [MSGID: 114018] >> [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from >> vol01-client-1. Client process will keep trying to connect to glusterd >> until brick's port is available >> [2018-06-29 07:52:51.019378] E [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> >> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> >> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] >> (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) >> 0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called >> at 2018-06-29 07:52:51.018893 (xid=0x54) >> [2018-06-29 07:52:51.019408] W [MSGID: 114032] >> [client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: >> received RPC status error [Transport endpoint is not connected] >> >> >> >> ====================================================>> >> After trying to copy the archive, I end up with different sized files on >> the two bricks: >> >> [root at gluster1 brick1]# pwd >> /data/glusterfs/gluster1/vol01/brick1 >> [root at gluster1 brick1]# ll >> total 384 >> -rw------- 2 root root 393216 Jun 29 10:52 mybee.tar.gz >> >> >> >> [root at gluster2 brick1]# pwd >> /data/glusterfs/gluster2/vol01/brick1 >> [root at gluster2 brick1]# ll >> total 1280 >> -rw------- 2 root root 1310720 Jun 29 10:52 mybee.tar.gz >> >> >> ====================================================>> >> These are the informations regarding the gluster volume: >> >> [root at gluster1 glusterfs]# gluster volume info vol01 >> >> Volume Name: vol01 >> Type: Replicate >> Volume ID: d3b39520-3fbb-4cfc-b736-338d7726b5a1 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: gluster1:/data/glusterfs/gluster1/vol01/brick1 >> Brick2: gluster2:/data/glusterfs/gluster2/vol01/brick1 >> Options Reconfigured: >> network.ping-timeout: 5 >> server.ssl: on >> client.ssl: on >> auth.ssl-allow: gluster01,gluster02,gluster-client >> transport.address-family: inet >> nfs.disable: on >> performance.client-io-threads: off >> >> ====================================================>> >> Here is the peers information: >> >> [root at gluster1 brick1]# gluster peer status >> Number of Peers: 1 >> >> Hostname: gluster2 >> Uuid: 9ccac8e3-71c1-43e4-b723-fb7374be8a1d >> State: Peer in Cluster (Connected) >> >> >> ====================================================>> >> Here is the volume status: >> >> [root at gluster1 brick1]# gluster volume status vol01 >> Status of volume: vol01 >> Gluster process TCP Port RDMA Port Online >> Pid >> ------------------------------------------------------------ >> ------------------ >> Brick gluster1:/data/glusterfs/gluster1/vol >> 01/brick1 49152 0 Y 11108 >> Brick gluster2:/data/glusterfs/gluster2/vol >> 01/brick1 49152 0 Y 10942 >> Self-heal Daemon on localhost N/A N/A Y 11133 >> Self-heal Daemon on gluster2 N/A N/A Y 10968 >> >> Task Status of Volume vol01 >> ------------------------------------------------------------ >> ------------------ >> There are no active volume tasks >> >> >> ====================================================>> >> >> These are the installed packages on gluster-client: >> >> [root at gluster-client glusterfs]# rpm -qa | grep "gluster\|fuse" >> centos-release-gluster40-1.0-2.el7.centos.x86_64 >> glusterfs-client-xlators-4.0.2-1.el7.x86_64 >> fuse-2.9.2-10.el7.x86_64 >> glusterfs-4.0.2-1.el7.x86_64 >> glusterfs-fuse-4.0.2-1.el7.x86_64 >> fuse-libs-2.9.2-10.el7.x86_64 >> glusterfs-libs-4.0.2-1.el7.x86_64 >> >> >> >> And these are the installed packages on gluster1 and gluster2 storage >> nodes: >> >> >> [root at gluster1 glusterfs]# rpm -qa | grep "gluster\|fuse" >> glusterfs-api-4.0.2-1.el7.x86_64 >> centos-release-gluster40-1.0-2.el7.centos.x86_64 >> glusterfs-libs-4.0.2-1.el7.x86_64 >> glusterfs-client-xlators-4.0.2-1.el7.x86_64 >> glusterfs-cli-4.0.2-1.el7.x86_64 >> glusterfs-fuse-4.0.2-1.el7.x86_64 >> glusterfs-server-4.0.2-1.el7.x86_64 >> glusterfs-4.0.2-1.el7.x86_64 >> >> >> >> >> Can anyone point me in the right direction? What am I doing wrong? The >> mount works fine until I try to copy an archive, multiple smaller files or >> a bigger file on it (meaning it shows correctly in df -Th and I can create >> several files with "touch file1 file2..."). Basically, after any data >> transfer, I get these errors. >> >> >> I followed the indications from the redhat page: >> >> https://access.redhat.com/documentation/en-us/red_hat_gluste >> r_storage/3.1/html/administration_guide/chap-network_encryption >> >> UPDATE 1: >> I tried doing the exact same steps in Gluster 3.12 and had no problem. >> The steps worked and SSL/TLS was enabled. There was no transport error or >> anything and I also checked if SSL/TLS was enabled. Afterwards, I also >> tried with the new release 4.1 and the problem persists (same error with >> "Transport endpoint is not connected"). >> Let me know if you need any other info. Any help is much appreciated. >> >> Thank you. >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180713/c35147b0/attachment.html>