[Gluster-users] Problem with SSL/TLS encryption on Gluster 4.0 & 4.1

Havriliuc Andrei
Fri, 13 Jul 2018 09:44:50 -0000

Hello David,


Yeah. I also disabled SSL/TLS for management and communication for
Gluster. The problem is that I will have traffic over public WAN for a
geo replication instance and I cannot go forward until this is solved.


Was there a bug filed about this? I've looked at the issues on the
Glusterfs github page but I didn't find anything related to this:


https://github.com/gluster/glusterfs/issues


Regards,

Andrei


On 7/12/18 4:03 PM, David Spisla wrote:
> Hello Andrei,
>
> I am also using Gluster 4.1 on CentOS and I have the same problem. I
> tested it with a volume which had no network encryption and one with
> network encryption. You are not the only one.
> It seems to be a bug. At the moment there is no other choice to
> disable client.ssl and sever.ssl on a volume to have stable I/O.
>
> Regards
> David
>
>
> 2018-07-12 10:45 GMT+02:00 Havriliuc Andrei <andrei at hostvision.ro
> <mailto:andrei at hostvision.ro>>:
>
>     Hello,
>
>     I am doing some tests with GlusterFS 4.0 and I can't seem to solve
>     some SSL/TLS issues. I am trying to set up a 2 node replicated
>     gluster volume with SSL/TLS. For this setup, I use 3 KVM VMs (2
>     storage nodes + 1 client node). For the networking part, I use a
>     dedicated private LAN for the KVM VMs. Each VM is able to ping the
>     other, so there's no problem with the connectivity.
>
>     To try to make the procedure I used as clear as possible, I will
>     put all commands in chronological order:
>
>
>
>     ====================================================>
>     1. First, I update the systems, install ntp and then reboot:
>
>     yum update
>
>     yum install ntp
>
>     systemctl status ntpd
>     systemctl start ntpd
>     systemctl enable ntpd
>     systemctl status ntpd
>
>
>     ====================================================>
>
>     2. I use a separate partition within the two VM storage nodes.
>     Each of the two nodes see the separate partition as /dev/sdb.
>     After creating a thinly provisioned LV on each of the two nodes, I
>     create an XFS filesystem on them:
>
>
>     pvcreate /dev/sdb1
>
>     vgcreate -s 32M vg_glusterfs /dev/sdb1
>
>     lvcreate -L 20G --thinpool glusterfs_thin_pool vg_glusterfs
>
>     lvcreate -V 15G --thin -n glusterfs_thin_vol1
>     vg_glusterfs/glusterfs_thin_pool
>
>     mkfs.xfs -i size=512 /dev/vg_glusterfs/glusterfs_thin_vol1
>
>
>     ====================================================>
>
>     3. On each node, I configured the brick data partition and added
>     the following to /etc/fstab:
>
>     mkdir /data
>
>     echo "/dev/vg_glusterfs/glusterfs_thin_vol1 /data xfs defaults 1
>     2" >> /etc/fstab
>
>     mount -a
>
>
>     ====================================================>
>     4. After mounting the volume, I see the following in df -Th, which
>     is correct:
>
>     [root at gluster1 brick1]# df -Th
>     Filesystem                                   Type      Size Used
>     Avail Use% Mounted on
>     /dev/sda1                                    ext4       46G 1.4G
>     42G   4% /
>     devtmpfs                                     devtmpfs  3.9G    0
>     3.9G   0% /dev
>     tmpfs                                        tmpfs     3.9G    0
>     3.9G   0% /dev/shm
>     tmpfs                                        tmpfs     3.9G 8.6M
>     3.9G   1% /run
>     tmpfs                                        tmpfs     3.9G    0
>     3.9G   0% /sys/fs/cgroup
>     tmpfs                                        tmpfs     783M    0
>     783M   0% /run/user/0
>     /dev/mapper/vg_glusterfs-glusterfs_thin_vol1 xfs 15G   34M 15G
>      1% /data
>
>
>     ====================================================>
>     5. Create specific volume dirs on all storage nodes:
>
>
>     mkdir -pv /data/glusterfs/${HOSTNAME%%.*}/vol01
>
>
>     ====================================================>
>     6. Add entries in /etc/hosts:
>
>     vim /etc/hosts
>
>     192.168.10.233    gluster1
>     192.168.10.234    gluster2
>     192.168.10.237    gluster-client
>
>     ====================================================>
>     7. Install gluster from the CentOS SIG:
>
>
>     yum search centos-release-gluster
>     yum install centos-release-gluster40
>
>
>     yum install glusterfs-server
>
>     ====================================================>
>     8. Set up TLS/SSL encryption on all nodes and clients (gluster1,
>     gluster2, gluster-client):
>
>     openssl genrsa -out /etc/ssl/glusterfs.key 2048
>
>     In gluster1 node:
>     openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj
>     "/CN=gluster1" -out /etc/ssl/glusterfs.pem
>     In gluster2 node:
>     openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj
>     "/CN=gluster2" -out /etc/ssl/glusterfs.pem
>     In gluster-client node:
>     openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj
>     "/CN=gluster-client" -out /etc/ssl/glusterfs.pem
>
>
>     ====================================================>
>     9. On another box, I concatenate all of the .pem certificates into
>     a .ca file.
>
>     Bring all .pem files locally:
>
>     scp gluster1:/etc/ssl/glusterfs.pe <http://glusterfs.pe>m
>     gluster01.pem
>     scp gluster2:/etc/ssl/glusterfs.pe <http://glusterfs.pe>m
>     gluster02.pem
>     scp gluster-client:/etc/ssl/glusterfs.pem gluster-client.pem
>
>     For storage nodes, I concatenate all .pem certificates (including
>     the client's .pem):
>
>     cat gluster01.pem gluster02.pem gluster-client.pem >
>     glusterfs-nodes.ca <http://glusterfs-nodes.ca>
>
>     For server clients:
>
>     cat gluster01.pem gluster02.pem > glusterfs-client.ca
>     <http://glusterfs-client.ca>
>
>
>     ====================================================>
>
>     10. Put glusterfs-nodes.ca <http://glusterfs-nodes.ca> file on
all
>     the storage nodes (this includes storage nodes .pem + client's
.pem):
>
>     scp glusterfs-nodes.ca <http://glusterfs-nodes.ca>
>     gluster1:/etc/ssl/glusterfs.ca <http://glusterfs.ca>
>     scp glusterfs-nodes.ca <http://glusterfs-nodes.ca>
>     gluster2:/etc/ssl/glusterfs.ca <http://glusterfs.ca>
>
>     Put glusterfs-client.ca <http://glusterfs-client.ca> file on all
>     the storage nodes (this includes only storage nodes .pem):
>
>     scp glusterfs-client.ca <http://glusterfs-client.ca>
>     gluster-client:/etc/ssl/glusterfs.ca <http://glusterfs.ca>
>
>
>     ====================================================>
>
>     11. Enable management encryption on each node (command ran on
>     gluster1, gluster2, gluster-client):
>
>
>     touch /var/lib/glusterd/secure-access
>
>
>     ====================================================>
>
>     12. Start, enable and check status of glusterd on gluster1 and
>     gluster2:
>
>     systemctl start glusterd
>     systemctl enable glusterd
>     systemctl status glusterd
>
>     ====================================================>
>     13. Configure trusted storage pool (TSP):
>
>     >From gluster1:
>
>
>     gluster peer status
>
>
>     gluster peer probe gluster2
>
>     ====================================================>
>     14. Create the replicated gluster volume but don't start it:
>
>
>     gluster volume create vol01 replica 2 transport tcp
>     gluster1:/data/glusterfs/gluster1/vol01/brick1
>     gluster2:/data/glusterfs/gluster2/vol01/brick1
>
>     ====================================================>
>
>     15. Setup SSL/TLS access to the volume:
>
>     gluster volume set vol01 auth.ssl-allow
>     'gluster01,gluster02,gluster-client'
>
>
>     gluster volume set vol01 client.ssl on
>     gluster volume set vol01 server.ssl on
>
>     gluster volume set vol01 network.ping-timeout "5"
>
>     gluster volume start vol01
>
>
>     ====================================================>
>
>     16. Mount the volume on gluster-client:
>
>     mount -t glusterfs gluster1:/vol01 /mnt
>
>
>     After mounting the volume, df -h has the correct output:
>
>     [root at gluster-client mnt]# df -h
>     Filesystem       Size  Used Avail Use% Mounted on
>     /dev/sda1         48G   36G  9.3G  80% /
>     devtmpfs         1.9G     0  1.9G   0% /dev
>     tmpfs            1.9G     0  1.9G   0% /dev/shm
>     tmpfs            1.9G  8.5M  1.9G   1% /run
>     tmpfs            1.9G     0  1.9G   0% /sys/fs/cgroup
>     tmpfs            379M     0  379M   0% /run/user/0
>     gluster1:/vol01   15G  187M   15G   2% /mnt
>
>
>     ====================================================>
>     17. If I try to copy an archive, it throws an error after
>     transferring part of the file and the /mnt folder becomes
>     inaccessible for a while:
>
>     [root at gluster-client mnt]# cp /root/mybee.tar.gz /mnt/
>     cp: error writing ?/mnt/mybee.tar.gz?: Transport endpoint is not
>     connected
>     cp: failed to extend ?/mnt/mybee.tar.gz?: Transport endpoint is
>     not connected
>     cp: failed to close ?/mnt/mybee.tar.gz?: Transport endpoint is not
>     connected
>     [root at gluster-client mnt]# ll /mnt
>     ls: cannot open directory .: Transport endpoint is not connected
>
>
>     ====================================================>
>     This is the log from gluster-client for the mnt mountpoint (mnt.log):
>
>
>     [2018-06-29 07:52:03.869382] I [MSGID: 100030]
>     [glusterfsd.c:2625:main] 0-/usr/sbin/glusterfs: Started running
>     /usr/sbin/glusterfs version 4.0.2 (args: /usr/sbin/glusterfs
>     --process-name fuse --volfile-server=gluster1 --volfile-id=/vol01
>     /mnt)
>     [2018-06-29 07:52:03.878709] I [socket.c:4470:socket_init]
>     0-glusterfs: SSL support on the I/O path is ENABLED
>     [2018-06-29 07:52:03.878782] I [socket.c:4473:socket_init]
>     0-glusterfs: SSL support for glusterd is ENABLED
>     [2018-06-29 07:52:03.878795] I [socket.c:4490:socket_init]
>     0-glusterfs: using private polling thread
>     [2018-06-29 07:52:03.879108] E [socket.c:4541:socket_init]
>     0-glusterfs: failed to open /etc/ssl/dhparam.pem, DH ciphers are
>     disabled
>     [2018-06-29 07:52:03.885718] I [MSGID: 101190]
>     [event-epoll.c:609:event_dispatch_epoll_worker] 0-epoll: Started
>     thread with index 1
>     [2018-06-29 07:52:03.901983] I [MSGID: 101190]
>     [event-epoll.c:609:event_dispatch_epoll_worker] 0-epoll: Started
>     thread with index 2
>     [2018-06-29 07:52:03.902092] I [socket.c:4470:socket_init]
>     0-vol01-client-1: SSL support on the I/O path is ENABLED
>     [2018-06-29 07:52:03.902117] I [socket.c:4473:socket_init]
>     0-vol01-client-1: SSL support for glusterd is ENABLED
>     [2018-06-29 07:52:03.902142] I [socket.c:4490:socket_init]
>     0-vol01-client-1: using private polling thread
>     [2018-06-29 07:52:03.902266] E [socket.c:4541:socket_init]
>     0-vol01-client-1: failed to open /etc/ssl/dhparam.pem, DH ciphers
>     are disabled
>     [2018-06-29 07:52:03.902977] I [socket.c:4470:socket_init]
>     0-vol01-client-0: SSL support on the I/O path is ENABLED
>     [2018-06-29 07:52:03.903018] I [socket.c:4473:socket_init]
>     0-vol01-client-0: SSL support for glusterd is ENABLED
>     [2018-06-29 07:52:03.903028] I [socket.c:4490:socket_init]
>     0-vol01-client-0: using private polling thread
>     [2018-06-29 07:52:03.903161] E [socket.c:4541:socket_init]
>     0-vol01-client-0: failed to open /etc/ssl/dhparam.pem, DH ciphers
>     are disabled
>     [2018-06-29 07:52:03.903756] I [MSGID: 114020]
>     [client.c:2300:notify] 0-vol01-client-0: parent translators are
>     ready, attempting connect on transport
>     [2018-06-29 07:52:03.908432] I [MSGID: 114020]
>     [client.c:2300:notify] 0-vol01-client-1: parent translators are
>     ready, attempting connect on transport
>     Final graph:
>
+------------------------------------------------------------------------------+
>       1: volume vol01-client-0
>       2:     type protocol/client
>       3:     option ping-timeout 5
>       4:     option remote-host gluster1
>       5:     option remote-subvolume /data/glusterfs/gluster1/vol01/brick1
>       6:     option transport-type socket
>       7:     option transport.address-family inet
>       8:     option transport.socket.ssl-enabled on
>       9:     option transport.tcp-user-timeout 0
>      10:     option transport.socket.keepalive-time 20
>      11:     option transport.socket.keepalive-interval 2
>      12:     option transport.socket.keepalive-count 9
>      13:     option send-gids true
>      14: end-volume
>      15:
>      16: volume vol01-client-1
>      17:     type protocol/client
>      18:     option ping-timeout 5
>      19:     option remote-host gluster2
>      20:     option remote-subvolume /data/glusterfs/gluster2/vol01/brick1
>      21:     option transport-type socket
>      22:     option transport.address-family inet
>      23:     option transport.socket.ssl-enabled on
>      24:     option transport.tcp-user-timeout 0
>      25:     option transport.socket.keepalive-time 20
>      26:     option transport.socket.keepalive-interval 2
>      27:     option transport.socket.keepalive-count 9
>      28:     option send-gids true
>      29: end-volume
>      30:
>      31: volume vol01-replicate-0
>      32:     type cluster/replicate
>      33:     option afr-pending-xattr vol01-client-0,vol01-client-1
>      34:     option use-compound-fops off
>      35:     subvolumes vol01-client-0 vol01-client-1
>      36: end-volume
>      37:
>      38: volume vol01-dht
>      39:     type cluster/distribute
>      40:     option lock-migration off
>      41:     option force-migration off
>      42:     subvolumes vol01-replicate-0
>      43: end-volume
>      44:
>      45: volume vol01-write-behind
>      46:     type performance/write-behind
>      47:     subvolumes vol01-dht
>      48: end-volume
>      49:
>      50: volume vol01-read-ahead
>      51:     type performance/read-ahead
>      52:     subvolumes vol01-write-behind
>      53: end-volume
>      54:
>      55: volume vol01-readdir-ahead
>      56:     type performance/readdir-ahead
>      57:     option parallel-readdir off
>      58:     option rda-request-size 131072
>      59:     option rda-cache-limit 10MB
>      60:     subvolumes vol01-read-ahead
>      61: end-volume
>      62:
>      63: volume vol01-io-cache
>      64:     type performance/io-cache
>      65:     subvolumes vol01-readdir-ahead
>      66: end-volume
>      67:
>      68: volume vol01-quick-read
>      69:     type performance/quick-read
>      70:     subvolumes vol01-io-cache
>      71: end-volume
>      72:
>      73: volume vol01-open-behind
>      74:     type performance/open-behind
>      75:     subvolumes vol01-quick-read
>      76: end-volume
>      77:
>      78: volume vol01-md-cache
>      79:     type performance/md-cache
>      80:     subvolumes vol01-open-behind
>      81: end-volume
>      82:
>      83: volume vol01
>      84:     type debug/io-stats
>      85:     option log-level INFO
>      86:     option latency-measurement off
>      87:     option count-fop-hits off
>      88:     subvolumes vol01-md-cache
>      89: end-volume
>      90:
>      91: volume meta-autoload
>      92:     type meta
>      93:     subvolumes vol01
>      94: end-volume
>      95:
>
+------------------------------------------------------------------------------+
>     [2018-06-29 07:52:03.917518] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:03.918459] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:03.919289] I [rpc-clnt.c:2071:rpc_clnt_reconfig]
>     0-vol01-client-0: changing port to 49152 (from 0)
>     [2018-06-29 07:52:03.922903] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:03.923922] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:03.925115] I [rpc-clnt.c:2071:rpc_clnt_reconfig]
>     0-vol01-client-1: changing port to 49152 (from 0)
>     [2018-06-29 07:52:03.930023] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:03.930756] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:03.931859] I [MSGID: 114046]
>     [client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-0:
>     Connected to vol01-client-0, attached to remote volume
>     '/data/glusterfs/gluster1/vol01/brick1'.
>     [2018-06-29 07:52:03.931894] I [MSGID: 108005]
>     [afr-common.c:5081:__afr_handle_child_up_event]
>     0-vol01-replicate-0: Subvolume 'vol01-client-0' came back up;
>     going online.
>     [2018-06-29 07:52:03.938903] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:03.939803] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:03.941300] I [MSGID: 114046]
>     [client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-1:
>     Connected to vol01-client-1, attached to remote volume
>     '/data/glusterfs/gluster2/vol01/brick1'.
>     [2018-06-29 07:52:03.942884] I [fuse-bridge.c:4234:fuse_init]
>     0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs
>     7.24 kernel 7.22
>     [2018-06-29 07:52:03.942907] I
>     [fuse-bridge.c:4864:fuse_graph_sync] 0-fuse: switched to graph 0
>     [2018-06-29 07:52:03.946767] W [socket.c:592:__socket_rwv]
>     0-vol01-client-0: readv on 192.168.10.233:49152
>     <http://192.168.10.233:49152> failed (Input/output error)
>     [2018-06-29 07:52:03.946791] E [socket.c:2785:socket_poller]
>     0-vol01-client-0: socket_poller 192.168.10.233:49152
>     <http://192.168.10.233:49152> failed (Input/output error)
>     [2018-06-29 07:52:03.946837] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:03.947264] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1)
>     op(LOOKUP(27)) called at 2018-06-29 07:52:03.945958 (xid=0xb)
>     [2018-06-29 07:52:03.947297] W [MSGID: 114031]
>     [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-vol01-client-0:
>     remote operation failed. Path: /
>     (00000000-0000-0000-0000-000000000001) [Transport endpoint is not
>     connected]
>     [2018-06-29 07:52:03.947455] I [MSGID: 109005]
>     [dht-selfheal.c:2328:dht_selfheal_directory] 0-vol01-dht:
>     Directory selfheal failed: Unable to form layout for directory /
>     [2018-06-29 07:52:07.885389] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:07.886457] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:07.887452] I [rpc-clnt.c:2071:rpc_clnt_reconfig]
>     0-vol01-client-0: changing port to 49152 (from 0)
>     [2018-06-29 07:52:07.900627] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:07.901594] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:07.903220] I [MSGID: 114046]
>     [client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-0:
>     Connected to vol01-client-0, attached to remote volume
>     '/data/glusterfs/gluster1/vol01/brick1'.
>     [2018-06-29 07:52:13.062404] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:13.062780] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1)
>     op(WRITE(13)) called at 2018-06-29 07:52:13.048761 (xid=0x22)
>     [2018-06-29 07:52:13.062826] W [MSGID: 114031]
>     [client-rpc-fops_v2.c:658:client4_0_writev_cbk] 0-vol01-client-0:
>     remote operation failed [Transport endpoint is not connected]
>     [2018-06-29 07:52:13.063017] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1)
>     op(WRITE(13)) called at 2018-06-29 07:52:13.049162 (xid=0x23)
>     [2018-06-29 07:52:13.063175] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1)
>     op(WRITE(13)) called at 2018-06-29 07:52:13.049865 (xid=0x24)
>     [2018-06-29 07:52:13.063319] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1)
>     op(WRITE(13)) called at 2018-06-29 07:52:13.050199 (xid=0x25)
>     [2018-06-29 07:52:13.063473] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1)
>     op(WRITE(13)) called at 2018-06-29 07:52:13.050489 (xid=0x26)
>     [2018-06-29 07:52:13.063600] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1)
>     op(WRITE(13)) called at 2018-06-29 07:52:13.050757 (xid=0x27)
>     [2018-06-29 07:52:13.063745] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1)
>     op(WRITE(13)) called at 2018-06-29 07:52:13.051031 (xid=0x28)
>     [2018-06-29 07:52:13.063883] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1)
>     op(WRITE(13)) called at 2018-06-29 07:52:13.053035 (xid=0x29)
>     The message "W [MSGID: 114031]
>     [client-rpc-fops_v2.c:658:client4_0_writev_cbk] 0-vol01-client-0:
>     remote operation failed [Transport endpoint is not connected]"
>     repeated 7 times between [2018-06-29 07:52:13.062826] and
>     [2018-06-29 07:52:13.063903]
>     [2018-06-29 07:52:13.104580] W [MSGID: 114031]
>     [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk]
>     0-vol01-client-0: remote operation failed
>     [2018-06-29 07:52:13.109230] W [socket.c:592:__socket_rwv]
>     0-vol01-client-1: writev on 192.168.10.234:49152
>     <http://192.168.10.234:49152> failed (No data available)
>     [2018-06-29 07:52:13.109282] E [socket.c:2777:socket_poller]
>     0-vol01-client-1: poll error on socket
>     [2018-06-29 07:52:13.109340] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected
>     from vol01-client-1. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:13.109357] E [MSGID: 108006]
>     [afr-common.c:5158:__afr_handle_child_down_event]
>     0-vol01-replicate-0: All subvolumes are down. Going offline until
>     atleast one of them comes back up.
>     [2018-06-29 07:52:13.109461] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:13.109679] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1)
>     op(FSTAT(25)) called at 2018-06-29 07:52:13.104513 (xid=0x42)
>     [2018-06-29 07:52:13.109700] W [MSGID: 114031]
>     [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] 0-vol01-client-1:
>     remote operation failed [Transport endpoint is not connected]
>     [2018-06-29 07:52:13.109854] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1)
>     op(FXATTROP(34)) called at 2018-06-29 07:52:13.104729 (xid=0x43)
>     [2018-06-29 07:52:13.109877] W [MSGID: 114031]
>     [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk]
>     0-vol01-client-1: remote operation failed
>     [2018-06-29 07:52:13.109897] E [MSGID: 114031]
>     [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
>     0-vol01-client-0: remote operation failed [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:13.110036] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1)
>     op(WRITE(13)) called at 2018-06-29 07:52:13.105623 (xid=0x44)
>     [2018-06-29 07:52:13.110053] W [MSGID: 114031]
>     [client-rpc-fops_v2.c:658:client4_0_writev_cbk] 0-vol01-client-1:
>     remote operation failed [Transport endpoint is not connected]
>     [2018-06-29 07:52:13.110232] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1)
>     op(FSTAT(25)) called at 2018-06-29 07:52:13.105658 (xid=0x45)
>     [2018-06-29 07:52:13.110250] W [MSGID: 114031]
>     [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] 0-vol01-client-1:
>     remote operation failed [Transport endpoint is not connected]
>     [2018-06-29 07:52:13.110386] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1)
>     op(FSTAT(25)) called at 2018-06-29 07:52:13.106862 (xid=0x46)
>     [2018-06-29 07:52:13.110530] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1)
>     op(FSTAT(25)) called at 2018-06-29 07:52:13.108103 (xid=0x47)
>     [2018-06-29 07:52:13.110660] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1)
>     op(FSTAT(25)) called at 2018-06-29 07:52:13.108969 (xid=0x48)
>     [2018-06-29 07:52:13.110793] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1)
>     op(FXATTROP(34)) called at 2018-06-29 07:52:13.109256 (xid=0x49)
>     The message "W [MSGID: 114031]
>     [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] 0-vol01-client-1:
>     remote operation failed [Transport endpoint is not connected]"
>     repeated 3 times between [2018-06-29 07:52:13.110250] and
>     [2018-06-29 07:52:13.110675]
>     [2018-06-29 07:52:13.110808] W [MSGID: 114031]
>     [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk]
>     0-vol01-client-1: remote operation failed
>     [2018-06-29 07:52:13.110933] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1)
>     op(FXATTROP(34)) called at 2018-06-29 07:52:13.109284 (xid=0x4a)
>     The message "W [MSGID: 114031]
>     [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk]
>     0-vol01-client-1: remote operation failed" repeated 3 times
>     between [2018-06-29 07:52:13.110808] and [2018-06-29 07:52:14.878874]
>     [2018-06-29 07:52:14.878895] E [MSGID: 114031]
>     [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
>     0-vol01-client-1: remote operation failed [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:14.879095] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:14.879217] W
>     [fuse-bridge.c:2427:fuse_writev_cbk] 0-glusterfs-fuse: 76: WRITE
>     => -1 gfid=a8848be6-1c05-4a72-88a7-5209eeef7c2c fd=0x7f7c50023a68
>     (Transport endpoint is not connected)
>     [2018-06-29 07:52:14.880509] W [fuse-bridge.c:1406:fuse_err_cbk]
>     0-glusterfs-fuse: 77: FLUSH() ERR => -1 (Transport endpoint is not
>     connected)
>     The message "I [MSGID: 108006] [afr-common.c:5444:afr_local_init]
>     0-vol01-replicate-0: no subvolumes up" repeated 11 times between
>     [2018-06-29 07:52:14.879095] and [2018-06-29 07:52:17.908056]
>     [2018-06-29 07:52:17.908066] E [MSGID: 101046]
>     [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
>     [2018-06-29 07:52:17.908175] W [fuse-bridge.c:896:fuse_attr_cbk]
>     0-glusterfs-fuse: 79: LOOKUP() / => -1 (Transport endpoint is not
>     connected)
>     [2018-06-29 07:52:18.758975] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:18.759155] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:18.759165] E [MSGID: 101046]
>     [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
>     [2018-06-29 07:52:18.759214] W [fuse-bridge.c:896:fuse_attr_cbk]
>     0-glusterfs-fuse: 80: LOOKUP() / => -1 (Transport endpoint is not
>     connected)
>     [2018-06-29 07:52:21.662626] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:21.662756] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:21.662765] E [MSGID: 101046]
>     [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
>     [2018-06-29 07:52:21.662811] W [fuse-bridge.c:896:fuse_attr_cbk]
>     0-glusterfs-fuse: 81: LOOKUP() / => -1 (Transport endpoint is not
>     connected)
>     [2018-06-29 07:52:21.663567] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:21.663630] W
>     [fuse-bridge.c:3205:fuse_statfs_cbk] 0-glusterfs-fuse: 82: ERR =>
>     -1 (Transport endpoint is not connected)
>     [2018-06-29 07:52:23.896362] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:23.896687] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:23.897321] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:23.896427 (xid=0x2a)
>     [2018-06-29 07:52:23.897353] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-0: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:23.903241] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:23.903566] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected
>     from vol01-client-1. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:23.904166] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:23.903281 (xid=0x4b)
>     [2018-06-29 07:52:23.904199] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-1: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:26.909819] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:26.910175] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:26.910858] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:26.909871 (xid=0x2b)
>     [2018-06-29 07:52:26.910888] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-0: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:26.913921] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:26.914163] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected
>     from vol01-client-1. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:26.914417] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:26.913961 (xid=0x4c)
>     [2018-06-29 07:52:26.914443] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-1: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:29.921628] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:29.921867] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:29.922215] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:29.921674 (xid=0x2c)
>     [2018-06-29 07:52:29.922257] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-0: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:29.926462] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:29.926774] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected
>     from vol01-client-1. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:29.927415] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:29.926499 (xid=0x4d)
>     [2018-06-29 07:52:29.927447] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-1: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:32.935218] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:32.935553] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:32.936196] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:32.935271 (xid=0x2d)
>     [2018-06-29 07:52:32.936230] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-0: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:32.938940] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:32.939232] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected
>     from vol01-client-1. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:32.939497] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:32.938982 (xid=0x4e)
>     [2018-06-29 07:52:32.939526] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-1: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:35.948789] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:35.949172] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:35.949860] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:35.948839 (xid=0x2e)
>     [2018-06-29 07:52:35.949901] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-0: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:35.952154] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:35.952408] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected
>     from vol01-client-1. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:35.952670] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:35.952199 (xid=0x4f)
>     [2018-06-29 07:52:35.952699] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-1: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:38.959369] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:38.959733] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:38.960536] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:38.959426 (xid=0x2f)
>     [2018-06-29 07:52:38.960573] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-0: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:38.963381] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:38.963618] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected
>     from vol01-client-1. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:38.963877] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:38.963428 (xid=0x50)
>     [2018-06-29 07:52:38.963906] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-1: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:41.969152] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:41.969397] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:41.969731] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:41.969204 (xid=0x30)
>     [2018-06-29 07:52:41.969769] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-0: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:41.974130] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:41.974381] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected
>     from vol01-client-1. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:41.974669] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:41.974198 (xid=0x51)
>     [2018-06-29 07:52:41.974698] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-1: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:44.984465] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:44.984781] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:44.985444] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:44.984514 (xid=0x31)
>     [2018-06-29 07:52:44.985477] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-0: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:44.990039] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:44.990352] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected
>     from vol01-client-1. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:44.990729] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:44.990094 (xid=0x52)
>     [2018-06-29 07:52:44.990771] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-1: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:47.994448] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:47.994684] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:47.994947] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:47.994499 (xid=0x32)
>     [2018-06-29 07:52:47.994984] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-0: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:48.002544] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:48.002809] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected
>     from vol01-client-1. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:48.003388] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:48.002577 (xid=0x53)
>     [2018-06-29 07:52:48.003416] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-1: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:49.440101] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:49.440257] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:49.440266] E [MSGID: 101046]
>     [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
>     [2018-06-29 07:52:49.440310] W [fuse-bridge.c:896:fuse_attr_cbk]
>     0-glusterfs-fuse: 83: LOOKUP() / => -1 (Transport endpoint is not
>     connected)
>     [2018-06-29 07:52:50.182948] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:50.183108] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:50.183116] E [MSGID: 101046]
>     [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
>     [2018-06-29 07:52:50.197572] W [fuse-bridge.c:896:fuse_attr_cbk]
>     0-glusterfs-fuse: 84: LOOKUP() / => -1 (Transport endpoint is not
>     connected)
>     [2018-06-29 07:52:50.575887] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:50.576030] I [MSGID: 108006]
>     [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no
>     subvolumes up
>     [2018-06-29 07:52:50.576039] E [MSGID: 101046]
>     [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
>     [2018-06-29 07:52:50.576087] W [fuse-bridge.c:896:fuse_attr_cbk]
>     0-glusterfs-fuse: 85: LOOKUP() / => -1 (Transport endpoint is not
>     connected)
>     [2018-06-29 07:52:51.012626] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-0: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:51.012924] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected
>     from vol01-client-0. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:51.013615] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:51.012667 (xid=0x33)
>     [2018-06-29 07:52:51.013647] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-0: received RPC status error [Transport endpoint is
>     not connected]
>     [2018-06-29 07:52:51.018855] W [rpc-clnt.c:1739:rpc_clnt_submit]
>     0-vol01-client-1: error returned while attempting to connect to
>     host:(null), port:0
>     [2018-06-29 07:52:51.019093] I [MSGID: 114018]
>     [client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected
>     from vol01-client-1. Client process will keep trying to connect to
>     glusterd until brick's port is available
>     [2018-06-29 07:52:51.019378] E
>     [rpc-clnt.c:350:saved_frames_unwind] (-->
>     /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
>     (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (-->
>     /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (-->
>     /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
>     (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] )))))
>     0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1))
>     called at 2018-06-29 07:52:51.018893 (xid=0x54)
>     [2018-06-29 07:52:51.019408] W [MSGID: 114032]
>     [client-handshake.c:1620:client_dump_version_cbk]
>     0-vol01-client-1: received RPC status error [Transport endpoint is
>     not connected]
>
>
>
>     ====================================================>
>     After trying to copy the archive, I end up with different sized
>     files on the two bricks:
>
>     [root at gluster1 brick1]# pwd
>     /data/glusterfs/gluster1/vol01/brick1
>     [root at gluster1 brick1]# ll
>     total 384
>     -rw------- 2 root root 393216 Jun 29 10:52 mybee.tar.gz
>
>
>
>     [root at gluster2 brick1]# pwd
>     /data/glusterfs/gluster2/vol01/brick1
>     [root at gluster2 brick1]# ll
>     total 1280
>     -rw------- 2 root root 1310720 Jun 29 10:52 mybee.tar.gz
>
>
>     ====================================================>
>     These are the informations regarding the gluster volume:
>
>     [root at gluster1 glusterfs]# gluster volume info vol01
>
>     Volume Name: vol01
>     Type: Replicate
>     Volume ID: d3b39520-3fbb-4cfc-b736-338d7726b5a1
>     Status: Started
>     Snapshot Count: 0
>     Number of Bricks: 1 x 2 = 2
>     Transport-type: tcp
>     Bricks:
>     Brick1: gluster1:/data/glusterfs/gluster1/vol01/brick1
>     Brick2: gluster2:/data/glusterfs/gluster2/vol01/brick1
>     Options Reconfigured:
>     network.ping-timeout: 5
>     server.ssl: on
>     client.ssl: on
>     auth.ssl-allow: gluster01,gluster02,gluster-client
>     transport.address-family: inet
>     nfs.disable: on
>     performance.client-io-threads: off
>
>     ====================================================>
>     Here is the peers information:
>
>     [root at gluster1 brick1]# gluster peer status
>     Number of Peers: 1
>
>     Hostname: gluster2
>     Uuid: 9ccac8e3-71c1-43e4-b723-fb7374be8a1d
>     State: Peer in Cluster (Connected)
>
>
>     ====================================================>
>     Here is the volume status:
>
>     [root at gluster1 brick1]# gluster volume status vol01
>     Status of volume: vol01
>     Gluster process                             TCP Port  RDMA Port
>     Online  Pid
>
------------------------------------------------------------------------------
>     Brick gluster1:/data/glusterfs/gluster1/vol
>     01/brick1                                   49152     0 Y    11108
>     Brick gluster2:/data/glusterfs/gluster2/vol
>     01/brick1                                   49152     0 Y    10942
>     Self-heal Daemon on localhost               N/A       N/A Y      11133
>     Self-heal Daemon on gluster2                N/A       N/A Y      10968
>
>     Task Status of Volume vol01
>
------------------------------------------------------------------------------
>     There are no active volume tasks
>
>
>     ====================================================>
>
>     These are the installed packages on gluster-client:
>
>     [root at gluster-client glusterfs]# rpm -qa | grep
"gluster\|fuse"
>     centos-release-gluster40-1.0-2.el7.centos.x86_64
>     glusterfs-client-xlators-4.0.2-1.el7.x86_64
>     fuse-2.9.2-10.el7.x86_64
>     glusterfs-4.0.2-1.el7.x86_64
>     glusterfs-fuse-4.0.2-1.el7.x86_64
>     fuse-libs-2.9.2-10.el7.x86_64
>     glusterfs-libs-4.0.2-1.el7.x86_64
>
>
>
>     And these are the installed packages on gluster1 and gluster2
>     storage nodes:
>
>
>     [root at gluster1 glusterfs]# rpm -qa | grep "gluster\|fuse"
>     glusterfs-api-4.0.2-1.el7.x86_64
>     centos-release-gluster40-1.0-2.el7.centos.x86_64
>     glusterfs-libs-4.0.2-1.el7.x86_64
>     glusterfs-client-xlators-4.0.2-1.el7.x86_64
>     glusterfs-cli-4.0.2-1.el7.x86_64
>     glusterfs-fuse-4.0.2-1.el7.x86_64
>     glusterfs-server-4.0.2-1.el7.x86_64
>     glusterfs-4.0.2-1.el7.x86_64
>
>
>
>
>     Can anyone point me in the right direction? What am I doing wrong?
>     The mount works fine until I try to copy an archive, multiple
>     smaller files or a bigger file on it (meaning it shows correctly
>     in df -Th and I can create several files with "touch file1
>     file2..."). Basically, after any data transfer, I get these
errors.
>
>
>     I followed the indications from the redhat page:
>
>
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/chap-network_encryption
>
<https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/chap-network_encryption>
>
>     UPDATE 1:
>     I tried doing the exact same steps in Gluster 3.12 and had no
>     problem. The steps worked and SSL/TLS was enabled. There was no
>     transport error or anything and I also checked if SSL/TLS was
>     enabled. Afterwards, I also tried with the new release 4.1 and the
>     problem persists (same error with "Transport endpoint is not
>     connected").
>     Let me know if you need any other info. Any help is much appreciated.
>
>     Thank you.
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-users
>     <https://lists.gluster.org/mailman/listinfo/gluster-users>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180713/0e686085/attachment.html>