Gluster users - Sep 2017 - GlusterFS as virtual machine storage

I think this should be considered a bug
If you have a server crash, glusterfsd process obviously doesn't exit
properly and thus this could least to IO stop ?

And server crashes are the main reason to use a redundant filesystem like
gluster


Il 8 set 2017 12:43 PM, "Diego Remolina" <dijuremo at gmail.com>
ha scritto:

This is exactly the problem,

Systemctl stop glusterd does *not* kill the brick processes.

On CentOS with gluster 3.10.x there is also a service, meant to only stop
glusterfsd (brick processes). I think the reboot process may not be
properly stopping glusterfsd or network or firewall may be stopped before
glusterfsd and so the nodes go into the long timeout.

Once again , in my case a simple reboot hangs my VMs, whereas stopping
glusterd *and* glusterfsd processes, then reboot works perfectly.

Diego

On Sep 8, 2017 6:36 AM, "Pavel Szalbot" <pavel.szalbot at
gmail.com> wrote:

Hi Diego,

indeed glusterfsd processes are runnin and it is the reason I do
server reboot instead of systemctl glusterd stop. Is killall different
from reboot in a way glusterfsd processes are terminated in CentOS
(init 1?)?

However I will try this and let you know.


-ps


On Fri, Sep 8, 2017 at 12:19 PM, Diego Remolina <dijuremo at gmail.com>
wrote:
> The issue of I/O stopping may also be with glusterfsd not being properly
> killed before rebooting the server.
>
> For example in RHEL 7.4 with official Gluster 3.8.4, the glusterd service
> does *not* stop glusterfsd when you run systemctl stop glusterd
>
> So give this a try on the nose you wish to reboot:
>
> 1. Stop glusterd
> 2. Check if glusterfsd processes are still running. If they are, use:
> killall glusterfsd
> Double check no more glusterfsd processes are running.
> 3. Now reboot the server, there should be no stopping of I/O on the VMs.
>
> I was able to reproduce the stopping I/O behaviour yesterday on server
> reboots without stopping glusterfsd processes. JoeJulian recommends
against
> changing network ping timeout. We discussed this via IRC recently.
>
> Diego
>
> On Sep 8, 2017 5:56 AM, "Pavel Szalbot" <pavel.szalbot at
gmail.com> wrote:
>
> This is the qemu log of instance:
>
> [2017-09-08 09:31:48.381077] C
> [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
> 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded
> in the last 1 seconds, disconnecting.
> [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>  0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:31:45.626882 (xid=0x27ab5)
> [2017-09-08 09:31:48.382451] E [MSGID: 114031]
> [client-rpc-fops.c:1593:client3_3_finodelk_cbk]
> 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
> is not connected]
> [2017-09-08 09:31:48.382835] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>  0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:31:45.627341 (xid=0x27ab6)
> [2017-09-08 09:31:48.383293] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>  0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:31:45.627403 (xid=0x27ab7)
> [2017-09-08 09:31:48.383639] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>  0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:31:45.627444 (xid=0x27ab8)
> [2017-09-08 09:31:48.384015] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>  0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(ENTRYLK(31)) called at 2017-09-08 09:31:45.627464 (xid=0x27ab9)
> [2017-09-08 09:31:48.384056] E [MSGID: 114031]
> [client-rpc-fops.c:1646:client3_3_entrylk_cbk]
> 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
> is not connected]
> [2017-09-08 09:31:48.384077] E [MSGID: 108007]
> [afr-lk-common.c:825:afr_unlock_entrylk_cbk]
> 0-gv_openstack_1-replicate-0:
> /.shard/fe56e79d-7800-4191-8cad-276545017ce7.4223: unlock failed on
> gv_openstack_1-client
> -1 [Transport endpoint is not connected]
> [2017-09-08 09:31:48.384336] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>  0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(WRITE(13)) called at 2017-09-08 09:31:45.626871 (xid=0x27ab4)
> [2017-09-08 09:31:48.388095] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>  0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(READ(12)) called at 2017-09-08 09:31:45.628760 (xid=0x27aba)
> [2017-09-08 09:31:48.388450] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>  0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(READ(12)) called at 2017-09-08 09:31:45.629421 (xid=0x27abb)
> [2017-09-08 09:31:48.388757] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>  0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.647966 (xid=0x27abc)
> [2017-09-08 09:31:48.389158] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>  0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.648128 (xid=0x27abd)
> [2017-09-08 09:31:48.389502] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170]
(-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
>> op(LOOKUP(27)) called at 2017-09-08 09:31:45.648260 (xid=0x27abe)
> [2017-09-08 09:31:48.389838] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.652420 (xid=0x27abf)
> [2017-09-08 09:31:48.390217] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.652552 (xid=0x27ac0)
> [2017-09-08 09:31:48.390558] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.652678 (xid=0x27ac1)
> [2017-09-08 09:31:48.390886] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.652810 (xid=0x27ac2)
> [2017-09-08 09:31:48.391263] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.652938 (xid=0x27ac3)
> [2017-09-08 09:31:48.391650] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371223 (xid=0x27ac4)
> [2017-09-08 09:31:48.392007] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371319 (xid=0x27ac5)
> [2017-09-08 09:31:48.392428] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371406 (xid=0x27ac6)
> [2017-09-08 09:31:48.392727] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371470 (xid=0x27ac7)
> [2017-09-08 09:31:48.393029] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371533 (xid=0x27ac8)
> [2017-09-08 09:31:48.393397] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371607 (xid=0x27ac9)
> [2017-09-08 09:31:48.393720] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371676 (xid=0x27aca)
> [2017-09-08 09:31:48.394092] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371739 (xid=0x27acb)
> [2017-09-08 09:31:48.396857] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371815 (xid=0x27acc)
> [2017-09-08 09:31:48.397273] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371913 (xid=0x27acd)
> [2017-09-08 09:31:48.399192] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.372010 (xid=0x27ace)
> [2017-09-08 09:31:48.402229] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.372147 (xid=0x27acf)
> [2017-09-08 09:31:48.402716] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.372247 (xid=0x27ad0)
> [2017-09-08 09:31:48.403171] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.372347 (xid=0x27ad1)
> [2017-09-08 09:31:48.403786] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.372438 (xid=0x27ad2)
> [2017-09-08 09:31:48.404281] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.372535 (xid=0x27ad3)
> [2017-09-08 09:31:48.406718] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.372632 (xid=0x27ad4)
> [2017-09-08 09:31:48.407210] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.372752 (xid=0x27ad5)
> [2017-09-08 09:31:48.407771] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.372840 (xid=0x27ad6)
> [2017-09-08 09:31:48.408253] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.372943 (xid=0x27ad7)
> [2017-09-08 09:31:48.408710] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.373054 (xid=0x27ad8)
> [2017-09-08 09:31:48.409190] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.373175 (xid=0x27ad9)
> [2017-09-08 09:31:48.409671] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.373276 (xid=0x27ada)
> [2017-09-08 09:31:48.410135] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.373375 (xid=0x27adb)
> [2017-09-08 09:31:48.410638] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.373466 (xid=0x27adc)
> [2017-09-08 09:31:48.411136] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.373552 (xid=0x27add)
> [2017-09-08 09:31:48.412471] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.373645 (xid=0x27ade)
> [2017-09-08 09:31:48.413783] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.373754 (xid=0x27adf)
> [2017-09-08 09:31:48.414972] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.373852 (xid=0x27ae0)
> [2017-09-08 09:31:48.415517] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.373936 (xid=0x27ae1)
> [2017-09-08 09:31:48.416428] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.374046 (xid=0x27ae2)
> [2017-09-08 09:31:48.417311] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.374148 (xid=0x27ae3)
> [2017-09-08 09:31:48.418227] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.374235 (xid=0x27ae4)
> [2017-09-08 09:31:48.419323] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.374326 (xid=0x27ae5)
> [2017-09-08 09:31:48.419910] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.374409 (xid=0x27ae6)
> [2017-09-08 09:31:48.420479] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.374525 (xid=0x27ae7)
> [2017-09-08 09:31:48.422134] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.374607 (xid=0x27ae8)
> [2017-09-08 09:31:48.424896] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.374704 (xid=0x27ae9)
> [2017-09-08 09:31:48.427236] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.374817 (xid=0x27aea)
> [2017-09-08 09:31:48.427802] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.374923 (xid=0x27aeb)
> [2017-09-08 09:31:48.428202] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.375012 (xid=0x27aec)
> [2017-09-08 09:31:48.428472] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.375130 (xid=0x27aed)
> [2017-09-08 09:31:48.428728] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.375245 (xid=0x27aee)
> [2017-09-08 09:31:48.429042] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.375333 (xid=0x27aef)
> [2017-09-08 09:31:48.429308] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.375427 (xid=0x27af0)
> [2017-09-08 09:31:48.429557] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.375531 (xid=0x27af1)
> [2017-09-08 09:31:48.429800] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.375651 (xid=0x27af2)
> [2017-09-08 09:31:48.430054] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.375759 (xid=0x27af3)
> [2017-09-08 09:31:48.430351] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.375857 (xid=0x27af4)
> [2017-09-08 09:31:48.430722] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.375956 (xid=0x27af5)
> [2017-09-08 09:31:48.431245] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.376060 (xid=0x27af6)
> [2017-09-08 09:31:48.431796] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.376157 (xid=0x27af7)
> [2017-09-08 09:31:48.432181] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.376251 (xid=0x27af8)
> [2017-09-08 09:31:48.432487] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.376346 (xid=0x27af9)
> [2017-09-08 09:31:48.432785] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.376457 (xid=0x27afa)
> [2017-09-08 09:31:48.433089] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.376540 (xid=0x27afb)
> [2017-09-08 09:31:48.433383] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.376630 (xid=0x27afc)
> [2017-09-08 09:31:48.433678] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.376734 (xid=0x27afd)
> [2017-09-08 09:31:48.433967] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.376827 (xid=0x27afe)
> [2017-09-08 09:31:48.434274] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.376950 (xid=0x27aff)
> [2017-09-08 09:31:48.434496] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.377054 (xid=0x27b00)
> [2017-09-08 09:31:48.434778] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.377153 (xid=0x27b01)
> [2017-09-08 09:31:48.435069] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.377249 (xid=0x27b02)
> [2017-09-08 09:31:48.435356] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.377352 (xid=0x27b03)
> [2017-09-08 09:31:48.435637] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.377448 (xid=0x27b04)
> [2017-09-08 09:31:48.435917] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.377554 (xid=0x27b05)
> [2017-09-08 09:31:48.436210] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.377637 (xid=0x27b06)
> [2017-09-08 09:31:48.436487] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.377739 (xid=0x27b07)
> [2017-09-08 09:31:48.436765] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.377836 (xid=0x27b08)
> [2017-09-08 09:31:48.437050] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.377938 (xid=0x27b09)
> [2017-09-08 09:31:48.437332] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.378052 (xid=0x27b0a)
> [2017-09-08 09:31:48.437610] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.378149 (xid=0x27b0b)
> [2017-09-08 09:31:48.437888] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.378411 (xid=0x27b0c)
> [2017-09-08 09:31:48.438176] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.378515 (xid=0x27b0d)
> [2017-09-08 09:31:48.438451] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.378645 (xid=0x27b0e)
> [2017-09-08 09:31:48.438727] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GF-DUMP)
> op(NULL(2)) called at 2017-09-08 09:31:47.378696 (xid=0x27b0f)
> [2017-09-08 09:31:48.438894] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.378881 (xid=0x27b10)
> [2017-09-08 09:31:48.439196] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.379090 (xid=0x27b11)
> [2017-09-08 09:31:48.439466] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.379200 (xid=0x27b12)
> [2017-09-08 09:31:48.439740] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.379303 (xid=0x27b13)
> [2017-09-08 09:31:48.440016] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.379403 (xid=0x27b14)
> [2017-09-08 09:31:48.440302] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.379537 (xid=0x27b15)
> [2017-09-08 09:31:48.440578] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.379679 (xid=0x27b16)
> [2017-09-08 09:31:48.440853] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.379831 (xid=0x27b17)
> [2017-09-08 09:31:48.441137] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.379921 (xid=0x27b18)
> [2017-09-08 09:31:48.441416] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.380023 (xid=0x27b19)
> [2017-09-08 09:31:48.441688] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.380156 (xid=0x27b1a)
> [2017-09-08 09:31:48.441960] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.380270 (xid=0x27b1b)
> [2017-09-08 09:31:48.442244] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.380399 (xid=0x27b1c)
> [2017-09-08 09:31:48.442520] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.380523 (xid=0x27b1d)
> [2017-09-08 09:31:48.442792] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.380635 (xid=0x27b1e)
> [2017-09-08 09:31:48.443079] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.380741 (xid=0x27b1f)
> [2017-09-08 09:31:48.443355] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.380859 (xid=0x27b20)
> [2017-09-08 09:31:48.443627] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:47.380969 (xid=0x27b21)
> The message "E [MSGID: 114031]
> [client-rpc-fops.c:1593:client3_3_finodelk_cbk]
> 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
> is not connected]" repeated 109 times between [2017-09-08
> 09:31:48.382451] and [2017-09-08 09:31:48.626176]
> [2017-09-08 09:32:17.489261] C
> [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
> 0-gv_openstack_1-client-2: server 10.0.1.200:49152 has not responded
> in the last 1 seconds, disconnecting.
> [2017-09-08 09:32:17.490362] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:32:16.055593 (xid=0x3071f)
> [2017-09-08 09:32:17.490401] E [MSGID: 114031]
> [client-rpc-fops.c:1593:client3_3_finodelk_cbk]
> 0-gv_openstack_1-client-2: remote operation failed [Transport endpoint
> is not connected]
> [2017-09-08 09:32:17.490736] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:32:16.055623 (xid=0x30720)
> [2017-09-08 09:32:17.491133] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:32:16.055654 (xid=0x30721)
> [2017-09-08 09:32:17.491480] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:32:16.055684 (xid=0x30722)
> [2017-09-08 09:32:17.491791] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:32:16.055712 (xid=0x30723)
> [2017-09-08 09:32:17.492181] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:32:16.055741 (xid=0x30724)
> [2017-09-08 09:32:17.492506] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:32:16.055770 (xid=0x30725)
> [2017-09-08 09:32:17.492811] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:32:16.055799 (xid=0x30726)
> [2017-09-08 09:32:17.493131] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FSYNC(16)) called at 2017-09-08 09:32:13.675475 (xid=0x2f186)
> [2017-09-08 09:32:17.493461] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.036379 (xid=0x3071d)
> [2017-09-08 09:32:17.494173] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.055558 (xid=0x3071e)
> [2017-09-08 09:32:17.495483] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FSTAT(25)) called at 2017-09-08 09:32:16.077348 (xid=0x30727)
> [2017-09-08 09:32:17.495787] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FSTAT(25)) called at 2017-09-08 09:32:16.078270 (xid=0x30728)
> [2017-09-08 09:32:17.496077] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484276 (xid=0x30729)
> [2017-09-08 09:32:17.496378] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484349 (xid=0x3072a)
> [2017-09-08 09:32:17.496655] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484435 (xid=0x3072b)
> [2017-09-08 09:32:17.496923] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484493 (xid=0x3072c)
> [2017-09-08 09:32:17.497202] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484553 (xid=0x3072d)
> [2017-09-08 09:32:17.497509] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484606 (xid=0x3072e)
> [2017-09-08 09:32:17.497780] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484669 (xid=0x3072f)
> [2017-09-08 09:32:17.498072] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484722 (xid=0x30730)
> [2017-09-08 09:32:17.498397] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484776 (xid=0x30731)
> [2017-09-08 09:32:17.498666] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484830 (xid=0x30732)
> [2017-09-08 09:32:17.499121] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484873 (xid=0x30733)
> [2017-09-08 09:32:17.499711] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.484956 (xid=0x30734)
> [2017-09-08 09:32:17.500893] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.485011 (xid=0x30735)
> [2017-09-08 09:32:17.501880] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.485128 (xid=0x30736)
> [2017-09-08 09:32:17.502892] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb
cad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:32:16.485243 (xid=0x30737)
> [2017-09-08 09:32:17.504237] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
...
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170908/db79fa2d/attachment.html>

On Fri, Sep 8, 2017 at 12:48 PM, Gandalf Corvotempesta
<gandalf.corvotempesta at gmail.com> wrote:
> I think this should be considered a bug
> If you have a server crash, glusterfsd process obviously doesn't exit
> properly and thus this could least to IO stop ?
I agree with you completely in this.

Btw now I am experiencing "Transport endpoint disconnects" because of
1s ping-timeout even though nodes are up. This sucks. The network is
not overloaded at all, switches are used only by gluster network and
network consists only of three gluster nodes and one VM hypervisor and
Cinder controller (no traffic here).
-ps


On Fri, Sep 8, 2017 at 12:50 PM, Pavel Szalbot <pavel.szalbot at
gmail.com> wrote:
> On Fri, Sep 8, 2017 at 12:48 PM, Gandalf Corvotempesta
> <gandalf.corvotempesta at gmail.com> wrote:
>> I think this should be considered a bug
>> If you have a server crash, glusterfsd process obviously doesn't
exit
>> properly and thus this could least to IO stop ?
>
> I agree with you completely in this.

I would prefer the behavior was different to what it is of I/O stopping.
The argument I heard for the long 42 second time out was that MTBF on a
server was high, and that the client reconnection operation was *costly*.
Those were arguments to *not* change the ping timeout value down from 42
seconds. I think it was mentioned that low ping timeout settings could lead
to high cpu loads with many clients trying to reconnect if a short timeout
was set. This is all hearsay, so the experts should explain it better... ?

Diego

On Sep 8, 2017 6:50 AM, "Pavel Szalbot" <pavel.szalbot at
gmail.com> wrote:
> On Fri, Sep 8, 2017 at 12:48 PM, Gandalf Corvotempesta
> <gandalf.corvotempesta at gmail.com> wrote:
> > I think this should be considered a bug
> > If you have a server crash, glusterfsd process obviously doesn't
exit
> > properly and thus this could least to IO stop ?
>
> I agree with you completely in this.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170908/da2bef78/attachment.html>