lemonnierk at ulrar.net
2017-Sep-08 09:42 UTC
[Gluster-users] GlusterFS as virtual machine storage
Oh, you really don't want to go below 30s, I was told. I'm using 30 seconds for the timeout, and indeed when a node goes down the VM freez for 30 seconds, but I've never seen them go read only for that. I _only_ use virtio though, maybe it's that. What are you using ? On Fri, Sep 08, 2017 at 11:41:13AM +0200, Pavel Szalbot wrote:> Back to replica 3 w/o arbiter. Two fio jobs running (direct=1 and > direct=0), rebooting one node... and VM dmesg looks like: > > [ 483.862664] blk_update_request: I/O error, dev vda, sector 23125016 > [ 483.898034] blk_update_request: I/O error, dev vda, sector 2161832 > [ 483.901103] blk_update_request: I/O error, dev vda, sector 2161832 > [ 483.904045] Aborting journal on device vda1-8. > [ 483.906959] blk_update_request: I/O error, dev vda, sector 2099200 > [ 483.908306] blk_update_request: I/O error, dev vda, sector 2099200 > [ 483.909585] Buffer I/O error on dev vda1, logical block 262144, > lost sync page write > [ 483.911121] blk_update_request: I/O error, dev vda, sector 2048 > [ 483.912192] blk_update_request: I/O error, dev vda, sector 2048 > [ 483.913221] Buffer I/O error on dev vda1, logical block 0, lost > sync page write > [ 483.914546] EXT4-fs error (device vda1): > ext4_journal_check_start:56: Detected aborted journal > [ 483.916230] EXT4-fs (vda1): Remounting filesystem read-only > [ 483.917231] EXT4-fs (vda1): previous I/O error to superblock detected > [ 483.917353] JBD2: Error -5 detected when updating journal > superblock for vda1-8. > [ 483.921106] blk_update_request: I/O error, dev vda, sector 2048 > [ 483.922147] blk_update_request: I/O error, dev vda, sector 2048 > [ 483.923107] Buffer I/O error on dev vda1, logical block 0, lost > sync page write > > Root fs is read-only even with 1s ping-timeout... > > I really hope I have been idiot for almost a year now and someone > shows what am I doing completely wrong because I dream about joining > the hordes of fellow colleagues who store multiple VMs in gluster and > never had a problem with it. I also suspect the CentOS libvirt version > to be the cause. > > -ps > > > On Fri, Sep 8, 2017 at 10:50 AM, Pavel Szalbot <pavel.szalbot at gmail.com> wrote: > > FYI I set up replica 3 (no arbiter this time), did the same thing - > > rebooted one node during lots of file IO on VM and IO stopped. > > > > As I mentioned either here or in another thread, this behavior is > > caused by high default of network.ping-timeout. My main problem used > > to be that setting it to low values like 3s or even 2s did not prevent > > the FS to be mounted as read-only in the past (at least with arbiter) > > and docs describe reconnect as very costly. If I set ping-timeout to > > 1s disaster of read-only mount is now prevented. > > > > However I find it very strange because in the past I actually did end > > up with read-only filesystem despite of the low ping-timeout. > > > > With replica 3 after node reboot iftop shows data flowing only to the > > one of remaining two nodes and there is no entry in heal info for the > > volume. Explanation would be very much appreciated ;-) > > > > Few minutes later I reverted back to replica 3 with arbiter (group > > virt, ping-timeout 1). All nodes are up. During first fio run, VM > > disconnected my ssh session, so I reconnected and saw ext4 problems in > > dmesg. I deleted the VM and started a new one. Glustershd.log fills > > with metadata heal shortly after fio job starts, but this time system > > is stable. > > Rebooting one of the nodes does not cause any problem (watching heal > > log, i/o on vm). > > > > So I decided to put more stress on VMs disk - I added second job with > > direct=1 and started it (now both are running) while one gluster node > > is still booting. What happened? One fio job reports "Bus error" and > > VM segfaults when trying to run dmesg... > > > > Is this gfapi related? Is this bug in arbiter? > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Digital signature URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170908/6c5ecce4/attachment.sig>
On Fri, Sep 8, 2017 at 11:42 AM, <lemonnierk at ulrar.net> wrote:> Oh, you really don't want to go below 30s, I was told. > I'm using 30 seconds for the timeout, and indeed when a node goes down > the VM freez for 30 seconds, but I've never seen them go read only for > that. > > I _only_ use virtio though, maybe it's that. What are you using ?As previously mentioned (long thread:-)) - virtio.
This is the qemu log of instance: [2017-09-08 09:31:48.381077] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded in the last 1 seconds, disconnecting. [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:31:45.626882 (xid=0x27ab5) [2017-09-08 09:31:48.382451] E [MSGID: 114031] [client-rpc-fops.c:1593:client3_3_finodelk_cbk] 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint is not connected] [2017-09-08 09:31:48.382835] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:31:45.627341 (xid=0x27ab6) [2017-09-08 09:31:48.383293] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:31:45.627403 (xid=0x27ab7) [2017-09-08 09:31:48.383639] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:31:45.627444 (xid=0x27ab8) [2017-09-08 09:31:48.384015] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(ENTRYLK(31)) called at 2017-09-08 09:31:45.627464 (xid=0x27ab9) [2017-09-08 09:31:48.384056] E [MSGID: 114031] [client-rpc-fops.c:1646:client3_3_entrylk_cbk] 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint is not connected] [2017-09-08 09:31:48.384077] E [MSGID: 108007] [afr-lk-common.c:825:afr_unlock_entrylk_cbk] 0-gv_openstack_1-replicate-0: /.shard/fe56e79d-7800-4191-8cad-276545017ce7.4223: unlock failed on gv_openstack_1-client -1 [Transport endpoint is not connected] [2017-09-08 09:31:48.384336] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2017-09-08 09:31:45.626871 (xid=0x27ab4) [2017-09-08 09:31:48.388095] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2017-09-08 09:31:45.628760 (xid=0x27aba) [2017-09-08 09:31:48.388450] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2017-09-08 09:31:45.629421 (xid=0x27abb) [2017-09-08 09:31:48.388757] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-09-08 09:31:45.647966 (xid=0x27abc) [2017-09-08 09:31:48.389158] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-09-08 09:31:45.648128 (xid=0x27abd) [2017-09-08 09:31:48.389502] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-09-08 09:31:45.648260 (xid=0x27abe)[2017-09-08 09:31:48.389838] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-09-08 09:31:45.652420 (xid=0x27abf) [2017-09-08 09:31:48.390217] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-09-08 09:31:45.652552 (xid=0x27ac0) [2017-09-08 09:31:48.390558] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-09-08 09:31:45.652678 (xid=0x27ac1) [2017-09-08 09:31:48.390886] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-09-08 09:31:45.652810 (xid=0x27ac2) [2017-09-08 09:31:48.391263] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-09-08 09:31:45.652938 (xid=0x27ac3) [2017-09-08 09:31:48.391650] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.371223 (xid=0x27ac4) [2017-09-08 09:31:48.392007] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.371319 (xid=0x27ac5) [2017-09-08 09:31:48.392428] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.371406 (xid=0x27ac6) [2017-09-08 09:31:48.392727] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.371470 (xid=0x27ac7) [2017-09-08 09:31:48.393029] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.371533 (xid=0x27ac8) [2017-09-08 09:31:48.393397] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.371607 (xid=0x27ac9) [2017-09-08 09:31:48.393720] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.371676 (xid=0x27aca) [2017-09-08 09:31:48.394092] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.371739 (xid=0x27acb) [2017-09-08 09:31:48.396857] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.371815 (xid=0x27acc) [2017-09-08 09:31:48.397273] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.371913 (xid=0x27acd) [2017-09-08 09:31:48.399192] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.372010 (xid=0x27ace) [2017-09-08 09:31:48.402229] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.372147 (xid=0x27acf) [2017-09-08 09:31:48.402716] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.372247 (xid=0x27ad0) [2017-09-08 09:31:48.403171] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.372347 (xid=0x27ad1) [2017-09-08 09:31:48.403786] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.372438 (xid=0x27ad2) [2017-09-08 09:31:48.404281] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.372535 (xid=0x27ad3) [2017-09-08 09:31:48.406718] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.372632 (xid=0x27ad4) [2017-09-08 09:31:48.407210] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.372752 (xid=0x27ad5) [2017-09-08 09:31:48.407771] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.372840 (xid=0x27ad6) [2017-09-08 09:31:48.408253] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.372943 (xid=0x27ad7) [2017-09-08 09:31:48.408710] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.373054 (xid=0x27ad8) [2017-09-08 09:31:48.409190] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.373175 (xid=0x27ad9) [2017-09-08 09:31:48.409671] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.373276 (xid=0x27ada) [2017-09-08 09:31:48.410135] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.373375 (xid=0x27adb) [2017-09-08 09:31:48.410638] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.373466 (xid=0x27adc) [2017-09-08 09:31:48.411136] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.373552 (xid=0x27add) [2017-09-08 09:31:48.412471] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.373645 (xid=0x27ade) [2017-09-08 09:31:48.413783] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.373754 (xid=0x27adf) [2017-09-08 09:31:48.414972] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.373852 (xid=0x27ae0) [2017-09-08 09:31:48.415517] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.373936 (xid=0x27ae1) [2017-09-08 09:31:48.416428] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.374046 (xid=0x27ae2) [2017-09-08 09:31:48.417311] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.374148 (xid=0x27ae3) [2017-09-08 09:31:48.418227] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.374235 (xid=0x27ae4) [2017-09-08 09:31:48.419323] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.374326 (xid=0x27ae5) [2017-09-08 09:31:48.419910] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.374409 (xid=0x27ae6) [2017-09-08 09:31:48.420479] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.374525 (xid=0x27ae7) [2017-09-08 09:31:48.422134] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.374607 (xid=0x27ae8) [2017-09-08 09:31:48.424896] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.374704 (xid=0x27ae9) [2017-09-08 09:31:48.427236] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.374817 (xid=0x27aea) [2017-09-08 09:31:48.427802] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.374923 (xid=0x27aeb) [2017-09-08 09:31:48.428202] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.375012 (xid=0x27aec) [2017-09-08 09:31:48.428472] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.375130 (xid=0x27aed) [2017-09-08 09:31:48.428728] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.375245 (xid=0x27aee) [2017-09-08 09:31:48.429042] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.375333 (xid=0x27aef) [2017-09-08 09:31:48.429308] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.375427 (xid=0x27af0) [2017-09-08 09:31:48.429557] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.375531 (xid=0x27af1) [2017-09-08 09:31:48.429800] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.375651 (xid=0x27af2) [2017-09-08 09:31:48.430054] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.375759 (xid=0x27af3) [2017-09-08 09:31:48.430351] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.375857 (xid=0x27af4) [2017-09-08 09:31:48.430722] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.375956 (xid=0x27af5) [2017-09-08 09:31:48.431245] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.376060 (xid=0x27af6) [2017-09-08 09:31:48.431796] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.376157 (xid=0x27af7) [2017-09-08 09:31:48.432181] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.376251 (xid=0x27af8) [2017-09-08 09:31:48.432487] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.376346 (xid=0x27af9) [2017-09-08 09:31:48.432785] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.376457 (xid=0x27afa) [2017-09-08 09:31:48.433089] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.376540 (xid=0x27afb) [2017-09-08 09:31:48.433383] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.376630 (xid=0x27afc) [2017-09-08 09:31:48.433678] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.376734 (xid=0x27afd) [2017-09-08 09:31:48.433967] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.376827 (xid=0x27afe) [2017-09-08 09:31:48.434274] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.376950 (xid=0x27aff) [2017-09-08 09:31:48.434496] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.377054 (xid=0x27b00) [2017-09-08 09:31:48.434778] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.377153 (xid=0x27b01) [2017-09-08 09:31:48.435069] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.377249 (xid=0x27b02) [2017-09-08 09:31:48.435356] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.377352 (xid=0x27b03) [2017-09-08 09:31:48.435637] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.377448 (xid=0x27b04) [2017-09-08 09:31:48.435917] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.377554 (xid=0x27b05) [2017-09-08 09:31:48.436210] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.377637 (xid=0x27b06) [2017-09-08 09:31:48.436487] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.377739 (xid=0x27b07) [2017-09-08 09:31:48.436765] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.377836 (xid=0x27b08) [2017-09-08 09:31:48.437050] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.377938 (xid=0x27b09) [2017-09-08 09:31:48.437332] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.378052 (xid=0x27b0a) [2017-09-08 09:31:48.437610] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:46.378149 (xid=0x27b0b) [2017-09-08 09:31:48.437888] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.378411 (xid=0x27b0c) [2017-09-08 09:31:48.438176] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.378515 (xid=0x27b0d) [2017-09-08 09:31:48.438451] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.378645 (xid=0x27b0e) [2017-09-08 09:31:48.438727] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2017-09-08 09:31:47.378696 (xid=0x27b0f) [2017-09-08 09:31:48.438894] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.378881 (xid=0x27b10) [2017-09-08 09:31:48.439196] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.379090 (xid=0x27b11) [2017-09-08 09:31:48.439466] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.379200 (xid=0x27b12) [2017-09-08 09:31:48.439740] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.379303 (xid=0x27b13) [2017-09-08 09:31:48.440016] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.379403 (xid=0x27b14) [2017-09-08 09:31:48.440302] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.379537 (xid=0x27b15) [2017-09-08 09:31:48.440578] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.379679 (xid=0x27b16) [2017-09-08 09:31:48.440853] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.379831 (xid=0x27b17) [2017-09-08 09:31:48.441137] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.379921 (xid=0x27b18) [2017-09-08 09:31:48.441416] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.380023 (xid=0x27b19) [2017-09-08 09:31:48.441688] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.380156 (xid=0x27b1a) [2017-09-08 09:31:48.441960] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.380270 (xid=0x27b1b) [2017-09-08 09:31:48.442244] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.380399 (xid=0x27b1c) [2017-09-08 09:31:48.442520] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.380523 (xid=0x27b1d) [2017-09-08 09:31:48.442792] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.380635 (xid=0x27b1e) [2017-09-08 09:31:48.443079] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.380741 (xid=0x27b1f) [2017-09-08 09:31:48.443355] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.380859 (xid=0x27b20) [2017-09-08 09:31:48.443627] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:31:47.380969 (xid=0x27b21) The message "E [MSGID: 114031] [client-rpc-fops.c:1593:client3_3_finodelk_cbk] 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint is not connected]" repeated 109 times between [2017-09-08 09:31:48.382451] and [2017-09-08 09:31:48.626176] [2017-09-08 09:32:17.489261] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gv_openstack_1-client-2: server 10.0.1.200:49152 has not responded in the last 1 seconds, disconnecting. [2017-09-08 09:32:17.490362] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:32:16.055593 (xid=0x3071f) [2017-09-08 09:32:17.490401] E [MSGID: 114031] [client-rpc-fops.c:1593:client3_3_finodelk_cbk] 0-gv_openstack_1-client-2: remote operation failed [Transport endpoint is not connected] [2017-09-08 09:32:17.490736] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:32:16.055623 (xid=0x30720) [2017-09-08 09:32:17.491133] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:32:16.055654 (xid=0x30721) [2017-09-08 09:32:17.491480] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:32:16.055684 (xid=0x30722) [2017-09-08 09:32:17.491791] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:32:16.055712 (xid=0x30723) [2017-09-08 09:32:17.492181] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:32:16.055741 (xid=0x30724) [2017-09-08 09:32:17.492506] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:32:16.055770 (xid=0x30725) [2017-09-08 09:32:17.492811] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:32:16.055799 (xid=0x30726) [2017-09-08 09:32:17.493131] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FSYNC(16)) called at 2017-09-08 09:32:13.675475 (xid=0x2f186) [2017-09-08 09:32:17.493461] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.036379 (xid=0x3071d) [2017-09-08 09:32:17.494173] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.055558 (xid=0x3071e) [2017-09-08 09:32:17.495483] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FSTAT(25)) called at 2017-09-08 09:32:16.077348 (xid=0x30727) [2017-09-08 09:32:17.495787] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FSTAT(25)) called at 2017-09-08 09:32:16.078270 (xid=0x30728) [2017-09-08 09:32:17.496077] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484276 (xid=0x30729) [2017-09-08 09:32:17.496378] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484349 (xid=0x3072a) [2017-09-08 09:32:17.496655] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484435 (xid=0x3072b) [2017-09-08 09:32:17.496923] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484493 (xid=0x3072c) [2017-09-08 09:32:17.497202] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484553 (xid=0x3072d) [2017-09-08 09:32:17.497509] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484606 (xid=0x3072e) [2017-09-08 09:32:17.497780] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484669 (xid=0x3072f) [2017-09-08 09:32:17.498072] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484722 (xid=0x30730) [2017-09-08 09:32:17.498397] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484776 (xid=0x30731) [2017-09-08 09:32:17.498666] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484830 (xid=0x30732) [2017-09-08 09:32:17.499121] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484873 (xid=0x30733) [2017-09-08 09:32:17.499711] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.484956 (xid=0x30734) [2017-09-08 09:32:17.500893] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.485011 (xid=0x30735) [2017-09-08 09:32:17.501880] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.485128 (xid=0x30736) [2017-09-08 09:32:17.502892] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.485243 (xid=0x30737) [2017-09-08 09:32:17.504237] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.485307 (xid=0x30738) [2017-09-08 09:32:17.505447] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.485370 (xid=0x30739) [2017-09-08 09:32:17.506503] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.485462 (xid=0x3073a) [2017-09-08 09:32:17.507727] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.485521 (xid=0x3073b) [2017-09-08 09:32:17.508173] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.485578 (xid=0x3073c) [2017-09-08 09:32:17.508925] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.485630 (xid=0x3073d) [2017-09-08 09:32:17.509262] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.485688 (xid=0x3073e) [2017-09-08 09:32:17.509641] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:16.485764 (xid=0x3073f) [2017-09-08 09:32:17.509957] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2017-09-08 09:32:16.485786 (xid=0x30740) [2017-09-08 09:32:17.510224] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.486075 (xid=0x30741) [2017-09-08 09:32:17.511645] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.486217 (xid=0x30742) [2017-09-08 09:32:17.511983] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.486340 (xid=0x30743) [2017-09-08 09:32:17.512326] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.486399 (xid=0x30744) [2017-09-08 09:32:17.512860] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.486543 (xid=0x30745) [2017-09-08 09:32:17.514077] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.486637 (xid=0x30746) [2017-09-08 09:32:17.514838] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.486734 (xid=0x30747) [2017-09-08 09:32:17.515413] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.486842 (xid=0x30748) [2017-09-08 09:32:17.516271] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.486964 (xid=0x30749) [2017-09-08 09:32:17.516775] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.487082 (xid=0x3074a) [2017-09-08 09:32:17.517195] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.487190 (xid=0x3074b) [2017-09-08 09:32:17.517635] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.487289 (xid=0x3074c) [2017-09-08 09:32:17.519469] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.487400 (xid=0x3074d) [2017-09-08 09:32:17.520008] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.487485 (xid=0x3074e) [2017-09-08 09:32:17.520411] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.487617 (xid=0x3074f) [2017-09-08 09:32:17.520891] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.487699 (xid=0x30750) [2017-09-08 09:32:17.521190] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.487836 (xid=0x30751) [2017-09-08 09:32:17.521489] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.487914 (xid=0x30752) [2017-09-08 09:32:17.521932] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488019 (xid=0x30753) [2017-09-08 09:32:17.522218] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488125 (xid=0x30754) [2017-09-08 09:32:17.522548] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488201 (xid=0x30755) [2017-09-08 09:32:17.524259] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488302 (xid=0x30756) [2017-09-08 09:32:17.524610] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488375 (xid=0x30757) [2017-09-08 09:32:17.524892] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488463 (xid=0x30758) [2017-09-08 09:32:17.525911] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488555 (xid=0x30759) [2017-09-08 09:32:17.526272] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488635 (xid=0x3075a) [2017-09-08 09:32:17.526555] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488713 (xid=0x3075b) [2017-09-08 09:32:17.528152] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488793 (xid=0x3075c) [2017-09-08 09:32:17.528524] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488870 (xid=0x3075d) [2017-09-08 09:32:17.529167] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.488948 (xid=0x3075e) [2017-09-08 09:32:17.529571] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.489064 (xid=0x3075f) [2017-09-08 09:32:17.529863] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.489164 (xid=0x30760) [2017-09-08 09:32:17.530152] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb09e8b] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08ee] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2017-09-08 09:32:17.489247 (xid=0x30761) [2017-09-08 09:32:33.490188] E [socket.c:2316:socket_connect_finish] 0-gv_openstack_1-client-1: connection to 10.0.1.202:24007 failed (No route to host); disconnecting socket The message "E [MSGID: 114031] [client-rpc-fops.c:1593:client3_3_finodelk_cbk] 0-gv_openstack_1-client-2: remote operation failed [Transport endpoint is not connected]" repeated 72 times between [2017-09-08 09:32:17.490401] and [2017-09-08 09:32:17.530223] 2017-09-08T09:41:36.512856Z qemu-kvm: terminating on signal 15 from pid 1528 [2017-09-08 09:41:36.616581] E [MSGID: 108006] [afr-common.c:4799:afr_notify] 0-gv_openstack_1-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. 2017-09-08 09:41:37.714+0000: shutting down -ps On Fri, Sep 8, 2017 at 11:42 AM, <lemonnierk at ulrar.net> wrote:> Oh, you really don't want to go below 30s, I was told. > I'm using 30 seconds for the timeout, and indeed when a node goes down > the VM freez for 30 seconds, but I've never seen them go read only for > that. > > I _only_ use virtio though, maybe it's that. What are you using ? > > > On Fri, Sep 08, 2017 at 11:41:13AM +0200, Pavel Szalbot wrote: >> Back to replica 3 w/o arbiter. Two fio jobs running (direct=1 and >> direct=0), rebooting one node... and VM dmesg looks like: >> >> [ 483.862664] blk_update_request: I/O error, dev vda, sector 23125016 >> [ 483.898034] blk_update_request: I/O error, dev vda, sector 2161832 >> [ 483.901103] blk_update_request: I/O error, dev vda, sector 2161832 >> [ 483.904045] Aborting journal on device vda1-8. >> [ 483.906959] blk_update_request: I/O error, dev vda, sector 2099200 >> [ 483.908306] blk_update_request: I/O error, dev vda, sector 2099200 >> [ 483.909585] Buffer I/O error on dev vda1, logical block 262144, >> lost sync page write >> [ 483.911121] blk_update_request: I/O error, dev vda, sector 2048 >> [ 483.912192] blk_update_request: I/O error, dev vda, sector 2048 >> [ 483.913221] Buffer I/O error on dev vda1, logical block 0, lost >> sync page write >> [ 483.914546] EXT4-fs error (device vda1): >> ext4_journal_check_start:56: Detected aborted journal >> [ 483.916230] EXT4-fs (vda1): Remounting filesystem read-only >> [ 483.917231] EXT4-fs (vda1): previous I/O error to superblock detected >> [ 483.917353] JBD2: Error -5 detected when updating journal >> superblock for vda1-8. >> [ 483.921106] blk_update_request: I/O error, dev vda, sector 2048 >> [ 483.922147] blk_update_request: I/O error, dev vda, sector 2048 >> [ 483.923107] Buffer I/O error on dev vda1, logical block 0, lost >> sync page write >> >> Root fs is read-only even with 1s ping-timeout... >> >> I really hope I have been idiot for almost a year now and someone >> shows what am I doing completely wrong because I dream about joining >> the hordes of fellow colleagues who store multiple VMs in gluster and >> never had a problem with it. I also suspect the CentOS libvirt version >> to be the cause. >> >> -ps >> >> >> On Fri, Sep 8, 2017 at 10:50 AM, Pavel Szalbot <pavel.szalbot at gmail.com> wrote: >> > FYI I set up replica 3 (no arbiter this time), did the same thing - >> > rebooted one node during lots of file IO on VM and IO stopped. >> > >> > As I mentioned either here or in another thread, this behavior is >> > caused by high default of network.ping-timeout. My main problem used >> > to be that setting it to low values like 3s or even 2s did not prevent >> > the FS to be mounted as read-only in the past (at least with arbiter) >> > and docs describe reconnect as very costly. If I set ping-timeout to >> > 1s disaster of read-only mount is now prevented. >> > >> > However I find it very strange because in the past I actually did end >> > up with read-only filesystem despite of the low ping-timeout. >> > >> > With replica 3 after node reboot iftop shows data flowing only to the >> > one of remaining two nodes and there is no entry in heal info for the >> > volume. Explanation would be very much appreciated ;-) >> > >> > Few minutes later I reverted back to replica 3 with arbiter (group >> > virt, ping-timeout 1). All nodes are up. During first fio run, VM >> > disconnected my ssh session, so I reconnected and saw ext4 problems in >> > dmesg. I deleted the VM and started a new one. Glustershd.log fills >> > with metadata heal shortly after fio job starts, but this time system >> > is stable. >> > Rebooting one of the nodes does not cause any problem (watching heal >> > log, i/o on vm). >> > >> > So I decided to put more stress on VMs disk - I added second job with >> > direct=1 and started it (now both are running) while one gluster node >> > is still booting. What happened? One fio job reports "Bus error" and >> > VM segfaults when trying to run dmesg... >> > >> > Is this gfapi related? Is this bug in arbiter? >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users
On Fri, Sep 8, 2017 at 5:56 AM, Pavel Szalbot <pavel.szalbot at gmail.com> wrote:> This is the qemu log of instance: > > [2017-09-08 09:31:48.381077] C > [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] > 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded > in the last 1 seconds, disconnecting. > >1 second is not an ideal value for ping timeout. Can you please set it to 30 seconds or so and simulate the problem? I would be interested in observing your logs with a higher ping timeout value. Thanks, Vijay -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170910/a608b97b/attachment.html>