search for: finodelk

...0.00 us 0.00 us 1 > RELEASE > 0.00 18.00 us 18.00 us 18.00 us 1 > STATFS > 0.00 20.50 us 11.00 us 30.00 us 2 > FLUSH > 0.00 22.50 us 17.00 us 28.00 us 2 > FINODELK > 0.01 76.50 us 65.00 us 88.00 us 2 > FXATTROP > 0.01 177.00 us 177.00 us 177.00 us 1 > CREATE > 0.02 56.14 us 23.00 us 128.00 us 7 > LOOKUP > 0.02 259.00 us 20.0...

...1 >> RELEASE >> 0.00 18.00 us 18.00 us 18.00 us 1 >> STATFS >> 0.00 20.50 us 11.00 us 30.00 us 2 >> FLUSH >> 0.00 22.50 us 17.00 us 28.00 us 2 >> FINODELK >> 0.01 76.50 us 65.00 us 88.00 us 2 >> FXATTROP >> 0.01 177.00 us 177.00 us 177.00 us 1 >> CREATE >> 0.02 56.14 us 23.00 us 128.00 us 7 >> LOOKUP >>...

...>>> 0.00 18.00 us 18.00 us 18.00 us 1 >>> STATFS >>> 0.00 20.50 us 11.00 us 30.00 us 2 >>> FLUSH >>> 0.00 22.50 us 17.00 us 28.00 us 2 >>> FINODELK >>> 0.01 76.50 us 65.00 us 88.00 us 2 >>> FXATTROP >>> 0.01 177.00 us 177.00 us 177.00 us 1 >>> CREATE >>> 0.02 56.14 us 23.00 us 128.00 us 7 >>>...

...hared-client-0-0 [2012-12-15 00:53:24.743459] I [server-helpers.c:474:do_fd_cleanup] 0-RedhawkShared-server: fd cleanup on /mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQswL [2012-12-15 00:53:24.743430] I [server-helpers.c:330:do_lock_table_cleanup] 0-RedhawkShared-server: finodelk released on /mkennedy/tramelot_nwfs/rpr3/rpr3/rpr3_sparky/matrix/.4d_ccnoesy.ucsf.QTQswL [2012-12-15 00:53:24.743400] I [server-helpers.c:741:server_connection_put] 0-RedhawkShared-server: Shutting down connection mualhpcp01.hpc.muohio.edu-17684-2012/12/13-17:25:16:994209-RedhawkShared-client-0-0 [...

...with absurd latency figures. I stopped the node and the problem moved to another brick: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 99.94 551292.41 us 10.00 us 1996709.00 us 361 FINODELK Could anyone provide some information how to debug this problem? Currently the volume is not usable due to the horrible delay. Thank you very much in advance, Samuel. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/glust...

...107.00 us 550203.00 us 11840 > FXATTROP > 0.34 14820.49 us 38.00 us 1935527.00 us 444 > STAT > 0.88 3765.77 us 42.00 us 1341978.00 us 4581 > LOOKUP > 15.06 19624.04 us 26.00 us 6412511.00 us 14971 > FINODELK > 21.63 59817.26 us 45.00 us 4008832.00 us 7054 > FSYNC > 23.19 263941.14 us 86.00 us 4130892.00 us 1714 > READ > 38.10 125578.64 us 207.00 us 7682338.00 us 5919 > WRITE > > Duration: 6957 seconds > D...

OK my understanding is that with preallocated disks the performance with and without shard will be the same. In any case, please attach the volume profile[1], so we can see what else is slowing things down. -Krutika [1] - https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/#running-glusterfs-volume-profile-command On Tue, Sep 5, 2017 at 2:32 PM, Abi Askushi

...11840 >> FXATTROP >> 0.34 14820.49 us 38.00 us 1935527.00 us 444 >> STAT >> 0.88 3765.77 us 42.00 us 1341978.00 us 4581 >> LOOKUP >> 15.06 19624.04 us 26.00 us 6412511.00 us 14971 >> FINODELK >> 21.63 59817.26 us 45.00 us 4008832.00 us 7054 >> FSYNC >> 23.19 263941.14 us 86.00 us 4130892.00 us 1714 >> READ >> 38.10 125578.64 us 207.00 us 7682338.00 us 5919 >> WRITE >> >>...

...ROP >>> 0.34 14820.49 us 38.00 us 1935527.00 us 444 >>> STAT >>> 0.88 3765.77 us 42.00 us 1341978.00 us 4581 >>> LOOKUP >>> 15.06 19624.04 us 26.00 us 6412511.00 us 14971 >>> FINODELK >>> 21.63 59817.26 us 45.00 us 4008832.00 us 7054 >>> FSYNC >>> 23.19 263941.14 us 86.00 us 4130892.00 us 1714 >>> READ >>> 38.10 125578.64 us 207.00 us 7682338.00 us 5919 >>> WR...

....34 14820.49 us 38.00 us 1935527.00 us 444 >>>> STAT >>>> 0.88 3765.77 us 42.00 us 1341978.00 us 4581 >>>> LOOKUP >>>> 15.06 19624.04 us 26.00 us 6412511.00 us 14971 >>>> FINODELK >>>> 21.63 59817.26 us 45.00 us 4008832.00 us 7054 >>>> FSYNC >>>> 23.19 263941.14 us 86.00 us 4130892.00 us 1714 >>>> READ >>>> 38.10 125578.64 us 207.00 us 7682338.00 us...

....0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) > 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) > op(FINODELK(30)) called at 2017-09-08 09:31:45.626882 (xid=0x27ab5) > [2017-09-08 09:31:48.382451] E [MSGID: 114031] > [client-rpc-fops.c:1593:client3_3_finodelk_cbk] > 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint > is not connected] > [2017-09-08 09:31:48.382835] E [rp...

...gfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] > (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) > 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) > op(FINODELK(30)) called at 2017-09-08 09:31:45.626882 (xid=0x27ab5) > [2017-09-08 09:31:48.382451] E [MSGID: 114031] > [client-rpc-fops.c:1593:client3_3_finodelk_cbk] > 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint > is not connected] > [2017-09-08 09:31:48.382835] E [rp...

...313379 >> FSTAT >> 0.23 1059.84 us 25.00 us 2716124.00 us 38255 >> LOOKUP >> 0.47 1024.11 us 54.00 us 6197164.00 us 81455 >> FXATTROP >> 1.72 2984.00 us 15.00 us 37098954.00 us 103020 >> FINODELK >> 5.92 44315.32 us 51.00 us 24731536.00 us 23957 >> FSYNC >> 13.27 2399.78 us 25.00 us 22089540.00 us 991005 >> READ >> 37.00 5980.43 us 52.00 us 22099889.00 us 1108976 >> WRITE >> 41....

...(-- > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:31:45.626882 (xid=0x27ab5) [2017-09-08 09:31:48.382451] E [MSGID: 114031] [client-rpc-fops.c:1593:client3_3_finodelk_cbk] 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint is not connected] [2017-09-08 09:31:48.382835] E [rpc-clnt.c:365:saved_frames...

....0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) > 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) > op(FINODELK(30)) called at 2017-09-08 09:31:45.626882 (xid=0x27ab5) > [2017-09-08 09:31:48.382451] E [MSGID: 114031] > [client-rpc-fops.c:1593:client3_3_finodelk_cbk] > 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint > is not connected] > [2017-09-08 09:31:48.382835] E [rp...

....0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) > 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) > op(FINODELK(30)) called at 2017-09-08 09:31:45.626882 (xid=0x27ab5) > [2017-09-08 09:31:48.382451] E [MSGID: 114031] > [client-rpc-fops.c:1593:client3_3_finodelk_cbk] > 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint > is not connected] > [2017-09-08 09:31:48.382835] E [rp...

...0.42 2574.98 us 208.73 us 1710282.73 us 11877 FXATTROP 2.80 609.20 us 468.51 us 321422.91 us 333946 RCHECKSUM 5.04 548.76 us 14.83 us 76288179.46 us 668188 INODELK 18.46 149940.70 us 13.59 us 79966278.04 us 8949 FINODELK 20.04 395073.91 us 84.99 us 3835355.67 us 3688 FSYNC 53.17 131171.66 us 85.76 us 3838020.34 us 29470 WRITE 0.00 0.00 us 0.00 us 0.00 us 7238 UPCALL 0.00 0.00 us 0.00 us 0.00 us 7238 CI_IAT...

Hi Krutika, I already have a preallocated disk on VM. Now I am checking performance with dd on the hypervisors which have the gluster volume configured. I tried also several values of shard-block-size and I keep getting the same low values on write performance. Enabling client-io-threads also did not have any affect. The version of gluster I am using is glusterfs 3.8.12 built on May 11 2017

...(-- > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fbcad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-09-08 09:31:45.626882 (xid=0x27ab5) [2017-09-08 09:31:48.382451] E [MSGID: 114031] [client-rpc-fops.c:1593:client3_3_finodelk_cbk] 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint is not connected] [2017-09-08 09:31:48.382835] E [rpc-clnt.c:365:saved_frames...

Oh, you really don't want to go below 30s, I was told. I'm using 30 seconds for the timeout, and indeed when a node goes down the VM freez for 30 seconds, but I've never seen them go read only for that. I _only_ use virtio though, maybe it's that. What are you using ? On Fri, Sep 08, 2017 at 11:41:13AM +0200, Pavel Szalbot wrote: > Back to replica 3 w/o arbiter. Two fio jobs