Hu Bert
2018-Aug-17 06:48 UTC
[Gluster-users] Gluter 3.12.12: performance during heal and in general
i hope i did get it right. gluster volume profile shared start wait 10 minutes gluster volume profile shared info gluster volume profile shared stop If that's ok, i've attached the output of the info command. 2018-08-17 8:31 GMT+02:00 Pranith Kumar Karampuri <pkarampu at redhat.com>:> Please do volume profile also for around 10 minutes when CPU% is high. > > On Fri, Aug 17, 2018 at 11:56 AM Pranith Kumar Karampuri > <pkarampu at redhat.com> wrote: >> >> As per the output, all io-threads are using a lot of CPU. It is better to >> check what the volume profile is to see what is leading to so much work for >> io-threads. Please follow the documentation at >> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/ >> section: " >> >> Running GlusterFS Volume Profile Command" >> >> and attach output of "gluster volume profile info", >> >> On Fri, Aug 17, 2018 at 11:24 AM Hu Bert <revirii at googlemail.com> wrote: >>> >>> Good morning, >>> >>> i ran the command during 100% CPU usage and attached the file. >>> Hopefully it helps. >>> >>> 2018-08-17 7:33 GMT+02:00 Pranith Kumar Karampuri <pkarampu at redhat.com>: >>> > Could you do the following on one of the nodes where you are observing >>> > high >>> > CPU usage and attach that file to this thread? We can find what >>> > threads/processes are leading to high usage. Do this for say 10 minutes >>> > when >>> > you see the ~100% CPU. >>> > >>> > top -bHd 5 > /tmp/top.${HOSTNAME}.txt >>> > >>> > On Wed, Aug 15, 2018 at 2:37 PM Hu Bert <revirii at googlemail.com> wrote: >>> >> >>> >> Hello again :-) >>> >> >>> >> The self heal must have finished as there are no log entries in >>> >> glustershd.log files anymore. According to munin disk latency (average >>> >> io wait) has gone down to 100 ms, and disk utilization has gone down >>> >> to ~60% - both on all servers and hard disks. >>> >> >>> >> But now system load on 2 servers (which were in the good state) >>> >> fluctuates between 60 and 100; the server with the formerly failed >>> >> disk has a load of 20-30.I've uploaded some munin graphics of the cpu >>> >> usage: >>> >> >>> >> https://abload.de/img/gluster11_cpu31d3a.png >>> >> https://abload.de/img/gluster12_cpu8sem7.png >>> >> https://abload.de/img/gluster13_cpud7eni.png >>> >> >>> >> This can't be normal. 2 of the servers under heavy load and one not >>> >> that much. Does anyone have an explanation of this strange behaviour? >>> >> >>> >> >>> >> Thx :-) >>> >> >>> >> 2018-08-14 9:37 GMT+02:00 Hu Bert <revirii at googlemail.com>: >>> >> > Hi there, >>> >> > >>> >> > well, it seems the heal has finally finished. Couldn't see/find any >>> >> > related log message; is there such a message in a specific log file? >>> >> > >>> >> > But i see the same behaviour when the last heal finished: all CPU >>> >> > cores are consumed by brick processes; not only by the formerly >>> >> > failed >>> >> > bricksdd1, but by all 4 brick processes (and their threads). Load >>> >> > goes >>> >> > up to > 100 on the 2 servers with the not-failed brick, and >>> >> > glustershd.log gets filled with a lot of entries. Load on the server >>> >> > with the then failed brick not that high, but still ~60. >>> >> > >>> >> > Is this behaviour normal? Is there some post-heal after a heal has >>> >> > finished? >>> >> > >>> >> > thx in advance :-) >>> > >>> > >>> > >>> > -- >>> > Pranith >> >> >> >> -- >> Pranith > > > > -- > Pranith-------------- next part -------------- Brick: gluster11:/gluster/bricksda1/shared ------------------------------------------ Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 4 23 46 No. of Writes: 27 68 130 Block Size: 8b+ 16b+ 32b+ No. of Reads: 87 162 386 No. of Writes: 252 496 1524 Block Size: 64b+ 128b+ 256b+ No. of Reads: 600 1236 2849 No. of Writes: 5862 18485 8234 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 8971 12202 22124 No. of Writes: 17856 89024 94131 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 52532 109761 340599 No. of Writes: 231908 2099445 401203 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 241543 1160765 3331181 No. of Writes: 843531 1576773 767822 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 2344788 FORGET 0.00 0.00 us 0.00 us 0.00 us 5473519 RELEASE 0.00 0.00 us 0.00 us 0.00 us 12395171 RELEASEDIR 0.00 51.62 us 17.00 us 468.00 us 32 STAT 0.00 697.67 us 349.00 us 1402.00 us 6 READDIR 0.00 100.25 us 60.00 us 802.00 us 170 LINK 0.00 31.47 us 12.00 us 1160.00 us 700 FINODELK 0.00 22.62 us 9.00 us 8936.00 us 1742 FLUSH 0.00 281.50 us 37.00 us 36034.00 us 165 SETATTR 0.00 62.46 us 16.00 us 2438.00 us 1430 INODELK 0.00 578.92 us 41.00 us 45998.00 us 202 SETXATTR 0.01 85.70 us 26.00 us 2733.00 us 1719 WRITE 0.01 52.99 us 9.00 us 7225.00 us 3546 ENTRYLK 0.01 27525.71 us 14.00 us 191408.00 us 7 GETXATTR 0.01 184.57 us 27.00 us 25950.00 us 1408 OPEN 0.01 737.37 us 16.00 us 150796.00 us 535 FSTAT 0.02 311.64 us 1.00 us 140788.00 us 1455 OPENDIR 0.03 750.04 us 15.00 us 547410.00 us 1159 STATFS 0.03 854.53 us 39.00 us 324138.00 us 1018 FXATTROP 0.03 4071.05 us 46.00 us 229415.00 us 236 UNLINK 0.05 23184.31 us 73.00 us 309197.00 us 55 RMDIR 0.06 6320.49 us 70.00 us 1485786.00 us 248 RENAME 0.30 41301.41 us 115.00 us 1561610.00 us 203 MKDIR 0.30 49189.99 us 124.00 us 451130.00 us 173 MKNOD 0.36 32907.19 us 43.00 us 1635027.00 us 309 READDIRP 0.61 51495.51 us 113.00 us 1682847.00 us 334 CREATE 1.60 30916.75 us 29.00 us 720507.00 us 1457 READ 96.56 30836.48 us 25.00 us 1586553.00 us 88261 LOOKUP Duration: 2739757 seconds Data Read: 564905511584 bytes Data Written: 315335673187 bytes Interval 2 Stats: Block Size: 16b+ 32b+ 64b+ No. of Reads: 0 0 0 No. of Writes: 1 2 2 Block Size: 128b+ 512b+ 1024b+ No. of Reads: 1 2 6 No. of Writes: 1 3 8 Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 6 10 31 No. of Writes: 20 44 428 Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 58 66 294 No. of Writes: 95 275 447 Block Size: 131072b+ No. of Reads: 866 No. of Writes: 171 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 455 FORGET 0.00 0.00 us 0.00 us 0.00 us 1701 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1278 RELEASEDIR 0.00 50.93 us 17.00 us 468.00 us 29 STAT 0.00 697.67 us 349.00 us 1402.00 us 6 READDIR 0.00 100.95 us 60.00 us 802.00 us 153 LINK 0.00 31.99 us 12.00 us 1160.00 us 616 FINODELK 0.00 22.63 us 9.00 us 8936.00 us 1701 FLUSH 0.00 300.61 us 37.00 us 36034.00 us 149 SETATTR 0.00 57.51 us 16.00 us 2438.00 us 1276 INODELK 0.00 639.78 us 41.00 us 45998.00 us 179 SETXATTR 0.00 84.76 us 28.00 us 2733.00 us 1497 WRITE 0.01 52.80 us 9.00 us 7225.00 us 3168 ENTRYLK 0.01 27525.71 us 14.00 us 191408.00 us 7 GETXATTR 0.01 184.57 us 27.00 us 25950.00 us 1408 OPEN 0.01 662.93 us 16.00 us 150796.00 us 490 FSTAT 0.02 345.44 us 1.00 us 140788.00 us 1279 OPENDIR 0.03 929.35 us 39.00 us 324138.00 us 912 FXATTROP 0.03 834.23 us 15.00 us 547410.00 us 1032 STATFS 0.04 4417.67 us 46.00 us 229415.00 us 215 UNLINK 0.05 23109.69 us 73.00 us 309197.00 us 51 RMDIR 0.06 7129.71 us 70.00 us 1485786.00 us 220 RENAME 0.20 28429.32 us 115.00 us 324214.00 us 179 MKDIR 0.32 51764.08 us 124.00 us 451130.00 us 156 MKNOD 0.36 34963.33 us 43.00 us 1635027.00 us 261 READDIRP 0.52 45042.62 us 113.00 us 729668.00 us 293 CREATE 1.68 31806.69 us 29.00 us 720507.00 us 1340 READ 96.64 31290.27 us 25.00 us 1506833.00 us 78465 LOOKUP Duration: 573 seconds Data Read: 145648404 bytes Data Written: 79382855 bytes Brick: gluster11:/gluster/bricksdb1/shared ------------------------------------------ Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 5 38 76 No. of Writes: 27 58 108 Block Size: 8b+ 16b+ 32b+ No. of Reads: 112 218 307 No. of Writes: 199 492 1437 Block Size: 64b+ 128b+ 256b+ No. of Reads: 565 1297 2861 No. of Writes: 5854 18833 7909 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 8627 12616 32357 No. of Writes: 17272 89783 93053 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 81797 82849 159042 No. of Writes: 231058 2091324 399860 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 242967 1158759 3336763 No. of Writes: 830475 1586621 802093 Block Size: 262144b+ 524288b+ No. of Reads: 1 1 No. of Writes: 0 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 2343190 FORGET 0.00 0.00 us 0.00 us 0.00 us 2483956 RELEASE 0.00 0.00 us 0.00 us 0.00 us 12395109 RELEASEDIR 0.00 41.40 us 22.00 us 90.00 us 5 GETXATTR 0.00 105.50 us 79.00 us 127.00 us 4 XATTROP 0.00 820.67 us 366.00 us 1674.00 us 6 READDIR 0.00 22.04 us 8.00 us 852.00 us 837 FLUSH 0.00 36.36 us 12.00 us 1092.00 us 706 FINODELK 0.00 147.54 us 37.00 us 8220.00 us 174 SETATTR 0.00 1855.09 us 19.00 us 57441.00 us 32 STAT 0.00 162.11 us 20.00 us 58558.00 us 525 FSTAT 0.00 88.73 us 26.00 us 3426.00 us 1485 WRITE 0.00 85.88 us 13.00 us 8276.00 us 1536 INODELK 0.01 391.52 us 24.00 us 24972.00 us 497 OPEN 0.01 105186.00 us 113.00 us 210259.00 us 2 TRUNCATE 0.01 59.46 us 10.00 us 12197.00 us 3680 ENTRYLK 0.01 1648.26 us 35.00 us 228719.00 us 202 SETXATTR 0.01 364.18 us 38.00 us 218867.00 us 1040 FXATTROP 0.02 402.53 us 1.00 us 216420.00 us 1455 OPENDIR 0.03 4985.97 us 60.00 us 680130.00 us 162 LINK 0.04 5440.45 us 50.00 us 303908.00 us 242 UNLINK 0.05 1247.74 us 16.00 us 442207.00 us 1159 STATFS 0.05 31279.73 us 74.00 us 373656.00 us 55 RMDIR 0.06 7484.67 us 61.00 us 1108625.00 us 266 RENAME 0.36 29970.54 us 42.00 us 738719.00 us 389 READDIRP 0.41 64512.28 us 103.00 us 1795277.00 us 203 MKDIR 0.44 77764.37 us 99.00 us 3311959.00 us 180 MKNOD 0.64 60153.29 us 91.00 us 933914.00 us 340 CREATE 1.64 33725.01 us 34.00 us 624710.00 us 1553 READ 96.20 34432.56 us 16.00 us 2017224.00 us 89254 LOOKUP Duration: 2739757 seconds Data Read: 562200652549 bytes Data Written: 320559185069 bytes Interval 2 Stats: Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 1 2 No. of Writes: 1 6 0 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2 4 10 No. of Writes: 2 19 11 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 16 24 40 No. of Writes: 52 417 91 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 81 298 950 No. of Writes: 195 382 185 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 407 FORGET 0.00 0.00 us 0.00 us 0.00 us 739 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1278 RELEASEDIR 0.00 41.40 us 22.00 us 90.00 us 5 GETXATTR 0.00 105.50 us 79.00 us 127.00 us 4 XATTROP 0.00 820.67 us 366.00 us 1674.00 us 6 READDIR 0.00 23.12 us 8.00 us 852.00 us 739 FLUSH 0.00 37.81 us 12.00 us 1092.00 us 620 FINODELK 0.00 151.42 us 37.00 us 8220.00 us 159 SETATTR 0.00 2039.48 us 19.00 us 57441.00 us 29 STAT 0.00 173.55 us 20.00 us 58558.00 us 481 FSTAT 0.00 549.73 us 35.00 us 26849.00 us 179 SETXATTR 0.00 83.22 us 13.00 us 8276.00 us 1376 INODELK 0.00 90.30 us 26.00 us 3426.00 us 1361 WRITE 0.01 435.64 us 28.00 us 24972.00 us 442 OPEN 0.01 60.27 us 10.00 us 12197.00 us 3294 ENTRYLK 0.01 105186.00 us 113.00 us 210259.00 us 2 TRUNCATE 0.01 403.77 us 38.00 us 218867.00 us 902 FXATTROP 0.02 432.04 us 1.00 us 216420.00 us 1279 OPENDIR 0.03 5473.30 us 63.00 us 680130.00 us 147 LINK 0.04 5631.87 us 50.00 us 303908.00 us 221 UNLINK 0.05 1382.59 us 16.00 us 442207.00 us 1032 STATFS 0.05 31009.37 us 74.00 us 373656.00 us 51 RMDIR 0.07 8420.41 us 61.00 us 1108625.00 us 236 RENAME 0.35 55808.69 us 103.00 us 554983.00 us 179 MKDIR 0.37 32525.43 us 42.00 us 738719.00 us 326 READDIRP 0.48 83770.63 us 99.00 us 3311959.00 us 164 MKNOD 0.62 60345.54 us 106.00 us 933914.00 us 297 CREATE 1.71 34434.86 us 34.00 us 624710.00 us 1428 READ 96.16 34900.81 us 16.00 us 2017224.00 us 79326 LOOKUP Duration: 573 seconds Data Read: 157139323 bytes Data Written: 73430641 bytes Brick: gluster11:/gluster/bricksdc1/shared ------------------------------------------ Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 3 50 38 No. of Writes: 30 81 111 Block Size: 8b+ 16b+ 32b+ No. of Reads: 44 147 349 No. of Writes: 220 426 1346 Block Size: 64b+ 128b+ 256b+ No. of Reads: 687 1276 2844 No. of Writes: 5639 18838 7886 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 8764 12476 91565 No. of Writes: 17414 89973 93464 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 42553 268572 255149 No. of Writes: 231649 2113323 398762 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 244169 1209170 3533625 No. of Writes: 833085 1597475 809580 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 3 1 1 No. of Writes: 3 0 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 2351853 FORGET 0.00 0.00 us 0.00 us 0.00 us 4338592 RELEASE 0.00 0.00 us 0.00 us 0.00 us 12395287 RELEASEDIR 0.00 110.50 us 101.00 us 120.00 us 2 XATTROP 0.00 389.00 us 389.00 us 389.00 us 1 TRUNCATE 0.00 144.14 us 17.00 us 228.00 us 7 GETXATTR 0.00 57.47 us 19.00 us 313.00 us 32 STAT 0.00 788.83 us 455.00 us 1261.00 us 6 READDIR 0.00 79.01 us 34.00 us 2396.00 us 173 SETATTR 0.00 41.77 us 13.00 us 1962.00 us 718 FINODELK 0.00 28.84 us 9.00 us 7826.00 us 1282 FLUSH 0.00 74.89 us 14.00 us 20179.00 us 1541 INODELK 0.00 84.96 us 24.00 us 2098.00 us 1569 WRITE 0.00 933.76 us 60.00 us 135280.00 us 164 LINK 0.00 48.35 us 8.00 us 4128.00 us 3678 ENTRYLK 0.02 579.39 us 43.00 us 203859.00 us 1108 FXATTROP 0.02 1479.30 us 20.00 us 209010.00 us 543 FSTAT 0.02 903.00 us 25.00 us 64579.00 us 943 OPEN 0.02 4386.01 us 33.00 us 390331.00 us 202 SETXATTR 0.03 926.68 us 15.00 us 289139.00 us 1159 STATFS 0.04 876.88 us 1.00 us 230680.00 us 1455 OPENDIR 0.06 8477.13 us 49.00 us 559560.00 us 241 UNLINK 0.08 10508.55 us 66.00 us 2478512.00 us 265 RENAME 0.11 70365.91 us 72.00 us 1673970.00 us 55 RMDIR 0.28 50279.27 us 101.00 us 1506163.00 us 203 MKDIR 0.29 58894.70 us 107.00 us 765427.00 us 178 MKNOD 0.40 35868.13 us 43.00 us 781724.00 us 403 READDIRP 0.51 54530.58 us 102.00 us 1318624.00 us 339 CREATE 1.87 40806.11 us 36.00 us 783714.00 us 1649 READ 96.22 39209.28 us 15.00 us 2135608.00 us 88293 LOOKUP Duration: 2739757 seconds Data Read: 599053082599 bytes Data Written: 323007303680 bytes Interval 2 Stats: Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 2 0 No. of Writes: 3 2 2 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 6 8 9 No. of Writes: 12 11 44 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 20 41 75 No. of Writes: 450 94 182 Block Size: 65536b+ 131072b+ No. of Reads: 331 1003 No. of Writes: 389 196 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 459 FORGET 0.00 0.00 us 0.00 us 0.00 us 1124 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1278 RELEASEDIR 0.00 110.50 us 101.00 us 120.00 us 2 XATTROP 0.00 389.00 us 389.00 us 389.00 us 1 TRUNCATE 0.00 144.14 us 17.00 us 228.00 us 7 GETXATTR 0.00 55.10 us 19.00 us 313.00 us 29 STAT 0.00 788.83 us 455.00 us 1261.00 us 6 READDIR 0.00 81.80 us 34.00 us 2396.00 us 157 SETATTR 0.00 43.03 us 13.00 us 1962.00 us 656 FINODELK 0.00 30.22 us 9.00 us 7826.00 us 1124 FLUSH 0.00 75.99 us 14.00 us 20179.00 us 1383 INODELK 0.00 85.05 us 24.00 us 2098.00 us 1385 WRITE 0.00 978.24 us 60.00 us 135280.00 us 153 LINK 0.00 46.64 us 8.00 us 4128.00 us 3344 ENTRYLK 0.02 1185.51 us 20.00 us 209010.00 us 498 FSTAT 0.02 624.16 us 43.00 us 203859.00 us 1016 FXATTROP 0.03 1037.94 us 25.00 us 64579.00 us 815 OPEN 0.03 4936.80 us 33.00 us 390331.00 us 179 SETXATTR 0.03 967.04 us 15.00 us 289139.00 us 1032 STATFS 0.03 869.43 us 1.00 us 230680.00 us 1279 OPENDIR 0.06 9113.05 us 49.00 us 559560.00 us 224 UNLINK 0.09 11592.38 us 66.00 us 2478512.00 us 240 RENAME 0.10 63148.55 us 72.00 us 1673970.00 us 51 RMDIR 0.24 43047.05 us 101.00 us 797954.00 us 179 MKDIR 0.30 61083.38 us 107.00 us 765427.00 us 160 MKNOD 0.39 38155.43 us 47.00 us 781724.00 us 336 READDIRP 0.52 54753.59 us 102.00 us 1318624.00 us 309 CREATE 1.87 40628.06 us 36.00 us 783714.00 us 1495 READ 96.26 39850.74 us 15.00 us 2135608.00 us 78539 LOOKUP Duration: 573 seconds Data Read: 167135536 bytes Data Written: 76661877 bytes Brick: gluster11:/gluster/bricksdd1/shared ------------------------------------------ Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 30 68 51 No. of Writes: 26 43 68 Block Size: 8b+ 16b+ 32b+ No. of Reads: 127 270 711 No. of Writes: 147 339 982 Block Size: 64b+ 128b+ 256b+ No. of Reads: 45937 38440 150143 No. of Writes: 3692 17208 6007 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 42492 862623 362777 No. of Writes: 14210 69991 80748 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 624029 1119114 2198331 No. of Writes: 206926 1788367 244242 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 1897227 2466872 7216571 No. of Writes: 461515 726707 1930832 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 161684 140769 112410 No. of Writes: 1 0 0 Block Size: 2097152b+ No. of Reads: 118521 No. of Writes: 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16247435 FORGET 0.00 0.00 us 0.00 us 0.00 us 17553038 RELEASE 0.00 0.00 us 0.00 us 0.00 us 23302990 RELEASEDIR 0.00 143.00 us 143.00 us 143.00 us 1 XATTROP 0.00 462.17 us 14.00 us 4797.00 us 12 GETXATTR 0.00 312.30 us 57.00 us 8805.00 us 167 LINK 0.00 413.08 us 34.00 us 14964.00 us 152 SETATTR 0.00 1223.67 us 33.00 us 151156.00 us 202 SETXATTR 0.00 489.94 us 7.00 us 21028.00 us 1374 FLUSH 0.00 169160.17 us 128.00 us 966235.00 us 6 READDIR 0.00 1156.19 us 8.00 us 36376.00 us 1180 FINODELK 0.00 1334.72 us 30.00 us 35530.00 us 1230 WRITE 0.00 11109.01 us 50.00 us 560354.00 us 222 UNLINK 0.00 10776.16 us 70.00 us 834677.00 us 234 RENAME 0.00 46699.47 us 72.00 us 612627.00 us 55 RMDIR 0.00 846.09 us 8.00 us 24840.00 us 3428 ENTRYLK 0.01 3666.50 us 11.00 us 3170599.00 us 1482 INODELK 0.01 204980.75 us 19.00 us 1211739.00 us 32 STAT 0.01 64174.80 us 115.00 us 875130.00 us 157 MKNOD 0.01 49753.23 us 103.00 us 1906516.00 us 203 MKDIR 0.02 60763.49 us 108.00 us 1298447.00 us 318 CREATE 0.11 65100.83 us 33.00 us 925001.00 us 1353 READ 0.13 100318.79 us 24.00 us 1780951.00 us 1058 OPEN 0.17 386621.02 us 50.00 us 5078133.00 us 345 READDIRP 0.44 243299.99 us 1.00 us 2793946.00 us 1454 OPENDIR 0.50 805671.42 us 25.00 us 5447049.00 us 500 FSTAT 0.68 475917.45 us 16.00 us 3964880.00 us 1155 STATFS 5.31 1850882.66 us 43.00 us 19684611.00 us 2305 FXATTROP 92.57 837528.51 us 17.00 us 6176236.00 us 88716 LOOKUP Duration: 2739126 seconds Data Read: 1933858814467 bytes Data Written: 366998246677 bytes Interval 2 Stats: Block Size: 128b+ 256b+ 512b+ No. of Reads: 0 0 1 No. of Writes: 1 2 1 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 8 13 7 No. of Writes: 3 6 43 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 17 44 73 No. of Writes: 319 48 63 Block Size: 65536b+ 131072b+ No. of Reads: 289 804 No. of Writes: 79 495 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 436 FORGET 0.00 0.00 us 0.00 us 0.00 us 1270 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1276 RELEASEDIR 0.00 143.00 us 143.00 us 143.00 us 1 XATTROP 0.00 462.17 us 14.00 us 4797.00 us 12 GETXATTR 0.00 293.07 us 57.00 us 8805.00 us 150 LINK 0.00 446.75 us 34.00 us 14964.00 us 138 SETATTR 0.00 1334.25 us 33.00 us 151156.00 us 179 SETXATTR 0.00 503.12 us 7.00 us 21028.00 us 1264 FLUSH 0.00 169160.17 us 128.00 us 966235.00 us 6 READDIR 0.00 1112.95 us 8.00 us 36376.00 us 1033 FINODELK 0.00 1253.04 us 31.00 us 32385.00 us 1060 WRITE 0.00 45056.63 us 72.00 us 612627.00 us 51 RMDIR 0.00 11130.30 us 70.00 us 834677.00 us 207 RENAME 0.00 11685.18 us 53.00 us 560354.00 us 201 UNLINK 0.00 831.87 us 8.00 us 24840.00 us 3059 ENTRYLK 0.01 165832.00 us 19.00 us 1211739.00 us 29 STAT 0.01 4003.48 us 11.00 us 3170599.00 us 1312 INODELK 0.01 41093.11 us 103.00 us 607790.00 us 179 MKDIR 0.01 62346.93 us 115.00 us 875130.00 us 144 MKNOD 0.02 49343.91 us 108.00 us 698323.00 us 281 CREATE 0.11 64436.11 us 33.00 us 925001.00 us 1256 READ 0.14 100480.31 us 24.00 us 1780951.00 us 987 OPEN 0.16 411991.64 us 50.00 us 5078133.00 us 291 READDIRP 0.44 251740.06 us 1.00 us 2793946.00 us 1279 OPENDIR 0.51 822516.08 us 25.00 us 5447049.00 us 458 FSTAT 0.70 499943.50 us 16.00 us 3964880.00 us 1029 STATFS 5.40 1922583.11 us 43.00 us 19684611.00 us 2057 FXATTROP 92.47 859387.75 us 17.00 us 6176236.00 us 78872 LOOKUP Duration: 573 seconds Data Read: 136840850 bytes Data Written: 79507233 bytes Brick: gluster13:/gluster/bricksdb1/shared ------------------------------------------ Cumulative Stats: Block Size: 2b+ 4b+ 8b+ No. of Reads: 1 1 3 No. of Writes: 1 2 7 Block Size: 16b+ 32b+ 64b+ No. of Reads: 4 11 19 No. of Writes: 10 41 207 Block Size: 128b+ 256b+ 512b+ No. of Reads: 47 81 264 No. of Writes: 806 253 584 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 854 776 1546 No. of Writes: 2651 2689 6919 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 2873 5142 8099 No. of Writes: 60164 12882 30029 Block Size: 65536b+ 131072b+ 262144b+ No. of Reads: 36630 107288 0 No. of Writes: 50949 26336 1 Block Size: 524288b+ No. of Reads: 0 No. of Writes: 3 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 67150 FORGET 0.00 0.00 us 0.00 us 0.00 us 76051 RELEASE 0.00 0.00 us 0.00 us 0.00 us 205997 RELEASEDIR 0.00 34.90 us 6.00 us 59.00 us 10 GETXATTR 0.00 96.75 us 87.00 us 117.00 us 4 XATTROP 0.00 157.50 us 95.00 us 416.00 us 6 READDIR 0.00 138.36 us 15.00 us 13062.00 us 173 STAT 0.00 35.88 us 13.00 us 917.00 us 716 FINODELK 0.00 33.88 us 7.00 us 9609.00 us 837 FLUSH 0.00 175.10 us 38.00 us 7447.00 us 174 SETATTR 0.00 309.38 us 33.00 us 31556.00 us 202 SETXATTR 0.00 69.19 us 14.00 us 4560.00 us 1536 INODELK 0.00 88.46 us 25.00 us 1036.00 us 1485 WRITE 0.01 980.67 us 62.00 us 80355.00 us 162 LINK 0.01 51.92 us 8.00 us 7563.00 us 3680 ENTRYLK 0.01 96038.50 us 131.00 us 191946.00 us 2 TRUNCATE 0.01 1314.41 us 71.00 us 151475.00 us 266 RENAME 0.01 901.46 us 31.00 us 47968.00 us 497 OPEN 0.01 308.76 us 1.00 us 110883.00 us 1454 OPENDIR 0.02 480.56 us 40.00 us 284617.00 us 1040 FXATTROP 0.02 1431.29 us 23.00 us 246838.00 us 502 FSTAT 0.02 3148.70 us 51.00 us 194760.00 us 242 UNLINK 0.03 675.77 us 15.00 us 134961.00 us 1159 STATFS 0.04 20830.80 us 76.00 us 365145.00 us 55 RMDIR 0.25 42080.72 us 100.00 us 749291.00 us 180 MKNOD 0.26 39434.13 us 103.00 us 758397.00 us 203 MKDIR 0.41 12001.52 us 36.00 us 691487.00 us 1043 READDIRP 0.41 37436.62 us 103.00 us 637111.00 us 340 CREATE 1.71 38444.66 us 33.00 us 605642.00 us 1372 READ 96.76 33398.16 us 4.00 us 1224156.00 us 89223 LOOKUP Duration: 92367 seconds Data Read: 18031828247 bytes Data Written: 10356957515 bytes Interval 2 Stats: Block Size: 64b+ 128b+ 512b+ No. of Reads: 0 0 0 No. of Writes: 1 6 2 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 2 5 15 No. of Writes: 19 11 52 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 21 42 60 No. of Writes: 417 91 195 Block Size: 65536b+ 131072b+ No. of Reads: 303 822 No. of Writes: 382 185 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 380 FORGET 0.00 0.00 us 0.00 us 0.00 us 739 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1278 RELEASEDIR 0.00 34.90 us 6.00 us 59.00 us 10 GETXATTR 0.00 96.75 us 87.00 us 117.00 us 4 XATTROP 0.00 157.50 us 95.00 us 416.00 us 6 READDIR 0.00 36.39 us 13.00 us 917.00 us 626 FINODELK 0.00 146.56 us 17.00 us 13062.00 us 156 STAT 0.00 36.02 us 7.00 us 9609.00 us 739 FLUSH 0.00 177.58 us 38.00 us 7447.00 us 159 SETATTR 0.00 340.23 us 33.00 us 31556.00 us 179 SETXATTR 0.00 68.33 us 14.00 us 4560.00 us 1376 INODELK 0.00 89.79 us 25.00 us 1036.00 us 1361 WRITE 0.01 46.95 us 8.00 us 3517.00 us 3294 ENTRYLK 0.01 1071.88 us 62.00 us 80355.00 us 147 LINK 0.01 96038.50 us 131.00 us 191946.00 us 2 TRUNCATE 0.01 1303.35 us 71.00 us 151475.00 us 236 RENAME 0.02 337.11 us 1.00 us 110883.00 us 1279 OPENDIR 0.02 1007.14 us 31.00 us 47968.00 us 442 OPEN 0.02 539.27 us 42.00 us 284617.00 us 902 FXATTROP 0.03 3241.54 us 51.00 us 194760.00 us 221 UNLINK 0.03 1609.87 us 25.00 us 246838.00 us 445 FSTAT 0.03 746.89 us 15.00 us 134961.00 us 1032 STATFS 0.04 20766.22 us 76.00 us 365145.00 us 51 RMDIR 0.25 41606.35 us 100.00 us 749291.00 us 164 MKNOD 0.27 42549.23 us 103.00 us 758397.00 us 179 MKDIR 0.36 10614.44 us 36.00 us 691487.00 us 930 READDIRP 0.42 39173.12 us 103.00 us 637111.00 us 297 CREATE 1.72 37553.72 us 33.00 us 605642.00 us 1270 READ 96.78 33892.06 us 4.00 us 1224156.00 us 79361 LOOKUP Duration: 573 seconds Data Read: 139543491 bytes Data Written: 73430641 bytes Brick: gluster13:/gluster/bricksda1/shared ------------------------------------------ Cumulative Stats: Block Size: 2b+ 4b+ 8b+ No. of Reads: 0 1 1 No. of Writes: 9 8 6 Block Size: 16b+ 32b+ 64b+ No. of Reads: 6 11 29 No. of Writes: 12 45 190 Block Size: 128b+ 256b+ 512b+ No. of Reads: 39 73 271 No. of Writes: 774 254 532 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 833 753 1491 No. of Writes: 2590 2687 6763 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 2981 5165 8076 No. of Writes: 60316 12474 28751 Block Size: 65536b+ 131072b+ 262144b+ No. of Reads: 37211 110473 0 No. of Writes: 48649 24723 1 Block Size: 524288b+ No. of Reads: 0 No. of Writes: 1 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 67031 FORGET 0.00 0.00 us 0.00 us 0.00 us 185052 RELEASE 0.00 0.00 us 0.00 us 0.00 us 206018 RELEASEDIR 0.00 59.10 us 6.00 us 263.00 us 10 GETXATTR 0.00 160.67 us 97.00 us 422.00 us 6 READDIR 0.00 57.01 us 32.00 us 367.00 us 165 SETATTR 0.00 144.99 us 14.00 us 15187.00 us 171 STAT 0.00 35.60 us 12.00 us 882.00 us 704 FINODELK 0.00 17.54 us 8.00 us 523.00 us 1742 FLUSH 0.00 226.13 us 64.00 us 25189.00 us 248 RENAME 0.00 65.58 us 12.00 us 3012.00 us 1430 INODELK 0.00 499.66 us 34.00 us 73141.00 us 202 SETXATTR 0.01 106.74 us 25.00 us 10466.00 us 1719 WRITE 0.01 54.28 us 8.00 us 7332.00 us 3546 ENTRYLK 0.01 1557.83 us 57.00 us 239298.00 us 170 LINK 0.01 301.96 us 41.00 us 180837.00 us 1018 FXATTROP 0.02 424.77 us 23.00 us 40643.00 us 1408 OPEN 0.03 1872.03 us 22.00 us 187148.00 us 490 FSTAT 0.03 812.47 us 15.00 us 131726.00 us 1159 STATFS 0.03 792.80 us 1.00 us 144988.00 us 1454 OPENDIR 0.04 5593.56 us 48.00 us 349101.00 us 236 UNLINK 0.04 25842.87 us 77.00 us 258710.00 us 55 RMDIR 0.20 38253.91 us 111.00 us 680459.00 us 173 MKNOD 0.29 46860.67 us 91.00 us 1565808.00 us 203 MKDIR 0.43 42989.39 us 98.00 us 944106.00 us 334 CREATE 0.54 16062.02 us 27.00 us 1932465.00 us 1108 READDIRP 1.69 48055.61 us 32.00 us 809523.00 us 1160 READ 96.62 36166.97 us 5.00 us 1536797.00 us 88200 LOOKUP Duration: 92367 seconds Data Read: 18500076400 bytes Data Written: 9875063816 bytes Interval 2 Stats: Block Size: 16b+ 32b+ 64b+ No. of Reads: 0 0 1 No. of Writes: 1 2 2 Block Size: 128b+ 512b+ 1024b+ No. of Reads: 1 1 4 No. of Writes: 1 3 8 Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 0 11 22 No. of Writes: 20 44 428 Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 30 70 302 No. of Writes: 95 275 447 Block Size: 131072b+ No. of Reads: 617 No. of Writes: 171 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 427 FORGET 0.00 0.00 us 0.00 us 0.00 us 1701 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1275 RELEASEDIR 0.00 88.50 us 37.00 us 263.00 us 6 GETXATTR 0.00 57.46 us 32.00 us 367.00 us 149 SETATTR 0.00 35.54 us 13.00 us 882.00 us 618 FINODELK 0.00 156.00 us 14.00 us 15187.00 us 154 STAT 0.00 17.48 us 8.00 us 523.00 us 1701 FLUSH 0.00 241.10 us 64.00 us 25189.00 us 220 RENAME 0.00 65.55 us 12.00 us 3012.00 us 1276 INODELK 0.00 536.02 us 34.00 us 73141.00 us 179 SETXATTR 0.01 104.42 us 25.00 us 10466.00 us 1497 WRITE 0.01 49.53 us 8.00 us 5423.00 us 3168 ENTRYLK 0.01 1718.32 us 57.00 us 239298.00 us 153 LINK 0.01 323.99 us 41.00 us 180837.00 us 912 FXATTROP 0.02 424.77 us 23.00 us 40643.00 us 1408 OPEN 0.02 1568.43 us 22.00 us 187148.00 us 445 FSTAT 0.03 627.92 us 19.00 us 134328.00 us 1276 OPENDIR 0.03 783.17 us 15.00 us 131726.00 us 1032 STATFS 0.04 23457.69 us 77.00 us 258710.00 us 51 RMDIR 0.04 6010.28 us 48.00 us 349101.00 us 215 UNLINK 0.21 40112.39 us 111.00 us 680459.00 us 156 MKNOD 0.31 50857.75 us 91.00 us 1565808.00 us 179 MKDIR 0.44 43844.91 us 98.00 us 944106.00 us 293 CREATE 0.49 14748.60 us 27.00 us 857674.00 us 978 READDIRP 1.69 47243.56 us 34.00 us 809523.00 us 1059 READ 96.64 36365.94 us 27.00 us 1536797.00 us 78472 LOOKUP Duration: 573 seconds Data Read: 113025697 bytes Data Written: 79382855 bytes Brick: gluster13:/gluster/bricksdd1_new/shared ---------------------------------------------- Cumulative Stats: Block Size: 2b+ 4b+ 8b+ No. of Reads: 0 1 9 No. of Writes: 1 2 2 Block Size: 16b+ 32b+ 64b+ No. of Reads: 5 10 22 No. of Writes: 3 22 100 Block Size: 128b+ 256b+ 512b+ No. of Reads: 52 85 282 No. of Writes: 648 142 399 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 884 763 1576 No. of Writes: 1758 2256 5550 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 2842 5281 7924 No. of Writes: 49740 7395 10830 Block Size: 65536b+ 131072b+ 262144b+ No. of Reads: 36143 115846 0 No. of Writes: 20263 80615 45 Block Size: 524288b+ 1048576b+ 2097152b+ No. of Reads: 0 0 0 No. of Writes: 30 17 16 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 67993 FORGET 0.00 0.00 us 0.00 us 0.00 us 114625 RELEASE 0.00 0.00 us 0.00 us 0.00 us 200165 RELEASEDIR 0.00 57.91 us 14.00 us 140.00 us 11 GETXATTR 0.00 139.20 us 36.00 us 3198.00 us 150 SETATTR 0.00 115839.83 us 114.00 us 692876.00 us 6 READDIR 0.00 507.80 us 7.00 us 34624.00 us 1373 FLUSH 0.00 3554.15 us 31.00 us 663793.00 us 202 SETXATTR 0.00 1188694.00 us 1188694.00 us 1188694.00 us 1 XATTROP 0.00 835.80 us 11.00 us 36286.00 us 1481 INODELK 0.00 1193.00 us 8.00 us 54933.00 us 1268 FINODELK 0.00 8813.80 us 47.00 us 364686.00 us 222 UNLINK 0.00 811.83 us 6.00 us 44721.00 us 3430 ENTRYLK 0.00 58652.45 us 71.00 us 1023311.00 us 55 RMDIR 0.00 2725.98 us 26.00 us 608635.00 us 1230 WRITE 0.01 31858.56 us 106.00 us 770392.00 us 157 MKNOD 0.01 22657.84 us 70.00 us 1812912.00 us 235 RENAME 0.01 33418.32 us 57.00 us 1862383.00 us 167 LINK 0.02 58044.30 us 89.00 us 1552491.00 us 203 MKDIR 0.02 48356.09 us 101.00 us 1537756.00 us 317 CREATE 0.09 379025.96 us 14.00 us 3128177.00 us 171 STAT 0.15 96009.18 us 20.00 us 2163025.00 us 1059 OPEN 0.26 113257.56 us 16.00 us 1910193.00 us 1573 READ 0.30 204565.68 us 33.00 us 2788769.00 us 1021 READDIRP 0.45 577226.44 us 13.00 us 2737901.00 us 537 FSTAT 0.46 219151.28 us 1.00 us 3687701.00 us 1455 OPENDIR 0.64 384261.01 us 14.00 us 2961007.00 us 1157 STATFS 4.24 1281429.92 us 45.00 us 22052627.00 us 2300 FXATTROP 93.33 730726.57 us 7.00 us 5885721.00 us 88809 LOOKUP Duration: 89555 seconds Data Read: 19096790737 bytes Data Written: 13829796557 bytes Interval 2 Stats: Block Size: 64b+ 128b+ 256b+ No. of Reads: 2 3 1 No. of Writes: 0 1 2 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3 9 9 No. of Writes: 1 3 6 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 10 27 49 No. of Writes: 43 319 48 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 67 291 960 No. of Writes: 63 79 495 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 406 FORGET 0.00 0.00 us 0.00 us 0.00 us 1270 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1276 RELEASEDIR 0.00 57.91 us 14.00 us 140.00 us 11 GETXATTR 0.00 124.68 us 36.00 us 3198.00 us 138 SETATTR 0.00 509.20 us 7.00 us 34624.00 us 1264 FLUSH 0.00 115839.83 us 114.00 us 692876.00 us 6 READDIR 0.00 4003.32 us 32.00 us 663793.00 us 179 SETXATTR 0.00 756.04 us 11.00 us 36286.00 us 1312 INODELK 0.00 1030.90 us 8.00 us 24606.00 us 1096 FINODELK 0.00 1188694.00 us 1188694.00 us 1188694.00 us 1 XATTROP 0.00 8933.85 us 49.00 us 364686.00 us 201 UNLINK 0.00 800.29 us 6.00 us 44721.00 us 3061 ENTRYLK 0.00 12335.48 us 70.00 us 1812912.00 us 208 RENAME 0.00 60380.94 us 71.00 us 1023311.00 us 51 RMDIR 0.01 3091.23 us 31.00 us 608635.00 us 1060 WRITE 0.01 30230.15 us 57.00 us 1862383.00 us 150 LINK 0.01 32657.23 us 108.00 us 770392.00 us 144 MKNOD 0.02 58913.75 us 89.00 us 1552491.00 us 179 MKDIR 0.02 45790.00 us 101.00 us 1537756.00 us 281 CREATE 0.10 394494.63 us 14.00 us 3128177.00 us 154 STAT 0.15 96886.65 us 20.00 us 2163025.00 us 988 OPEN 0.26 112537.11 us 16.00 us 1910193.00 us 1435 READ 0.30 207173.94 us 33.00 us 2788769.00 us 902 READDIRP 0.44 218473.48 us 1.00 us 3687701.00 us 1280 OPENDIR 0.46 593604.47 us 13.00 us 2737901.00 us 486 FSTAT 0.64 392722.50 us 14.00 us 2961007.00 us 1031 STATFS 4.23 1293852.85 us 45.00 us 22052627.00 us 2061 FXATTROP 93.35 745335.50 us 7.00 us 5885721.00 us 79033 LOOKUP Duration: 573 seconds Data Read: 157973480 bytes Data Written: 79507233 bytes Brick: gluster13:/gluster/bricksdc1/shared ------------------------------------------ Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 1 1 4 No. of Writes: 2 3 1 Block Size: 8b+ 16b+ 32b+ No. of Reads: 0 6 13 No. of Writes: 6 9 40 Block Size: 64b+ 128b+ 256b+ No. of Reads: 23 42 88 No. of Writes: 200 725 246 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 309 900 810 No. of Writes: 532 2620 2654 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 1518 2779 5125 No. of Writes: 6953 60644 12838 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 8173 40897 117337 No. of Writes: 30653 50676 27373 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 4 1 2 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 67398 FORGET 0.00 0.00 us 0.00 us 0.00 us 149493 RELEASE 0.00 0.00 us 0.00 us 0.00 us 206015 RELEASEDIR 0.00 99.00 us 82.00 us 116.00 us 2 XATTROP 0.00 36.90 us 10.00 us 64.00 us 10 GETXATTR 0.00 382.00 us 382.00 us 382.00 us 1 TRUNCATE 0.00 193.67 us 109.00 us 532.00 us 6 READDIR 0.00 117.35 us 33.00 us 1460.00 us 202 SETXATTR 0.00 38.92 us 12.00 us 1097.00 us 722 FINODELK 0.00 24.83 us 6.00 us 3878.00 us 1282 FLUSH 0.00 469.80 us 15.00 us 69152.00 us 174 STAT 0.00 66.22 us 13.00 us 4392.00 us 1541 INODELK 0.00 104.83 us 27.00 us 11654.00 us 1569 WRITE 0.01 192.17 us 20.00 us 35121.00 us 943 OPEN 0.01 72.84 us 8.00 us 52122.00 us 3678 ENTRYLK 0.01 627.99 us 15.00 us 85448.00 us 483 FSTAT 0.01 1941.11 us 32.00 us 323367.00 us 173 SETATTR 0.02 534.98 us 15.00 us 225607.00 us 1159 STATFS 0.02 512.21 us 1.00 us 117639.00 us 1454 OPENDIR 0.02 4611.68 us 55.00 us 634364.00 us 164 LINK 0.02 3388.79 us 50.00 us 143866.00 us 241 UNLINK 0.07 41091.56 us 77.00 us 661708.00 us 55 RMDIR 0.09 11185.69 us 60.00 us 2896998.00 us 265 RENAME 0.13 4000.70 us 42.00 us 382520.00 us 1108 FXATTROP 0.26 44306.56 us 87.00 us 2933023.00 us 203 MKDIR 0.34 64964.70 us 111.00 us 855656.00 us 178 MKNOD 0.39 39025.22 us 95.00 us 660183.00 us 339 CREATE 0.43 13765.14 us 33.00 us 734381.00 us 1064 READDIRP 1.65 45941.31 us 28.00 us 530561.00 us 1221 READ 96.50 37123.26 us 8.00 us 2489216.00 us 88247 LOOKUP Duration: 92367 seconds Data Read: 19888635900 bytes Data Written: 10512483717 bytes Interval 2 Stats: Block Size: 64b+ 128b+ 256b+ No. of Reads: 1 1 0 No. of Writes: 3 2 2 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3 14 9 No. of Writes: 0 12 11 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 12 16 36 No. of Writes: 44 450 94 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 62 272 647 No. of Writes: 182 389 196 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 441 FORGET 0.00 0.00 us 0.00 us 0.00 us 1124 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1278 RELEASEDIR 0.00 99.00 us 82.00 us 116.00 us 2 XATTROP 0.00 36.90 us 10.00 us 64.00 us 10 GETXATTR 0.00 382.00 us 382.00 us 382.00 us 1 TRUNCATE 0.00 193.67 us 109.00 us 532.00 us 6 READDIR 0.00 113.18 us 33.00 us 1460.00 us 179 SETXATTR 0.00 38.87 us 12.00 us 1097.00 us 658 FINODELK 0.00 22.79 us 6.00 us 1746.00 us 1124 FLUSH 0.00 517.46 us 16.00 us 69152.00 us 157 STAT 0.00 61.05 us 13.00 us 4392.00 us 1383 INODELK 0.00 107.91 us 27.00 us 11654.00 us 1385 WRITE 0.01 215.98 us 20.00 us 35121.00 us 815 OPEN 0.01 73.16 us 8.00 us 52122.00 us 3344 ENTRYLK 0.01 688.31 us 15.00 us 85448.00 us 438 FSTAT 0.01 2133.62 us 32.00 us 323367.00 us 157 SETATTR 0.02 580.70 us 15.00 us 225607.00 us 1032 STATFS 0.02 536.47 us 1.00 us 117639.00 us 1279 OPENDIR 0.02 4935.99 us 55.00 us 634364.00 us 153 LINK 0.02 3399.37 us 50.00 us 143866.00 us 224 UNLINK 0.07 42437.00 us 77.00 us 661708.00 us 51 RMDIR 0.10 12289.22 us 60.00 us 2896998.00 us 240 RENAME 0.14 4352.15 us 42.00 us 382520.00 us 1016 FXATTROP 0.27 45958.13 us 87.00 us 2933023.00 us 179 MKDIR 0.34 66369.21 us 111.00 us 855656.00 us 160 MKNOD 0.38 38324.23 us 95.00 us 660183.00 us 309 CREATE 0.43 14170.13 us 33.00 us 734381.00 us 938 READDIRP 1.67 47878.64 us 28.00 us 530561.00 us 1073 READ 96.46 37854.72 us 8.00 us 2489216.00 us 78562 LOOKUP Duration: 573 seconds Data Read: 113932210 bytes Data Written: 76661877 bytes Brick: gluster12:/gluster/bricksda1/shared ------------------------------------------ Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 3 7 21 No. of Writes: 27 67 126 Block Size: 8b+ 16b+ 32b+ No. of Reads: 87 108 277 No. of Writes: 250 488 1506 Block Size: 64b+ 128b+ 256b+ No. of Reads: 727 1372 2541 No. of Writes: 5771 18241 8058 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 8573 11834 21264 No. of Writes: 17415 87159 92183 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 113222 86044 146012 No. of Writes: 227640 2030483 390792 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 236433 1133314 3256543 No. of Writes: 820892 1537412 743425 Block Size: 262144b+ 524288b+ No. of Reads: 1 1 No. of Writes: 0 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 2274337 FORGET 0.00 0.00 us 0.00 us 0.00 us 5344669 RELEASE 0.00 0.00 us 0.00 us 0.00 us 13376115 RELEASEDIR 0.00 44.75 us 19.00 us 100.00 us 8 GETXATTR 0.00 377.33 us 251.00 us 770.00 us 6 READDIR 0.00 45.51 us 15.00 us 164.00 us 112 STAT 0.00 75.85 us 34.00 us 128.00 us 165 SETATTR 0.00 36.34 us 12.00 us 103.00 us 694 FINODELK 0.00 26.32 us 8.00 us 105.00 us 1742 FLUSH 0.00 43.55 us 14.00 us 393.00 us 1430 INODELK 0.00 491.26 us 60.00 us 39670.00 us 248 RENAME 0.00 38.04 us 7.00 us 236.00 us 3546 ENTRYLK 0.01 112.08 us 23.00 us 679.00 us 1719 WRITE 0.01 214.32 us 22.00 us 35145.00 us 1408 OPEN 0.01 382.70 us 43.00 us 97879.00 us 1018 FXATTROP 0.02 4538.56 us 56.00 us 628768.00 us 170 LINK 0.03 1720.91 us 13.00 us 204537.00 us 498 FSTAT 0.03 800.00 us 13.00 us 148610.00 us 1159 STATFS 0.03 5736.05 us 35.00 us 563247.00 us 202 SETXATTR 0.05 1142.27 us 2.00 us 207392.00 us 1454 OPENDIR 0.06 9060.71 us 49.00 us 442578.00 us 236 UNLINK 0.07 41283.67 us 68.00 us 320626.00 us 55 RMDIR 0.32 53265.48 us 92.00 us 1375178.00 us 203 MKDIR 0.33 64772.78 us 100.00 us 1082540.00 us 173 MKNOD 0.41 17276.59 us 37.00 us 747774.00 us 797 READDIRP 0.58 59422.85 us 99.00 us 1005489.00 us 334 CREATE 1.76 46128.36 us 25.00 us 624482.00 us 1297 READ 96.28 37092.87 us 11.00 us 1805931.00 us 88219 LOOKUP Duration: 2671916 seconds Data Read: 548822336953 bytes Data Written: 306583588574 bytes Interval 2 Stats: Block Size: 16b+ 32b+ 64b+ No. of Reads: 0 0 0 No. of Writes: 1 2 2 Block Size: 128b+ 256b+ 512b+ No. of Reads: 0 1 2 No. of Writes: 1 0 3 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 5 9 9 No. of Writes: 8 20 44 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 22 46 62 No. of Writes: 428 95 275 Block Size: 65536b+ 131072b+ No. of Reads: 298 696 No. of Writes: 447 171 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 446 FORGET 0.00 0.00 us 0.00 us 0.00 us 1701 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1278 RELEASEDIR 0.00 44.75 us 19.00 us 100.00 us 8 GETXATTR 0.00 377.33 us 251.00 us 770.00 us 6 READDIR 0.00 45.49 us 15.00 us 164.00 us 100 STAT 0.00 76.28 us 34.00 us 128.00 us 149 SETATTR 0.00 36.24 us 12.00 us 103.00 us 610 FINODELK 0.00 26.28 us 8.00 us 105.00 us 1701 FLUSH 0.00 43.39 us 14.00 us 393.00 us 1276 INODELK 0.00 537.28 us 60.00 us 39670.00 us 220 RENAME 0.00 38.15 us 7.00 us 236.00 us 3168 ENTRYLK 0.01 109.62 us 23.00 us 679.00 us 1497 WRITE 0.01 214.32 us 22.00 us 35145.00 us 1408 OPEN 0.01 393.08 us 43.00 us 97879.00 us 912 FXATTROP 0.03 5028.49 us 56.00 us 628768.00 us 153 LINK 0.03 1791.15 us 13.00 us 204537.00 us 457 FSTAT 0.03 892.39 us 13.00 us 148610.00 us 1032 STATFS 0.04 6456.17 us 35.00 us 563247.00 us 179 SETXATTR 0.04 1001.33 us 2.00 us 185595.00 us 1279 OPENDIR 0.07 40199.57 us 90.00 us 320626.00 us 51 RMDIR 0.07 9898.99 us 49.00 us 442578.00 us 215 UNLINK 0.29 49429.69 us 101.00 us 490185.00 us 179 MKDIR 0.34 66794.53 us 100.00 us 1082540.00 us 156 MKNOD 0.39 16647.41 us 37.00 us 747774.00 us 706 READDIRP 0.61 62792.22 us 99.00 us 1005489.00 us 293 CREATE 1.80 47470.78 us 25.00 us 624482.00 us 1150 READ 96.22 37128.10 us 11.00 us 1805931.00 us 78488 LOOKUP Duration: 573 seconds Data Read: 124126008 bytes Data Written: 79382855 bytes Brick: gluster12:/gluster/bricksdb1/shared ------------------------------------------ Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 6 33 49 No. of Writes: 27 56 107 Block Size: 8b+ 16b+ 32b+ No. of Reads: 94 200 312 No. of Writes: 195 485 1410 Block Size: 64b+ 128b+ 256b+ No. of Reads: 630 1461 2983 No. of Writes: 5773 18591 7735 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 8402 12341 26650 No. of Writes: 16778 87884 91099 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 42375 189746 239259 No. of Writes: 226843 2023248 388982 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 236595 1135093 3521486 No. of Writes: 806697 1546538 775293 Block Size: 524288b+ No. of Reads: 2 No. of Writes: 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 2271301 FORGET 0.00 0.00 us 0.00 us 0.00 us 2414775 RELEASE 0.00 0.00 us 0.00 us 0.00 us 13251269 RELEASEDIR 0.00 122.75 us 56.00 us 227.00 us 4 XATTROP 0.00 53.80 us 13.00 us 102.00 us 10 GETXATTR 0.00 323.67 us 197.00 us 727.00 us 6 READDIR 0.00 102.46 us 15.00 us 6719.00 us 111 STAT 0.00 77.79 us 33.00 us 216.00 us 174 SETATTR 0.00 23.54 us 8.00 us 85.00 us 837 FLUSH 0.00 35.86 us 12.00 us 86.00 us 710 FINODELK 0.00 48.75 us 12.00 us 11261.00 us 1536 INODELK 0.00 37.57 us 7.00 us 1785.00 us 3680 ENTRYLK 0.00 302.59 us 26.00 us 32486.00 us 497 OPEN 0.01 113.50 us 27.00 us 292.00 us 1485 WRITE 0.01 131319.50 us 97.00 us 262542.00 us 2 TRUNCATE 0.02 927.44 us 15.00 us 136281.00 us 543 FSTAT 0.02 364.87 us 2.00 us 159594.00 us 1454 OPENDIR 0.02 512.43 us 33.00 us 154535.00 us 1040 FXATTROP 0.03 775.13 us 14.00 us 111798.00 us 1159 STATFS 0.06 39010.53 us 67.00 us 1008643.00 us 55 RMDIR 0.07 9683.62 us 48.00 us 807197.00 us 242 UNLINK 0.07 8904.95 us 62.00 us 957415.00 us 266 RENAME 0.10 21092.46 us 53.00 us 3334761.00 us 162 LINK 0.10 17193.10 us 34.00 us 3455193.00 us 202 SETXATTR 0.35 11671.79 us 34.00 us 755573.00 us 1006 READDIRP 0.38 70264.52 us 101.00 us 2017829.00 us 180 MKNOD 0.45 74099.28 us 94.00 us 2736488.00 us 203 MKDIR 0.69 66975.51 us 98.00 us 2346509.00 us 340 CREATE 1.71 36065.24 us 33.00 us 832660.00 us 1573 READ 95.90 35673.07 us 22.00 us 3442363.00 us 89270 LOOKUP Duration: 2671916 seconds Data Read: 586700928256 bytes Data Written: 311358279278 bytes Interval 2 Stats: Block Size: 64b+ 128b+ 512b+ No. of Reads: 0 2 4 No. of Writes: 1 6 2 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 4 7 11 No. of Writes: 19 11 52 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 23 39 70 No. of Writes: 417 91 195 Block Size: 65536b+ 131072b+ No. of Reads: 334 999 No. of Writes: 382 185 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 394 FORGET 0.00 0.00 us 0.00 us 0.00 us 739 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1278 RELEASEDIR 0.00 122.75 us 56.00 us 227.00 us 4 XATTROP 0.00 53.80 us 13.00 us 102.00 us 10 GETXATTR 0.00 323.67 us 197.00 us 727.00 us 6 READDIR 0.00 110.30 us 15.00 us 6719.00 us 99 STAT 0.00 76.99 us 33.00 us 144.00 us 159 SETATTR 0.00 89.08 us 34.00 us 1295.00 us 179 SETXATTR 0.00 23.97 us 8.00 us 85.00 us 739 FLUSH 0.00 35.96 us 12.00 us 86.00 us 622 FINODELK 0.00 49.13 us 12.00 us 11261.00 us 1376 INODELK 0.00 548.42 us 53.00 us 61956.00 us 147 LINK 0.00 37.44 us 8.00 us 1785.00 us 3294 ENTRYLK 0.00 335.30 us 27.00 us 32486.00 us 442 OPEN 0.01 115.17 us 27.00 us 292.00 us 1361 WRITE 0.01 131319.50 us 97.00 us 262542.00 us 2 TRUNCATE 0.02 353.77 us 2.00 us 159594.00 us 1279 OPENDIR 0.02 997.75 us 15.00 us 136281.00 us 502 FSTAT 0.02 568.25 us 33.00 us 154535.00 us 902 FXATTROP 0.03 790.72 us 14.00 us 111798.00 us 1032 STATFS 0.07 40222.00 us 67.00 us 1008643.00 us 51 RMDIR 0.08 10575.04 us 48.00 us 807197.00 us 221 UNLINK 0.08 10012.92 us 62.00 us 957415.00 us 236 RENAME 0.29 48994.55 us 94.00 us 1430184.00 us 179 MKDIR 0.35 11719.52 us 34.00 us 755573.00 us 890 READDIRP 0.41 74290.62 us 101.00 us 2017829.00 us 164 MKNOD 0.52 52039.70 us 98.00 us 1335975.00 us 297 CREATE 1.79 35819.73 us 33.00 us 832660.00 us 1493 READ 96.31 36222.16 us 22.00 us 1928696.00 us 79367 LOOKUP Duration: 573 seconds Data Read: 166300930 bytes Data Written: 73430641 bytes Brick: gluster12:/gluster/bricksdc1/shared ------------------------------------------ Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 13 26 36 No. of Writes: 28 79 111 Block Size: 8b+ 16b+ 32b+ No. of Reads: 82 147 315 No. of Writes: 217 418 1321 Block Size: 64b+ 128b+ 256b+ No. of Reads: 679 1325 2582 No. of Writes: 5561 18597 7697 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 8751 12053 21737 No. of Writes: 16905 88077 91467 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 41616 80941 157855 No. of Writes: 227214 2041353 387993 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 238670 1129305 3279369 No. of Writes: 810613 1558177 784456 Block Size: 262144b+ 1048576b+ No. of Reads: 1 1 No. of Writes: 3 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 2281926 FORGET 0.00 0.00 us 0.00 us 0.00 us 4218326 RELEASE 0.00 0.00 us 0.00 us 0.00 us 13572978 RELEASEDIR 0.00 188.50 us 152.00 us 225.00 us 2 XATTROP 0.00 63.71 us 33.00 us 113.00 us 7 GETXATTR 0.00 476.00 us 476.00 us 476.00 us 1 TRUNCATE 0.00 453.67 us 275.00 us 859.00 us 6 READDIR 0.00 40.58 us 15.00 us 168.00 us 109 STAT 0.00 36.13 us 12.00 us 90.00 us 716 FINODELK 0.00 22.28 us 7.00 us 482.00 us 1282 FLUSH 0.00 41.82 us 12.00 us 623.00 us 1541 INODELK 0.00 139.21 us 23.00 us 37082.00 us 943 OPEN 0.00 36.84 us 9.00 us 279.00 us 3678 ENTRYLK 0.01 646.07 us 59.00 us 68054.00 us 265 RENAME 0.01 120.02 us 23.00 us 837.00 us 1569 WRITE 0.01 1103.93 us 34.00 us 196372.00 us 202 SETXATTR 0.01 201.51 us 15.00 us 60778.00 us 1159 STATFS 0.01 585.85 us 19.00 us 111958.00 us 514 FSTAT 0.01 279.08 us 2.00 us 62138.00 us 1454 OPENDIR 0.01 2496.55 us 31.00 us 418721.00 us 173 SETATTR 0.02 3278.91 us 55.00 us 421679.00 us 164 LINK 0.03 801.93 us 46.00 us 302679.00 us 1108 FXATTROP 0.03 3793.83 us 45.00 us 257743.00 us 241 UNLINK 0.05 29446.55 us 80.00 us 333212.00 us 55 RMDIR 0.27 39254.36 us 93.00 us 595453.00 us 203 MKDIR 0.31 51748.69 us 101.00 us 540932.00 us 178 MKNOD 0.41 11840.57 us 33.00 us 658333.00 us 1029 READDIRP 0.64 56314.76 us 100.00 us 1507447.00 us 339 CREATE 1.57 35413.64 us 32.00 us 509326.00 us 1326 READ 96.61 32823.06 us 25.00 us 1544052.00 us 88284 LOOKUP Duration: 2671916 seconds Data Read: 551381965668 bytes Data Written: 314134202776 bytes Interval 2 Stats: Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 1 1 No. of Writes: 3 2 2 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2 6 3 No. of Writes: 0 12 11 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 7 16 42 No. of Writes: 44 450 94 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 72 306 766 No. of Writes: 182 389 196 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 449 FORGET 0.00 0.00 us 0.00 us 0.00 us 1124 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1278 RELEASEDIR 0.00 188.50 us 152.00 us 225.00 us 2 XATTROP 0.00 63.71 us 33.00 us 113.00 us 7 GETXATTR 0.00 476.00 us 476.00 us 476.00 us 1 TRUNCATE 0.00 453.67 us 275.00 us 859.00 us 6 READDIR 0.00 39.63 us 15.00 us 168.00 us 98 STAT 0.00 36.03 us 12.00 us 90.00 us 654 FINODELK 0.00 22.67 us 7.00 us 482.00 us 1124 FLUSH 0.00 41.73 us 12.00 us 623.00 us 1383 INODELK 0.00 36.73 us 9.00 us 237.00 us 3344 ENTRYLK 0.00 153.20 us 23.00 us 37082.00 us 815 OPEN 0.01 678.75 us 59.00 us 68054.00 us 240 RENAME 0.01 120.39 us 23.00 us 837.00 us 1385 WRITE 0.01 1180.55 us 34.00 us 196372.00 us 179 SETXATTR 0.01 207.78 us 15.00 us 60778.00 us 1032 STATFS 0.01 228.32 us 2.00 us 62138.00 us 1279 OPENDIR 0.01 638.47 us 19.00 us 111958.00 us 467 FSTAT 0.02 2743.67 us 31.00 us 418721.00 us 157 SETATTR 0.02 3505.03 us 55.00 us 421679.00 us 153 LINK 0.03 3832.47 us 45.00 us 257743.00 us 224 UNLINK 0.03 860.06 us 46.00 us 302679.00 us 1016 FXATTROP 0.05 25534.71 us 80.00 us 333212.00 us 51 RMDIR 0.27 40137.07 us 93.00 us 595453.00 us 179 MKDIR 0.31 52768.29 us 101.00 us 540932.00 us 160 MKNOD 0.38 11260.86 us 33.00 us 652794.00 us 914 READDIRP 0.66 57339.62 us 100.00 us 1507447.00 us 309 CREATE 1.57 34542.64 us 32.00 us 509326.00 us 1222 READ 96.60 33122.80 us 25.00 us 1544052.00 us 78558 LOOKUP Duration: 573 seconds Data Read: 133051044 bytes Data Written: 76661877 bytes Brick: gluster12:/gluster/bricksdd1_new/shared ---------------------------------------------- Cumulative Stats: Block Size: 1b+ 2b+ 4b+ No. of Reads: 7 36 25 No. of Writes: 29 59 90 Block Size: 8b+ 16b+ 32b+ No. of Reads: 26 116 312 No. of Writes: 200 431 1314 Block Size: 64b+ 128b+ 256b+ No. of Reads: 591 1033 2464 No. of Writes: 90771 88347 289700 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 6624 10299 17539 No. of Writes: 62877 656288 478799 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 42737 63962 115139 No. of Writes: 825596 2955354 2423568 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 186515 898809 2611338 No. of Writes: 2087776 2297998 5521485 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 20 16 6 No. of Writes: 296499 261174 219355 Block Size: 2097152b+ No. of Reads: 1 No. of Writes: 227764 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 27036608 FORGET 0.00 0.00 us 0.00 us 0.00 us 17439197 RELEASE 0.00 0.00 us 0.00 us 0.00 us 23286594 RELEASEDIR 0.00 201.00 us 201.00 us 201.00 us 1 XATTROP 0.00 66.25 us 41.00 us 99.00 us 8 GETXATTR 0.00 363.17 us 308.00 us 519.00 us 6 READDIR 0.00 43.74 us 18.00 us 172.00 us 110 STAT 0.00 41.05 us 15.00 us 109.00 us 659 FINODELK 0.00 63.79 us 21.00 us 155.00 us 475 FSTAT 0.00 27.98 us 8.00 us 727.00 us 1372 FLUSH 0.00 43.89 us 14.00 us 670.00 us 1482 INODELK 0.00 85.11 us 25.00 us 17111.00 us 1058 OPEN 0.01 39.02 us 7.00 us 1889.00 us 3432 ENTRYLK 0.01 223.37 us 13.00 us 140488.00 us 1159 STATFS 0.01 1151.06 us 66.00 us 117375.00 us 235 RENAME 0.01 2028.70 us 35.00 us 109373.00 us 150 SETATTR 0.02 317.23 us 2.00 us 147686.00 us 1454 OPENDIR 0.02 2785.28 us 35.00 us 390871.00 us 202 SETXATTR 0.03 478.01 us 26.00 us 333031.00 us 1229 WRITE 0.06 591.92 us 37.00 us 333853.00 us 2308 FXATTROP 0.07 7633.68 us 45.00 us 821788.00 us 222 UNLINK 0.09 12430.85 us 60.00 us 453180.00 us 167 LINK 0.11 43320.51 us 66.00 us 687523.00 us 55 RMDIR 0.22 24393.20 us 97.00 us 753920.00 us 203 MKDIR 0.23 32908.52 us 94.00 us 582705.00 us 157 MKNOD 0.43 30866.61 us 94.00 us 784122.00 us 317 CREATE 0.66 14557.10 us 31.00 us 1065265.00 us 1029 READDIRP 3.86 65513.70 us 34.00 us 1005239.00 us 1336 READ 94.14 24031.53 us 14.00 us 1671808.00 us 88819 LOOKUP Duration: 2668907 seconds Data Read: 438736244479 bytes Data Written: 2272016832701 bytes Interval 2 Stats: Block Size: 64b+ 128b+ 256b+ No. of Reads: 1 2 0 No. of Writes: 0 1 2 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3 7 6 No. of Writes: 1 3 6 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 15 14 49 No. of Writes: 43 318 48 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 61 267 776 No. of Writes: 63 79 495 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 424 FORGET 0.00 0.00 us 0.00 us 0.00 us 1270 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1278 RELEASEDIR 0.00 201.00 us 201.00 us 201.00 us 1 XATTROP 0.00 66.25 us 41.00 us 99.00 us 8 GETXATTR 0.00 363.17 us 308.00 us 519.00 us 6 READDIR 0.00 43.97 us 18.00 us 172.00 us 99 STAT 0.00 41.07 us 15.00 us 109.00 us 583 FINODELK 0.00 63.41 us 21.00 us 155.00 us 428 FSTAT 0.00 27.77 us 8.00 us 727.00 us 1264 FLUSH 0.00 43.87 us 14.00 us 670.00 us 1313 INODELK 0.00 85.89 us 25.00 us 17111.00 us 988 OPEN 0.01 38.65 us 7.00 us 1889.00 us 3060 ENTRYLK 0.01 652.64 us 66.00 us 100441.00 us 208 RENAME 0.01 245.56 us 13.00 us 140488.00 us 1032 STATFS 0.01 2197.16 us 35.00 us 109373.00 us 138 SETATTR 0.02 353.42 us 2.00 us 147686.00 us 1279 OPENDIR 0.03 529.00 us 26.00 us 333031.00 us 1059 WRITE 0.03 3133.22 us 35.00 us 390871.00 us 179 SETXATTR 0.06 637.80 us 37.00 us 333853.00 us 2036 FXATTROP 0.08 30944.43 us 70.00 us 327928.00 us 51 RMDIR 0.08 8077.65 us 45.00 us 821788.00 us 200 UNLINK 0.10 13824.17 us 60.00 us 453180.00 us 150 LINK 0.21 30601.44 us 94.00 us 523007.00 us 144 MKNOD 0.23 25950.57 us 97.00 us 753920.00 us 179 MKDIR 0.43 31232.90 us 94.00 us 784122.00 us 281 CREATE 0.64 14631.45 us 31.00 us 1065265.00 us 908 READDIRP 3.98 68407.79 us 34.00 us 1005239.00 us 1201 READ 94.06 24557.29 us 14.00 us 1671808.00 us 78990 LOOKUP Duration: 573 seconds Data Read: 130735659 bytes Data Written: 79499041 bytes
Pranith Kumar Karampuri
2018-Aug-17 07:30 UTC
[Gluster-users] Gluter 3.12.12: performance during heal and in general
There seems to be too many lookup operations compared to any other operations. What is the workload on the volume? On Fri, Aug 17, 2018 at 12:47 PM Hu Bert <revirii at googlemail.com> wrote:> i hope i did get it right. > > gluster volume profile shared start > wait 10 minutes > gluster volume profile shared info > gluster volume profile shared stop > > If that's ok, i've attached the output of the info command. > > > 2018-08-17 8:31 GMT+02:00 Pranith Kumar Karampuri <pkarampu at redhat.com>: > > Please do volume profile also for around 10 minutes when CPU% is high. > > > > On Fri, Aug 17, 2018 at 11:56 AM Pranith Kumar Karampuri > > <pkarampu at redhat.com> wrote: > >> > >> As per the output, all io-threads are using a lot of CPU. It is better > to > >> check what the volume profile is to see what is leading to so much work > for > >> io-threads. Please follow the documentation at > >> > https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/ > >> section: " > >> > >> Running GlusterFS Volume Profile Command" > >> > >> and attach output of "gluster volume profile info", > >> > >> On Fri, Aug 17, 2018 at 11:24 AM Hu Bert <revirii at googlemail.com> > wrote: > >>> > >>> Good morning, > >>> > >>> i ran the command during 100% CPU usage and attached the file. > >>> Hopefully it helps. > >>> > >>> 2018-08-17 7:33 GMT+02:00 Pranith Kumar Karampuri <pkarampu at redhat.com > >: > >>> > Could you do the following on one of the nodes where you are > observing > >>> > high > >>> > CPU usage and attach that file to this thread? We can find what > >>> > threads/processes are leading to high usage. Do this for say 10 > minutes > >>> > when > >>> > you see the ~100% CPU. > >>> > > >>> > top -bHd 5 > /tmp/top.${HOSTNAME}.txt > >>> > > >>> > On Wed, Aug 15, 2018 at 2:37 PM Hu Bert <revirii at googlemail.com> > wrote: > >>> >> > >>> >> Hello again :-) > >>> >> > >>> >> The self heal must have finished as there are no log entries in > >>> >> glustershd.log files anymore. According to munin disk latency > (average > >>> >> io wait) has gone down to 100 ms, and disk utilization has gone down > >>> >> to ~60% - both on all servers and hard disks. > >>> >> > >>> >> But now system load on 2 servers (which were in the good state) > >>> >> fluctuates between 60 and 100; the server with the formerly failed > >>> >> disk has a load of 20-30.I've uploaded some munin graphics of the > cpu > >>> >> usage: > >>> >> > >>> >> https://abload.de/img/gluster11_cpu31d3a.png > >>> >> https://abload.de/img/gluster12_cpu8sem7.png > >>> >> https://abload.de/img/gluster13_cpud7eni.png > >>> >> > >>> >> This can't be normal. 2 of the servers under heavy load and one not > >>> >> that much. Does anyone have an explanation of this strange > behaviour? > >>> >> > >>> >> > >>> >> Thx :-) > >>> >> > >>> >> 2018-08-14 9:37 GMT+02:00 Hu Bert <revirii at googlemail.com>: > >>> >> > Hi there, > >>> >> > > >>> >> > well, it seems the heal has finally finished. Couldn't see/find > any > >>> >> > related log message; is there such a message in a specific log > file? > >>> >> > > >>> >> > But i see the same behaviour when the last heal finished: all CPU > >>> >> > cores are consumed by brick processes; not only by the formerly > >>> >> > failed > >>> >> > bricksdd1, but by all 4 brick processes (and their threads). Load > >>> >> > goes > >>> >> > up to > 100 on the 2 servers with the not-failed brick, and > >>> >> > glustershd.log gets filled with a lot of entries. Load on the > server > >>> >> > with the then failed brick not that high, but still ~60. > >>> >> > > >>> >> > Is this behaviour normal? Is there some post-heal after a heal has > >>> >> > finished? > >>> >> > > >>> >> > thx in advance :-) > >>> > > >>> > > >>> > > >>> > -- > >>> > Pranith > >> > >> > >> > >> -- > >> Pranith > > > > > > > > -- > > Pranith >-- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180817/3c495475/attachment.html>