Hi All, So we've got a cluster currently running on glusterfs 7.3 Heal performance is absolutely terrible (it starts of great, runs for a while then completely trips up). Once the SHD trips up, access speeds to the cluster also becomes poor.? And with poor I mean directory listings that normally takes well under 500ms takes upwards of 10s, but I've seen upwards of 40s (although a follow up ls immediately after is nice and fast again, temporarily).? With all SHDs killed this particular ls is as per normal (around 30-50ms). The SHD logs doesn't provide any usable clues, the only variance in logs is this, just before all daemons across the cluster crawls to a near halt. [2020-05-02 19:55:58.627036] I [MSGID: 108026] [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0: Completed entry selfheal on 5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1 ? [2020-05-02 19:55:58.627058] I [MSGID: 108026] [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0: Completed entry selfheal on c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1 ? ? [2020-05-02 19:55:59.125001] W [MSGID: 114031] [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] 0-vpbx_shared-client-1: remote operation failed [Invalid argument] ? [2020-05-02 19:55:59.339803] I [MSGID: 108026] [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0: Completed entry selfheal on 2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1? [2020-05-02 19:55:59.490387] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-vpbx_shared-replicate-0: performing metadata selfheal on 527a63a3-9acd-4c3f-b95c-5c719325c973 A few more entries will complete overall and then it'll be a file every other second.? Once I saw this I ran a few more tests, and on every test just prior to slowing to a crawl, that entry is there. There is a perceived improvement when lowering the thread count, but not significant enough to confirm a correlation.? Even with**cluster.shd-max-threads=1 this still happens (gets a few tens thousand files completed instead of few hundred, at 32 it gets to maybe 400 to 500 before it stalls, 16 does maybe a thousand).* * There are probably performant bursts hereafter, because from experience we know this process takes about two months.? But this time around it seems to be much, much worse.? Might be because that was healing a single brick to another brick, so less locking going?? At the moment the replicas are basically trying to populate the newly added arbiters.? Also, we haven't messed as greatly with the health check settings (queues and threads and locks nearly as much as since the arbiter addition because performance has simply dropped to non-acceptable levels. Unfortunately granular entry heal isn't on at the moment (nor can I switch it on due to a few million outstanding heals).? This used to be a 2x2 cluster.? We then moved one of the bricks to a newly added server using replace-brick.? What a frightening process this is.? We then proceeded to add the arbiters. Some of the settings below was absolute shots in the dark, based on the little information we've been able to track down so far. Kind Regards, Jaco The current gluster volume info (this used to be 2x2 distributed-replicate, aiming to get it to 3x (2 + 1) ... at current rates we might get there by 2021): Volume Name: glvol Type: Distributed-Replicate Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb Status: Started Snapshot Count: 0 Number of Bricks: 2 x (2 + 1) = 6 Transport-type: tcp Bricks: Brick1: zoidberg:/mnt/gluster/a Brick2: flexo:/mnt/gluster/b Brick3: bender:/mnt/gluster/c (arbiter) Brick4: bender:/mnt/gluster/a Brick5: zoidberg:/mnt/gluster/b Brick6: flexo:/mnt/gluster/c (arbiter) Options Reconfigured: *cluster.heal-timeout: 5 <-- hoping this would at least shorten the blockages.* cluster.locking-scheme: granular cluster.heal-wait-queue-length: 10240 *diagnostics.count-fop-hits: on** **diagnostics.latency-measurement: on <-- was hoping profiling might provide some insight.? Going to switch this off now.* cluster.background-self-heal-count: 128 *cluster.shd-wait-qlength: 131072** **cluster.shd-max-threads: 32? <-- even with this disk IO never goes over 10% utilization.? CPU goes to about 2 cores.? So it's lock contention that's slowing it down we reckon.* *cluster.metadata-self-heal: off** **cluster.entry-self-heal: off** **cluster.data-self-heal: off**<-- these kill performance. *transport.address-family: inet nfs.disable: on performance.client-io-threads: on features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.stat-prefetch: on performance.cache-invalidation: on performance.md-cache-timeout: 600 *cluster.self-heal-daemon: yes <-- disabling/enabling this doesn't help, have to kill the shd and start ... force to get things moving along again.* performance.io-thread-count: 16 performance.high-prio-threads: 16 performance.normal-prio-threads: 16 performance.low-prio-threads: 16 performance.least-prio-threads: 4 cluster.self-heal-window-size: 512 cluster.self-heal-readdir-size: 131072 client.event-threads: 8 server.event-threads: 16 network.ping-timeout: 2 *cluster.data-self-heal-algorithm: full <-- most files are small (<1MB), on dual 1gbit links (so up to 2gbit, but b/w never goes over 5mbit anyway).* performance info: Brick: zoidberg:/mnt/gluster/a ----------------------------------------- Cumulative Stats: ?? Block Size:????????????????? 1b+?????????????????? 8b+????????????????? 16b+ ?No. of Reads:??????????????????? 0???????????????????? 0???????????????????? 0 No. of Writes:????????????????? 117?????????????????? 195?????????????????? 367 ? ?? Block Size:???????????????? 32b+????????????????? 64b+???????????????? 128b+ ?No. of Reads:????????????????? 306???????????????????? 0???????????????????? 6 No. of Writes:???????????????? 1016???????????????????? 9?????????????????? 134 ? ?? Block Size:??????????????? 256b+???????????????? 512b+??????????????? 1024b+ ?No. of Reads:????????????????? 107??????????????????? 58?????????????????? 485 No. of Writes:????????????????? 731??????????????????? 73?????????????????? 168 ? ?? Block Size:?????????????? 2048b+??????????????? 4096b+??????????????? 8192b+ ?No. of Reads:?????????????????? 43??????????????????? 59?????????????????? 383 No. of Writes:????????????????? 295?????????????????? 475???????????????? 10562 ? ?? Block Size:????????????? 16384b+?????????????? 32768b+?????????????? 65536b+ ?No. of Reads:????????????????? 347?????????????????? 380?????????????????? 634 No. of Writes:???????????????? 1487????????????????? 6369????????????????? 3970 ? ?? Block Size:???????????? 131072b+ ?No. of Reads:??????????????? 35704 No. of Writes:??????????????? 12398 ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 957293????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1455091???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 69951? RELEASEDIR ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us??????????? 113???????? IPC ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 us???????????? 30 REMOVEXATTR ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 us???????????? 52?? FTRUNCATE ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us?????????? 1385??????? STAT ????? 0.00???? 364.93 us????? 63.45 us?? 15957.66 us??????????? 810?? FGETXATTR ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us??????????? 137??????? LINK ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 us???????????? 12?????? FSYNC ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us??????????? 698??? TRUNCATE ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us??????????? 188????? RENAME ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us?????????? 1680???? SETATTR ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us??????????? 127?????? MKNOD ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us????????? 11412??? FINODELK ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us??????????? 221??? SETXATTR ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 us???????????? 30???? SYMLINK ????? 0.00????? 56.18 us?????? 7.31 us?? 41484.37 us????????? 22640?????? FLUSH ????? 0.00???? 105.64 us????? 19.34 us? 351055.72 us????????? 21693????? STATFS ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us??????????? 178?????? MKDIR ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us?????????? 9653?????? FSTAT ????? 0.00???? 214.87 us????? 25.71 us 1712020.99 us????????? 39054??????? READ ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us????????? 69951???? OPENDIR ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us?????????? 3180?????? RMDIR ????? 0.01???? 327.08 us????? 55.69 us? 420376.80 us????????? 38366?????? WRITE ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us?????????? 6169????? CREATE ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us??????? 1159936??? READLINK ????? 0.40???? 543.50 us????? 36.45 us 2430653.54 us??????? 1449307??????? OPEN ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us????????? 10787??? FXATTROP ????? 0.53??? 2342.72 us????? 20.91 us 4157542.43 us???????? 447821????? UNLINK ????? 0.53? 558734.79 us????? 24.40 us 1399487.63 us?????????? 1889????????? LK ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us????????? 10687???? READDIR ????? 0.99??? 2426.23 us????? 12.47 us 24176123.13 us???????? 809430??? GETXATTR ????? 1.45???? 676.83 us?????? 9.30 us 193177067.25 us??????? 4260021???? ENTRYLK ????? 1.76?? 38325.38 us????? 21.44 us 46693575.93 us????????? 91439??? READDIRP ????? 4.44??? 1200.31 us?????? 9.10 us 2559377.90 us??????? 7352811???? INODELK ????? 9.30??? 1849.24 us?????? 9.28 us 35299050.00 us??????? 9989221????? LOOKUP ???? 79.24?? 47791.46 us????? 76.77 us 10488824.26 us??????? 3294642???? XATTROP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 3194806????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 5311???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????? 54??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 1907?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 3189457?? CI_FORGET ? ??? Duration: 209524 seconds ?? Data Read: 4775365531 bytes Data Written: 2354108664 bytes ? Interval 0 Stats: ?? Block Size:????????????????? 1b+?????????????????? 8b+????????????????? 16b+ ?No. of Reads:??????????????????? 0???????????????????? 0???????????????????? 0 No. of Writes:????????????????? 117?????????????????? 195?????????????????? 367 ? ?? Block Size:???????????????? 32b+????????????????? 64b+???????????????? 128b+ ?No. of Reads:????????????????? 306???????????????????? 0???????????????????? 6 No. of Writes:???????????????? 1016???????????????????? 9?????????????????? 134 ? ?? Block Size:??????????????? 256b+???????????????? 512b+??????????????? 1024b+ ?No. of Reads:????????????????? 107??????????????????? 58?????????????????? 485 No. of Writes:????????????????? 731??????????????????? 73?????????????????? 168 ? ?? Block Size:?????????????? 2048b+??????????????? 4096b+??????????????? 8192b+ ?No. of Reads:?????????????????? 43??????????????????? 59?????????????????? 383 No. of Writes:????????????????? 295?????????????????? 475???????????????? 10562 ? ?? Block Size:????????????? 16384b+?????????????? 32768b+?????????????? 65536b+ ?No. of Reads:????????????????? 347?????????????????? 380?????????????????? 634 No. of Writes:???????????????? 1487????????????????? 6369????????????????? 3970 ? ?? Block Size:???????????? 131072b+ ?No. of Reads:??????????????? 35704 No. of Writes:??????????????? 12398 ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 957293????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1455091???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 69951? RELEASEDIR ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us??????????? 113???????? IPC ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 us???????????? 30 REMOVEXATTR ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 us???????????? 52?? FTRUNCATE ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us?????????? 1385??????? STAT ????? 0.00???? 363.20 us????? 63.45 us?? 15957.66 us??????????? 810?? FGETXATTR ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us??????????? 137??????? LINK ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 us???????????? 12?????? FSYNC ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us??????????? 698??? TRUNCATE ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us??????????? 188????? RENAME ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us?????????? 1680???? SETATTR ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us??????????? 127?????? MKNOD ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us????????? 11412??? FINODELK ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us??????????? 221??? SETXATTR ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 us???????????? 30???? SYMLINK ????? 0.00????? 56.19 us?????? 7.31 us?? 41484.37 us????????? 22640?????? FLUSH ????? 0.00???? 105.61 us????? 19.34 us? 351055.72 us????????? 21693????? STATFS ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us??????????? 178?????? MKDIR ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us?????????? 9653?????? FSTAT ????? 0.00???? 214.91 us????? 25.71 us 1712020.99 us????????? 39054??????? READ ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us????????? 69951???? OPENDIR ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us?????????? 3180?????? RMDIR ????? 0.01???? 327.13 us????? 55.69 us? 420376.80 us????????? 38366?????? WRITE ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us?????????? 6169????? CREATE ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us??????? 1159936??? READLINK ????? 0.40???? 543.49 us????? 36.45 us 2430653.54 us??????? 1449307??????? OPEN ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us????????? 10787??? FXATTROP ????? 0.53??? 2342.57 us????? 20.91 us 4157542.43 us???????? 447821????? UNLINK ????? 0.53? 558416.10 us????? 24.40 us 1399487.63 us?????????? 1889????????? LK ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us????????? 10687???? READDIR ????? 0.99??? 2426.24 us????? 12.47 us 24176123.13 us???????? 809430??? GETXATTR ????? 1.45???? 676.75 us?????? 9.30 us 193177067.25 us??????? 4260021???? ENTRYLK ????? 1.76?? 38324.59 us????? 21.44 us 46693575.93 us????????? 91439??? READDIRP ????? 4.44??? 1200.37 us?????? 9.10 us 2559377.90 us??????? 7352811???? INODELK ????? 9.29??? 1848.94 us?????? 9.28 us 35299050.00 us??????? 9989221????? LOOKUP ???? 79.24?? 47792.42 us????? 76.77 us 10488824.26 us??????? 3294642???? XATTROP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 3194806????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 5311???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????? 54??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 1907?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 3189457?? CI_FORGET ? ??? Duration: 209524 seconds ?? Data Read: 4775365531 bytes Data Written: 2354108664 bytes ? Brick: zoidberg:/mnt/gluster/b ----------------------------------------- Cumulative Stats: ?? Block Size:????????????????? 1b+?????????????????? 2b+?????????????????? 8b+ ?No. of Reads:??????????????????? 0???????????????????? 0???????????????????? 0 No. of Writes:?????????????????? 53???????????????????? 1??????????????????? 60 ? ?? Block Size:???????????????? 16b+????????????????? 32b+????????????????? 64b+ ?No. of Reads:??????????????????? 0?????????????????? 213???????????????????? 2 No. of Writes:?????????????????? 14????????????????? 1057??????????????????? 15 ? ?? Block Size:??????????????? 128b+???????????????? 256b+???????????????? 512b+ ?No. of Reads:??????????????????? 0??????????????????? 41?????????????????? 146 No. of Writes:?????????????????? 59?????????????????? 715??????????????????? 80 ? ?? Block Size:?????????????? 1024b+??????????????? 2048b+??????????????? 4096b+ ?No. of Reads:????????????????? 224???????????????????? 9??????????????????? 15 No. of Writes:????????????????? 163?????????????????? 282?????????????????? 404 ? ?? Block Size:?????????????? 8192b+?????????????? 16384b+?????????????? 32768b+ ?No. of Reads:?????????????????? 52??????????????????? 74?????????????????? 210 No. of Writes:???????????????? 7415????????????????? 1430????????????????? 6860 ? ?? Block Size:????????????? 65536b+????????????? 131072b+????????????? 262144b+ ?No. of Reads:????????????????? 299???????????????? 15313???????????????????? 1 No. of Writes:???????????????? 2458???????????????? 12378???????????????????? 1 ? ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 686733????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 837434???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 87338? RELEASEDIR ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us??????????? 116???????? IPC ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 us???????????? 57?? FTRUNCATE ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 us???????????? 65?? FGETXATTR ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 us???????????? 23 REMOVEXATTR ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us??????????? 121??????? LINK ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us??????????? 490????????? LK ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us??????????? 540??????? STAT ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us??????????? 200??? TRUNCATE ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us?????????? 1482???? SETATTR ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 us???????????? 12?????? FSYNC ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us????????? 10047?????? FLUSH ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us?????????? 9977??? FINODELK ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us??????????? 188????? RENAME ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 us???????????? 25???? SYMLINK ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us??????????? 118?????? MKNOD ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us????????? 12587?????? FSTAT ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us????????? 21695????? STATFS ????? 0.00???? 307.86 us????? 28.00 us? 292666.83 us????????? 16841??????? READ ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us??????????? 178?????? MKDIR ????? 0.00???? 319.24 us????? 51.19 us? 136353.39 us????????? 33445?????? WRITE ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us????????? 87338???? OPENDIR ????? 0.01???? 249.96 us????? 58.42 us? 594774.86 us????????? 85089??? SETXATTR ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us?????????? 3180?????? RMDIR ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us?????????? 6067????? CREATE ????? 0.05????? 89.45 us????? 30.14 us? 687961.60 us??????? 1219542??? READLINK ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us???????? 831371??????? OPEN ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us?????????? 9493??? FXATTROP ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us??????? 3764714???? ENTRYLK ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us????????? 22253???? READDIR ????? 1.06??? 6913.71 us????? 11.58 us 36968532.72 us???????? 364963??? GETXATTR ????? 1.29?? 62130.12 us????? 36.89 us 8649871.11 us????????? 49472??? READDIRP ????? 2.89??? 1477.53 us????? 11.10 us 2807637.64 us??????? 4650500???? INODELK ????? 4.12?? 23681.31 us????? 20.98 us 23291414.50 us???????? 414352????? UNLINK ???? 16.09??? 5000.31 us?????? 9.04 us 23866016.88 us??????? 7655621????? LOOKUP ???? 72.04?? 68953.31 us????? 68.40 us 27542499.65 us??????? 2486126???? XATTROP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 2433405????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 2453???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????? 69??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 1544?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 2430885?? CI_FORGET ? ??? Duration: 209524 seconds ?? Data Read: 2051388524 bytes Data Written: 2208261499 bytes ? Interval 0 Stats: ?? Block Size:????????????????? 1b+?????????????????? 2b+?????????????????? 8b+ ?No. of Reads:??????????????????? 0???????????????????? 0???????????????????? 0 No. of Writes:?????????????????? 53???????????????????? 1??????????????????? 60 ? ?? Block Size:???????????????? 16b+????????????????? 32b+????????????????? 64b+ ?No. of Reads:??????????????????? 0?????????????????? 213???????????????????? 2 No. of Writes:?????????????????? 14????????????????? 1057??????????????????? 15 ? ?? Block Size:??????????????? 128b+???????????????? 256b+???????????????? 512b+ ?No. of Reads:??????????????????? 0??????????????????? 41?????????????????? 146 No. of Writes:?????????????????? 59?????????????????? 715??????????????????? 80 ? ?? Block Size:?????????????? 1024b+??????????????? 2048b+??????????????? 4096b+ ?No. of Reads:????????????????? 224???????????????????? 9??????????????????? 15 No. of Writes:????????????????? 163?????????????????? 282?????????????????? 404 ? ?? Block Size:?????????????? 8192b+?????????????? 16384b+?????????????? 32768b+ ?No. of Reads:?????????????????? 52??????????????????? 74?????????????????? 210 No. of Writes:???????????????? 7415????????????????? 1430????????????????? 6860 ? ?? Block Size:????????????? 65536b+????????????? 131072b+????????????? 262144b+ ?No. of Reads:????????????????? 299???????????????? 15313???????????????????? 1 No. of Writes:???????????????? 2458???????????????? 12378???????????????????? 1 ? ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 686733????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 837434???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 87338? RELEASEDIR ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us??????????? 116???????? IPC ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 us???????????? 57?? FTRUNCATE ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 us???????????? 65?? FGETXATTR ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 us???????????? 23 REMOVEXATTR ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us??????????? 121??????? LINK ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us??????????? 490????????? LK ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us??????????? 540??????? STAT ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us??????????? 200??? TRUNCATE ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us?????????? 1482???? SETATTR ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 us???????????? 12?????? FSYNC ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us????????? 10047?????? FLUSH ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us?????????? 9977??? FINODELK ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us??????????? 188????? RENAME ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 us???????????? 25???? SYMLINK ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us??????????? 118?????? MKNOD ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us????????? 12587?????? FSTAT ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us????????? 21695????? STATFS ????? 0.00???? 307.91 us????? 28.00 us? 292666.83 us????????? 16841??????? READ ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us??????????? 178?????? MKDIR ????? 0.00???? 319.22 us????? 51.19 us? 136353.39 us????????? 33445?????? WRITE ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us????????? 87338???? OPENDIR ????? 0.01???? 249.95 us????? 58.42 us? 594774.86 us????????? 85089??? SETXATTR ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us?????????? 3180?????? RMDIR ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us?????????? 6067????? CREATE ????? 0.05????? 89.46 us????? 30.14 us? 687961.60 us??????? 1219542??? READLINK ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us???????? 831371??????? OPEN ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us?????????? 9493??? FXATTROP ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us??????? 3764714???? ENTRYLK ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us????????? 22253???? READDIR ????? 1.06??? 6913.26 us????? 11.58 us 36968532.72 us???????? 364963??? GETXATTR ????? 1.29?? 62121.95 us????? 36.89 us 8649871.11 us????????? 49472??? READDIRP ????? 2.89??? 1477.58 us????? 11.10 us 2807637.64 us??????? 4650500???? INODELK ????? 4.12?? 23681.29 us????? 20.98 us 23291414.50 us???????? 414352????? UNLINK ???? 16.09??? 5000.02 us?????? 9.04 us 23866016.88 us??????? 7655621????? LOOKUP ???? 72.04?? 68956.14 us????? 68.40 us 27542499.65 us??????? 2486126???? XATTROP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 2433405????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 2453???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????? 69??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 1544?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 2430885?? CI_FORGET ? ??? Duration: 209524 seconds ?? Data Read: 2051388524 bytes Data Written: 2208261499 bytes ? Brick: flexo:/mnt/gluster/b -------------------------------------- Cumulative Stats: ?? Block Size:????????????????? 1b+?????????????????? 8b+????????????????? 16b+ ?No. of Reads:??????????????????? 0???????????????????? 0???????????????????? 0 No. of Writes:????????????????? 117?????????????????? 195?????????????????? 367 ? ?? Block Size:???????????????? 32b+????????????????? 64b+???????????????? 128b+ ?No. of Reads:????????????????? 130???????????????????? 0???????????????????? 3 No. of Writes:???????????????? 1012???????????????????? 9?????????????????? 134 ? ?? Block Size:??????????????? 256b+???????????????? 512b+??????????????? 1024b+ ?No. of Reads:?????????????????? 47???????????????????? 8?????????????????? 257 No. of Writes:????????????????? 729??????????????????? 73?????????????????? 168 ? ?? Block Size:?????????????? 2048b+??????????????? 4096b+??????????????? 8192b+ ?No. of Reads:?????????????????? 10??????????????????? 27?????????????????? 240 No. of Writes:????????????????? 294?????????????????? 466???????????????? 10472 ? ?? Block Size:????????????? 16384b+?????????????? 32768b+?????????????? 65536b+ ?No. of Reads:????????????????? 139?????????????????? 119?????????????????? 135 No. of Writes:???????????????? 1480????????????????? 6273????????????????? 3935 ? ?? Block Size:???????????? 131072b+ ?No. of Reads:???????????????? 7892 No. of Writes:??????????????? 12327 ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 955058????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1450749???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 81377? RELEASEDIR ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us??????????? 108???????? IPC ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 us???????????? 44?? FTRUNCATE ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 us???????????? 30 REMOVEXATTR ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 us????????????? 4?????? FSYNC ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 us???????????? 30???? SYMLINK ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us??????????? 137??????? LINK ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us??????????? 119?????? MKNOD ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us??????????? 221??? SETXATTR ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us??????????? 354??????? STAT ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us??????????? 777????????? LK ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us??????????? 188????? RENAME ????? 0.00???? 150.40 us????? 75.59 us???? 749.54 us??????????? 810?? FGETXATTR ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us??????????? 698??? TRUNCATE ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us??????????? 178?????? MKDIR ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us?????????? 1665???? SETATTR ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us????????? 11372??? FINODELK ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us?????????? 7398?????? FSTAT ????? 0.00???? 120.71 us????? 31.43 us???? 847.97 us?????????? 9094??????? READ ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us????????? 22614?????? FLUSH ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us????????? 21656????? STATFS ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us?????????? 6145????? CREATE ????? 0.00???? 217.55 us????? 50.60 us?? 30734.99 us????????? 38051?????? WRITE ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us????????? 81377???? OPENDIR ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us?????????? 3177?????? RMDIR ????? 0.03???? 100.05 us????? 30.93 us?? 49756.93 us???????? 432439??? READLINK ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us???????? 448381????? UNLINK ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us????????? 52185??? READDIRP ????? 0.35???? 414.05 us????? 36.58 us 1038229.93 us??????? 1444606??????? OPEN ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us????????? 10745??? FXATTROP ????? 0.72???? 776.21 us????? 11.24 us 4271792.57 us??????? 1605822??? GETXATTR ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us????????? 13532???? READDIR ????? 0.79???? 320.44 us?????? 7.71 us 2369511.06 us??????? 4257297???? ENTRYLK ????? 1.97???? 464.02 us?????? 9.38 us 1038460.80 us??????? 7333722???? INODELK ???? 13.73??? 2372.50 us?????? 8.33 us 6258159.87 us??????? 9979799????? LOOKUP ???? 80.79?? 42372.04 us????? 61.49 us 2238308.13 us??????? 3289103???? XATTROP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 2250268????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 3756???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? 167??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 1908?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 2246352?? CI_FORGET ? ??? Duration: 209334 seconds ?? Data Read: 1059777395 bytes Data Written: 2336921014 bytes ? Interval 0 Stats: ?? Block Size:????????????????? 1b+?????????????????? 8b+????????????????? 16b+ ?No. of Reads:??????????????????? 0???????????????????? 0???????????????????? 0 No. of Writes:????????????????? 117?????????????????? 195?????????????????? 367 ? ?? Block Size:???????????????? 32b+????????????????? 64b+???????????????? 128b+ ?No. of Reads:????????????????? 130???????????????????? 0???????????????????? 3 No. of Writes:???????????????? 1012???????????????????? 9?????????????????? 134 ? ?? Block Size:??????????????? 256b+???????????????? 512b+??????????????? 1024b+ ?No. of Reads:?????????????????? 47???????????????????? 8?????????????????? 257 No. of Writes:????????????????? 729??????????????????? 73?????????????????? 168 ? ?? Block Size:?????????????? 2048b+??????????????? 4096b+??????????????? 8192b+ ?No. of Reads:?????????????????? 10??????????????????? 27?????????????????? 240 No. of Writes:????????????????? 294?????????????????? 466???????????????? 10472 ? ?? Block Size:????????????? 16384b+?????????????? 32768b+?????????????? 65536b+ ?No. of Reads:????????????????? 139?????????????????? 119?????????????????? 135 No. of Writes:???????????????? 1480????????????????? 6273????????????????? 3935 ? ?? Block Size:???????????? 131072b+ ?No. of Reads:???????????????? 7892 No. of Writes:??????????????? 12327 ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 955058????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1450749???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 81377? RELEASEDIR ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us??????????? 108???????? IPC ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 us???????????? 44?? FTRUNCATE ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 us???????????? 30 REMOVEXATTR ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 us????????????? 4?????? FSYNC ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 us???????????? 30???? SYMLINK ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us??????????? 137??????? LINK ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us??????????? 119?????? MKNOD ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us??????????? 221??? SETXATTR ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us??????????? 354??????? STAT ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us??????????? 777????????? LK ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us??????????? 188????? RENAME ????? 0.00???? 150.43 us????? 75.59 us???? 749.54 us??????????? 810?? FGETXATTR ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us??????????? 698??? TRUNCATE ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us??????????? 178?????? MKDIR ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us?????????? 1665???? SETATTR ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us????????? 11372??? FINODELK ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us?????????? 7398?????? FSTAT ????? 0.00???? 120.74 us????? 31.43 us???? 847.97 us?????????? 9094??????? READ ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us????????? 22614?????? FLUSH ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us????????? 21656????? STATFS ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us?????????? 6145????? CREATE ????? 0.00???? 217.50 us????? 50.60 us?? 30734.99 us????????? 38051?????? WRITE ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us????????? 81377???? OPENDIR ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us?????????? 3177?????? RMDIR ????? 0.03???? 100.06 us????? 30.93 us?? 49756.93 us???????? 432439??? READLINK ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us???????? 448381????? UNLINK ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us????????? 52185??? READDIRP ????? 0.35???? 414.01 us????? 36.58 us 1038229.93 us??????? 1444606??????? OPEN ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us????????? 10745??? FXATTROP ????? 0.72???? 776.24 us????? 11.24 us 4271792.57 us??????? 1605822??? GETXATTR ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us????????? 13532???? READDIR ????? 0.79???? 320.45 us?????? 7.71 us 2369511.06 us??????? 4257297???? ENTRYLK ????? 1.97???? 464.07 us?????? 9.38 us 1038460.80 us??????? 7333722???? INODELK ???? 13.72??? 2372.42 us?????? 8.33 us 6258159.87 us??????? 9979799????? LOOKUP ???? 80.79?? 42373.50 us????? 61.49 us 2238308.13 us??????? 3289103???? XATTROP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 2250268????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 3756???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? 167??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 1908?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 2246352?? CI_FORGET ? ??? Duration: 209334 seconds ?? Data Read: 1059777395 bytes Data Written: 2336921014 bytes ? Brick: flexo:/mnt/gluster/c -------------------------------------- Cumulative Stats: ?? Block Size:????????????????? 1b+ ?No. of Reads:??????????????????? 0 No. of Writes:??????????????? 32843 ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 686562????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 834596???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 93757? RELEASEDIR ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us??????????? 107???????? IPC ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 us???????????? 36??????? LINK ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 us???????????? 65?? FGETXATTR ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 us???????????? 88????? RENAME ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us??????????? 153??? TRUNCATE ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us??????????? 491????????? LK ????? 0.01????? 51.73 us?????? 7.70 us??? 1174.64 us?????????? 9339?????? FLUSH ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us?????????? 9011??? FINODELK ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us????????? 10824?????? FSTAT ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us?????????? 2594?????? MKDIR ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us?????????? 9009??? FXATTROP ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us?????????? 5754????? CREATE ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us????????? 32843?????? WRITE ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us?????????? 5798???? SYMLINK ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us????????? 34904???? READDIR ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us?????????? 2732?????? RMDIR ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us????????? 93757???? OPENDIR ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us???????? 170292????? UNLINK ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us????????? 13840??? READDIRP ????? 1.94???? 200.88 us????? 25.49 us? 461654.32 us???????? 468842?? FTRUNCATE ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us???????? 486882 REMOVEXATTR ????? 2.27???? 223.01 us????? 52.03 us? 178897.40 us???????? 493240??? SETXATTR ????? 3.53???? 155.31 us?????? 7.96 us?? 67873.00 us??????? 1101338??? GETXATTR ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us???????? 829002??????? OPEN ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us??????? 3660425???? ENTRYLK ????? 6.23???? 304.93 us????? 42.51 us? 661394.43 us???????? 990644???? SETATTR ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us???????? 990766???? XATTROP ???? 12.39???? 549.11 us???? 135.57 us 1659697.83 us??????? 1093387?????? MKNOD ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us??????? 4325281???? INODELK ???? 39.53???? 206.94 us?????? 8.56 us? 964823.33 us??????? 9259429????? LOOKUP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1856377????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 933300???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? 251??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? 293?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 923077?? CI_FORGET ? ??? Duration: 209334 seconds ?? Data Read: 0 bytes Data Written: 32843 bytes ? Interval 0 Stats: ?? Block Size:????????????????? 1b+ ?No. of Reads:??????????????????? 0 No. of Writes:??????????????? 32843 ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 686562????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 834596???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 93757? RELEASEDIR ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us??????????? 107???????? IPC ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 us???????????? 36??????? LINK ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 us???????????? 65?? FGETXATTR ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 us???????????? 88????? RENAME ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us??????????? 153??? TRUNCATE ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us??????????? 491????????? LK ????? 0.01????? 51.71 us?????? 7.70 us??? 1174.64 us?????????? 9339?????? FLUSH ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us?????????? 9011??? FINODELK ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us????????? 10824?????? FSTAT ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us?????????? 2594?????? MKDIR ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us?????????? 9009??? FXATTROP ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us?????????? 5754????? CREATE ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us????????? 32843?????? WRITE ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us?????????? 5798???? SYMLINK ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us????????? 34904???? READDIR ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us?????????? 2732?????? RMDIR ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us????????? 93757???? OPENDIR ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us???????? 170292????? UNLINK ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us????????? 13840??? READDIRP ????? 1.94???? 200.87 us????? 25.49 us? 461654.32 us???????? 468842?? FTRUNCATE ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us???????? 486882 REMOVEXATTR ????? 2.27???? 223.00 us????? 52.03 us? 178897.40 us???????? 493240??? SETXATTR ????? 3.53???? 155.32 us?????? 7.96 us?? 67873.00 us??????? 1101338??? GETXATTR ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us???????? 829002??????? OPEN ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us??????? 3660425???? ENTRYLK ????? 6.23???? 304.88 us????? 42.51 us? 661394.43 us???????? 990644???? SETATTR ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us???????? 990766???? XATTROP ???? 12.39???? 549.25 us???? 135.57 us 1659697.83 us??????? 1093387?????? MKNOD ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us??????? 4325281???? INODELK ???? 39.53???? 206.95 us?????? 8.56 us? 964823.33 us??????? 9259429????? LOOKUP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1856377????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 933300???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? 251??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? 293?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 923077?? CI_FORGET ? ??? Duration: 209334 seconds ?? Data Read: 0 bytes Data Written: 32843 bytes ? Brick: bender:/mnt/gluster/c --------------------------------------- Cumulative Stats: ?? Block Size:????????????????? 1b+ ?No. of Reads:??????????????????? 0 No. of Writes:??????????????? 37050 ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 928119????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1401524???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 70092? RELEASEDIR ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us??????????? 100???????? IPC ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 us???????????? 38??????? LINK ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 us???????????? 30???? SYMLINK ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 us????????????? 4?????? FSYNC ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us??????????? 122????? RENAME ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us??????????? 777????????? LK ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us??????????? 810?? FGETXATTR ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us??????????? 603??? TRUNCATE ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us??????????? 302?????? MKDIR ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us????????? 10060??? FINODELK ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us?????????? 6409?????? FSTAT ????? 0.01????? 49.38 us?????? 8.08 us??? 1264.41 us????????? 21719?????? FLUSH ????? 0.03????? 74.76 us????? 20.86 us??? 2008.41 us????????? 37050?????? WRITE ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us?????????? 5901????? CREATE ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us????????? 10057??? FXATTROP ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us????????? 70092???? OPENDIR ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us????????? 10779???? READDIR ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us?????????? 3008?????? RMDIR ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us???????? 109478????? UNLINK ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us???????? 722669 REMOVEXATTR ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us???????? 789294??? GETXATTR ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us???????? 718835?? FTRUNCATE ????? 1.89???? 219.47 us????? 49.92 us? 506767.78 us???????? 722856??? SETXATTR ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us??????? 1395957??????? OPEN ????? 5.56???? 322.08 us????? 43.37 us? 634069.61 us??????? 1450160???? SETATTR ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us??????? 4233520???? ENTRYLK ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us??????? 1449601???? XATTROP ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us??????? 7097226???? INODELK ???? 19.37??? 1104.98 us???? 101.28 us 1612346.75 us??????? 1472813?????? MKNOD ???? 41.85???? 295.61 us?????? 8.98 us? 996805.54 us?????? 11896091????? LOOKUP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 3531139????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1862470???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? 244??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? 632?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1668669?? CI_FORGET ? ??? Duration: 208643 seconds ?? Data Read: 0 bytes Data Written: 37050 bytes ? Interval 0 Stats: ?? Block Size:????????????????? 1b+ ?No. of Reads:??????????????????? 0 No. of Writes:??????????????? 37050 ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 928119????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1401524???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 70092? RELEASEDIR ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us??????????? 100???????? IPC ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 us???????????? 38??????? LINK ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 us???????????? 30???? SYMLINK ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 us????????????? 4?????? FSYNC ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us??????????? 122????? RENAME ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us??????????? 777????????? LK ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us??????????? 810?? FGETXATTR ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us??????????? 603??? TRUNCATE ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us??????????? 302?????? MKDIR ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us????????? 10060??? FINODELK ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us?????????? 6409?????? FSTAT ????? 0.01????? 49.39 us?????? 8.08 us??? 1264.41 us????????? 21719?????? FLUSH ????? 0.03????? 74.77 us????? 20.86 us??? 2008.41 us????????? 37050?????? WRITE ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us?????????? 5901????? CREATE ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us????????? 10057??? FXATTROP ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us????????? 70092???? OPENDIR ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us????????? 10779???? READDIR ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us?????????? 3008?????? RMDIR ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us???????? 109478????? UNLINK ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us???????? 722669 REMOVEXATTR ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us???????? 789294??? GETXATTR ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us???????? 718835?? FTRUNCATE ????? 1.89???? 219.48 us????? 49.92 us? 506767.78 us???????? 722856??? SETXATTR ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us??????? 1395957??????? OPEN ????? 5.56???? 322.09 us????? 43.37 us? 634069.61 us??????? 1450160???? SETATTR ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us??????? 4233520???? ENTRYLK ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us??????? 1449601???? XATTROP ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us??????? 7097226???? INODELK ???? 19.37??? 1105.06 us???? 101.28 us 1612346.75 us??????? 1472813?????? MKNOD ???? 41.84???? 295.54 us?????? 8.98 us? 996805.54 us?????? 11896091????? LOOKUP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 3531139????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1862470???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? 244??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? 632?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1668669?? CI_FORGET ? ??? Duration: 208643 seconds ?? Data Read: 0 bytes Data Written: 37050 bytes ? Brick: bender:/mnt/gluster/a --------------------------------------- Cumulative Stats: ?? Block Size:????????????????? 1b+?????????????????? 2b+?????????????????? 8b+ ?No. of Reads:??????????????????? 0???????????????????? 0???????????????????? 0 No. of Writes:?????????????????? 53???????????????????? 1??????????????????? 60 ? ?? Block Size:???????????????? 16b+????????????????? 32b+????????????????? 64b+ ?No. of Reads:??????????????????? 0?????????????????? 235???????????????????? 4 No. of Writes:?????????????????? 14????????????????? 1033??????????????????? 15 ? ?? Block Size:??????????????? 128b+???????????????? 256b+???????????????? 512b+ ?No. of Reads:??????????????????? 4?????????????????? 115??????????????????? 81 No. of Writes:?????????????????? 58?????????????????? 703??????????????????? 80 ? ?? Block Size:?????????????? 1024b+??????????????? 2048b+??????????????? 4096b+ ?No. of Reads:????????????????? 181??????????????????? 26??????????????????? 74 No. of Writes:????????????????? 163?????????????????? 278?????????????????? 397 ? ?? Block Size:?????????????? 8192b+?????????????? 16384b+?????????????? 32768b+ ?No. of Reads:????????????????? 131?????????????????? 161?????????????????? 267 No. of Writes:???????????????? 7264????????????????? 1398????????????????? 6742 ? ?? Block Size:????????????? 65536b+????????????? 131072b+????????????? 262144b+ ?No. of Reads:????????????????? 443???????????????? 25528???????????????????? 0 No. of Writes:???????????????? 2426???????????????? 12240???????????????????? 1 ? ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 684942????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 837092???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 87297? RELEASEDIR ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 us????????????? 8 REMOVEXATTR ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 us???????????? 97???????? IPC ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 us???????????? 46?? FTRUNCATE ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 us???????????? 65?? FGETXATTR ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us??????????? 121??????? LINK ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 us???????????? 25???? SYMLINK ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us??????????? 107?????? MKNOD ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 us????????????? 1?????? FSYNC ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us??????????? 188????? RENAME ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us??????????? 745??????? STAT ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us??????????? 200??? TRUNCATE ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us?????????? 1444???? SETATTR ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us??????????? 177?????? MKDIR ????? 0.00????? 52.68 us?????? 7.28 us???? 826.01 us?????????? 9890?????? FLUSH ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us?????????? 9792??? FINODELK ????? 0.00????? 74.26 us????? 20.00 us??? 1522.39 us????????? 21521????? STATFS ????? 0.00???? 129.21 us????? 34.15 us??? 9068.67 us????????? 13094?????? FSTAT ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us?????????? 5967????? CREATE ????? 0.00???? 112.06 us????? 28.14 us??? 9545.53 us????????? 27474??????? READ ????? 0.01???? 313.76 us????? 52.98 us?? 71635.74 us????????? 32926?????? WRITE ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us????????? 87297???? OPENDIR ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us?????????? 3177?????? RMDIR ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us????????? 85073??? SETXATTR ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us????????? 15090??? READDIRP ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us??????????? 564????????? LK ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us???????? 850938??? READLINK ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us???????? 831212??????? OPEN ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us????????? 22482???? READDIR ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us???????? 415226????? UNLINK ????? 0.54??? 1272.83 us?????? 9.66 us 1604368.93 us???????? 686945??? GETXATTR ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us??????? 3739718???? ENTRYLK ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us?????????? 9311??? FXATTROP ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us??????? 4649291???? INODELK ????? 5.82??? 1240.20 us?????? 8.78 us 1526868.50 us??????? 7625383????? LOOKUP ???? 88.60?? 58005.98 us????? 67.31 us 1448810.06 us??????? 2481914???? XATTROP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1396875????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 1854???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????? 28??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 1051?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1394996?? CI_FORGET ? ??? Duration: 208643 seconds ?? Data Read: 3409342848 bytes Data Written: 2180686507 bytes ? Interval 0 Stats: ?? Block Size:????????????????? 1b+?????????????????? 2b+?????????????????? 8b+ ?No. of Reads:??????????????????? 0???????????????????? 0???????????????????? 0 No. of Writes:?????????????????? 53???????????????????? 1??????????????????? 60 ? ?? Block Size:???????????????? 16b+????????????????? 32b+????????????????? 64b+ ?No. of Reads:??????????????????? 0?????????????????? 235???????????????????? 4 No. of Writes:?????????????????? 14????????????????? 1033??????????????????? 15 ? ?? Block Size:??????????????? 128b+???????????????? 256b+???????????????? 512b+ ?No. of Reads:??????????????????? 4?????????????????? 115??????????????????? 81 No. of Writes:?????????????????? 58?????????????????? 703??????????????????? 80 ? ?? Block Size:?????????????? 1024b+??????????????? 2048b+??????????????? 4096b+ ?No. of Reads:????????????????? 181??????????????????? 26??????????????????? 74 No. of Writes:????????????????? 163?????????????????? 278?????????????????? 397 ? ?? Block Size:?????????????? 8192b+?????????????? 16384b+?????????????? 32768b+ ?No. of Reads:????????????????? 131?????????????????? 161?????????????????? 267 No. of Writes:???????????????? 7264????????????????? 1398????????????????? 6742 ? ?? Block Size:????????????? 65536b+????????????? 131072b+????????????? 262144b+ ?No. of Reads:????????????????? 443???????????????? 25528???????????????????? 0 No. of Writes:???????????????? 2426???????????????? 12240???????????????????? 1 ? ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of calls???????? Fop ?---------?? -----------?? -----------?? -----------?? ------------??????? ---- ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 684942????? FORGET ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? 837092???? RELEASE ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? 87297? RELEASEDIR ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 us????????????? 8 REMOVEXATTR ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 us???????????? 97???????? IPC ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 us???????????? 46?? FTRUNCATE ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 us???????????? 65?? FGETXATTR ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us??????????? 121??????? LINK ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 us???????????? 25???? SYMLINK ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us??????????? 107?????? MKNOD ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 us????????????? 1?????? FSYNC ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us??????????? 188????? RENAME ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us??????????? 745??????? STAT ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us??????????? 200??? TRUNCATE ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us?????????? 1444???? SETATTR ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us??????????? 177?????? MKDIR ????? 0.00????? 52.69 us?????? 7.28 us???? 826.01 us?????????? 9890?????? FLUSH ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us?????????? 9792??? FINODELK ????? 0.00????? 74.25 us????? 20.00 us??? 1522.39 us????????? 21521????? STATFS ????? 0.00???? 129.23 us????? 34.15 us??? 9068.67 us????????? 13094?????? FSTAT ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us?????????? 5967????? CREATE ????? 0.00???? 112.45 us????? 28.14 us??? 9545.53 us????????? 27474??????? READ ????? 0.01???? 313.77 us????? 52.98 us?? 71635.74 us????????? 32926?????? WRITE ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us????????? 87297???? OPENDIR ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us?????????? 3177?????? RMDIR ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us????????? 85073??? SETXATTR ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us????????? 15090??? READDIRP ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us??????????? 564????????? LK ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us???????? 850938??? READLINK ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us???????? 831212??????? OPEN ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us????????? 22482???? READDIR ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us???????? 415226????? UNLINK ????? 0.54??? 1272.79 us?????? 9.66 us 1604368.93 us???????? 686945??? GETXATTR ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us??????? 3739718???? ENTRYLK ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us?????????? 9311??? FXATTROP ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us??????? 4649291???? INODELK ????? 5.82??? 1240.10 us?????? 8.78 us 1526868.50 us??????? 7625383????? LOOKUP ???? 88.60?? 58006.02 us????? 67.31 us 1448810.06 us??????? 2481914???? XATTROP ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1396875????? UPCALL ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 1854???? CI_IATT ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????? 28??? CI_XATTR ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? 1051?? CI_UNLINK ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? 1394996?? CI_FORGET ? ??? Duration: 208643 seconds ?? Data Read: 3409342848 bytes Data Written: 2180686507 bytes ? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200502/5962bb36/attachment.html>
I reported something similar recently to this list, and I highly advise you to update to 7.5 (in my case it was 5.11 -> 5.13), then see if it keeps happening. The self heal issues should be fixed there properly at least. Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | @ArtemR <http://twitter.com/ArtemR> On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za> wrote:> Hi All, > > So we've got a cluster currently running on glusterfs 7.3 > > Heal performance is absolutely terrible (it starts of great, runs for a > while then completely trips up). > > Once the SHD trips up, access speeds to the cluster also becomes poor. > And with poor I mean directory listings that normally takes well under > 500ms takes upwards of 10s, but I've seen upwards of 40s (although a follow > up ls immediately after is nice and fast again, temporarily). With all > SHDs killed this particular ls is as per normal (around 30-50ms). > > The SHD logs doesn't provide any usable clues, the only variance in logs > is this, just before all daemons across the cluster crawls to a near halt. > [2020-05-02 19:55:58.627036] I [MSGID: 108026] > [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0: > Completed entry selfheal on 5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources> sinks=0 1 > [2020-05-02 19:55:58.627058] I [MSGID: 108026] > [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0: > Completed entry selfheal on c736c9b6-f116-468b-adbc-f102f5b57d01. sources> sinks=0 1 > > [2020-05-02 19:55:59.125001] W [MSGID: 114031] > [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] 0-vpbx_shared-client-1: > remote operation failed [Invalid argument] > > [2020-05-02 19:55:59.339803] I [MSGID: 108026] > [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0: > Completed entry selfheal on 2e128eeb-e991-4544-8316-4ddfcb4942da. sources> sinks=0 1 > [2020-05-02 19:55:59.490387] I [MSGID: 108026] > [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] > 0-vpbx_shared-replicate-0: performing metadata selfheal on > 527a63a3-9acd-4c3f-b95c-5c719325c973 > > A few more entries will complete overall and then it'll be a file every > other second. Once I saw this I ran a few more tests, and on every test > just prior to slowing to a crawl, that entry is there. > > There is a perceived improvement when lowering the thread count, but not > significant enough to confirm a correlation. Even with cluster.shd-max-threads=1 > this still happens (gets a few tens thousand files completed instead of few > hundred, at 32 it gets to maybe 400 to 500 before it stalls, 16 does maybe > a thousand). > > There are probably performant bursts hereafter, because from experience we > know this process takes about two months. But this time around it seems to > be much, much worse. Might be because that was healing a single brick to > another brick, so less locking going? At the moment the replicas are > basically trying to populate the newly added arbiters. Also, we haven't > messed as greatly with the health check settings (queues and threads and > locks nearly as much as since the arbiter addition because performance has > simply dropped to non-acceptable levels. > > Unfortunately granular entry heal isn't on at the moment (nor can I switch > it on due to a few million outstanding heals). This used to be a 2x2 > cluster. We then moved one of the bricks to a newly added server using > replace-brick. What a frightening process this is. We then proceeded to > add the arbiters. > > Some of the settings below was absolute shots in the dark, based on the > little information we've been able to track down so far. > > Kind Regards, > Jaco > > The current gluster volume info (this used to be 2x2 > distributed-replicate, aiming to get it to 3x (2 + 1) ... at current rates > we might get there by 2021): > > Volume Name: glvol > Type: Distributed-Replicate > Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb > Status: Started > Snapshot Count: 0 > Number of Bricks: 2 x (2 + 1) = 6 > Transport-type: tcp > Bricks: > Brick1: zoidberg:/mnt/gluster/a > Brick2: flexo:/mnt/gluster/b > Brick3: bender:/mnt/gluster/c (arbiter) > Brick4: bender:/mnt/gluster/a > Brick5: zoidberg:/mnt/gluster/b > Brick6: flexo:/mnt/gluster/c (arbiter) > Options Reconfigured: > *cluster.heal-timeout: 5 <-- hoping this would at least shorten the > blockages.* > cluster.locking-scheme: granular > cluster.heal-wait-queue-length: 10240 > *diagnostics.count-fop-hits: on* > *diagnostics.latency-measurement: on <-- was hoping profiling might > provide some insight. Going to switch this off now.* > cluster.background-self-heal-count: 128 > *cluster.shd-wait-qlength: 131072* > *cluster.shd-max-threads: 32 <-- even with this disk IO never goes over > 10% utilization. CPU goes to about 2 cores. So it's lock contention > that's slowing it down we reckon.* > *cluster.metadata-self-heal: off* > *cluster.entry-self-heal: off* > *cluster.data-self-heal: off* > * <-- these kill performance. *transport.address-family: inet > nfs.disable: on > performance.client-io-threads: on > features.cache-invalidation: on > features.cache-invalidation-timeout: 600 > performance.stat-prefetch: on > performance.cache-invalidation: on > performance.md-cache-timeout: 600 > *cluster.self-heal-daemon: yes <-- disabling/enabling this doesn't help, > have to kill the shd and start ... force to get things moving along again.* > performance.io-thread-count: 16 > performance.high-prio-threads: 16 > performance.normal-prio-threads: 16 > performance.low-prio-threads: 16 > performance.least-prio-threads: 4 > cluster.self-heal-window-size: 512 > cluster.self-heal-readdir-size: 131072 > client.event-threads: 8 > server.event-threads: 16 > network.ping-timeout: 2 > *cluster.data-self-heal-algorithm: full <-- most files are small (<1MB), > on dual 1gbit links (so up to 2gbit, but b/w never goes over 5mbit anyway).* > > performance info: > > Brick: zoidberg:/mnt/gluster/a > ----------------------------------------- > Cumulative Stats: > Block Size: 1b+ 8b+ > 16b+ > No. of Reads: 0 > 0 0 > No. of Writes: 117 > 195 367 > > Block Size: 32b+ 64b+ > 128b+ > No. of Reads: 306 > 0 6 > No. of Writes: 1016 > 9 134 > > Block Size: 256b+ 512b+ > 1024b+ > No. of Reads: 107 > 58 485 > No. of Writes: 731 > 73 168 > > Block Size: 2048b+ 4096b+ > 8192b+ > No. of Reads: 43 > 59 383 > No. of Writes: 295 475 > 10562 > > Block Size: 16384b+ 32768b+ > 65536b+ > No. of Reads: 347 > 380 634 > No. of Writes: 1487 6369 > 3970 > > Block Size: 131072b+ > No. of Reads: 35704 > No. of Writes: 12398 > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 957293 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 1455091 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 69951 > RELEASEDIR > 0.00 16.20 us 5.43 us 100.95 us > 113 IPC > 0.00 687.33 us 80.53 us 15414.03 us 30 > REMOVEXATTR > 0.00 840.76 us 72.71 us 37648.39 us 52 > FTRUNCATE > 0.00 138.49 us 36.66 us 25280.24 us 1385 > STAT > 0.00 364.93 us 63.45 us 15957.66 us 810 > FGETXATTR > 0.00 2208.82 us 127.23 us 267092.17 us 137 > LINK > 0.00 37595.33 us 1677.46 us 218313.32 us 12 > FSYNC > 0.00 675.61 us 106.82 us 114351.01 us 698 > TRUNCATE > 0.00 3995.66 us 147.00 us 109544.46 us 188 > RENAME > 0.00 453.11 us 53.37 us 104848.94 us 1680 > SETATTR > 0.00 6004.32 us 229.33 us 151322.58 us 127 > MKNOD > 0.00 68.74 us 16.11 us 61050.93 us 11412 > FINODELK > 0.00 3793.44 us 68.61 us 411523.70 us 221 > SETXATTR > 0.00 28669.99 us 227.28 us 138988.41 us 30 > SYMLINK > 0.00 56.18 us 7.31 us 41484.37 us 22640 > FLUSH > 0.00 105.64 us 19.34 us 351055.72 us 21693 > STATFS > 0.00 23642.90 us 244.19 us 234912.25 us 178 > MKDIR > 0.00 537.38 us 31.73 us 296911.67 us 9653 > FSTAT > 0.00 214.87 us 25.71 us 1712020.99 us 39054 > READ > 0.01 147.08 us 1.47 us 241611.80 us 69951 > OPENDIR > 0.01 3899.84 us 117.22 us 503422.21 us 3180 > RMDIR > 0.01 327.08 us 55.69 us 420376.80 us 38366 > WRITE > 0.02 5580.36 us 210.11 us 1029702.30 us 6169 > CREATE > 0.05 92.69 us 29.94 us 687955.86 us 1159936 > READLINK > 0.40 543.50 us 36.45 us 2430653.54 us 1449307 > OPEN > 0.48 87643.67 us 70.20 us 661999.51 us 10787 > FXATTROP > 0.53 2342.72 us 20.91 us 4157542.43 us 447821 > UNLINK > 0.53 558734.79 us 24.40 us 1399487.63 us > 1889 LK > 0.78 145786.31 us 19.06 us 37490825.73 us 10687 > READDIR > 0.99 2426.23 us 12.47 us 24176123.13 us 809430 > GETXATTR > 1.45 676.83 us 9.30 us 193177067.25 us 4260021 > ENTRYLK > 1.76 38325.38 us 21.44 us 46693575.93 us 91439 > READDIRP > 4.44 1200.31 us 9.10 us 2559377.90 us 7352811 > INODELK > 9.30 1849.24 us 9.28 us 35299050.00 us 9989221 > LOOKUP > 79.24 47791.46 us 76.77 us 10488824.26 us 3294642 > XATTROP > 0.00 0.00 us 0.00 us 0.00 us 3194806 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 5311 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 54 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 1907 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 3189457 > CI_FORGET > > Duration: 209524 seconds > Data Read: 4775365531 bytes > Data Written: 2354108664 bytes > > Interval 0 Stats: > Block Size: 1b+ 8b+ > 16b+ > No. of Reads: 0 > 0 0 > No. of Writes: 117 > 195 367 > > Block Size: 32b+ 64b+ > 128b+ > No. of Reads: 306 > 0 6 > No. of Writes: 1016 > 9 134 > > Block Size: 256b+ 512b+ > 1024b+ > No. of Reads: 107 > 58 485 > No. of Writes: 731 > 73 168 > > Block Size: 2048b+ 4096b+ > 8192b+ > No. of Reads: 43 > 59 383 > No. of Writes: 295 475 > 10562 > > Block Size: 16384b+ 32768b+ > 65536b+ > No. of Reads: 347 > 380 634 > No. of Writes: 1487 6369 > 3970 > > Block Size: 131072b+ > No. of Reads: 35704 > No. of Writes: 12398 > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 957293 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 1455091 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 69951 > RELEASEDIR > 0.00 16.20 us 5.43 us 100.95 us > 113 IPC > 0.00 687.33 us 80.53 us 15414.03 us 30 > REMOVEXATTR > 0.00 840.76 us 72.71 us 37648.39 us 52 > FTRUNCATE > 0.00 138.49 us 36.66 us 25280.24 us 1385 > STAT > 0.00 363.20 us 63.45 us 15957.66 us 810 > FGETXATTR > 0.00 2208.82 us 127.23 us 267092.17 us 137 > LINK > 0.00 37595.33 us 1677.46 us 218313.32 us 12 > FSYNC > 0.00 675.61 us 106.82 us 114351.01 us 698 > TRUNCATE > 0.00 3995.66 us 147.00 us 109544.46 us 188 > RENAME > 0.00 453.11 us 53.37 us 104848.94 us 1680 > SETATTR > 0.00 6004.32 us 229.33 us 151322.58 us 127 > MKNOD > 0.00 68.74 us 16.11 us 61050.93 us 11412 > FINODELK > 0.00 3793.44 us 68.61 us 411523.70 us 221 > SETXATTR > 0.00 28669.99 us 227.28 us 138988.41 us 30 > SYMLINK > 0.00 56.19 us 7.31 us 41484.37 us 22640 > FLUSH > 0.00 105.61 us 19.34 us 351055.72 us 21693 > STATFS > 0.00 23642.90 us 244.19 us 234912.25 us 178 > MKDIR > 0.00 537.38 us 31.73 us 296911.67 us 9653 > FSTAT > 0.00 214.91 us 25.71 us 1712020.99 us 39054 > READ > 0.01 147.08 us 1.47 us 241611.80 us 69951 > OPENDIR > 0.01 3899.84 us 117.22 us 503422.21 us 3180 > RMDIR > 0.01 327.13 us 55.69 us 420376.80 us 38366 > WRITE > 0.02 5580.36 us 210.11 us 1029702.30 us 6169 > CREATE > 0.05 92.69 us 29.94 us 687955.86 us 1159936 > READLINK > 0.40 543.49 us 36.45 us 2430653.54 us 1449307 > OPEN > 0.48 87643.67 us 70.20 us 661999.51 us 10787 > FXATTROP > 0.53 2342.57 us 20.91 us 4157542.43 us 447821 > UNLINK > 0.53 558416.10 us 24.40 us 1399487.63 us > 1889 LK > 0.78 145786.31 us 19.06 us 37490825.73 us 10687 > READDIR > 0.99 2426.24 us 12.47 us 24176123.13 us 809430 > GETXATTR > 1.45 676.75 us 9.30 us 193177067.25 us 4260021 > ENTRYLK > 1.76 38324.59 us 21.44 us 46693575.93 us 91439 > READDIRP > 4.44 1200.37 us 9.10 us 2559377.90 us 7352811 > INODELK > 9.29 1848.94 us 9.28 us 35299050.00 us 9989221 > LOOKUP > 79.24 47792.42 us 76.77 us 10488824.26 us 3294642 > XATTROP > 0.00 0.00 us 0.00 us 0.00 us 3194806 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 5311 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 54 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 1907 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 3189457 > CI_FORGET > > Duration: 209524 seconds > Data Read: 4775365531 bytes > Data Written: 2354108664 bytes > > Brick: zoidberg:/mnt/gluster/b > ----------------------------------------- > Cumulative Stats: > Block Size: 1b+ > 2b+ 8b+ > No. of Reads: 0 > 0 0 > No. of Writes: 53 > 1 60 > > Block Size: 16b+ 32b+ > 64b+ > No. of Reads: 0 > 213 2 > No. of Writes: 14 > 1057 15 > > Block Size: 128b+ 256b+ > 512b+ > No. of Reads: 0 > 41 146 > No. of Writes: 59 > 715 80 > > Block Size: 1024b+ 2048b+ > 4096b+ > No. of Reads: 224 > 9 15 > No. of Writes: 163 > 282 404 > > Block Size: 8192b+ 16384b+ > 32768b+ > No. of Reads: 52 > 74 210 > No. of Writes: 7415 1430 > 6860 > > Block Size: 65536b+ 131072b+ > 262144b+ > No. of Reads: 299 > 15313 1 > No. of Writes: 2458 > 12378 1 > > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 686733 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 837434 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 87338 > RELEASEDIR > 0.00 22.69 us 7.16 us 332.61 us > 116 IPC > 0.00 119.93 us 70.56 us 272.28 us 57 > FTRUNCATE > 0.00 152.25 us 86.82 us 340.63 us 65 > FGETXATTR > 0.00 2336.06 us 90.05 us 20509.81 us 23 > REMOVEXATTR > 0.00 631.26 us 123.17 us 51803.77 us 121 > LINK > 0.00 369.27 us 22.64 us 145090.48 us > 490 LK > 0.00 360.85 us 45.14 us 46846.98 us 540 > STAT > 0.00 1190.96 us 117.40 us 69781.85 us 200 > TRUNCATE > 0.00 245.72 us 54.91 us 123985.76 us 1482 > SETATTR > 0.00 42139.10 us 3313.56 us 312273.21 us 12 > FSYNC > 0.00 55.83 us 8.26 us 49649.35 us 10047 > FLUSH > 0.00 57.52 us 14.62 us 820.59 us 9977 > FINODELK > 0.00 3191.60 us 143.06 us 96214.75 us 188 > RENAME > 0.00 32862.86 us 215.38 us 83851.13 us 25 > SYMLINK > 0.00 11392.03 us 230.70 us 105471.90 us 118 > MKNOD > 0.00 134.63 us 28.08 us 36326.43 us 12587 > FSTAT > 0.00 112.32 us 19.91 us 381241.34 us 21695 > STATFS > 0.00 307.86 us 28.00 us 292666.83 us 16841 > READ > 0.00 29625.60 us 239.57 us 239532.26 us 178 > MKDIR > 0.00 319.24 us 51.19 us 136353.39 us 33445 > WRITE > 0.01 175.83 us 1.57 us 513635.22 us 87338 > OPENDIR > 0.01 249.96 us 58.42 us 594774.86 us 85089 > SETXATTR > 0.01 11001.25 us 119.87 us 612765.03 us 3180 > RMDIR > 0.04 14240.66 us 203.89 us 1376758.99 us 6067 > CREATE > 0.05 89.45 us 30.14 us 687961.60 us 1219542 > READLINK > 0.39 1114.30 us 44.29 us 2412169.89 us 831371 > OPEN > 0.51 127078.45 us 78.55 us 694364.09 us 9493 > FXATTROP > 0.63 399.16 us 8.43 us 3815689.36 us 3764714 > ENTRYLK > 0.86 91973.00 us 19.89 us 24258016.33 us 22253 > READDIR > 1.06 6913.71 us 11.58 us 36968532.72 us 364963 > GETXATTR > 1.29 62130.12 us 36.89 us 8649871.11 us 49472 > READDIRP > 2.89 1477.53 us 11.10 us 2807637.64 us 4650500 > INODELK > 4.12 23681.31 us 20.98 us 23291414.50 us 414352 > UNLINK > 16.09 5000.31 us 9.04 us 23866016.88 us 7655621 > LOOKUP > 72.04 68953.31 us 68.40 us 27542499.65 us 2486126 > XATTROP > 0.00 0.00 us 0.00 us 0.00 us 2433405 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 2453 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 69 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 1544 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 2430885 > CI_FORGET > > Duration: 209524 seconds > Data Read: 2051388524 bytes > Data Written: 2208261499 bytes > > Interval 0 Stats: > Block Size: 1b+ > 2b+ 8b+ > No. of Reads: 0 > 0 0 > No. of Writes: 53 > 1 60 > > Block Size: 16b+ 32b+ > 64b+ > No. of Reads: 0 > 213 2 > No. of Writes: 14 > 1057 15 > > Block Size: 128b+ 256b+ > 512b+ > No. of Reads: 0 > 41 146 > No. of Writes: 59 > 715 80 > > Block Size: 1024b+ 2048b+ > 4096b+ > No. of Reads: 224 > 9 15 > No. of Writes: 163 > 282 404 > > Block Size: 8192b+ 16384b+ > 32768b+ > No. of Reads: 52 > 74 210 > No. of Writes: 7415 1430 > 6860 > > Block Size: 65536b+ 131072b+ > 262144b+ > No. of Reads: 299 > 15313 1 > No. of Writes: 2458 > 12378 1 > > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 686733 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 837434 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 87338 > RELEASEDIR > 0.00 22.69 us 7.16 us 332.61 us > 116 IPC > 0.00 119.93 us 70.56 us 272.28 us 57 > FTRUNCATE > 0.00 152.25 us 86.82 us 340.63 us 65 > FGETXATTR > 0.00 2336.06 us 90.05 us 20509.81 us 23 > REMOVEXATTR > 0.00 631.26 us 123.17 us 51803.77 us 121 > LINK > 0.00 369.27 us 22.64 us 145090.48 us > 490 LK > 0.00 360.85 us 45.14 us 46846.98 us 540 > STAT > 0.00 1190.96 us 117.40 us 69781.85 us 200 > TRUNCATE > 0.00 245.72 us 54.91 us 123985.76 us 1482 > SETATTR > 0.00 42139.10 us 3313.56 us 312273.21 us 12 > FSYNC > 0.00 55.83 us 8.26 us 49649.35 us 10047 > FLUSH > 0.00 57.52 us 14.62 us 820.59 us 9977 > FINODELK > 0.00 3191.60 us 143.06 us 96214.75 us 188 > RENAME > 0.00 32862.86 us 215.38 us 83851.13 us 25 > SYMLINK > 0.00 11392.03 us 230.70 us 105471.90 us 118 > MKNOD > 0.00 134.63 us 28.08 us 36326.43 us 12587 > FSTAT > 0.00 112.32 us 19.91 us 381241.34 us 21695 > STATFS > 0.00 307.91 us 28.00 us 292666.83 us 16841 > READ > 0.00 29625.60 us 239.57 us 239532.26 us 178 > MKDIR > 0.00 319.22 us 51.19 us 136353.39 us 33445 > WRITE > 0.01 175.83 us 1.57 us 513635.22 us 87338 > OPENDIR > 0.01 249.95 us 58.42 us 594774.86 us 85089 > SETXATTR > 0.01 11001.25 us 119.87 us 612765.03 us 3180 > RMDIR > 0.04 14240.66 us 203.89 us 1376758.99 us 6067 > CREATE > 0.05 89.46 us 30.14 us 687961.60 us 1219542 > READLINK > 0.39 1114.30 us 44.29 us 2412169.89 us 831371 > OPEN > 0.51 127078.45 us 78.55 us 694364.09 us 9493 > FXATTROP > 0.63 399.16 us 8.43 us 3815689.36 us 3764714 > ENTRYLK > 0.86 91973.00 us 19.89 us 24258016.33 us 22253 > READDIR > 1.06 6913.26 us 11.58 us 36968532.72 us 364963 > GETXATTR > 1.29 62121.95 us 36.89 us 8649871.11 us 49472 > READDIRP > 2.89 1477.58 us 11.10 us 2807637.64 us 4650500 > INODELK > 4.12 23681.29 us 20.98 us 23291414.50 us 414352 > UNLINK > 16.09 5000.02 us 9.04 us 23866016.88 us 7655621 > LOOKUP > 72.04 68956.14 us 68.40 us 27542499.65 us 2486126 > XATTROP > 0.00 0.00 us 0.00 us 0.00 us 2433405 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 2453 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 69 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 1544 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 2430885 > CI_FORGET > > Duration: 209524 seconds > Data Read: 2051388524 bytes > Data Written: 2208261499 bytes > > Brick: flexo:/mnt/gluster/b > -------------------------------------- > Cumulative Stats: > Block Size: 1b+ 8b+ > 16b+ > No. of Reads: 0 > 0 0 > No. of Writes: 117 > 195 367 > > Block Size: 32b+ 64b+ > 128b+ > No. of Reads: 130 > 0 3 > No. of Writes: 1012 > 9 134 > > Block Size: 256b+ 512b+ > 1024b+ > No. of Reads: 47 > 8 257 > No. of Writes: 729 > 73 168 > > Block Size: 2048b+ 4096b+ > 8192b+ > No. of Reads: 10 > 27 240 > No. of Writes: 294 466 > 10472 > > Block Size: 16384b+ 32768b+ > 65536b+ > No. of Reads: 139 > 119 135 > No. of Writes: 1480 6273 > 3935 > > Block Size: 131072b+ > No. of Reads: 7892 > No. of Writes: 12327 > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 955058 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 1450749 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 81377 > RELEASEDIR > 0.00 15.55 us 5.35 us 31.88 us > 108 IPC > 0.00 125.85 us 59.77 us 224.15 us 44 > FTRUNCATE > 0.00 198.38 us 81.68 us 1332.45 us 30 > REMOVEXATTR > 0.00 4254.16 us 621.52 us 10602.99 us 4 > FSYNC > 0.00 795.08 us 190.30 us 16518.64 us 30 > SYMLINK > 0.00 195.90 us 111.02 us 953.39 us 137 > LINK > 0.00 286.07 us 179.64 us 437.82 us 119 > MKNOD > 0.00 162.61 us 65.62 us 3864.77 us 221 > SETXATTR > 0.00 117.29 us 48.50 us 317.17 us 354 > STAT > 0.00 70.35 us 23.10 us 317.32 us > 777 LK > 0.00 461.72 us 114.54 us 16013.81 us 188 > RENAME > 0.00 150.40 us 75.59 us 749.54 us 810 > FGETXATTR > 0.00 267.80 us 99.67 us 994.58 us 698 > TRUNCATE > 0.00 1176.00 us 194.89 us 78261.12 us 178 > MKDIR > 0.00 439.83 us 54.38 us 75270.51 us 1665 > SETATTR > 0.00 65.89 us 17.39 us 354.77 us 11372 > FINODELK > 0.00 131.57 us 32.68 us 2107.52 us 7398 > FSTAT > 0.00 120.71 us 31.43 us 847.97 us 9094 > READ > 0.00 60.75 us 7.25 us 22520.98 us 22614 > FLUSH > 0.00 73.03 us 22.41 us 750.17 us 21656 > STATFS > 0.00 323.63 us 166.56 us 23034.44 us 6145 > CREATE > 0.00 217.55 us 50.60 us 30734.99 us 38051 > WRITE > 0.01 146.39 us 1.80 us 318612.45 us 81377 > OPENDIR > 0.01 5641.37 us 108.84 us 468960.33 us 3177 > RMDIR > 0.03 100.05 us 30.93 us 49756.93 us 432439 > READLINK > 0.13 483.47 us 16.81 us 2608013.14 us 448381 > UNLINK > 0.21 7101.56 us 22.26 us 2554390.49 us 52185 > READDIRP > 0.35 414.05 us 36.58 us 1038229.93 us 1444606 > OPEN > 0.49 78518.02 us 73.39 us 373999.31 us 10745 > FXATTROP > 0.72 776.21 us 11.24 us 4271792.57 us 1605822 > GETXATTR > 0.77 97829.38 us 18.84 us 9477446.11 us 13532 > READDIR > 0.79 320.44 us 7.71 us 2369511.06 us 4257297 > ENTRYLK > 1.97 464.02 us 9.38 us 1038460.80 us 7333722 > INODELK > 13.73 2372.50 us 8.33 us 6258159.87 us 9979799 > LOOKUP > 80.79 42372.04 us 61.49 us 2238308.13 us 3289103 > XATTROP > 0.00 0.00 us 0.00 us 0.00 us 2250268 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 3756 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 167 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 1908 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 2246352 > CI_FORGET > > Duration: 209334 seconds > Data Read: 1059777395 bytes > Data Written: 2336921014 bytes > > Interval 0 Stats: > Block Size: 1b+ 8b+ > 16b+ > No. of Reads: 0 > 0 0 > No. of Writes: 117 > 195 367 > > Block Size: 32b+ 64b+ > 128b+ > No. of Reads: 130 > 0 3 > No. of Writes: 1012 > 9 134 > > Block Size: 256b+ 512b+ > 1024b+ > No. of Reads: 47 > 8 257 > No. of Writes: 729 > 73 168 > > Block Size: 2048b+ 4096b+ > 8192b+ > No. of Reads: 10 > 27 240 > No. of Writes: 294 466 > 10472 > > Block Size: 16384b+ 32768b+ > 65536b+ > No. of Reads: 139 > 119 135 > No. of Writes: 1480 6273 > 3935 > > Block Size: 131072b+ > No. of Reads: 7892 > No. of Writes: 12327 > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 955058 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 1450749 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 81377 > RELEASEDIR > 0.00 15.55 us 5.35 us 31.88 us > 108 IPC > 0.00 125.85 us 59.77 us 224.15 us 44 > FTRUNCATE > 0.00 198.38 us 81.68 us 1332.45 us 30 > REMOVEXATTR > 0.00 4254.16 us 621.52 us 10602.99 us 4 > FSYNC > 0.00 795.08 us 190.30 us 16518.64 us 30 > SYMLINK > 0.00 195.90 us 111.02 us 953.39 us 137 > LINK > 0.00 286.07 us 179.64 us 437.82 us 119 > MKNOD > 0.00 162.61 us 65.62 us 3864.77 us 221 > SETXATTR > 0.00 117.29 us 48.50 us 317.17 us 354 > STAT > 0.00 70.35 us 23.10 us 317.32 us > 777 LK > 0.00 461.72 us 114.54 us 16013.81 us 188 > RENAME > 0.00 150.43 us 75.59 us 749.54 us 810 > FGETXATTR > 0.00 267.80 us 99.67 us 994.58 us 698 > TRUNCATE > 0.00 1176.00 us 194.89 us 78261.12 us 178 > MKDIR > 0.00 439.83 us 54.38 us 75270.51 us 1665 > SETATTR > 0.00 65.89 us 17.39 us 354.77 us 11372 > FINODELK > 0.00 131.57 us 32.68 us 2107.52 us 7398 > FSTAT > 0.00 120.74 us 31.43 us 847.97 us 9094 > READ > 0.00 60.75 us 7.25 us 22520.98 us 22614 > FLUSH > 0.00 73.03 us 22.41 us 750.17 us 21656 > STATFS > 0.00 323.63 us 166.56 us 23034.44 us 6145 > CREATE > 0.00 217.50 us 50.60 us 30734.99 us 38051 > WRITE > 0.01 146.39 us 1.80 us 318612.45 us 81377 > OPENDIR > 0.01 5641.37 us 108.84 us 468960.33 us 3177 > RMDIR > 0.03 100.06 us 30.93 us 49756.93 us 432439 > READLINK > 0.13 483.47 us 16.81 us 2608013.14 us 448381 > UNLINK > 0.21 7101.56 us 22.26 us 2554390.49 us 52185 > READDIRP > 0.35 414.01 us 36.58 us 1038229.93 us 1444606 > OPEN > 0.49 78518.02 us 73.39 us 373999.31 us 10745 > FXATTROP > 0.72 776.24 us 11.24 us 4271792.57 us 1605822 > GETXATTR > 0.77 97829.38 us 18.84 us 9477446.11 us 13532 > READDIR > 0.79 320.45 us 7.71 us 2369511.06 us 4257297 > ENTRYLK > 1.97 464.07 us 9.38 us 1038460.80 us 7333722 > INODELK > 13.72 2372.42 us 8.33 us 6258159.87 us 9979799 > LOOKUP > 80.79 42373.50 us 61.49 us 2238308.13 us 3289103 > XATTROP > 0.00 0.00 us 0.00 us 0.00 us 2250268 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 3756 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 167 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 1908 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 2246352 > CI_FORGET > > Duration: 209334 seconds > Data Read: 1059777395 bytes > Data Written: 2336921014 bytes > > Brick: flexo:/mnt/gluster/c > -------------------------------------- > Cumulative Stats: > Block Size: 1b+ > No. of Reads: 0 > No. of Writes: 32843 > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 686562 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 834596 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 93757 > RELEASEDIR > 0.00 18.50 us 8.21 us 50.20 us > 107 IPC > 0.00 205.83 us 126.30 us 412.06 us 36 > LINK > 0.00 167.33 us 75.01 us 272.47 us 65 > FGETXATTR > 0.00 203.58 us 117.35 us 648.28 us 88 > RENAME > 0.00 203.20 us 66.60 us 809.96 us 153 > TRUNCATE > 0.00 67.06 us 23.64 us 555.59 us > 491 LK > 0.01 51.73 us 7.70 us 1174.64 us 9339 > FLUSH > 0.01 62.41 us 15.13 us 636.35 us 9011 > FINODELK > 0.03 126.23 us 43.15 us 4760.14 us 10824 > FSTAT > 0.03 571.94 us 94.85 us 51697.03 us 2594 > MKDIR > 0.05 264.87 us 81.58 us 53152.06 us 9009 > FXATTROP > 0.05 423.05 us 38.26 us 548686.55 us 5754 > CREATE > 0.05 78.14 us 19.17 us 678.32 us 32843 > WRITE > 0.06 526.39 us 130.84 us 41405.46 us 5798 > SYMLINK > 0.10 133.73 us 16.25 us 676494.72 us 34904 > READDIR > 0.12 2054.48 us 99.28 us 88061.83 us 2732 > RMDIR > 0.21 110.60 us 1.19 us 16833.78 us 93757 > OPENDIR > 0.72 205.89 us 33.28 us 261023.99 us 170292 > UNLINK > 0.85 2971.45 us 21.24 us 257839.19 us 13840 > READDIRP > 1.94 200.88 us 25.49 us 461654.32 us 468842 > FTRUNCATE > 2.13 212.44 us 54.02 us 461878.34 us 486882 > REMOVEXATTR > 2.27 223.01 us 52.03 us 178897.40 us 493240 > SETXATTR > 3.53 155.31 us 7.96 us 67873.00 us 1101338 > GETXATTR > 3.63 212.32 us 44.42 us 551264.02 us 829002 > OPEN > 5.77 76.36 us 7.76 us 962634.73 us 3660425 > ENTRYLK > 6.23 304.93 us 42.51 us 661394.43 us 990644 > SETATTR > 7.81 382.20 us 107.42 us 846502.78 us 990766 > XATTROP > 12.39 549.11 us 135.57 us 1659697.83 us 1093387 > MKNOD > 12.47 139.78 us 9.76 us 552067.56 us 4325281 > INODELK > 39.53 206.94 us 8.56 us 964823.33 us 9259429 > LOOKUP > 0.00 0.00 us 0.00 us 0.00 us 1856377 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 933300 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 251 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 293 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 923077 > CI_FORGET > > Duration: 209334 seconds > Data Read: 0 bytes > Data Written: 32843 bytes > > Interval 0 Stats: > Block Size: 1b+ > No. of Reads: 0 > No. of Writes: 32843 > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 686562 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 834596 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 93757 > RELEASEDIR > 0.00 18.50 us 8.21 us 50.20 us > 107 IPC > 0.00 205.83 us 126.30 us 412.06 us 36 > LINK > 0.00 167.33 us 75.01 us 272.47 us 65 > FGETXATTR > 0.00 203.58 us 117.35 us 648.28 us 88 > RENAME > 0.00 203.20 us 66.60 us 809.96 us 153 > TRUNCATE > 0.00 67.06 us 23.64 us 555.59 us > 491 LK > 0.01 51.71 us 7.70 us 1174.64 us 9339 > FLUSH > 0.01 62.41 us 15.13 us 636.35 us 9011 > FINODELK > 0.03 126.23 us 43.15 us 4760.14 us 10824 > FSTAT > 0.03 571.94 us 94.85 us 51697.03 us 2594 > MKDIR > 0.05 264.87 us 81.58 us 53152.06 us 9009 > FXATTROP > 0.05 423.05 us 38.26 us 548686.55 us 5754 > CREATE > 0.05 78.14 us 19.17 us 678.32 us 32843 > WRITE > 0.06 526.39 us 130.84 us 41405.46 us 5798 > SYMLINK > 0.10 133.73 us 16.25 us 676494.72 us 34904 > READDIR > 0.12 2054.48 us 99.28 us 88061.83 us 2732 > RMDIR > 0.21 110.60 us 1.19 us 16833.78 us 93757 > OPENDIR > 0.72 205.89 us 33.28 us 261023.99 us 170292 > UNLINK > 0.85 2971.45 us 21.24 us 257839.19 us 13840 > READDIRP > 1.94 200.87 us 25.49 us 461654.32 us 468842 > FTRUNCATE > 2.13 212.44 us 54.02 us 461878.34 us 486882 > REMOVEXATTR > 2.27 223.00 us 52.03 us 178897.40 us 493240 > SETXATTR > 3.53 155.32 us 7.96 us 67873.00 us 1101338 > GETXATTR > 3.63 212.32 us 44.42 us 551264.02 us 829002 > OPEN > 5.77 76.36 us 7.76 us 962634.73 us 3660425 > ENTRYLK > 6.23 304.88 us 42.51 us 661394.43 us 990644 > SETATTR > 7.81 382.20 us 107.42 us 846502.78 us 990766 > XATTROP > 12.39 549.25 us 135.57 us 1659697.83 us 1093387 > MKNOD > 12.47 139.78 us 9.76 us 552067.56 us 4325281 > INODELK > 39.53 206.95 us 8.56 us 964823.33 us 9259429 > LOOKUP > 0.00 0.00 us 0.00 us 0.00 us 1856377 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 933300 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 251 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 293 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 923077 > CI_FORGET > > Duration: 209334 seconds > Data Read: 0 bytes > Data Written: 32843 bytes > > Brick: bender:/mnt/gluster/c > --------------------------------------- > Cumulative Stats: > Block Size: 1b+ > No. of Reads: 0 > No. of Writes: 37050 > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 928119 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 1401524 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 70092 > RELEASEDIR > 0.00 19.49 us 8.28 us 180.61 us > 100 IPC > 0.00 238.88 us 163.97 us 486.93 us 38 > LINK > 0.00 306.43 us 196.67 us 788.36 us 30 > SYMLINK > 0.00 2670.18 us 765.93 us 4151.64 us 4 > FSYNC > 0.00 350.65 us 141.92 us 10573.64 us 122 > RENAME > 0.00 67.02 us 20.66 us 453.90 us > 777 LK > 0.00 158.32 us 64.22 us 532.21 us 810 > FGETXATTR > 0.00 229.87 us 66.47 us 826.31 us 603 > TRUNCATE > 0.01 1742.85 us 202.81 us 47471.71 us 302 > MKDIR > 0.01 59.05 us 15.40 us 1813.56 us 10060 > FINODELK > 0.01 118.54 us 36.65 us 4002.58 us 6409 > FSTAT > 0.01 49.38 us 8.08 us 1264.41 us 21719 > FLUSH > 0.03 74.76 us 20.86 us 2008.41 us 37050 > WRITE > 0.04 553.12 us 40.81 us 386485.29 us 5901 > CREATE > 0.06 534.01 us 100.86 us 527770.44 us 10057 > FXATTROP > 0.12 138.96 us 1.00 us 601208.70 us 70092 > OPENDIR > 0.12 947.76 us 18.54 us 712568.33 us 10779 > READDIR > 0.15 4099.07 us 124.69 us 421939.47 us 3008 > RMDIR > 0.61 465.30 us 20.67 us 936959.86 us 109478 > UNLINK > 1.57 182.41 us 49.54 us 73938.09 us 722669 > REMOVEXATTR > 1.61 171.41 us 6.98 us 83809.88 us 789294 > GETXATTR > 1.86 217.88 us 26.40 us 556446.19 us 718835 > FTRUNCATE > 1.89 219.47 us 49.92 us 506767.78 us 722856 > SETXATTR > 3.02 181.66 us 44.40 us 344656.69 us 1395957 > OPEN > 5.56 322.08 us 43.37 us 634069.61 us 1450160 > SETATTR > 5.84 115.88 us 7.67 us 747387.51 us 4233520 > ENTRYLK > 6.53 378.33 us 122.84 us 766939.14 us 1449601 > XATTROP > 9.75 115.48 us 10.70 us 556675.13 us 7097226 > INODELK > 19.37 1104.98 us 101.28 us 1612346.75 us 1472813 > MKNOD > 41.85 295.61 us 8.98 us 996805.54 us 11896091 > LOOKUP > 0.00 0.00 us 0.00 us 0.00 us 3531139 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 1862470 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 244 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 632 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 1668669 > CI_FORGET > > Duration: 208643 seconds > Data Read: 0 bytes > Data Written: 37050 bytes > > Interval 0 Stats: > Block Size: 1b+ > No. of Reads: 0 > No. of Writes: 37050 > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 928119 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 1401524 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 70092 > RELEASEDIR > 0.00 19.49 us 8.28 us 180.61 us > 100 IPC > 0.00 238.88 us 163.97 us 486.93 us 38 > LINK > 0.00 306.43 us 196.67 us 788.36 us 30 > SYMLINK > 0.00 2670.18 us 765.93 us 4151.64 us 4 > FSYNC > 0.00 350.65 us 141.92 us 10573.64 us 122 > RENAME > 0.00 67.02 us 20.66 us 453.90 us > 777 LK > 0.00 158.32 us 64.22 us 532.21 us 810 > FGETXATTR > 0.00 229.87 us 66.47 us 826.31 us 603 > TRUNCATE > 0.01 1742.85 us 202.81 us 47471.71 us 302 > MKDIR > 0.01 59.05 us 15.40 us 1813.56 us 10060 > FINODELK > 0.01 118.54 us 36.65 us 4002.58 us 6409 > FSTAT > 0.01 49.39 us 8.08 us 1264.41 us 21719 > FLUSH > 0.03 74.77 us 20.86 us 2008.41 us 37050 > WRITE > 0.04 553.12 us 40.81 us 386485.29 us 5901 > CREATE > 0.06 534.01 us 100.86 us 527770.44 us 10057 > FXATTROP > 0.12 138.96 us 1.00 us 601208.70 us 70092 > OPENDIR > 0.12 947.76 us 18.54 us 712568.33 us 10779 > READDIR > 0.15 4099.07 us 124.69 us 421939.47 us 3008 > RMDIR > 0.61 465.30 us 20.67 us 936959.86 us 109478 > UNLINK > 1.57 182.41 us 49.54 us 73938.09 us 722669 > REMOVEXATTR > 1.61 171.41 us 6.98 us 83809.88 us 789294 > GETXATTR > 1.86 217.88 us 26.40 us 556446.19 us 718835 > FTRUNCATE > 1.89 219.48 us 49.92 us 506767.78 us 722856 > SETXATTR > 3.02 181.66 us 44.40 us 344656.69 us 1395957 > OPEN > 5.56 322.09 us 43.37 us 634069.61 us 1450160 > SETATTR > 5.84 115.88 us 7.67 us 747387.51 us 4233520 > ENTRYLK > 6.53 378.33 us 122.84 us 766939.14 us 1449601 > XATTROP > 9.75 115.48 us 10.70 us 556675.13 us 7097226 > INODELK > 19.37 1105.06 us 101.28 us 1612346.75 us 1472813 > MKNOD > 41.84 295.54 us 8.98 us 996805.54 us 11896091 > LOOKUP > 0.00 0.00 us 0.00 us 0.00 us 3531139 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 1862470 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 244 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 632 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 1668669 > CI_FORGET > > Duration: 208643 seconds > Data Read: 0 bytes > Data Written: 37050 bytes > > Brick: bender:/mnt/gluster/a > --------------------------------------- > Cumulative Stats: > Block Size: 1b+ > 2b+ 8b+ > No. of Reads: 0 > 0 0 > No. of Writes: 53 > 1 60 > > Block Size: 16b+ 32b+ > 64b+ > No. of Reads: 0 > 235 4 > No. of Writes: 14 > 1033 15 > > Block Size: 128b+ 256b+ > 512b+ > No. of Reads: 4 > 115 81 > No. of Writes: 58 > 703 80 > > Block Size: 1024b+ 2048b+ > 4096b+ > No. of Reads: 181 > 26 74 > No. of Writes: 163 > 278 397 > > Block Size: 8192b+ 16384b+ > 32768b+ > No. of Reads: 131 > 161 267 > No. of Writes: 7264 1398 > 6742 > > Block Size: 65536b+ 131072b+ > 262144b+ > No. of Reads: 443 > 25528 0 > No. of Writes: 2426 > 12240 1 > > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 684942 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 837092 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 87297 > RELEASEDIR > 0.00 219.22 us 121.08 us 306.12 us 8 > REMOVEXATTR > 0.00 21.70 us 7.24 us 196.07 us > 97 IPC > 0.00 152.77 us 78.90 us 1059.82 us 46 > FTRUNCATE > 0.00 295.73 us 167.10 us 606.96 us 65 > FGETXATTR > 0.00 243.66 us 140.50 us 3392.16 us 121 > LINK > 0.00 1216.47 us 194.37 us 23411.42 us 25 > SYMLINK > 0.00 327.87 us 234.27 us 551.71 us 107 > MKNOD > 0.00 36094.61 us 36094.61 us 36094.61 us 1 > FSYNC > 0.00 219.63 us 147.34 us 783.10 us 188 > RENAME > 0.00 104.27 us 40.34 us 595.29 us 745 > STAT > 0.00 531.71 us 113.57 us 53973.08 us 200 > TRUNCATE > 0.00 289.05 us 63.10 us 70107.41 us 1444 > SETATTR > 0.00 2430.15 us 232.78 us 68120.54 us 177 > MKDIR > 0.00 52.68 us 7.28 us 826.01 us 9890 > FLUSH > 0.00 60.96 us 17.54 us 1393.95 us 9792 > FINODELK > 0.00 74.26 us 20.00 us 1522.39 us 21521 > STATFS > 0.00 129.21 us 34.15 us 9068.67 us 13094 > FSTAT > 0.00 456.99 us 195.50 us 130182.02 us 5967 > CREATE > 0.00 112.06 us 28.14 us 9545.53 us 27474 > READ > 0.01 313.76 us 52.98 us 71635.74 us 32926 > WRITE > 0.01 135.84 us 1.52 us 671373.08 us 87297 > OPENDIR > 0.01 5225.28 us 131.28 us 951323.69 us 3177 > RMDIR > 0.01 203.34 us 56.13 us 486756.90 us 85073 > SETXATTR > 0.03 3521.00 us 145.28 us 3396306.00 us 15090 > READDIRP > 0.04 113758.50 us 26.72 us 1147074.44 us > 564 LK > 0.05 92.18 us 29.71 us 39069.99 us 850938 > READLINK > 0.23 453.79 us 41.84 us 655999.23 us 831212 > OPEN > 0.41 29425.34 us 19.20 us 1526187.08 us 22482 > READDIR > 0.50 1954.70 us 21.51 us 1526772.28 us 415226 > UNLINK > 0.54 1272.83 us 9.66 us 1604368.93 us 686945 > GETXATTR > 0.55 237.57 us 7.91 us 1277047.91 us 3739718 > ENTRYLK > 0.60 103888.49 us 89.32 us 761509.35 us 9311 > FXATTROP > 2.60 909.04 us 9.68 us 2272778.49 us 4649291 > INODELK > 5.82 1240.20 us 8.78 us 1526868.50 us 7625383 > LOOKUP > 88.60 58005.98 us 67.31 us 1448810.06 us 2481914 > XATTROP > 0.00 0.00 us 0.00 us 0.00 us 1396875 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 1854 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 28 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 1051 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 1394996 > CI_FORGET > > Duration: 208643 seconds > Data Read: 3409342848 bytes > Data Written: 2180686507 bytes > > Interval 0 Stats: > Block Size: 1b+ > 2b+ 8b+ > No. of Reads: 0 > 0 0 > No. of Writes: 53 > 1 60 > > Block Size: 16b+ 32b+ > 64b+ > No. of Reads: 0 > 235 4 > No. of Writes: 14 > 1033 15 > > Block Size: 128b+ 256b+ > 512b+ > No. of Reads: 4 > 115 81 > No. of Writes: 58 > 703 80 > > Block Size: 1024b+ 2048b+ > 4096b+ > No. of Reads: 181 > 26 74 > No. of Writes: 163 > 278 397 > > Block Size: 8192b+ 16384b+ > 32768b+ > No. of Reads: 131 > 161 267 > No. of Writes: 7264 1398 > 6742 > > Block Size: 65536b+ 131072b+ > 262144b+ > No. of Reads: 443 > 25528 0 > No. of Writes: 2426 > 12240 1 > > %-latency Avg-latency Min-Latency Max-Latency No. of > calls Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 684942 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 837092 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 87297 > RELEASEDIR > 0.00 219.22 us 121.08 us 306.12 us 8 > REMOVEXATTR > 0.00 21.70 us 7.24 us 196.07 us > 97 IPC > 0.00 152.77 us 78.90 us 1059.82 us 46 > FTRUNCATE > 0.00 295.73 us 167.10 us 606.96 us 65 > FGETXATTR > 0.00 243.66 us 140.50 us 3392.16 us 121 > LINK > 0.00 1216.47 us 194.37 us 23411.42 us 25 > SYMLINK > 0.00 327.87 us 234.27 us 551.71 us 107 > MKNOD > 0.00 36094.61 us 36094.61 us 36094.61 us 1 > FSYNC > 0.00 219.63 us 147.34 us 783.10 us 188 > RENAME > 0.00 104.27 us 40.34 us 595.29 us 745 > STAT > 0.00 531.71 us 113.57 us 53973.08 us 200 > TRUNCATE > 0.00 289.05 us 63.10 us 70107.41 us 1444 > SETATTR > 0.00 2430.15 us 232.78 us 68120.54 us 177 > MKDIR > 0.00 52.69 us 7.28 us 826.01 us 9890 > FLUSH > 0.00 60.96 us 17.54 us 1393.95 us 9792 > FINODELK > 0.00 74.25 us 20.00 us 1522.39 us 21521 > STATFS > 0.00 129.23 us 34.15 us 9068.67 us 13094 > FSTAT > 0.00 456.99 us 195.50 us 130182.02 us 5967 > CREATE > 0.00 112.45 us 28.14 us 9545.53 us 27474 > READ > 0.01 313.77 us 52.98 us 71635.74 us 32926 > WRITE > 0.01 135.84 us 1.52 us 671373.08 us 87297 > OPENDIR > 0.01 5225.28 us 131.28 us 951323.69 us 3177 > RMDIR > 0.01 203.34 us 56.13 us 486756.90 us 85073 > SETXATTR > 0.03 3521.00 us 145.28 us 3396306.00 us 15090 > READDIRP > 0.04 113758.50 us 26.72 us 1147074.44 us > 564 LK > 0.05 92.18 us 29.71 us 39069.99 us 850938 > READLINK > 0.23 453.79 us 41.84 us 655999.23 us 831212 > OPEN > 0.41 29425.34 us 19.20 us 1526187.08 us 22482 > READDIR > 0.50 1954.70 us 21.51 us 1526772.28 us 415226 > UNLINK > 0.54 1272.79 us 9.66 us 1604368.93 us 686945 > GETXATTR > 0.55 237.57 us 7.91 us 1277047.91 us 3739718 > ENTRYLK > 0.60 103888.49 us 89.32 us 761509.35 us 9311 > FXATTROP > 2.60 909.04 us 9.68 us 2272778.49 us 4649291 > INODELK > 5.82 1240.10 us 8.78 us 1526868.50 us 7625383 > LOOKUP > 88.60 58006.02 us 67.31 us 1448810.06 us 2481914 > XATTROP > 0.00 0.00 us 0.00 us 0.00 us 1396875 > UPCALL > 0.00 0.00 us 0.00 us 0.00 us 1854 > CI_IATT > 0.00 0.00 us 0.00 us 0.00 us 28 > CI_XATTR > 0.00 0.00 us 0.00 us 0.00 us 1051 > CI_UNLINK > 0.00 0.00 us 0.00 us 0.00 us 1394996 > CI_FORGET > > Duration: 208643 seconds > Data Read: 3409342848 bytes > Data Written: 2180686507 bytes > > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200504/76239ebb/attachment.html>