Gluster users - May 2020 - Possible SHD stalling

Hi Artem,

I'll do that.? In the meantime consider this (inferred from the changelogs).

It seems like glusterfs SHD is queueing into a bounded queue under lock
conditions, and waiting for space to open up.? At least, this is what's
hinted in the changelogs, I've been trying to grasp the SHD code for two
days now and am not getting to wrap my head around that.? If this is
true, then the only way to avoid lock-ups are to ensure that the queue
is larger than the total number of files that can ever require healing
(effectively making the queue unbounded).

Kind Regards,
Jaco

On 2020/05/04 21:56, Artem Russakovskii wrote:
> I reported something similar recently to this list, and I highly
> advise you to update to 7.5 (in my case it was 5.11 -> 5.13), then see
> if it keeps happening. The self heal issues should be fixed there
> properly at least.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>,?APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net <http://beerpla.net/> | @ArtemR
<http://twitter.com/ArtemR>
>
>
> On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za
> <mailto:jaco at uls.co.za>> wrote:
>
>     Hi All,
>
>     So we've got a cluster currently running on glusterfs 7.3
>
>     Heal performance is absolutely terrible (it starts of great, runs
>     for a while then completely trips up).
>
>     Once the SHD trips up, access speeds to the cluster also becomes
>     poor.? And with poor I mean directory listings that normally takes
>     well under 500ms takes upwards of 10s, but I've seen upwards of
>     40s (although a follow up ls immediately after is nice and fast
>     again, temporarily).? With all SHDs killed this particular ls is
>     as per normal (around 30-50ms).
>
>     The SHD logs doesn't provide any usable clues, the only variance
>     in logs is this, just before all daemons across the cluster crawls
>     to a near halt.
>
>     [2020-05-02 19:55:58.627036] I [MSGID: 108026]
>     [afr-self-heal-common.c:1747:afr_log_selfheal]
>     0-vpbx_shared-replicate-0: Completed entry selfheal on
>     5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1 ?
>     [2020-05-02 19:55:58.627058] I [MSGID: 108026]
>     [afr-self-heal-common.c:1747:afr_log_selfheal]
>     0-vpbx_shared-replicate-0: Completed entry selfheal on
>     c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1 ?
>     ?
>     [2020-05-02 19:55:59.125001] W [MSGID: 114031]
>     [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
>     0-vpbx_shared-client-1: remote operation failed [Invalid argument]
>     ?
>     [2020-05-02 19:55:59.339803] I [MSGID: 108026]
>     [afr-self-heal-common.c:1747:afr_log_selfheal]
>     0-vpbx_shared-replicate-0: Completed entry selfheal on
>     2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1?
>     [2020-05-02 19:55:59.490387] I [MSGID: 108026]
>     [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
>     0-vpbx_shared-replicate-0: performing metadata selfheal on
>     527a63a3-9acd-4c3f-b95c-5c719325c973
>
>     A few more entries will complete overall and then it'll be a file
>     every other second.? Once I saw this I ran a few more tests, and
>     on every test just prior to slowing to a crawl, that entry is there.
>
>     There is a perceived improvement when lowering the thread count,
>     but not significant enough to confirm a correlation.? Even
>     with**cluster.shd-max-threads=1 this still happens (gets a few
>     tens thousand files completed instead of few hundred, at 32 it
>     gets to maybe 400 to 500 before it stalls, 16 does maybe a thousand).*
>     *
>
>     There are probably performant bursts hereafter, because from
>     experience we know this process takes about two months.? But this
>     time around it seems to be much, much worse.? Might be because
>     that was healing a single brick to another brick, so less locking
>     going?? At the moment the replicas are basically trying to
>     populate the newly added arbiters.? Also, we haven't messed as
>     greatly with the health check settings (queues and threads and
>     locks nearly as much as since the arbiter addition because
>     performance has simply dropped to non-acceptable levels.
>
>     Unfortunately granular entry heal isn't on at the moment (nor can
>     I switch it on due to a few million outstanding heals).? This used
>     to be a 2x2 cluster.? We then moved one of the bricks to a newly
>     added server using replace-brick.? What a frightening process this
>     is.? We then proceeded to add the arbiters.
>
>     Some of the settings below was absolute shots in the dark, based
>     on the little information we've been able to track down so far.
>
>     Kind Regards,
>     Jaco
>
>     The current gluster volume info (this used to be 2x2
>     distributed-replicate, aiming to get it to 3x (2 + 1) ... at
>     current rates we might get there by 2021):
>
>     Volume Name: glvol
>     Type: Distributed-Replicate
>     Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb
>     Status: Started
>     Snapshot Count: 0
>     Number of Bricks: 2 x (2 + 1) = 6
>     Transport-type: tcp
>     Bricks:
>     Brick1: zoidberg:/mnt/gluster/a
>     Brick2: flexo:/mnt/gluster/b
>     Brick3: bender:/mnt/gluster/c (arbiter)
>     Brick4: bender:/mnt/gluster/a
>     Brick5: zoidberg:/mnt/gluster/b
>     Brick6: flexo:/mnt/gluster/c (arbiter)
>     Options Reconfigured:
>     *cluster.heal-timeout: 5 <-- hoping this would at least shorten
>     the blockages.*
>     cluster.locking-scheme: granular
>     cluster.heal-wait-queue-length: 10240
>     *diagnostics.count-fop-hits: on**
>     **diagnostics.latency-measurement: on <-- was hoping profiling
>     might provide some insight.? Going to switch this off now.*
>     cluster.background-self-heal-count: 128
>     *cluster.shd-wait-qlength: 131072**
>     **cluster.shd-max-threads: 32? <-- even with this disk IO never
>     goes over 10% utilization.? CPU goes to about 2 cores.? So it's
>     lock contention that's slowing it down we reckon.*
>     *cluster.metadata-self-heal: off**
>     **cluster.entry-self-heal: off**
>     **cluster.data-self-heal: off**<-- these kill performance.
>     *transport.address-family: inet
>     nfs.disable: on
>     performance.client-io-threads: on
>     features.cache-invalidation: on
>     features.cache-invalidation-timeout: 600
>     performance.stat-prefetch: on
>     performance.cache-invalidation: on
>     performance.md-cache-timeout: 600
>     *cluster.self-heal-daemon: yes <-- disabling/enabling this
doesn't
>     help, have to kill the shd and start ... force to get things
>     moving along again.*
>     performance.io-thread-count: 16
>     performance.high-prio-threads: 16
>     performance.normal-prio-threads: 16
>     performance.low-prio-threads: 16
>     performance.least-prio-threads: 4
>     cluster.self-heal-window-size: 512
>     cluster.self-heal-readdir-size: 131072
>     client.event-threads: 8
>     server.event-threads: 16
>     network.ping-timeout: 2
>     *cluster.data-self-heal-algorithm: full <-- most files are small
>     (<1MB), on dual 1gbit links (so up to 2gbit, but b/w never goes
>     over 5mbit anyway).*
>
>     performance info:
>
>     Brick: zoidberg:/mnt/gluster/a
>     -----------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     8b+????????????????? 16b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:????????????????? 117??????????????????
>     195?????????????????? 367
>     ?
>     ?? Block Size:???????????????? 32b+?????????????????
>     64b+???????????????? 128b+
>     ?No. of Reads:????????????????? 306????????????????????
>     0???????????????????? 6
>     No. of Writes:???????????????? 1016????????????????????
>     9?????????????????? 134
>     ?
>     ?? Block Size:??????????????? 256b+????????????????
>     512b+??????????????? 1024b+
>     ?No. of Reads:????????????????? 107???????????????????
>     58?????????????????? 485
>     No. of Writes:????????????????? 731???????????????????
>     73?????????????????? 168
>     ?
>     ?? Block Size:?????????????? 2048b+???????????????
>     4096b+??????????????? 8192b+
>     ?No. of Reads:?????????????????? 43???????????????????
>     59?????????????????? 383
>     No. of Writes:????????????????? 295??????????????????
>     475???????????????? 10562
>     ?
>     ?? Block Size:????????????? 16384b+??????????????
>     32768b+?????????????? 65536b+
>     ?No. of Reads:????????????????? 347??????????????????
>     380?????????????????? 634
>     No. of Writes:???????????????? 1487?????????????????
>     6369????????????????? 3970
>     ?
>     ?? Block Size:???????????? 131072b+
>     ?No. of Reads:??????????????? 35704
>     No. of Writes:??????????????? 12398
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     957293????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1455091???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     69951? RELEASEDIR
>     ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us???????????
>     113???????? IPC
>     ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 us????????????
>     30 REMOVEXATTR
>     ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 us????????????
>     52?? FTRUNCATE
>     ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us??????????
>     1385??????? STAT
>     ????? 0.00???? 364.93 us????? 63.45 us?? 15957.66 us???????????
>     810?? FGETXATTR
>     ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us???????????
>     137??????? LINK
>     ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 us????????????
>     12?????? FSYNC
>     ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us???????????
>     698??? TRUNCATE
>     ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us???????????
>     188????? RENAME
>     ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us??????????
>     1680???? SETATTR
>     ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us???????????
>     127?????? MKNOD
>     ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us?????????
>     11412??? FINODELK
>     ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us???????????
>     221??? SETXATTR
>     ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 us????????????
>     30???? SYMLINK
>     ????? 0.00????? 56.18 us?????? 7.31 us?? 41484.37 us?????????
>     22640?????? FLUSH
>     ????? 0.00???? 105.64 us????? 19.34 us? 351055.72 us?????????
>     21693????? STATFS
>     ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us??????????
>     9653?????? FSTAT
>     ????? 0.00???? 214.87 us????? 25.71 us 1712020.99 us?????????
>     39054??????? READ
>     ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us?????????
>     69951???? OPENDIR
>     ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us??????????
>     3180?????? RMDIR
>     ????? 0.01???? 327.08 us????? 55.69 us? 420376.80 us?????????
>     38366?????? WRITE
>     ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us??????????
>     6169????? CREATE
>     ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us???????
>     1159936??? READLINK
>     ????? 0.40???? 543.50 us????? 36.45 us 2430653.54 us???????
>     1449307??????? OPEN
>     ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us?????????
>     10787??? FXATTROP
>     ????? 0.53??? 2342.72 us????? 20.91 us 4157542.43 us????????
>     447821????? UNLINK
>     ????? 0.53? 558734.79 us????? 24.40 us 1399487.63 us??????????
>     1889????????? LK
>     ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us?????????
>     10687???? READDIR
>     ????? 0.99??? 2426.23 us????? 12.47 us 24176123.13 us????????
>     809430??? GETXATTR
>     ????? 1.45???? 676.83 us?????? 9.30 us 193177067.25 us???????
>     4260021???? ENTRYLK
>     ????? 1.76?? 38325.38 us????? 21.44 us 46693575.93 us?????????
>     91439??? READDIRP
>     ????? 4.44??? 1200.31 us?????? 9.10 us 2559377.90 us???????
>     7352811???? INODELK
>     ????? 9.30??? 1849.24 us?????? 9.28 us 35299050.00 us???????
>     9989221????? LOOKUP
>     ???? 79.24?? 47791.46 us????? 76.77 us 10488824.26 us???????
>     3294642???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3194806????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     5311???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     54??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1907?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3189457?? CI_FORGET
>     ?
>     ??? Duration: 209524 seconds
>     ?? Data Read: 4775365531 bytes
>     Data Written: 2354108664 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     8b+????????????????? 16b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:????????????????? 117??????????????????
>     195?????????????????? 367
>     ?
>     ?? Block Size:???????????????? 32b+?????????????????
>     64b+???????????????? 128b+
>     ?No. of Reads:????????????????? 306????????????????????
>     0???????????????????? 6
>     No. of Writes:???????????????? 1016????????????????????
>     9?????????????????? 134
>     ?
>     ?? Block Size:??????????????? 256b+????????????????
>     512b+??????????????? 1024b+
>     ?No. of Reads:????????????????? 107???????????????????
>     58?????????????????? 485
>     No. of Writes:????????????????? 731???????????????????
>     73?????????????????? 168
>     ?
>     ?? Block Size:?????????????? 2048b+???????????????
>     4096b+??????????????? 8192b+
>     ?No. of Reads:?????????????????? 43???????????????????
>     59?????????????????? 383
>     No. of Writes:????????????????? 295??????????????????
>     475???????????????? 10562
>     ?
>     ?? Block Size:????????????? 16384b+??????????????
>     32768b+?????????????? 65536b+
>     ?No. of Reads:????????????????? 347??????????????????
>     380?????????????????? 634
>     No. of Writes:???????????????? 1487?????????????????
>     6369????????????????? 3970
>     ?
>     ?? Block Size:???????????? 131072b+
>     ?No. of Reads:??????????????? 35704
>     No. of Writes:??????????????? 12398
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     957293????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1455091???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     69951? RELEASEDIR
>     ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us???????????
>     113???????? IPC
>     ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 us????????????
>     30 REMOVEXATTR
>     ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 us????????????
>     52?? FTRUNCATE
>     ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us??????????
>     1385??????? STAT
>     ????? 0.00???? 363.20 us????? 63.45 us?? 15957.66 us???????????
>     810?? FGETXATTR
>     ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us???????????
>     137??????? LINK
>     ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 us????????????
>     12?????? FSYNC
>     ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us???????????
>     698??? TRUNCATE
>     ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us???????????
>     188????? RENAME
>     ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us??????????
>     1680???? SETATTR
>     ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us???????????
>     127?????? MKNOD
>     ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us?????????
>     11412??? FINODELK
>     ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us???????????
>     221??? SETXATTR
>     ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 us????????????
>     30???? SYMLINK
>     ????? 0.00????? 56.19 us?????? 7.31 us?? 41484.37 us?????????
>     22640?????? FLUSH
>     ????? 0.00???? 105.61 us????? 19.34 us? 351055.72 us?????????
>     21693????? STATFS
>     ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us??????????
>     9653?????? FSTAT
>     ????? 0.00???? 214.91 us????? 25.71 us 1712020.99 us?????????
>     39054??????? READ
>     ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us?????????
>     69951???? OPENDIR
>     ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us??????????
>     3180?????? RMDIR
>     ????? 0.01???? 327.13 us????? 55.69 us? 420376.80 us?????????
>     38366?????? WRITE
>     ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us??????????
>     6169????? CREATE
>     ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us???????
>     1159936??? READLINK
>     ????? 0.40???? 543.49 us????? 36.45 us 2430653.54 us???????
>     1449307??????? OPEN
>     ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us?????????
>     10787??? FXATTROP
>     ????? 0.53??? 2342.57 us????? 20.91 us 4157542.43 us????????
>     447821????? UNLINK
>     ????? 0.53? 558416.10 us????? 24.40 us 1399487.63 us??????????
>     1889????????? LK
>     ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us?????????
>     10687???? READDIR
>     ????? 0.99??? 2426.24 us????? 12.47 us 24176123.13 us????????
>     809430??? GETXATTR
>     ????? 1.45???? 676.75 us?????? 9.30 us 193177067.25 us???????
>     4260021???? ENTRYLK
>     ????? 1.76?? 38324.59 us????? 21.44 us 46693575.93 us?????????
>     91439??? READDIRP
>     ????? 4.44??? 1200.37 us?????? 9.10 us 2559377.90 us???????
>     7352811???? INODELK
>     ????? 9.29??? 1848.94 us?????? 9.28 us 35299050.00 us???????
>     9989221????? LOOKUP
>     ???? 79.24?? 47792.42 us????? 76.77 us 10488824.26 us???????
>     3294642???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3194806????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     5311???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     54??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1907?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3189457?? CI_FORGET
>     ?
>     ??? Duration: 209524 seconds
>     ?? Data Read: 4775365531 bytes
>     Data Written: 2354108664 bytes
>     ?
>     Brick: zoidberg:/mnt/gluster/b
>     -----------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     2b+?????????????????? 8b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:?????????????????? 53????????????????????
>     1??????????????????? 60
>     ?
>     ?? Block Size:???????????????? 16b+?????????????????
>     32b+????????????????? 64b+
>     ?No. of Reads:??????????????????? 0??????????????????
>     213???????????????????? 2
>     No. of Writes:?????????????????? 14?????????????????
>     1057??????????????????? 15
>     ?
>     ?? Block Size:??????????????? 128b+????????????????
>     256b+???????????????? 512b+
>     ?No. of Reads:??????????????????? 0???????????????????
>     41?????????????????? 146
>     No. of Writes:?????????????????? 59??????????????????
>     715??????????????????? 80
>     ?
>     ?? Block Size:?????????????? 1024b+???????????????
>     2048b+??????????????? 4096b+
>     ?No. of Reads:????????????????? 224????????????????????
>     9??????????????????? 15
>     No. of Writes:????????????????? 163??????????????????
>     282?????????????????? 404
>     ?
>     ?? Block Size:?????????????? 8192b+??????????????
>     16384b+?????????????? 32768b+
>     ?No. of Reads:?????????????????? 52???????????????????
>     74?????????????????? 210
>     No. of Writes:???????????????? 7415?????????????????
>     1430????????????????? 6860
>     ?
>     ?? Block Size:????????????? 65536b+?????????????
>     131072b+????????????? 262144b+
>     ?No. of Reads:????????????????? 299????????????????
>     15313???????????????????? 1
>     No. of Writes:???????????????? 2458????????????????
>     12378???????????????????? 1
>     ?
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     686733????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     837434???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     87338? RELEASEDIR
>     ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us???????????
>     116???????? IPC
>     ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 us????????????
>     57?? FTRUNCATE
>     ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 us????????????
>     65?? FGETXATTR
>     ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 us????????????
>     23 REMOVEXATTR
>     ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us???????????
>     121??????? LINK
>     ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us???????????
>     490????????? LK
>     ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us???????????
>     540??????? STAT
>     ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us???????????
>     200??? TRUNCATE
>     ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us??????????
>     1482???? SETATTR
>     ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 us????????????
>     12?????? FSYNC
>     ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us?????????
>     10047?????? FLUSH
>     ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us??????????
>     9977??? FINODELK
>     ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us???????????
>     188????? RENAME
>     ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 us????????????
>     25???? SYMLINK
>     ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us???????????
>     118?????? MKNOD
>     ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us?????????
>     12587?????? FSTAT
>     ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us?????????
>     21695????? STATFS
>     ????? 0.00???? 307.86 us????? 28.00 us? 292666.83 us?????????
>     16841??????? READ
>     ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 319.24 us????? 51.19 us? 136353.39 us?????????
>     33445?????? WRITE
>     ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us?????????
>     87338???? OPENDIR
>     ????? 0.01???? 249.96 us????? 58.42 us? 594774.86 us?????????
>     85089??? SETXATTR
>     ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us??????????
>     3180?????? RMDIR
>     ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us??????????
>     6067????? CREATE
>     ????? 0.05????? 89.45 us????? 30.14 us? 687961.60 us???????
>     1219542??? READLINK
>     ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us????????
>     831371??????? OPEN
>     ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us??????????
>     9493??? FXATTROP
>     ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us???????
>     3764714???? ENTRYLK
>     ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us?????????
>     22253???? READDIR
>     ????? 1.06??? 6913.71 us????? 11.58 us 36968532.72 us????????
>     364963??? GETXATTR
>     ????? 1.29?? 62130.12 us????? 36.89 us 8649871.11 us?????????
>     49472??? READDIRP
>     ????? 2.89??? 1477.53 us????? 11.10 us 2807637.64 us???????
>     4650500???? INODELK
>     ????? 4.12?? 23681.31 us????? 20.98 us 23291414.50 us????????
>     414352????? UNLINK
>     ???? 16.09??? 5000.31 us?????? 9.04 us 23866016.88 us???????
>     7655621????? LOOKUP
>     ???? 72.04?? 68953.31 us????? 68.40 us 27542499.65 us???????
>     2486126???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2433405????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     2453???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     69??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1544?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2430885?? CI_FORGET
>     ?
>     ??? Duration: 209524 seconds
>     ?? Data Read: 2051388524 bytes
>     Data Written: 2208261499 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     2b+?????????????????? 8b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:?????????????????? 53????????????????????
>     1??????????????????? 60
>     ?
>     ?? Block Size:???????????????? 16b+?????????????????
>     32b+????????????????? 64b+
>     ?No. of Reads:??????????????????? 0??????????????????
>     213???????????????????? 2
>     No. of Writes:?????????????????? 14?????????????????
>     1057??????????????????? 15
>     ?
>     ?? Block Size:??????????????? 128b+????????????????
>     256b+???????????????? 512b+
>     ?No. of Reads:??????????????????? 0???????????????????
>     41?????????????????? 146
>     No. of Writes:?????????????????? 59??????????????????
>     715??????????????????? 80
>     ?
>     ?? Block Size:?????????????? 1024b+???????????????
>     2048b+??????????????? 4096b+
>     ?No. of Reads:????????????????? 224????????????????????
>     9??????????????????? 15
>     No. of Writes:????????????????? 163??????????????????
>     282?????????????????? 404
>     ?
>     ?? Block Size:?????????????? 8192b+??????????????
>     16384b+?????????????? 32768b+
>     ?No. of Reads:?????????????????? 52???????????????????
>     74?????????????????? 210
>     No. of Writes:???????????????? 7415?????????????????
>     1430????????????????? 6860
>     ?
>     ?? Block Size:????????????? 65536b+?????????????
>     131072b+????????????? 262144b+
>     ?No. of Reads:????????????????? 299????????????????
>     15313???????????????????? 1
>     No. of Writes:???????????????? 2458????????????????
>     12378???????????????????? 1
>     ?
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     686733????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     837434???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     87338? RELEASEDIR
>     ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us???????????
>     116???????? IPC
>     ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 us????????????
>     57?? FTRUNCATE
>     ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 us????????????
>     65?? FGETXATTR
>     ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 us????????????
>     23 REMOVEXATTR
>     ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us???????????
>     121??????? LINK
>     ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us???????????
>     490????????? LK
>     ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us???????????
>     540??????? STAT
>     ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us???????????
>     200??? TRUNCATE
>     ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us??????????
>     1482???? SETATTR
>     ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 us????????????
>     12?????? FSYNC
>     ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us?????????
>     10047?????? FLUSH
>     ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us??????????
>     9977??? FINODELK
>     ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us???????????
>     188????? RENAME
>     ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 us????????????
>     25???? SYMLINK
>     ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us???????????
>     118?????? MKNOD
>     ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us?????????
>     12587?????? FSTAT
>     ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us?????????
>     21695????? STATFS
>     ????? 0.00???? 307.91 us????? 28.00 us? 292666.83 us?????????
>     16841??????? READ
>     ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 319.22 us????? 51.19 us? 136353.39 us?????????
>     33445?????? WRITE
>     ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us?????????
>     87338???? OPENDIR
>     ????? 0.01???? 249.95 us????? 58.42 us? 594774.86 us?????????
>     85089??? SETXATTR
>     ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us??????????
>     3180?????? RMDIR
>     ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us??????????
>     6067????? CREATE
>     ????? 0.05????? 89.46 us????? 30.14 us? 687961.60 us???????
>     1219542??? READLINK
>     ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us????????
>     831371??????? OPEN
>     ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us??????????
>     9493??? FXATTROP
>     ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us???????
>     3764714???? ENTRYLK
>     ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us?????????
>     22253???? READDIR
>     ????? 1.06??? 6913.26 us????? 11.58 us 36968532.72 us????????
>     364963??? GETXATTR
>     ????? 1.29?? 62121.95 us????? 36.89 us 8649871.11 us?????????
>     49472??? READDIRP
>     ????? 2.89??? 1477.58 us????? 11.10 us 2807637.64 us???????
>     4650500???? INODELK
>     ????? 4.12?? 23681.29 us????? 20.98 us 23291414.50 us????????
>     414352????? UNLINK
>     ???? 16.09??? 5000.02 us?????? 9.04 us 23866016.88 us???????
>     7655621????? LOOKUP
>     ???? 72.04?? 68956.14 us????? 68.40 us 27542499.65 us???????
>     2486126???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2433405????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     2453???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     69??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1544?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2430885?? CI_FORGET
>     ?
>     ??? Duration: 209524 seconds
>     ?? Data Read: 2051388524 bytes
>     Data Written: 2208261499 bytes
>     ?
>     Brick: flexo:/mnt/gluster/b
>     --------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     8b+????????????????? 16b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:????????????????? 117??????????????????
>     195?????????????????? 367
>     ?
>     ?? Block Size:???????????????? 32b+?????????????????
>     64b+???????????????? 128b+
>     ?No. of Reads:????????????????? 130????????????????????
>     0???????????????????? 3
>     No. of Writes:???????????????? 1012????????????????????
>     9?????????????????? 134
>     ?
>     ?? Block Size:??????????????? 256b+????????????????
>     512b+??????????????? 1024b+
>     ?No. of Reads:?????????????????? 47????????????????????
>     8?????????????????? 257
>     No. of Writes:????????????????? 729???????????????????
>     73?????????????????? 168
>     ?
>     ?? Block Size:?????????????? 2048b+???????????????
>     4096b+??????????????? 8192b+
>     ?No. of Reads:?????????????????? 10???????????????????
>     27?????????????????? 240
>     No. of Writes:????????????????? 294??????????????????
>     466???????????????? 10472
>     ?
>     ?? Block Size:????????????? 16384b+??????????????
>     32768b+?????????????? 65536b+
>     ?No. of Reads:????????????????? 139??????????????????
>     119?????????????????? 135
>     No. of Writes:???????????????? 1480?????????????????
>     6273????????????????? 3935
>     ?
>     ?? Block Size:???????????? 131072b+
>     ?No. of Reads:???????????????? 7892
>     No. of Writes:??????????????? 12327
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     955058????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1450749???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     81377? RELEASEDIR
>     ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us???????????
>     108???????? IPC
>     ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 us????????????
>     44?? FTRUNCATE
>     ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 us????????????
>     30 REMOVEXATTR
>     ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 us?????????????
>     4?????? FSYNC
>     ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 us????????????
>     30???? SYMLINK
>     ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us???????????
>     137??????? LINK
>     ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us???????????
>     119?????? MKNOD
>     ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us???????????
>     221??? SETXATTR
>     ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us???????????
>     354??????? STAT
>     ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us???????????
>     777????????? LK
>     ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us???????????
>     188????? RENAME
>     ????? 0.00???? 150.40 us????? 75.59 us???? 749.54 us???????????
>     810?? FGETXATTR
>     ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us???????????
>     698??? TRUNCATE
>     ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us??????????
>     1665???? SETATTR
>     ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us?????????
>     11372??? FINODELK
>     ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us??????????
>     7398?????? FSTAT
>     ????? 0.00???? 120.71 us????? 31.43 us???? 847.97 us??????????
>     9094??????? READ
>     ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us?????????
>     22614?????? FLUSH
>     ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us?????????
>     21656????? STATFS
>     ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us??????????
>     6145????? CREATE
>     ????? 0.00???? 217.55 us????? 50.60 us?? 30734.99 us?????????
>     38051?????? WRITE
>     ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us?????????
>     81377???? OPENDIR
>     ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us??????????
>     3177?????? RMDIR
>     ????? 0.03???? 100.05 us????? 30.93 us?? 49756.93 us????????
>     432439??? READLINK
>     ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us????????
>     448381????? UNLINK
>     ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us?????????
>     52185??? READDIRP
>     ????? 0.35???? 414.05 us????? 36.58 us 1038229.93 us???????
>     1444606??????? OPEN
>     ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us?????????
>     10745??? FXATTROP
>     ????? 0.72???? 776.21 us????? 11.24 us 4271792.57 us???????
>     1605822??? GETXATTR
>     ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us?????????
>     13532???? READDIR
>     ????? 0.79???? 320.44 us?????? 7.71 us 2369511.06 us???????
>     4257297???? ENTRYLK
>     ????? 1.97???? 464.02 us?????? 9.38 us 1038460.80 us???????
>     7333722???? INODELK
>     ???? 13.73??? 2372.50 us?????? 8.33 us 6258159.87 us???????
>     9979799????? LOOKUP
>     ???? 80.79?? 42372.04 us????? 61.49 us 2238308.13 us???????
>     3289103???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2250268????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     3756???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     167??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1908?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2246352?? CI_FORGET
>     ?
>     ??? Duration: 209334 seconds
>     ?? Data Read: 1059777395 bytes
>     Data Written: 2336921014 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     8b+????????????????? 16b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:????????????????? 117??????????????????
>     195?????????????????? 367
>     ?
>     ?? Block Size:???????????????? 32b+?????????????????
>     64b+???????????????? 128b+
>     ?No. of Reads:????????????????? 130????????????????????
>     0???????????????????? 3
>     No. of Writes:???????????????? 1012????????????????????
>     9?????????????????? 134
>     ?
>     ?? Block Size:??????????????? 256b+????????????????
>     512b+??????????????? 1024b+
>     ?No. of Reads:?????????????????? 47????????????????????
>     8?????????????????? 257
>     No. of Writes:????????????????? 729???????????????????
>     73?????????????????? 168
>     ?
>     ?? Block Size:?????????????? 2048b+???????????????
>     4096b+??????????????? 8192b+
>     ?No. of Reads:?????????????????? 10???????????????????
>     27?????????????????? 240
>     No. of Writes:????????????????? 294??????????????????
>     466???????????????? 10472
>     ?
>     ?? Block Size:????????????? 16384b+??????????????
>     32768b+?????????????? 65536b+
>     ?No. of Reads:????????????????? 139??????????????????
>     119?????????????????? 135
>     No. of Writes:???????????????? 1480?????????????????
>     6273????????????????? 3935
>     ?
>     ?? Block Size:???????????? 131072b+
>     ?No. of Reads:???????????????? 7892
>     No. of Writes:??????????????? 12327
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     955058????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1450749???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     81377? RELEASEDIR
>     ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us???????????
>     108???????? IPC
>     ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 us????????????
>     44?? FTRUNCATE
>     ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 us????????????
>     30 REMOVEXATTR
>     ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 us?????????????
>     4?????? FSYNC
>     ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 us????????????
>     30???? SYMLINK
>     ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us???????????
>     137??????? LINK
>     ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us???????????
>     119?????? MKNOD
>     ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us???????????
>     221??? SETXATTR
>     ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us???????????
>     354??????? STAT
>     ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us???????????
>     777????????? LK
>     ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us???????????
>     188????? RENAME
>     ????? 0.00???? 150.43 us????? 75.59 us???? 749.54 us???????????
>     810?? FGETXATTR
>     ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us???????????
>     698??? TRUNCATE
>     ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us??????????
>     1665???? SETATTR
>     ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us?????????
>     11372??? FINODELK
>     ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us??????????
>     7398?????? FSTAT
>     ????? 0.00???? 120.74 us????? 31.43 us???? 847.97 us??????????
>     9094??????? READ
>     ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us?????????
>     22614?????? FLUSH
>     ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us?????????
>     21656????? STATFS
>     ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us??????????
>     6145????? CREATE
>     ????? 0.00???? 217.50 us????? 50.60 us?? 30734.99 us?????????
>     38051?????? WRITE
>     ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us?????????
>     81377???? OPENDIR
>     ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us??????????
>     3177?????? RMDIR
>     ????? 0.03???? 100.06 us????? 30.93 us?? 49756.93 us????????
>     432439??? READLINK
>     ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us????????
>     448381????? UNLINK
>     ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us?????????
>     52185??? READDIRP
>     ????? 0.35???? 414.01 us????? 36.58 us 1038229.93 us???????
>     1444606??????? OPEN
>     ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us?????????
>     10745??? FXATTROP
>     ????? 0.72???? 776.24 us????? 11.24 us 4271792.57 us???????
>     1605822??? GETXATTR
>     ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us?????????
>     13532???? READDIR
>     ????? 0.79???? 320.45 us?????? 7.71 us 2369511.06 us???????
>     4257297???? ENTRYLK
>     ????? 1.97???? 464.07 us?????? 9.38 us 1038460.80 us???????
>     7333722???? INODELK
>     ???? 13.72??? 2372.42 us?????? 8.33 us 6258159.87 us???????
>     9979799????? LOOKUP
>     ???? 80.79?? 42373.50 us????? 61.49 us 2238308.13 us???????
>     3289103???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2250268????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     3756???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     167??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1908?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2246352?? CI_FORGET
>     ?
>     ??? Duration: 209334 seconds
>     ?? Data Read: 1059777395 bytes
>     Data Written: 2336921014 bytes
>     ?
>     Brick: flexo:/mnt/gluster/c
>     --------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+
>     ?No. of Reads:??????????????????? 0
>     No. of Writes:??????????????? 32843
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     686562????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     834596???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     93757? RELEASEDIR
>     ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us???????????
>     107???????? IPC
>     ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 us????????????
>     36??????? LINK
>     ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 us????????????
>     65?? FGETXATTR
>     ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 us????????????
>     88????? RENAME
>     ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us???????????
>     153??? TRUNCATE
>     ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us???????????
>     491????????? LK
>     ????? 0.01????? 51.73 us?????? 7.70 us??? 1174.64 us??????????
>     9339?????? FLUSH
>     ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us??????????
>     9011??? FINODELK
>     ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us?????????
>     10824?????? FSTAT
>     ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us??????????
>     2594?????? MKDIR
>     ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us??????????
>     9009??? FXATTROP
>     ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us??????????
>     5754????? CREATE
>     ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us?????????
>     32843?????? WRITE
>     ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us??????????
>     5798???? SYMLINK
>     ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us?????????
>     34904???? READDIR
>     ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us??????????
>     2732?????? RMDIR
>     ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us?????????
>     93757???? OPENDIR
>     ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us????????
>     170292????? UNLINK
>     ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us?????????
>     13840??? READDIRP
>     ????? 1.94???? 200.88 us????? 25.49 us? 461654.32 us????????
>     468842?? FTRUNCATE
>     ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us????????
>     486882 REMOVEXATTR
>     ????? 2.27???? 223.01 us????? 52.03 us? 178897.40 us????????
>     493240??? SETXATTR
>     ????? 3.53???? 155.31 us?????? 7.96 us?? 67873.00 us???????
>     1101338??? GETXATTR
>     ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us????????
>     829002??????? OPEN
>     ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us???????
>     3660425???? ENTRYLK
>     ????? 6.23???? 304.93 us????? 42.51 us? 661394.43 us????????
>     990644???? SETATTR
>     ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us????????
>     990766???? XATTROP
>     ???? 12.39???? 549.11 us???? 135.57 us 1659697.83 us???????
>     1093387?????? MKNOD
>     ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us???????
>     4325281???? INODELK
>     ???? 39.53???? 206.94 us?????? 8.56 us? 964823.33 us???????
>     9259429????? LOOKUP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1856377????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     933300???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     251??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     293?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     923077?? CI_FORGET
>     ?
>     ??? Duration: 209334 seconds
>     ?? Data Read: 0 bytes
>     Data Written: 32843 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+
>     ?No. of Reads:??????????????????? 0
>     No. of Writes:??????????????? 32843
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     686562????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     834596???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     93757? RELEASEDIR
>     ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us???????????
>     107???????? IPC
>     ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 us????????????
>     36??????? LINK
>     ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 us????????????
>     65?? FGETXATTR
>     ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 us????????????
>     88????? RENAME
>     ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us???????????
>     153??? TRUNCATE
>     ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us???????????
>     491????????? LK
>     ????? 0.01????? 51.71 us?????? 7.70 us??? 1174.64 us??????????
>     9339?????? FLUSH
>     ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us??????????
>     9011??? FINODELK
>     ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us?????????
>     10824?????? FSTAT
>     ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us??????????
>     2594?????? MKDIR
>     ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us??????????
>     9009??? FXATTROP
>     ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us??????????
>     5754????? CREATE
>     ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us?????????
>     32843?????? WRITE
>     ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us??????????
>     5798???? SYMLINK
>     ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us?????????
>     34904???? READDIR
>     ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us??????????
>     2732?????? RMDIR
>     ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us?????????
>     93757???? OPENDIR
>     ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us????????
>     170292????? UNLINK
>     ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us?????????
>     13840??? READDIRP
>     ????? 1.94???? 200.87 us????? 25.49 us? 461654.32 us????????
>     468842?? FTRUNCATE
>     ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us????????
>     486882 REMOVEXATTR
>     ????? 2.27???? 223.00 us????? 52.03 us? 178897.40 us????????
>     493240??? SETXATTR
>     ????? 3.53???? 155.32 us?????? 7.96 us?? 67873.00 us???????
>     1101338??? GETXATTR
>     ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us????????
>     829002??????? OPEN
>     ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us???????
>     3660425???? ENTRYLK
>     ????? 6.23???? 304.88 us????? 42.51 us? 661394.43 us????????
>     990644???? SETATTR
>     ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us????????
>     990766???? XATTROP
>     ???? 12.39???? 549.25 us???? 135.57 us 1659697.83 us???????
>     1093387?????? MKNOD
>     ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us???????
>     4325281???? INODELK
>     ???? 39.53???? 206.95 us?????? 8.56 us? 964823.33 us???????
>     9259429????? LOOKUP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1856377????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     933300???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     251??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     293?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     923077?? CI_FORGET
>     ?
>     ??? Duration: 209334 seconds
>     ?? Data Read: 0 bytes
>     Data Written: 32843 bytes
>     ?
>     Brick: bender:/mnt/gluster/c
>     ---------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+
>     ?No. of Reads:??????????????????? 0
>     No. of Writes:??????????????? 37050
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     928119????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1401524???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     70092? RELEASEDIR
>     ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us???????????
>     100???????? IPC
>     ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 us????????????
>     38??????? LINK
>     ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 us????????????
>     30???? SYMLINK
>     ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 us?????????????
>     4?????? FSYNC
>     ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us???????????
>     122????? RENAME
>     ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us???????????
>     777????????? LK
>     ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us???????????
>     810?? FGETXATTR
>     ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us???????????
>     603??? TRUNCATE
>     ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us???????????
>     302?????? MKDIR
>     ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us?????????
>     10060??? FINODELK
>     ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us??????????
>     6409?????? FSTAT
>     ????? 0.01????? 49.38 us?????? 8.08 us??? 1264.41 us?????????
>     21719?????? FLUSH
>     ????? 0.03????? 74.76 us????? 20.86 us??? 2008.41 us?????????
>     37050?????? WRITE
>     ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us??????????
>     5901????? CREATE
>     ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us?????????
>     10057??? FXATTROP
>     ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us?????????
>     70092???? OPENDIR
>     ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us?????????
>     10779???? READDIR
>     ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us??????????
>     3008?????? RMDIR
>     ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us????????
>     109478????? UNLINK
>     ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us????????
>     722669 REMOVEXATTR
>     ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us????????
>     789294??? GETXATTR
>     ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us????????
>     718835?? FTRUNCATE
>     ????? 1.89???? 219.47 us????? 49.92 us? 506767.78 us????????
>     722856??? SETXATTR
>     ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us???????
>     1395957??????? OPEN
>     ????? 5.56???? 322.08 us????? 43.37 us? 634069.61 us???????
>     1450160???? SETATTR
>     ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us???????
>     4233520???? ENTRYLK
>     ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us???????
>     1449601???? XATTROP
>     ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us???????
>     7097226???? INODELK
>     ???? 19.37??? 1104.98 us???? 101.28 us 1612346.75 us???????
>     1472813?????? MKNOD
>     ???? 41.85???? 295.61 us?????? 8.98 us? 996805.54 us??????
>     11896091????? LOOKUP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3531139????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1862470???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     244??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     632?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1668669?? CI_FORGET
>     ?
>     ??? Duration: 208643 seconds
>     ?? Data Read: 0 bytes
>     Data Written: 37050 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+
>     ?No. of Reads:??????????????????? 0
>     No. of Writes:??????????????? 37050
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     928119????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1401524???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     70092? RELEASEDIR
>     ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us???????????
>     100???????? IPC
>     ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 us????????????
>     38??????? LINK
>     ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 us????????????
>     30???? SYMLINK
>     ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 us?????????????
>     4?????? FSYNC
>     ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us???????????
>     122????? RENAME
>     ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us???????????
>     777????????? LK
>     ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us???????????
>     810?? FGETXATTR
>     ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us???????????
>     603??? TRUNCATE
>     ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us???????????
>     302?????? MKDIR
>     ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us?????????
>     10060??? FINODELK
>     ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us??????????
>     6409?????? FSTAT
>     ????? 0.01????? 49.39 us?????? 8.08 us??? 1264.41 us?????????
>     21719?????? FLUSH
>     ????? 0.03????? 74.77 us????? 20.86 us??? 2008.41 us?????????
>     37050?????? WRITE
>     ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us??????????
>     5901????? CREATE
>     ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us?????????
>     10057??? FXATTROP
>     ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us?????????
>     70092???? OPENDIR
>     ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us?????????
>     10779???? READDIR
>     ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us??????????
>     3008?????? RMDIR
>     ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us????????
>     109478????? UNLINK
>     ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us????????
>     722669 REMOVEXATTR
>     ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us????????
>     789294??? GETXATTR
>     ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us????????
>     718835?? FTRUNCATE
>     ????? 1.89???? 219.48 us????? 49.92 us? 506767.78 us????????
>     722856??? SETXATTR
>     ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us???????
>     1395957??????? OPEN
>     ????? 5.56???? 322.09 us????? 43.37 us? 634069.61 us???????
>     1450160???? SETATTR
>     ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us???????
>     4233520???? ENTRYLK
>     ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us???????
>     1449601???? XATTROP
>     ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us???????
>     7097226???? INODELK
>     ???? 19.37??? 1105.06 us???? 101.28 us 1612346.75 us???????
>     1472813?????? MKNOD
>     ???? 41.84???? 295.54 us?????? 8.98 us? 996805.54 us??????
>     11896091????? LOOKUP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3531139????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1862470???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     244??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     632?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1668669?? CI_FORGET
>     ?
>     ??? Duration: 208643 seconds
>     ?? Data Read: 0 bytes
>     Data Written: 37050 bytes
>     ?
>     Brick: bender:/mnt/gluster/a
>     ---------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     2b+?????????????????? 8b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:?????????????????? 53????????????????????
>     1??????????????????? 60
>     ?
>     ?? Block Size:???????????????? 16b+?????????????????
>     32b+????????????????? 64b+
>     ?No. of Reads:??????????????????? 0??????????????????
>     235???????????????????? 4
>     No. of Writes:?????????????????? 14?????????????????
>     1033??????????????????? 15
>     ?
>     ?? Block Size:??????????????? 128b+????????????????
>     256b+???????????????? 512b+
>     ?No. of Reads:??????????????????? 4??????????????????
>     115??????????????????? 81
>     No. of Writes:?????????????????? 58??????????????????
>     703??????????????????? 80
>     ?
>     ?? Block Size:?????????????? 1024b+???????????????
>     2048b+??????????????? 4096b+
>     ?No. of Reads:????????????????? 181???????????????????
>     26??????????????????? 74
>     No. of Writes:????????????????? 163??????????????????
>     278?????????????????? 397
>     ?
>     ?? Block Size:?????????????? 8192b+??????????????
>     16384b+?????????????? 32768b+
>     ?No. of Reads:????????????????? 131??????????????????
>     161?????????????????? 267
>     No. of Writes:???????????????? 7264?????????????????
>     1398????????????????? 6742
>     ?
>     ?? Block Size:????????????? 65536b+?????????????
>     131072b+????????????? 262144b+
>     ?No. of Reads:????????????????? 443????????????????
>     25528???????????????????? 0
>     No. of Writes:???????????????? 2426????????????????
>     12240???????????????????? 1
>     ?
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     684942????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     837092???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     87297? RELEASEDIR
>     ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 us?????????????
>     8 REMOVEXATTR
>     ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 us????????????
>     97???????? IPC
>     ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 us????????????
>     46?? FTRUNCATE
>     ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 us????????????
>     65?? FGETXATTR
>     ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us???????????
>     121??????? LINK
>     ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 us????????????
>     25???? SYMLINK
>     ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us???????????
>     107?????? MKNOD
>     ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 us?????????????
>     1?????? FSYNC
>     ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us???????????
>     188????? RENAME
>     ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us???????????
>     745??????? STAT
>     ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us???????????
>     200??? TRUNCATE
>     ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us??????????
>     1444???? SETATTR
>     ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us???????????
>     177?????? MKDIR
>     ????? 0.00????? 52.68 us?????? 7.28 us???? 826.01 us??????????
>     9890?????? FLUSH
>     ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us??????????
>     9792??? FINODELK
>     ????? 0.00????? 74.26 us????? 20.00 us??? 1522.39 us?????????
>     21521????? STATFS
>     ????? 0.00???? 129.21 us????? 34.15 us??? 9068.67 us?????????
>     13094?????? FSTAT
>     ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us??????????
>     5967????? CREATE
>     ????? 0.00???? 112.06 us????? 28.14 us??? 9545.53 us?????????
>     27474??????? READ
>     ????? 0.01???? 313.76 us????? 52.98 us?? 71635.74 us?????????
>     32926?????? WRITE
>     ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us?????????
>     87297???? OPENDIR
>     ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us??????????
>     3177?????? RMDIR
>     ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us?????????
>     85073??? SETXATTR
>     ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us?????????
>     15090??? READDIRP
>     ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us???????????
>     564????????? LK
>     ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us????????
>     850938??? READLINK
>     ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us????????
>     831212??????? OPEN
>     ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us?????????
>     22482???? READDIR
>     ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us????????
>     415226????? UNLINK
>     ????? 0.54??? 1272.83 us?????? 9.66 us 1604368.93 us????????
>     686945??? GETXATTR
>     ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us???????
>     3739718???? ENTRYLK
>     ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us??????????
>     9311??? FXATTROP
>     ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us???????
>     4649291???? INODELK
>     ????? 5.82??? 1240.20 us?????? 8.78 us 1526868.50 us???????
>     7625383????? LOOKUP
>     ???? 88.60?? 58005.98 us????? 67.31 us 1448810.06 us???????
>     2481914???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1396875????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1854???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     28??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1051?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1394996?? CI_FORGET
>     ?
>     ??? Duration: 208643 seconds
>     ?? Data Read: 3409342848 bytes
>     Data Written: 2180686507 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     2b+?????????????????? 8b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:?????????????????? 53????????????????????
>     1??????????????????? 60
>     ?
>     ?? Block Size:???????????????? 16b+?????????????????
>     32b+????????????????? 64b+
>     ?No. of Reads:??????????????????? 0??????????????????
>     235???????????????????? 4
>     No. of Writes:?????????????????? 14?????????????????
>     1033??????????????????? 15
>     ?
>     ?? Block Size:??????????????? 128b+????????????????
>     256b+???????????????? 512b+
>     ?No. of Reads:??????????????????? 4??????????????????
>     115??????????????????? 81
>     No. of Writes:?????????????????? 58??????????????????
>     703??????????????????? 80
>     ?
>     ?? Block Size:?????????????? 1024b+???????????????
>     2048b+??????????????? 4096b+
>     ?No. of Reads:????????????????? 181???????????????????
>     26??????????????????? 74
>     No. of Writes:????????????????? 163??????????????????
>     278?????????????????? 397
>     ?
>     ?? Block Size:?????????????? 8192b+??????????????
>     16384b+?????????????? 32768b+
>     ?No. of Reads:????????????????? 131??????????????????
>     161?????????????????? 267
>     No. of Writes:???????????????? 7264?????????????????
>     1398????????????????? 6742
>     ?
>     ?? Block Size:????????????? 65536b+?????????????
>     131072b+????????????? 262144b+
>     ?No. of Reads:????????????????? 443????????????????
>     25528???????????????????? 0
>     No. of Writes:???????????????? 2426????????????????
>     12240???????????????????? 1
>     ?
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     684942????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     837092???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     87297? RELEASEDIR
>     ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 us?????????????
>     8 REMOVEXATTR
>     ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 us????????????
>     97???????? IPC
>     ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 us????????????
>     46?? FTRUNCATE
>     ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 us????????????
>     65?? FGETXATTR
>     ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us???????????
>     121??????? LINK
>     ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 us????????????
>     25???? SYMLINK
>     ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us???????????
>     107?????? MKNOD
>     ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 us?????????????
>     1?????? FSYNC
>     ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us???????????
>     188????? RENAME
>     ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us???????????
>     745??????? STAT
>     ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us???????????
>     200??? TRUNCATE
>     ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us??????????
>     1444???? SETATTR
>     ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us???????????
>     177?????? MKDIR
>     ????? 0.00????? 52.69 us?????? 7.28 us???? 826.01 us??????????
>     9890?????? FLUSH
>     ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us??????????
>     9792??? FINODELK
>     ????? 0.00????? 74.25 us????? 20.00 us??? 1522.39 us?????????
>     21521????? STATFS
>     ????? 0.00???? 129.23 us????? 34.15 us??? 9068.67 us?????????
>     13094?????? FSTAT
>     ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us??????????
>     5967????? CREATE
>     ????? 0.00???? 112.45 us????? 28.14 us??? 9545.53 us?????????
>     27474??????? READ
>     ????? 0.01???? 313.77 us????? 52.98 us?? 71635.74 us?????????
>     32926?????? WRITE
>     ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us?????????
>     87297???? OPENDIR
>     ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us??????????
>     3177?????? RMDIR
>     ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us?????????
>     85073??? SETXATTR
>     ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us?????????
>     15090??? READDIRP
>     ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us???????????
>     564????????? LK
>     ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us????????
>     850938??? READLINK
>     ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us????????
>     831212??????? OPEN
>     ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us?????????
>     22482???? READDIR
>     ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us????????
>     415226????? UNLINK
>     ????? 0.54??? 1272.79 us?????? 9.66 us 1604368.93 us????????
>     686945??? GETXATTR
>     ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us???????
>     3739718???? ENTRYLK
>     ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us??????????
>     9311??? FXATTROP
>     ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us???????
>     4649291???? INODELK
>     ????? 5.82??? 1240.10 us?????? 8.78 us 1526868.50 us???????
>     7625383????? LOOKUP
>     ???? 88.60?? 58006.02 us????? 67.31 us 1448810.06 us???????
>     2481914???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1396875????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1854???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     28??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1051?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1394996?? CI_FORGET
>     ?
>     ??? Duration: 208643 seconds
>     ?? Data Read: 3409342848 bytes
>     Data Written: 2180686507 bytes
>     ?
>
>     ________
>
>
>
>     Community Meeting Calendar:
>
>     Schedule -
>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>     Bridge: https://bluejeans.com/441850968
>
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200505/afca4a51/attachment-0001.html>

Hi,

So upgrading to 7.5 made a difference in that it takes a bit longer to
cause chaos, but it still does the exact same thing.

To be precise:

I've now had to set:

gluster volume set vpbx_shared cluster.heal-wait-queue-length 1
gluster volume set vpbx_shared cluster.shd-wait-qlength 1
gluster volume set vpbx_shared cluster.shd-max-threads 1

With any of these settings >1 I end up in a situation where simple
directory listings of < 100 entries in a folder takes upwards of a
minute to complete *from time to time*.

It seems that when the logs

With these the maximum stall I can reproduce is usually around 3 seconds
before the directory listing will complete, but about 5s worst case.?
I've now reverted these as well, which are known for us to cause issues
under heal required conditions:

cluster.metadata-self-heal?????????????
on?????????????????????????????????????
cluster.data-self-heal?????????????????
on?????????????????????????????????????
cluster.entry-self-heal????????????????
on?????????????????????????????????????

Not sure if I need to remount for these to become effective but I've
done so just to be on the safe side.

I'm still seeing these errors:

[2020-05-07 19:20:12.763311] W [MSGID: 114031]
[client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
0-vpbx_shared-client-1: remote operation failed [Invalid argument]
[2020-05-07 19:20:12.995675] W [MSGID: 114031]
[client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
0-vpbx_shared-client-1: remote operation failed [Invalid argument]
[2020-05-07 19:20:59.199776] W [MSGID: 114031]
[client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
0-vpbx_shared-client-1: remote operation failed [Invalid argument]

And the start of my issues still seem to co-incide with that.? Sample case:

# time ls
...
real??? 0m5.140s
user??? 0m0.009s
sys??? 0m0.000s
# f=(*); echo "${#f[@]}"
165

This just does not seem right.? Note, again, if I set *any* of the above
parameters >1 these delays get far, far worse (I've had over a minute
for this exact same ls).

Any advise would be greatly appreciated.? To me, it looks like some lock
is being taken somewhere, and takes a while to release.? I don't
understand the glusterfs internals nearly well enough to understand
exactly what's going on here, nor do I know what information to provide
to help drill down on this.? Happy to go look at code, if I can get
pointed in a direction.

Kind Regards,
Jaco

On 2020/05/05 08:43, Jaco Kroon wrote:
>
> Hi Artem,
>
> I'll do that.? In the meantime consider this (inferred from the
> changelogs).
>
> It seems like glusterfs SHD is queueing into a bounded queue under
> lock conditions, and waiting for space to open up.? At least, this is
> what's hinted in the changelogs, I've been trying to grasp the SHD
> code for two days now and am not getting to wrap my head around that.?
> If this is true, then the only way to avoid lock-ups are to ensure
> that the queue is larger than the total number of files that can ever
> require healing (effectively making the queue unbounded).
>
> Kind Regards,
> Jaco
>
> On 2020/05/04 21:56, Artem Russakovskii wrote:
>> I reported something similar recently to this list, and I highly
>> advise you to update to 7.5 (in my case it was 5.11 -> 5.13), then
>> see if it keeps happening. The self heal issues should be fixed there
>> properly at least.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>,?APK
Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net <http://beerpla.net/> | @ArtemR
<http://twitter.com/ArtemR>
>>
>>
>> On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za
>> <mailto:jaco at uls.co.za>> wrote:
>>
>>     Hi All,
>>
>>     So we've got a cluster currently running on glusterfs 7.3
>>
>>     Heal performance is absolutely terrible (it starts of great, runs
>>     for a while then completely trips up).
>>
>>     Once the SHD trips up, access speeds to the cluster also becomes
>>     poor.? And with poor I mean directory listings that normally
>>     takes well under 500ms takes upwards of 10s, but I've seen
>>     upwards of 40s (although a follow up ls immediately after is nice
>>     and fast again, temporarily).? With all SHDs killed this
>>     particular ls is as per normal (around 30-50ms).
>>
>>     The SHD logs doesn't provide any usable clues, the only
variance
>>     in logs is this, just before all daemons across the cluster
>>     crawls to a near halt.
>>
>>     [2020-05-02 19:55:58.627036] I [MSGID: 108026]
>>     [afr-self-heal-common.c:1747:afr_log_selfheal]
>>     0-vpbx_shared-replicate-0: Completed entry selfheal on
>>     5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1 ?
>>     [2020-05-02 19:55:58.627058] I [MSGID: 108026]
>>     [afr-self-heal-common.c:1747:afr_log_selfheal]
>>     0-vpbx_shared-replicate-0: Completed entry selfheal on
>>     c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1 ?
>>     ?
>>     [2020-05-02 19:55:59.125001] W [MSGID: 114031]
>>     [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
>>     0-vpbx_shared-client-1: remote operation failed [Invalid argument]
>>     ?
>>     [2020-05-02 19:55:59.339803] I [MSGID: 108026]
>>     [afr-self-heal-common.c:1747:afr_log_selfheal]
>>     0-vpbx_shared-replicate-0: Completed entry selfheal on
>>     2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1?
>>     [2020-05-02 19:55:59.490387] I [MSGID: 108026]
>>     [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
>>     0-vpbx_shared-replicate-0: performing metadata selfheal on
>>     527a63a3-9acd-4c3f-b95c-5c719325c973
>>
>>     A few more entries will complete overall and then it'll be a
file
>>     every other second.? Once I saw this I ran a few more tests, and
>>     on every test just prior to slowing to a crawl, that entry is
there.
>>
>>     There is a perceived improvement when lowering the thread count,
>>     but not significant enough to confirm a correlation.? Even
>>     with**cluster.shd-max-threads=1 this still happens (gets a few
>>     tens thousand files completed instead of few hundred, at 32 it
>>     gets to maybe 400 to 500 before it stalls, 16 does maybe a
>>     thousand).*
>>     *
>>
>>     There are probably performant bursts hereafter, because from
>>     experience we know this process takes about two months.? But this
>>     time around it seems to be much, much worse.? Might be because
>>     that was healing a single brick to another brick, so less locking
>>     going?? At the moment the replicas are basically trying to
>>     populate the newly added arbiters.? Also, we haven't messed as
>>     greatly with the health check settings (queues and threads and
>>     locks nearly as much as since the arbiter addition because
>>     performance has simply dropped to non-acceptable levels.
>>
>>     Unfortunately granular entry heal isn't on at the moment (nor
can
>>     I switch it on due to a few million outstanding heals).? This
>>     used to be a 2x2 cluster.? We then moved one of the bricks to a
>>     newly added server using replace-brick.? What a frightening
>>     process this is.? We then proceeded to add the arbiters.
>>
>>     Some of the settings below was absolute shots in the dark, based
>>     on the little information we've been able to track down so far.
>>
>>     Kind Regards,
>>     Jaco
>>
>>     The current gluster volume info (this used to be 2x2
>>     distributed-replicate, aiming to get it to 3x (2 + 1) ... at
>>     current rates we might get there by 2021):
>>
>>     Volume Name: glvol
>>     Type: Distributed-Replicate
>>     Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb
>>     Status: Started
>>     Snapshot Count: 0
>>     Number of Bricks: 2 x (2 + 1) = 6
>>     Transport-type: tcp
>>     Bricks:
>>     Brick1: zoidberg:/mnt/gluster/a
>>     Brick2: flexo:/mnt/gluster/b
>>     Brick3: bender:/mnt/gluster/c (arbiter)
>>     Brick4: bender:/mnt/gluster/a
>>     Brick5: zoidberg:/mnt/gluster/b
>>     Brick6: flexo:/mnt/gluster/c (arbiter)
>>     Options Reconfigured:
>>     *cluster.heal-timeout: 5 <-- hoping this would at least shorten
>>     the blockages.*
>>     cluster.locking-scheme: granular
>>     cluster.heal-wait-queue-length: 10240
>>     *diagnostics.count-fop-hits: on**
>>     **diagnostics.latency-measurement: on <-- was hoping profiling
>>     might provide some insight.? Going to switch this off now.*
>>     cluster.background-self-heal-count: 128
>>     *cluster.shd-wait-qlength: 131072**
>>     **cluster.shd-max-threads: 32? <-- even with this disk IO never
>>     goes over 10% utilization.? CPU goes to about 2 cores.? So it's
>>     lock contention that's slowing it down we reckon.*
>>     *cluster.metadata-self-heal: off**
>>     **cluster.entry-self-heal: off**
>>     **cluster.data-self-heal: off**<-- these kill performance.
>>     *transport.address-family: inet
>>     nfs.disable: on
>>     performance.client-io-threads: on
>>     features.cache-invalidation: on
>>     features.cache-invalidation-timeout: 600
>>     performance.stat-prefetch: on
>>     performance.cache-invalidation: on
>>     performance.md-cache-timeout: 600
>>     *cluster.self-heal-daemon: yes <-- disabling/enabling this
>>     doesn't help, have to kill the shd and start ... force to get
>>     things moving along again.*
>>     performance.io-thread-count: 16
>>     performance.high-prio-threads: 16
>>     performance.normal-prio-threads: 16
>>     performance.low-prio-threads: 16
>>     performance.least-prio-threads: 4
>>     cluster.self-heal-window-size: 512
>>     cluster.self-heal-readdir-size: 131072
>>     client.event-threads: 8
>>     server.event-threads: 16
>>     network.ping-timeout: 2
>>     *cluster.data-self-heal-algorithm: full <-- most files are small
>>     (<1MB), on dual 1gbit links (so up to 2gbit, but b/w never goes
>>     over 5mbit anyway).*
>>
>>     performance info:
>>
>>     Brick: zoidberg:/mnt/gluster/a
>>     -----------------------------------------
>>     Cumulative Stats:
>>     ?? Block Size:????????????????? 1b+??????????????????
>>     8b+????????????????? 16b+
>>     ?No. of Reads:??????????????????? 0????????????????????
>>     0???????????????????? 0
>>     No. of Writes:????????????????? 117??????????????????
>>     195?????????????????? 367
>>     ?
>>     ?? Block Size:???????????????? 32b+?????????????????
>>     64b+???????????????? 128b+
>>     ?No. of Reads:????????????????? 306????????????????????
>>     0???????????????????? 6
>>     No. of Writes:???????????????? 1016????????????????????
>>     9?????????????????? 134
>>     ?
>>     ?? Block Size:??????????????? 256b+????????????????
>>     512b+??????????????? 1024b+
>>     ?No. of Reads:????????????????? 107???????????????????
>>     58?????????????????? 485
>>     No. of Writes:????????????????? 731???????????????????
>>     73?????????????????? 168
>>     ?
>>     ?? Block Size:?????????????? 2048b+???????????????
>>     4096b+??????????????? 8192b+
>>     ?No. of Reads:?????????????????? 43???????????????????
>>     59?????????????????? 383
>>     No. of Writes:????????????????? 295??????????????????
>>     475???????????????? 10562
>>     ?
>>     ?? Block Size:????????????? 16384b+??????????????
>>     32768b+?????????????? 65536b+
>>     ?No. of Reads:????????????????? 347??????????????????
>>     380?????????????????? 634
>>     No. of Writes:???????????????? 1487?????????????????
>>     6369????????????????? 3970
>>     ?
>>     ?? Block Size:???????????? 131072b+
>>     ?No. of Reads:??????????????? 35704
>>     No. of Writes:??????????????? 12398
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     957293????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1455091???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     69951? RELEASEDIR
>>     ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us???????????
>>     113???????? IPC
>>     ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 us????????????
>>     30 REMOVEXATTR
>>     ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 us????????????
>>     52?? FTRUNCATE
>>     ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us??????????
>>     1385??????? STAT
>>     ????? 0.00???? 364.93 us????? 63.45 us?? 15957.66 us???????????
>>     810?? FGETXATTR
>>     ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us???????????
>>     137??????? LINK
>>     ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 us????????????
>>     12?????? FSYNC
>>     ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us???????????
>>     698??? TRUNCATE
>>     ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us???????????
>>     188????? RENAME
>>     ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us??????????
>>     1680???? SETATTR
>>     ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us???????????
>>     127?????? MKNOD
>>     ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us?????????
>>     11412??? FINODELK
>>     ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us???????????
>>     221??? SETXATTR
>>     ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 us????????????
>>     30???? SYMLINK
>>     ????? 0.00????? 56.18 us?????? 7.31 us?? 41484.37 us?????????
>>     22640?????? FLUSH
>>     ????? 0.00???? 105.64 us????? 19.34 us? 351055.72 us?????????
>>     21693????? STATFS
>>     ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us???????????
>>     178?????? MKDIR
>>     ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us??????????
>>     9653?????? FSTAT
>>     ????? 0.00???? 214.87 us????? 25.71 us 1712020.99 us?????????
>>     39054??????? READ
>>     ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us?????????
>>     69951???? OPENDIR
>>     ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us??????????
>>     3180?????? RMDIR
>>     ????? 0.01???? 327.08 us????? 55.69 us? 420376.80 us?????????
>>     38366?????? WRITE
>>     ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us??????????
>>     6169????? CREATE
>>     ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us???????
>>     1159936??? READLINK
>>     ????? 0.40???? 543.50 us????? 36.45 us 2430653.54 us???????
>>     1449307??????? OPEN
>>     ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us?????????
>>     10787??? FXATTROP
>>     ????? 0.53??? 2342.72 us????? 20.91 us 4157542.43 us????????
>>     447821????? UNLINK
>>     ????? 0.53? 558734.79 us????? 24.40 us 1399487.63 us??????????
>>     1889????????? LK
>>     ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us?????????
>>     10687???? READDIR
>>     ????? 0.99??? 2426.23 us????? 12.47 us 24176123.13 us????????
>>     809430??? GETXATTR
>>     ????? 1.45???? 676.83 us?????? 9.30 us 193177067.25 us???????
>>     4260021???? ENTRYLK
>>     ????? 1.76?? 38325.38 us????? 21.44 us 46693575.93 us?????????
>>     91439??? READDIRP
>>     ????? 4.44??? 1200.31 us?????? 9.10 us 2559377.90 us???????
>>     7352811???? INODELK
>>     ????? 9.30??? 1849.24 us?????? 9.28 us 35299050.00 us???????
>>     9989221????? LOOKUP
>>     ???? 79.24?? 47791.46 us????? 76.77 us 10488824.26 us???????
>>     3294642???? XATTROP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     3194806????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     5311???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>>     54??? CI_XATTR
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     1907?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     3189457?? CI_FORGET
>>     ?
>>     ??? Duration: 209524 seconds
>>     ?? Data Read: 4775365531 bytes
>>     Data Written: 2354108664 bytes
>>     ?
>>     Interval 0 Stats:
>>     ?? Block Size:????????????????? 1b+??????????????????
>>     8b+????????????????? 16b+
>>     ?No. of Reads:??????????????????? 0????????????????????
>>     0???????????????????? 0
>>     No. of Writes:????????????????? 117??????????????????
>>     195?????????????????? 367
>>     ?
>>     ?? Block Size:???????????????? 32b+?????????????????
>>     64b+???????????????? 128b+
>>     ?No. of Reads:????????????????? 306????????????????????
>>     0???????????????????? 6
>>     No. of Writes:???????????????? 1016????????????????????
>>     9?????????????????? 134
>>     ?
>>     ?? Block Size:??????????????? 256b+????????????????
>>     512b+??????????????? 1024b+
>>     ?No. of Reads:????????????????? 107???????????????????
>>     58?????????????????? 485
>>     No. of Writes:????????????????? 731???????????????????
>>     73?????????????????? 168
>>     ?
>>     ?? Block Size:?????????????? 2048b+???????????????
>>     4096b+??????????????? 8192b+
>>     ?No. of Reads:?????????????????? 43???????????????????
>>     59?????????????????? 383
>>     No. of Writes:????????????????? 295??????????????????
>>     475???????????????? 10562
>>     ?
>>     ?? Block Size:????????????? 16384b+??????????????
>>     32768b+?????????????? 65536b+
>>     ?No. of Reads:????????????????? 347??????????????????
>>     380?????????????????? 634
>>     No. of Writes:???????????????? 1487?????????????????
>>     6369????????????????? 3970
>>     ?
>>     ?? Block Size:???????????? 131072b+
>>     ?No. of Reads:??????????????? 35704
>>     No. of Writes:??????????????? 12398
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     957293????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1455091???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     69951? RELEASEDIR
>>     ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us???????????
>>     113???????? IPC
>>     ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 us????????????
>>     30 REMOVEXATTR
>>     ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 us????????????
>>     52?? FTRUNCATE
>>     ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us??????????
>>     1385??????? STAT
>>     ????? 0.00???? 363.20 us????? 63.45 us?? 15957.66 us???????????
>>     810?? FGETXATTR
>>     ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us???????????
>>     137??????? LINK
>>     ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 us????????????
>>     12?????? FSYNC
>>     ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us???????????
>>     698??? TRUNCATE
>>     ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us???????????
>>     188????? RENAME
>>     ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us??????????
>>     1680???? SETATTR
>>     ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us???????????
>>     127?????? MKNOD
>>     ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us?????????
>>     11412??? FINODELK
>>     ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us???????????
>>     221??? SETXATTR
>>     ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 us????????????
>>     30???? SYMLINK
>>     ????? 0.00????? 56.19 us?????? 7.31 us?? 41484.37 us?????????
>>     22640?????? FLUSH
>>     ????? 0.00???? 105.61 us????? 19.34 us? 351055.72 us?????????
>>     21693????? STATFS
>>     ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us???????????
>>     178?????? MKDIR
>>     ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us??????????
>>     9653?????? FSTAT
>>     ????? 0.00???? 214.91 us????? 25.71 us 1712020.99 us?????????
>>     39054??????? READ
>>     ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us?????????
>>     69951???? OPENDIR
>>     ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us??????????
>>     3180?????? RMDIR
>>     ????? 0.01???? 327.13 us????? 55.69 us? 420376.80 us?????????
>>     38366?????? WRITE
>>     ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us??????????
>>     6169????? CREATE
>>     ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us???????
>>     1159936??? READLINK
>>     ????? 0.40???? 543.49 us????? 36.45 us 2430653.54 us???????
>>     1449307??????? OPEN
>>     ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us?????????
>>     10787??? FXATTROP
>>     ????? 0.53??? 2342.57 us????? 20.91 us 4157542.43 us????????
>>     447821????? UNLINK
>>     ????? 0.53? 558416.10 us????? 24.40 us 1399487.63 us??????????
>>     1889????????? LK
>>     ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us?????????
>>     10687???? READDIR
>>     ????? 0.99??? 2426.24 us????? 12.47 us 24176123.13 us????????
>>     809430??? GETXATTR
>>     ????? 1.45???? 676.75 us?????? 9.30 us 193177067.25 us???????
>>     4260021???? ENTRYLK
>>     ????? 1.76?? 38324.59 us????? 21.44 us 46693575.93 us?????????
>>     91439??? READDIRP
>>     ????? 4.44??? 1200.37 us?????? 9.10 us 2559377.90 us???????
>>     7352811???? INODELK
>>     ????? 9.29??? 1848.94 us?????? 9.28 us 35299050.00 us???????
>>     9989221????? LOOKUP
>>     ???? 79.24?? 47792.42 us????? 76.77 us 10488824.26 us???????
>>     3294642???? XATTROP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     3194806????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     5311???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>>     54??? CI_XATTR
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     1907?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     3189457?? CI_FORGET
>>     ?
>>     ??? Duration: 209524 seconds
>>     ?? Data Read: 4775365531 bytes
>>     Data Written: 2354108664 bytes
>>     ?
>>     Brick: zoidberg:/mnt/gluster/b
>>     -----------------------------------------
>>     Cumulative Stats:
>>     ?? Block Size:????????????????? 1b+??????????????????
>>     2b+?????????????????? 8b+
>>     ?No. of Reads:??????????????????? 0????????????????????
>>     0???????????????????? 0
>>     No. of Writes:?????????????????? 53????????????????????
>>     1??????????????????? 60
>>     ?
>>     ?? Block Size:???????????????? 16b+?????????????????
>>     32b+????????????????? 64b+
>>     ?No. of Reads:??????????????????? 0??????????????????
>>     213???????????????????? 2
>>     No. of Writes:?????????????????? 14?????????????????
>>     1057??????????????????? 15
>>     ?
>>     ?? Block Size:??????????????? 128b+????????????????
>>     256b+???????????????? 512b+
>>     ?No. of Reads:??????????????????? 0???????????????????
>>     41?????????????????? 146
>>     No. of Writes:?????????????????? 59??????????????????
>>     715??????????????????? 80
>>     ?
>>     ?? Block Size:?????????????? 1024b+???????????????
>>     2048b+??????????????? 4096b+
>>     ?No. of Reads:????????????????? 224????????????????????
>>     9??????????????????? 15
>>     No. of Writes:????????????????? 163??????????????????
>>     282?????????????????? 404
>>     ?
>>     ?? Block Size:?????????????? 8192b+??????????????
>>     16384b+?????????????? 32768b+
>>     ?No. of Reads:?????????????????? 52???????????????????
>>     74?????????????????? 210
>>     No. of Writes:???????????????? 7415?????????????????
>>     1430????????????????? 6860
>>     ?
>>     ?? Block Size:????????????? 65536b+?????????????
>>     131072b+????????????? 262144b+
>>     ?No. of Reads:????????????????? 299????????????????
>>     15313???????????????????? 1
>>     No. of Writes:???????????????? 2458????????????????
>>     12378???????????????????? 1
>>     ?
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     686733????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     837434???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     87338? RELEASEDIR
>>     ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us???????????
>>     116???????? IPC
>>     ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 us????????????
>>     57?? FTRUNCATE
>>     ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 us????????????
>>     65?? FGETXATTR
>>     ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 us????????????
>>     23 REMOVEXATTR
>>     ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us???????????
>>     121??????? LINK
>>     ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us???????????
>>     490????????? LK
>>     ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us???????????
>>     540??????? STAT
>>     ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us???????????
>>     200??? TRUNCATE
>>     ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us??????????
>>     1482???? SETATTR
>>     ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 us????????????
>>     12?????? FSYNC
>>     ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us?????????
>>     10047?????? FLUSH
>>     ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us??????????
>>     9977??? FINODELK
>>     ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us???????????
>>     188????? RENAME
>>     ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 us????????????
>>     25???? SYMLINK
>>     ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us???????????
>>     118?????? MKNOD
>>     ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us?????????
>>     12587?????? FSTAT
>>     ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us?????????
>>     21695????? STATFS
>>     ????? 0.00???? 307.86 us????? 28.00 us? 292666.83 us?????????
>>     16841??????? READ
>>     ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us???????????
>>     178?????? MKDIR
>>     ????? 0.00???? 319.24 us????? 51.19 us? 136353.39 us?????????
>>     33445?????? WRITE
>>     ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us?????????
>>     87338???? OPENDIR
>>     ????? 0.01???? 249.96 us????? 58.42 us? 594774.86 us?????????
>>     85089??? SETXATTR
>>     ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us??????????
>>     3180?????? RMDIR
>>     ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us??????????
>>     6067????? CREATE
>>     ????? 0.05????? 89.45 us????? 30.14 us? 687961.60 us???????
>>     1219542??? READLINK
>>     ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us????????
>>     831371??????? OPEN
>>     ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us??????????
>>     9493??? FXATTROP
>>     ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us???????
>>     3764714???? ENTRYLK
>>     ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us?????????
>>     22253???? READDIR
>>     ????? 1.06??? 6913.71 us????? 11.58 us 36968532.72 us????????
>>     364963??? GETXATTR
>>     ????? 1.29?? 62130.12 us????? 36.89 us 8649871.11 us?????????
>>     49472??? READDIRP
>>     ????? 2.89??? 1477.53 us????? 11.10 us 2807637.64 us???????
>>     4650500???? INODELK
>>     ????? 4.12?? 23681.31 us????? 20.98 us 23291414.50 us????????
>>     414352????? UNLINK
>>     ???? 16.09??? 5000.31 us?????? 9.04 us 23866016.88 us???????
>>     7655621????? LOOKUP
>>     ???? 72.04?? 68953.31 us????? 68.40 us 27542499.65 us???????
>>     2486126???? XATTROP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     2433405????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     2453???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>>     69??? CI_XATTR
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     1544?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     2430885?? CI_FORGET
>>     ?
>>     ??? Duration: 209524 seconds
>>     ?? Data Read: 2051388524 bytes
>>     Data Written: 2208261499 bytes
>>     ?
>>     Interval 0 Stats:
>>     ?? Block Size:????????????????? 1b+??????????????????
>>     2b+?????????????????? 8b+
>>     ?No. of Reads:??????????????????? 0????????????????????
>>     0???????????????????? 0
>>     No. of Writes:?????????????????? 53????????????????????
>>     1??????????????????? 60
>>     ?
>>     ?? Block Size:???????????????? 16b+?????????????????
>>     32b+????????????????? 64b+
>>     ?No. of Reads:??????????????????? 0??????????????????
>>     213???????????????????? 2
>>     No. of Writes:?????????????????? 14?????????????????
>>     1057??????????????????? 15
>>     ?
>>     ?? Block Size:??????????????? 128b+????????????????
>>     256b+???????????????? 512b+
>>     ?No. of Reads:??????????????????? 0???????????????????
>>     41?????????????????? 146
>>     No. of Writes:?????????????????? 59??????????????????
>>     715??????????????????? 80
>>     ?
>>     ?? Block Size:?????????????? 1024b+???????????????
>>     2048b+??????????????? 4096b+
>>     ?No. of Reads:????????????????? 224????????????????????
>>     9??????????????????? 15
>>     No. of Writes:????????????????? 163??????????????????
>>     282?????????????????? 404
>>     ?
>>     ?? Block Size:?????????????? 8192b+??????????????
>>     16384b+?????????????? 32768b+
>>     ?No. of Reads:?????????????????? 52???????????????????
>>     74?????????????????? 210
>>     No. of Writes:???????????????? 7415?????????????????
>>     1430????????????????? 6860
>>     ?
>>     ?? Block Size:????????????? 65536b+?????????????
>>     131072b+????????????? 262144b+
>>     ?No. of Reads:????????????????? 299????????????????
>>     15313???????????????????? 1
>>     No. of Writes:???????????????? 2458????????????????
>>     12378???????????????????? 1
>>     ?
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     686733????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     837434???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     87338? RELEASEDIR
>>     ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us???????????
>>     116???????? IPC
>>     ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 us????????????
>>     57?? FTRUNCATE
>>     ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 us????????????
>>     65?? FGETXATTR
>>     ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 us????????????
>>     23 REMOVEXATTR
>>     ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us???????????
>>     121??????? LINK
>>     ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us???????????
>>     490????????? LK
>>     ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us???????????
>>     540??????? STAT
>>     ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us???????????
>>     200??? TRUNCATE
>>     ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us??????????
>>     1482???? SETATTR
>>     ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 us????????????
>>     12?????? FSYNC
>>     ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us?????????
>>     10047?????? FLUSH
>>     ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us??????????
>>     9977??? FINODELK
>>     ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us???????????
>>     188????? RENAME
>>     ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 us????????????
>>     25???? SYMLINK
>>     ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us???????????
>>     118?????? MKNOD
>>     ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us?????????
>>     12587?????? FSTAT
>>     ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us?????????
>>     21695????? STATFS
>>     ????? 0.00???? 307.91 us????? 28.00 us? 292666.83 us?????????
>>     16841??????? READ
>>     ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us???????????
>>     178?????? MKDIR
>>     ????? 0.00???? 319.22 us????? 51.19 us? 136353.39 us?????????
>>     33445?????? WRITE
>>     ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us?????????
>>     87338???? OPENDIR
>>     ????? 0.01???? 249.95 us????? 58.42 us? 594774.86 us?????????
>>     85089??? SETXATTR
>>     ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us??????????
>>     3180?????? RMDIR
>>     ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us??????????
>>     6067????? CREATE
>>     ????? 0.05????? 89.46 us????? 30.14 us? 687961.60 us???????
>>     1219542??? READLINK
>>     ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us????????
>>     831371??????? OPEN
>>     ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us??????????
>>     9493??? FXATTROP
>>     ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us???????
>>     3764714???? ENTRYLK
>>     ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us?????????
>>     22253???? READDIR
>>     ????? 1.06??? 6913.26 us????? 11.58 us 36968532.72 us????????
>>     364963??? GETXATTR
>>     ????? 1.29?? 62121.95 us????? 36.89 us 8649871.11 us?????????
>>     49472??? READDIRP
>>     ????? 2.89??? 1477.58 us????? 11.10 us 2807637.64 us???????
>>     4650500???? INODELK
>>     ????? 4.12?? 23681.29 us????? 20.98 us 23291414.50 us????????
>>     414352????? UNLINK
>>     ???? 16.09??? 5000.02 us?????? 9.04 us 23866016.88 us???????
>>     7655621????? LOOKUP
>>     ???? 72.04?? 68956.14 us????? 68.40 us 27542499.65 us???????
>>     2486126???? XATTROP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     2433405????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     2453???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>>     69??? CI_XATTR
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     1544?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     2430885?? CI_FORGET
>>     ?
>>     ??? Duration: 209524 seconds
>>     ?? Data Read: 2051388524 bytes
>>     Data Written: 2208261499 bytes
>>     ?
>>     Brick: flexo:/mnt/gluster/b
>>     --------------------------------------
>>     Cumulative Stats:
>>     ?? Block Size:????????????????? 1b+??????????????????
>>     8b+????????????????? 16b+
>>     ?No. of Reads:??????????????????? 0????????????????????
>>     0???????????????????? 0
>>     No. of Writes:????????????????? 117??????????????????
>>     195?????????????????? 367
>>     ?
>>     ?? Block Size:???????????????? 32b+?????????????????
>>     64b+???????????????? 128b+
>>     ?No. of Reads:????????????????? 130????????????????????
>>     0???????????????????? 3
>>     No. of Writes:???????????????? 1012????????????????????
>>     9?????????????????? 134
>>     ?
>>     ?? Block Size:??????????????? 256b+????????????????
>>     512b+??????????????? 1024b+
>>     ?No. of Reads:?????????????????? 47????????????????????
>>     8?????????????????? 257
>>     No. of Writes:????????????????? 729???????????????????
>>     73?????????????????? 168
>>     ?
>>     ?? Block Size:?????????????? 2048b+???????????????
>>     4096b+??????????????? 8192b+
>>     ?No. of Reads:?????????????????? 10???????????????????
>>     27?????????????????? 240
>>     No. of Writes:????????????????? 294??????????????????
>>     466???????????????? 10472
>>     ?
>>     ?? Block Size:????????????? 16384b+??????????????
>>     32768b+?????????????? 65536b+
>>     ?No. of Reads:????????????????? 139??????????????????
>>     119?????????????????? 135
>>     No. of Writes:???????????????? 1480?????????????????
>>     6273????????????????? 3935
>>     ?
>>     ?? Block Size:???????????? 131072b+
>>     ?No. of Reads:???????????????? 7892
>>     No. of Writes:??????????????? 12327
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     955058????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1450749???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     81377? RELEASEDIR
>>     ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us???????????
>>     108???????? IPC
>>     ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 us????????????
>>     44?? FTRUNCATE
>>     ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 us????????????
>>     30 REMOVEXATTR
>>     ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 us?????????????
>>     4?????? FSYNC
>>     ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 us????????????
>>     30???? SYMLINK
>>     ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us???????????
>>     137??????? LINK
>>     ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us???????????
>>     119?????? MKNOD
>>     ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us???????????
>>     221??? SETXATTR
>>     ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us???????????
>>     354??????? STAT
>>     ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us???????????
>>     777????????? LK
>>     ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us???????????
>>     188????? RENAME
>>     ????? 0.00???? 150.40 us????? 75.59 us???? 749.54 us???????????
>>     810?? FGETXATTR
>>     ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us???????????
>>     698??? TRUNCATE
>>     ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us???????????
>>     178?????? MKDIR
>>     ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us??????????
>>     1665???? SETATTR
>>     ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us?????????
>>     11372??? FINODELK
>>     ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us??????????
>>     7398?????? FSTAT
>>     ????? 0.00???? 120.71 us????? 31.43 us???? 847.97 us??????????
>>     9094??????? READ
>>     ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us?????????
>>     22614?????? FLUSH
>>     ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us?????????
>>     21656????? STATFS
>>     ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us??????????
>>     6145????? CREATE
>>     ????? 0.00???? 217.55 us????? 50.60 us?? 30734.99 us?????????
>>     38051?????? WRITE
>>     ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us?????????
>>     81377???? OPENDIR
>>     ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us??????????
>>     3177?????? RMDIR
>>     ????? 0.03???? 100.05 us????? 30.93 us?? 49756.93 us????????
>>     432439??? READLINK
>>     ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us????????
>>     448381????? UNLINK
>>     ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us?????????
>>     52185??? READDIRP
>>     ????? 0.35???? 414.05 us????? 36.58 us 1038229.93 us???????
>>     1444606??????? OPEN
>>     ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us?????????
>>     10745??? FXATTROP
>>     ????? 0.72???? 776.21 us????? 11.24 us 4271792.57 us???????
>>     1605822??? GETXATTR
>>     ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us?????????
>>     13532???? READDIR
>>     ????? 0.79???? 320.44 us?????? 7.71 us 2369511.06 us???????
>>     4257297???? ENTRYLK
>>     ????? 1.97???? 464.02 us?????? 9.38 us 1038460.80 us???????
>>     7333722???? INODELK
>>     ???? 13.73??? 2372.50 us?????? 8.33 us 6258159.87 us???????
>>     9979799????? LOOKUP
>>     ???? 80.79?? 42372.04 us????? 61.49 us 2238308.13 us???????
>>     3289103???? XATTROP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     2250268????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     3756???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>>     167??? CI_XATTR
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     1908?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     2246352?? CI_FORGET
>>     ?
>>     ??? Duration: 209334 seconds
>>     ?? Data Read: 1059777395 bytes
>>     Data Written: 2336921014 bytes
>>     ?
>>     Interval 0 Stats:
>>     ?? Block Size:????????????????? 1b+??????????????????
>>     8b+????????????????? 16b+
>>     ?No. of Reads:??????????????????? 0????????????????????
>>     0???????????????????? 0
>>     No. of Writes:????????????????? 117??????????????????
>>     195?????????????????? 367
>>     ?
>>     ?? Block Size:???????????????? 32b+?????????????????
>>     64b+???????????????? 128b+
>>     ?No. of Reads:????????????????? 130????????????????????
>>     0???????????????????? 3
>>     No. of Writes:???????????????? 1012????????????????????
>>     9?????????????????? 134
>>     ?
>>     ?? Block Size:??????????????? 256b+????????????????
>>     512b+??????????????? 1024b+
>>     ?No. of Reads:?????????????????? 47????????????????????
>>     8?????????????????? 257
>>     No. of Writes:????????????????? 729???????????????????
>>     73?????????????????? 168
>>     ?
>>     ?? Block Size:?????????????? 2048b+???????????????
>>     4096b+??????????????? 8192b+
>>     ?No. of Reads:?????????????????? 10???????????????????
>>     27?????????????????? 240
>>     No. of Writes:????????????????? 294??????????????????
>>     466???????????????? 10472
>>     ?
>>     ?? Block Size:????????????? 16384b+??????????????
>>     32768b+?????????????? 65536b+
>>     ?No. of Reads:????????????????? 139??????????????????
>>     119?????????????????? 135
>>     No. of Writes:???????????????? 1480?????????????????
>>     6273????????????????? 3935
>>     ?
>>     ?? Block Size:???????????? 131072b+
>>     ?No. of Reads:???????????????? 7892
>>     No. of Writes:??????????????? 12327
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     955058????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1450749???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     81377? RELEASEDIR
>>     ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us???????????
>>     108???????? IPC
>>     ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 us????????????
>>     44?? FTRUNCATE
>>     ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 us????????????
>>     30 REMOVEXATTR
>>     ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 us?????????????
>>     4?????? FSYNC
>>     ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 us????????????
>>     30???? SYMLINK
>>     ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us???????????
>>     137??????? LINK
>>     ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us???????????
>>     119?????? MKNOD
>>     ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us???????????
>>     221??? SETXATTR
>>     ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us???????????
>>     354??????? STAT
>>     ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us???????????
>>     777????????? LK
>>     ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us???????????
>>     188????? RENAME
>>     ????? 0.00???? 150.43 us????? 75.59 us???? 749.54 us???????????
>>     810?? FGETXATTR
>>     ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us???????????
>>     698??? TRUNCATE
>>     ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us???????????
>>     178?????? MKDIR
>>     ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us??????????
>>     1665???? SETATTR
>>     ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us?????????
>>     11372??? FINODELK
>>     ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us??????????
>>     7398?????? FSTAT
>>     ????? 0.00???? 120.74 us????? 31.43 us???? 847.97 us??????????
>>     9094??????? READ
>>     ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us?????????
>>     22614?????? FLUSH
>>     ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us?????????
>>     21656????? STATFS
>>     ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us??????????
>>     6145????? CREATE
>>     ????? 0.00???? 217.50 us????? 50.60 us?? 30734.99 us?????????
>>     38051?????? WRITE
>>     ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us?????????
>>     81377???? OPENDIR
>>     ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us??????????
>>     3177?????? RMDIR
>>     ????? 0.03???? 100.06 us????? 30.93 us?? 49756.93 us????????
>>     432439??? READLINK
>>     ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us????????
>>     448381????? UNLINK
>>     ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us?????????
>>     52185??? READDIRP
>>     ????? 0.35???? 414.01 us????? 36.58 us 1038229.93 us???????
>>     1444606??????? OPEN
>>     ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us?????????
>>     10745??? FXATTROP
>>     ????? 0.72???? 776.24 us????? 11.24 us 4271792.57 us???????
>>     1605822??? GETXATTR
>>     ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us?????????
>>     13532???? READDIR
>>     ????? 0.79???? 320.45 us?????? 7.71 us 2369511.06 us???????
>>     4257297???? ENTRYLK
>>     ????? 1.97???? 464.07 us?????? 9.38 us 1038460.80 us???????
>>     7333722???? INODELK
>>     ???? 13.72??? 2372.42 us?????? 8.33 us 6258159.87 us???????
>>     9979799????? LOOKUP
>>     ???? 80.79?? 42373.50 us????? 61.49 us 2238308.13 us???????
>>     3289103???? XATTROP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     2250268????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     3756???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>>     167??? CI_XATTR
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     1908?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     2246352?? CI_FORGET
>>     ?
>>     ??? Duration: 209334 seconds
>>     ?? Data Read: 1059777395 bytes
>>     Data Written: 2336921014 bytes
>>     ?
>>     Brick: flexo:/mnt/gluster/c
>>     --------------------------------------
>>     Cumulative Stats:
>>     ?? Block Size:????????????????? 1b+
>>     ?No. of Reads:??????????????????? 0
>>     No. of Writes:??????????????? 32843
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     686562????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     834596???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     93757? RELEASEDIR
>>     ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us???????????
>>     107???????? IPC
>>     ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 us????????????
>>     36??????? LINK
>>     ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 us????????????
>>     65?? FGETXATTR
>>     ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 us????????????
>>     88????? RENAME
>>     ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us???????????
>>     153??? TRUNCATE
>>     ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us???????????
>>     491????????? LK
>>     ????? 0.01????? 51.73 us?????? 7.70 us??? 1174.64 us??????????
>>     9339?????? FLUSH
>>     ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us??????????
>>     9011??? FINODELK
>>     ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us?????????
>>     10824?????? FSTAT
>>     ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us??????????
>>     2594?????? MKDIR
>>     ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us??????????
>>     9009??? FXATTROP
>>     ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us??????????
>>     5754????? CREATE
>>     ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us?????????
>>     32843?????? WRITE
>>     ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us??????????
>>     5798???? SYMLINK
>>     ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us?????????
>>     34904???? READDIR
>>     ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us??????????
>>     2732?????? RMDIR
>>     ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us?????????
>>     93757???? OPENDIR
>>     ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us????????
>>     170292????? UNLINK
>>     ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us?????????
>>     13840??? READDIRP
>>     ????? 1.94???? 200.88 us????? 25.49 us? 461654.32 us????????
>>     468842?? FTRUNCATE
>>     ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us????????
>>     486882 REMOVEXATTR
>>     ????? 2.27???? 223.01 us????? 52.03 us? 178897.40 us????????
>>     493240??? SETXATTR
>>     ????? 3.53???? 155.31 us?????? 7.96 us?? 67873.00 us???????
>>     1101338??? GETXATTR
>>     ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us????????
>>     829002??????? OPEN
>>     ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us???????
>>     3660425???? ENTRYLK
>>     ????? 6.23???? 304.93 us????? 42.51 us? 661394.43 us????????
>>     990644???? SETATTR
>>     ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us????????
>>     990766???? XATTROP
>>     ???? 12.39???? 549.11 us???? 135.57 us 1659697.83 us???????
>>     1093387?????? MKNOD
>>     ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us???????
>>     4325281???? INODELK
>>     ???? 39.53???? 206.94 us?????? 8.56 us? 964823.33 us???????
>>     9259429????? LOOKUP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1856377????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     933300???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>>     251??? CI_XATTR
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>>     293?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     923077?? CI_FORGET
>>     ?
>>     ??? Duration: 209334 seconds
>>     ?? Data Read: 0 bytes
>>     Data Written: 32843 bytes
>>     ?
>>     Interval 0 Stats:
>>     ?? Block Size:????????????????? 1b+
>>     ?No. of Reads:??????????????????? 0
>>     No. of Writes:??????????????? 32843
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     686562????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     834596???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     93757? RELEASEDIR
>>     ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us???????????
>>     107???????? IPC
>>     ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 us????????????
>>     36??????? LINK
>>     ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 us????????????
>>     65?? FGETXATTR
>>     ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 us????????????
>>     88????? RENAME
>>     ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us???????????
>>     153??? TRUNCATE
>>     ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us???????????
>>     491????????? LK
>>     ????? 0.01????? 51.71 us?????? 7.70 us??? 1174.64 us??????????
>>     9339?????? FLUSH
>>     ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us??????????
>>     9011??? FINODELK
>>     ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us?????????
>>     10824?????? FSTAT
>>     ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us??????????
>>     2594?????? MKDIR
>>     ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us??????????
>>     9009??? FXATTROP
>>     ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us??????????
>>     5754????? CREATE
>>     ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us?????????
>>     32843?????? WRITE
>>     ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us??????????
>>     5798???? SYMLINK
>>     ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us?????????
>>     34904???? READDIR
>>     ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us??????????
>>     2732?????? RMDIR
>>     ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us?????????
>>     93757???? OPENDIR
>>     ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us????????
>>     170292????? UNLINK
>>     ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us?????????
>>     13840??? READDIRP
>>     ????? 1.94???? 200.87 us????? 25.49 us? 461654.32 us????????
>>     468842?? FTRUNCATE
>>     ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us????????
>>     486882 REMOVEXATTR
>>     ????? 2.27???? 223.00 us????? 52.03 us? 178897.40 us????????
>>     493240??? SETXATTR
>>     ????? 3.53???? 155.32 us?????? 7.96 us?? 67873.00 us???????
>>     1101338??? GETXATTR
>>     ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us????????
>>     829002??????? OPEN
>>     ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us???????
>>     3660425???? ENTRYLK
>>     ????? 6.23???? 304.88 us????? 42.51 us? 661394.43 us????????
>>     990644???? SETATTR
>>     ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us????????
>>     990766???? XATTROP
>>     ???? 12.39???? 549.25 us???? 135.57 us 1659697.83 us???????
>>     1093387?????? MKNOD
>>     ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us???????
>>     4325281???? INODELK
>>     ???? 39.53???? 206.95 us?????? 8.56 us? 964823.33 us???????
>>     9259429????? LOOKUP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1856377????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     933300???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>>     251??? CI_XATTR
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>>     293?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     923077?? CI_FORGET
>>     ?
>>     ??? Duration: 209334 seconds
>>     ?? Data Read: 0 bytes
>>     Data Written: 32843 bytes
>>     ?
>>     Brick: bender:/mnt/gluster/c
>>     ---------------------------------------
>>     Cumulative Stats:
>>     ?? Block Size:????????????????? 1b+
>>     ?No. of Reads:??????????????????? 0
>>     No. of Writes:??????????????? 37050
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     928119????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1401524???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     70092? RELEASEDIR
>>     ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us???????????
>>     100???????? IPC
>>     ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 us????????????
>>     38??????? LINK
>>     ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 us????????????
>>     30???? SYMLINK
>>     ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 us?????????????
>>     4?????? FSYNC
>>     ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us???????????
>>     122????? RENAME
>>     ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us???????????
>>     777????????? LK
>>     ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us???????????
>>     810?? FGETXATTR
>>     ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us???????????
>>     603??? TRUNCATE
>>     ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us???????????
>>     302?????? MKDIR
>>     ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us?????????
>>     10060??? FINODELK
>>     ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us??????????
>>     6409?????? FSTAT
>>     ????? 0.01????? 49.38 us?????? 8.08 us??? 1264.41 us?????????
>>     21719?????? FLUSH
>>     ????? 0.03????? 74.76 us????? 20.86 us??? 2008.41 us?????????
>>     37050?????? WRITE
>>     ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us??????????
>>     5901????? CREATE
>>     ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us?????????
>>     10057??? FXATTROP
>>     ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us?????????
>>     70092???? OPENDIR
>>     ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us?????????
>>     10779???? READDIR
>>     ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us??????????
>>     3008?????? RMDIR
>>     ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us????????
>>     109478????? UNLINK
>>     ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us????????
>>     722669 REMOVEXATTR
>>     ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us????????
>>     789294??? GETXATTR
>>     ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us????????
>>     718835?? FTRUNCATE
>>     ????? 1.89???? 219.47 us????? 49.92 us? 506767.78 us????????
>>     722856??? SETXATTR
>>     ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us???????
>>     1395957??????? OPEN
>>     ????? 5.56???? 322.08 us????? 43.37 us? 634069.61 us???????
>>     1450160???? SETATTR
>>     ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us???????
>>     4233520???? ENTRYLK
>>     ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us???????
>>     1449601???? XATTROP
>>     ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us???????
>>     7097226???? INODELK
>>     ???? 19.37??? 1104.98 us???? 101.28 us 1612346.75 us???????
>>     1472813?????? MKNOD
>>     ???? 41.85???? 295.61 us?????? 8.98 us? 996805.54 us??????
>>     11896091????? LOOKUP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     3531139????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1862470???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>>     244??? CI_XATTR
>>     ?????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>>     632?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1668669?? CI_FORGET
>>     ?
>>     ??? Duration: 208643 seconds
>>     ?? Data Read: 0 bytes
>>     Data Written: 37050 bytes
>>     ?
>>     Interval 0 Stats:
>>     ?? Block Size:????????????????? 1b+
>>     ?No. of Reads:??????????????????? 0
>>     No. of Writes:??????????????? 37050
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     928119????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1401524???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     70092? RELEASEDIR
>>     ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us???????????
>>     100???????? IPC
>>     ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 us????????????
>>     38??????? LINK
>>     ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 us????????????
>>     30???? SYMLINK
>>     ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 us?????????????
>>     4?????? FSYNC
>>     ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us???????????
>>     122????? RENAME
>>     ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us???????????
>>     777????????? LK
>>     ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us???????????
>>     810?? FGETXATTR
>>     ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us???????????
>>     603??? TRUNCATE
>>     ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us???????????
>>     302?????? MKDIR
>>     ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us?????????
>>     10060??? FINODELK
>>     ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us??????????
>>     6409?????? FSTAT
>>     ????? 0.01????? 49.39 us?????? 8.08 us??? 1264.41 us?????????
>>     21719?????? FLUSH
>>     ????? 0.03????? 74.77 us????? 20.86 us??? 2008.41 us?????????
>>     37050?????? WRITE
>>     ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us??????????
>>     5901????? CREATE
>>     ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us?????????
>>     10057??? FXATTROP
>>     ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us?????????
>>     70092???? OPENDIR
>>     ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us?????????
>>     10779???? READDIR
>>     ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us??????????
>>     3008?????? RMDIR
>>     ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us????????
>>     109478????? UNLINK
>>     ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us????????
>>     722669 REMOVEXATTR
>>     ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us????????
>>     789294??? GETXATTR
>>     ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us????????
>>     718835?? FTRUNCATE
>>     ????? 1.89???? 219.48 us????? 49.92 us? 506767.78 us????????
>>     722856??? SETXATTR
>>     ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us???????
>>     1395957??????? OPEN
>>     ????? 5.56???? 322.09 us????? 43.37 us? 634069.61 us???????
>>     1450160???? SETATTR
>>     ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us???????
>>     4233520???? ENTRYLK
>>     ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us???????
>>     1449601???? XATTROP
>>     ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us???????
>>     7097226???? INODELK
>>     ???? 19.37??? 1105.06 us???? 101.28 us 1612346.75 us???????
>>     1472813?????? MKNOD
>>     ???? 41.84???? 295.54 us?????? 8.98 us? 996805.54 us??????
>>     11896091????? LOOKUP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     3531139????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1862470???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>>     244??? CI_XATTR
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>>     632?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1668669?? CI_FORGET
>>     ?
>>     ??? Duration: 208643 seconds
>>     ?? Data Read: 0 bytes
>>     Data Written: 37050 bytes
>>     ?
>>     Brick: bender:/mnt/gluster/a
>>     ---------------------------------------
>>     Cumulative Stats:
>>     ?? Block Size:????????????????? 1b+??????????????????
>>     2b+?????????????????? 8b+
>>     ?No. of Reads:??????????????????? 0????????????????????
>>     0???????????????????? 0
>>     No. of Writes:?????????????????? 53????????????????????
>>     1??????????????????? 60
>>     ?
>>     ?? Block Size:???????????????? 16b+?????????????????
>>     32b+????????????????? 64b+
>>     ?No. of Reads:??????????????????? 0??????????????????
>>     235???????????????????? 4
>>     No. of Writes:?????????????????? 14?????????????????
>>     1033??????????????????? 15
>>     ?
>>     ?? Block Size:??????????????? 128b+????????????????
>>     256b+???????????????? 512b+
>>     ?No. of Reads:??????????????????? 4??????????????????
>>     115??????????????????? 81
>>     No. of Writes:?????????????????? 58??????????????????
>>     703??????????????????? 80
>>     ?
>>     ?? Block Size:?????????????? 1024b+???????????????
>>     2048b+??????????????? 4096b+
>>     ?No. of Reads:????????????????? 181???????????????????
>>     26??????????????????? 74
>>     No. of Writes:????????????????? 163??????????????????
>>     278?????????????????? 397
>>     ?
>>     ?? Block Size:?????????????? 8192b+??????????????
>>     16384b+?????????????? 32768b+
>>     ?No. of Reads:????????????????? 131??????????????????
>>     161?????????????????? 267
>>     No. of Writes:???????????????? 7264?????????????????
>>     1398????????????????? 6742
>>     ?
>>     ?? Block Size:????????????? 65536b+?????????????
>>     131072b+????????????? 262144b+
>>     ?No. of Reads:????????????????? 443????????????????
>>     25528???????????????????? 0
>>     No. of Writes:???????????????? 2426????????????????
>>     12240???????????????????? 1
>>     ?
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     684942????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     837092???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     87297? RELEASEDIR
>>     ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 us?????????????
>>     8 REMOVEXATTR
>>     ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 us????????????
>>     97???????? IPC
>>     ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 us????????????
>>     46?? FTRUNCATE
>>     ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 us????????????
>>     65?? FGETXATTR
>>     ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us???????????
>>     121??????? LINK
>>     ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 us????????????
>>     25???? SYMLINK
>>     ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us???????????
>>     107?????? MKNOD
>>     ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 us?????????????
>>     1?????? FSYNC
>>     ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us???????????
>>     188????? RENAME
>>     ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us???????????
>>     745??????? STAT
>>     ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us???????????
>>     200??? TRUNCATE
>>     ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us??????????
>>     1444???? SETATTR
>>     ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us???????????
>>     177?????? MKDIR
>>     ????? 0.00????? 52.68 us?????? 7.28 us???? 826.01 us??????????
>>     9890?????? FLUSH
>>     ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us??????????
>>     9792??? FINODELK
>>     ????? 0.00????? 74.26 us????? 20.00 us??? 1522.39 us?????????
>>     21521????? STATFS
>>     ????? 0.00???? 129.21 us????? 34.15 us??? 9068.67 us?????????
>>     13094?????? FSTAT
>>     ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us??????????
>>     5967????? CREATE
>>     ????? 0.00???? 112.06 us????? 28.14 us??? 9545.53 us?????????
>>     27474??????? READ
>>     ????? 0.01???? 313.76 us????? 52.98 us?? 71635.74 us?????????
>>     32926?????? WRITE
>>     ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us?????????
>>     87297???? OPENDIR
>>     ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us??????????
>>     3177?????? RMDIR
>>     ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us?????????
>>     85073??? SETXATTR
>>     ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us?????????
>>     15090??? READDIRP
>>     ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us???????????
>>     564????????? LK
>>     ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us????????
>>     850938??? READLINK
>>     ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us????????
>>     831212??????? OPEN
>>     ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us?????????
>>     22482???? READDIR
>>     ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us????????
>>     415226????? UNLINK
>>     ????? 0.54??? 1272.83 us?????? 9.66 us 1604368.93 us????????
>>     686945??? GETXATTR
>>     ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us???????
>>     3739718???? ENTRYLK
>>     ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us??????????
>>     9311??? FXATTROP
>>     ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us???????
>>     4649291???? INODELK
>>     ????? 5.82??? 1240.20 us?????? 8.78 us 1526868.50 us???????
>>     7625383????? LOOKUP
>>     ???? 88.60?? 58005.98 us????? 67.31 us 1448810.06 us???????
>>     2481914???? XATTROP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1396875????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     1854???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>>     28??? CI_XATTR
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     1051?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1394996?? CI_FORGET
>>     ?
>>     ??? Duration: 208643 seconds
>>     ?? Data Read: 3409342848 bytes
>>     Data Written: 2180686507 bytes
>>     ?
>>     Interval 0 Stats:
>>     ?? Block Size:????????????????? 1b+??????????????????
>>     2b+?????????????????? 8b+
>>     ?No. of Reads:??????????????????? 0????????????????????
>>     0???????????????????? 0
>>     No. of Writes:?????????????????? 53????????????????????
>>     1??????????????????? 60
>>     ?
>>     ?? Block Size:???????????????? 16b+?????????????????
>>     32b+????????????????? 64b+
>>     ?No. of Reads:??????????????????? 0??????????????????
>>     235???????????????????? 4
>>     No. of Writes:?????????????????? 14?????????????????
>>     1033??????????????????? 15
>>     ?
>>     ?? Block Size:??????????????? 128b+????????????????
>>     256b+???????????????? 512b+
>>     ?No. of Reads:??????????????????? 4??????????????????
>>     115??????????????????? 81
>>     No. of Writes:?????????????????? 58??????????????????
>>     703??????????????????? 80
>>     ?
>>     ?? Block Size:?????????????? 1024b+???????????????
>>     2048b+??????????????? 4096b+
>>     ?No. of Reads:????????????????? 181???????????????????
>>     26??????????????????? 74
>>     No. of Writes:????????????????? 163??????????????????
>>     278?????????????????? 397
>>     ?
>>     ?? Block Size:?????????????? 8192b+??????????????
>>     16384b+?????????????? 32768b+
>>     ?No. of Reads:????????????????? 131??????????????????
>>     161?????????????????? 267
>>     No. of Writes:???????????????? 7264?????????????????
>>     1398????????????????? 6742
>>     ?
>>     ?? Block Size:????????????? 65536b+?????????????
>>     131072b+????????????? 262144b+
>>     ?No. of Reads:????????????????? 443????????????????
>>     25528???????????????????? 0
>>     No. of Writes:???????????????? 2426????????????????
>>     12240???????????????????? 1
>>     ?
>>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>>     calls???????? Fop
>>     ?---------?? -----------?? -----------?? -----------??
>>     ------------??????? ----
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     684942????? FORGET
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>>     837092???? RELEASE
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>>     87297? RELEASEDIR
>>     ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 us?????????????
>>     8 REMOVEXATTR
>>     ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 us????????????
>>     97???????? IPC
>>     ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 us????????????
>>     46?? FTRUNCATE
>>     ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 us????????????
>>     65?? FGETXATTR
>>     ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us???????????
>>     121??????? LINK
>>     ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 us????????????
>>     25???? SYMLINK
>>     ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us???????????
>>     107?????? MKNOD
>>     ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 us?????????????
>>     1?????? FSYNC
>>     ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us???????????
>>     188????? RENAME
>>     ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us???????????
>>     745??????? STAT
>>     ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us???????????
>>     200??? TRUNCATE
>>     ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us??????????
>>     1444???? SETATTR
>>     ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us???????????
>>     177?????? MKDIR
>>     ????? 0.00????? 52.69 us?????? 7.28 us???? 826.01 us??????????
>>     9890?????? FLUSH
>>     ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us??????????
>>     9792??? FINODELK
>>     ????? 0.00????? 74.25 us????? 20.00 us??? 1522.39 us?????????
>>     21521????? STATFS
>>     ????? 0.00???? 129.23 us????? 34.15 us??? 9068.67 us?????????
>>     13094?????? FSTAT
>>     ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us??????????
>>     5967????? CREATE
>>     ????? 0.00???? 112.45 us????? 28.14 us??? 9545.53 us?????????
>>     27474??????? READ
>>     ????? 0.01???? 313.77 us????? 52.98 us?? 71635.74 us?????????
>>     32926?????? WRITE
>>     ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us?????????
>>     87297???? OPENDIR
>>     ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us??????????
>>     3177?????? RMDIR
>>     ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us?????????
>>     85073??? SETXATTR
>>     ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us?????????
>>     15090??? READDIRP
>>     ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us???????????
>>     564????????? LK
>>     ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us????????
>>     850938??? READLINK
>>     ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us????????
>>     831212??????? OPEN
>>     ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us?????????
>>     22482???? READDIR
>>     ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us????????
>>     415226????? UNLINK
>>     ????? 0.54??? 1272.79 us?????? 9.66 us 1604368.93 us????????
>>     686945??? GETXATTR
>>     ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us???????
>>     3739718???? ENTRYLK
>>     ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us??????????
>>     9311??? FXATTROP
>>     ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us???????
>>     4649291???? INODELK
>>     ????? 5.82??? 1240.10 us?????? 8.78 us 1526868.50 us???????
>>     7625383????? LOOKUP
>>     ???? 88.60?? 58006.02 us????? 67.31 us 1448810.06 us???????
>>     2481914???? XATTROP
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1396875????? UPCALL
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     1854???? CI_IATT
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>>     28??? CI_XATTR
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>>     1051?? CI_UNLINK
>>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>>     1394996?? CI_FORGET
>>     ?
>>     ??? Duration: 208643 seconds
>>     ?? Data Read: 3409342848 bytes
>>     Data Written: 2180686507 bytes
>>     ?
>>
>>     ________
>>
>>
>>
>>     Community Meeting Calendar:
>>
>>     Schedule -
>>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>     Bridge: https://bluejeans.com/441850968
>>
>>     Gluster-users mailing list
>>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>>     https://lists.gluster.org/mailman/listinfo/gluster-users
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200507/0ddc505a/attachment.html>