thr3ads.net - Gluster users - [Gluster-users] Possible SHD stalling [May 2020]

If this information is useful, please help other people find it:
Share via:

Artem Russakovskii

2020-May-04 19:56 UTC

[Gluster-users] Possible SHD stalling

I reported something similar recently to this list, and I highly advise you
to update to 7.5 (in my case it was 5.11 -> 5.13), then see if it keeps
happening. The self heal issues should be fixed there properly at least.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>


On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za> wrote:
> Hi All,
>
> So we've got a cluster currently running on glusterfs 7.3
>
> Heal performance is absolutely terrible (it starts of great, runs for a
> while then completely trips up).
>
> Once the SHD trips up, access speeds to the cluster also becomes poor.
> And with poor I mean directory listings that normally takes well under
> 500ms takes upwards of 10s, but I've seen upwards of 40s (although a
follow
> up ls immediately after is nice and fast again, temporarily).  With all
> SHDs killed this particular ls is as per normal (around 30-50ms).
>
> The SHD logs doesn't provide any usable clues, the only variance in
logs
> is this, just before all daemons across the cluster crawls to a near halt.
> [2020-05-02 19:55:58.627036] I [MSGID: 108026]
> [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0:
> Completed entry selfheal on 5b9aa3d8-5085-46e2-9500-145b8cbf9a61.
sources> sinks=0 1
> [2020-05-02 19:55:58.627058] I [MSGID: 108026]
> [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0:
> Completed entry selfheal on c736c9b6-f116-468b-adbc-f102f5b57d01.
sources> sinks=0 1
>
> [2020-05-02 19:55:59.125001] W [MSGID: 114031]
> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] 0-vpbx_shared-client-1:
> remote operation failed [Invalid argument]
>
> [2020-05-02 19:55:59.339803] I [MSGID: 108026]
> [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0:
> Completed entry selfheal on 2e128eeb-e991-4544-8316-4ddfcb4942da.
sources> sinks=0 1
> [2020-05-02 19:55:59.490387] I [MSGID: 108026]
> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
> 0-vpbx_shared-replicate-0: performing metadata selfheal on
> 527a63a3-9acd-4c3f-b95c-5c719325c973
>
> A few more entries will complete overall and then it'll be a file every
> other second.  Once I saw this I ran a few more tests, and on every test
> just prior to slowing to a crawl, that entry is there.
>
> There is a perceived improvement when lowering the thread count, but not
> significant enough to confirm a correlation.  Even with
cluster.shd-max-threads=1
> this still happens (gets a few tens thousand files completed instead of few
> hundred, at 32 it gets to maybe 400 to 500 before it stalls, 16 does maybe
> a thousand).
>
> There are probably performant bursts hereafter, because from experience we
> know this process takes about two months.  But this time around it seems to
> be much, much worse.  Might be because that was healing a single brick to
> another brick, so less locking going?  At the moment the replicas are
> basically trying to populate the newly added arbiters.  Also, we
haven't
> messed as greatly with the health check settings (queues and threads and
> locks nearly as much as since the arbiter addition because performance has
> simply dropped to non-acceptable levels.
>
> Unfortunately granular entry heal isn't on at the moment (nor can I
switch
> it on due to a few million outstanding heals).  This used to be a 2x2
> cluster.  We then moved one of the bricks to a newly added server using
> replace-brick.  What a frightening process this is.  We then proceeded to
> add the arbiters.
>
> Some of the settings below was absolute shots in the dark, based on the
> little information we've been able to track down so far.
>
> Kind Regards,
> Jaco
>
> The current gluster volume info (this used to be 2x2
> distributed-replicate, aiming to get it to 3x (2 + 1) ... at current rates
> we might get there by 2021):
>
> Volume Name: glvol
> Type: Distributed-Replicate
> Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x (2 + 1) = 6
> Transport-type: tcp
> Bricks:
> Brick1: zoidberg:/mnt/gluster/a
> Brick2: flexo:/mnt/gluster/b
> Brick3: bender:/mnt/gluster/c (arbiter)
> Brick4: bender:/mnt/gluster/a
> Brick5: zoidberg:/mnt/gluster/b
> Brick6: flexo:/mnt/gluster/c (arbiter)
> Options Reconfigured:
> *cluster.heal-timeout: 5 <-- hoping this would at least shorten the
> blockages.*
> cluster.locking-scheme: granular
> cluster.heal-wait-queue-length: 10240
> *diagnostics.count-fop-hits: on*
> *diagnostics.latency-measurement: on <-- was hoping profiling might
> provide some insight.  Going to switch this off now.*
> cluster.background-self-heal-count: 128
> *cluster.shd-wait-qlength: 131072*
> *cluster.shd-max-threads: 32  <-- even with this disk IO never goes over
> 10% utilization.  CPU goes to about 2 cores.  So it's lock contention
> that's slowing it down we reckon.*
> *cluster.metadata-self-heal: off*
> *cluster.entry-self-heal: off*
> *cluster.data-self-heal: off*
> * <-- these kill performance. *transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: on
> features.cache-invalidation: on
> features.cache-invalidation-timeout: 600
> performance.stat-prefetch: on
> performance.cache-invalidation: on
> performance.md-cache-timeout: 600
> *cluster.self-heal-daemon: yes <-- disabling/enabling this doesn't
help,
> have to kill the shd and start ... force to get things moving along again.*
> performance.io-thread-count: 16
> performance.high-prio-threads: 16
> performance.normal-prio-threads: 16
> performance.low-prio-threads: 16
> performance.least-prio-threads: 4
> cluster.self-heal-window-size: 512
> cluster.self-heal-readdir-size: 131072
> client.event-threads: 8
> server.event-threads: 16
> network.ping-timeout: 2
> *cluster.data-self-heal-algorithm: full <-- most files are small
(<1MB),
> on dual 1gbit links (so up to 2gbit, but b/w never goes over 5mbit
anyway).*
>
> performance info:
>
> Brick: zoidberg:/mnt/gluster/a
> -----------------------------------------
> Cumulative Stats:
>    Block Size:                  1b+                   8b+
> 16b+
>  No. of Reads:                    0
> 0                     0
> No. of Writes:                  117
> 195                   367
>
>    Block Size:                 32b+                  64b+
> 128b+
>  No. of Reads:                  306
> 0                     6
> No. of Writes:                 1016
> 9                   134
>
>    Block Size:                256b+                 512b+
> 1024b+
>  No. of Reads:                  107
> 58                   485
> No. of Writes:                  731
> 73                   168
>
>    Block Size:               2048b+                4096b+
> 8192b+
>  No. of Reads:                   43
> 59                   383
> No. of Writes:                  295                   475
> 10562
>
>    Block Size:              16384b+               32768b+
> 65536b+
>  No. of Reads:                  347
> 380                   634
> No. of Writes:                 1487                  6369
> 3970
>
>    Block Size:             131072b+
>  No. of Reads:                35704
> No. of Writes:                12398
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         957293
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us        1455091
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          69951
> RELEASEDIR
>       0.00      16.20 us       5.43 us     100.95 us
> 113         IPC
>       0.00     687.33 us      80.53 us   15414.03 us             30
> REMOVEXATTR
>       0.00     840.76 us      72.71 us   37648.39 us             52
> FTRUNCATE
>       0.00     138.49 us      36.66 us   25280.24 us           1385
> STAT
>       0.00     364.93 us      63.45 us   15957.66 us            810
> FGETXATTR
>       0.00    2208.82 us     127.23 us  267092.17 us            137
> LINK
>       0.00   37595.33 us    1677.46 us  218313.32 us             12
> FSYNC
>       0.00     675.61 us     106.82 us  114351.01 us            698
> TRUNCATE
>       0.00    3995.66 us     147.00 us  109544.46 us            188
> RENAME
>       0.00     453.11 us      53.37 us  104848.94 us           1680
> SETATTR
>       0.00    6004.32 us     229.33 us  151322.58 us            127
> MKNOD
>       0.00      68.74 us      16.11 us   61050.93 us          11412
> FINODELK
>       0.00    3793.44 us      68.61 us  411523.70 us            221
> SETXATTR
>       0.00   28669.99 us     227.28 us  138988.41 us             30
> SYMLINK
>       0.00      56.18 us       7.31 us   41484.37 us          22640
> FLUSH
>       0.00     105.64 us      19.34 us  351055.72 us          21693
> STATFS
>       0.00   23642.90 us     244.19 us  234912.25 us            178
> MKDIR
>       0.00     537.38 us      31.73 us  296911.67 us           9653
> FSTAT
>       0.00     214.87 us      25.71 us 1712020.99 us          39054
> READ
>       0.01     147.08 us       1.47 us  241611.80 us          69951
> OPENDIR
>       0.01    3899.84 us     117.22 us  503422.21 us           3180
> RMDIR
>       0.01     327.08 us      55.69 us  420376.80 us          38366
> WRITE
>       0.02    5580.36 us     210.11 us 1029702.30 us           6169
> CREATE
>       0.05      92.69 us      29.94 us  687955.86 us        1159936
> READLINK
>       0.40     543.50 us      36.45 us 2430653.54 us        1449307
> OPEN
>       0.48   87643.67 us      70.20 us  661999.51 us          10787
> FXATTROP
>       0.53    2342.72 us      20.91 us 4157542.43 us         447821
> UNLINK
>       0.53  558734.79 us      24.40 us 1399487.63 us
> 1889          LK
>       0.78  145786.31 us      19.06 us 37490825.73 us          10687
> READDIR
>       0.99    2426.23 us      12.47 us 24176123.13 us         809430
> GETXATTR
>       1.45     676.83 us       9.30 us 193177067.25 us        4260021
> ENTRYLK
>       1.76   38325.38 us      21.44 us 46693575.93 us          91439
> READDIRP
>       4.44    1200.31 us       9.10 us 2559377.90 us        7352811
> INODELK
>       9.30    1849.24 us       9.28 us 35299050.00 us        9989221
> LOOKUP
>      79.24   47791.46 us      76.77 us 10488824.26 us        3294642
> XATTROP
>       0.00       0.00 us       0.00 us       0.00 us        3194806
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us           5311
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us             54
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us           1907
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us        3189457
> CI_FORGET
>
>     Duration: 209524 seconds
>    Data Read: 4775365531 bytes
> Data Written: 2354108664 bytes
>
> Interval 0 Stats:
>    Block Size:                  1b+                   8b+
> 16b+
>  No. of Reads:                    0
> 0                     0
> No. of Writes:                  117
> 195                   367
>
>    Block Size:                 32b+                  64b+
> 128b+
>  No. of Reads:                  306
> 0                     6
> No. of Writes:                 1016
> 9                   134
>
>    Block Size:                256b+                 512b+
> 1024b+
>  No. of Reads:                  107
> 58                   485
> No. of Writes:                  731
> 73                   168
>
>    Block Size:               2048b+                4096b+
> 8192b+
>  No. of Reads:                   43
> 59                   383
> No. of Writes:                  295                   475
> 10562
>
>    Block Size:              16384b+               32768b+
> 65536b+
>  No. of Reads:                  347
> 380                   634
> No. of Writes:                 1487                  6369
> 3970
>
>    Block Size:             131072b+
>  No. of Reads:                35704
> No. of Writes:                12398
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         957293
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us        1455091
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          69951
> RELEASEDIR
>       0.00      16.20 us       5.43 us     100.95 us
> 113         IPC
>       0.00     687.33 us      80.53 us   15414.03 us             30
> REMOVEXATTR
>       0.00     840.76 us      72.71 us   37648.39 us             52
> FTRUNCATE
>       0.00     138.49 us      36.66 us   25280.24 us           1385
> STAT
>       0.00     363.20 us      63.45 us   15957.66 us            810
> FGETXATTR
>       0.00    2208.82 us     127.23 us  267092.17 us            137
> LINK
>       0.00   37595.33 us    1677.46 us  218313.32 us             12
> FSYNC
>       0.00     675.61 us     106.82 us  114351.01 us            698
> TRUNCATE
>       0.00    3995.66 us     147.00 us  109544.46 us            188
> RENAME
>       0.00     453.11 us      53.37 us  104848.94 us           1680
> SETATTR
>       0.00    6004.32 us     229.33 us  151322.58 us            127
> MKNOD
>       0.00      68.74 us      16.11 us   61050.93 us          11412
> FINODELK
>       0.00    3793.44 us      68.61 us  411523.70 us            221
> SETXATTR
>       0.00   28669.99 us     227.28 us  138988.41 us             30
> SYMLINK
>       0.00      56.19 us       7.31 us   41484.37 us          22640
> FLUSH
>       0.00     105.61 us      19.34 us  351055.72 us          21693
> STATFS
>       0.00   23642.90 us     244.19 us  234912.25 us            178
> MKDIR
>       0.00     537.38 us      31.73 us  296911.67 us           9653
> FSTAT
>       0.00     214.91 us      25.71 us 1712020.99 us          39054
> READ
>       0.01     147.08 us       1.47 us  241611.80 us          69951
> OPENDIR
>       0.01    3899.84 us     117.22 us  503422.21 us           3180
> RMDIR
>       0.01     327.13 us      55.69 us  420376.80 us          38366
> WRITE
>       0.02    5580.36 us     210.11 us 1029702.30 us           6169
> CREATE
>       0.05      92.69 us      29.94 us  687955.86 us        1159936
> READLINK
>       0.40     543.49 us      36.45 us 2430653.54 us        1449307
> OPEN
>       0.48   87643.67 us      70.20 us  661999.51 us          10787
> FXATTROP
>       0.53    2342.57 us      20.91 us 4157542.43 us         447821
> UNLINK
>       0.53  558416.10 us      24.40 us 1399487.63 us
> 1889          LK
>       0.78  145786.31 us      19.06 us 37490825.73 us          10687
> READDIR
>       0.99    2426.24 us      12.47 us 24176123.13 us         809430
> GETXATTR
>       1.45     676.75 us       9.30 us 193177067.25 us        4260021
> ENTRYLK
>       1.76   38324.59 us      21.44 us 46693575.93 us          91439
> READDIRP
>       4.44    1200.37 us       9.10 us 2559377.90 us        7352811
> INODELK
>       9.29    1848.94 us       9.28 us 35299050.00 us        9989221
> LOOKUP
>      79.24   47792.42 us      76.77 us 10488824.26 us        3294642
> XATTROP
>       0.00       0.00 us       0.00 us       0.00 us        3194806
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us           5311
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us             54
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us           1907
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us        3189457
> CI_FORGET
>
>     Duration: 209524 seconds
>    Data Read: 4775365531 bytes
> Data Written: 2354108664 bytes
>
> Brick: zoidberg:/mnt/gluster/b
> -----------------------------------------
> Cumulative Stats:
>    Block Size:                  1b+
> 2b+                   8b+
>  No. of Reads:                    0
> 0                     0
> No. of Writes:                   53
> 1                    60
>
>    Block Size:                 16b+                  32b+
> 64b+
>  No. of Reads:                    0
> 213                     2
> No. of Writes:                   14
> 1057                    15
>
>    Block Size:                128b+                 256b+
> 512b+
>  No. of Reads:                    0
> 41                   146
> No. of Writes:                   59
> 715                    80
>
>    Block Size:               1024b+                2048b+
> 4096b+
>  No. of Reads:                  224
> 9                    15
> No. of Writes:                  163
> 282                   404
>
>    Block Size:               8192b+               16384b+
> 32768b+
>  No. of Reads:                   52
> 74                   210
> No. of Writes:                 7415                  1430
> 6860
>
>    Block Size:              65536b+              131072b+
> 262144b+
>  No. of Reads:                  299
> 15313                     1
> No. of Writes:                 2458
> 12378                     1
>
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         686733
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us         837434
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          87338
> RELEASEDIR
>       0.00      22.69 us       7.16 us     332.61 us
> 116         IPC
>       0.00     119.93 us      70.56 us     272.28 us             57
> FTRUNCATE
>       0.00     152.25 us      86.82 us     340.63 us             65
> FGETXATTR
>       0.00    2336.06 us      90.05 us   20509.81 us             23
> REMOVEXATTR
>       0.00     631.26 us     123.17 us   51803.77 us            121
> LINK
>       0.00     369.27 us      22.64 us  145090.48 us
> 490          LK
>       0.00     360.85 us      45.14 us   46846.98 us            540
> STAT
>       0.00    1190.96 us     117.40 us   69781.85 us            200
> TRUNCATE
>       0.00     245.72 us      54.91 us  123985.76 us           1482
> SETATTR
>       0.00   42139.10 us    3313.56 us  312273.21 us             12
> FSYNC
>       0.00      55.83 us       8.26 us   49649.35 us          10047
> FLUSH
>       0.00      57.52 us      14.62 us     820.59 us           9977
> FINODELK
>       0.00    3191.60 us     143.06 us   96214.75 us            188
> RENAME
>       0.00   32862.86 us     215.38 us   83851.13 us             25
> SYMLINK
>       0.00   11392.03 us     230.70 us  105471.90 us            118
> MKNOD
>       0.00     134.63 us      28.08 us   36326.43 us          12587
> FSTAT
>       0.00     112.32 us      19.91 us  381241.34 us          21695
> STATFS
>       0.00     307.86 us      28.00 us  292666.83 us          16841
> READ
>       0.00   29625.60 us     239.57 us  239532.26 us            178
> MKDIR
>       0.00     319.24 us      51.19 us  136353.39 us          33445
> WRITE
>       0.01     175.83 us       1.57 us  513635.22 us          87338
> OPENDIR
>       0.01     249.96 us      58.42 us  594774.86 us          85089
> SETXATTR
>       0.01   11001.25 us     119.87 us  612765.03 us           3180
> RMDIR
>       0.04   14240.66 us     203.89 us 1376758.99 us           6067
> CREATE
>       0.05      89.45 us      30.14 us  687961.60 us        1219542
> READLINK
>       0.39    1114.30 us      44.29 us 2412169.89 us         831371
> OPEN
>       0.51  127078.45 us      78.55 us  694364.09 us           9493
> FXATTROP
>       0.63     399.16 us       8.43 us 3815689.36 us        3764714
> ENTRYLK
>       0.86   91973.00 us      19.89 us 24258016.33 us          22253
> READDIR
>       1.06    6913.71 us      11.58 us 36968532.72 us         364963
> GETXATTR
>       1.29   62130.12 us      36.89 us 8649871.11 us          49472
> READDIRP
>       2.89    1477.53 us      11.10 us 2807637.64 us        4650500
> INODELK
>       4.12   23681.31 us      20.98 us 23291414.50 us         414352
> UNLINK
>      16.09    5000.31 us       9.04 us 23866016.88 us        7655621
> LOOKUP
>      72.04   68953.31 us      68.40 us 27542499.65 us        2486126
> XATTROP
>       0.00       0.00 us       0.00 us       0.00 us        2433405
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us           2453
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us             69
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us           1544
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us        2430885
> CI_FORGET
>
>     Duration: 209524 seconds
>    Data Read: 2051388524 bytes
> Data Written: 2208261499 bytes
>
> Interval 0 Stats:
>    Block Size:                  1b+
> 2b+                   8b+
>  No. of Reads:                    0
> 0                     0
> No. of Writes:                   53
> 1                    60
>
>    Block Size:                 16b+                  32b+
> 64b+
>  No. of Reads:                    0
> 213                     2
> No. of Writes:                   14
> 1057                    15
>
>    Block Size:                128b+                 256b+
> 512b+
>  No. of Reads:                    0
> 41                   146
> No. of Writes:                   59
> 715                    80
>
>    Block Size:               1024b+                2048b+
> 4096b+
>  No. of Reads:                  224
> 9                    15
> No. of Writes:                  163
> 282                   404
>
>    Block Size:               8192b+               16384b+
> 32768b+
>  No. of Reads:                   52
> 74                   210
> No. of Writes:                 7415                  1430
> 6860
>
>    Block Size:              65536b+              131072b+
> 262144b+
>  No. of Reads:                  299
> 15313                     1
> No. of Writes:                 2458
> 12378                     1
>
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         686733
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us         837434
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          87338
> RELEASEDIR
>       0.00      22.69 us       7.16 us     332.61 us
> 116         IPC
>       0.00     119.93 us      70.56 us     272.28 us             57
> FTRUNCATE
>       0.00     152.25 us      86.82 us     340.63 us             65
> FGETXATTR
>       0.00    2336.06 us      90.05 us   20509.81 us             23
> REMOVEXATTR
>       0.00     631.26 us     123.17 us   51803.77 us            121
> LINK
>       0.00     369.27 us      22.64 us  145090.48 us
> 490          LK
>       0.00     360.85 us      45.14 us   46846.98 us            540
> STAT
>       0.00    1190.96 us     117.40 us   69781.85 us            200
> TRUNCATE
>       0.00     245.72 us      54.91 us  123985.76 us           1482
> SETATTR
>       0.00   42139.10 us    3313.56 us  312273.21 us             12
> FSYNC
>       0.00      55.83 us       8.26 us   49649.35 us          10047
> FLUSH
>       0.00      57.52 us      14.62 us     820.59 us           9977
> FINODELK
>       0.00    3191.60 us     143.06 us   96214.75 us            188
> RENAME
>       0.00   32862.86 us     215.38 us   83851.13 us             25
> SYMLINK
>       0.00   11392.03 us     230.70 us  105471.90 us            118
> MKNOD
>       0.00     134.63 us      28.08 us   36326.43 us          12587
> FSTAT
>       0.00     112.32 us      19.91 us  381241.34 us          21695
> STATFS
>       0.00     307.91 us      28.00 us  292666.83 us          16841
> READ
>       0.00   29625.60 us     239.57 us  239532.26 us            178
> MKDIR
>       0.00     319.22 us      51.19 us  136353.39 us          33445
> WRITE
>       0.01     175.83 us       1.57 us  513635.22 us          87338
> OPENDIR
>       0.01     249.95 us      58.42 us  594774.86 us          85089
> SETXATTR
>       0.01   11001.25 us     119.87 us  612765.03 us           3180
> RMDIR
>       0.04   14240.66 us     203.89 us 1376758.99 us           6067
> CREATE
>       0.05      89.46 us      30.14 us  687961.60 us        1219542
> READLINK
>       0.39    1114.30 us      44.29 us 2412169.89 us         831371
> OPEN
>       0.51  127078.45 us      78.55 us  694364.09 us           9493
> FXATTROP
>       0.63     399.16 us       8.43 us 3815689.36 us        3764714
> ENTRYLK
>       0.86   91973.00 us      19.89 us 24258016.33 us          22253
> READDIR
>       1.06    6913.26 us      11.58 us 36968532.72 us         364963
> GETXATTR
>       1.29   62121.95 us      36.89 us 8649871.11 us          49472
> READDIRP
>       2.89    1477.58 us      11.10 us 2807637.64 us        4650500
> INODELK
>       4.12   23681.29 us      20.98 us 23291414.50 us         414352
> UNLINK
>      16.09    5000.02 us       9.04 us 23866016.88 us        7655621
> LOOKUP
>      72.04   68956.14 us      68.40 us 27542499.65 us        2486126
> XATTROP
>       0.00       0.00 us       0.00 us       0.00 us        2433405
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us           2453
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us             69
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us           1544
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us        2430885
> CI_FORGET
>
>     Duration: 209524 seconds
>    Data Read: 2051388524 bytes
> Data Written: 2208261499 bytes
>
> Brick: flexo:/mnt/gluster/b
> --------------------------------------
> Cumulative Stats:
>    Block Size:                  1b+                   8b+
> 16b+
>  No. of Reads:                    0
> 0                     0
> No. of Writes:                  117
> 195                   367
>
>    Block Size:                 32b+                  64b+
> 128b+
>  No. of Reads:                  130
> 0                     3
> No. of Writes:                 1012
> 9                   134
>
>    Block Size:                256b+                 512b+
> 1024b+
>  No. of Reads:                   47
> 8                   257
> No. of Writes:                  729
> 73                   168
>
>    Block Size:               2048b+                4096b+
> 8192b+
>  No. of Reads:                   10
> 27                   240
> No. of Writes:                  294                   466
> 10472
>
>    Block Size:              16384b+               32768b+
> 65536b+
>  No. of Reads:                  139
> 119                   135
> No. of Writes:                 1480                  6273
> 3935
>
>    Block Size:             131072b+
>  No. of Reads:                 7892
> No. of Writes:                12327
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         955058
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us        1450749
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          81377
> RELEASEDIR
>       0.00      15.55 us       5.35 us      31.88 us
> 108         IPC
>       0.00     125.85 us      59.77 us     224.15 us             44
> FTRUNCATE
>       0.00     198.38 us      81.68 us    1332.45 us             30
> REMOVEXATTR
>       0.00    4254.16 us     621.52 us   10602.99 us              4
> FSYNC
>       0.00     795.08 us     190.30 us   16518.64 us             30
> SYMLINK
>       0.00     195.90 us     111.02 us     953.39 us            137
> LINK
>       0.00     286.07 us     179.64 us     437.82 us            119
> MKNOD
>       0.00     162.61 us      65.62 us    3864.77 us            221
> SETXATTR
>       0.00     117.29 us      48.50 us     317.17 us            354
> STAT
>       0.00      70.35 us      23.10 us     317.32 us
> 777          LK
>       0.00     461.72 us     114.54 us   16013.81 us            188
> RENAME
>       0.00     150.40 us      75.59 us     749.54 us            810
> FGETXATTR
>       0.00     267.80 us      99.67 us     994.58 us            698
> TRUNCATE
>       0.00    1176.00 us     194.89 us   78261.12 us            178
> MKDIR
>       0.00     439.83 us      54.38 us   75270.51 us           1665
> SETATTR
>       0.00      65.89 us      17.39 us     354.77 us          11372
> FINODELK
>       0.00     131.57 us      32.68 us    2107.52 us           7398
> FSTAT
>       0.00     120.71 us      31.43 us     847.97 us           9094
> READ
>       0.00      60.75 us       7.25 us   22520.98 us          22614
> FLUSH
>       0.00      73.03 us      22.41 us     750.17 us          21656
> STATFS
>       0.00     323.63 us     166.56 us   23034.44 us           6145
> CREATE
>       0.00     217.55 us      50.60 us   30734.99 us          38051
> WRITE
>       0.01     146.39 us       1.80 us  318612.45 us          81377
> OPENDIR
>       0.01    5641.37 us     108.84 us  468960.33 us           3177
> RMDIR
>       0.03     100.05 us      30.93 us   49756.93 us         432439
> READLINK
>       0.13     483.47 us      16.81 us 2608013.14 us         448381
> UNLINK
>       0.21    7101.56 us      22.26 us 2554390.49 us          52185
> READDIRP
>       0.35     414.05 us      36.58 us 1038229.93 us        1444606
> OPEN
>       0.49   78518.02 us      73.39 us  373999.31 us          10745
> FXATTROP
>       0.72     776.21 us      11.24 us 4271792.57 us        1605822
> GETXATTR
>       0.77   97829.38 us      18.84 us 9477446.11 us          13532
> READDIR
>       0.79     320.44 us       7.71 us 2369511.06 us        4257297
> ENTRYLK
>       1.97     464.02 us       9.38 us 1038460.80 us        7333722
> INODELK
>      13.73    2372.50 us       8.33 us 6258159.87 us        9979799
> LOOKUP
>      80.79   42372.04 us      61.49 us 2238308.13 us        3289103
> XATTROP
>       0.00       0.00 us       0.00 us       0.00 us        2250268
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us           3756
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us            167
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us           1908
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us        2246352
> CI_FORGET
>
>     Duration: 209334 seconds
>    Data Read: 1059777395 bytes
> Data Written: 2336921014 bytes
>
> Interval 0 Stats:
>    Block Size:                  1b+                   8b+
> 16b+
>  No. of Reads:                    0
> 0                     0
> No. of Writes:                  117
> 195                   367
>
>    Block Size:                 32b+                  64b+
> 128b+
>  No. of Reads:                  130
> 0                     3
> No. of Writes:                 1012
> 9                   134
>
>    Block Size:                256b+                 512b+
> 1024b+
>  No. of Reads:                   47
> 8                   257
> No. of Writes:                  729
> 73                   168
>
>    Block Size:               2048b+                4096b+
> 8192b+
>  No. of Reads:                   10
> 27                   240
> No. of Writes:                  294                   466
> 10472
>
>    Block Size:              16384b+               32768b+
> 65536b+
>  No. of Reads:                  139
> 119                   135
> No. of Writes:                 1480                  6273
> 3935
>
>    Block Size:             131072b+
>  No. of Reads:                 7892
> No. of Writes:                12327
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         955058
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us        1450749
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          81377
> RELEASEDIR
>       0.00      15.55 us       5.35 us      31.88 us
> 108         IPC
>       0.00     125.85 us      59.77 us     224.15 us             44
> FTRUNCATE
>       0.00     198.38 us      81.68 us    1332.45 us             30
> REMOVEXATTR
>       0.00    4254.16 us     621.52 us   10602.99 us              4
> FSYNC
>       0.00     795.08 us     190.30 us   16518.64 us             30
> SYMLINK
>       0.00     195.90 us     111.02 us     953.39 us            137
> LINK
>       0.00     286.07 us     179.64 us     437.82 us            119
> MKNOD
>       0.00     162.61 us      65.62 us    3864.77 us            221
> SETXATTR
>       0.00     117.29 us      48.50 us     317.17 us            354
> STAT
>       0.00      70.35 us      23.10 us     317.32 us
> 777          LK
>       0.00     461.72 us     114.54 us   16013.81 us            188
> RENAME
>       0.00     150.43 us      75.59 us     749.54 us            810
> FGETXATTR
>       0.00     267.80 us      99.67 us     994.58 us            698
> TRUNCATE
>       0.00    1176.00 us     194.89 us   78261.12 us            178
> MKDIR
>       0.00     439.83 us      54.38 us   75270.51 us           1665
> SETATTR
>       0.00      65.89 us      17.39 us     354.77 us          11372
> FINODELK
>       0.00     131.57 us      32.68 us    2107.52 us           7398
> FSTAT
>       0.00     120.74 us      31.43 us     847.97 us           9094
> READ
>       0.00      60.75 us       7.25 us   22520.98 us          22614
> FLUSH
>       0.00      73.03 us      22.41 us     750.17 us          21656
> STATFS
>       0.00     323.63 us     166.56 us   23034.44 us           6145
> CREATE
>       0.00     217.50 us      50.60 us   30734.99 us          38051
> WRITE
>       0.01     146.39 us       1.80 us  318612.45 us          81377
> OPENDIR
>       0.01    5641.37 us     108.84 us  468960.33 us           3177
> RMDIR
>       0.03     100.06 us      30.93 us   49756.93 us         432439
> READLINK
>       0.13     483.47 us      16.81 us 2608013.14 us         448381
> UNLINK
>       0.21    7101.56 us      22.26 us 2554390.49 us          52185
> READDIRP
>       0.35     414.01 us      36.58 us 1038229.93 us        1444606
> OPEN
>       0.49   78518.02 us      73.39 us  373999.31 us          10745
> FXATTROP
>       0.72     776.24 us      11.24 us 4271792.57 us        1605822
> GETXATTR
>       0.77   97829.38 us      18.84 us 9477446.11 us          13532
> READDIR
>       0.79     320.45 us       7.71 us 2369511.06 us        4257297
> ENTRYLK
>       1.97     464.07 us       9.38 us 1038460.80 us        7333722
> INODELK
>      13.72    2372.42 us       8.33 us 6258159.87 us        9979799
> LOOKUP
>      80.79   42373.50 us      61.49 us 2238308.13 us        3289103
> XATTROP
>       0.00       0.00 us       0.00 us       0.00 us        2250268
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us           3756
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us            167
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us           1908
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us        2246352
> CI_FORGET
>
>     Duration: 209334 seconds
>    Data Read: 1059777395 bytes
> Data Written: 2336921014 bytes
>
> Brick: flexo:/mnt/gluster/c
> --------------------------------------
> Cumulative Stats:
>    Block Size:                  1b+
>  No. of Reads:                    0
> No. of Writes:                32843
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         686562
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us         834596
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          93757
> RELEASEDIR
>       0.00      18.50 us       8.21 us      50.20 us
> 107         IPC
>       0.00     205.83 us     126.30 us     412.06 us             36
> LINK
>       0.00     167.33 us      75.01 us     272.47 us             65
> FGETXATTR
>       0.00     203.58 us     117.35 us     648.28 us             88
> RENAME
>       0.00     203.20 us      66.60 us     809.96 us            153
> TRUNCATE
>       0.00      67.06 us      23.64 us     555.59 us
> 491          LK
>       0.01      51.73 us       7.70 us    1174.64 us           9339
> FLUSH
>       0.01      62.41 us      15.13 us     636.35 us           9011
> FINODELK
>       0.03     126.23 us      43.15 us    4760.14 us          10824
> FSTAT
>       0.03     571.94 us      94.85 us   51697.03 us           2594
> MKDIR
>       0.05     264.87 us      81.58 us   53152.06 us           9009
> FXATTROP
>       0.05     423.05 us      38.26 us  548686.55 us           5754
> CREATE
>       0.05      78.14 us      19.17 us     678.32 us          32843
> WRITE
>       0.06     526.39 us     130.84 us   41405.46 us           5798
> SYMLINK
>       0.10     133.73 us      16.25 us  676494.72 us          34904
> READDIR
>       0.12    2054.48 us      99.28 us   88061.83 us           2732
> RMDIR
>       0.21     110.60 us       1.19 us   16833.78 us          93757
> OPENDIR
>       0.72     205.89 us      33.28 us  261023.99 us         170292
> UNLINK
>       0.85    2971.45 us      21.24 us  257839.19 us          13840
> READDIRP
>       1.94     200.88 us      25.49 us  461654.32 us         468842
> FTRUNCATE
>       2.13     212.44 us      54.02 us  461878.34 us         486882
> REMOVEXATTR
>       2.27     223.01 us      52.03 us  178897.40 us         493240
> SETXATTR
>       3.53     155.31 us       7.96 us   67873.00 us        1101338
> GETXATTR
>       3.63     212.32 us      44.42 us  551264.02 us         829002
> OPEN
>       5.77      76.36 us       7.76 us  962634.73 us        3660425
> ENTRYLK
>       6.23     304.93 us      42.51 us  661394.43 us         990644
> SETATTR
>       7.81     382.20 us     107.42 us  846502.78 us         990766
> XATTROP
>      12.39     549.11 us     135.57 us 1659697.83 us        1093387
> MKNOD
>      12.47     139.78 us       9.76 us  552067.56 us        4325281
> INODELK
>      39.53     206.94 us       8.56 us  964823.33 us        9259429
> LOOKUP
>       0.00       0.00 us       0.00 us       0.00 us        1856377
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us         933300
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us            251
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us            293
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us         923077
> CI_FORGET
>
>     Duration: 209334 seconds
>    Data Read: 0 bytes
> Data Written: 32843 bytes
>
> Interval 0 Stats:
>    Block Size:                  1b+
>  No. of Reads:                    0
> No. of Writes:                32843
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         686562
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us         834596
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          93757
> RELEASEDIR
>       0.00      18.50 us       8.21 us      50.20 us
> 107         IPC
>       0.00     205.83 us     126.30 us     412.06 us             36
> LINK
>       0.00     167.33 us      75.01 us     272.47 us             65
> FGETXATTR
>       0.00     203.58 us     117.35 us     648.28 us             88
> RENAME
>       0.00     203.20 us      66.60 us     809.96 us            153
> TRUNCATE
>       0.00      67.06 us      23.64 us     555.59 us
> 491          LK
>       0.01      51.71 us       7.70 us    1174.64 us           9339
> FLUSH
>       0.01      62.41 us      15.13 us     636.35 us           9011
> FINODELK
>       0.03     126.23 us      43.15 us    4760.14 us          10824
> FSTAT
>       0.03     571.94 us      94.85 us   51697.03 us           2594
> MKDIR
>       0.05     264.87 us      81.58 us   53152.06 us           9009
> FXATTROP
>       0.05     423.05 us      38.26 us  548686.55 us           5754
> CREATE
>       0.05      78.14 us      19.17 us     678.32 us          32843
> WRITE
>       0.06     526.39 us     130.84 us   41405.46 us           5798
> SYMLINK
>       0.10     133.73 us      16.25 us  676494.72 us          34904
> READDIR
>       0.12    2054.48 us      99.28 us   88061.83 us           2732
> RMDIR
>       0.21     110.60 us       1.19 us   16833.78 us          93757
> OPENDIR
>       0.72     205.89 us      33.28 us  261023.99 us         170292
> UNLINK
>       0.85    2971.45 us      21.24 us  257839.19 us          13840
> READDIRP
>       1.94     200.87 us      25.49 us  461654.32 us         468842
> FTRUNCATE
>       2.13     212.44 us      54.02 us  461878.34 us         486882
> REMOVEXATTR
>       2.27     223.00 us      52.03 us  178897.40 us         493240
> SETXATTR
>       3.53     155.32 us       7.96 us   67873.00 us        1101338
> GETXATTR
>       3.63     212.32 us      44.42 us  551264.02 us         829002
> OPEN
>       5.77      76.36 us       7.76 us  962634.73 us        3660425
> ENTRYLK
>       6.23     304.88 us      42.51 us  661394.43 us         990644
> SETATTR
>       7.81     382.20 us     107.42 us  846502.78 us         990766
> XATTROP
>      12.39     549.25 us     135.57 us 1659697.83 us        1093387
> MKNOD
>      12.47     139.78 us       9.76 us  552067.56 us        4325281
> INODELK
>      39.53     206.95 us       8.56 us  964823.33 us        9259429
> LOOKUP
>       0.00       0.00 us       0.00 us       0.00 us        1856377
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us         933300
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us            251
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us            293
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us         923077
> CI_FORGET
>
>     Duration: 209334 seconds
>    Data Read: 0 bytes
> Data Written: 32843 bytes
>
> Brick: bender:/mnt/gluster/c
> ---------------------------------------
> Cumulative Stats:
>    Block Size:                  1b+
>  No. of Reads:                    0
> No. of Writes:                37050
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         928119
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us        1401524
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          70092
> RELEASEDIR
>       0.00      19.49 us       8.28 us     180.61 us
> 100         IPC
>       0.00     238.88 us     163.97 us     486.93 us             38
> LINK
>       0.00     306.43 us     196.67 us     788.36 us             30
> SYMLINK
>       0.00    2670.18 us     765.93 us    4151.64 us              4
> FSYNC
>       0.00     350.65 us     141.92 us   10573.64 us            122
> RENAME
>       0.00      67.02 us      20.66 us     453.90 us
> 777          LK
>       0.00     158.32 us      64.22 us     532.21 us            810
> FGETXATTR
>       0.00     229.87 us      66.47 us     826.31 us            603
> TRUNCATE
>       0.01    1742.85 us     202.81 us   47471.71 us            302
> MKDIR
>       0.01      59.05 us      15.40 us    1813.56 us          10060
> FINODELK
>       0.01     118.54 us      36.65 us    4002.58 us           6409
> FSTAT
>       0.01      49.38 us       8.08 us    1264.41 us          21719
> FLUSH
>       0.03      74.76 us      20.86 us    2008.41 us          37050
> WRITE
>       0.04     553.12 us      40.81 us  386485.29 us           5901
> CREATE
>       0.06     534.01 us     100.86 us  527770.44 us          10057
> FXATTROP
>       0.12     138.96 us       1.00 us  601208.70 us          70092
> OPENDIR
>       0.12     947.76 us      18.54 us  712568.33 us          10779
> READDIR
>       0.15    4099.07 us     124.69 us  421939.47 us           3008
> RMDIR
>       0.61     465.30 us      20.67 us  936959.86 us         109478
> UNLINK
>       1.57     182.41 us      49.54 us   73938.09 us         722669
> REMOVEXATTR
>       1.61     171.41 us       6.98 us   83809.88 us         789294
> GETXATTR
>       1.86     217.88 us      26.40 us  556446.19 us         718835
> FTRUNCATE
>       1.89     219.47 us      49.92 us  506767.78 us         722856
> SETXATTR
>       3.02     181.66 us      44.40 us  344656.69 us        1395957
> OPEN
>       5.56     322.08 us      43.37 us  634069.61 us        1450160
> SETATTR
>       5.84     115.88 us       7.67 us  747387.51 us        4233520
> ENTRYLK
>       6.53     378.33 us     122.84 us  766939.14 us        1449601
> XATTROP
>       9.75     115.48 us      10.70 us  556675.13 us        7097226
> INODELK
>      19.37    1104.98 us     101.28 us 1612346.75 us        1472813
> MKNOD
>      41.85     295.61 us       8.98 us  996805.54 us       11896091
> LOOKUP
>       0.00       0.00 us       0.00 us       0.00 us        3531139
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us        1862470
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us            244
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us            632
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us        1668669
> CI_FORGET
>
>     Duration: 208643 seconds
>    Data Read: 0 bytes
> Data Written: 37050 bytes
>
> Interval 0 Stats:
>    Block Size:                  1b+
>  No. of Reads:                    0
> No. of Writes:                37050
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         928119
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us        1401524
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          70092
> RELEASEDIR
>       0.00      19.49 us       8.28 us     180.61 us
> 100         IPC
>       0.00     238.88 us     163.97 us     486.93 us             38
> LINK
>       0.00     306.43 us     196.67 us     788.36 us             30
> SYMLINK
>       0.00    2670.18 us     765.93 us    4151.64 us              4
> FSYNC
>       0.00     350.65 us     141.92 us   10573.64 us            122
> RENAME
>       0.00      67.02 us      20.66 us     453.90 us
> 777          LK
>       0.00     158.32 us      64.22 us     532.21 us            810
> FGETXATTR
>       0.00     229.87 us      66.47 us     826.31 us            603
> TRUNCATE
>       0.01    1742.85 us     202.81 us   47471.71 us            302
> MKDIR
>       0.01      59.05 us      15.40 us    1813.56 us          10060
> FINODELK
>       0.01     118.54 us      36.65 us    4002.58 us           6409
> FSTAT
>       0.01      49.39 us       8.08 us    1264.41 us          21719
> FLUSH
>       0.03      74.77 us      20.86 us    2008.41 us          37050
> WRITE
>       0.04     553.12 us      40.81 us  386485.29 us           5901
> CREATE
>       0.06     534.01 us     100.86 us  527770.44 us          10057
> FXATTROP
>       0.12     138.96 us       1.00 us  601208.70 us          70092
> OPENDIR
>       0.12     947.76 us      18.54 us  712568.33 us          10779
> READDIR
>       0.15    4099.07 us     124.69 us  421939.47 us           3008
> RMDIR
>       0.61     465.30 us      20.67 us  936959.86 us         109478
> UNLINK
>       1.57     182.41 us      49.54 us   73938.09 us         722669
> REMOVEXATTR
>       1.61     171.41 us       6.98 us   83809.88 us         789294
> GETXATTR
>       1.86     217.88 us      26.40 us  556446.19 us         718835
> FTRUNCATE
>       1.89     219.48 us      49.92 us  506767.78 us         722856
> SETXATTR
>       3.02     181.66 us      44.40 us  344656.69 us        1395957
> OPEN
>       5.56     322.09 us      43.37 us  634069.61 us        1450160
> SETATTR
>       5.84     115.88 us       7.67 us  747387.51 us        4233520
> ENTRYLK
>       6.53     378.33 us     122.84 us  766939.14 us        1449601
> XATTROP
>       9.75     115.48 us      10.70 us  556675.13 us        7097226
> INODELK
>      19.37    1105.06 us     101.28 us 1612346.75 us        1472813
> MKNOD
>      41.84     295.54 us       8.98 us  996805.54 us       11896091
> LOOKUP
>       0.00       0.00 us       0.00 us       0.00 us        3531139
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us        1862470
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us            244
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us            632
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us        1668669
> CI_FORGET
>
>     Duration: 208643 seconds
>    Data Read: 0 bytes
> Data Written: 37050 bytes
>
> Brick: bender:/mnt/gluster/a
> ---------------------------------------
> Cumulative Stats:
>    Block Size:                  1b+
> 2b+                   8b+
>  No. of Reads:                    0
> 0                     0
> No. of Writes:                   53
> 1                    60
>
>    Block Size:                 16b+                  32b+
> 64b+
>  No. of Reads:                    0
> 235                     4
> No. of Writes:                   14
> 1033                    15
>
>    Block Size:                128b+                 256b+
> 512b+
>  No. of Reads:                    4
> 115                    81
> No. of Writes:                   58
> 703                    80
>
>    Block Size:               1024b+                2048b+
> 4096b+
>  No. of Reads:                  181
> 26                    74
> No. of Writes:                  163
> 278                   397
>
>    Block Size:               8192b+               16384b+
> 32768b+
>  No. of Reads:                  131
> 161                   267
> No. of Writes:                 7264                  1398
> 6742
>
>    Block Size:              65536b+              131072b+
> 262144b+
>  No. of Reads:                  443
> 25528                     0
> No. of Writes:                 2426
> 12240                     1
>
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         684942
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us         837092
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          87297
> RELEASEDIR
>       0.00     219.22 us     121.08 us     306.12 us              8
> REMOVEXATTR
>       0.00      21.70 us       7.24 us     196.07 us
> 97         IPC
>       0.00     152.77 us      78.90 us    1059.82 us             46
> FTRUNCATE
>       0.00     295.73 us     167.10 us     606.96 us             65
> FGETXATTR
>       0.00     243.66 us     140.50 us    3392.16 us            121
> LINK
>       0.00    1216.47 us     194.37 us   23411.42 us             25
> SYMLINK
>       0.00     327.87 us     234.27 us     551.71 us            107
> MKNOD
>       0.00   36094.61 us   36094.61 us   36094.61 us              1
> FSYNC
>       0.00     219.63 us     147.34 us     783.10 us            188
> RENAME
>       0.00     104.27 us      40.34 us     595.29 us            745
> STAT
>       0.00     531.71 us     113.57 us   53973.08 us            200
> TRUNCATE
>       0.00     289.05 us      63.10 us   70107.41 us           1444
> SETATTR
>       0.00    2430.15 us     232.78 us   68120.54 us            177
> MKDIR
>       0.00      52.68 us       7.28 us     826.01 us           9890
> FLUSH
>       0.00      60.96 us      17.54 us    1393.95 us           9792
> FINODELK
>       0.00      74.26 us      20.00 us    1522.39 us          21521
> STATFS
>       0.00     129.21 us      34.15 us    9068.67 us          13094
> FSTAT
>       0.00     456.99 us     195.50 us  130182.02 us           5967
> CREATE
>       0.00     112.06 us      28.14 us    9545.53 us          27474
> READ
>       0.01     313.76 us      52.98 us   71635.74 us          32926
> WRITE
>       0.01     135.84 us       1.52 us  671373.08 us          87297
> OPENDIR
>       0.01    5225.28 us     131.28 us  951323.69 us           3177
> RMDIR
>       0.01     203.34 us      56.13 us  486756.90 us          85073
> SETXATTR
>       0.03    3521.00 us     145.28 us 3396306.00 us          15090
> READDIRP
>       0.04  113758.50 us      26.72 us 1147074.44 us
> 564          LK
>       0.05      92.18 us      29.71 us   39069.99 us         850938
> READLINK
>       0.23     453.79 us      41.84 us  655999.23 us         831212
> OPEN
>       0.41   29425.34 us      19.20 us 1526187.08 us          22482
> READDIR
>       0.50    1954.70 us      21.51 us 1526772.28 us         415226
> UNLINK
>       0.54    1272.83 us       9.66 us 1604368.93 us         686945
> GETXATTR
>       0.55     237.57 us       7.91 us 1277047.91 us        3739718
> ENTRYLK
>       0.60  103888.49 us      89.32 us  761509.35 us           9311
> FXATTROP
>       2.60     909.04 us       9.68 us 2272778.49 us        4649291
> INODELK
>       5.82    1240.20 us       8.78 us 1526868.50 us        7625383
> LOOKUP
>      88.60   58005.98 us      67.31 us 1448810.06 us        2481914
> XATTROP
>       0.00       0.00 us       0.00 us       0.00 us        1396875
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us           1854
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us             28
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us           1051
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us        1394996
> CI_FORGET
>
>     Duration: 208643 seconds
>    Data Read: 3409342848 bytes
> Data Written: 2180686507 bytes
>
> Interval 0 Stats:
>    Block Size:                  1b+
> 2b+                   8b+
>  No. of Reads:                    0
> 0                     0
> No. of Writes:                   53
> 1                    60
>
>    Block Size:                 16b+                  32b+
> 64b+
>  No. of Reads:                    0
> 235                     4
> No. of Writes:                   14
> 1033                    15
>
>    Block Size:                128b+                 256b+
> 512b+
>  No. of Reads:                    4
> 115                    81
> No. of Writes:                   58
> 703                    80
>
>    Block Size:               1024b+                2048b+
> 4096b+
>  No. of Reads:                  181
> 26                    74
> No. of Writes:                  163
> 278                   397
>
>    Block Size:               8192b+               16384b+
> 32768b+
>  No. of Reads:                  131
> 161                   267
> No. of Writes:                 7264                  1398
> 6742
>
>    Block Size:              65536b+              131072b+
> 262144b+
>  No. of Reads:                  443
> 25528                     0
> No. of Writes:                 2426
> 12240                     1
>
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls         Fop
>  ---------   -----------   -----------   -----------   ------------
> ----
>       0.00       0.00 us       0.00 us       0.00 us         684942
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us         837092
> RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          87297
> RELEASEDIR
>       0.00     219.22 us     121.08 us     306.12 us              8
> REMOVEXATTR
>       0.00      21.70 us       7.24 us     196.07 us
> 97         IPC
>       0.00     152.77 us      78.90 us    1059.82 us             46
> FTRUNCATE
>       0.00     295.73 us     167.10 us     606.96 us             65
> FGETXATTR
>       0.00     243.66 us     140.50 us    3392.16 us            121
> LINK
>       0.00    1216.47 us     194.37 us   23411.42 us             25
> SYMLINK
>       0.00     327.87 us     234.27 us     551.71 us            107
> MKNOD
>       0.00   36094.61 us   36094.61 us   36094.61 us              1
> FSYNC
>       0.00     219.63 us     147.34 us     783.10 us            188
> RENAME
>       0.00     104.27 us      40.34 us     595.29 us            745
> STAT
>       0.00     531.71 us     113.57 us   53973.08 us            200
> TRUNCATE
>       0.00     289.05 us      63.10 us   70107.41 us           1444
> SETATTR
>       0.00    2430.15 us     232.78 us   68120.54 us            177
> MKDIR
>       0.00      52.69 us       7.28 us     826.01 us           9890
> FLUSH
>       0.00      60.96 us      17.54 us    1393.95 us           9792
> FINODELK
>       0.00      74.25 us      20.00 us    1522.39 us          21521
> STATFS
>       0.00     129.23 us      34.15 us    9068.67 us          13094
> FSTAT
>       0.00     456.99 us     195.50 us  130182.02 us           5967
> CREATE
>       0.00     112.45 us      28.14 us    9545.53 us          27474
> READ
>       0.01     313.77 us      52.98 us   71635.74 us          32926
> WRITE
>       0.01     135.84 us       1.52 us  671373.08 us          87297
> OPENDIR
>       0.01    5225.28 us     131.28 us  951323.69 us           3177
> RMDIR
>       0.01     203.34 us      56.13 us  486756.90 us          85073
> SETXATTR
>       0.03    3521.00 us     145.28 us 3396306.00 us          15090
> READDIRP
>       0.04  113758.50 us      26.72 us 1147074.44 us
> 564          LK
>       0.05      92.18 us      29.71 us   39069.99 us         850938
> READLINK
>       0.23     453.79 us      41.84 us  655999.23 us         831212
> OPEN
>       0.41   29425.34 us      19.20 us 1526187.08 us          22482
> READDIR
>       0.50    1954.70 us      21.51 us 1526772.28 us         415226
> UNLINK
>       0.54    1272.79 us       9.66 us 1604368.93 us         686945
> GETXATTR
>       0.55     237.57 us       7.91 us 1277047.91 us        3739718
> ENTRYLK
>       0.60  103888.49 us      89.32 us  761509.35 us           9311
> FXATTROP
>       2.60     909.04 us       9.68 us 2272778.49 us        4649291
> INODELK
>       5.82    1240.10 us       8.78 us 1526868.50 us        7625383
> LOOKUP
>      88.60   58006.02 us      67.31 us 1448810.06 us        2481914
> XATTROP
>       0.00       0.00 us       0.00 us       0.00 us        1396875
> UPCALL
>       0.00       0.00 us       0.00 us       0.00 us           1854
> CI_IATT
>       0.00       0.00 us       0.00 us       0.00 us             28
> CI_XATTR
>       0.00       0.00 us       0.00 us       0.00 us           1051
> CI_UNLINK
>       0.00       0.00 us       0.00 us       0.00 us        1394996
> CI_FORGET
>
>     Duration: 208643 seconds
>    Data Read: 3409342848 bytes
> Data Written: 2180686507 bytes
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200504/76239ebb/attachment.html>

Jaco Kroon

2020-May-05 06:43 UTC

head link

[Gluster-users] Possible SHD stalling

Hi Artem,

I'll do that.? In the meantime consider this (inferred from the changelogs).

It seems like glusterfs SHD is queueing into a bounded queue under lock
conditions, and waiting for space to open up.? At least, this is what's
hinted in the changelogs, I've been trying to grasp the SHD code for two
days now and am not getting to wrap my head around that.? If this is
true, then the only way to avoid lock-ups are to ensure that the queue
is larger than the total number of files that can ever require healing
(effectively making the queue unbounded).

Kind Regards,
Jaco

On 2020/05/04 21:56, Artem Russakovskii wrote:> I reported something similar recently to this list, and I highly
> advise you to update to 7.5 (in my case it was 5.11 -> 5.13), then see
> if it keeps happening. The self heal issues should be fixed there
> properly at least.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>,?APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net <http://beerpla.net/> | @ArtemR
<http://twitter.com/ArtemR>
>
>
> On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za
> <mailto:jaco at uls.co.za>> wrote:
>
>     Hi All,
>
>     So we've got a cluster currently running on glusterfs 7.3
>
>     Heal performance is absolutely terrible (it starts of great, runs
>     for a while then completely trips up).
>
>     Once the SHD trips up, access speeds to the cluster also becomes
>     poor.? And with poor I mean directory listings that normally takes
>     well under 500ms takes upwards of 10s, but I've seen upwards of
>     40s (although a follow up ls immediately after is nice and fast
>     again, temporarily).? With all SHDs killed this particular ls is
>     as per normal (around 30-50ms).
>
>     The SHD logs doesn't provide any usable clues, the only variance
>     in logs is this, just before all daemons across the cluster crawls
>     to a near halt.
>
>     [2020-05-02 19:55:58.627036] I [MSGID: 108026]
>     [afr-self-heal-common.c:1747:afr_log_selfheal]
>     0-vpbx_shared-replicate-0: Completed entry selfheal on
>     5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1 ?
>     [2020-05-02 19:55:58.627058] I [MSGID: 108026]
>     [afr-self-heal-common.c:1747:afr_log_selfheal]
>     0-vpbx_shared-replicate-0: Completed entry selfheal on
>     c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1 ?
>     ?
>     [2020-05-02 19:55:59.125001] W [MSGID: 114031]
>     [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
>     0-vpbx_shared-client-1: remote operation failed [Invalid argument]
>     ?
>     [2020-05-02 19:55:59.339803] I [MSGID: 108026]
>     [afr-self-heal-common.c:1747:afr_log_selfheal]
>     0-vpbx_shared-replicate-0: Completed entry selfheal on
>     2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1?
>     [2020-05-02 19:55:59.490387] I [MSGID: 108026]
>     [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
>     0-vpbx_shared-replicate-0: performing metadata selfheal on
>     527a63a3-9acd-4c3f-b95c-5c719325c973
>
>     A few more entries will complete overall and then it'll be a file
>     every other second.? Once I saw this I ran a few more tests, and
>     on every test just prior to slowing to a crawl, that entry is there.
>
>     There is a perceived improvement when lowering the thread count,
>     but not significant enough to confirm a correlation.? Even
>     with**cluster.shd-max-threads=1 this still happens (gets a few
>     tens thousand files completed instead of few hundred, at 32 it
>     gets to maybe 400 to 500 before it stalls, 16 does maybe a thousand).*
>     *
>
>     There are probably performant bursts hereafter, because from
>     experience we know this process takes about two months.? But this
>     time around it seems to be much, much worse.? Might be because
>     that was healing a single brick to another brick, so less locking
>     going?? At the moment the replicas are basically trying to
>     populate the newly added arbiters.? Also, we haven't messed as
>     greatly with the health check settings (queues and threads and
>     locks nearly as much as since the arbiter addition because
>     performance has simply dropped to non-acceptable levels.
>
>     Unfortunately granular entry heal isn't on at the moment (nor can
>     I switch it on due to a few million outstanding heals).? This used
>     to be a 2x2 cluster.? We then moved one of the bricks to a newly
>     added server using replace-brick.? What a frightening process this
>     is.? We then proceeded to add the arbiters.
>
>     Some of the settings below was absolute shots in the dark, based
>     on the little information we've been able to track down so far.
>
>     Kind Regards,
>     Jaco
>
>     The current gluster volume info (this used to be 2x2
>     distributed-replicate, aiming to get it to 3x (2 + 1) ... at
>     current rates we might get there by 2021):
>
>     Volume Name: glvol
>     Type: Distributed-Replicate
>     Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb
>     Status: Started
>     Snapshot Count: 0
>     Number of Bricks: 2 x (2 + 1) = 6
>     Transport-type: tcp
>     Bricks:
>     Brick1: zoidberg:/mnt/gluster/a
>     Brick2: flexo:/mnt/gluster/b
>     Brick3: bender:/mnt/gluster/c (arbiter)
>     Brick4: bender:/mnt/gluster/a
>     Brick5: zoidberg:/mnt/gluster/b
>     Brick6: flexo:/mnt/gluster/c (arbiter)
>     Options Reconfigured:
>     *cluster.heal-timeout: 5 <-- hoping this would at least shorten
>     the blockages.*
>     cluster.locking-scheme: granular
>     cluster.heal-wait-queue-length: 10240
>     *diagnostics.count-fop-hits: on**
>     **diagnostics.latency-measurement: on <-- was hoping profiling
>     might provide some insight.? Going to switch this off now.*
>     cluster.background-self-heal-count: 128
>     *cluster.shd-wait-qlength: 131072**
>     **cluster.shd-max-threads: 32? <-- even with this disk IO never
>     goes over 10% utilization.? CPU goes to about 2 cores.? So it's
>     lock contention that's slowing it down we reckon.*
>     *cluster.metadata-self-heal: off**
>     **cluster.entry-self-heal: off**
>     **cluster.data-self-heal: off**<-- these kill performance.
>     *transport.address-family: inet
>     nfs.disable: on
>     performance.client-io-threads: on
>     features.cache-invalidation: on
>     features.cache-invalidation-timeout: 600
>     performance.stat-prefetch: on
>     performance.cache-invalidation: on
>     performance.md-cache-timeout: 600
>     *cluster.self-heal-daemon: yes <-- disabling/enabling this
doesn't
>     help, have to kill the shd and start ... force to get things
>     moving along again.*
>     performance.io-thread-count: 16
>     performance.high-prio-threads: 16
>     performance.normal-prio-threads: 16
>     performance.low-prio-threads: 16
>     performance.least-prio-threads: 4
>     cluster.self-heal-window-size: 512
>     cluster.self-heal-readdir-size: 131072
>     client.event-threads: 8
>     server.event-threads: 16
>     network.ping-timeout: 2
>     *cluster.data-self-heal-algorithm: full <-- most files are small
>     (<1MB), on dual 1gbit links (so up to 2gbit, but b/w never goes
>     over 5mbit anyway).*
>
>     performance info:
>
>     Brick: zoidberg:/mnt/gluster/a
>     -----------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     8b+????????????????? 16b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:????????????????? 117??????????????????
>     195?????????????????? 367
>     ?
>     ?? Block Size:???????????????? 32b+?????????????????
>     64b+???????????????? 128b+
>     ?No. of Reads:????????????????? 306????????????????????
>     0???????????????????? 6
>     No. of Writes:???????????????? 1016????????????????????
>     9?????????????????? 134
>     ?
>     ?? Block Size:??????????????? 256b+????????????????
>     512b+??????????????? 1024b+
>     ?No. of Reads:????????????????? 107???????????????????
>     58?????????????????? 485
>     No. of Writes:????????????????? 731???????????????????
>     73?????????????????? 168
>     ?
>     ?? Block Size:?????????????? 2048b+???????????????
>     4096b+??????????????? 8192b+
>     ?No. of Reads:?????????????????? 43???????????????????
>     59?????????????????? 383
>     No. of Writes:????????????????? 295??????????????????
>     475???????????????? 10562
>     ?
>     ?? Block Size:????????????? 16384b+??????????????
>     32768b+?????????????? 65536b+
>     ?No. of Reads:????????????????? 347??????????????????
>     380?????????????????? 634
>     No. of Writes:???????????????? 1487?????????????????
>     6369????????????????? 3970
>     ?
>     ?? Block Size:???????????? 131072b+
>     ?No. of Reads:??????????????? 35704
>     No. of Writes:??????????????? 12398
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     957293????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1455091???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     69951? RELEASEDIR
>     ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us???????????
>     113???????? IPC
>     ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 us????????????
>     30 REMOVEXATTR
>     ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 us????????????
>     52?? FTRUNCATE
>     ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us??????????
>     1385??????? STAT
>     ????? 0.00???? 364.93 us????? 63.45 us?? 15957.66 us???????????
>     810?? FGETXATTR
>     ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us???????????
>     137??????? LINK
>     ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 us????????????
>     12?????? FSYNC
>     ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us???????????
>     698??? TRUNCATE
>     ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us???????????
>     188????? RENAME
>     ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us??????????
>     1680???? SETATTR
>     ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us???????????
>     127?????? MKNOD
>     ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us?????????
>     11412??? FINODELK
>     ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us???????????
>     221??? SETXATTR
>     ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 us????????????
>     30???? SYMLINK
>     ????? 0.00????? 56.18 us?????? 7.31 us?? 41484.37 us?????????
>     22640?????? FLUSH
>     ????? 0.00???? 105.64 us????? 19.34 us? 351055.72 us?????????
>     21693????? STATFS
>     ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us??????????
>     9653?????? FSTAT
>     ????? 0.00???? 214.87 us????? 25.71 us 1712020.99 us?????????
>     39054??????? READ
>     ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us?????????
>     69951???? OPENDIR
>     ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us??????????
>     3180?????? RMDIR
>     ????? 0.01???? 327.08 us????? 55.69 us? 420376.80 us?????????
>     38366?????? WRITE
>     ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us??????????
>     6169????? CREATE
>     ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us???????
>     1159936??? READLINK
>     ????? 0.40???? 543.50 us????? 36.45 us 2430653.54 us???????
>     1449307??????? OPEN
>     ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us?????????
>     10787??? FXATTROP
>     ????? 0.53??? 2342.72 us????? 20.91 us 4157542.43 us????????
>     447821????? UNLINK
>     ????? 0.53? 558734.79 us????? 24.40 us 1399487.63 us??????????
>     1889????????? LK
>     ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us?????????
>     10687???? READDIR
>     ????? 0.99??? 2426.23 us????? 12.47 us 24176123.13 us????????
>     809430??? GETXATTR
>     ????? 1.45???? 676.83 us?????? 9.30 us 193177067.25 us???????
>     4260021???? ENTRYLK
>     ????? 1.76?? 38325.38 us????? 21.44 us 46693575.93 us?????????
>     91439??? READDIRP
>     ????? 4.44??? 1200.31 us?????? 9.10 us 2559377.90 us???????
>     7352811???? INODELK
>     ????? 9.30??? 1849.24 us?????? 9.28 us 35299050.00 us???????
>     9989221????? LOOKUP
>     ???? 79.24?? 47791.46 us????? 76.77 us 10488824.26 us???????
>     3294642???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3194806????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     5311???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     54??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1907?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3189457?? CI_FORGET
>     ?
>     ??? Duration: 209524 seconds
>     ?? Data Read: 4775365531 bytes
>     Data Written: 2354108664 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     8b+????????????????? 16b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:????????????????? 117??????????????????
>     195?????????????????? 367
>     ?
>     ?? Block Size:???????????????? 32b+?????????????????
>     64b+???????????????? 128b+
>     ?No. of Reads:????????????????? 306????????????????????
>     0???????????????????? 6
>     No. of Writes:???????????????? 1016????????????????????
>     9?????????????????? 134
>     ?
>     ?? Block Size:??????????????? 256b+????????????????
>     512b+??????????????? 1024b+
>     ?No. of Reads:????????????????? 107???????????????????
>     58?????????????????? 485
>     No. of Writes:????????????????? 731???????????????????
>     73?????????????????? 168
>     ?
>     ?? Block Size:?????????????? 2048b+???????????????
>     4096b+??????????????? 8192b+
>     ?No. of Reads:?????????????????? 43???????????????????
>     59?????????????????? 383
>     No. of Writes:????????????????? 295??????????????????
>     475???????????????? 10562
>     ?
>     ?? Block Size:????????????? 16384b+??????????????
>     32768b+?????????????? 65536b+
>     ?No. of Reads:????????????????? 347??????????????????
>     380?????????????????? 634
>     No. of Writes:???????????????? 1487?????????????????
>     6369????????????????? 3970
>     ?
>     ?? Block Size:???????????? 131072b+
>     ?No. of Reads:??????????????? 35704
>     No. of Writes:??????????????? 12398
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     957293????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1455091???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     69951? RELEASEDIR
>     ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us???????????
>     113???????? IPC
>     ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 us????????????
>     30 REMOVEXATTR
>     ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 us????????????
>     52?? FTRUNCATE
>     ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us??????????
>     1385??????? STAT
>     ????? 0.00???? 363.20 us????? 63.45 us?? 15957.66 us???????????
>     810?? FGETXATTR
>     ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us???????????
>     137??????? LINK
>     ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 us????????????
>     12?????? FSYNC
>     ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us???????????
>     698??? TRUNCATE
>     ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us???????????
>     188????? RENAME
>     ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us??????????
>     1680???? SETATTR
>     ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us???????????
>     127?????? MKNOD
>     ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us?????????
>     11412??? FINODELK
>     ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us???????????
>     221??? SETXATTR
>     ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 us????????????
>     30???? SYMLINK
>     ????? 0.00????? 56.19 us?????? 7.31 us?? 41484.37 us?????????
>     22640?????? FLUSH
>     ????? 0.00???? 105.61 us????? 19.34 us? 351055.72 us?????????
>     21693????? STATFS
>     ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us??????????
>     9653?????? FSTAT
>     ????? 0.00???? 214.91 us????? 25.71 us 1712020.99 us?????????
>     39054??????? READ
>     ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us?????????
>     69951???? OPENDIR
>     ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us??????????
>     3180?????? RMDIR
>     ????? 0.01???? 327.13 us????? 55.69 us? 420376.80 us?????????
>     38366?????? WRITE
>     ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us??????????
>     6169????? CREATE
>     ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us???????
>     1159936??? READLINK
>     ????? 0.40???? 543.49 us????? 36.45 us 2430653.54 us???????
>     1449307??????? OPEN
>     ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us?????????
>     10787??? FXATTROP
>     ????? 0.53??? 2342.57 us????? 20.91 us 4157542.43 us????????
>     447821????? UNLINK
>     ????? 0.53? 558416.10 us????? 24.40 us 1399487.63 us??????????
>     1889????????? LK
>     ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us?????????
>     10687???? READDIR
>     ????? 0.99??? 2426.24 us????? 12.47 us 24176123.13 us????????
>     809430??? GETXATTR
>     ????? 1.45???? 676.75 us?????? 9.30 us 193177067.25 us???????
>     4260021???? ENTRYLK
>     ????? 1.76?? 38324.59 us????? 21.44 us 46693575.93 us?????????
>     91439??? READDIRP
>     ????? 4.44??? 1200.37 us?????? 9.10 us 2559377.90 us???????
>     7352811???? INODELK
>     ????? 9.29??? 1848.94 us?????? 9.28 us 35299050.00 us???????
>     9989221????? LOOKUP
>     ???? 79.24?? 47792.42 us????? 76.77 us 10488824.26 us???????
>     3294642???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3194806????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     5311???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     54??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1907?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3189457?? CI_FORGET
>     ?
>     ??? Duration: 209524 seconds
>     ?? Data Read: 4775365531 bytes
>     Data Written: 2354108664 bytes
>     ?
>     Brick: zoidberg:/mnt/gluster/b
>     -----------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     2b+?????????????????? 8b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:?????????????????? 53????????????????????
>     1??????????????????? 60
>     ?
>     ?? Block Size:???????????????? 16b+?????????????????
>     32b+????????????????? 64b+
>     ?No. of Reads:??????????????????? 0??????????????????
>     213???????????????????? 2
>     No. of Writes:?????????????????? 14?????????????????
>     1057??????????????????? 15
>     ?
>     ?? Block Size:??????????????? 128b+????????????????
>     256b+???????????????? 512b+
>     ?No. of Reads:??????????????????? 0???????????????????
>     41?????????????????? 146
>     No. of Writes:?????????????????? 59??????????????????
>     715??????????????????? 80
>     ?
>     ?? Block Size:?????????????? 1024b+???????????????
>     2048b+??????????????? 4096b+
>     ?No. of Reads:????????????????? 224????????????????????
>     9??????????????????? 15
>     No. of Writes:????????????????? 163??????????????????
>     282?????????????????? 404
>     ?
>     ?? Block Size:?????????????? 8192b+??????????????
>     16384b+?????????????? 32768b+
>     ?No. of Reads:?????????????????? 52???????????????????
>     74?????????????????? 210
>     No. of Writes:???????????????? 7415?????????????????
>     1430????????????????? 6860
>     ?
>     ?? Block Size:????????????? 65536b+?????????????
>     131072b+????????????? 262144b+
>     ?No. of Reads:????????????????? 299????????????????
>     15313???????????????????? 1
>     No. of Writes:???????????????? 2458????????????????
>     12378???????????????????? 1
>     ?
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     686733????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     837434???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     87338? RELEASEDIR
>     ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us???????????
>     116???????? IPC
>     ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 us????????????
>     57?? FTRUNCATE
>     ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 us????????????
>     65?? FGETXATTR
>     ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 us????????????
>     23 REMOVEXATTR
>     ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us???????????
>     121??????? LINK
>     ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us???????????
>     490????????? LK
>     ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us???????????
>     540??????? STAT
>     ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us???????????
>     200??? TRUNCATE
>     ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us??????????
>     1482???? SETATTR
>     ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 us????????????
>     12?????? FSYNC
>     ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us?????????
>     10047?????? FLUSH
>     ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us??????????
>     9977??? FINODELK
>     ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us???????????
>     188????? RENAME
>     ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 us????????????
>     25???? SYMLINK
>     ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us???????????
>     118?????? MKNOD
>     ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us?????????
>     12587?????? FSTAT
>     ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us?????????
>     21695????? STATFS
>     ????? 0.00???? 307.86 us????? 28.00 us? 292666.83 us?????????
>     16841??????? READ
>     ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 319.24 us????? 51.19 us? 136353.39 us?????????
>     33445?????? WRITE
>     ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us?????????
>     87338???? OPENDIR
>     ????? 0.01???? 249.96 us????? 58.42 us? 594774.86 us?????????
>     85089??? SETXATTR
>     ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us??????????
>     3180?????? RMDIR
>     ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us??????????
>     6067????? CREATE
>     ????? 0.05????? 89.45 us????? 30.14 us? 687961.60 us???????
>     1219542??? READLINK
>     ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us????????
>     831371??????? OPEN
>     ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us??????????
>     9493??? FXATTROP
>     ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us???????
>     3764714???? ENTRYLK
>     ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us?????????
>     22253???? READDIR
>     ????? 1.06??? 6913.71 us????? 11.58 us 36968532.72 us????????
>     364963??? GETXATTR
>     ????? 1.29?? 62130.12 us????? 36.89 us 8649871.11 us?????????
>     49472??? READDIRP
>     ????? 2.89??? 1477.53 us????? 11.10 us 2807637.64 us???????
>     4650500???? INODELK
>     ????? 4.12?? 23681.31 us????? 20.98 us 23291414.50 us????????
>     414352????? UNLINK
>     ???? 16.09??? 5000.31 us?????? 9.04 us 23866016.88 us???????
>     7655621????? LOOKUP
>     ???? 72.04?? 68953.31 us????? 68.40 us 27542499.65 us???????
>     2486126???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2433405????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     2453???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     69??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1544?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2430885?? CI_FORGET
>     ?
>     ??? Duration: 209524 seconds
>     ?? Data Read: 2051388524 bytes
>     Data Written: 2208261499 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     2b+?????????????????? 8b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:?????????????????? 53????????????????????
>     1??????????????????? 60
>     ?
>     ?? Block Size:???????????????? 16b+?????????????????
>     32b+????????????????? 64b+
>     ?No. of Reads:??????????????????? 0??????????????????
>     213???????????????????? 2
>     No. of Writes:?????????????????? 14?????????????????
>     1057??????????????????? 15
>     ?
>     ?? Block Size:??????????????? 128b+????????????????
>     256b+???????????????? 512b+
>     ?No. of Reads:??????????????????? 0???????????????????
>     41?????????????????? 146
>     No. of Writes:?????????????????? 59??????????????????
>     715??????????????????? 80
>     ?
>     ?? Block Size:?????????????? 1024b+???????????????
>     2048b+??????????????? 4096b+
>     ?No. of Reads:????????????????? 224????????????????????
>     9??????????????????? 15
>     No. of Writes:????????????????? 163??????????????????
>     282?????????????????? 404
>     ?
>     ?? Block Size:?????????????? 8192b+??????????????
>     16384b+?????????????? 32768b+
>     ?No. of Reads:?????????????????? 52???????????????????
>     74?????????????????? 210
>     No. of Writes:???????????????? 7415?????????????????
>     1430????????????????? 6860
>     ?
>     ?? Block Size:????????????? 65536b+?????????????
>     131072b+????????????? 262144b+
>     ?No. of Reads:????????????????? 299????????????????
>     15313???????????????????? 1
>     No. of Writes:???????????????? 2458????????????????
>     12378???????????????????? 1
>     ?
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     686733????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     837434???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     87338? RELEASEDIR
>     ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us???????????
>     116???????? IPC
>     ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 us????????????
>     57?? FTRUNCATE
>     ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 us????????????
>     65?? FGETXATTR
>     ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 us????????????
>     23 REMOVEXATTR
>     ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us???????????
>     121??????? LINK
>     ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us???????????
>     490????????? LK
>     ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us???????????
>     540??????? STAT
>     ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us???????????
>     200??? TRUNCATE
>     ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us??????????
>     1482???? SETATTR
>     ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 us????????????
>     12?????? FSYNC
>     ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us?????????
>     10047?????? FLUSH
>     ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us??????????
>     9977??? FINODELK
>     ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us???????????
>     188????? RENAME
>     ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 us????????????
>     25???? SYMLINK
>     ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us???????????
>     118?????? MKNOD
>     ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us?????????
>     12587?????? FSTAT
>     ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us?????????
>     21695????? STATFS
>     ????? 0.00???? 307.91 us????? 28.00 us? 292666.83 us?????????
>     16841??????? READ
>     ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 319.22 us????? 51.19 us? 136353.39 us?????????
>     33445?????? WRITE
>     ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us?????????
>     87338???? OPENDIR
>     ????? 0.01???? 249.95 us????? 58.42 us? 594774.86 us?????????
>     85089??? SETXATTR
>     ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us??????????
>     3180?????? RMDIR
>     ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us??????????
>     6067????? CREATE
>     ????? 0.05????? 89.46 us????? 30.14 us? 687961.60 us???????
>     1219542??? READLINK
>     ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us????????
>     831371??????? OPEN
>     ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us??????????
>     9493??? FXATTROP
>     ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us???????
>     3764714???? ENTRYLK
>     ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us?????????
>     22253???? READDIR
>     ????? 1.06??? 6913.26 us????? 11.58 us 36968532.72 us????????
>     364963??? GETXATTR
>     ????? 1.29?? 62121.95 us????? 36.89 us 8649871.11 us?????????
>     49472??? READDIRP
>     ????? 2.89??? 1477.58 us????? 11.10 us 2807637.64 us???????
>     4650500???? INODELK
>     ????? 4.12?? 23681.29 us????? 20.98 us 23291414.50 us????????
>     414352????? UNLINK
>     ???? 16.09??? 5000.02 us?????? 9.04 us 23866016.88 us???????
>     7655621????? LOOKUP
>     ???? 72.04?? 68956.14 us????? 68.40 us 27542499.65 us???????
>     2486126???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2433405????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     2453???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     69??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1544?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2430885?? CI_FORGET
>     ?
>     ??? Duration: 209524 seconds
>     ?? Data Read: 2051388524 bytes
>     Data Written: 2208261499 bytes
>     ?
>     Brick: flexo:/mnt/gluster/b
>     --------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     8b+????????????????? 16b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:????????????????? 117??????????????????
>     195?????????????????? 367
>     ?
>     ?? Block Size:???????????????? 32b+?????????????????
>     64b+???????????????? 128b+
>     ?No. of Reads:????????????????? 130????????????????????
>     0???????????????????? 3
>     No. of Writes:???????????????? 1012????????????????????
>     9?????????????????? 134
>     ?
>     ?? Block Size:??????????????? 256b+????????????????
>     512b+??????????????? 1024b+
>     ?No. of Reads:?????????????????? 47????????????????????
>     8?????????????????? 257
>     No. of Writes:????????????????? 729???????????????????
>     73?????????????????? 168
>     ?
>     ?? Block Size:?????????????? 2048b+???????????????
>     4096b+??????????????? 8192b+
>     ?No. of Reads:?????????????????? 10???????????????????
>     27?????????????????? 240
>     No. of Writes:????????????????? 294??????????????????
>     466???????????????? 10472
>     ?
>     ?? Block Size:????????????? 16384b+??????????????
>     32768b+?????????????? 65536b+
>     ?No. of Reads:????????????????? 139??????????????????
>     119?????????????????? 135
>     No. of Writes:???????????????? 1480?????????????????
>     6273????????????????? 3935
>     ?
>     ?? Block Size:???????????? 131072b+
>     ?No. of Reads:???????????????? 7892
>     No. of Writes:??????????????? 12327
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     955058????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1450749???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     81377? RELEASEDIR
>     ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us???????????
>     108???????? IPC
>     ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 us????????????
>     44?? FTRUNCATE
>     ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 us????????????
>     30 REMOVEXATTR
>     ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 us?????????????
>     4?????? FSYNC
>     ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 us????????????
>     30???? SYMLINK
>     ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us???????????
>     137??????? LINK
>     ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us???????????
>     119?????? MKNOD
>     ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us???????????
>     221??? SETXATTR
>     ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us???????????
>     354??????? STAT
>     ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us???????????
>     777????????? LK
>     ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us???????????
>     188????? RENAME
>     ????? 0.00???? 150.40 us????? 75.59 us???? 749.54 us???????????
>     810?? FGETXATTR
>     ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us???????????
>     698??? TRUNCATE
>     ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us??????????
>     1665???? SETATTR
>     ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us?????????
>     11372??? FINODELK
>     ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us??????????
>     7398?????? FSTAT
>     ????? 0.00???? 120.71 us????? 31.43 us???? 847.97 us??????????
>     9094??????? READ
>     ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us?????????
>     22614?????? FLUSH
>     ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us?????????
>     21656????? STATFS
>     ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us??????????
>     6145????? CREATE
>     ????? 0.00???? 217.55 us????? 50.60 us?? 30734.99 us?????????
>     38051?????? WRITE
>     ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us?????????
>     81377???? OPENDIR
>     ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us??????????
>     3177?????? RMDIR
>     ????? 0.03???? 100.05 us????? 30.93 us?? 49756.93 us????????
>     432439??? READLINK
>     ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us????????
>     448381????? UNLINK
>     ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us?????????
>     52185??? READDIRP
>     ????? 0.35???? 414.05 us????? 36.58 us 1038229.93 us???????
>     1444606??????? OPEN
>     ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us?????????
>     10745??? FXATTROP
>     ????? 0.72???? 776.21 us????? 11.24 us 4271792.57 us???????
>     1605822??? GETXATTR
>     ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us?????????
>     13532???? READDIR
>     ????? 0.79???? 320.44 us?????? 7.71 us 2369511.06 us???????
>     4257297???? ENTRYLK
>     ????? 1.97???? 464.02 us?????? 9.38 us 1038460.80 us???????
>     7333722???? INODELK
>     ???? 13.73??? 2372.50 us?????? 8.33 us 6258159.87 us???????
>     9979799????? LOOKUP
>     ???? 80.79?? 42372.04 us????? 61.49 us 2238308.13 us???????
>     3289103???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2250268????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     3756???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     167??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1908?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2246352?? CI_FORGET
>     ?
>     ??? Duration: 209334 seconds
>     ?? Data Read: 1059777395 bytes
>     Data Written: 2336921014 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     8b+????????????????? 16b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:????????????????? 117??????????????????
>     195?????????????????? 367
>     ?
>     ?? Block Size:???????????????? 32b+?????????????????
>     64b+???????????????? 128b+
>     ?No. of Reads:????????????????? 130????????????????????
>     0???????????????????? 3
>     No. of Writes:???????????????? 1012????????????????????
>     9?????????????????? 134
>     ?
>     ?? Block Size:??????????????? 256b+????????????????
>     512b+??????????????? 1024b+
>     ?No. of Reads:?????????????????? 47????????????????????
>     8?????????????????? 257
>     No. of Writes:????????????????? 729???????????????????
>     73?????????????????? 168
>     ?
>     ?? Block Size:?????????????? 2048b+???????????????
>     4096b+??????????????? 8192b+
>     ?No. of Reads:?????????????????? 10???????????????????
>     27?????????????????? 240
>     No. of Writes:????????????????? 294??????????????????
>     466???????????????? 10472
>     ?
>     ?? Block Size:????????????? 16384b+??????????????
>     32768b+?????????????? 65536b+
>     ?No. of Reads:????????????????? 139??????????????????
>     119?????????????????? 135
>     No. of Writes:???????????????? 1480?????????????????
>     6273????????????????? 3935
>     ?
>     ?? Block Size:???????????? 131072b+
>     ?No. of Reads:???????????????? 7892
>     No. of Writes:??????????????? 12327
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     955058????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1450749???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     81377? RELEASEDIR
>     ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us???????????
>     108???????? IPC
>     ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 us????????????
>     44?? FTRUNCATE
>     ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 us????????????
>     30 REMOVEXATTR
>     ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 us?????????????
>     4?????? FSYNC
>     ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 us????????????
>     30???? SYMLINK
>     ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us???????????
>     137??????? LINK
>     ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us???????????
>     119?????? MKNOD
>     ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us???????????
>     221??? SETXATTR
>     ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us???????????
>     354??????? STAT
>     ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us???????????
>     777????????? LK
>     ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us???????????
>     188????? RENAME
>     ????? 0.00???? 150.43 us????? 75.59 us???? 749.54 us???????????
>     810?? FGETXATTR
>     ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us???????????
>     698??? TRUNCATE
>     ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us???????????
>     178?????? MKDIR
>     ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us??????????
>     1665???? SETATTR
>     ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us?????????
>     11372??? FINODELK
>     ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us??????????
>     7398?????? FSTAT
>     ????? 0.00???? 120.74 us????? 31.43 us???? 847.97 us??????????
>     9094??????? READ
>     ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us?????????
>     22614?????? FLUSH
>     ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us?????????
>     21656????? STATFS
>     ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us??????????
>     6145????? CREATE
>     ????? 0.00???? 217.50 us????? 50.60 us?? 30734.99 us?????????
>     38051?????? WRITE
>     ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us?????????
>     81377???? OPENDIR
>     ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us??????????
>     3177?????? RMDIR
>     ????? 0.03???? 100.06 us????? 30.93 us?? 49756.93 us????????
>     432439??? READLINK
>     ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us????????
>     448381????? UNLINK
>     ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us?????????
>     52185??? READDIRP
>     ????? 0.35???? 414.01 us????? 36.58 us 1038229.93 us???????
>     1444606??????? OPEN
>     ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us?????????
>     10745??? FXATTROP
>     ????? 0.72???? 776.24 us????? 11.24 us 4271792.57 us???????
>     1605822??? GETXATTR
>     ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us?????????
>     13532???? READDIR
>     ????? 0.79???? 320.45 us?????? 7.71 us 2369511.06 us???????
>     4257297???? ENTRYLK
>     ????? 1.97???? 464.07 us?????? 9.38 us 1038460.80 us???????
>     7333722???? INODELK
>     ???? 13.72??? 2372.42 us?????? 8.33 us 6258159.87 us???????
>     9979799????? LOOKUP
>     ???? 80.79?? 42373.50 us????? 61.49 us 2238308.13 us???????
>     3289103???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2250268????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     3756???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     167??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1908?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     2246352?? CI_FORGET
>     ?
>     ??? Duration: 209334 seconds
>     ?? Data Read: 1059777395 bytes
>     Data Written: 2336921014 bytes
>     ?
>     Brick: flexo:/mnt/gluster/c
>     --------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+
>     ?No. of Reads:??????????????????? 0
>     No. of Writes:??????????????? 32843
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     686562????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     834596???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     93757? RELEASEDIR
>     ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us???????????
>     107???????? IPC
>     ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 us????????????
>     36??????? LINK
>     ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 us????????????
>     65?? FGETXATTR
>     ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 us????????????
>     88????? RENAME
>     ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us???????????
>     153??? TRUNCATE
>     ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us???????????
>     491????????? LK
>     ????? 0.01????? 51.73 us?????? 7.70 us??? 1174.64 us??????????
>     9339?????? FLUSH
>     ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us??????????
>     9011??? FINODELK
>     ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us?????????
>     10824?????? FSTAT
>     ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us??????????
>     2594?????? MKDIR
>     ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us??????????
>     9009??? FXATTROP
>     ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us??????????
>     5754????? CREATE
>     ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us?????????
>     32843?????? WRITE
>     ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us??????????
>     5798???? SYMLINK
>     ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us?????????
>     34904???? READDIR
>     ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us??????????
>     2732?????? RMDIR
>     ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us?????????
>     93757???? OPENDIR
>     ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us????????
>     170292????? UNLINK
>     ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us?????????
>     13840??? READDIRP
>     ????? 1.94???? 200.88 us????? 25.49 us? 461654.32 us????????
>     468842?? FTRUNCATE
>     ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us????????
>     486882 REMOVEXATTR
>     ????? 2.27???? 223.01 us????? 52.03 us? 178897.40 us????????
>     493240??? SETXATTR
>     ????? 3.53???? 155.31 us?????? 7.96 us?? 67873.00 us???????
>     1101338??? GETXATTR
>     ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us????????
>     829002??????? OPEN
>     ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us???????
>     3660425???? ENTRYLK
>     ????? 6.23???? 304.93 us????? 42.51 us? 661394.43 us????????
>     990644???? SETATTR
>     ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us????????
>     990766???? XATTROP
>     ???? 12.39???? 549.11 us???? 135.57 us 1659697.83 us???????
>     1093387?????? MKNOD
>     ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us???????
>     4325281???? INODELK
>     ???? 39.53???? 206.94 us?????? 8.56 us? 964823.33 us???????
>     9259429????? LOOKUP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1856377????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     933300???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     251??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     293?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     923077?? CI_FORGET
>     ?
>     ??? Duration: 209334 seconds
>     ?? Data Read: 0 bytes
>     Data Written: 32843 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+
>     ?No. of Reads:??????????????????? 0
>     No. of Writes:??????????????? 32843
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     686562????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     834596???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     93757? RELEASEDIR
>     ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us???????????
>     107???????? IPC
>     ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 us????????????
>     36??????? LINK
>     ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 us????????????
>     65?? FGETXATTR
>     ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 us????????????
>     88????? RENAME
>     ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us???????????
>     153??? TRUNCATE
>     ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us???????????
>     491????????? LK
>     ????? 0.01????? 51.71 us?????? 7.70 us??? 1174.64 us??????????
>     9339?????? FLUSH
>     ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us??????????
>     9011??? FINODELK
>     ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us?????????
>     10824?????? FSTAT
>     ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us??????????
>     2594?????? MKDIR
>     ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us??????????
>     9009??? FXATTROP
>     ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us??????????
>     5754????? CREATE
>     ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us?????????
>     32843?????? WRITE
>     ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us??????????
>     5798???? SYMLINK
>     ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us?????????
>     34904???? READDIR
>     ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us??????????
>     2732?????? RMDIR
>     ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us?????????
>     93757???? OPENDIR
>     ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us????????
>     170292????? UNLINK
>     ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us?????????
>     13840??? READDIRP
>     ????? 1.94???? 200.87 us????? 25.49 us? 461654.32 us????????
>     468842?? FTRUNCATE
>     ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us????????
>     486882 REMOVEXATTR
>     ????? 2.27???? 223.00 us????? 52.03 us? 178897.40 us????????
>     493240??? SETXATTR
>     ????? 3.53???? 155.32 us?????? 7.96 us?? 67873.00 us???????
>     1101338??? GETXATTR
>     ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us????????
>     829002??????? OPEN
>     ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us???????
>     3660425???? ENTRYLK
>     ????? 6.23???? 304.88 us????? 42.51 us? 661394.43 us????????
>     990644???? SETATTR
>     ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us????????
>     990766???? XATTROP
>     ???? 12.39???? 549.25 us???? 135.57 us 1659697.83 us???????
>     1093387?????? MKNOD
>     ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us???????
>     4325281???? INODELK
>     ???? 39.53???? 206.95 us?????? 8.56 us? 964823.33 us???????
>     9259429????? LOOKUP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1856377????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     933300???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     251??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     293?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     923077?? CI_FORGET
>     ?
>     ??? Duration: 209334 seconds
>     ?? Data Read: 0 bytes
>     Data Written: 32843 bytes
>     ?
>     Brick: bender:/mnt/gluster/c
>     ---------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+
>     ?No. of Reads:??????????????????? 0
>     No. of Writes:??????????????? 37050
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     928119????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1401524???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     70092? RELEASEDIR
>     ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us???????????
>     100???????? IPC
>     ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 us????????????
>     38??????? LINK
>     ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 us????????????
>     30???? SYMLINK
>     ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 us?????????????
>     4?????? FSYNC
>     ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us???????????
>     122????? RENAME
>     ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us???????????
>     777????????? LK
>     ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us???????????
>     810?? FGETXATTR
>     ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us???????????
>     603??? TRUNCATE
>     ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us???????????
>     302?????? MKDIR
>     ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us?????????
>     10060??? FINODELK
>     ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us??????????
>     6409?????? FSTAT
>     ????? 0.01????? 49.38 us?????? 8.08 us??? 1264.41 us?????????
>     21719?????? FLUSH
>     ????? 0.03????? 74.76 us????? 20.86 us??? 2008.41 us?????????
>     37050?????? WRITE
>     ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us??????????
>     5901????? CREATE
>     ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us?????????
>     10057??? FXATTROP
>     ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us?????????
>     70092???? OPENDIR
>     ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us?????????
>     10779???? READDIR
>     ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us??????????
>     3008?????? RMDIR
>     ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us????????
>     109478????? UNLINK
>     ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us????????
>     722669 REMOVEXATTR
>     ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us????????
>     789294??? GETXATTR
>     ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us????????
>     718835?? FTRUNCATE
>     ????? 1.89???? 219.47 us????? 49.92 us? 506767.78 us????????
>     722856??? SETXATTR
>     ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us???????
>     1395957??????? OPEN
>     ????? 5.56???? 322.08 us????? 43.37 us? 634069.61 us???????
>     1450160???? SETATTR
>     ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us???????
>     4233520???? ENTRYLK
>     ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us???????
>     1449601???? XATTROP
>     ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us???????
>     7097226???? INODELK
>     ???? 19.37??? 1104.98 us???? 101.28 us 1612346.75 us???????
>     1472813?????? MKNOD
>     ???? 41.85???? 295.61 us?????? 8.98 us? 996805.54 us??????
>     11896091????? LOOKUP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3531139????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1862470???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     244??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     632?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1668669?? CI_FORGET
>     ?
>     ??? Duration: 208643 seconds
>     ?? Data Read: 0 bytes
>     Data Written: 37050 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+
>     ?No. of Reads:??????????????????? 0
>     No. of Writes:??????????????? 37050
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     928119????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1401524???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     70092? RELEASEDIR
>     ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us???????????
>     100???????? IPC
>     ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 us????????????
>     38??????? LINK
>     ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 us????????????
>     30???? SYMLINK
>     ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 us?????????????
>     4?????? FSYNC
>     ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us???????????
>     122????? RENAME
>     ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us???????????
>     777????????? LK
>     ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us???????????
>     810?? FGETXATTR
>     ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us???????????
>     603??? TRUNCATE
>     ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us???????????
>     302?????? MKDIR
>     ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us?????????
>     10060??? FINODELK
>     ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us??????????
>     6409?????? FSTAT
>     ????? 0.01????? 49.39 us?????? 8.08 us??? 1264.41 us?????????
>     21719?????? FLUSH
>     ????? 0.03????? 74.77 us????? 20.86 us??? 2008.41 us?????????
>     37050?????? WRITE
>     ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us??????????
>     5901????? CREATE
>     ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us?????????
>     10057??? FXATTROP
>     ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us?????????
>     70092???? OPENDIR
>     ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us?????????
>     10779???? READDIR
>     ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us??????????
>     3008?????? RMDIR
>     ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us????????
>     109478????? UNLINK
>     ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us????????
>     722669 REMOVEXATTR
>     ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us????????
>     789294??? GETXATTR
>     ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us????????
>     718835?? FTRUNCATE
>     ????? 1.89???? 219.48 us????? 49.92 us? 506767.78 us????????
>     722856??? SETXATTR
>     ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us???????
>     1395957??????? OPEN
>     ????? 5.56???? 322.09 us????? 43.37 us? 634069.61 us???????
>     1450160???? SETATTR
>     ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us???????
>     4233520???? ENTRYLK
>     ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us???????
>     1449601???? XATTROP
>     ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us???????
>     7097226???? INODELK
>     ???? 19.37??? 1105.06 us???? 101.28 us 1612346.75 us???????
>     1472813?????? MKNOD
>     ???? 41.84???? 295.54 us?????? 8.98 us? 996805.54 us??????
>     11896091????? LOOKUP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     3531139????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1862470???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     244??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>     632?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1668669?? CI_FORGET
>     ?
>     ??? Duration: 208643 seconds
>     ?? Data Read: 0 bytes
>     Data Written: 37050 bytes
>     ?
>     Brick: bender:/mnt/gluster/a
>     ---------------------------------------
>     Cumulative Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     2b+?????????????????? 8b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:?????????????????? 53????????????????????
>     1??????????????????? 60
>     ?
>     ?? Block Size:???????????????? 16b+?????????????????
>     32b+????????????????? 64b+
>     ?No. of Reads:??????????????????? 0??????????????????
>     235???????????????????? 4
>     No. of Writes:?????????????????? 14?????????????????
>     1033??????????????????? 15
>     ?
>     ?? Block Size:??????????????? 128b+????????????????
>     256b+???????????????? 512b+
>     ?No. of Reads:??????????????????? 4??????????????????
>     115??????????????????? 81
>     No. of Writes:?????????????????? 58??????????????????
>     703??????????????????? 80
>     ?
>     ?? Block Size:?????????????? 1024b+???????????????
>     2048b+??????????????? 4096b+
>     ?No. of Reads:????????????????? 181???????????????????
>     26??????????????????? 74
>     No. of Writes:????????????????? 163??????????????????
>     278?????????????????? 397
>     ?
>     ?? Block Size:?????????????? 8192b+??????????????
>     16384b+?????????????? 32768b+
>     ?No. of Reads:????????????????? 131??????????????????
>     161?????????????????? 267
>     No. of Writes:???????????????? 7264?????????????????
>     1398????????????????? 6742
>     ?
>     ?? Block Size:????????????? 65536b+?????????????
>     131072b+????????????? 262144b+
>     ?No. of Reads:????????????????? 443????????????????
>     25528???????????????????? 0
>     No. of Writes:???????????????? 2426????????????????
>     12240???????????????????? 1
>     ?
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     684942????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     837092???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     87297? RELEASEDIR
>     ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 us?????????????
>     8 REMOVEXATTR
>     ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 us????????????
>     97???????? IPC
>     ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 us????????????
>     46?? FTRUNCATE
>     ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 us????????????
>     65?? FGETXATTR
>     ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us???????????
>     121??????? LINK
>     ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 us????????????
>     25???? SYMLINK
>     ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us???????????
>     107?????? MKNOD
>     ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 us?????????????
>     1?????? FSYNC
>     ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us???????????
>     188????? RENAME
>     ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us???????????
>     745??????? STAT
>     ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us???????????
>     200??? TRUNCATE
>     ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us??????????
>     1444???? SETATTR
>     ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us???????????
>     177?????? MKDIR
>     ????? 0.00????? 52.68 us?????? 7.28 us???? 826.01 us??????????
>     9890?????? FLUSH
>     ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us??????????
>     9792??? FINODELK
>     ????? 0.00????? 74.26 us????? 20.00 us??? 1522.39 us?????????
>     21521????? STATFS
>     ????? 0.00???? 129.21 us????? 34.15 us??? 9068.67 us?????????
>     13094?????? FSTAT
>     ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us??????????
>     5967????? CREATE
>     ????? 0.00???? 112.06 us????? 28.14 us??? 9545.53 us?????????
>     27474??????? READ
>     ????? 0.01???? 313.76 us????? 52.98 us?? 71635.74 us?????????
>     32926?????? WRITE
>     ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us?????????
>     87297???? OPENDIR
>     ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us??????????
>     3177?????? RMDIR
>     ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us?????????
>     85073??? SETXATTR
>     ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us?????????
>     15090??? READDIRP
>     ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us???????????
>     564????????? LK
>     ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us????????
>     850938??? READLINK
>     ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us????????
>     831212??????? OPEN
>     ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us?????????
>     22482???? READDIR
>     ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us????????
>     415226????? UNLINK
>     ????? 0.54??? 1272.83 us?????? 9.66 us 1604368.93 us????????
>     686945??? GETXATTR
>     ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us???????
>     3739718???? ENTRYLK
>     ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us??????????
>     9311??? FXATTROP
>     ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us???????
>     4649291???? INODELK
>     ????? 5.82??? 1240.20 us?????? 8.78 us 1526868.50 us???????
>     7625383????? LOOKUP
>     ???? 88.60?? 58005.98 us????? 67.31 us 1448810.06 us???????
>     2481914???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1396875????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1854???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     28??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1051?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1394996?? CI_FORGET
>     ?
>     ??? Duration: 208643 seconds
>     ?? Data Read: 3409342848 bytes
>     Data Written: 2180686507 bytes
>     ?
>     Interval 0 Stats:
>     ?? Block Size:????????????????? 1b+??????????????????
>     2b+?????????????????? 8b+
>     ?No. of Reads:??????????????????? 0????????????????????
>     0???????????????????? 0
>     No. of Writes:?????????????????? 53????????????????????
>     1??????????????????? 60
>     ?
>     ?? Block Size:???????????????? 16b+?????????????????
>     32b+????????????????? 64b+
>     ?No. of Reads:??????????????????? 0??????????????????
>     235???????????????????? 4
>     No. of Writes:?????????????????? 14?????????????????
>     1033??????????????????? 15
>     ?
>     ?? Block Size:??????????????? 128b+????????????????
>     256b+???????????????? 512b+
>     ?No. of Reads:??????????????????? 4??????????????????
>     115??????????????????? 81
>     No. of Writes:?????????????????? 58??????????????????
>     703??????????????????? 80
>     ?
>     ?? Block Size:?????????????? 1024b+???????????????
>     2048b+??????????????? 4096b+
>     ?No. of Reads:????????????????? 181???????????????????
>     26??????????????????? 74
>     No. of Writes:????????????????? 163??????????????????
>     278?????????????????? 397
>     ?
>     ?? Block Size:?????????????? 8192b+??????????????
>     16384b+?????????????? 32768b+
>     ?No. of Reads:????????????????? 131??????????????????
>     161?????????????????? 267
>     No. of Writes:???????????????? 7264?????????????????
>     1398????????????????? 6742
>     ?
>     ?? Block Size:????????????? 65536b+?????????????
>     131072b+????????????? 262144b+
>     ?No. of Reads:????????????????? 443????????????????
>     25528???????????????????? 0
>     No. of Writes:???????????????? 2426????????????????
>     12240???????????????????? 1
>     ?
>     ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>     calls???????? Fop
>     ?---------?? -----------?? -----------?? -----------??
>     ------------??????? ----
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     684942????? FORGET
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>     837092???? RELEASE
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>     87297? RELEASEDIR
>     ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 us?????????????
>     8 REMOVEXATTR
>     ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 us????????????
>     97???????? IPC
>     ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 us????????????
>     46?? FTRUNCATE
>     ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 us????????????
>     65?? FGETXATTR
>     ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us???????????
>     121??????? LINK
>     ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 us????????????
>     25???? SYMLINK
>     ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us???????????
>     107?????? MKNOD
>     ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 us?????????????
>     1?????? FSYNC
>     ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us???????????
>     188????? RENAME
>     ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us???????????
>     745??????? STAT
>     ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us???????????
>     200??? TRUNCATE
>     ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us??????????
>     1444???? SETATTR
>     ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us???????????
>     177?????? MKDIR
>     ????? 0.00????? 52.69 us?????? 7.28 us???? 826.01 us??????????
>     9890?????? FLUSH
>     ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us??????????
>     9792??? FINODELK
>     ????? 0.00????? 74.25 us????? 20.00 us??? 1522.39 us?????????
>     21521????? STATFS
>     ????? 0.00???? 129.23 us????? 34.15 us??? 9068.67 us?????????
>     13094?????? FSTAT
>     ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us??????????
>     5967????? CREATE
>     ????? 0.00???? 112.45 us????? 28.14 us??? 9545.53 us?????????
>     27474??????? READ
>     ????? 0.01???? 313.77 us????? 52.98 us?? 71635.74 us?????????
>     32926?????? WRITE
>     ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us?????????
>     87297???? OPENDIR
>     ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us??????????
>     3177?????? RMDIR
>     ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us?????????
>     85073??? SETXATTR
>     ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us?????????
>     15090??? READDIRP
>     ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us???????????
>     564????????? LK
>     ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us????????
>     850938??? READLINK
>     ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us????????
>     831212??????? OPEN
>     ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us?????????
>     22482???? READDIR
>     ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us????????
>     415226????? UNLINK
>     ????? 0.54??? 1272.79 us?????? 9.66 us 1604368.93 us????????
>     686945??? GETXATTR
>     ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us???????
>     3739718???? ENTRYLK
>     ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us??????????
>     9311??? FXATTROP
>     ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us???????
>     4649291???? INODELK
>     ????? 5.82??? 1240.10 us?????? 8.78 us 1526868.50 us???????
>     7625383????? LOOKUP
>     ???? 88.60?? 58006.02 us????? 67.31 us 1448810.06 us???????
>     2481914???? XATTROP
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1396875????? UPCALL
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1854???? CI_IATT
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>     28??? CI_XATTR
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>     1051?? CI_UNLINK
>     ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>     1394996?? CI_FORGET
>     ?
>     ??? Duration: 208643 seconds
>     ?? Data Read: 3409342848 bytes
>     Data Written: 2180686507 bytes
>     ?
>
>     ________
>
>
>
>     Community Meeting Calendar:
>
>     Schedule -
>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>     Bridge: https://bluejeans.com/441850968
>
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-users
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200505/afca4a51/attachment-0001.html>

Gluster users - May 2020 - Possible SHD stalling

[Gluster-users] Possible SHD stalling

[Gluster-users] Possible SHD stalling