Hi,
So upgrading to 7.5 made a difference in that it takes a bit longer to
cause chaos, but it still does the exact same thing.
To be precise:
I've now had to set:
gluster volume set vpbx_shared cluster.heal-wait-queue-length 1
gluster volume set vpbx_shared cluster.shd-wait-qlength 1
gluster volume set vpbx_shared cluster.shd-max-threads 1
With any of these settings >1 I end up in a situation where simple
directory listings of < 100 entries in a folder takes upwards of a
minute to complete *from time to time*.
It seems that when the logs
With these the maximum stall I can reproduce is usually around 3 seconds
before the directory listing will complete, but about 5s worst case.?
I've now reverted these as well, which are known for us to cause issues
under heal required conditions:
cluster.metadata-self-heal?????????????
on?????????????????????????????????????
cluster.data-self-heal?????????????????
on?????????????????????????????????????
cluster.entry-self-heal????????????????
on?????????????????????????????????????
Not sure if I need to remount for these to become effective but I've
done so just to be on the safe side.
I'm still seeing these errors:
[2020-05-07 19:20:12.763311] W [MSGID: 114031]
[client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
0-vpbx_shared-client-1: remote operation failed [Invalid argument]
[2020-05-07 19:20:12.995675] W [MSGID: 114031]
[client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
0-vpbx_shared-client-1: remote operation failed [Invalid argument]
[2020-05-07 19:20:59.199776] W [MSGID: 114031]
[client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
0-vpbx_shared-client-1: remote operation failed [Invalid argument]
And the start of my issues still seem to co-incide with that.? Sample case:
# time ls
...
real??? 0m5.140s
user??? 0m0.009s
sys??? 0m0.000s
# f=(*); echo "${#f[@]}"
165
This just does not seem right.? Note, again, if I set *any* of the above
parameters >1 these delays get far, far worse (I've had over a minute
for this exact same ls).
Any advise would be greatly appreciated.? To me, it looks like some lock
is being taken somewhere, and takes a while to release.? I don't
understand the glusterfs internals nearly well enough to understand
exactly what's going on here, nor do I know what information to provide
to help drill down on this.? Happy to go look at code, if I can get
pointed in a direction.
Kind Regards,
Jaco
On 2020/05/05 08:43, Jaco Kroon wrote:>
> Hi Artem,
>
> I'll do that.? In the meantime consider this (inferred from the
> changelogs).
>
> It seems like glusterfs SHD is queueing into a bounded queue under
> lock conditions, and waiting for space to open up.? At least, this is
> what's hinted in the changelogs, I've been trying to grasp the SHD
> code for two days now and am not getting to wrap my head around that.?
> If this is true, then the only way to avoid lock-ups are to ensure
> that the queue is larger than the total number of files that can ever
> require healing (effectively making the queue unbounded).
>
> Kind Regards,
> Jaco
>
> On 2020/05/04 21:56, Artem Russakovskii wrote:
>> I reported something similar recently to this list, and I highly
>> advise you to update to 7.5 (in my case it was 5.11 -> 5.13), then
>> see if it keeps happening. The self heal issues should be fixed there
>> properly at least.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>,?APK
Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net <http://beerpla.net/> | @ArtemR
<http://twitter.com/ArtemR>
>>
>>
>> On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za
>> <mailto:jaco at uls.co.za>> wrote:
>>
>> Hi All,
>>
>> So we've got a cluster currently running on glusterfs 7.3
>>
>> Heal performance is absolutely terrible (it starts of great, runs
>> for a while then completely trips up).
>>
>> Once the SHD trips up, access speeds to the cluster also becomes
>> poor.? And with poor I mean directory listings that normally
>> takes well under 500ms takes upwards of 10s, but I've seen
>> upwards of 40s (although a follow up ls immediately after is nice
>> and fast again, temporarily).? With all SHDs killed this
>> particular ls is as per normal (around 30-50ms).
>>
>> The SHD logs doesn't provide any usable clues, the only
variance
>> in logs is this, just before all daemons across the cluster
>> crawls to a near halt.
>>
>> [2020-05-02 19:55:58.627036] I [MSGID: 108026]
>> [afr-self-heal-common.c:1747:afr_log_selfheal]
>> 0-vpbx_shared-replicate-0: Completed entry selfheal on
>> 5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1 ?
>> [2020-05-02 19:55:58.627058] I [MSGID: 108026]
>> [afr-self-heal-common.c:1747:afr_log_selfheal]
>> 0-vpbx_shared-replicate-0: Completed entry selfheal on
>> c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1 ?
>> ?
>> [2020-05-02 19:55:59.125001] W [MSGID: 114031]
>> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
>> 0-vpbx_shared-client-1: remote operation failed [Invalid argument]
>> ?
>> [2020-05-02 19:55:59.339803] I [MSGID: 108026]
>> [afr-self-heal-common.c:1747:afr_log_selfheal]
>> 0-vpbx_shared-replicate-0: Completed entry selfheal on
>> 2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1?
>> [2020-05-02 19:55:59.490387] I [MSGID: 108026]
>> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
>> 0-vpbx_shared-replicate-0: performing metadata selfheal on
>> 527a63a3-9acd-4c3f-b95c-5c719325c973
>>
>> A few more entries will complete overall and then it'll be a
file
>> every other second.? Once I saw this I ran a few more tests, and
>> on every test just prior to slowing to a crawl, that entry is
there.
>>
>> There is a perceived improvement when lowering the thread count,
>> but not significant enough to confirm a correlation.? Even
>> with**cluster.shd-max-threads=1 this still happens (gets a few
>> tens thousand files completed instead of few hundred, at 32 it
>> gets to maybe 400 to 500 before it stalls, 16 does maybe a
>> thousand).*
>> *
>>
>> There are probably performant bursts hereafter, because from
>> experience we know this process takes about two months.? But this
>> time around it seems to be much, much worse.? Might be because
>> that was healing a single brick to another brick, so less locking
>> going?? At the moment the replicas are basically trying to
>> populate the newly added arbiters.? Also, we haven't messed as
>> greatly with the health check settings (queues and threads and
>> locks nearly as much as since the arbiter addition because
>> performance has simply dropped to non-acceptable levels.
>>
>> Unfortunately granular entry heal isn't on at the moment (nor
can
>> I switch it on due to a few million outstanding heals).? This
>> used to be a 2x2 cluster.? We then moved one of the bricks to a
>> newly added server using replace-brick.? What a frightening
>> process this is.? We then proceeded to add the arbiters.
>>
>> Some of the settings below was absolute shots in the dark, based
>> on the little information we've been able to track down so far.
>>
>> Kind Regards,
>> Jaco
>>
>> The current gluster volume info (this used to be 2x2
>> distributed-replicate, aiming to get it to 3x (2 + 1) ... at
>> current rates we might get there by 2021):
>>
>> Volume Name: glvol
>> Type: Distributed-Replicate
>> Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 2 x (2 + 1) = 6
>> Transport-type: tcp
>> Bricks:
>> Brick1: zoidberg:/mnt/gluster/a
>> Brick2: flexo:/mnt/gluster/b
>> Brick3: bender:/mnt/gluster/c (arbiter)
>> Brick4: bender:/mnt/gluster/a
>> Brick5: zoidberg:/mnt/gluster/b
>> Brick6: flexo:/mnt/gluster/c (arbiter)
>> Options Reconfigured:
>> *cluster.heal-timeout: 5 <-- hoping this would at least shorten
>> the blockages.*
>> cluster.locking-scheme: granular
>> cluster.heal-wait-queue-length: 10240
>> *diagnostics.count-fop-hits: on**
>> **diagnostics.latency-measurement: on <-- was hoping profiling
>> might provide some insight.? Going to switch this off now.*
>> cluster.background-self-heal-count: 128
>> *cluster.shd-wait-qlength: 131072**
>> **cluster.shd-max-threads: 32? <-- even with this disk IO never
>> goes over 10% utilization.? CPU goes to about 2 cores.? So it's
>> lock contention that's slowing it down we reckon.*
>> *cluster.metadata-self-heal: off**
>> **cluster.entry-self-heal: off**
>> **cluster.data-self-heal: off**<-- these kill performance.
>> *transport.address-family: inet
>> nfs.disable: on
>> performance.client-io-threads: on
>> features.cache-invalidation: on
>> features.cache-invalidation-timeout: 600
>> performance.stat-prefetch: on
>> performance.cache-invalidation: on
>> performance.md-cache-timeout: 600
>> *cluster.self-heal-daemon: yes <-- disabling/enabling this
>> doesn't help, have to kill the shd and start ... force to get
>> things moving along again.*
>> performance.io-thread-count: 16
>> performance.high-prio-threads: 16
>> performance.normal-prio-threads: 16
>> performance.low-prio-threads: 16
>> performance.least-prio-threads: 4
>> cluster.self-heal-window-size: 512
>> cluster.self-heal-readdir-size: 131072
>> client.event-threads: 8
>> server.event-threads: 16
>> network.ping-timeout: 2
>> *cluster.data-self-heal-algorithm: full <-- most files are small
>> (<1MB), on dual 1gbit links (so up to 2gbit, but b/w never goes
>> over 5mbit anyway).*
>>
>> performance info:
>>
>> Brick: zoidberg:/mnt/gluster/a
>> -----------------------------------------
>> Cumulative Stats:
>> ?? Block Size:????????????????? 1b+??????????????????
>> 8b+????????????????? 16b+
>> ?No. of Reads:??????????????????? 0????????????????????
>> 0???????????????????? 0
>> No. of Writes:????????????????? 117??????????????????
>> 195?????????????????? 367
>> ?
>> ?? Block Size:???????????????? 32b+?????????????????
>> 64b+???????????????? 128b+
>> ?No. of Reads:????????????????? 306????????????????????
>> 0???????????????????? 6
>> No. of Writes:???????????????? 1016????????????????????
>> 9?????????????????? 134
>> ?
>> ?? Block Size:??????????????? 256b+????????????????
>> 512b+??????????????? 1024b+
>> ?No. of Reads:????????????????? 107???????????????????
>> 58?????????????????? 485
>> No. of Writes:????????????????? 731???????????????????
>> 73?????????????????? 168
>> ?
>> ?? Block Size:?????????????? 2048b+???????????????
>> 4096b+??????????????? 8192b+
>> ?No. of Reads:?????????????????? 43???????????????????
>> 59?????????????????? 383
>> No. of Writes:????????????????? 295??????????????????
>> 475???????????????? 10562
>> ?
>> ?? Block Size:????????????? 16384b+??????????????
>> 32768b+?????????????? 65536b+
>> ?No. of Reads:????????????????? 347??????????????????
>> 380?????????????????? 634
>> No. of Writes:???????????????? 1487?????????????????
>> 6369????????????????? 3970
>> ?
>> ?? Block Size:???????????? 131072b+
>> ?No. of Reads:??????????????? 35704
>> No. of Writes:??????????????? 12398
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 957293????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1455091???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 69951? RELEASEDIR
>> ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us???????????
>> 113???????? IPC
>> ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 us????????????
>> 30 REMOVEXATTR
>> ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 us????????????
>> 52?? FTRUNCATE
>> ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us??????????
>> 1385??????? STAT
>> ????? 0.00???? 364.93 us????? 63.45 us?? 15957.66 us???????????
>> 810?? FGETXATTR
>> ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us???????????
>> 137??????? LINK
>> ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 us????????????
>> 12?????? FSYNC
>> ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us???????????
>> 698??? TRUNCATE
>> ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us???????????
>> 188????? RENAME
>> ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us??????????
>> 1680???? SETATTR
>> ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us???????????
>> 127?????? MKNOD
>> ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us?????????
>> 11412??? FINODELK
>> ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us???????????
>> 221??? SETXATTR
>> ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 us????????????
>> 30???? SYMLINK
>> ????? 0.00????? 56.18 us?????? 7.31 us?? 41484.37 us?????????
>> 22640?????? FLUSH
>> ????? 0.00???? 105.64 us????? 19.34 us? 351055.72 us?????????
>> 21693????? STATFS
>> ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us???????????
>> 178?????? MKDIR
>> ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us??????????
>> 9653?????? FSTAT
>> ????? 0.00???? 214.87 us????? 25.71 us 1712020.99 us?????????
>> 39054??????? READ
>> ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us?????????
>> 69951???? OPENDIR
>> ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us??????????
>> 3180?????? RMDIR
>> ????? 0.01???? 327.08 us????? 55.69 us? 420376.80 us?????????
>> 38366?????? WRITE
>> ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us??????????
>> 6169????? CREATE
>> ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us???????
>> 1159936??? READLINK
>> ????? 0.40???? 543.50 us????? 36.45 us 2430653.54 us???????
>> 1449307??????? OPEN
>> ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us?????????
>> 10787??? FXATTROP
>> ????? 0.53??? 2342.72 us????? 20.91 us 4157542.43 us????????
>> 447821????? UNLINK
>> ????? 0.53? 558734.79 us????? 24.40 us 1399487.63 us??????????
>> 1889????????? LK
>> ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us?????????
>> 10687???? READDIR
>> ????? 0.99??? 2426.23 us????? 12.47 us 24176123.13 us????????
>> 809430??? GETXATTR
>> ????? 1.45???? 676.83 us?????? 9.30 us 193177067.25 us???????
>> 4260021???? ENTRYLK
>> ????? 1.76?? 38325.38 us????? 21.44 us 46693575.93 us?????????
>> 91439??? READDIRP
>> ????? 4.44??? 1200.31 us?????? 9.10 us 2559377.90 us???????
>> 7352811???? INODELK
>> ????? 9.30??? 1849.24 us?????? 9.28 us 35299050.00 us???????
>> 9989221????? LOOKUP
>> ???? 79.24?? 47791.46 us????? 76.77 us 10488824.26 us???????
>> 3294642???? XATTROP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 3194806????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 5311???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>> 54??? CI_XATTR
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 1907?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 3189457?? CI_FORGET
>> ?
>> ??? Duration: 209524 seconds
>> ?? Data Read: 4775365531 bytes
>> Data Written: 2354108664 bytes
>> ?
>> Interval 0 Stats:
>> ?? Block Size:????????????????? 1b+??????????????????
>> 8b+????????????????? 16b+
>> ?No. of Reads:??????????????????? 0????????????????????
>> 0???????????????????? 0
>> No. of Writes:????????????????? 117??????????????????
>> 195?????????????????? 367
>> ?
>> ?? Block Size:???????????????? 32b+?????????????????
>> 64b+???????????????? 128b+
>> ?No. of Reads:????????????????? 306????????????????????
>> 0???????????????????? 6
>> No. of Writes:???????????????? 1016????????????????????
>> 9?????????????????? 134
>> ?
>> ?? Block Size:??????????????? 256b+????????????????
>> 512b+??????????????? 1024b+
>> ?No. of Reads:????????????????? 107???????????????????
>> 58?????????????????? 485
>> No. of Writes:????????????????? 731???????????????????
>> 73?????????????????? 168
>> ?
>> ?? Block Size:?????????????? 2048b+???????????????
>> 4096b+??????????????? 8192b+
>> ?No. of Reads:?????????????????? 43???????????????????
>> 59?????????????????? 383
>> No. of Writes:????????????????? 295??????????????????
>> 475???????????????? 10562
>> ?
>> ?? Block Size:????????????? 16384b+??????????????
>> 32768b+?????????????? 65536b+
>> ?No. of Reads:????????????????? 347??????????????????
>> 380?????????????????? 634
>> No. of Writes:???????????????? 1487?????????????????
>> 6369????????????????? 3970
>> ?
>> ?? Block Size:???????????? 131072b+
>> ?No. of Reads:??????????????? 35704
>> No. of Writes:??????????????? 12398
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 957293????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1455091???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 69951? RELEASEDIR
>> ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us???????????
>> 113???????? IPC
>> ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 us????????????
>> 30 REMOVEXATTR
>> ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 us????????????
>> 52?? FTRUNCATE
>> ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us??????????
>> 1385??????? STAT
>> ????? 0.00???? 363.20 us????? 63.45 us?? 15957.66 us???????????
>> 810?? FGETXATTR
>> ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us???????????
>> 137??????? LINK
>> ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 us????????????
>> 12?????? FSYNC
>> ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us???????????
>> 698??? TRUNCATE
>> ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us???????????
>> 188????? RENAME
>> ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us??????????
>> 1680???? SETATTR
>> ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us???????????
>> 127?????? MKNOD
>> ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us?????????
>> 11412??? FINODELK
>> ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us???????????
>> 221??? SETXATTR
>> ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 us????????????
>> 30???? SYMLINK
>> ????? 0.00????? 56.19 us?????? 7.31 us?? 41484.37 us?????????
>> 22640?????? FLUSH
>> ????? 0.00???? 105.61 us????? 19.34 us? 351055.72 us?????????
>> 21693????? STATFS
>> ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us???????????
>> 178?????? MKDIR
>> ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us??????????
>> 9653?????? FSTAT
>> ????? 0.00???? 214.91 us????? 25.71 us 1712020.99 us?????????
>> 39054??????? READ
>> ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us?????????
>> 69951???? OPENDIR
>> ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us??????????
>> 3180?????? RMDIR
>> ????? 0.01???? 327.13 us????? 55.69 us? 420376.80 us?????????
>> 38366?????? WRITE
>> ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us??????????
>> 6169????? CREATE
>> ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us???????
>> 1159936??? READLINK
>> ????? 0.40???? 543.49 us????? 36.45 us 2430653.54 us???????
>> 1449307??????? OPEN
>> ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us?????????
>> 10787??? FXATTROP
>> ????? 0.53??? 2342.57 us????? 20.91 us 4157542.43 us????????
>> 447821????? UNLINK
>> ????? 0.53? 558416.10 us????? 24.40 us 1399487.63 us??????????
>> 1889????????? LK
>> ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us?????????
>> 10687???? READDIR
>> ????? 0.99??? 2426.24 us????? 12.47 us 24176123.13 us????????
>> 809430??? GETXATTR
>> ????? 1.45???? 676.75 us?????? 9.30 us 193177067.25 us???????
>> 4260021???? ENTRYLK
>> ????? 1.76?? 38324.59 us????? 21.44 us 46693575.93 us?????????
>> 91439??? READDIRP
>> ????? 4.44??? 1200.37 us?????? 9.10 us 2559377.90 us???????
>> 7352811???? INODELK
>> ????? 9.29??? 1848.94 us?????? 9.28 us 35299050.00 us???????
>> 9989221????? LOOKUP
>> ???? 79.24?? 47792.42 us????? 76.77 us 10488824.26 us???????
>> 3294642???? XATTROP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 3194806????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 5311???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>> 54??? CI_XATTR
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 1907?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 3189457?? CI_FORGET
>> ?
>> ??? Duration: 209524 seconds
>> ?? Data Read: 4775365531 bytes
>> Data Written: 2354108664 bytes
>> ?
>> Brick: zoidberg:/mnt/gluster/b
>> -----------------------------------------
>> Cumulative Stats:
>> ?? Block Size:????????????????? 1b+??????????????????
>> 2b+?????????????????? 8b+
>> ?No. of Reads:??????????????????? 0????????????????????
>> 0???????????????????? 0
>> No. of Writes:?????????????????? 53????????????????????
>> 1??????????????????? 60
>> ?
>> ?? Block Size:???????????????? 16b+?????????????????
>> 32b+????????????????? 64b+
>> ?No. of Reads:??????????????????? 0??????????????????
>> 213???????????????????? 2
>> No. of Writes:?????????????????? 14?????????????????
>> 1057??????????????????? 15
>> ?
>> ?? Block Size:??????????????? 128b+????????????????
>> 256b+???????????????? 512b+
>> ?No. of Reads:??????????????????? 0???????????????????
>> 41?????????????????? 146
>> No. of Writes:?????????????????? 59??????????????????
>> 715??????????????????? 80
>> ?
>> ?? Block Size:?????????????? 1024b+???????????????
>> 2048b+??????????????? 4096b+
>> ?No. of Reads:????????????????? 224????????????????????
>> 9??????????????????? 15
>> No. of Writes:????????????????? 163??????????????????
>> 282?????????????????? 404
>> ?
>> ?? Block Size:?????????????? 8192b+??????????????
>> 16384b+?????????????? 32768b+
>> ?No. of Reads:?????????????????? 52???????????????????
>> 74?????????????????? 210
>> No. of Writes:???????????????? 7415?????????????????
>> 1430????????????????? 6860
>> ?
>> ?? Block Size:????????????? 65536b+?????????????
>> 131072b+????????????? 262144b+
>> ?No. of Reads:????????????????? 299????????????????
>> 15313???????????????????? 1
>> No. of Writes:???????????????? 2458????????????????
>> 12378???????????????????? 1
>> ?
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 686733????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 837434???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 87338? RELEASEDIR
>> ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us???????????
>> 116???????? IPC
>> ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 us????????????
>> 57?? FTRUNCATE
>> ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 us????????????
>> 65?? FGETXATTR
>> ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 us????????????
>> 23 REMOVEXATTR
>> ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us???????????
>> 121??????? LINK
>> ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us???????????
>> 490????????? LK
>> ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us???????????
>> 540??????? STAT
>> ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us???????????
>> 200??? TRUNCATE
>> ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us??????????
>> 1482???? SETATTR
>> ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 us????????????
>> 12?????? FSYNC
>> ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us?????????
>> 10047?????? FLUSH
>> ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us??????????
>> 9977??? FINODELK
>> ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us???????????
>> 188????? RENAME
>> ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 us????????????
>> 25???? SYMLINK
>> ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us???????????
>> 118?????? MKNOD
>> ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us?????????
>> 12587?????? FSTAT
>> ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us?????????
>> 21695????? STATFS
>> ????? 0.00???? 307.86 us????? 28.00 us? 292666.83 us?????????
>> 16841??????? READ
>> ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us???????????
>> 178?????? MKDIR
>> ????? 0.00???? 319.24 us????? 51.19 us? 136353.39 us?????????
>> 33445?????? WRITE
>> ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us?????????
>> 87338???? OPENDIR
>> ????? 0.01???? 249.96 us????? 58.42 us? 594774.86 us?????????
>> 85089??? SETXATTR
>> ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us??????????
>> 3180?????? RMDIR
>> ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us??????????
>> 6067????? CREATE
>> ????? 0.05????? 89.45 us????? 30.14 us? 687961.60 us???????
>> 1219542??? READLINK
>> ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us????????
>> 831371??????? OPEN
>> ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us??????????
>> 9493??? FXATTROP
>> ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us???????
>> 3764714???? ENTRYLK
>> ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us?????????
>> 22253???? READDIR
>> ????? 1.06??? 6913.71 us????? 11.58 us 36968532.72 us????????
>> 364963??? GETXATTR
>> ????? 1.29?? 62130.12 us????? 36.89 us 8649871.11 us?????????
>> 49472??? READDIRP
>> ????? 2.89??? 1477.53 us????? 11.10 us 2807637.64 us???????
>> 4650500???? INODELK
>> ????? 4.12?? 23681.31 us????? 20.98 us 23291414.50 us????????
>> 414352????? UNLINK
>> ???? 16.09??? 5000.31 us?????? 9.04 us 23866016.88 us???????
>> 7655621????? LOOKUP
>> ???? 72.04?? 68953.31 us????? 68.40 us 27542499.65 us???????
>> 2486126???? XATTROP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 2433405????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 2453???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>> 69??? CI_XATTR
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 1544?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 2430885?? CI_FORGET
>> ?
>> ??? Duration: 209524 seconds
>> ?? Data Read: 2051388524 bytes
>> Data Written: 2208261499 bytes
>> ?
>> Interval 0 Stats:
>> ?? Block Size:????????????????? 1b+??????????????????
>> 2b+?????????????????? 8b+
>> ?No. of Reads:??????????????????? 0????????????????????
>> 0???????????????????? 0
>> No. of Writes:?????????????????? 53????????????????????
>> 1??????????????????? 60
>> ?
>> ?? Block Size:???????????????? 16b+?????????????????
>> 32b+????????????????? 64b+
>> ?No. of Reads:??????????????????? 0??????????????????
>> 213???????????????????? 2
>> No. of Writes:?????????????????? 14?????????????????
>> 1057??????????????????? 15
>> ?
>> ?? Block Size:??????????????? 128b+????????????????
>> 256b+???????????????? 512b+
>> ?No. of Reads:??????????????????? 0???????????????????
>> 41?????????????????? 146
>> No. of Writes:?????????????????? 59??????????????????
>> 715??????????????????? 80
>> ?
>> ?? Block Size:?????????????? 1024b+???????????????
>> 2048b+??????????????? 4096b+
>> ?No. of Reads:????????????????? 224????????????????????
>> 9??????????????????? 15
>> No. of Writes:????????????????? 163??????????????????
>> 282?????????????????? 404
>> ?
>> ?? Block Size:?????????????? 8192b+??????????????
>> 16384b+?????????????? 32768b+
>> ?No. of Reads:?????????????????? 52???????????????????
>> 74?????????????????? 210
>> No. of Writes:???????????????? 7415?????????????????
>> 1430????????????????? 6860
>> ?
>> ?? Block Size:????????????? 65536b+?????????????
>> 131072b+????????????? 262144b+
>> ?No. of Reads:????????????????? 299????????????????
>> 15313???????????????????? 1
>> No. of Writes:???????????????? 2458????????????????
>> 12378???????????????????? 1
>> ?
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 686733????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 837434???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 87338? RELEASEDIR
>> ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us???????????
>> 116???????? IPC
>> ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 us????????????
>> 57?? FTRUNCATE
>> ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 us????????????
>> 65?? FGETXATTR
>> ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 us????????????
>> 23 REMOVEXATTR
>> ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us???????????
>> 121??????? LINK
>> ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us???????????
>> 490????????? LK
>> ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us???????????
>> 540??????? STAT
>> ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us???????????
>> 200??? TRUNCATE
>> ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us??????????
>> 1482???? SETATTR
>> ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 us????????????
>> 12?????? FSYNC
>> ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us?????????
>> 10047?????? FLUSH
>> ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us??????????
>> 9977??? FINODELK
>> ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us???????????
>> 188????? RENAME
>> ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 us????????????
>> 25???? SYMLINK
>> ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us???????????
>> 118?????? MKNOD
>> ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us?????????
>> 12587?????? FSTAT
>> ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us?????????
>> 21695????? STATFS
>> ????? 0.00???? 307.91 us????? 28.00 us? 292666.83 us?????????
>> 16841??????? READ
>> ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us???????????
>> 178?????? MKDIR
>> ????? 0.00???? 319.22 us????? 51.19 us? 136353.39 us?????????
>> 33445?????? WRITE
>> ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us?????????
>> 87338???? OPENDIR
>> ????? 0.01???? 249.95 us????? 58.42 us? 594774.86 us?????????
>> 85089??? SETXATTR
>> ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us??????????
>> 3180?????? RMDIR
>> ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us??????????
>> 6067????? CREATE
>> ????? 0.05????? 89.46 us????? 30.14 us? 687961.60 us???????
>> 1219542??? READLINK
>> ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us????????
>> 831371??????? OPEN
>> ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us??????????
>> 9493??? FXATTROP
>> ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us???????
>> 3764714???? ENTRYLK
>> ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us?????????
>> 22253???? READDIR
>> ????? 1.06??? 6913.26 us????? 11.58 us 36968532.72 us????????
>> 364963??? GETXATTR
>> ????? 1.29?? 62121.95 us????? 36.89 us 8649871.11 us?????????
>> 49472??? READDIRP
>> ????? 2.89??? 1477.58 us????? 11.10 us 2807637.64 us???????
>> 4650500???? INODELK
>> ????? 4.12?? 23681.29 us????? 20.98 us 23291414.50 us????????
>> 414352????? UNLINK
>> ???? 16.09??? 5000.02 us?????? 9.04 us 23866016.88 us???????
>> 7655621????? LOOKUP
>> ???? 72.04?? 68956.14 us????? 68.40 us 27542499.65 us???????
>> 2486126???? XATTROP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 2433405????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 2453???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>> 69??? CI_XATTR
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 1544?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 2430885?? CI_FORGET
>> ?
>> ??? Duration: 209524 seconds
>> ?? Data Read: 2051388524 bytes
>> Data Written: 2208261499 bytes
>> ?
>> Brick: flexo:/mnt/gluster/b
>> --------------------------------------
>> Cumulative Stats:
>> ?? Block Size:????????????????? 1b+??????????????????
>> 8b+????????????????? 16b+
>> ?No. of Reads:??????????????????? 0????????????????????
>> 0???????????????????? 0
>> No. of Writes:????????????????? 117??????????????????
>> 195?????????????????? 367
>> ?
>> ?? Block Size:???????????????? 32b+?????????????????
>> 64b+???????????????? 128b+
>> ?No. of Reads:????????????????? 130????????????????????
>> 0???????????????????? 3
>> No. of Writes:???????????????? 1012????????????????????
>> 9?????????????????? 134
>> ?
>> ?? Block Size:??????????????? 256b+????????????????
>> 512b+??????????????? 1024b+
>> ?No. of Reads:?????????????????? 47????????????????????
>> 8?????????????????? 257
>> No. of Writes:????????????????? 729???????????????????
>> 73?????????????????? 168
>> ?
>> ?? Block Size:?????????????? 2048b+???????????????
>> 4096b+??????????????? 8192b+
>> ?No. of Reads:?????????????????? 10???????????????????
>> 27?????????????????? 240
>> No. of Writes:????????????????? 294??????????????????
>> 466???????????????? 10472
>> ?
>> ?? Block Size:????????????? 16384b+??????????????
>> 32768b+?????????????? 65536b+
>> ?No. of Reads:????????????????? 139??????????????????
>> 119?????????????????? 135
>> No. of Writes:???????????????? 1480?????????????????
>> 6273????????????????? 3935
>> ?
>> ?? Block Size:???????????? 131072b+
>> ?No. of Reads:???????????????? 7892
>> No. of Writes:??????????????? 12327
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 955058????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1450749???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 81377? RELEASEDIR
>> ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us???????????
>> 108???????? IPC
>> ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 us????????????
>> 44?? FTRUNCATE
>> ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 us????????????
>> 30 REMOVEXATTR
>> ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 us?????????????
>> 4?????? FSYNC
>> ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 us????????????
>> 30???? SYMLINK
>> ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us???????????
>> 137??????? LINK
>> ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us???????????
>> 119?????? MKNOD
>> ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us???????????
>> 221??? SETXATTR
>> ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us???????????
>> 354??????? STAT
>> ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us???????????
>> 777????????? LK
>> ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us???????????
>> 188????? RENAME
>> ????? 0.00???? 150.40 us????? 75.59 us???? 749.54 us???????????
>> 810?? FGETXATTR
>> ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us???????????
>> 698??? TRUNCATE
>> ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us???????????
>> 178?????? MKDIR
>> ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us??????????
>> 1665???? SETATTR
>> ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us?????????
>> 11372??? FINODELK
>> ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us??????????
>> 7398?????? FSTAT
>> ????? 0.00???? 120.71 us????? 31.43 us???? 847.97 us??????????
>> 9094??????? READ
>> ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us?????????
>> 22614?????? FLUSH
>> ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us?????????
>> 21656????? STATFS
>> ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us??????????
>> 6145????? CREATE
>> ????? 0.00???? 217.55 us????? 50.60 us?? 30734.99 us?????????
>> 38051?????? WRITE
>> ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us?????????
>> 81377???? OPENDIR
>> ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us??????????
>> 3177?????? RMDIR
>> ????? 0.03???? 100.05 us????? 30.93 us?? 49756.93 us????????
>> 432439??? READLINK
>> ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us????????
>> 448381????? UNLINK
>> ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us?????????
>> 52185??? READDIRP
>> ????? 0.35???? 414.05 us????? 36.58 us 1038229.93 us???????
>> 1444606??????? OPEN
>> ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us?????????
>> 10745??? FXATTROP
>> ????? 0.72???? 776.21 us????? 11.24 us 4271792.57 us???????
>> 1605822??? GETXATTR
>> ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us?????????
>> 13532???? READDIR
>> ????? 0.79???? 320.44 us?????? 7.71 us 2369511.06 us???????
>> 4257297???? ENTRYLK
>> ????? 1.97???? 464.02 us?????? 9.38 us 1038460.80 us???????
>> 7333722???? INODELK
>> ???? 13.73??? 2372.50 us?????? 8.33 us 6258159.87 us???????
>> 9979799????? LOOKUP
>> ???? 80.79?? 42372.04 us????? 61.49 us 2238308.13 us???????
>> 3289103???? XATTROP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 2250268????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 3756???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>> 167??? CI_XATTR
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 1908?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 2246352?? CI_FORGET
>> ?
>> ??? Duration: 209334 seconds
>> ?? Data Read: 1059777395 bytes
>> Data Written: 2336921014 bytes
>> ?
>> Interval 0 Stats:
>> ?? Block Size:????????????????? 1b+??????????????????
>> 8b+????????????????? 16b+
>> ?No. of Reads:??????????????????? 0????????????????????
>> 0???????????????????? 0
>> No. of Writes:????????????????? 117??????????????????
>> 195?????????????????? 367
>> ?
>> ?? Block Size:???????????????? 32b+?????????????????
>> 64b+???????????????? 128b+
>> ?No. of Reads:????????????????? 130????????????????????
>> 0???????????????????? 3
>> No. of Writes:???????????????? 1012????????????????????
>> 9?????????????????? 134
>> ?
>> ?? Block Size:??????????????? 256b+????????????????
>> 512b+??????????????? 1024b+
>> ?No. of Reads:?????????????????? 47????????????????????
>> 8?????????????????? 257
>> No. of Writes:????????????????? 729???????????????????
>> 73?????????????????? 168
>> ?
>> ?? Block Size:?????????????? 2048b+???????????????
>> 4096b+??????????????? 8192b+
>> ?No. of Reads:?????????????????? 10???????????????????
>> 27?????????????????? 240
>> No. of Writes:????????????????? 294??????????????????
>> 466???????????????? 10472
>> ?
>> ?? Block Size:????????????? 16384b+??????????????
>> 32768b+?????????????? 65536b+
>> ?No. of Reads:????????????????? 139??????????????????
>> 119?????????????????? 135
>> No. of Writes:???????????????? 1480?????????????????
>> 6273????????????????? 3935
>> ?
>> ?? Block Size:???????????? 131072b+
>> ?No. of Reads:???????????????? 7892
>> No. of Writes:??????????????? 12327
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 955058????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1450749???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 81377? RELEASEDIR
>> ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us???????????
>> 108???????? IPC
>> ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 us????????????
>> 44?? FTRUNCATE
>> ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 us????????????
>> 30 REMOVEXATTR
>> ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 us?????????????
>> 4?????? FSYNC
>> ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 us????????????
>> 30???? SYMLINK
>> ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us???????????
>> 137??????? LINK
>> ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us???????????
>> 119?????? MKNOD
>> ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us???????????
>> 221??? SETXATTR
>> ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us???????????
>> 354??????? STAT
>> ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us???????????
>> 777????????? LK
>> ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us???????????
>> 188????? RENAME
>> ????? 0.00???? 150.43 us????? 75.59 us???? 749.54 us???????????
>> 810?? FGETXATTR
>> ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us???????????
>> 698??? TRUNCATE
>> ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us???????????
>> 178?????? MKDIR
>> ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us??????????
>> 1665???? SETATTR
>> ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us?????????
>> 11372??? FINODELK
>> ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us??????????
>> 7398?????? FSTAT
>> ????? 0.00???? 120.74 us????? 31.43 us???? 847.97 us??????????
>> 9094??????? READ
>> ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us?????????
>> 22614?????? FLUSH
>> ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us?????????
>> 21656????? STATFS
>> ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us??????????
>> 6145????? CREATE
>> ????? 0.00???? 217.50 us????? 50.60 us?? 30734.99 us?????????
>> 38051?????? WRITE
>> ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us?????????
>> 81377???? OPENDIR
>> ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us??????????
>> 3177?????? RMDIR
>> ????? 0.03???? 100.06 us????? 30.93 us?? 49756.93 us????????
>> 432439??? READLINK
>> ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us????????
>> 448381????? UNLINK
>> ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us?????????
>> 52185??? READDIRP
>> ????? 0.35???? 414.01 us????? 36.58 us 1038229.93 us???????
>> 1444606??????? OPEN
>> ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us?????????
>> 10745??? FXATTROP
>> ????? 0.72???? 776.24 us????? 11.24 us 4271792.57 us???????
>> 1605822??? GETXATTR
>> ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us?????????
>> 13532???? READDIR
>> ????? 0.79???? 320.45 us?????? 7.71 us 2369511.06 us???????
>> 4257297???? ENTRYLK
>> ????? 1.97???? 464.07 us?????? 9.38 us 1038460.80 us???????
>> 7333722???? INODELK
>> ???? 13.72??? 2372.42 us?????? 8.33 us 6258159.87 us???????
>> 9979799????? LOOKUP
>> ???? 80.79?? 42373.50 us????? 61.49 us 2238308.13 us???????
>> 3289103???? XATTROP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 2250268????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 3756???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>> 167??? CI_XATTR
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 1908?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 2246352?? CI_FORGET
>> ?
>> ??? Duration: 209334 seconds
>> ?? Data Read: 1059777395 bytes
>> Data Written: 2336921014 bytes
>> ?
>> Brick: flexo:/mnt/gluster/c
>> --------------------------------------
>> Cumulative Stats:
>> ?? Block Size:????????????????? 1b+
>> ?No. of Reads:??????????????????? 0
>> No. of Writes:??????????????? 32843
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 686562????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 834596???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 93757? RELEASEDIR
>> ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us???????????
>> 107???????? IPC
>> ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 us????????????
>> 36??????? LINK
>> ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 us????????????
>> 65?? FGETXATTR
>> ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 us????????????
>> 88????? RENAME
>> ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us???????????
>> 153??? TRUNCATE
>> ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us???????????
>> 491????????? LK
>> ????? 0.01????? 51.73 us?????? 7.70 us??? 1174.64 us??????????
>> 9339?????? FLUSH
>> ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us??????????
>> 9011??? FINODELK
>> ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us?????????
>> 10824?????? FSTAT
>> ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us??????????
>> 2594?????? MKDIR
>> ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us??????????
>> 9009??? FXATTROP
>> ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us??????????
>> 5754????? CREATE
>> ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us?????????
>> 32843?????? WRITE
>> ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us??????????
>> 5798???? SYMLINK
>> ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us?????????
>> 34904???? READDIR
>> ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us??????????
>> 2732?????? RMDIR
>> ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us?????????
>> 93757???? OPENDIR
>> ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us????????
>> 170292????? UNLINK
>> ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us?????????
>> 13840??? READDIRP
>> ????? 1.94???? 200.88 us????? 25.49 us? 461654.32 us????????
>> 468842?? FTRUNCATE
>> ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us????????
>> 486882 REMOVEXATTR
>> ????? 2.27???? 223.01 us????? 52.03 us? 178897.40 us????????
>> 493240??? SETXATTR
>> ????? 3.53???? 155.31 us?????? 7.96 us?? 67873.00 us???????
>> 1101338??? GETXATTR
>> ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us????????
>> 829002??????? OPEN
>> ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us???????
>> 3660425???? ENTRYLK
>> ????? 6.23???? 304.93 us????? 42.51 us? 661394.43 us????????
>> 990644???? SETATTR
>> ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us????????
>> 990766???? XATTROP
>> ???? 12.39???? 549.11 us???? 135.57 us 1659697.83 us???????
>> 1093387?????? MKNOD
>> ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us???????
>> 4325281???? INODELK
>> ???? 39.53???? 206.94 us?????? 8.56 us? 964823.33 us???????
>> 9259429????? LOOKUP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1856377????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 933300???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>> 251??? CI_XATTR
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>> 293?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 923077?? CI_FORGET
>> ?
>> ??? Duration: 209334 seconds
>> ?? Data Read: 0 bytes
>> Data Written: 32843 bytes
>> ?
>> Interval 0 Stats:
>> ?? Block Size:????????????????? 1b+
>> ?No. of Reads:??????????????????? 0
>> No. of Writes:??????????????? 32843
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 686562????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 834596???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 93757? RELEASEDIR
>> ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us???????????
>> 107???????? IPC
>> ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 us????????????
>> 36??????? LINK
>> ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 us????????????
>> 65?? FGETXATTR
>> ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 us????????????
>> 88????? RENAME
>> ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us???????????
>> 153??? TRUNCATE
>> ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us???????????
>> 491????????? LK
>> ????? 0.01????? 51.71 us?????? 7.70 us??? 1174.64 us??????????
>> 9339?????? FLUSH
>> ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us??????????
>> 9011??? FINODELK
>> ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us?????????
>> 10824?????? FSTAT
>> ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us??????????
>> 2594?????? MKDIR
>> ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us??????????
>> 9009??? FXATTROP
>> ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us??????????
>> 5754????? CREATE
>> ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us?????????
>> 32843?????? WRITE
>> ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us??????????
>> 5798???? SYMLINK
>> ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us?????????
>> 34904???? READDIR
>> ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us??????????
>> 2732?????? RMDIR
>> ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us?????????
>> 93757???? OPENDIR
>> ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us????????
>> 170292????? UNLINK
>> ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us?????????
>> 13840??? READDIRP
>> ????? 1.94???? 200.87 us????? 25.49 us? 461654.32 us????????
>> 468842?? FTRUNCATE
>> ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us????????
>> 486882 REMOVEXATTR
>> ????? 2.27???? 223.00 us????? 52.03 us? 178897.40 us????????
>> 493240??? SETXATTR
>> ????? 3.53???? 155.32 us?????? 7.96 us?? 67873.00 us???????
>> 1101338??? GETXATTR
>> ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us????????
>> 829002??????? OPEN
>> ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us???????
>> 3660425???? ENTRYLK
>> ????? 6.23???? 304.88 us????? 42.51 us? 661394.43 us????????
>> 990644???? SETATTR
>> ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us????????
>> 990766???? XATTROP
>> ???? 12.39???? 549.25 us???? 135.57 us 1659697.83 us???????
>> 1093387?????? MKNOD
>> ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us???????
>> 4325281???? INODELK
>> ???? 39.53???? 206.95 us?????? 8.56 us? 964823.33 us???????
>> 9259429????? LOOKUP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1856377????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 933300???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>> 251??? CI_XATTR
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>> 293?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 923077?? CI_FORGET
>> ?
>> ??? Duration: 209334 seconds
>> ?? Data Read: 0 bytes
>> Data Written: 32843 bytes
>> ?
>> Brick: bender:/mnt/gluster/c
>> ---------------------------------------
>> Cumulative Stats:
>> ?? Block Size:????????????????? 1b+
>> ?No. of Reads:??????????????????? 0
>> No. of Writes:??????????????? 37050
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 928119????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1401524???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 70092? RELEASEDIR
>> ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us???????????
>> 100???????? IPC
>> ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 us????????????
>> 38??????? LINK
>> ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 us????????????
>> 30???? SYMLINK
>> ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 us?????????????
>> 4?????? FSYNC
>> ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us???????????
>> 122????? RENAME
>> ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us???????????
>> 777????????? LK
>> ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us???????????
>> 810?? FGETXATTR
>> ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us???????????
>> 603??? TRUNCATE
>> ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us???????????
>> 302?????? MKDIR
>> ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us?????????
>> 10060??? FINODELK
>> ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us??????????
>> 6409?????? FSTAT
>> ????? 0.01????? 49.38 us?????? 8.08 us??? 1264.41 us?????????
>> 21719?????? FLUSH
>> ????? 0.03????? 74.76 us????? 20.86 us??? 2008.41 us?????????
>> 37050?????? WRITE
>> ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us??????????
>> 5901????? CREATE
>> ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us?????????
>> 10057??? FXATTROP
>> ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us?????????
>> 70092???? OPENDIR
>> ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us?????????
>> 10779???? READDIR
>> ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us??????????
>> 3008?????? RMDIR
>> ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us????????
>> 109478????? UNLINK
>> ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us????????
>> 722669 REMOVEXATTR
>> ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us????????
>> 789294??? GETXATTR
>> ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us????????
>> 718835?? FTRUNCATE
>> ????? 1.89???? 219.47 us????? 49.92 us? 506767.78 us????????
>> 722856??? SETXATTR
>> ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us???????
>> 1395957??????? OPEN
>> ????? 5.56???? 322.08 us????? 43.37 us? 634069.61 us???????
>> 1450160???? SETATTR
>> ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us???????
>> 4233520???? ENTRYLK
>> ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us???????
>> 1449601???? XATTROP
>> ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us???????
>> 7097226???? INODELK
>> ???? 19.37??? 1104.98 us???? 101.28 us 1612346.75 us???????
>> 1472813?????? MKNOD
>> ???? 41.85???? 295.61 us?????? 8.98 us? 996805.54 us??????
>> 11896091????? LOOKUP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 3531139????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1862470???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>> 244??? CI_XATTR
>> ?????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>> 632?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1668669?? CI_FORGET
>> ?
>> ??? Duration: 208643 seconds
>> ?? Data Read: 0 bytes
>> Data Written: 37050 bytes
>> ?
>> Interval 0 Stats:
>> ?? Block Size:????????????????? 1b+
>> ?No. of Reads:??????????????????? 0
>> No. of Writes:??????????????? 37050
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 928119????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1401524???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 70092? RELEASEDIR
>> ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us???????????
>> 100???????? IPC
>> ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 us????????????
>> 38??????? LINK
>> ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 us????????????
>> 30???? SYMLINK
>> ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 us?????????????
>> 4?????? FSYNC
>> ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us???????????
>> 122????? RENAME
>> ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us???????????
>> 777????????? LK
>> ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us???????????
>> 810?? FGETXATTR
>> ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us???????????
>> 603??? TRUNCATE
>> ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us???????????
>> 302?????? MKDIR
>> ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us?????????
>> 10060??? FINODELK
>> ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us??????????
>> 6409?????? FSTAT
>> ????? 0.01????? 49.39 us?????? 8.08 us??? 1264.41 us?????????
>> 21719?????? FLUSH
>> ????? 0.03????? 74.77 us????? 20.86 us??? 2008.41 us?????????
>> 37050?????? WRITE
>> ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us??????????
>> 5901????? CREATE
>> ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us?????????
>> 10057??? FXATTROP
>> ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us?????????
>> 70092???? OPENDIR
>> ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us?????????
>> 10779???? READDIR
>> ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us??????????
>> 3008?????? RMDIR
>> ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us????????
>> 109478????? UNLINK
>> ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us????????
>> 722669 REMOVEXATTR
>> ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us????????
>> 789294??? GETXATTR
>> ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us????????
>> 718835?? FTRUNCATE
>> ????? 1.89???? 219.48 us????? 49.92 us? 506767.78 us????????
>> 722856??? SETXATTR
>> ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us???????
>> 1395957??????? OPEN
>> ????? 5.56???? 322.09 us????? 43.37 us? 634069.61 us???????
>> 1450160???? SETATTR
>> ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us???????
>> 4233520???? ENTRYLK
>> ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us???????
>> 1449601???? XATTROP
>> ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us???????
>> 7097226???? INODELK
>> ???? 19.37??? 1105.06 us???? 101.28 us 1612346.75 us???????
>> 1472813?????? MKNOD
>> ???? 41.84???? 295.54 us?????? 8.98 us? 996805.54 us??????
>> 11896091????? LOOKUP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 3531139????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1862470???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>> 244??? CI_XATTR
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????????
>> 632?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1668669?? CI_FORGET
>> ?
>> ??? Duration: 208643 seconds
>> ?? Data Read: 0 bytes
>> Data Written: 37050 bytes
>> ?
>> Brick: bender:/mnt/gluster/a
>> ---------------------------------------
>> Cumulative Stats:
>> ?? Block Size:????????????????? 1b+??????????????????
>> 2b+?????????????????? 8b+
>> ?No. of Reads:??????????????????? 0????????????????????
>> 0???????????????????? 0
>> No. of Writes:?????????????????? 53????????????????????
>> 1??????????????????? 60
>> ?
>> ?? Block Size:???????????????? 16b+?????????????????
>> 32b+????????????????? 64b+
>> ?No. of Reads:??????????????????? 0??????????????????
>> 235???????????????????? 4
>> No. of Writes:?????????????????? 14?????????????????
>> 1033??????????????????? 15
>> ?
>> ?? Block Size:??????????????? 128b+????????????????
>> 256b+???????????????? 512b+
>> ?No. of Reads:??????????????????? 4??????????????????
>> 115??????????????????? 81
>> No. of Writes:?????????????????? 58??????????????????
>> 703??????????????????? 80
>> ?
>> ?? Block Size:?????????????? 1024b+???????????????
>> 2048b+??????????????? 4096b+
>> ?No. of Reads:????????????????? 181???????????????????
>> 26??????????????????? 74
>> No. of Writes:????????????????? 163??????????????????
>> 278?????????????????? 397
>> ?
>> ?? Block Size:?????????????? 8192b+??????????????
>> 16384b+?????????????? 32768b+
>> ?No. of Reads:????????????????? 131??????????????????
>> 161?????????????????? 267
>> No. of Writes:???????????????? 7264?????????????????
>> 1398????????????????? 6742
>> ?
>> ?? Block Size:????????????? 65536b+?????????????
>> 131072b+????????????? 262144b+
>> ?No. of Reads:????????????????? 443????????????????
>> 25528???????????????????? 0
>> No. of Writes:???????????????? 2426????????????????
>> 12240???????????????????? 1
>> ?
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 684942????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 837092???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 87297? RELEASEDIR
>> ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 us?????????????
>> 8 REMOVEXATTR
>> ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 us????????????
>> 97???????? IPC
>> ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 us????????????
>> 46?? FTRUNCATE
>> ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 us????????????
>> 65?? FGETXATTR
>> ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us???????????
>> 121??????? LINK
>> ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 us????????????
>> 25???? SYMLINK
>> ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us???????????
>> 107?????? MKNOD
>> ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 us?????????????
>> 1?????? FSYNC
>> ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us???????????
>> 188????? RENAME
>> ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us???????????
>> 745??????? STAT
>> ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us???????????
>> 200??? TRUNCATE
>> ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us??????????
>> 1444???? SETATTR
>> ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us???????????
>> 177?????? MKDIR
>> ????? 0.00????? 52.68 us?????? 7.28 us???? 826.01 us??????????
>> 9890?????? FLUSH
>> ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us??????????
>> 9792??? FINODELK
>> ????? 0.00????? 74.26 us????? 20.00 us??? 1522.39 us?????????
>> 21521????? STATFS
>> ????? 0.00???? 129.21 us????? 34.15 us??? 9068.67 us?????????
>> 13094?????? FSTAT
>> ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us??????????
>> 5967????? CREATE
>> ????? 0.00???? 112.06 us????? 28.14 us??? 9545.53 us?????????
>> 27474??????? READ
>> ????? 0.01???? 313.76 us????? 52.98 us?? 71635.74 us?????????
>> 32926?????? WRITE
>> ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us?????????
>> 87297???? OPENDIR
>> ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us??????????
>> 3177?????? RMDIR
>> ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us?????????
>> 85073??? SETXATTR
>> ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us?????????
>> 15090??? READDIRP
>> ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us???????????
>> 564????????? LK
>> ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us????????
>> 850938??? READLINK
>> ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us????????
>> 831212??????? OPEN
>> ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us?????????
>> 22482???? READDIR
>> ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us????????
>> 415226????? UNLINK
>> ????? 0.54??? 1272.83 us?????? 9.66 us 1604368.93 us????????
>> 686945??? GETXATTR
>> ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us???????
>> 3739718???? ENTRYLK
>> ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us??????????
>> 9311??? FXATTROP
>> ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us???????
>> 4649291???? INODELK
>> ????? 5.82??? 1240.20 us?????? 8.78 us 1526868.50 us???????
>> 7625383????? LOOKUP
>> ???? 88.60?? 58005.98 us????? 67.31 us 1448810.06 us???????
>> 2481914???? XATTROP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1396875????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 1854???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>> 28??? CI_XATTR
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 1051?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1394996?? CI_FORGET
>> ?
>> ??? Duration: 208643 seconds
>> ?? Data Read: 3409342848 bytes
>> Data Written: 2180686507 bytes
>> ?
>> Interval 0 Stats:
>> ?? Block Size:????????????????? 1b+??????????????????
>> 2b+?????????????????? 8b+
>> ?No. of Reads:??????????????????? 0????????????????????
>> 0???????????????????? 0
>> No. of Writes:?????????????????? 53????????????????????
>> 1??????????????????? 60
>> ?
>> ?? Block Size:???????????????? 16b+?????????????????
>> 32b+????????????????? 64b+
>> ?No. of Reads:??????????????????? 0??????????????????
>> 235???????????????????? 4
>> No. of Writes:?????????????????? 14?????????????????
>> 1033??????????????????? 15
>> ?
>> ?? Block Size:??????????????? 128b+????????????????
>> 256b+???????????????? 512b+
>> ?No. of Reads:??????????????????? 4??????????????????
>> 115??????????????????? 81
>> No. of Writes:?????????????????? 58??????????????????
>> 703??????????????????? 80
>> ?
>> ?? Block Size:?????????????? 1024b+???????????????
>> 2048b+??????????????? 4096b+
>> ?No. of Reads:????????????????? 181???????????????????
>> 26??????????????????? 74
>> No. of Writes:????????????????? 163??????????????????
>> 278?????????????????? 397
>> ?
>> ?? Block Size:?????????????? 8192b+??????????????
>> 16384b+?????????????? 32768b+
>> ?No. of Reads:????????????????? 131??????????????????
>> 161?????????????????? 267
>> No. of Writes:???????????????? 7264?????????????????
>> 1398????????????????? 6742
>> ?
>> ?? Block Size:????????????? 65536b+?????????????
>> 131072b+????????????? 262144b+
>> ?No. of Reads:????????????????? 443????????????????
>> 25528???????????????????? 0
>> No. of Writes:???????????????? 2426????????????????
>> 12240???????????????????? 1
>> ?
>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of
>> calls???????? Fop
>> ?---------?? -----------?? -----------?? -----------??
>> ------------??????? ----
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 684942????? FORGET
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????
>> 837092???? RELEASE
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????
>> 87297? RELEASEDIR
>> ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 us?????????????
>> 8 REMOVEXATTR
>> ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 us????????????
>> 97???????? IPC
>> ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 us????????????
>> 46?? FTRUNCATE
>> ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 us????????????
>> 65?? FGETXATTR
>> ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us???????????
>> 121??????? LINK
>> ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 us????????????
>> 25???? SYMLINK
>> ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us???????????
>> 107?????? MKNOD
>> ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 us?????????????
>> 1?????? FSYNC
>> ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us???????????
>> 188????? RENAME
>> ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us???????????
>> 745??????? STAT
>> ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us???????????
>> 200??? TRUNCATE
>> ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us??????????
>> 1444???? SETATTR
>> ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us???????????
>> 177?????? MKDIR
>> ????? 0.00????? 52.69 us?????? 7.28 us???? 826.01 us??????????
>> 9890?????? FLUSH
>> ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us??????????
>> 9792??? FINODELK
>> ????? 0.00????? 74.25 us????? 20.00 us??? 1522.39 us?????????
>> 21521????? STATFS
>> ????? 0.00???? 129.23 us????? 34.15 us??? 9068.67 us?????????
>> 13094?????? FSTAT
>> ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us??????????
>> 5967????? CREATE
>> ????? 0.00???? 112.45 us????? 28.14 us??? 9545.53 us?????????
>> 27474??????? READ
>> ????? 0.01???? 313.77 us????? 52.98 us?? 71635.74 us?????????
>> 32926?????? WRITE
>> ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us?????????
>> 87297???? OPENDIR
>> ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us??????????
>> 3177?????? RMDIR
>> ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us?????????
>> 85073??? SETXATTR
>> ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us?????????
>> 15090??? READDIRP
>> ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us???????????
>> 564????????? LK
>> ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us????????
>> 850938??? READLINK
>> ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us????????
>> 831212??????? OPEN
>> ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us?????????
>> 22482???? READDIR
>> ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us????????
>> 415226????? UNLINK
>> ????? 0.54??? 1272.79 us?????? 9.66 us 1604368.93 us????????
>> 686945??? GETXATTR
>> ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us???????
>> 3739718???? ENTRYLK
>> ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us??????????
>> 9311??? FXATTROP
>> ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us???????
>> 4649291???? INODELK
>> ????? 5.82??? 1240.10 us?????? 8.78 us 1526868.50 us???????
>> 7625383????? LOOKUP
>> ???? 88.60?? 58006.02 us????? 67.31 us 1448810.06 us???????
>> 2481914???? XATTROP
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1396875????? UPCALL
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 1854???? CI_IATT
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????????
>> 28??? CI_XATTR
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????
>> 1051?? CI_UNLINK
>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????
>> 1394996?? CI_FORGET
>> ?
>> ??? Duration: 208643 seconds
>> ?? Data Read: 3409342848 bytes
>> Data Written: 2180686507 bytes
>> ?
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200507/0ddc505a/attachment.html>
On 08/05/20 1:10 am, Jaco Kroon wrote:> > Hi, > > So upgrading to 7.5 made a difference in that it takes a bit longer to > cause chaos, but it still does the exact same thing. > > To be precise: > > I've now had to set: > > gluster volume set vpbx_shared cluster.heal-wait-queue-length 1 > gluster volume set vpbx_shared cluster.shd-wait-qlength 1 > gluster volume set vpbx_shared cluster.shd-max-threads 1 > > With any of these settings >1 I end up in a situation where simple > directory listings of < 100 entries in a folder takes upwards of a > minute to complete *from time to time*. >Hi Jaco, these options are to be increased only if your hardware is capable of handling the load.It was contributed initially by facebook as it worked for them. See https://bugzilla.redhat.com/show_bug.cgi?id=1221737#c0.? So I think this is expected. Unless you are saying that the `ls` slowness happens even when there is no self-heal going on, in which case we need to see what the issue is. Regards, Ravi> It seems that when the logs > > With these the maximum stall I can reproduce is usually around 3 > seconds before the directory listing will complete, but about 5s worst > case.? I've now reverted these as well, which are known for us to > cause issues under heal required conditions: > > cluster.metadata-self-heal on > cluster.data-self-heal on > cluster.entry-self-heal on > > Not sure if I need to remount for these to become effective but I've > done so just to be on the safe side. > > I'm still seeing these errors: > > [2020-05-07 19:20:12.763311] W [MSGID: 114031] > [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] > 0-vpbx_shared-client-1: remote operation failed [Invalid argument] > [2020-05-07 19:20:12.995675] W [MSGID: 114031] > [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] > 0-vpbx_shared-client-1: remote operation failed [Invalid argument] > [2020-05-07 19:20:59.199776] W [MSGID: 114031] > [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] > 0-vpbx_shared-client-1: remote operation failed [Invalid argument] > > And the start of my issues still seem to co-incide with that. Sample case: > > # time ls > ... > real??? 0m5.140s > user??? 0m0.009s > sys??? 0m0.000s > # f=(*); echo "${#f[@]}" > 165 > > This just does not seem right.? Note, again, if I set *any* of the > above parameters >1 these delays get far, far worse (I've had over a > minute for this exact same ls). > > Any advise would be greatly appreciated.? To me, it looks like some > lock is being taken somewhere, and takes a while to release.? I don't > understand the glusterfs internals nearly well enough to understand > exactly what's going on here, nor do I know what information to > provide to help drill down on this.? Happy to go look at code, if I > can get pointed in a direction. > > Kind Regards, > Jaco > > On 2020/05/05 08:43, Jaco Kroon wrote: >> >> Hi Artem, >> >> I'll do that.? In the meantime consider this (inferred from the >> changelogs). >> >> It seems like glusterfs SHD is queueing into a bounded queue under >> lock conditions, and waiting for space to open up.? At least, this is >> what's hinted in the changelogs, I've been trying to grasp the SHD >> code for two days now and am not getting to wrap my head around >> that.? If this is true, then the only way to avoid lock-ups are to >> ensure that the queue is larger than the total number of files that >> can ever require healing (effectively making the queue unbounded). >> >> Kind Regards, >> Jaco >> >> On 2020/05/04 21:56, Artem Russakovskii wrote: >>> I reported something similar recently to this list, and I highly >>> advise you to update to 7.5 (in my case it was 5.11 -> 5.13), then >>> see if it keeps happening. The self heal issues should be fixed >>> there properly at least. >>> >>> Sincerely, >>> Artem >>> >>> -- >>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror >>> <http://www.apkmirror.com/>, Illogical Robot LLC >>> beerpla.net <http://beerpla.net/> | @ArtemR <http://twitter.com/ArtemR> >>> >>> >>> On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za >>> <mailto:jaco at uls.co.za>> wrote: >>> >>> Hi All, >>> >>> So we've got a cluster currently running on glusterfs 7.3 >>> >>> Heal performance is absolutely terrible (it starts of great, >>> runs for a while then completely trips up). >>> >>> Once the SHD trips up, access speeds to the cluster also becomes >>> poor.? And with poor I mean directory listings that normally >>> takes well under 500ms takes upwards of 10s, but I've seen >>> upwards of 40s (although a follow up ls immediately after is >>> nice and fast again, temporarily).? With all SHDs killed this >>> particular ls is as per normal (around 30-50ms). >>> >>> The SHD logs doesn't provide any usable clues, the only variance >>> in logs is this, just before all daemons across the cluster >>> crawls to a near halt. >>> >>> [2020-05-02 19:55:58.627036] I [MSGID: 108026] >>> [afr-self-heal-common.c:1747:afr_log_selfheal] >>> 0-vpbx_shared-replicate-0: Completed entry selfheal on >>> 5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1 >>> [2020-05-02 19:55:58.627058] I [MSGID: 108026] >>> [afr-self-heal-common.c:1747:afr_log_selfheal] >>> 0-vpbx_shared-replicate-0: Completed entry selfheal on >>> c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1 >>> [2020-05-02 19:55:59.125001] W [MSGID: 114031] >>> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] >>> 0-vpbx_shared-client-1: remote operation failed [Invalid argument] >>> [2020-05-02 19:55:59.339803] I [MSGID: 108026] >>> [afr-self-heal-common.c:1747:afr_log_selfheal] >>> 0-vpbx_shared-replicate-0: Completed entry selfheal on >>> 2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1 >>> [2020-05-02 19:55:59.490387] I [MSGID: 108026] >>> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] >>> 0-vpbx_shared-replicate-0: performing metadata selfheal on >>> 527a63a3-9acd-4c3f-b95c-5c719325c973 >>> >>> A few more entries will complete overall and then it'll be a >>> file every other second.? Once I saw this I ran a few more >>> tests, and on every test just prior to slowing to a crawl, that >>> entry is there. >>> >>> There is a perceived improvement when lowering the thread count, >>> but not significant enough to confirm a correlation.? Even >>> with**cluster.shd-max-threads=1 this still happens (gets a few >>> tens thousand files completed instead of few hundred, at 32 it >>> gets to maybe 400 to 500 before it stalls, 16 does maybe a >>> thousand).* >>> * >>> >>> There are probably performant bursts hereafter, because from >>> experience we know this process takes about two months.? But >>> this time around it seems to be much, much worse.? Might be >>> because that was healing a single brick to another brick, so >>> less locking going? At the moment the replicas are basically >>> trying to populate the newly added arbiters.? Also, we haven't >>> messed as greatly with the health check settings (queues and >>> threads and locks nearly as much as since the arbiter addition >>> because performance has simply dropped to non-acceptable levels. >>> >>> Unfortunately granular entry heal isn't on at the moment (nor >>> can I switch it on due to a few million outstanding heals).? >>> This used to be a 2x2 cluster. We then moved one of the bricks >>> to a newly added server using replace-brick.? What a frightening >>> process this is.? We then proceeded to add the arbiters. >>> >>> Some of the settings below was absolute shots in the dark, based >>> on the little information we've been able to track down so far. >>> >>> Kind Regards, >>> Jaco >>> >>> The current gluster volume info (this used to be 2x2 >>> distributed-replicate, aiming to get it to 3x (2 + 1) ... at >>> current rates we might get there by 2021): >>> >>> Volume Name: glvol >>> Type: Distributed-Replicate >>> Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 2 x (2 + 1) = 6 >>> Transport-type: tcp >>> Bricks: >>> Brick1: zoidberg:/mnt/gluster/a >>> Brick2: flexo:/mnt/gluster/b >>> Brick3: bender:/mnt/gluster/c (arbiter) >>> Brick4: bender:/mnt/gluster/a >>> Brick5: zoidberg:/mnt/gluster/b >>> Brick6: flexo:/mnt/gluster/c (arbiter) >>> Options Reconfigured: >>> *cluster.heal-timeout: 5 <-- hoping this would at least shorten >>> the blockages.* >>> cluster.locking-scheme: granular >>> cluster.heal-wait-queue-length: 10240 >>> *diagnostics.count-fop-hits: on** >>> **diagnostics.latency-measurement: on <-- was hoping profiling >>> might provide some insight.? Going to switch this off now.* >>> cluster.background-self-heal-count: 128 >>> *cluster.shd-wait-qlength: 131072** >>> **cluster.shd-max-threads: 32? <-- even with this disk IO never >>> goes over 10% utilization.? CPU goes to about 2 cores.? So it's >>> lock contention that's slowing it down we reckon.* >>> *cluster.metadata-self-heal: off** >>> **cluster.entry-self-heal: off** >>> **cluster.data-self-heal: off**<-- these kill performance. >>> *transport.address-family: inet >>> nfs.disable: on >>> performance.client-io-threads: on >>> features.cache-invalidation: on >>> features.cache-invalidation-timeout: 600 >>> performance.stat-prefetch: on >>> performance.cache-invalidation: on >>> performance.md-cache-timeout: 600 >>> *cluster.self-heal-daemon: yes <-- disabling/enabling this >>> doesn't help, have to kill the shd and start ... force to get >>> things moving along again.* >>> performance.io-thread-count: 16 >>> performance.high-prio-threads: 16 >>> performance.normal-prio-threads: 16 >>> performance.low-prio-threads: 16 >>> performance.least-prio-threads: 4 >>> cluster.self-heal-window-size: 512 >>> cluster.self-heal-readdir-size: 131072 >>> client.event-threads: 8 >>> server.event-threads: 16 >>> network.ping-timeout: 2 >>> *cluster.data-self-heal-algorithm: full <-- most files are small >>> (<1MB), on dual 1gbit links (so up to 2gbit, but b/w never goes >>> over 5mbit anyway).* >>> >>> performance info: >>> >>> Brick: zoidberg:/mnt/gluster/a >>> ----------------------------------------- >>> Cumulative Stats: >>> ?? Block Size: 1b+?????????????????? 8b+????????????????? 16b+ >>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>> No. of Writes: 117?????????????????? 195?????????????????? 367 >>> >>> ?? Block Size: 32b+????????????????? 64b+???????????????? 128b+ >>> ?No. of Reads: 306???????????????????? 0???????????????????? 6 >>> No. of Writes: 1016???????????????????? 9?????????????????? 134 >>> >>> ?? Block Size: 256b+???????????????? 512b+??????????????? 1024b+ >>> ?No. of Reads: 107??????????????????? 58?????????????????? 485 >>> No. of Writes: 731??????????????????? 73?????????????????? 168 >>> >>> ?? Block Size: 2048b+??????????????? 4096b+??????????????? 8192b+ >>> ?No. of Reads: 43??????????????????? 59?????????????????? 383 >>> No. of Writes: 295?????????????????? 475???????????????? 10562 >>> >>> ?? Block Size: 16384b+?????????????? 32768b+?????????????? 65536b+ >>> ?No. of Reads: 347?????????????????? 380?????????????????? 634 >>> No. of Writes: 1487????????????????? 6369????????????????? 3970 >>> >>> ?? Block Size:???????????? 131072b+ >>> ?No. of Reads:??????????????? 35704 >>> No. of Writes:??????????????? 12398 >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 957293????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1455091???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 69951? >>> RELEASEDIR >>> ????? 0.00????? 16.20 us?????? 5.43 us 100.95 us??????????? >>> 113???????? IPC >>> ????? 0.00???? 687.33 us????? 80.53 us 15414.03 us???????????? >>> 30 REMOVEXATTR >>> ????? 0.00???? 840.76 us????? 72.71 us 37648.39 us???????????? >>> 52?? FTRUNCATE >>> ????? 0.00???? 138.49 us????? 36.66 us 25280.24 us?????????? >>> 1385??????? STAT >>> ????? 0.00???? 364.93 us????? 63.45 us 15957.66 us??????????? >>> 810?? FGETXATTR >>> ????? 0.00??? 2208.82 us???? 127.23 us 267092.17 us??????????? >>> 137??????? LINK >>> ????? 0.00?? 37595.33 us??? 1677.46 us 218313.32 us???????????? >>> 12?????? FSYNC >>> ????? 0.00???? 675.61 us???? 106.82 us 114351.01 us??????????? >>> 698??? TRUNCATE >>> ????? 0.00??? 3995.66 us???? 147.00 us 109544.46 us??????????? >>> 188????? RENAME >>> ????? 0.00???? 453.11 us????? 53.37 us 104848.94 us?????????? >>> 1680???? SETATTR >>> ????? 0.00??? 6004.32 us???? 229.33 us 151322.58 us??????????? >>> 127?????? MKNOD >>> ????? 0.00????? 68.74 us????? 16.11 us 61050.93 us????????? >>> 11412??? FINODELK >>> ????? 0.00??? 3793.44 us????? 68.61 us 411523.70 us??????????? >>> 221??? SETXATTR >>> ????? 0.00?? 28669.99 us???? 227.28 us 138988.41 us???????????? >>> 30???? SYMLINK >>> ????? 0.00????? 56.18 us?????? 7.31 us 41484.37 us????????? >>> 22640?????? FLUSH >>> ????? 0.00???? 105.64 us????? 19.34 us 351055.72 us????????? >>> 21693????? STATFS >>> ????? 0.00?? 23642.90 us???? 244.19 us 234912.25 us??????????? >>> 178?????? MKDIR >>> ????? 0.00???? 537.38 us????? 31.73 us 296911.67 us?????????? >>> 9653?????? FSTAT >>> ????? 0.00???? 214.87 us????? 25.71 us 1712020.99 us????????? >>> 39054??????? READ >>> ????? 0.01???? 147.08 us?????? 1.47 us 241611.80 us????????? >>> 69951???? OPENDIR >>> ????? 0.01??? 3899.84 us???? 117.22 us 503422.21 us?????????? >>> 3180?????? RMDIR >>> ????? 0.01???? 327.08 us????? 55.69 us 420376.80 us????????? >>> 38366?????? WRITE >>> ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us?????????? >>> 6169????? CREATE >>> ????? 0.05????? 92.69 us????? 29.94 us 687955.86 us??????? >>> 1159936??? READLINK >>> ????? 0.40???? 543.50 us????? 36.45 us 2430653.54 us??????? >>> 1449307??????? OPEN >>> ????? 0.48?? 87643.67 us????? 70.20 us 661999.51 us????????? >>> 10787??? FXATTROP >>> ????? 0.53??? 2342.72 us????? 20.91 us 4157542.43 us???????? >>> 447821????? UNLINK >>> ????? 0.53? 558734.79 us????? 24.40 us 1399487.63 us?????????? >>> 1889????????? LK >>> ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us????????? >>> 10687???? READDIR >>> ????? 0.99??? 2426.23 us????? 12.47 us 24176123.13 us???????? >>> 809430??? GETXATTR >>> ????? 1.45???? 676.83 us?????? 9.30 us 193177067.25 us??????? >>> 4260021???? ENTRYLK >>> ????? 1.76?? 38325.38 us????? 21.44 us 46693575.93 us????????? >>> 91439??? READDIRP >>> ????? 4.44??? 1200.31 us?????? 9.10 us 2559377.90 us??????? >>> 7352811???? INODELK >>> ????? 9.30??? 1849.24 us?????? 9.28 us 35299050.00 us??????? >>> 9989221????? LOOKUP >>> ???? 79.24?? 47791.46 us????? 76.77 us 10488824.26 us??????? >>> 3294642???? XATTROP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 3194806????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? >>> 5311???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????????? 54??? >>> CI_XATTR >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? 1907?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? 3189457?? >>> CI_FORGET >>> >>> ??? Duration: 209524 seconds >>> ?? Data Read: 4775365531 bytes >>> Data Written: 2354108664 bytes >>> >>> Interval 0 Stats: >>> ?? Block Size: 1b+?????????????????? 8b+????????????????? 16b+ >>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>> No. of Writes: 117?????????????????? 195?????????????????? 367 >>> >>> ?? Block Size: 32b+????????????????? 64b+???????????????? 128b+ >>> ?No. of Reads: 306???????????????????? 0???????????????????? 6 >>> No. of Writes: 1016???????????????????? 9?????????????????? 134 >>> >>> ?? Block Size: 256b+???????????????? 512b+??????????????? 1024b+ >>> ?No. of Reads: 107??????????????????? 58?????????????????? 485 >>> No. of Writes: 731??????????????????? 73?????????????????? 168 >>> >>> ?? Block Size: 2048b+??????????????? 4096b+??????????????? 8192b+ >>> ?No. of Reads: 43??????????????????? 59?????????????????? 383 >>> No. of Writes: 295?????????????????? 475???????????????? 10562 >>> >>> ?? Block Size: 16384b+?????????????? 32768b+?????????????? 65536b+ >>> ?No. of Reads: 347?????????????????? 380?????????????????? 634 >>> No. of Writes: 1487????????????????? 6369????????????????? 3970 >>> >>> ?? Block Size:???????????? 131072b+ >>> ?No. of Reads:??????????????? 35704 >>> No. of Writes:??????????????? 12398 >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 957293????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1455091???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 69951? >>> RELEASEDIR >>> ????? 0.00????? 16.20 us?????? 5.43 us 100.95 us??????????? >>> 113???????? IPC >>> ????? 0.00???? 687.33 us????? 80.53 us 15414.03 us???????????? >>> 30 REMOVEXATTR >>> ????? 0.00???? 840.76 us????? 72.71 us 37648.39 us???????????? >>> 52?? FTRUNCATE >>> ????? 0.00???? 138.49 us????? 36.66 us 25280.24 us?????????? >>> 1385??????? STAT >>> ????? 0.00???? 363.20 us????? 63.45 us 15957.66 us??????????? >>> 810?? FGETXATTR >>> ????? 0.00??? 2208.82 us???? 127.23 us 267092.17 us??????????? >>> 137??????? LINK >>> ????? 0.00?? 37595.33 us??? 1677.46 us 218313.32 us???????????? >>> 12?????? FSYNC >>> ????? 0.00???? 675.61 us???? 106.82 us 114351.01 us??????????? >>> 698??? TRUNCATE >>> ????? 0.00??? 3995.66 us???? 147.00 us 109544.46 us??????????? >>> 188????? RENAME >>> ????? 0.00???? 453.11 us????? 53.37 us 104848.94 us?????????? >>> 1680???? SETATTR >>> ????? 0.00??? 6004.32 us???? 229.33 us 151322.58 us??????????? >>> 127?????? MKNOD >>> ????? 0.00????? 68.74 us????? 16.11 us 61050.93 us????????? >>> 11412??? FINODELK >>> ????? 0.00??? 3793.44 us????? 68.61 us 411523.70 us??????????? >>> 221??? SETXATTR >>> ????? 0.00?? 28669.99 us???? 227.28 us 138988.41 us???????????? >>> 30???? SYMLINK >>> ????? 0.00????? 56.19 us?????? 7.31 us 41484.37 us????????? >>> 22640?????? FLUSH >>> ????? 0.00???? 105.61 us????? 19.34 us 351055.72 us????????? >>> 21693????? STATFS >>> ????? 0.00?? 23642.90 us???? 244.19 us 234912.25 us??????????? >>> 178?????? MKDIR >>> ????? 0.00???? 537.38 us????? 31.73 us 296911.67 us?????????? >>> 9653?????? FSTAT >>> ????? 0.00???? 214.91 us????? 25.71 us 1712020.99 us????????? >>> 39054??????? READ >>> ????? 0.01???? 147.08 us?????? 1.47 us 241611.80 us????????? >>> 69951???? OPENDIR >>> ????? 0.01??? 3899.84 us???? 117.22 us 503422.21 us?????????? >>> 3180?????? RMDIR >>> ????? 0.01???? 327.13 us????? 55.69 us 420376.80 us????????? >>> 38366?????? WRITE >>> ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us?????????? >>> 6169????? CREATE >>> ????? 0.05????? 92.69 us????? 29.94 us 687955.86 us??????? >>> 1159936??? READLINK >>> ????? 0.40???? 543.49 us????? 36.45 us 2430653.54 us??????? >>> 1449307??????? OPEN >>> ????? 0.48?? 87643.67 us????? 70.20 us 661999.51 us????????? >>> 10787??? FXATTROP >>> ????? 0.53??? 2342.57 us????? 20.91 us 4157542.43 us???????? >>> 447821????? UNLINK >>> ????? 0.53? 558416.10 us????? 24.40 us 1399487.63 us?????????? >>> 1889????????? LK >>> ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us????????? >>> 10687???? READDIR >>> ????? 0.99??? 2426.24 us????? 12.47 us 24176123.13 us???????? >>> 809430??? GETXATTR >>> ????? 1.45???? 676.75 us?????? 9.30 us 193177067.25 us??????? >>> 4260021???? ENTRYLK >>> ????? 1.76?? 38324.59 us????? 21.44 us 46693575.93 us????????? >>> 91439??? READDIRP >>> ????? 4.44??? 1200.37 us?????? 9.10 us 2559377.90 us??????? >>> 7352811???? INODELK >>> ????? 9.29??? 1848.94 us?????? 9.28 us 35299050.00 us??????? >>> 9989221????? LOOKUP >>> ???? 79.24?? 47792.42 us????? 76.77 us 10488824.26 us??????? >>> 3294642???? XATTROP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 3194806????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? >>> 5311???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????????? 54??? >>> CI_XATTR >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? 1907?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? 3189457?? >>> CI_FORGET >>> >>> ??? Duration: 209524 seconds >>> ?? Data Read: 4775365531 bytes >>> Data Written: 2354108664 bytes >>> >>> Brick: zoidberg:/mnt/gluster/b >>> ----------------------------------------- >>> Cumulative Stats: >>> ?? Block Size: 1b+?????????????????? 2b+?????????????????? 8b+ >>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>> No. of Writes: 53???????????????????? 1??????????????????? 60 >>> >>> ?? Block Size: 16b+????????????????? 32b+????????????????? 64b+ >>> ?No. of Reads: 0?????????????????? 213???????????????????? 2 >>> No. of Writes: 14????????????????? 1057??????????????????? 15 >>> >>> ?? Block Size: 128b+???????????????? 256b+???????????????? 512b+ >>> ?No. of Reads: 0??????????????????? 41?????????????????? 146 >>> No. of Writes: 59?????????????????? 715??????????????????? 80 >>> >>> ?? Block Size: 1024b+??????????????? 2048b+??????????????? 4096b+ >>> ?No. of Reads: 224???????????????????? 9??????????????????? 15 >>> No. of Writes: 163?????????????????? 282?????????????????? 404 >>> >>> ?? Block Size: 8192b+?????????????? 16384b+?????????????? 32768b+ >>> ?No. of Reads: 52??????????????????? 74?????????????????? 210 >>> No. of Writes: 7415????????????????? 1430????????????????? 6860 >>> >>> ?? Block Size: 65536b+????????????? 131072b+????????????? 262144b+ >>> ?No. of Reads: 299???????????????? 15313???????????????????? 1 >>> No. of Writes: 2458???????????????? 12378???????????????????? 1 >>> >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 686733????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 837434???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 87338? >>> RELEASEDIR >>> ????? 0.00????? 22.69 us?????? 7.16 us 332.61 us??????????? >>> 116???????? IPC >>> ????? 0.00???? 119.93 us????? 70.56 us 272.28 us???????????? >>> 57?? FTRUNCATE >>> ????? 0.00???? 152.25 us????? 86.82 us 340.63 us???????????? >>> 65?? FGETXATTR >>> ????? 0.00??? 2336.06 us????? 90.05 us 20509.81 us???????????? >>> 23 REMOVEXATTR >>> ????? 0.00???? 631.26 us???? 123.17 us 51803.77 us??????????? >>> 121??????? LINK >>> ????? 0.00???? 369.27 us????? 22.64 us 145090.48 us??????????? >>> 490????????? LK >>> ????? 0.00???? 360.85 us????? 45.14 us 46846.98 us??????????? >>> 540??????? STAT >>> ????? 0.00??? 1190.96 us???? 117.40 us 69781.85 us??????????? >>> 200??? TRUNCATE >>> ????? 0.00???? 245.72 us????? 54.91 us 123985.76 us?????????? >>> 1482???? SETATTR >>> ????? 0.00?? 42139.10 us??? 3313.56 us 312273.21 us???????????? >>> 12?????? FSYNC >>> ????? 0.00????? 55.83 us?????? 8.26 us 49649.35 us????????? >>> 10047?????? FLUSH >>> ????? 0.00????? 57.52 us????? 14.62 us 820.59 us?????????? >>> 9977??? FINODELK >>> ????? 0.00??? 3191.60 us???? 143.06 us 96214.75 us??????????? >>> 188????? RENAME >>> ????? 0.00?? 32862.86 us???? 215.38 us 83851.13 us???????????? >>> 25???? SYMLINK >>> ????? 0.00?? 11392.03 us???? 230.70 us 105471.90 us??????????? >>> 118?????? MKNOD >>> ????? 0.00???? 134.63 us????? 28.08 us 36326.43 us????????? >>> 12587?????? FSTAT >>> ????? 0.00???? 112.32 us????? 19.91 us 381241.34 us????????? >>> 21695????? STATFS >>> ????? 0.00???? 307.86 us????? 28.00 us 292666.83 us????????? >>> 16841??????? READ >>> ????? 0.00?? 29625.60 us???? 239.57 us 239532.26 us??????????? >>> 178?????? MKDIR >>> ????? 0.00???? 319.24 us????? 51.19 us 136353.39 us????????? >>> 33445?????? WRITE >>> ????? 0.01???? 175.83 us?????? 1.57 us 513635.22 us????????? >>> 87338???? OPENDIR >>> ????? 0.01???? 249.96 us????? 58.42 us 594774.86 us????????? >>> 85089??? SETXATTR >>> ????? 0.01?? 11001.25 us???? 119.87 us 612765.03 us?????????? >>> 3180?????? RMDIR >>> ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us?????????? >>> 6067????? CREATE >>> ????? 0.05????? 89.45 us????? 30.14 us 687961.60 us??????? >>> 1219542??? READLINK >>> ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us???????? >>> 831371??????? OPEN >>> ????? 0.51? 127078.45 us????? 78.55 us 694364.09 us?????????? >>> 9493??? FXATTROP >>> ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us??????? >>> 3764714???? ENTRYLK >>> ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us????????? >>> 22253???? READDIR >>> ????? 1.06??? 6913.71 us????? 11.58 us 36968532.72 us???????? >>> 364963??? GETXATTR >>> ????? 1.29?? 62130.12 us????? 36.89 us 8649871.11 us????????? >>> 49472??? READDIRP >>> ????? 2.89??? 1477.53 us????? 11.10 us 2807637.64 us??????? >>> 4650500???? INODELK >>> ????? 4.12?? 23681.31 us????? 20.98 us 23291414.50 us???????? >>> 414352????? UNLINK >>> ???? 16.09??? 5000.31 us?????? 9.04 us 23866016.88 us??????? >>> 7655621????? LOOKUP >>> ???? 72.04?? 68953.31 us????? 68.40 us 27542499.65 us??????? >>> 2486126???? XATTROP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 2433405????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? >>> 2453???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????????? 69??? >>> CI_XATTR >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? 1544?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? 2430885?? >>> CI_FORGET >>> >>> ??? Duration: 209524 seconds >>> ?? Data Read: 2051388524 bytes >>> Data Written: 2208261499 bytes >>> >>> Interval 0 Stats: >>> ?? Block Size: 1b+?????????????????? 2b+?????????????????? 8b+ >>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>> No. of Writes: 53???????????????????? 1??????????????????? 60 >>> >>> ?? Block Size: 16b+????????????????? 32b+????????????????? 64b+ >>> ?No. of Reads: 0?????????????????? 213???????????????????? 2 >>> No. of Writes: 14????????????????? 1057??????????????????? 15 >>> >>> ?? Block Size: 128b+???????????????? 256b+???????????????? 512b+ >>> ?No. of Reads: 0??????????????????? 41?????????????????? 146 >>> No. of Writes: 59?????????????????? 715??????????????????? 80 >>> >>> ?? Block Size: 1024b+??????????????? 2048b+??????????????? 4096b+ >>> ?No. of Reads: 224???????????????????? 9??????????????????? 15 >>> No. of Writes: 163?????????????????? 282?????????????????? 404 >>> >>> ?? Block Size: 8192b+?????????????? 16384b+?????????????? 32768b+ >>> ?No. of Reads: 52??????????????????? 74?????????????????? 210 >>> No. of Writes: 7415????????????????? 1430????????????????? 6860 >>> >>> ?? Block Size: 65536b+????????????? 131072b+????????????? 262144b+ >>> ?No. of Reads: 299???????????????? 15313???????????????????? 1 >>> No. of Writes: 2458???????????????? 12378???????????????????? 1 >>> >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 686733????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 837434???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 87338? >>> RELEASEDIR >>> ????? 0.00????? 22.69 us?????? 7.16 us 332.61 us??????????? >>> 116???????? IPC >>> ????? 0.00???? 119.93 us????? 70.56 us 272.28 us???????????? >>> 57?? FTRUNCATE >>> ????? 0.00???? 152.25 us????? 86.82 us 340.63 us???????????? >>> 65?? FGETXATTR >>> ????? 0.00??? 2336.06 us????? 90.05 us 20509.81 us???????????? >>> 23 REMOVEXATTR >>> ????? 0.00???? 631.26 us???? 123.17 us 51803.77 us??????????? >>> 121??????? LINK >>> ????? 0.00???? 369.27 us????? 22.64 us 145090.48 us??????????? >>> 490????????? LK >>> ????? 0.00???? 360.85 us????? 45.14 us 46846.98 us??????????? >>> 540??????? STAT >>> ????? 0.00??? 1190.96 us???? 117.40 us 69781.85 us??????????? >>> 200??? TRUNCATE >>> ????? 0.00???? 245.72 us????? 54.91 us 123985.76 us?????????? >>> 1482???? SETATTR >>> ????? 0.00?? 42139.10 us??? 3313.56 us 312273.21 us???????????? >>> 12?????? FSYNC >>> ????? 0.00????? 55.83 us?????? 8.26 us 49649.35 us????????? >>> 10047?????? FLUSH >>> ????? 0.00????? 57.52 us????? 14.62 us 820.59 us?????????? >>> 9977??? FINODELK >>> ????? 0.00??? 3191.60 us???? 143.06 us 96214.75 us??????????? >>> 188????? RENAME >>> ????? 0.00?? 32862.86 us???? 215.38 us 83851.13 us???????????? >>> 25???? SYMLINK >>> ????? 0.00?? 11392.03 us???? 230.70 us 105471.90 us??????????? >>> 118?????? MKNOD >>> ????? 0.00???? 134.63 us????? 28.08 us 36326.43 us????????? >>> 12587?????? FSTAT >>> ????? 0.00???? 112.32 us????? 19.91 us 381241.34 us????????? >>> 21695????? STATFS >>> ????? 0.00???? 307.91 us????? 28.00 us 292666.83 us????????? >>> 16841??????? READ >>> ????? 0.00?? 29625.60 us???? 239.57 us 239532.26 us??????????? >>> 178?????? MKDIR >>> ????? 0.00???? 319.22 us????? 51.19 us 136353.39 us????????? >>> 33445?????? WRITE >>> ????? 0.01???? 175.83 us?????? 1.57 us 513635.22 us????????? >>> 87338???? OPENDIR >>> ????? 0.01???? 249.95 us????? 58.42 us 594774.86 us????????? >>> 85089??? SETXATTR >>> ????? 0.01?? 11001.25 us???? 119.87 us 612765.03 us?????????? >>> 3180?????? RMDIR >>> ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us?????????? >>> 6067????? CREATE >>> ????? 0.05????? 89.46 us????? 30.14 us 687961.60 us??????? >>> 1219542??? READLINK >>> ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us???????? >>> 831371??????? OPEN >>> ????? 0.51? 127078.45 us????? 78.55 us 694364.09 us?????????? >>> 9493??? FXATTROP >>> ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us??????? >>> 3764714???? ENTRYLK >>> ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us????????? >>> 22253???? READDIR >>> ????? 1.06??? 6913.26 us????? 11.58 us 36968532.72 us???????? >>> 364963??? GETXATTR >>> ????? 1.29?? 62121.95 us????? 36.89 us 8649871.11 us????????? >>> 49472??? READDIRP >>> ????? 2.89??? 1477.58 us????? 11.10 us 2807637.64 us??????? >>> 4650500???? INODELK >>> ????? 4.12?? 23681.29 us????? 20.98 us 23291414.50 us???????? >>> 414352????? UNLINK >>> ???? 16.09??? 5000.02 us?????? 9.04 us 23866016.88 us??????? >>> 7655621????? LOOKUP >>> ???? 72.04?? 68956.14 us????? 68.40 us 27542499.65 us??????? >>> 2486126???? XATTROP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 2433405????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? >>> 2453???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????????? 69??? >>> CI_XATTR >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? 1544?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? 2430885?? >>> CI_FORGET >>> >>> ??? Duration: 209524 seconds >>> ?? Data Read: 2051388524 bytes >>> Data Written: 2208261499 bytes >>> >>> Brick: flexo:/mnt/gluster/b >>> -------------------------------------- >>> Cumulative Stats: >>> ?? Block Size: 1b+?????????????????? 8b+????????????????? 16b+ >>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>> No. of Writes: 117?????????????????? 195?????????????????? 367 >>> >>> ?? Block Size: 32b+????????????????? 64b+???????????????? 128b+ >>> ?No. of Reads: 130???????????????????? 0???????????????????? 3 >>> No. of Writes: 1012???????????????????? 9?????????????????? 134 >>> >>> ?? Block Size: 256b+???????????????? 512b+??????????????? 1024b+ >>> ?No. of Reads: 47???????????????????? 8?????????????????? 257 >>> No. of Writes: 729??????????????????? 73?????????????????? 168 >>> >>> ?? Block Size: 2048b+??????????????? 4096b+??????????????? 8192b+ >>> ?No. of Reads: 10??????????????????? 27?????????????????? 240 >>> No. of Writes: 294?????????????????? 466???????????????? 10472 >>> >>> ?? Block Size: 16384b+?????????????? 32768b+?????????????? 65536b+ >>> ?No. of Reads: 139?????????????????? 119?????????????????? 135 >>> No. of Writes: 1480????????????????? 6273????????????????? 3935 >>> >>> ?? Block Size:???????????? 131072b+ >>> ?No. of Reads:???????????????? 7892 >>> No. of Writes:??????????????? 12327 >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 955058????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1450749???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 81377? >>> RELEASEDIR >>> ????? 0.00????? 15.55 us?????? 5.35 us 31.88 us??????????? >>> 108???????? IPC >>> ????? 0.00???? 125.85 us????? 59.77 us 224.15 us???????????? >>> 44?? FTRUNCATE >>> ????? 0.00???? 198.38 us????? 81.68 us 1332.45 us???????????? 30 >>> REMOVEXATTR >>> ????? 0.00??? 4254.16 us???? 621.52 us 10602.99 us????????????? >>> 4?????? FSYNC >>> ????? 0.00???? 795.08 us???? 190.30 us 16518.64 us???????????? >>> 30???? SYMLINK >>> ????? 0.00???? 195.90 us???? 111.02 us 953.39 us??????????? >>> 137??????? LINK >>> ????? 0.00???? 286.07 us???? 179.64 us 437.82 us??????????? >>> 119?????? MKNOD >>> ????? 0.00???? 162.61 us????? 65.62 us 3864.77 us??????????? >>> 221??? SETXATTR >>> ????? 0.00???? 117.29 us????? 48.50 us 317.17 us??????????? >>> 354??????? STAT >>> ????? 0.00????? 70.35 us????? 23.10 us 317.32 us??????????? >>> 777????????? LK >>> ????? 0.00???? 461.72 us???? 114.54 us 16013.81 us??????????? >>> 188????? RENAME >>> ????? 0.00???? 150.40 us????? 75.59 us 749.54 us??????????? >>> 810?? FGETXATTR >>> ????? 0.00???? 267.80 us????? 99.67 us 994.58 us??????????? >>> 698??? TRUNCATE >>> ????? 0.00??? 1176.00 us???? 194.89 us 78261.12 us??????????? >>> 178?????? MKDIR >>> ????? 0.00???? 439.83 us????? 54.38 us 75270.51 us?????????? >>> 1665???? SETATTR >>> ????? 0.00????? 65.89 us????? 17.39 us 354.77 us????????? >>> 11372??? FINODELK >>> ????? 0.00???? 131.57 us????? 32.68 us 2107.52 us?????????? >>> 7398?????? FSTAT >>> ????? 0.00???? 120.71 us????? 31.43 us 847.97 us?????????? >>> 9094??????? READ >>> ????? 0.00????? 60.75 us?????? 7.25 us 22520.98 us????????? >>> 22614?????? FLUSH >>> ????? 0.00????? 73.03 us????? 22.41 us 750.17 us????????? >>> 21656????? STATFS >>> ????? 0.00???? 323.63 us???? 166.56 us 23034.44 us?????????? >>> 6145????? CREATE >>> ????? 0.00???? 217.55 us????? 50.60 us 30734.99 us????????? >>> 38051?????? WRITE >>> ????? 0.01???? 146.39 us?????? 1.80 us 318612.45 us????????? >>> 81377???? OPENDIR >>> ????? 0.01??? 5641.37 us???? 108.84 us 468960.33 us?????????? >>> 3177?????? RMDIR >>> ????? 0.03???? 100.05 us????? 30.93 us 49756.93 us???????? >>> 432439??? READLINK >>> ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us???????? >>> 448381????? UNLINK >>> ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us????????? >>> 52185??? READDIRP >>> ????? 0.35???? 414.05 us????? 36.58 us 1038229.93 us??????? >>> 1444606??????? OPEN >>> ????? 0.49?? 78518.02 us????? 73.39 us 373999.31 us????????? >>> 10745??? FXATTROP >>> ????? 0.72???? 776.21 us????? 11.24 us 4271792.57 us??????? >>> 1605822??? GETXATTR >>> ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us????????? >>> 13532???? READDIR >>> ????? 0.79???? 320.44 us?????? 7.71 us 2369511.06 us??????? >>> 4257297???? ENTRYLK >>> ????? 1.97???? 464.02 us?????? 9.38 us 1038460.80 us??????? >>> 7333722???? INODELK >>> ???? 13.73??? 2372.50 us?????? 8.33 us 6258159.87 us??????? >>> 9979799????? LOOKUP >>> ???? 80.79?? 42372.04 us????? 61.49 us 2238308.13 us??????? >>> 3289103???? XATTROP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 2250268????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? >>> 3756???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????????? 167??? >>> CI_XATTR >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? 1908?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? 2246352?? >>> CI_FORGET >>> >>> ??? Duration: 209334 seconds >>> ?? Data Read: 1059777395 bytes >>> Data Written: 2336921014 bytes >>> >>> Interval 0 Stats: >>> ?? Block Size: 1b+?????????????????? 8b+????????????????? 16b+ >>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>> No. of Writes: 117?????????????????? 195?????????????????? 367 >>> >>> ?? Block Size: 32b+????????????????? 64b+???????????????? 128b+ >>> ?No. of Reads: 130???????????????????? 0???????????????????? 3 >>> No. of Writes: 1012???????????????????? 9?????????????????? 134 >>> >>> ?? Block Size: 256b+???????????????? 512b+??????????????? 1024b+ >>> ?No. of Reads: 47???????????????????? 8?????????????????? 257 >>> No. of Writes: 729??????????????????? 73?????????????????? 168 >>> >>> ?? Block Size: 2048b+??????????????? 4096b+??????????????? 8192b+ >>> ?No. of Reads: 10??????????????????? 27?????????????????? 240 >>> No. of Writes: 294?????????????????? 466???????????????? 10472 >>> >>> ?? Block Size: 16384b+?????????????? 32768b+?????????????? 65536b+ >>> ?No. of Reads: 139?????????????????? 119?????????????????? 135 >>> No. of Writes: 1480????????????????? 6273????????????????? 3935 >>> >>> ?? Block Size:???????????? 131072b+ >>> ?No. of Reads:???????????????? 7892 >>> No. of Writes:??????????????? 12327 >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 955058????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1450749???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 81377? >>> RELEASEDIR >>> ????? 0.00????? 15.55 us?????? 5.35 us 31.88 us??????????? >>> 108???????? IPC >>> ????? 0.00???? 125.85 us????? 59.77 us 224.15 us???????????? >>> 44?? FTRUNCATE >>> ????? 0.00???? 198.38 us????? 81.68 us 1332.45 us???????????? 30 >>> REMOVEXATTR >>> ????? 0.00??? 4254.16 us???? 621.52 us 10602.99 us????????????? >>> 4?????? FSYNC >>> ????? 0.00???? 795.08 us???? 190.30 us 16518.64 us???????????? >>> 30???? SYMLINK >>> ????? 0.00???? 195.90 us???? 111.02 us 953.39 us??????????? >>> 137??????? LINK >>> ????? 0.00???? 286.07 us???? 179.64 us 437.82 us??????????? >>> 119?????? MKNOD >>> ????? 0.00???? 162.61 us????? 65.62 us 3864.77 us??????????? >>> 221??? SETXATTR >>> ????? 0.00???? 117.29 us????? 48.50 us 317.17 us??????????? >>> 354??????? STAT >>> ????? 0.00????? 70.35 us????? 23.10 us 317.32 us??????????? >>> 777????????? LK >>> ????? 0.00???? 461.72 us???? 114.54 us 16013.81 us??????????? >>> 188????? RENAME >>> ????? 0.00???? 150.43 us????? 75.59 us 749.54 us??????????? >>> 810?? FGETXATTR >>> ????? 0.00???? 267.80 us????? 99.67 us 994.58 us??????????? >>> 698??? TRUNCATE >>> ????? 0.00??? 1176.00 us???? 194.89 us 78261.12 us??????????? >>> 178?????? MKDIR >>> ????? 0.00???? 439.83 us????? 54.38 us 75270.51 us?????????? >>> 1665???? SETATTR >>> ????? 0.00????? 65.89 us????? 17.39 us 354.77 us????????? >>> 11372??? FINODELK >>> ????? 0.00???? 131.57 us????? 32.68 us 2107.52 us?????????? >>> 7398?????? FSTAT >>> ????? 0.00???? 120.74 us????? 31.43 us 847.97 us?????????? >>> 9094??????? READ >>> ????? 0.00????? 60.75 us?????? 7.25 us 22520.98 us????????? >>> 22614?????? FLUSH >>> ????? 0.00????? 73.03 us????? 22.41 us 750.17 us????????? >>> 21656????? STATFS >>> ????? 0.00???? 323.63 us???? 166.56 us 23034.44 us?????????? >>> 6145????? CREATE >>> ????? 0.00???? 217.50 us????? 50.60 us 30734.99 us????????? >>> 38051?????? WRITE >>> ????? 0.01???? 146.39 us?????? 1.80 us 318612.45 us????????? >>> 81377???? OPENDIR >>> ????? 0.01??? 5641.37 us???? 108.84 us 468960.33 us?????????? >>> 3177?????? RMDIR >>> ????? 0.03???? 100.06 us????? 30.93 us 49756.93 us???????? >>> 432439??? READLINK >>> ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us???????? >>> 448381????? UNLINK >>> ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us????????? >>> 52185??? READDIRP >>> ????? 0.35???? 414.01 us????? 36.58 us 1038229.93 us??????? >>> 1444606??????? OPEN >>> ????? 0.49?? 78518.02 us????? 73.39 us 373999.31 us????????? >>> 10745??? FXATTROP >>> ????? 0.72???? 776.24 us????? 11.24 us 4271792.57 us??????? >>> 1605822??? GETXATTR >>> ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us????????? >>> 13532???? READDIR >>> ????? 0.79???? 320.45 us?????? 7.71 us 2369511.06 us??????? >>> 4257297???? ENTRYLK >>> ????? 1.97???? 464.07 us?????? 9.38 us 1038460.80 us??????? >>> 7333722???? INODELK >>> ???? 13.72??? 2372.42 us?????? 8.33 us 6258159.87 us??????? >>> 9979799????? LOOKUP >>> ???? 80.79?? 42373.50 us????? 61.49 us 2238308.13 us??????? >>> 3289103???? XATTROP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 2250268????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? >>> 3756???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????????? 167??? >>> CI_XATTR >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? 1908?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? 2246352?? >>> CI_FORGET >>> >>> ??? Duration: 209334 seconds >>> ?? Data Read: 1059777395 bytes >>> Data Written: 2336921014 bytes >>> >>> Brick: flexo:/mnt/gluster/c >>> -------------------------------------- >>> Cumulative Stats: >>> ?? Block Size:????????????????? 1b+ >>> ?No. of Reads:??????????????????? 0 >>> No. of Writes:??????????????? 32843 >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 686562????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 834596???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 93757? >>> RELEASEDIR >>> ????? 0.00????? 18.50 us?????? 8.21 us 50.20 us??????????? >>> 107???????? IPC >>> ????? 0.00???? 205.83 us???? 126.30 us 412.06 us???????????? >>> 36??????? LINK >>> ????? 0.00???? 167.33 us????? 75.01 us 272.47 us???????????? >>> 65?? FGETXATTR >>> ????? 0.00???? 203.58 us???? 117.35 us 648.28 us???????????? >>> 88????? RENAME >>> ????? 0.00???? 203.20 us????? 66.60 us 809.96 us??????????? >>> 153??? TRUNCATE >>> ????? 0.00????? 67.06 us????? 23.64 us 555.59 us??????????? >>> 491????????? LK >>> ????? 0.01????? 51.73 us?????? 7.70 us 1174.64 us?????????? >>> 9339?????? FLUSH >>> ????? 0.01????? 62.41 us????? 15.13 us 636.35 us?????????? >>> 9011??? FINODELK >>> ????? 0.03???? 126.23 us????? 43.15 us 4760.14 us????????? >>> 10824?????? FSTAT >>> ????? 0.03???? 571.94 us????? 94.85 us 51697.03 us?????????? >>> 2594?????? MKDIR >>> ????? 0.05???? 264.87 us????? 81.58 us 53152.06 us?????????? >>> 9009??? FXATTROP >>> ????? 0.05???? 423.05 us????? 38.26 us 548686.55 us?????????? >>> 5754????? CREATE >>> ????? 0.05????? 78.14 us????? 19.17 us 678.32 us????????? >>> 32843?????? WRITE >>> ????? 0.06???? 526.39 us???? 130.84 us 41405.46 us?????????? >>> 5798???? SYMLINK >>> ????? 0.10???? 133.73 us????? 16.25 us 676494.72 us????????? >>> 34904???? READDIR >>> ????? 0.12??? 2054.48 us????? 99.28 us 88061.83 us?????????? >>> 2732?????? RMDIR >>> ????? 0.21???? 110.60 us?????? 1.19 us 16833.78 us????????? >>> 93757???? OPENDIR >>> ????? 0.72???? 205.89 us????? 33.28 us 261023.99 us???????? >>> 170292????? UNLINK >>> ????? 0.85??? 2971.45 us????? 21.24 us 257839.19 us????????? >>> 13840??? READDIRP >>> ????? 1.94???? 200.88 us????? 25.49 us 461654.32 us???????? >>> 468842?? FTRUNCATE >>> ????? 2.13???? 212.44 us????? 54.02 us 461878.34 us???????? >>> 486882 REMOVEXATTR >>> ????? 2.27???? 223.01 us????? 52.03 us 178897.40 us???????? >>> 493240??? SETXATTR >>> ????? 3.53???? 155.31 us?????? 7.96 us 67873.00 us??????? >>> 1101338??? GETXATTR >>> ????? 3.63???? 212.32 us????? 44.42 us 551264.02 us???????? >>> 829002??????? OPEN >>> ????? 5.77????? 76.36 us?????? 7.76 us 962634.73 us??????? >>> 3660425???? ENTRYLK >>> ????? 6.23???? 304.93 us????? 42.51 us 661394.43 us???????? >>> 990644???? SETATTR >>> ????? 7.81???? 382.20 us???? 107.42 us 846502.78 us???????? >>> 990766???? XATTROP >>> ???? 12.39???? 549.11 us???? 135.57 us 1659697.83 us??????? >>> 1093387?????? MKNOD >>> ???? 12.47???? 139.78 us?????? 9.76 us 552067.56 us??????? >>> 4325281???? INODELK >>> ???? 39.53???? 206.94 us?????? 8.56 us 964823.33 us??????? >>> 9259429????? LOOKUP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1856377????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 933300???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????????? 251??? >>> CI_XATTR >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????????? 293?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? 923077?? >>> CI_FORGET >>> >>> ??? Duration: 209334 seconds >>> ?? Data Read: 0 bytes >>> Data Written: 32843 bytes >>> >>> Interval 0 Stats: >>> ?? Block Size:????????????????? 1b+ >>> ?No. of Reads:??????????????????? 0 >>> No. of Writes:??????????????? 32843 >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 686562????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 834596???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 93757? >>> RELEASEDIR >>> ????? 0.00????? 18.50 us?????? 8.21 us 50.20 us??????????? >>> 107???????? IPC >>> ????? 0.00???? 205.83 us???? 126.30 us 412.06 us???????????? >>> 36??????? LINK >>> ????? 0.00???? 167.33 us????? 75.01 us 272.47 us???????????? >>> 65?? FGETXATTR >>> ????? 0.00???? 203.58 us???? 117.35 us 648.28 us???????????? >>> 88????? RENAME >>> ????? 0.00???? 203.20 us????? 66.60 us 809.96 us??????????? >>> 153??? TRUNCATE >>> ????? 0.00????? 67.06 us????? 23.64 us 555.59 us??????????? >>> 491????????? LK >>> ????? 0.01????? 51.71 us?????? 7.70 us 1174.64 us?????????? >>> 9339?????? FLUSH >>> ????? 0.01????? 62.41 us????? 15.13 us 636.35 us?????????? >>> 9011??? FINODELK >>> ????? 0.03???? 126.23 us????? 43.15 us 4760.14 us????????? >>> 10824?????? FSTAT >>> ????? 0.03???? 571.94 us????? 94.85 us 51697.03 us?????????? >>> 2594?????? MKDIR >>> ????? 0.05???? 264.87 us????? 81.58 us 53152.06 us?????????? >>> 9009??? FXATTROP >>> ????? 0.05???? 423.05 us????? 38.26 us 548686.55 us?????????? >>> 5754????? CREATE >>> ????? 0.05????? 78.14 us????? 19.17 us 678.32 us????????? >>> 32843?????? WRITE >>> ????? 0.06???? 526.39 us???? 130.84 us 41405.46 us?????????? >>> 5798???? SYMLINK >>> ????? 0.10???? 133.73 us????? 16.25 us 676494.72 us????????? >>> 34904???? READDIR >>> ????? 0.12??? 2054.48 us????? 99.28 us 88061.83 us?????????? >>> 2732?????? RMDIR >>> ????? 0.21???? 110.60 us?????? 1.19 us 16833.78 us????????? >>> 93757???? OPENDIR >>> ????? 0.72???? 205.89 us????? 33.28 us 261023.99 us???????? >>> 170292????? UNLINK >>> ????? 0.85??? 2971.45 us????? 21.24 us 257839.19 us????????? >>> 13840??? READDIRP >>> ????? 1.94???? 200.87 us????? 25.49 us 461654.32 us???????? >>> 468842?? FTRUNCATE >>> ????? 2.13???? 212.44 us????? 54.02 us 461878.34 us???????? >>> 486882 REMOVEXATTR >>> ????? 2.27???? 223.00 us????? 52.03 us 178897.40 us???????? >>> 493240??? SETXATTR >>> ????? 3.53???? 155.32 us?????? 7.96 us 67873.00 us??????? >>> 1101338??? GETXATTR >>> ????? 3.63???? 212.32 us????? 44.42 us 551264.02 us???????? >>> 829002??????? OPEN >>> ????? 5.77????? 76.36 us?????? 7.76 us 962634.73 us??????? >>> 3660425???? ENTRYLK >>> ????? 6.23???? 304.88 us????? 42.51 us 661394.43 us???????? >>> 990644???? SETATTR >>> ????? 7.81???? 382.20 us???? 107.42 us 846502.78 us???????? >>> 990766???? XATTROP >>> ???? 12.39???? 549.25 us???? 135.57 us 1659697.83 us??????? >>> 1093387?????? MKNOD >>> ???? 12.47???? 139.78 us?????? 9.76 us 552067.56 us??????? >>> 4325281???? INODELK >>> ???? 39.53???? 206.95 us?????? 8.56 us 964823.33 us??????? >>> 9259429????? LOOKUP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1856377????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 933300???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????????? 251??? >>> CI_XATTR >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????????? 293?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? 923077?? >>> CI_FORGET >>> >>> ??? Duration: 209334 seconds >>> ?? Data Read: 0 bytes >>> Data Written: 32843 bytes >>> >>> Brick: bender:/mnt/gluster/c >>> --------------------------------------- >>> Cumulative Stats: >>> ?? Block Size:????????????????? 1b+ >>> ?No. of Reads:??????????????????? 0 >>> No. of Writes:??????????????? 37050 >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ?????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 928119????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1401524???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 70092? >>> RELEASEDIR >>> ????? 0.00????? 19.49 us?????? 8.28 us 180.61 us??????????? >>> 100???????? IPC >>> ????? 0.00???? 238.88 us???? 163.97 us 486.93 us???????????? >>> 38??????? LINK >>> ????? 0.00???? 306.43 us???? 196.67 us 788.36 us???????????? >>> 30???? SYMLINK >>> ????? 0.00??? 2670.18 us???? 765.93 us 4151.64 us????????????? >>> 4?????? FSYNC >>> ????? 0.00???? 350.65 us???? 141.92 us 10573.64 us??????????? >>> 122????? RENAME >>> ????? 0.00????? 67.02 us????? 20.66 us 453.90 us??????????? >>> 777????????? LK >>> ????? 0.00???? 158.32 us????? 64.22 us 532.21 us??????????? >>> 810?? FGETXATTR >>> ????? 0.00???? 229.87 us????? 66.47 us 826.31 us??????????? >>> 603??? TRUNCATE >>> ????? 0.01??? 1742.85 us???? 202.81 us 47471.71 us??????????? >>> 302?????? MKDIR >>> ????? 0.01????? 59.05 us????? 15.40 us 1813.56 us????????? >>> 10060??? FINODELK >>> ????? 0.01???? 118.54 us????? 36.65 us 4002.58 us?????????? >>> 6409?????? FSTAT >>> ????? 0.01????? 49.38 us?????? 8.08 us 1264.41 us????????? >>> 21719?????? FLUSH >>> ????? 0.03????? 74.76 us????? 20.86 us 2008.41 us????????? >>> 37050?????? WRITE >>> ????? 0.04???? 553.12 us????? 40.81 us 386485.29 us?????????? >>> 5901????? CREATE >>> ????? 0.06???? 534.01 us???? 100.86 us 527770.44 us????????? >>> 10057??? FXATTROP >>> ????? 0.12???? 138.96 us?????? 1.00 us 601208.70 us????????? >>> 70092???? OPENDIR >>> ????? 0.12???? 947.76 us????? 18.54 us 712568.33 us????????? >>> 10779???? READDIR >>> ????? 0.15??? 4099.07 us???? 124.69 us 421939.47 us?????????? >>> 3008?????? RMDIR >>> ????? 0.61???? 465.30 us????? 20.67 us 936959.86 us???????? >>> 109478????? UNLINK >>> ????? 1.57???? 182.41 us????? 49.54 us 73938.09 us???????? >>> 722669 REMOVEXATTR >>> ????? 1.61???? 171.41 us?????? 6.98 us 83809.88 us???????? >>> 789294??? GETXATTR >>> ????? 1.86???? 217.88 us????? 26.40 us 556446.19 us???????? >>> 718835?? FTRUNCATE >>> ????? 1.89???? 219.47 us????? 49.92 us 506767.78 us???????? >>> 722856??? SETXATTR >>> ????? 3.02???? 181.66 us????? 44.40 us 344656.69 us??????? >>> 1395957??????? OPEN >>> ????? 5.56???? 322.08 us????? 43.37 us 634069.61 us??????? >>> 1450160???? SETATTR >>> ????? 5.84???? 115.88 us?????? 7.67 us 747387.51 us??????? >>> 4233520???? ENTRYLK >>> ????? 6.53???? 378.33 us???? 122.84 us 766939.14 us??????? >>> 1449601???? XATTROP >>> ????? 9.75???? 115.48 us????? 10.70 us 556675.13 us??????? >>> 7097226???? INODELK >>> ???? 19.37??? 1104.98 us???? 101.28 us 1612346.75 us??????? >>> 1472813?????? MKNOD >>> ???? 41.85???? 295.61 us?????? 8.98 us 996805.54 us?????? >>> 11896091????? LOOKUP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 3531139????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1862470???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????????? 244??? >>> CI_XATTR >>> ?????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????????? 632?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? 1668669?? >>> CI_FORGET >>> >>> ??? Duration: 208643 seconds >>> ?? Data Read: 0 bytes >>> Data Written: 37050 bytes >>> >>> Interval 0 Stats: >>> ?? Block Size:????????????????? 1b+ >>> ?No. of Reads:??????????????????? 0 >>> No. of Writes:??????????????? 37050 >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 928119????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1401524???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 70092? >>> RELEASEDIR >>> ????? 0.00????? 19.49 us?????? 8.28 us 180.61 us??????????? >>> 100???????? IPC >>> ????? 0.00???? 238.88 us???? 163.97 us 486.93 us???????????? >>> 38??????? LINK >>> ????? 0.00???? 306.43 us???? 196.67 us 788.36 us???????????? >>> 30???? SYMLINK >>> ????? 0.00??? 2670.18 us???? 765.93 us 4151.64 us????????????? >>> 4?????? FSYNC >>> ????? 0.00???? 350.65 us???? 141.92 us 10573.64 us??????????? >>> 122????? RENAME >>> ????? 0.00????? 67.02 us????? 20.66 us 453.90 us??????????? >>> 777????????? LK >>> ????? 0.00???? 158.32 us????? 64.22 us 532.21 us??????????? >>> 810?? FGETXATTR >>> ????? 0.00???? 229.87 us????? 66.47 us 826.31 us??????????? >>> 603??? TRUNCATE >>> ????? 0.01??? 1742.85 us???? 202.81 us 47471.71 us??????????? >>> 302?????? MKDIR >>> ????? 0.01????? 59.05 us????? 15.40 us 1813.56 us????????? >>> 10060??? FINODELK >>> ????? 0.01???? 118.54 us????? 36.65 us 4002.58 us?????????? >>> 6409?????? FSTAT >>> ????? 0.01????? 49.39 us?????? 8.08 us 1264.41 us????????? >>> 21719?????? FLUSH >>> ????? 0.03????? 74.77 us????? 20.86 us 2008.41 us????????? >>> 37050?????? WRITE >>> ????? 0.04???? 553.12 us????? 40.81 us 386485.29 us?????????? >>> 5901????? CREATE >>> ????? 0.06???? 534.01 us???? 100.86 us 527770.44 us????????? >>> 10057??? FXATTROP >>> ????? 0.12???? 138.96 us?????? 1.00 us 601208.70 us????????? >>> 70092???? OPENDIR >>> ????? 0.12???? 947.76 us????? 18.54 us 712568.33 us????????? >>> 10779???? READDIR >>> ????? 0.15??? 4099.07 us???? 124.69 us 421939.47 us?????????? >>> 3008?????? RMDIR >>> ????? 0.61???? 465.30 us????? 20.67 us 936959.86 us???????? >>> 109478????? UNLINK >>> ????? 1.57???? 182.41 us????? 49.54 us 73938.09 us???????? >>> 722669 REMOVEXATTR >>> ????? 1.61???? 171.41 us?????? 6.98 us 83809.88 us???????? >>> 789294??? GETXATTR >>> ????? 1.86???? 217.88 us????? 26.40 us 556446.19 us???????? >>> 718835?? FTRUNCATE >>> ????? 1.89???? 219.48 us????? 49.92 us 506767.78 us???????? >>> 722856??? SETXATTR >>> ????? 3.02???? 181.66 us????? 44.40 us 344656.69 us??????? >>> 1395957??????? OPEN >>> ????? 5.56???? 322.09 us????? 43.37 us 634069.61 us??????? >>> 1450160???? SETATTR >>> ????? 5.84???? 115.88 us?????? 7.67 us 747387.51 us??????? >>> 4233520???? ENTRYLK >>> ????? 6.53???? 378.33 us???? 122.84 us 766939.14 us??????? >>> 1449601???? XATTROP >>> ????? 9.75???? 115.48 us????? 10.70 us 556675.13 us??????? >>> 7097226???? INODELK >>> ???? 19.37??? 1105.06 us???? 101.28 us 1612346.75 us??????? >>> 1472813?????? MKNOD >>> ???? 41.84???? 295.54 us?????? 8.98 us 996805.54 us?????? >>> 11896091????? LOOKUP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 3531139????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1862470???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????????? 244??? >>> CI_XATTR >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????????? 632?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? 1668669?? >>> CI_FORGET >>> >>> ??? Duration: 208643 seconds >>> ?? Data Read: 0 bytes >>> Data Written: 37050 bytes >>> >>> Brick: bender:/mnt/gluster/a >>> --------------------------------------- >>> Cumulative Stats: >>> ?? Block Size: 1b+?????????????????? 2b+?????????????????? 8b+ >>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>> No. of Writes: 53???????????????????? 1??????????????????? 60 >>> >>> ?? Block Size: 16b+????????????????? 32b+????????????????? 64b+ >>> ?No. of Reads: 0?????????????????? 235???????????????????? 4 >>> No. of Writes: 14????????????????? 1033??????????????????? 15 >>> >>> ?? Block Size: 128b+???????????????? 256b+???????????????? 512b+ >>> ?No. of Reads: 4?????????????????? 115??????????????????? 81 >>> No. of Writes: 58?????????????????? 703??????????????????? 80 >>> >>> ?? Block Size: 1024b+??????????????? 2048b+??????????????? 4096b+ >>> ?No. of Reads: 181??????????????????? 26??????????????????? 74 >>> No. of Writes: 163?????????????????? 278?????????????????? 397 >>> >>> ?? Block Size: 8192b+?????????????? 16384b+?????????????? 32768b+ >>> ?No. of Reads: 131?????????????????? 161?????????????????? 267 >>> No. of Writes: 7264????????????????? 1398????????????????? 6742 >>> >>> ?? Block Size: 65536b+????????????? 131072b+????????????? 262144b+ >>> ?No. of Reads: 443???????????????? 25528???????????????????? 0 >>> No. of Writes: 2426???????????????? 12240???????????????????? 1 >>> >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 684942????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 837092???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 87297? >>> RELEASEDIR >>> ????? 0.00???? 219.22 us???? 121.08 us 306.12 us????????????? 8 >>> REMOVEXATTR >>> ????? 0.00????? 21.70 us?????? 7.24 us 196.07 us???????????? >>> 97???????? IPC >>> ????? 0.00???? 152.77 us????? 78.90 us 1059.82 us???????????? >>> 46?? FTRUNCATE >>> ????? 0.00???? 295.73 us???? 167.10 us 606.96 us???????????? >>> 65?? FGETXATTR >>> ????? 0.00???? 243.66 us???? 140.50 us 3392.16 us??????????? >>> 121??????? LINK >>> ????? 0.00??? 1216.47 us???? 194.37 us 23411.42 us???????????? >>> 25???? SYMLINK >>> ????? 0.00???? 327.87 us???? 234.27 us 551.71 us??????????? >>> 107?????? MKNOD >>> ????? 0.00?? 36094.61 us?? 36094.61 us 36094.61 us????????????? >>> 1?????? FSYNC >>> ????? 0.00???? 219.63 us???? 147.34 us 783.10 us??????????? >>> 188????? RENAME >>> ????? 0.00???? 104.27 us????? 40.34 us 595.29 us??????????? >>> 745??????? STAT >>> ????? 0.00???? 531.71 us???? 113.57 us 53973.08 us??????????? >>> 200??? TRUNCATE >>> ????? 0.00???? 289.05 us????? 63.10 us 70107.41 us?????????? >>> 1444???? SETATTR >>> ????? 0.00??? 2430.15 us???? 232.78 us 68120.54 us??????????? >>> 177?????? MKDIR >>> ????? 0.00????? 52.68 us?????? 7.28 us 826.01 us?????????? >>> 9890?????? FLUSH >>> ????? 0.00????? 60.96 us????? 17.54 us 1393.95 us?????????? >>> 9792??? FINODELK >>> ????? 0.00????? 74.26 us????? 20.00 us 1522.39 us????????? >>> 21521????? STATFS >>> ????? 0.00???? 129.21 us????? 34.15 us 9068.67 us????????? >>> 13094?????? FSTAT >>> ????? 0.00???? 456.99 us???? 195.50 us 130182.02 us?????????? >>> 5967????? CREATE >>> ????? 0.00???? 112.06 us????? 28.14 us 9545.53 us????????? >>> 27474??????? READ >>> ????? 0.01???? 313.76 us????? 52.98 us 71635.74 us????????? >>> 32926?????? WRITE >>> ????? 0.01???? 135.84 us?????? 1.52 us 671373.08 us????????? >>> 87297???? OPENDIR >>> ????? 0.01??? 5225.28 us???? 131.28 us 951323.69 us?????????? >>> 3177?????? RMDIR >>> ????? 0.01???? 203.34 us????? 56.13 us 486756.90 us????????? >>> 85073??? SETXATTR >>> ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us????????? >>> 15090??? READDIRP >>> ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us??????????? >>> 564????????? LK >>> ????? 0.05????? 92.18 us????? 29.71 us 39069.99 us???????? >>> 850938??? READLINK >>> ????? 0.23???? 453.79 us????? 41.84 us 655999.23 us???????? >>> 831212??????? OPEN >>> ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us????????? >>> 22482???? READDIR >>> ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us???????? >>> 415226????? UNLINK >>> ????? 0.54??? 1272.83 us?????? 9.66 us 1604368.93 us???????? >>> 686945??? GETXATTR >>> ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us??????? >>> 3739718???? ENTRYLK >>> ????? 0.60? 103888.49 us????? 89.32 us 761509.35 us?????????? >>> 9311??? FXATTROP >>> ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us??????? >>> 4649291???? INODELK >>> ????? 5.82??? 1240.20 us?????? 8.78 us 1526868.50 us??????? >>> 7625383????? LOOKUP >>> ???? 88.60?? 58005.98 us????? 67.31 us 1448810.06 us??????? >>> 2481914???? XATTROP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1396875????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? >>> 1854???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????????? 28??? >>> CI_XATTR >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? 1051?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? 1394996?? >>> CI_FORGET >>> >>> ??? Duration: 208643 seconds >>> ?? Data Read: 3409342848 bytes >>> Data Written: 2180686507 bytes >>> >>> Interval 0 Stats: >>> ?? Block Size: 1b+?????????????????? 2b+?????????????????? 8b+ >>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>> No. of Writes: 53???????????????????? 1??????????????????? 60 >>> >>> ?? Block Size: 16b+????????????????? 32b+????????????????? 64b+ >>> ?No. of Reads: 0?????????????????? 235???????????????????? 4 >>> No. of Writes: 14????????????????? 1033??????????????????? 15 >>> >>> ?? Block Size: 128b+???????????????? 256b+???????????????? 512b+ >>> ?No. of Reads: 4?????????????????? 115??????????????????? 81 >>> No. of Writes: 58?????????????????? 703??????????????????? 80 >>> >>> ?? Block Size: 1024b+??????????????? 2048b+??????????????? 4096b+ >>> ?No. of Reads: 181??????????????????? 26??????????????????? 74 >>> No. of Writes: 163?????????????????? 278?????????????????? 397 >>> >>> ?? Block Size: 8192b+?????????????? 16384b+?????????????? 32768b+ >>> ?No. of Reads: 131?????????????????? 161?????????????????? 267 >>> No. of Writes: 7264????????????????? 1398????????????????? 6742 >>> >>> ?? Block Size: 65536b+????????????? 131072b+????????????? 262144b+ >>> ?No. of Reads: 443???????????????? 25528???????????????????? 0 >>> No. of Writes: 2426???????????????? 12240???????????????????? 1 >>> >>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>> calls???????? Fop >>> ?---------?? -----------?? ----------- -----------?? >>> ------------??????? ---- >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 684942????? FORGET >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????? >>> 837092???? RELEASE >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us????????? 87297? >>> RELEASEDIR >>> ????? 0.00???? 219.22 us???? 121.08 us 306.12 us????????????? 8 >>> REMOVEXATTR >>> ????? 0.00????? 21.70 us?????? 7.24 us 196.07 us???????????? >>> 97???????? IPC >>> ????? 0.00???? 152.77 us????? 78.90 us 1059.82 us???????????? >>> 46?? FTRUNCATE >>> ????? 0.00???? 295.73 us???? 167.10 us 606.96 us???????????? >>> 65?? FGETXATTR >>> ????? 0.00???? 243.66 us???? 140.50 us 3392.16 us??????????? >>> 121??????? LINK >>> ????? 0.00??? 1216.47 us???? 194.37 us 23411.42 us???????????? >>> 25???? SYMLINK >>> ????? 0.00???? 327.87 us???? 234.27 us 551.71 us??????????? >>> 107?????? MKNOD >>> ????? 0.00?? 36094.61 us?? 36094.61 us 36094.61 us????????????? >>> 1?????? FSYNC >>> ????? 0.00???? 219.63 us???? 147.34 us 783.10 us??????????? >>> 188????? RENAME >>> ????? 0.00???? 104.27 us????? 40.34 us 595.29 us??????????? >>> 745??????? STAT >>> ????? 0.00???? 531.71 us???? 113.57 us 53973.08 us??????????? >>> 200??? TRUNCATE >>> ????? 0.00???? 289.05 us????? 63.10 us 70107.41 us?????????? >>> 1444???? SETATTR >>> ????? 0.00??? 2430.15 us???? 232.78 us 68120.54 us??????????? >>> 177?????? MKDIR >>> ????? 0.00????? 52.69 us?????? 7.28 us 826.01 us?????????? >>> 9890?????? FLUSH >>> ????? 0.00????? 60.96 us????? 17.54 us 1393.95 us?????????? >>> 9792??? FINODELK >>> ????? 0.00????? 74.25 us????? 20.00 us 1522.39 us????????? >>> 21521????? STATFS >>> ????? 0.00???? 129.23 us????? 34.15 us 9068.67 us????????? >>> 13094?????? FSTAT >>> ????? 0.00???? 456.99 us???? 195.50 us 130182.02 us?????????? >>> 5967????? CREATE >>> ????? 0.00???? 112.45 us????? 28.14 us 9545.53 us????????? >>> 27474??????? READ >>> ????? 0.01???? 313.77 us????? 52.98 us 71635.74 us????????? >>> 32926?????? WRITE >>> ????? 0.01???? 135.84 us?????? 1.52 us 671373.08 us????????? >>> 87297???? OPENDIR >>> ????? 0.01??? 5225.28 us???? 131.28 us 951323.69 us?????????? >>> 3177?????? RMDIR >>> ????? 0.01???? 203.34 us????? 56.13 us 486756.90 us????????? >>> 85073??? SETXATTR >>> ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us????????? >>> 15090??? READDIRP >>> ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us??????????? >>> 564????????? LK >>> ????? 0.05????? 92.18 us????? 29.71 us 39069.99 us???????? >>> 850938??? READLINK >>> ????? 0.23???? 453.79 us????? 41.84 us 655999.23 us???????? >>> 831212??????? OPEN >>> ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us????????? >>> 22482???? READDIR >>> ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us???????? >>> 415226????? UNLINK >>> ????? 0.54??? 1272.79 us?????? 9.66 us 1604368.93 us???????? >>> 686945??? GETXATTR >>> ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us??????? >>> 3739718???? ENTRYLK >>> ????? 0.60? 103888.49 us????? 89.32 us 761509.35 us?????????? >>> 9311??? FXATTROP >>> ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us??????? >>> 4649291???? INODELK >>> ????? 5.82??? 1240.10 us?????? 8.78 us 1526868.50 us??????? >>> 7625383????? LOOKUP >>> ???? 88.60?? 58006.02 us????? 67.31 us 1448810.06 us??????? >>> 2481914???? XATTROP >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? >>> 1396875????? UPCALL >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? >>> 1854???? CI_IATT >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us???????????? 28??? >>> CI_XATTR >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us?????????? 1051?? >>> CI_UNLINK >>> ????? 0.00?????? 0.00 us?????? 0.00 us 0.00 us??????? 1394996?? >>> CI_FORGET >>> >>> ??? Duration: 208643 seconds >>> ?? Data Read: 3409342848 bytes >>> Data Written: 2180686507 bytes >>> >>> >>> ________ >>> >>> >>> >>> Community Meeting Calendar: >>> >>> Schedule - >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>> Bridge: https://bluejeans.com/441850968 >>> >>> Gluster-users mailing list >>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200508/8dc8090e/attachment-0001.html>