Hi, On 2020/05/08 06:32, Ravishankar N wrote:> > On 08/05/20 1:10 am, Jaco Kroon wrote: >> >> Hi, >> >> So upgrading to 7.5 made a difference in that it takes a bit longer >> to cause chaos, but it still does the exact same thing. >> >> To be precise: >> >> I've now had to set: >> >> gluster volume set vpbx_shared cluster.heal-wait-queue-length 1 >> gluster volume set vpbx_shared cluster.shd-wait-qlength 1 >> gluster volume set vpbx_shared cluster.shd-max-threads 1 >> >> With any of these settings >1 I end up in a situation where simple >> directory listings of < 100 entries in a folder takes upwards of a >> minute to complete *from time to time*. >> > Hi Jaco, these options are to be increased only if your hardware is > capable of handling the load.It was contributed initially by facebook > as it worked for them. See > https://bugzilla.redhat.com/show_bug.cgi?id=1221737#c0.? So I think > this is expected. Unless you are saying that the `ls` slowness happens > even when there is no self-heal going on, in which case we need to see > what the issue is. >The defaults for the queue lengths are >1 (1024) if I recall. The default for max threads are 1. Even with all those at a value of 1 we reached an ls latency of 10s overnight, and I specicifically switched off backups before letting this run overnight to ensure that it would not interfere. Can the hardware handle it?? Well, let's look at the capacity status.? During threads above set to 16, and queue lengths at 10240.? All three hosts have approximately equal load values, with zoidberg (two data bricks compared to the other hosts one data brick and one "healing" arbiter that's newly added) being marginally (~2% on IO utilization) higher loaded, CPU was very similar.? I suspect this is because it's supplying two source bricks compared to the other hosts each have one source and one sink. During these stalls the IO wait is <10% on all hosts and according to iotop that's almost all glusterfsd.? This goes up slightly if there are LVM snapshots in place for backup purposes, which takes about an hour. CPU on glusterfsd < 50% according to top (ie, half a core's capacity).? 16 cores available. 64GB RAM total, and as of writing, free -m is reporting: ????????????? total??????? used??????? free????? shared? buff/cache?? available Mem:????????? 64413??????? 6600?????? 29011???????? 370?????? 28801?????? 56517 RAID controller on each host: ??????????????? description: RAID bus controller ??????????????? product: MegaRAID SAS 2108 [Liberator] ??????????????? vendor: Broadcom / LSI ?????????????????? description: SCSI Disk ?????????????????? product: PERC H700 ?????????????????? vendor: DELL Looking at the queue depth for the exposed drive: flexo /sys/class/block/sda # cat device/queue_depth 256 The disks themselves are in a RAID-1 configuration, 4TB NL-SAS drives from Seagate. On top of the RAID-exposed drive we run LVM and the gluster data is sitting in an ext4 filesystem on top of a 3TB LV inside of the single PV that makes up the VG I don't feel that there should be a problem with threads set to 4 at least, but probably even 8 or 16 should be OK.? I mean, until the IO capacity doesn't get to at least 70-80% it really should make little difference in my experience.? Technically even at 100% utilization it could just be a single IO request at any point in time (as we've seen cloning large disks) so in theory you can then still increase concurrency, but from experience we know that overall performance starts to degrade quite sharply at a certain tipping point and then you're in trouble with little to no warning. Under non-healing conditions IO utilization (iostat -dmx 1, iotop etc ... ie, percentage of time that has outstanding requests ignoring multi-concurrency) is generally about 0.3%. So somewhere there is definitely a bottleneck.? I don't mind deploying a custom "debug" patch on top of gluserfs 7.5 for the weekend but I generally need optimal performance again by Monday morning 6:30 SAST (GMT+2), then pushing the heal as hard as I possibly can overnight to get lock statistics or something.? If I can just know what options ./configure should be on/off, and which volume options needs to be set to get the right statistics, that's also helpful.? I'm even happy to then be pointed at the code and told "it's this process where the problem's at" (bottleneck, as it doesn't look like anything is broken really, just terribly unacceptably slow) - in other words, I'm happy to go attempt and contribute a patch on the issue, but I need a starting point of where to start scratching.? Can even do this Saturday after 15:00 till Sunday morning again. Kind Regards, Jaco> Regards, > > Ravi > >> It seems that when the logs >> >> With these the maximum stall I can reproduce is usually around 3 >> seconds before the directory listing will complete, but about 5s >> worst case.? I've now reverted these as well, which are known for us >> to cause issues under heal required conditions: >> >> cluster.metadata-self-heal????????????? >> on????????????????????????????????????? >> cluster.data-self-heal????????????????? >> on????????????????????????????????????? >> cluster.entry-self-heal???????????????? >> on????????????????????????????????????? >> >> Not sure if I need to remount for these to become effective but I've >> done so just to be on the safe side. >> >> I'm still seeing these errors: >> >> [2020-05-07 19:20:12.763311] W [MSGID: 114031] >> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] >> 0-vpbx_shared-client-1: remote operation failed [Invalid argument] >> [2020-05-07 19:20:12.995675] W [MSGID: 114031] >> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] >> 0-vpbx_shared-client-1: remote operation failed [Invalid argument] >> [2020-05-07 19:20:59.199776] W [MSGID: 114031] >> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] >> 0-vpbx_shared-client-1: remote operation failed [Invalid argument] >> >> And the start of my issues still seem to co-incide with that.? Sample >> case: >> >> # time ls >> ... >> real??? 0m5.140s >> user??? 0m0.009s >> sys??? 0m0.000s >> # f=(*); echo "${#f[@]}" >> 165 >> >> This just does not seem right.? Note, again, if I set *any* of the >> above parameters >1 these delays get far, far worse (I've had over a >> minute for this exact same ls). >> >> Any advise would be greatly appreciated.? To me, it looks like some >> lock is being taken somewhere, and takes a while to release.? I don't >> understand the glusterfs internals nearly well enough to understand >> exactly what's going on here, nor do I know what information to >> provide to help drill down on this.? Happy to go look at code, if I >> can get pointed in a direction. >> >> Kind Regards, >> Jaco >> >> On 2020/05/05 08:43, Jaco Kroon wrote: >>> >>> Hi Artem, >>> >>> I'll do that.? In the meantime consider this (inferred from the >>> changelogs). >>> >>> It seems like glusterfs SHD is queueing into a bounded queue under >>> lock conditions, and waiting for space to open up.? At least, this >>> is what's hinted in the changelogs, I've been trying to grasp the >>> SHD code for two days now and am not getting to wrap my head around >>> that.? If this is true, then the only way to avoid lock-ups are to >>> ensure that the queue is larger than the total number of files that >>> can ever require healing (effectively making the queue unbounded). >>> >>> Kind Regards, >>> Jaco >>> >>> On 2020/05/04 21:56, Artem Russakovskii wrote: >>>> I reported something similar recently to this list, and I highly >>>> advise you to update to 7.5 (in my case it was 5.11 -> 5.13), then >>>> see if it keeps happening. The self heal issues should be fixed >>>> there properly at least. >>>> >>>> Sincerely, >>>> Artem >>>> >>>> -- >>>> Founder, Android Police <http://www.androidpolice.com>,?APK Mirror >>>> <http://www.apkmirror.com/>, Illogical Robot LLC >>>> beerpla.net <http://beerpla.net/> | @ArtemR <http://twitter.com/ArtemR> >>>> >>>> >>>> On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za >>>> <mailto:jaco at uls.co.za>> wrote: >>>> >>>> Hi All, >>>> >>>> So we've got a cluster currently running on glusterfs 7.3 >>>> >>>> Heal performance is absolutely terrible (it starts of great, >>>> runs for a while then completely trips up). >>>> >>>> Once the SHD trips up, access speeds to the cluster also >>>> becomes poor.? And with poor I mean directory listings that >>>> normally takes well under 500ms takes upwards of 10s, but I've >>>> seen upwards of 40s (although a follow up ls immediately after >>>> is nice and fast again, temporarily).? With all SHDs killed >>>> this particular ls is as per normal (around 30-50ms). >>>> >>>> The SHD logs doesn't provide any usable clues, the only >>>> variance in logs is this, just before all daemons across the >>>> cluster crawls to a near halt. >>>> >>>> [2020-05-02 19:55:58.627036] I [MSGID: 108026] >>>> [afr-self-heal-common.c:1747:afr_log_selfheal] >>>> 0-vpbx_shared-replicate-0: Completed entry selfheal on >>>> 5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1 ? >>>> [2020-05-02 19:55:58.627058] I [MSGID: 108026] >>>> [afr-self-heal-common.c:1747:afr_log_selfheal] >>>> 0-vpbx_shared-replicate-0: Completed entry selfheal on >>>> c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1 ? >>>> ? >>>> [2020-05-02 19:55:59.125001] W [MSGID: 114031] >>>> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] >>>> 0-vpbx_shared-client-1: remote operation failed [Invalid argument] >>>> ? >>>> [2020-05-02 19:55:59.339803] I [MSGID: 108026] >>>> [afr-self-heal-common.c:1747:afr_log_selfheal] >>>> 0-vpbx_shared-replicate-0: Completed entry selfheal on >>>> 2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1? >>>> [2020-05-02 19:55:59.490387] I [MSGID: 108026] >>>> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] >>>> 0-vpbx_shared-replicate-0: performing metadata selfheal on >>>> 527a63a3-9acd-4c3f-b95c-5c719325c973 >>>> >>>> A few more entries will complete overall and then it'll be a >>>> file every other second.? Once I saw this I ran a few more >>>> tests, and on every test just prior to slowing to a crawl, that >>>> entry is there. >>>> >>>> There is a perceived improvement when lowering the thread >>>> count, but not significant enough to confirm a correlation.? >>>> Even with**cluster.shd-max-threads=1 this still happens (gets a >>>> few tens thousand files completed instead of few hundred, at 32 >>>> it gets to maybe 400 to 500 before it stalls, 16 does maybe a >>>> thousand).* >>>> * >>>> >>>> There are probably performant bursts hereafter, because from >>>> experience we know this process takes about two months.? But >>>> this time around it seems to be much, much worse.? Might be >>>> because that was healing a single brick to another brick, so >>>> less locking going?? At the moment the replicas are basically >>>> trying to populate the newly added arbiters.? Also, we haven't >>>> messed as greatly with the health check settings (queues and >>>> threads and locks nearly as much as since the arbiter addition >>>> because performance has simply dropped to non-acceptable levels. >>>> >>>> Unfortunately granular entry heal isn't on at the moment (nor >>>> can I switch it on due to a few million outstanding heals).? >>>> This used to be a 2x2 cluster.? We then moved one of the bricks >>>> to a newly added server using replace-brick.? What a >>>> frightening process this is.? We then proceeded to add the >>>> arbiters. >>>> >>>> Some of the settings below was absolute shots in the dark, >>>> based on the little information we've been able to track down >>>> so far. >>>> >>>> Kind Regards, >>>> Jaco >>>> >>>> The current gluster volume info (this used to be 2x2 >>>> distributed-replicate, aiming to get it to 3x (2 + 1) ... at >>>> current rates we might get there by 2021): >>>> >>>> Volume Name: glvol >>>> Type: Distributed-Replicate >>>> Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb >>>> Status: Started >>>> Snapshot Count: 0 >>>> Number of Bricks: 2 x (2 + 1) = 6 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: zoidberg:/mnt/gluster/a >>>> Brick2: flexo:/mnt/gluster/b >>>> Brick3: bender:/mnt/gluster/c (arbiter) >>>> Brick4: bender:/mnt/gluster/a >>>> Brick5: zoidberg:/mnt/gluster/b >>>> Brick6: flexo:/mnt/gluster/c (arbiter) >>>> Options Reconfigured: >>>> *cluster.heal-timeout: 5 <-- hoping this would at least shorten >>>> the blockages.* >>>> cluster.locking-scheme: granular >>>> cluster.heal-wait-queue-length: 10240 >>>> *diagnostics.count-fop-hits: on** >>>> **diagnostics.latency-measurement: on <-- was hoping profiling >>>> might provide some insight.? Going to switch this off now.* >>>> cluster.background-self-heal-count: 128 >>>> *cluster.shd-wait-qlength: 131072** >>>> **cluster.shd-max-threads: 32? <-- even with this disk IO never >>>> goes over 10% utilization.? CPU goes to about 2 cores.? So it's >>>> lock contention that's slowing it down we reckon.* >>>> *cluster.metadata-self-heal: off** >>>> **cluster.entry-self-heal: off** >>>> **cluster.data-self-heal: off**<-- these kill performance. >>>> *transport.address-family: inet >>>> nfs.disable: on >>>> performance.client-io-threads: on >>>> features.cache-invalidation: on >>>> features.cache-invalidation-timeout: 600 >>>> performance.stat-prefetch: on >>>> performance.cache-invalidation: on >>>> performance.md-cache-timeout: 600 >>>> *cluster.self-heal-daemon: yes <-- disabling/enabling this >>>> doesn't help, have to kill the shd and start ... force to get >>>> things moving along again.* >>>> performance.io-thread-count: 16 >>>> performance.high-prio-threads: 16 >>>> performance.normal-prio-threads: 16 >>>> performance.low-prio-threads: 16 >>>> performance.least-prio-threads: 4 >>>> cluster.self-heal-window-size: 512 >>>> cluster.self-heal-readdir-size: 131072 >>>> client.event-threads: 8 >>>> server.event-threads: 16 >>>> network.ping-timeout: 2 >>>> *cluster.data-self-heal-algorithm: full <-- most files are >>>> small (<1MB), on dual 1gbit links (so up to 2gbit, but b/w >>>> never goes over 5mbit anyway).* >>>> >>>> performance info: >>>> >>>> Brick: zoidberg:/mnt/gluster/a >>>> ----------------------------------------- >>>> Cumulative Stats: >>>> ?? Block Size:????????????????? 1b+?????????????????? >>>> 8b+????????????????? 16b+ >>>> ?No. of Reads:??????????????????? 0???????????????????? >>>> 0???????????????????? 0 >>>> No. of Writes:????????????????? 117?????????????????? >>>> 195?????????????????? 367 >>>> ? >>>> ?? Block Size:???????????????? 32b+????????????????? >>>> 64b+???????????????? 128b+ >>>> ?No. of Reads:????????????????? 306???????????????????? >>>> 0???????????????????? 6 >>>> No. of Writes:???????????????? 1016???????????????????? >>>> 9?????????????????? 134 >>>> ? >>>> ?? Block Size:??????????????? 256b+???????????????? >>>> 512b+??????????????? 1024b+ >>>> ?No. of Reads:????????????????? 107??????????????????? >>>> 58?????????????????? 485 >>>> No. of Writes:????????????????? 731??????????????????? >>>> 73?????????????????? 168 >>>> ? >>>> ?? Block Size:?????????????? 2048b+??????????????? >>>> 4096b+??????????????? 8192b+ >>>> ?No. of Reads:?????????????????? 43??????????????????? >>>> 59?????????????????? 383 >>>> No. of Writes:????????????????? 295?????????????????? >>>> 475???????????????? 10562 >>>> ? >>>> ?? Block Size:????????????? 16384b+?????????????? >>>> 32768b+?????????????? 65536b+ >>>> ?No. of Reads:????????????????? 347?????????????????? >>>> 380?????????????????? 634 >>>> No. of Writes:???????????????? 1487????????????????? >>>> 6369????????????????? 3970 >>>> ? >>>> ?? Block Size:???????????? 131072b+ >>>> ?No. of Reads:??????????????? 35704 >>>> No. of Writes:??????????????? 12398 >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 957293????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1455091???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 69951? RELEASEDIR >>>> ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us??????????? >>>> 113???????? IPC >>>> ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 >>>> us???????????? 30 REMOVEXATTR >>>> ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 >>>> us???????????? 52?? FTRUNCATE >>>> ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us?????????? >>>> 1385??????? STAT >>>> ????? 0.00???? 364.93 us????? 63.45 us?? 15957.66 us??????????? >>>> 810?? FGETXATTR >>>> ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us??????????? >>>> 137??????? LINK >>>> ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 >>>> us???????????? 12?????? FSYNC >>>> ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us??????????? >>>> 698??? TRUNCATE >>>> ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us??????????? >>>> 188????? RENAME >>>> ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us?????????? >>>> 1680???? SETATTR >>>> ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us??????????? >>>> 127?????? MKNOD >>>> ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us????????? >>>> 11412??? FINODELK >>>> ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us??????????? >>>> 221??? SETXATTR >>>> ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 >>>> us???????????? 30???? SYMLINK >>>> ????? 0.00????? 56.18 us?????? 7.31 us?? 41484.37 us????????? >>>> 22640?????? FLUSH >>>> ????? 0.00???? 105.64 us????? 19.34 us? 351055.72 us????????? >>>> 21693????? STATFS >>>> ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us??????????? >>>> 178?????? MKDIR >>>> ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us?????????? >>>> 9653?????? FSTAT >>>> ????? 0.00???? 214.87 us????? 25.71 us 1712020.99 us????????? >>>> 39054??????? READ >>>> ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us????????? >>>> 69951???? OPENDIR >>>> ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us?????????? >>>> 3180?????? RMDIR >>>> ????? 0.01???? 327.08 us????? 55.69 us? 420376.80 us????????? >>>> 38366?????? WRITE >>>> ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us?????????? >>>> 6169????? CREATE >>>> ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us??????? >>>> 1159936??? READLINK >>>> ????? 0.40???? 543.50 us????? 36.45 us 2430653.54 us??????? >>>> 1449307??????? OPEN >>>> ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us????????? >>>> 10787??? FXATTROP >>>> ????? 0.53??? 2342.72 us????? 20.91 us 4157542.43 us???????? >>>> 447821????? UNLINK >>>> ????? 0.53? 558734.79 us????? 24.40 us 1399487.63 us?????????? >>>> 1889????????? LK >>>> ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us????????? >>>> 10687???? READDIR >>>> ????? 0.99??? 2426.23 us????? 12.47 us 24176123.13 us???????? >>>> 809430??? GETXATTR >>>> ????? 1.45???? 676.83 us?????? 9.30 us 193177067.25 us??????? >>>> 4260021???? ENTRYLK >>>> ????? 1.76?? 38325.38 us????? 21.44 us 46693575.93 us????????? >>>> 91439??? READDIRP >>>> ????? 4.44??? 1200.31 us?????? 9.10 us 2559377.90 us??????? >>>> 7352811???? INODELK >>>> ????? 9.30??? 1849.24 us?????? 9.28 us 35299050.00 us??????? >>>> 9989221????? LOOKUP >>>> ???? 79.24?? 47791.46 us????? 76.77 us 10488824.26 us??????? >>>> 3294642???? XATTROP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 3194806????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 5311???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>> us???????????? 54??? CI_XATTR >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 1907?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 3189457?? CI_FORGET >>>> ? >>>> ??? Duration: 209524 seconds >>>> ?? Data Read: 4775365531 bytes >>>> Data Written: 2354108664 bytes >>>> ? >>>> Interval 0 Stats: >>>> ?? Block Size:????????????????? 1b+?????????????????? >>>> 8b+????????????????? 16b+ >>>> ?No. of Reads:??????????????????? 0???????????????????? >>>> 0???????????????????? 0 >>>> No. of Writes:????????????????? 117?????????????????? >>>> 195?????????????????? 367 >>>> ? >>>> ?? Block Size:???????????????? 32b+????????????????? >>>> 64b+???????????????? 128b+ >>>> ?No. of Reads:????????????????? 306???????????????????? >>>> 0???????????????????? 6 >>>> No. of Writes:???????????????? 1016???????????????????? >>>> 9?????????????????? 134 >>>> ? >>>> ?? Block Size:??????????????? 256b+???????????????? >>>> 512b+??????????????? 1024b+ >>>> ?No. of Reads:????????????????? 107??????????????????? >>>> 58?????????????????? 485 >>>> No. of Writes:????????????????? 731??????????????????? >>>> 73?????????????????? 168 >>>> ? >>>> ?? Block Size:?????????????? 2048b+??????????????? >>>> 4096b+??????????????? 8192b+ >>>> ?No. of Reads:?????????????????? 43??????????????????? >>>> 59?????????????????? 383 >>>> No. of Writes:????????????????? 295?????????????????? >>>> 475???????????????? 10562 >>>> ? >>>> ?? Block Size:????????????? 16384b+?????????????? >>>> 32768b+?????????????? 65536b+ >>>> ?No. of Reads:????????????????? 347?????????????????? >>>> 380?????????????????? 634 >>>> No. of Writes:???????????????? 1487????????????????? >>>> 6369????????????????? 3970 >>>> ? >>>> ?? Block Size:???????????? 131072b+ >>>> ?No. of Reads:??????????????? 35704 >>>> No. of Writes:??????????????? 12398 >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 957293????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1455091???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 69951? RELEASEDIR >>>> ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 us??????????? >>>> 113???????? IPC >>>> ????? 0.00???? 687.33 us????? 80.53 us?? 15414.03 >>>> us???????????? 30 REMOVEXATTR >>>> ????? 0.00???? 840.76 us????? 72.71 us?? 37648.39 >>>> us???????????? 52?? FTRUNCATE >>>> ????? 0.00???? 138.49 us????? 36.66 us?? 25280.24 us?????????? >>>> 1385??????? STAT >>>> ????? 0.00???? 363.20 us????? 63.45 us?? 15957.66 us??????????? >>>> 810?? FGETXATTR >>>> ????? 0.00??? 2208.82 us???? 127.23 us? 267092.17 us??????????? >>>> 137??????? LINK >>>> ????? 0.00?? 37595.33 us??? 1677.46 us? 218313.32 >>>> us???????????? 12?????? FSYNC >>>> ????? 0.00???? 675.61 us???? 106.82 us? 114351.01 us??????????? >>>> 698??? TRUNCATE >>>> ????? 0.00??? 3995.66 us???? 147.00 us? 109544.46 us??????????? >>>> 188????? RENAME >>>> ????? 0.00???? 453.11 us????? 53.37 us? 104848.94 us?????????? >>>> 1680???? SETATTR >>>> ????? 0.00??? 6004.32 us???? 229.33 us? 151322.58 us??????????? >>>> 127?????? MKNOD >>>> ????? 0.00????? 68.74 us????? 16.11 us?? 61050.93 us????????? >>>> 11412??? FINODELK >>>> ????? 0.00??? 3793.44 us????? 68.61 us? 411523.70 us??????????? >>>> 221??? SETXATTR >>>> ????? 0.00?? 28669.99 us???? 227.28 us? 138988.41 >>>> us???????????? 30???? SYMLINK >>>> ????? 0.00????? 56.19 us?????? 7.31 us?? 41484.37 us????????? >>>> 22640?????? FLUSH >>>> ????? 0.00???? 105.61 us????? 19.34 us? 351055.72 us????????? >>>> 21693????? STATFS >>>> ????? 0.00?? 23642.90 us???? 244.19 us? 234912.25 us??????????? >>>> 178?????? MKDIR >>>> ????? 0.00???? 537.38 us????? 31.73 us? 296911.67 us?????????? >>>> 9653?????? FSTAT >>>> ????? 0.00???? 214.91 us????? 25.71 us 1712020.99 us????????? >>>> 39054??????? READ >>>> ????? 0.01???? 147.08 us?????? 1.47 us? 241611.80 us????????? >>>> 69951???? OPENDIR >>>> ????? 0.01??? 3899.84 us???? 117.22 us? 503422.21 us?????????? >>>> 3180?????? RMDIR >>>> ????? 0.01???? 327.13 us????? 55.69 us? 420376.80 us????????? >>>> 38366?????? WRITE >>>> ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us?????????? >>>> 6169????? CREATE >>>> ????? 0.05????? 92.69 us????? 29.94 us? 687955.86 us??????? >>>> 1159936??? READLINK >>>> ????? 0.40???? 543.49 us????? 36.45 us 2430653.54 us??????? >>>> 1449307??????? OPEN >>>> ????? 0.48?? 87643.67 us????? 70.20 us? 661999.51 us????????? >>>> 10787??? FXATTROP >>>> ????? 0.53??? 2342.57 us????? 20.91 us 4157542.43 us???????? >>>> 447821????? UNLINK >>>> ????? 0.53? 558416.10 us????? 24.40 us 1399487.63 us?????????? >>>> 1889????????? LK >>>> ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us????????? >>>> 10687???? READDIR >>>> ????? 0.99??? 2426.24 us????? 12.47 us 24176123.13 us???????? >>>> 809430??? GETXATTR >>>> ????? 1.45???? 676.75 us?????? 9.30 us 193177067.25 us??????? >>>> 4260021???? ENTRYLK >>>> ????? 1.76?? 38324.59 us????? 21.44 us 46693575.93 us????????? >>>> 91439??? READDIRP >>>> ????? 4.44??? 1200.37 us?????? 9.10 us 2559377.90 us??????? >>>> 7352811???? INODELK >>>> ????? 9.29??? 1848.94 us?????? 9.28 us 35299050.00 us??????? >>>> 9989221????? LOOKUP >>>> ???? 79.24?? 47792.42 us????? 76.77 us 10488824.26 us??????? >>>> 3294642???? XATTROP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 3194806????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 5311???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>> us???????????? 54??? CI_XATTR >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 1907?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 3189457?? CI_FORGET >>>> ? >>>> ??? Duration: 209524 seconds >>>> ?? Data Read: 4775365531 bytes >>>> Data Written: 2354108664 bytes >>>> ? >>>> Brick: zoidberg:/mnt/gluster/b >>>> ----------------------------------------- >>>> Cumulative Stats: >>>> ?? Block Size:????????????????? 1b+?????????????????? >>>> 2b+?????????????????? 8b+ >>>> ?No. of Reads:??????????????????? 0???????????????????? >>>> 0???????????????????? 0 >>>> No. of Writes:?????????????????? 53???????????????????? >>>> 1??????????????????? 60 >>>> ? >>>> ?? Block Size:???????????????? 16b+????????????????? >>>> 32b+????????????????? 64b+ >>>> ?No. of Reads:??????????????????? 0?????????????????? >>>> 213???????????????????? 2 >>>> No. of Writes:?????????????????? 14????????????????? >>>> 1057??????????????????? 15 >>>> ? >>>> ?? Block Size:??????????????? 128b+???????????????? >>>> 256b+???????????????? 512b+ >>>> ?No. of Reads:??????????????????? 0??????????????????? >>>> 41?????????????????? 146 >>>> No. of Writes:?????????????????? 59?????????????????? >>>> 715??????????????????? 80 >>>> ? >>>> ?? Block Size:?????????????? 1024b+??????????????? >>>> 2048b+??????????????? 4096b+ >>>> ?No. of Reads:????????????????? 224???????????????????? >>>> 9??????????????????? 15 >>>> No. of Writes:????????????????? 163?????????????????? >>>> 282?????????????????? 404 >>>> ? >>>> ?? Block Size:?????????????? 8192b+?????????????? >>>> 16384b+?????????????? 32768b+ >>>> ?No. of Reads:?????????????????? 52??????????????????? >>>> 74?????????????????? 210 >>>> No. of Writes:???????????????? 7415????????????????? >>>> 1430????????????????? 6860 >>>> ? >>>> ?? Block Size:????????????? 65536b+????????????? >>>> 131072b+????????????? 262144b+ >>>> ?No. of Reads:????????????????? 299???????????????? >>>> 15313???????????????????? 1 >>>> No. of Writes:???????????????? 2458???????????????? >>>> 12378???????????????????? 1 >>>> ? >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 686733????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 837434???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 87338? RELEASEDIR >>>> ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us??????????? >>>> 116???????? IPC >>>> ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 >>>> us???????????? 57?? FTRUNCATE >>>> ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 >>>> us???????????? 65?? FGETXATTR >>>> ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 >>>> us???????????? 23 REMOVEXATTR >>>> ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us??????????? >>>> 121??????? LINK >>>> ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us??????????? >>>> 490????????? LK >>>> ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us??????????? >>>> 540??????? STAT >>>> ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us??????????? >>>> 200??? TRUNCATE >>>> ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us?????????? >>>> 1482???? SETATTR >>>> ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 >>>> us???????????? 12?????? FSYNC >>>> ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us????????? >>>> 10047?????? FLUSH >>>> ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us?????????? >>>> 9977??? FINODELK >>>> ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us??????????? >>>> 188????? RENAME >>>> ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 >>>> us???????????? 25???? SYMLINK >>>> ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us??????????? >>>> 118?????? MKNOD >>>> ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us????????? >>>> 12587?????? FSTAT >>>> ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us????????? >>>> 21695????? STATFS >>>> ????? 0.00???? 307.86 us????? 28.00 us? 292666.83 us????????? >>>> 16841??????? READ >>>> ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us??????????? >>>> 178?????? MKDIR >>>> ????? 0.00???? 319.24 us????? 51.19 us? 136353.39 us????????? >>>> 33445?????? WRITE >>>> ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us????????? >>>> 87338???? OPENDIR >>>> ????? 0.01???? 249.96 us????? 58.42 us? 594774.86 us????????? >>>> 85089??? SETXATTR >>>> ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us?????????? >>>> 3180?????? RMDIR >>>> ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us?????????? >>>> 6067????? CREATE >>>> ????? 0.05????? 89.45 us????? 30.14 us? 687961.60 us??????? >>>> 1219542??? READLINK >>>> ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us???????? >>>> 831371??????? OPEN >>>> ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us?????????? >>>> 9493??? FXATTROP >>>> ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us??????? >>>> 3764714???? ENTRYLK >>>> ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us????????? >>>> 22253???? READDIR >>>> ????? 1.06??? 6913.71 us????? 11.58 us 36968532.72 us???????? >>>> 364963??? GETXATTR >>>> ????? 1.29?? 62130.12 us????? 36.89 us 8649871.11 us????????? >>>> 49472??? READDIRP >>>> ????? 2.89??? 1477.53 us????? 11.10 us 2807637.64 us??????? >>>> 4650500???? INODELK >>>> ????? 4.12?? 23681.31 us????? 20.98 us 23291414.50 us???????? >>>> 414352????? UNLINK >>>> ???? 16.09??? 5000.31 us?????? 9.04 us 23866016.88 us??????? >>>> 7655621????? LOOKUP >>>> ???? 72.04?? 68953.31 us????? 68.40 us 27542499.65 us??????? >>>> 2486126???? XATTROP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 2433405????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 2453???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>> us???????????? 69??? CI_XATTR >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 1544?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 2430885?? CI_FORGET >>>> ? >>>> ??? Duration: 209524 seconds >>>> ?? Data Read: 2051388524 bytes >>>> Data Written: 2208261499 bytes >>>> ? >>>> Interval 0 Stats: >>>> ?? Block Size:????????????????? 1b+?????????????????? >>>> 2b+?????????????????? 8b+ >>>> ?No. of Reads:??????????????????? 0???????????????????? >>>> 0???????????????????? 0 >>>> No. of Writes:?????????????????? 53???????????????????? >>>> 1??????????????????? 60 >>>> ? >>>> ?? Block Size:???????????????? 16b+????????????????? >>>> 32b+????????????????? 64b+ >>>> ?No. of Reads:??????????????????? 0?????????????????? >>>> 213???????????????????? 2 >>>> No. of Writes:?????????????????? 14????????????????? >>>> 1057??????????????????? 15 >>>> ? >>>> ?? Block Size:??????????????? 128b+???????????????? >>>> 256b+???????????????? 512b+ >>>> ?No. of Reads:??????????????????? 0??????????????????? >>>> 41?????????????????? 146 >>>> No. of Writes:?????????????????? 59?????????????????? >>>> 715??????????????????? 80 >>>> ? >>>> ?? Block Size:?????????????? 1024b+??????????????? >>>> 2048b+??????????????? 4096b+ >>>> ?No. of Reads:????????????????? 224???????????????????? >>>> 9??????????????????? 15 >>>> No. of Writes:????????????????? 163?????????????????? >>>> 282?????????????????? 404 >>>> ? >>>> ?? Block Size:?????????????? 8192b+?????????????? >>>> 16384b+?????????????? 32768b+ >>>> ?No. of Reads:?????????????????? 52??????????????????? >>>> 74?????????????????? 210 >>>> No. of Writes:???????????????? 7415????????????????? >>>> 1430????????????????? 6860 >>>> ? >>>> ?? Block Size:????????????? 65536b+????????????? >>>> 131072b+????????????? 262144b+ >>>> ?No. of Reads:????????????????? 299???????????????? >>>> 15313???????????????????? 1 >>>> No. of Writes:???????????????? 2458???????????????? >>>> 12378???????????????????? 1 >>>> ? >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 686733????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 837434???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 87338? RELEASEDIR >>>> ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 us??????????? >>>> 116???????? IPC >>>> ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 >>>> us???????????? 57?? FTRUNCATE >>>> ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 >>>> us???????????? 65?? FGETXATTR >>>> ????? 0.00??? 2336.06 us????? 90.05 us?? 20509.81 >>>> us???????????? 23 REMOVEXATTR >>>> ????? 0.00???? 631.26 us???? 123.17 us?? 51803.77 us??????????? >>>> 121??????? LINK >>>> ????? 0.00???? 369.27 us????? 22.64 us? 145090.48 us??????????? >>>> 490????????? LK >>>> ????? 0.00???? 360.85 us????? 45.14 us?? 46846.98 us??????????? >>>> 540??????? STAT >>>> ????? 0.00??? 1190.96 us???? 117.40 us?? 69781.85 us??????????? >>>> 200??? TRUNCATE >>>> ????? 0.00???? 245.72 us????? 54.91 us? 123985.76 us?????????? >>>> 1482???? SETATTR >>>> ????? 0.00?? 42139.10 us??? 3313.56 us? 312273.21 >>>> us???????????? 12?????? FSYNC >>>> ????? 0.00????? 55.83 us?????? 8.26 us?? 49649.35 us????????? >>>> 10047?????? FLUSH >>>> ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us?????????? >>>> 9977??? FINODELK >>>> ????? 0.00??? 3191.60 us???? 143.06 us?? 96214.75 us??????????? >>>> 188????? RENAME >>>> ????? 0.00?? 32862.86 us???? 215.38 us?? 83851.13 >>>> us???????????? 25???? SYMLINK >>>> ????? 0.00?? 11392.03 us???? 230.70 us? 105471.90 us??????????? >>>> 118?????? MKNOD >>>> ????? 0.00???? 134.63 us????? 28.08 us?? 36326.43 us????????? >>>> 12587?????? FSTAT >>>> ????? 0.00???? 112.32 us????? 19.91 us? 381241.34 us????????? >>>> 21695????? STATFS >>>> ????? 0.00???? 307.91 us????? 28.00 us? 292666.83 us????????? >>>> 16841??????? READ >>>> ????? 0.00?? 29625.60 us???? 239.57 us? 239532.26 us??????????? >>>> 178?????? MKDIR >>>> ????? 0.00???? 319.22 us????? 51.19 us? 136353.39 us????????? >>>> 33445?????? WRITE >>>> ????? 0.01???? 175.83 us?????? 1.57 us? 513635.22 us????????? >>>> 87338???? OPENDIR >>>> ????? 0.01???? 249.95 us????? 58.42 us? 594774.86 us????????? >>>> 85089??? SETXATTR >>>> ????? 0.01?? 11001.25 us???? 119.87 us? 612765.03 us?????????? >>>> 3180?????? RMDIR >>>> ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us?????????? >>>> 6067????? CREATE >>>> ????? 0.05????? 89.46 us????? 30.14 us? 687961.60 us??????? >>>> 1219542??? READLINK >>>> ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us???????? >>>> 831371??????? OPEN >>>> ????? 0.51? 127078.45 us????? 78.55 us? 694364.09 us?????????? >>>> 9493??? FXATTROP >>>> ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us??????? >>>> 3764714???? ENTRYLK >>>> ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us????????? >>>> 22253???? READDIR >>>> ????? 1.06??? 6913.26 us????? 11.58 us 36968532.72 us???????? >>>> 364963??? GETXATTR >>>> ????? 1.29?? 62121.95 us????? 36.89 us 8649871.11 us????????? >>>> 49472??? READDIRP >>>> ????? 2.89??? 1477.58 us????? 11.10 us 2807637.64 us??????? >>>> 4650500???? INODELK >>>> ????? 4.12?? 23681.29 us????? 20.98 us 23291414.50 us???????? >>>> 414352????? UNLINK >>>> ???? 16.09??? 5000.02 us?????? 9.04 us 23866016.88 us??????? >>>> 7655621????? LOOKUP >>>> ???? 72.04?? 68956.14 us????? 68.40 us 27542499.65 us??????? >>>> 2486126???? XATTROP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 2433405????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 2453???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>> us???????????? 69??? CI_XATTR >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 1544?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 2430885?? CI_FORGET >>>> ? >>>> ??? Duration: 209524 seconds >>>> ?? Data Read: 2051388524 bytes >>>> Data Written: 2208261499 bytes >>>> ? >>>> Brick: flexo:/mnt/gluster/b >>>> -------------------------------------- >>>> Cumulative Stats: >>>> ?? Block Size:????????????????? 1b+?????????????????? >>>> 8b+????????????????? 16b+ >>>> ?No. of Reads:??????????????????? 0???????????????????? >>>> 0???????????????????? 0 >>>> No. of Writes:????????????????? 117?????????????????? >>>> 195?????????????????? 367 >>>> ? >>>> ?? Block Size:???????????????? 32b+????????????????? >>>> 64b+???????????????? 128b+ >>>> ?No. of Reads:????????????????? 130???????????????????? >>>> 0???????????????????? 3 >>>> No. of Writes:???????????????? 1012???????????????????? >>>> 9?????????????????? 134 >>>> ? >>>> ?? Block Size:??????????????? 256b+???????????????? >>>> 512b+??????????????? 1024b+ >>>> ?No. of Reads:?????????????????? 47???????????????????? >>>> 8?????????????????? 257 >>>> No. of Writes:????????????????? 729??????????????????? >>>> 73?????????????????? 168 >>>> ? >>>> ?? Block Size:?????????????? 2048b+??????????????? >>>> 4096b+??????????????? 8192b+ >>>> ?No. of Reads:?????????????????? 10??????????????????? >>>> 27?????????????????? 240 >>>> No. of Writes:????????????????? 294?????????????????? >>>> 466???????????????? 10472 >>>> ? >>>> ?? Block Size:????????????? 16384b+?????????????? >>>> 32768b+?????????????? 65536b+ >>>> ?No. of Reads:????????????????? 139?????????????????? >>>> 119?????????????????? 135 >>>> No. of Writes:???????????????? 1480????????????????? >>>> 6273????????????????? 3935 >>>> ? >>>> ?? Block Size:???????????? 131072b+ >>>> ?No. of Reads:???????????????? 7892 >>>> No. of Writes:??????????????? 12327 >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 955058????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1450749???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 81377? RELEASEDIR >>>> ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us??????????? >>>> 108???????? IPC >>>> ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 >>>> us???????????? 44?? FTRUNCATE >>>> ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 >>>> us???????????? 30 REMOVEXATTR >>>> ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 >>>> us????????????? 4?????? FSYNC >>>> ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 >>>> us???????????? 30???? SYMLINK >>>> ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us??????????? >>>> 137??????? LINK >>>> ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us??????????? >>>> 119?????? MKNOD >>>> ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us??????????? >>>> 221??? SETXATTR >>>> ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us??????????? >>>> 354??????? STAT >>>> ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us??????????? >>>> 777????????? LK >>>> ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us??????????? >>>> 188????? RENAME >>>> ????? 0.00???? 150.40 us????? 75.59 us???? 749.54 us??????????? >>>> 810?? FGETXATTR >>>> ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us??????????? >>>> 698??? TRUNCATE >>>> ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us??????????? >>>> 178?????? MKDIR >>>> ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us?????????? >>>> 1665???? SETATTR >>>> ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us????????? >>>> 11372??? FINODELK >>>> ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us?????????? >>>> 7398?????? FSTAT >>>> ????? 0.00???? 120.71 us????? 31.43 us???? 847.97 us?????????? >>>> 9094??????? READ >>>> ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us????????? >>>> 22614?????? FLUSH >>>> ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us????????? >>>> 21656????? STATFS >>>> ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us?????????? >>>> 6145????? CREATE >>>> ????? 0.00???? 217.55 us????? 50.60 us?? 30734.99 us????????? >>>> 38051?????? WRITE >>>> ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us????????? >>>> 81377???? OPENDIR >>>> ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us?????????? >>>> 3177?????? RMDIR >>>> ????? 0.03???? 100.05 us????? 30.93 us?? 49756.93 us???????? >>>> 432439??? READLINK >>>> ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us???????? >>>> 448381????? UNLINK >>>> ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us????????? >>>> 52185??? READDIRP >>>> ????? 0.35???? 414.05 us????? 36.58 us 1038229.93 us??????? >>>> 1444606??????? OPEN >>>> ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us????????? >>>> 10745??? FXATTROP >>>> ????? 0.72???? 776.21 us????? 11.24 us 4271792.57 us??????? >>>> 1605822??? GETXATTR >>>> ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us????????? >>>> 13532???? READDIR >>>> ????? 0.79???? 320.44 us?????? 7.71 us 2369511.06 us??????? >>>> 4257297???? ENTRYLK >>>> ????? 1.97???? 464.02 us?????? 9.38 us 1038460.80 us??????? >>>> 7333722???? INODELK >>>> ???? 13.73??? 2372.50 us?????? 8.33 us 6258159.87 us??????? >>>> 9979799????? LOOKUP >>>> ???? 80.79?? 42372.04 us????? 61.49 us 2238308.13 us??????? >>>> 3289103???? XATTROP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 2250268????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 3756???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? >>>> 167??? CI_XATTR >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 1908?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 2246352?? CI_FORGET >>>> ? >>>> ??? Duration: 209334 seconds >>>> ?? Data Read: 1059777395 bytes >>>> Data Written: 2336921014 bytes >>>> ? >>>> Interval 0 Stats: >>>> ?? Block Size:????????????????? 1b+?????????????????? >>>> 8b+????????????????? 16b+ >>>> ?No. of Reads:??????????????????? 0???????????????????? >>>> 0???????????????????? 0 >>>> No. of Writes:????????????????? 117?????????????????? >>>> 195?????????????????? 367 >>>> ? >>>> ?? Block Size:???????????????? 32b+????????????????? >>>> 64b+???????????????? 128b+ >>>> ?No. of Reads:????????????????? 130???????????????????? >>>> 0???????????????????? 3 >>>> No. of Writes:???????????????? 1012???????????????????? >>>> 9?????????????????? 134 >>>> ? >>>> ?? Block Size:??????????????? 256b+???????????????? >>>> 512b+??????????????? 1024b+ >>>> ?No. of Reads:?????????????????? 47???????????????????? >>>> 8?????????????????? 257 >>>> No. of Writes:????????????????? 729??????????????????? >>>> 73?????????????????? 168 >>>> ? >>>> ?? Block Size:?????????????? 2048b+??????????????? >>>> 4096b+??????????????? 8192b+ >>>> ?No. of Reads:?????????????????? 10??????????????????? >>>> 27?????????????????? 240 >>>> No. of Writes:????????????????? 294?????????????????? >>>> 466???????????????? 10472 >>>> ? >>>> ?? Block Size:????????????? 16384b+?????????????? >>>> 32768b+?????????????? 65536b+ >>>> ?No. of Reads:????????????????? 139?????????????????? >>>> 119?????????????????? 135 >>>> No. of Writes:???????????????? 1480????????????????? >>>> 6273????????????????? 3935 >>>> ? >>>> ?? Block Size:???????????? 131072b+ >>>> ?No. of Reads:???????????????? 7892 >>>> No. of Writes:??????????????? 12327 >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 955058????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1450749???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 81377? RELEASEDIR >>>> ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 us??????????? >>>> 108???????? IPC >>>> ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 >>>> us???????????? 44?? FTRUNCATE >>>> ????? 0.00???? 198.38 us????? 81.68 us??? 1332.45 >>>> us???????????? 30 REMOVEXATTR >>>> ????? 0.00??? 4254.16 us???? 621.52 us?? 10602.99 >>>> us????????????? 4?????? FSYNC >>>> ????? 0.00???? 795.08 us???? 190.30 us?? 16518.64 >>>> us???????????? 30???? SYMLINK >>>> ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 us??????????? >>>> 137??????? LINK >>>> ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 us??????????? >>>> 119?????? MKNOD >>>> ????? 0.00???? 162.61 us????? 65.62 us??? 3864.77 us??????????? >>>> 221??? SETXATTR >>>> ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 us??????????? >>>> 354??????? STAT >>>> ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 us??????????? >>>> 777????????? LK >>>> ????? 0.00???? 461.72 us???? 114.54 us?? 16013.81 us??????????? >>>> 188????? RENAME >>>> ????? 0.00???? 150.43 us????? 75.59 us???? 749.54 us??????????? >>>> 810?? FGETXATTR >>>> ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 us??????????? >>>> 698??? TRUNCATE >>>> ????? 0.00??? 1176.00 us???? 194.89 us?? 78261.12 us??????????? >>>> 178?????? MKDIR >>>> ????? 0.00???? 439.83 us????? 54.38 us?? 75270.51 us?????????? >>>> 1665???? SETATTR >>>> ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us????????? >>>> 11372??? FINODELK >>>> ????? 0.00???? 131.57 us????? 32.68 us??? 2107.52 us?????????? >>>> 7398?????? FSTAT >>>> ????? 0.00???? 120.74 us????? 31.43 us???? 847.97 us?????????? >>>> 9094??????? READ >>>> ????? 0.00????? 60.75 us?????? 7.25 us?? 22520.98 us????????? >>>> 22614?????? FLUSH >>>> ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us????????? >>>> 21656????? STATFS >>>> ????? 0.00???? 323.63 us???? 166.56 us?? 23034.44 us?????????? >>>> 6145????? CREATE >>>> ????? 0.00???? 217.50 us????? 50.60 us?? 30734.99 us????????? >>>> 38051?????? WRITE >>>> ????? 0.01???? 146.39 us?????? 1.80 us? 318612.45 us????????? >>>> 81377???? OPENDIR >>>> ????? 0.01??? 5641.37 us???? 108.84 us? 468960.33 us?????????? >>>> 3177?????? RMDIR >>>> ????? 0.03???? 100.06 us????? 30.93 us?? 49756.93 us???????? >>>> 432439??? READLINK >>>> ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us???????? >>>> 448381????? UNLINK >>>> ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us????????? >>>> 52185??? READDIRP >>>> ????? 0.35???? 414.01 us????? 36.58 us 1038229.93 us??????? >>>> 1444606??????? OPEN >>>> ????? 0.49?? 78518.02 us????? 73.39 us? 373999.31 us????????? >>>> 10745??? FXATTROP >>>> ????? 0.72???? 776.24 us????? 11.24 us 4271792.57 us??????? >>>> 1605822??? GETXATTR >>>> ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us????????? >>>> 13532???? READDIR >>>> ????? 0.79???? 320.45 us?????? 7.71 us 2369511.06 us??????? >>>> 4257297???? ENTRYLK >>>> ????? 1.97???? 464.07 us?????? 9.38 us 1038460.80 us??????? >>>> 7333722???? INODELK >>>> ???? 13.72??? 2372.42 us?????? 8.33 us 6258159.87 us??????? >>>> 9979799????? LOOKUP >>>> ???? 80.79?? 42373.50 us????? 61.49 us 2238308.13 us??????? >>>> 3289103???? XATTROP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 2250268????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 3756???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? >>>> 167??? CI_XATTR >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 1908?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 2246352?? CI_FORGET >>>> ? >>>> ??? Duration: 209334 seconds >>>> ?? Data Read: 1059777395 bytes >>>> Data Written: 2336921014 bytes >>>> ? >>>> Brick: flexo:/mnt/gluster/c >>>> -------------------------------------- >>>> Cumulative Stats: >>>> ?? Block Size:????????????????? 1b+ >>>> ?No. of Reads:??????????????????? 0 >>>> No. of Writes:??????????????? 32843 >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 686562????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 834596???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 93757? RELEASEDIR >>>> ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us??????????? >>>> 107???????? IPC >>>> ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 >>>> us???????????? 36??????? LINK >>>> ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 >>>> us???????????? 65?? FGETXATTR >>>> ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 >>>> us???????????? 88????? RENAME >>>> ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us??????????? >>>> 153??? TRUNCATE >>>> ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us??????????? >>>> 491????????? LK >>>> ????? 0.01????? 51.73 us?????? 7.70 us??? 1174.64 us?????????? >>>> 9339?????? FLUSH >>>> ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us?????????? >>>> 9011??? FINODELK >>>> ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us????????? >>>> 10824?????? FSTAT >>>> ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us?????????? >>>> 2594?????? MKDIR >>>> ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us?????????? >>>> 9009??? FXATTROP >>>> ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us?????????? >>>> 5754????? CREATE >>>> ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us????????? >>>> 32843?????? WRITE >>>> ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us?????????? >>>> 5798???? SYMLINK >>>> ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us????????? >>>> 34904???? READDIR >>>> ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us?????????? >>>> 2732?????? RMDIR >>>> ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us????????? >>>> 93757???? OPENDIR >>>> ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us???????? >>>> 170292????? UNLINK >>>> ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us????????? >>>> 13840??? READDIRP >>>> ????? 1.94???? 200.88 us????? 25.49 us? 461654.32 us???????? >>>> 468842?? FTRUNCATE >>>> ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us???????? >>>> 486882 REMOVEXATTR >>>> ????? 2.27???? 223.01 us????? 52.03 us? 178897.40 us???????? >>>> 493240??? SETXATTR >>>> ????? 3.53???? 155.31 us?????? 7.96 us?? 67873.00 us??????? >>>> 1101338??? GETXATTR >>>> ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us???????? >>>> 829002??????? OPEN >>>> ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us??????? >>>> 3660425???? ENTRYLK >>>> ????? 6.23???? 304.93 us????? 42.51 us? 661394.43 us???????? >>>> 990644???? SETATTR >>>> ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us???????? >>>> 990766???? XATTROP >>>> ???? 12.39???? 549.11 us???? 135.57 us 1659697.83 us??????? >>>> 1093387?????? MKNOD >>>> ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us??????? >>>> 4325281???? INODELK >>>> ???? 39.53???? 206.94 us?????? 8.56 us? 964823.33 us??????? >>>> 9259429????? LOOKUP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1856377????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 933300???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? >>>> 251??? CI_XATTR >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? >>>> 293?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 923077?? CI_FORGET >>>> ? >>>> ??? Duration: 209334 seconds >>>> ?? Data Read: 0 bytes >>>> Data Written: 32843 bytes >>>> ? >>>> Interval 0 Stats: >>>> ?? Block Size:????????????????? 1b+ >>>> ?No. of Reads:??????????????????? 0 >>>> No. of Writes:??????????????? 32843 >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 686562????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 834596???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 93757? RELEASEDIR >>>> ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 us??????????? >>>> 107???????? IPC >>>> ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 >>>> us???????????? 36??????? LINK >>>> ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 >>>> us???????????? 65?? FGETXATTR >>>> ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 >>>> us???????????? 88????? RENAME >>>> ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 us??????????? >>>> 153??? TRUNCATE >>>> ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 us??????????? >>>> 491????????? LK >>>> ????? 0.01????? 51.71 us?????? 7.70 us??? 1174.64 us?????????? >>>> 9339?????? FLUSH >>>> ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us?????????? >>>> 9011??? FINODELK >>>> ????? 0.03???? 126.23 us????? 43.15 us??? 4760.14 us????????? >>>> 10824?????? FSTAT >>>> ????? 0.03???? 571.94 us????? 94.85 us?? 51697.03 us?????????? >>>> 2594?????? MKDIR >>>> ????? 0.05???? 264.87 us????? 81.58 us?? 53152.06 us?????????? >>>> 9009??? FXATTROP >>>> ????? 0.05???? 423.05 us????? 38.26 us? 548686.55 us?????????? >>>> 5754????? CREATE >>>> ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us????????? >>>> 32843?????? WRITE >>>> ????? 0.06???? 526.39 us???? 130.84 us?? 41405.46 us?????????? >>>> 5798???? SYMLINK >>>> ????? 0.10???? 133.73 us????? 16.25 us? 676494.72 us????????? >>>> 34904???? READDIR >>>> ????? 0.12??? 2054.48 us????? 99.28 us?? 88061.83 us?????????? >>>> 2732?????? RMDIR >>>> ????? 0.21???? 110.60 us?????? 1.19 us?? 16833.78 us????????? >>>> 93757???? OPENDIR >>>> ????? 0.72???? 205.89 us????? 33.28 us? 261023.99 us???????? >>>> 170292????? UNLINK >>>> ????? 0.85??? 2971.45 us????? 21.24 us? 257839.19 us????????? >>>> 13840??? READDIRP >>>> ????? 1.94???? 200.87 us????? 25.49 us? 461654.32 us???????? >>>> 468842?? FTRUNCATE >>>> ????? 2.13???? 212.44 us????? 54.02 us? 461878.34 us???????? >>>> 486882 REMOVEXATTR >>>> ????? 2.27???? 223.00 us????? 52.03 us? 178897.40 us???????? >>>> 493240??? SETXATTR >>>> ????? 3.53???? 155.32 us?????? 7.96 us?? 67873.00 us??????? >>>> 1101338??? GETXATTR >>>> ????? 3.63???? 212.32 us????? 44.42 us? 551264.02 us???????? >>>> 829002??????? OPEN >>>> ????? 5.77????? 76.36 us?????? 7.76 us? 962634.73 us??????? >>>> 3660425???? ENTRYLK >>>> ????? 6.23???? 304.88 us????? 42.51 us? 661394.43 us???????? >>>> 990644???? SETATTR >>>> ????? 7.81???? 382.20 us???? 107.42 us? 846502.78 us???????? >>>> 990766???? XATTROP >>>> ???? 12.39???? 549.25 us???? 135.57 us 1659697.83 us??????? >>>> 1093387?????? MKNOD >>>> ???? 12.47???? 139.78 us?????? 9.76 us? 552067.56 us??????? >>>> 4325281???? INODELK >>>> ???? 39.53???? 206.95 us?????? 8.56 us? 964823.33 us??????? >>>> 9259429????? LOOKUP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1856377????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 933300???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? >>>> 251??? CI_XATTR >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? >>>> 293?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 923077?? CI_FORGET >>>> ? >>>> ??? Duration: 209334 seconds >>>> ?? Data Read: 0 bytes >>>> Data Written: 32843 bytes >>>> ? >>>> Brick: bender:/mnt/gluster/c >>>> --------------------------------------- >>>> Cumulative Stats: >>>> ?? Block Size:????????????????? 1b+ >>>> ?No. of Reads:??????????????????? 0 >>>> No. of Writes:??????????????? 37050 >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ?????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 928119????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1401524???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 70092? RELEASEDIR >>>> ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us??????????? >>>> 100???????? IPC >>>> ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 >>>> us???????????? 38??????? LINK >>>> ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 >>>> us???????????? 30???? SYMLINK >>>> ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 >>>> us????????????? 4?????? FSYNC >>>> ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us??????????? >>>> 122????? RENAME >>>> ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us??????????? >>>> 777????????? LK >>>> ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us??????????? >>>> 810?? FGETXATTR >>>> ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us??????????? >>>> 603??? TRUNCATE >>>> ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us??????????? >>>> 302?????? MKDIR >>>> ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us????????? >>>> 10060??? FINODELK >>>> ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us?????????? >>>> 6409?????? FSTAT >>>> ????? 0.01????? 49.38 us?????? 8.08 us??? 1264.41 us????????? >>>> 21719?????? FLUSH >>>> ????? 0.03????? 74.76 us????? 20.86 us??? 2008.41 us????????? >>>> 37050?????? WRITE >>>> ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us?????????? >>>> 5901????? CREATE >>>> ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us????????? >>>> 10057??? FXATTROP >>>> ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us????????? >>>> 70092???? OPENDIR >>>> ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us????????? >>>> 10779???? READDIR >>>> ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us?????????? >>>> 3008?????? RMDIR >>>> ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us???????? >>>> 109478????? UNLINK >>>> ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us???????? >>>> 722669 REMOVEXATTR >>>> ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us???????? >>>> 789294??? GETXATTR >>>> ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us???????? >>>> 718835?? FTRUNCATE >>>> ????? 1.89???? 219.47 us????? 49.92 us? 506767.78 us???????? >>>> 722856??? SETXATTR >>>> ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us??????? >>>> 1395957??????? OPEN >>>> ????? 5.56???? 322.08 us????? 43.37 us? 634069.61 us??????? >>>> 1450160???? SETATTR >>>> ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us??????? >>>> 4233520???? ENTRYLK >>>> ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us??????? >>>> 1449601???? XATTROP >>>> ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us??????? >>>> 7097226???? INODELK >>>> ???? 19.37??? 1104.98 us???? 101.28 us 1612346.75 us??????? >>>> 1472813?????? MKNOD >>>> ???? 41.85???? 295.61 us?????? 8.98 us? 996805.54 us?????? >>>> 11896091????? LOOKUP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 3531139????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1862470???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? >>>> 244??? CI_XATTR >>>> ?????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>> us??????????? 632?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1668669?? CI_FORGET >>>> ? >>>> ??? Duration: 208643 seconds >>>> ?? Data Read: 0 bytes >>>> Data Written: 37050 bytes >>>> ? >>>> Interval 0 Stats: >>>> ?? Block Size:????????????????? 1b+ >>>> ?No. of Reads:??????????????????? 0 >>>> No. of Writes:??????????????? 37050 >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 928119????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1401524???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 70092? RELEASEDIR >>>> ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 us??????????? >>>> 100???????? IPC >>>> ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 >>>> us???????????? 38??????? LINK >>>> ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 >>>> us???????????? 30???? SYMLINK >>>> ????? 0.00??? 2670.18 us???? 765.93 us??? 4151.64 >>>> us????????????? 4?????? FSYNC >>>> ????? 0.00???? 350.65 us???? 141.92 us?? 10573.64 us??????????? >>>> 122????? RENAME >>>> ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 us??????????? >>>> 777????????? LK >>>> ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 us??????????? >>>> 810?? FGETXATTR >>>> ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 us??????????? >>>> 603??? TRUNCATE >>>> ????? 0.01??? 1742.85 us???? 202.81 us?? 47471.71 us??????????? >>>> 302?????? MKDIR >>>> ????? 0.01????? 59.05 us????? 15.40 us??? 1813.56 us????????? >>>> 10060??? FINODELK >>>> ????? 0.01???? 118.54 us????? 36.65 us??? 4002.58 us?????????? >>>> 6409?????? FSTAT >>>> ????? 0.01????? 49.39 us?????? 8.08 us??? 1264.41 us????????? >>>> 21719?????? FLUSH >>>> ????? 0.03????? 74.77 us????? 20.86 us??? 2008.41 us????????? >>>> 37050?????? WRITE >>>> ????? 0.04???? 553.12 us????? 40.81 us? 386485.29 us?????????? >>>> 5901????? CREATE >>>> ????? 0.06???? 534.01 us???? 100.86 us? 527770.44 us????????? >>>> 10057??? FXATTROP >>>> ????? 0.12???? 138.96 us?????? 1.00 us? 601208.70 us????????? >>>> 70092???? OPENDIR >>>> ????? 0.12???? 947.76 us????? 18.54 us? 712568.33 us????????? >>>> 10779???? READDIR >>>> ????? 0.15??? 4099.07 us???? 124.69 us? 421939.47 us?????????? >>>> 3008?????? RMDIR >>>> ????? 0.61???? 465.30 us????? 20.67 us? 936959.86 us???????? >>>> 109478????? UNLINK >>>> ????? 1.57???? 182.41 us????? 49.54 us?? 73938.09 us???????? >>>> 722669 REMOVEXATTR >>>> ????? 1.61???? 171.41 us?????? 6.98 us?? 83809.88 us???????? >>>> 789294??? GETXATTR >>>> ????? 1.86???? 217.88 us????? 26.40 us? 556446.19 us???????? >>>> 718835?? FTRUNCATE >>>> ????? 1.89???? 219.48 us????? 49.92 us? 506767.78 us???????? >>>> 722856??? SETXATTR >>>> ????? 3.02???? 181.66 us????? 44.40 us? 344656.69 us??????? >>>> 1395957??????? OPEN >>>> ????? 5.56???? 322.09 us????? 43.37 us? 634069.61 us??????? >>>> 1450160???? SETATTR >>>> ????? 5.84???? 115.88 us?????? 7.67 us? 747387.51 us??????? >>>> 4233520???? ENTRYLK >>>> ????? 6.53???? 378.33 us???? 122.84 us? 766939.14 us??????? >>>> 1449601???? XATTROP >>>> ????? 9.75???? 115.48 us????? 10.70 us? 556675.13 us??????? >>>> 7097226???? INODELK >>>> ???? 19.37??? 1105.06 us???? 101.28 us 1612346.75 us??????? >>>> 1472813?????? MKNOD >>>> ???? 41.84???? 295.54 us?????? 8.98 us? 996805.54 us?????? >>>> 11896091????? LOOKUP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 3531139????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1862470???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? >>>> 244??? CI_XATTR >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????????? >>>> 632?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1668669?? CI_FORGET >>>> ? >>>> ??? Duration: 208643 seconds >>>> ?? Data Read: 0 bytes >>>> Data Written: 37050 bytes >>>> ? >>>> Brick: bender:/mnt/gluster/a >>>> --------------------------------------- >>>> Cumulative Stats: >>>> ?? Block Size:????????????????? 1b+?????????????????? >>>> 2b+?????????????????? 8b+ >>>> ?No. of Reads:??????????????????? 0???????????????????? >>>> 0???????????????????? 0 >>>> No. of Writes:?????????????????? 53???????????????????? >>>> 1??????????????????? 60 >>>> ? >>>> ?? Block Size:???????????????? 16b+????????????????? >>>> 32b+????????????????? 64b+ >>>> ?No. of Reads:??????????????????? 0?????????????????? >>>> 235???????????????????? 4 >>>> No. of Writes:?????????????????? 14????????????????? >>>> 1033??????????????????? 15 >>>> ? >>>> ?? Block Size:??????????????? 128b+???????????????? >>>> 256b+???????????????? 512b+ >>>> ?No. of Reads:??????????????????? 4?????????????????? >>>> 115??????????????????? 81 >>>> No. of Writes:?????????????????? 58?????????????????? >>>> 703??????????????????? 80 >>>> ? >>>> ?? Block Size:?????????????? 1024b+??????????????? >>>> 2048b+??????????????? 4096b+ >>>> ?No. of Reads:????????????????? 181??????????????????? >>>> 26??????????????????? 74 >>>> No. of Writes:????????????????? 163?????????????????? >>>> 278?????????????????? 397 >>>> ? >>>> ?? Block Size:?????????????? 8192b+?????????????? >>>> 16384b+?????????????? 32768b+ >>>> ?No. of Reads:????????????????? 131?????????????????? >>>> 161?????????????????? 267 >>>> No. of Writes:???????????????? 7264????????????????? >>>> 1398????????????????? 6742 >>>> ? >>>> ?? Block Size:????????????? 65536b+????????????? >>>> 131072b+????????????? 262144b+ >>>> ?No. of Reads:????????????????? 443???????????????? >>>> 25528???????????????????? 0 >>>> No. of Writes:???????????????? 2426???????????????? >>>> 12240???????????????????? 1 >>>> ? >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 684942????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 837092???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 87297? RELEASEDIR >>>> ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 >>>> us????????????? 8 REMOVEXATTR >>>> ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 >>>> us???????????? 97???????? IPC >>>> ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 >>>> us???????????? 46?? FTRUNCATE >>>> ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 >>>> us???????????? 65?? FGETXATTR >>>> ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us??????????? >>>> 121??????? LINK >>>> ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 >>>> us???????????? 25???? SYMLINK >>>> ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us??????????? >>>> 107?????? MKNOD >>>> ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 >>>> us????????????? 1?????? FSYNC >>>> ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us??????????? >>>> 188????? RENAME >>>> ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us??????????? >>>> 745??????? STAT >>>> ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us??????????? >>>> 200??? TRUNCATE >>>> ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us?????????? >>>> 1444???? SETATTR >>>> ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us??????????? >>>> 177?????? MKDIR >>>> ????? 0.00????? 52.68 us?????? 7.28 us???? 826.01 us?????????? >>>> 9890?????? FLUSH >>>> ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us?????????? >>>> 9792??? FINODELK >>>> ????? 0.00????? 74.26 us????? 20.00 us??? 1522.39 us????????? >>>> 21521????? STATFS >>>> ????? 0.00???? 129.21 us????? 34.15 us??? 9068.67 us????????? >>>> 13094?????? FSTAT >>>> ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us?????????? >>>> 5967????? CREATE >>>> ????? 0.00???? 112.06 us????? 28.14 us??? 9545.53 us????????? >>>> 27474??????? READ >>>> ????? 0.01???? 313.76 us????? 52.98 us?? 71635.74 us????????? >>>> 32926?????? WRITE >>>> ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us????????? >>>> 87297???? OPENDIR >>>> ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us?????????? >>>> 3177?????? RMDIR >>>> ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us????????? >>>> 85073??? SETXATTR >>>> ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us????????? >>>> 15090??? READDIRP >>>> ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us??????????? >>>> 564????????? LK >>>> ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us???????? >>>> 850938??? READLINK >>>> ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us???????? >>>> 831212??????? OPEN >>>> ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us????????? >>>> 22482???? READDIR >>>> ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us???????? >>>> 415226????? UNLINK >>>> ????? 0.54??? 1272.83 us?????? 9.66 us 1604368.93 us???????? >>>> 686945??? GETXATTR >>>> ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us??????? >>>> 3739718???? ENTRYLK >>>> ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us?????????? >>>> 9311??? FXATTROP >>>> ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us??????? >>>> 4649291???? INODELK >>>> ????? 5.82??? 1240.20 us?????? 8.78 us 1526868.50 us??????? >>>> 7625383????? LOOKUP >>>> ???? 88.60?? 58005.98 us????? 67.31 us 1448810.06 us??????? >>>> 2481914???? XATTROP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1396875????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 1854???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>> us???????????? 28??? CI_XATTR >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 1051?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1394996?? CI_FORGET >>>> ? >>>> ??? Duration: 208643 seconds >>>> ?? Data Read: 3409342848 bytes >>>> Data Written: 2180686507 bytes >>>> ? >>>> Interval 0 Stats: >>>> ?? Block Size:????????????????? 1b+?????????????????? >>>> 2b+?????????????????? 8b+ >>>> ?No. of Reads:??????????????????? 0???????????????????? >>>> 0???????????????????? 0 >>>> No. of Writes:?????????????????? 53???????????????????? >>>> 1??????????????????? 60 >>>> ? >>>> ?? Block Size:???????????????? 16b+????????????????? >>>> 32b+????????????????? 64b+ >>>> ?No. of Reads:??????????????????? 0?????????????????? >>>> 235???????????????????? 4 >>>> No. of Writes:?????????????????? 14????????????????? >>>> 1033??????????????????? 15 >>>> ? >>>> ?? Block Size:??????????????? 128b+???????????????? >>>> 256b+???????????????? 512b+ >>>> ?No. of Reads:??????????????????? 4?????????????????? >>>> 115??????????????????? 81 >>>> No. of Writes:?????????????????? 58?????????????????? >>>> 703??????????????????? 80 >>>> ? >>>> ?? Block Size:?????????????? 1024b+??????????????? >>>> 2048b+??????????????? 4096b+ >>>> ?No. of Reads:????????????????? 181??????????????????? >>>> 26??????????????????? 74 >>>> No. of Writes:????????????????? 163?????????????????? >>>> 278?????????????????? 397 >>>> ? >>>> ?? Block Size:?????????????? 8192b+?????????????? >>>> 16384b+?????????????? 32768b+ >>>> ?No. of Reads:????????????????? 131?????????????????? >>>> 161?????????????????? 267 >>>> No. of Writes:???????????????? 7264????????????????? >>>> 1398????????????????? 6742 >>>> ? >>>> ?? Block Size:????????????? 65536b+????????????? >>>> 131072b+????????????? 262144b+ >>>> ?No. of Reads:????????????????? 443???????????????? >>>> 25528???????????????????? 0 >>>> No. of Writes:???????????????? 2426???????????????? >>>> 12240???????????????????? 1 >>>> ? >>>> ?%-latency?? Avg-latency?? Min-Latency?? Max-Latency?? No. of >>>> calls???????? Fop >>>> ?---------?? -----------?? -----------?? -----------?? >>>> ------------??????? ---- >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 684942????? FORGET >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>> 837092???? RELEASE >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>> 87297? RELEASEDIR >>>> ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 >>>> us????????????? 8 REMOVEXATTR >>>> ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 >>>> us???????????? 97???????? IPC >>>> ????? 0.00???? 152.77 us????? 78.90 us??? 1059.82 >>>> us???????????? 46?? FTRUNCATE >>>> ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 >>>> us???????????? 65?? FGETXATTR >>>> ????? 0.00???? 243.66 us???? 140.50 us??? 3392.16 us??????????? >>>> 121??????? LINK >>>> ????? 0.00??? 1216.47 us???? 194.37 us?? 23411.42 >>>> us???????????? 25???? SYMLINK >>>> ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 us??????????? >>>> 107?????? MKNOD >>>> ????? 0.00?? 36094.61 us?? 36094.61 us?? 36094.61 >>>> us????????????? 1?????? FSYNC >>>> ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 us??????????? >>>> 188????? RENAME >>>> ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 us??????????? >>>> 745??????? STAT >>>> ????? 0.00???? 531.71 us???? 113.57 us?? 53973.08 us??????????? >>>> 200??? TRUNCATE >>>> ????? 0.00???? 289.05 us????? 63.10 us?? 70107.41 us?????????? >>>> 1444???? SETATTR >>>> ????? 0.00??? 2430.15 us???? 232.78 us?? 68120.54 us??????????? >>>> 177?????? MKDIR >>>> ????? 0.00????? 52.69 us?????? 7.28 us???? 826.01 us?????????? >>>> 9890?????? FLUSH >>>> ????? 0.00????? 60.96 us????? 17.54 us??? 1393.95 us?????????? >>>> 9792??? FINODELK >>>> ????? 0.00????? 74.25 us????? 20.00 us??? 1522.39 us????????? >>>> 21521????? STATFS >>>> ????? 0.00???? 129.23 us????? 34.15 us??? 9068.67 us????????? >>>> 13094?????? FSTAT >>>> ????? 0.00???? 456.99 us???? 195.50 us? 130182.02 us?????????? >>>> 5967????? CREATE >>>> ????? 0.00???? 112.45 us????? 28.14 us??? 9545.53 us????????? >>>> 27474??????? READ >>>> ????? 0.01???? 313.77 us????? 52.98 us?? 71635.74 us????????? >>>> 32926?????? WRITE >>>> ????? 0.01???? 135.84 us?????? 1.52 us? 671373.08 us????????? >>>> 87297???? OPENDIR >>>> ????? 0.01??? 5225.28 us???? 131.28 us? 951323.69 us?????????? >>>> 3177?????? RMDIR >>>> ????? 0.01???? 203.34 us????? 56.13 us? 486756.90 us????????? >>>> 85073??? SETXATTR >>>> ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us????????? >>>> 15090??? READDIRP >>>> ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 us??????????? >>>> 564????????? LK >>>> ????? 0.05????? 92.18 us????? 29.71 us?? 39069.99 us???????? >>>> 850938??? READLINK >>>> ????? 0.23???? 453.79 us????? 41.84 us? 655999.23 us???????? >>>> 831212??????? OPEN >>>> ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us????????? >>>> 22482???? READDIR >>>> ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us???????? >>>> 415226????? UNLINK >>>> ????? 0.54??? 1272.79 us?????? 9.66 us 1604368.93 us???????? >>>> 686945??? GETXATTR >>>> ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us??????? >>>> 3739718???? ENTRYLK >>>> ????? 0.60? 103888.49 us????? 89.32 us? 761509.35 us?????????? >>>> 9311??? FXATTROP >>>> ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us??????? >>>> 4649291???? INODELK >>>> ????? 5.82??? 1240.10 us?????? 8.78 us 1526868.50 us??????? >>>> 7625383????? LOOKUP >>>> ???? 88.60?? 58006.02 us????? 67.31 us 1448810.06 us??????? >>>> 2481914???? XATTROP >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1396875????? UPCALL >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 1854???? CI_IATT >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>> us???????????? 28??? CI_XATTR >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>> 1051?? CI_UNLINK >>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>> 1394996?? CI_FORGET >>>> ? >>>> ??? Duration: 208643 seconds >>>> ?? Data Read: 3409342848 bytes >>>> Data Written: 2180686507 bytes >>>> ? >>>> >>>> ________ >>>> >>>> >>>> >>>> Community Meeting Calendar: >>>> >>>> Schedule - >>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>>> Bridge: https://bluejeans.com/441850968 >>>> >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >> >> ________ >> >> >> >> Community Meeting Calendar: >> >> Schedule - >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >> Bridge: https://bluejeans.com/441850968 >> >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200508/79422395/attachment.html>
Hi, On 08/05/20 1:03 pm, Jaco Kroon wrote:> > Hi, > > On 2020/05/08 06:32, Ravishankar N wrote: >> >> On 08/05/20 1:10 am, Jaco Kroon wrote: >>> >>> Hi, >>> >>> So upgrading to 7.5 made a difference in that it takes a bit longer >>> to cause chaos, but it still does the exact same thing. >>> >>> To be precise: >>> >>> I've now had to set: >>> >>> gluster volume set vpbx_shared cluster.heal-wait-queue-length 1 >>> gluster volume set vpbx_shared cluster.shd-wait-qlength 1 >>> gluster volume set vpbx_shared cluster.shd-max-threads 1 >>> >>> With any of these settings >1 I end up in a situation where simple >>> directory listings of < 100 entries in a folder takes upwards of a >>> minute to complete *from time to time*. >>> >> Hi Jaco, these options are to be increased only if your hardware is >> capable of handling the load.It was contributed initially by facebook >> as it worked for them. See >> https://bugzilla.redhat.com/show_bug.cgi?id=1221737#c0. So I think >> this is expected. Unless you are saying that the `ls` slowness >> happens even when there is no self-heal going on, in which case we >> need to see what the issue is. >> > The defaults for the queue lengths are >1 (1024) if I recall. > > The default for max threads are 1. > > Even with all those at a value of 1 we reached an ls latency of 10s > overnight, and I specicifically switched off backups before letting > this run overnight to ensure that it would not interfere. > > Can the hardware handle it?? Well, let's look at the capacity status.? > During threads above set to 16, and queue lengths at 10240.? All three > hosts have approximately equal load values, with zoidberg (two data > bricks compared to the other hosts one data brick and one "healing" > arbiter that's newly added) being marginally (~2% on IO utilization) > higher loaded, CPU was very similar.? I suspect this is because it's > supplying two source bricks compared to the other hosts each have one > source and one sink. > > During these stalls the IO wait is <10% on all hosts and according to > iotop that's almost all glusterfsd.? This goes up slightly if there > are LVM snapshots in place for backup purposes, which takes about an hour. > > CPU on glusterfsd < 50% according to top (ie, half a core's > capacity).? 16 cores available. > > 64GB RAM total, and as of writing, free -m is reporting: > > ????????????? total??????? used??????? free????? shared buff/cache?? > available > Mem:????????? 64413??????? 6600?????? 29011 370?????? 28801?????? 56517 > > RAID controller on each host: > > ??????????????? description: RAID bus controller > ??????????????? product: MegaRAID SAS 2108 [Liberator] > ??????????????? vendor: Broadcom / LSI > > ?????????????????? description: SCSI Disk > ?????????????????? product: PERC H700 > ?????????????????? vendor: DELL > > Looking at the queue depth for the exposed drive: > > flexo /sys/class/block/sda # cat device/queue_depth > 256 > > The disks themselves are in a RAID-1 configuration, 4TB NL-SAS drives > from Seagate. > > On top of the RAID-exposed drive we run LVM and the gluster data is > sitting in an ext4 filesystem on top of a 3TB LV inside of the single > PV that makes up the VG > > I don't feel that there should be a problem with threads set to 4 at > least, but probably even 8 or 16 should be OK.? I mean, until the IO > capacity doesn't get to at least 70-80% it really should make little > difference in my experience.? Technically even at 100% utilization it > could just be a single IO request at any point in time (as we've seen > cloning large disks) so in theory you can then still increase > concurrency, but from experience we know that overall performance > starts to degrade quite sharply at a certain tipping point and then > you're in trouble with little to no warning. > > Under non-healing conditions IO utilization (iostat -dmx 1, iotop etc > ... ie, percentage of time that has outstanding requests ignoring > multi-concurrency) is generally about 0.3%. > > So somewhere there is definitely a bottleneck.? I don't mind deploying > a custom "debug" patch on top of gluserfs 7.5 for the weekend but I > generally need optimal performance again by Monday morning 6:30 SAST > (GMT+2), then pushing the heal as hard as I possibly can overnight to > get lock statistics or something.? If I can just know what options > ./configure should be on/off, and which volume options needs to be set > to get the right statistics, that's also helpful.? I'm even happy to > then be pointed at the code and told "it's this process where the > problem's at" (bottleneck, as it doesn't look like anything is broken > really, just terribly unacceptably slow) - in other words, I'm happy > to go attempt and contribute a patch on the issue, but I need a > starting point of where to start scratching.? Can even do this > Saturday after 15:00 till Sunday morning again. >I don't have a patch to offer but the entry point for the self-heal-daemon code itself is in syncop_mt_dir_scan()? which gets called for each of the 3 sub folders under /brick/.glusterfs/indices/ (for each brick). It uses gluster's synctask framework to readdir the entries in these sub folders, and eventually calls afr_shd_index_heal() for each of the entries. So if you think shd is stuck, maybe you can see at what point it is stuck - perhaps while afr_shd_selfheal() for a given entry is stuck on an inodelk held by the client. Maybe look at the pstack of the shd process. I also see from the profile info on your first email that XATTROP fop has the highest latency. Can you do an strace of the brick processes to see if there is any syscall (setxattr in particular, for the XATTROP fop) that is taking higher than usual times? For the slow ls from the client, an ongoing selfheal should not affect it, since readdir, stat etc are all read-only operations that do not take any locks that can compete with heal, and there is no hardware bottleneck at the brick side to process requests from what you say. Maybe a tcp dump between the client and the server can help find if the slowness is from the server side by looking at the request and response times of the FOPS in the dump. Since you're saying hardware is not saturated despite increasing the shd threads and wait-queues, you could try increasing performance.io-thread-count on the brick side from the default 16. Regards, Ravi> > Kind Regards, > Jaco > > >> Regards, >> >> Ravi >> >>> It seems that when the logs >>> >>> With these the maximum stall I can reproduce is usually around 3 >>> seconds before the directory listing will complete, but about 5s >>> worst case.? I've now reverted these as well, which are known for us >>> to cause issues under heal required conditions: >>> >>> cluster.metadata-self-heal on >>> cluster.data-self-heal on >>> cluster.entry-self-heal on >>> >>> Not sure if I need to remount for these to become effective but I've >>> done so just to be on the safe side. >>> >>> I'm still seeing these errors: >>> >>> [2020-05-07 19:20:12.763311] W [MSGID: 114031] >>> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] >>> 0-vpbx_shared-client-1: remote operation failed [Invalid argument] >>> [2020-05-07 19:20:12.995675] W [MSGID: 114031] >>> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] >>> 0-vpbx_shared-client-1: remote operation failed [Invalid argument] >>> [2020-05-07 19:20:59.199776] W [MSGID: 114031] >>> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] >>> 0-vpbx_shared-client-1: remote operation failed [Invalid argument] >>> >>> And the start of my issues still seem to co-incide with that.? >>> Sample case: >>> >>> # time ls >>> ... >>> real??? 0m5.140s >>> user??? 0m0.009s >>> sys??? 0m0.000s >>> # f=(*); echo "${#f[@]}" >>> 165 >>> >>> This just does not seem right.? Note, again, if I set *any* of the >>> above parameters >1 these delays get far, far worse (I've had over a >>> minute for this exact same ls). >>> >>> Any advise would be greatly appreciated.? To me, it looks like some >>> lock is being taken somewhere, and takes a while to release.? I >>> don't understand the glusterfs internals nearly well enough to >>> understand exactly what's going on here, nor do I know what >>> information to provide to help drill down on this.? Happy to go look >>> at code, if I can get pointed in a direction. >>> >>> Kind Regards, >>> Jaco >>> >>> On 2020/05/05 08:43, Jaco Kroon wrote: >>>> >>>> Hi Artem, >>>> >>>> I'll do that.? In the meantime consider this (inferred from the >>>> changelogs). >>>> >>>> It seems like glusterfs SHD is queueing into a bounded queue under >>>> lock conditions, and waiting for space to open up.? At least, this >>>> is what's hinted in the changelogs, I've been trying to grasp the >>>> SHD code for two days now and am not getting to wrap my head around >>>> that.? If this is true, then the only way to avoid lock-ups are to >>>> ensure that the queue is larger than the total number of files that >>>> can ever require healing (effectively making the queue unbounded). >>>> >>>> Kind Regards, >>>> Jaco >>>> >>>> On 2020/05/04 21:56, Artem Russakovskii wrote: >>>>> I reported something similar recently to this list, and I highly >>>>> advise you to update to 7.5 (in my case it was 5.11 -> 5.13), then >>>>> see if it keeps happening. The self heal issues should be fixed >>>>> there properly at least. >>>>> >>>>> Sincerely, >>>>> Artem >>>>> >>>>> -- >>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror >>>>> <http://www.apkmirror.com/>, Illogical Robot LLC >>>>> beerpla.net <http://beerpla.net/> | @ArtemR >>>>> <http://twitter.com/ArtemR> >>>>> >>>>> >>>>> On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za >>>>> <mailto:jaco at uls.co.za>> wrote: >>>>> >>>>> Hi All, >>>>> >>>>> So we've got a cluster currently running on glusterfs 7.3 >>>>> >>>>> Heal performance is absolutely terrible (it starts of great, >>>>> runs for a while then completely trips up). >>>>> >>>>> Once the SHD trips up, access speeds to the cluster also >>>>> becomes poor.? And with poor I mean directory listings that >>>>> normally takes well under 500ms takes upwards of 10s, but I've >>>>> seen upwards of 40s (although a follow up ls immediately after >>>>> is nice and fast again, temporarily).? With all SHDs killed >>>>> this particular ls is as per normal (around 30-50ms). >>>>> >>>>> The SHD logs doesn't provide any usable clues, the only >>>>> variance in logs is this, just before all daemons across the >>>>> cluster crawls to a near halt. >>>>> >>>>> [2020-05-02 19:55:58.627036] I [MSGID: 108026] >>>>> [afr-self-heal-common.c:1747:afr_log_selfheal] >>>>> 0-vpbx_shared-replicate-0: Completed entry selfheal on >>>>> 5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1 >>>>> [2020-05-02 19:55:58.627058] I [MSGID: 108026] >>>>> [afr-self-heal-common.c:1747:afr_log_selfheal] >>>>> 0-vpbx_shared-replicate-0: Completed entry selfheal on >>>>> c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1 >>>>> [2020-05-02 19:55:59.125001] W [MSGID: 114031] >>>>> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] >>>>> 0-vpbx_shared-client-1: remote operation failed [Invalid argument] >>>>> [2020-05-02 19:55:59.339803] I [MSGID: 108026] >>>>> [afr-self-heal-common.c:1747:afr_log_selfheal] >>>>> 0-vpbx_shared-replicate-0: Completed entry selfheal on >>>>> 2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1 >>>>> [2020-05-02 19:55:59.490387] I [MSGID: 108026] >>>>> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] >>>>> 0-vpbx_shared-replicate-0: performing metadata selfheal on >>>>> 527a63a3-9acd-4c3f-b95c-5c719325c973 >>>>> >>>>> A few more entries will complete overall and then it'll be a >>>>> file every other second.? Once I saw this I ran a few more >>>>> tests, and on every test just prior to slowing to a crawl, >>>>> that entry is there. >>>>> >>>>> There is a perceived improvement when lowering the thread >>>>> count, but not significant enough to confirm a correlation.? >>>>> Even with**cluster.shd-max-threads=1 this still happens (gets >>>>> a few tens thousand files completed instead of few hundred, at >>>>> 32 it gets to maybe 400 to 500 before it stalls, 16 does maybe >>>>> a thousand).* >>>>> * >>>>> >>>>> There are probably performant bursts hereafter, because from >>>>> experience we know this process takes about two months.? But >>>>> this time around it seems to be much, much worse.? Might be >>>>> because that was healing a single brick to another brick, so >>>>> less locking going?? At the moment the replicas are basically >>>>> trying to populate the newly added arbiters.? Also, we haven't >>>>> messed as greatly with the health check settings (queues and >>>>> threads and locks nearly as much as since the arbiter addition >>>>> because performance has simply dropped to non-acceptable levels. >>>>> >>>>> Unfortunately granular entry heal isn't on at the moment (nor >>>>> can I switch it on due to a few million outstanding heals).? >>>>> This used to be a 2x2 cluster.? We then moved one of the >>>>> bricks to a newly added server using replace-brick.? What a >>>>> frightening process this is.? We then proceeded to add the >>>>> arbiters. >>>>> >>>>> Some of the settings below was absolute shots in the dark, >>>>> based on the little information we've been able to track down >>>>> so far. >>>>> >>>>> Kind Regards, >>>>> Jaco >>>>> >>>>> The current gluster volume info (this used to be 2x2 >>>>> distributed-replicate, aiming to get it to 3x (2 + 1) ... at >>>>> current rates we might get there by 2021): >>>>> >>>>> Volume Name: glvol >>>>> Type: Distributed-Replicate >>>>> Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb >>>>> Status: Started >>>>> Snapshot Count: 0 >>>>> Number of Bricks: 2 x (2 + 1) = 6 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: zoidberg:/mnt/gluster/a >>>>> Brick2: flexo:/mnt/gluster/b >>>>> Brick3: bender:/mnt/gluster/c (arbiter) >>>>> Brick4: bender:/mnt/gluster/a >>>>> Brick5: zoidberg:/mnt/gluster/b >>>>> Brick6: flexo:/mnt/gluster/c (arbiter) >>>>> Options Reconfigured: >>>>> *cluster.heal-timeout: 5 <-- hoping this would at least >>>>> shorten the blockages.* >>>>> cluster.locking-scheme: granular >>>>> cluster.heal-wait-queue-length: 10240 >>>>> *diagnostics.count-fop-hits: on** >>>>> **diagnostics.latency-measurement: on <-- was hoping profiling >>>>> might provide some insight.? Going to switch this off now.* >>>>> cluster.background-self-heal-count: 128 >>>>> *cluster.shd-wait-qlength: 131072** >>>>> **cluster.shd-max-threads: 32? <-- even with this disk IO >>>>> never goes over 10% utilization.? CPU goes to about 2 cores.? >>>>> So it's lock contention that's slowing it down we reckon.* >>>>> *cluster.metadata-self-heal: off** >>>>> **cluster.entry-self-heal: off** >>>>> **cluster.data-self-heal: off**<-- these kill performance. >>>>> *transport.address-family: inet >>>>> nfs.disable: on >>>>> performance.client-io-threads: on >>>>> features.cache-invalidation: on >>>>> features.cache-invalidation-timeout: 600 >>>>> performance.stat-prefetch: on >>>>> performance.cache-invalidation: on >>>>> performance.md-cache-timeout: 600 >>>>> *cluster.self-heal-daemon: yes <-- disabling/enabling this >>>>> doesn't help, have to kill the shd and start ... force to get >>>>> things moving along again.* >>>>> performance.io-thread-count: 16 >>>>> performance.high-prio-threads: 16 >>>>> performance.normal-prio-threads: 16 >>>>> performance.low-prio-threads: 16 >>>>> performance.least-prio-threads: 4 >>>>> cluster.self-heal-window-size: 512 >>>>> cluster.self-heal-readdir-size: 131072 >>>>> client.event-threads: 8 >>>>> server.event-threads: 16 >>>>> network.ping-timeout: 2 >>>>> *cluster.data-self-heal-algorithm: full <-- most files are >>>>> small (<1MB), on dual 1gbit links (so up to 2gbit, but b/w >>>>> never goes over 5mbit anyway).* >>>>> >>>>> performance info: >>>>> >>>>> Brick: zoidberg:/mnt/gluster/a >>>>> ----------------------------------------- >>>>> Cumulative Stats: >>>>> ?? Block Size: 1b+?????????????????? 8b+????????????????? 16b+ >>>>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>>>> No. of Writes: 117?????????????????? 195?????????????????? 367 >>>>> >>>>> ?? Block Size: 32b+????????????????? 64b+???????????????? 128b+ >>>>> ?No. of Reads: 306???????????????????? 0???????????????????? 6 >>>>> No. of Writes: 1016???????????????????? 9?????????????????? 134 >>>>> >>>>> ?? Block Size: 256b+???????????????? 512b+ 1024b+ >>>>> ?No. of Reads: 107??????????????????? 58?????????????????? 485 >>>>> No. of Writes: 731??????????????????? 73?????????????????? 168 >>>>> >>>>> ?? Block Size: 2048b+??????????????? 4096b+ 8192b+ >>>>> ?No. of Reads: 43??????????????????? 59?????????????????? 383 >>>>> No. of Writes: 295?????????????????? 475???????????????? 10562 >>>>> >>>>> ?? Block Size: 16384b+?????????????? 32768b+ 65536b+ >>>>> ?No. of Reads: 347?????????????????? 380?????????????????? 634 >>>>> No. of Writes: 1487????????????????? 6369????????????????? 3970 >>>>> >>>>> ?? Block Size:???????????? 131072b+ >>>>> ?No. of Reads:??????????????? 35704 >>>>> No. of Writes:??????????????? 12398 >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 957293????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1455091???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 69951? RELEASEDIR >>>>> ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 >>>>> us??????????? 113???????? IPC >>>>> ????? 0.00???? 687.33 us????? 80.53 us 15414.03 us???????????? >>>>> 30 REMOVEXATTR >>>>> ????? 0.00???? 840.76 us????? 72.71 us 37648.39 us???????????? >>>>> 52?? FTRUNCATE >>>>> ????? 0.00???? 138.49 us????? 36.66 us 25280.24 us?????????? >>>>> 1385??????? STAT >>>>> ????? 0.00???? 364.93 us????? 63.45 us 15957.66 us??????????? >>>>> 810?? FGETXATTR >>>>> ????? 0.00??? 2208.82 us???? 127.23 us 267092.17 us??????????? >>>>> 137??????? LINK >>>>> ????? 0.00?? 37595.33 us??? 1677.46 us 218313.32 >>>>> us???????????? 12?????? FSYNC >>>>> ????? 0.00???? 675.61 us???? 106.82 us 114351.01 us??????????? >>>>> 698??? TRUNCATE >>>>> ????? 0.00??? 3995.66 us???? 147.00 us 109544.46 us??????????? >>>>> 188????? RENAME >>>>> ????? 0.00???? 453.11 us????? 53.37 us 104848.94 us?????????? >>>>> 1680???? SETATTR >>>>> ????? 0.00??? 6004.32 us???? 229.33 us 151322.58 us??????????? >>>>> 127?????? MKNOD >>>>> ????? 0.00????? 68.74 us????? 16.11 us 61050.93 us????????? >>>>> 11412??? FINODELK >>>>> ????? 0.00??? 3793.44 us????? 68.61 us 411523.70 us??????????? >>>>> 221??? SETXATTR >>>>> ????? 0.00?? 28669.99 us???? 227.28 us 138988.41 >>>>> us???????????? 30???? SYMLINK >>>>> ????? 0.00????? 56.18 us?????? 7.31 us 41484.37 us????????? >>>>> 22640?????? FLUSH >>>>> ????? 0.00???? 105.64 us????? 19.34 us 351055.72 us????????? >>>>> 21693????? STATFS >>>>> ????? 0.00?? 23642.90 us???? 244.19 us 234912.25 us??????????? >>>>> 178?????? MKDIR >>>>> ????? 0.00???? 537.38 us????? 31.73 us 296911.67 us?????????? >>>>> 9653?????? FSTAT >>>>> ????? 0.00???? 214.87 us????? 25.71 us 1712020.99 us????????? >>>>> 39054??????? READ >>>>> ????? 0.01???? 147.08 us?????? 1.47 us 241611.80 us????????? >>>>> 69951???? OPENDIR >>>>> ????? 0.01??? 3899.84 us???? 117.22 us 503422.21 us?????????? >>>>> 3180?????? RMDIR >>>>> ????? 0.01???? 327.08 us????? 55.69 us 420376.80 us????????? >>>>> 38366?????? WRITE >>>>> ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us?????????? >>>>> 6169????? CREATE >>>>> ????? 0.05????? 92.69 us????? 29.94 us 687955.86 us??????? >>>>> 1159936??? READLINK >>>>> ????? 0.40???? 543.50 us????? 36.45 us 2430653.54 us??????? >>>>> 1449307??????? OPEN >>>>> ????? 0.48?? 87643.67 us????? 70.20 us 661999.51 us????????? >>>>> 10787??? FXATTROP >>>>> ????? 0.53??? 2342.72 us????? 20.91 us 4157542.43 us???????? >>>>> 447821????? UNLINK >>>>> ????? 0.53? 558734.79 us????? 24.40 us 1399487.63 us?????????? >>>>> 1889????????? LK >>>>> ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us????????? >>>>> 10687???? READDIR >>>>> ????? 0.99??? 2426.23 us????? 12.47 us 24176123.13 us???????? >>>>> 809430??? GETXATTR >>>>> ????? 1.45???? 676.83 us?????? 9.30 us 193177067.25 us??????? >>>>> 4260021???? ENTRYLK >>>>> ????? 1.76?? 38325.38 us????? 21.44 us 46693575.93 us????????? >>>>> 91439??? READDIRP >>>>> ????? 4.44??? 1200.31 us?????? 9.10 us 2559377.90 us??????? >>>>> 7352811???? INODELK >>>>> ????? 9.30??? 1849.24 us?????? 9.28 us 35299050.00 us??????? >>>>> 9989221????? LOOKUP >>>>> ???? 79.24?? 47791.46 us????? 76.77 us 10488824.26 us??????? >>>>> 3294642???? XATTROP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 3194806????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 5311???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us???????????? 54??? CI_XATTR >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 1907?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 3189457?? CI_FORGET >>>>> >>>>> ??? Duration: 209524 seconds >>>>> ?? Data Read: 4775365531 bytes >>>>> Data Written: 2354108664 bytes >>>>> >>>>> Interval 0 Stats: >>>>> ?? Block Size: 1b+?????????????????? 8b+????????????????? 16b+ >>>>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>>>> No. of Writes: 117?????????????????? 195?????????????????? 367 >>>>> >>>>> ?? Block Size: 32b+????????????????? 64b+???????????????? 128b+ >>>>> ?No. of Reads: 306???????????????????? 0???????????????????? 6 >>>>> No. of Writes: 1016???????????????????? 9?????????????????? 134 >>>>> >>>>> ?? Block Size: 256b+???????????????? 512b+ 1024b+ >>>>> ?No. of Reads: 107??????????????????? 58?????????????????? 485 >>>>> No. of Writes: 731??????????????????? 73?????????????????? 168 >>>>> >>>>> ?? Block Size: 2048b+??????????????? 4096b+ 8192b+ >>>>> ?No. of Reads: 43??????????????????? 59?????????????????? 383 >>>>> No. of Writes: 295?????????????????? 475???????????????? 10562 >>>>> >>>>> ?? Block Size: 16384b+?????????????? 32768b+ 65536b+ >>>>> ?No. of Reads: 347?????????????????? 380?????????????????? 634 >>>>> No. of Writes: 1487????????????????? 6369????????????????? 3970 >>>>> >>>>> ?? Block Size:???????????? 131072b+ >>>>> ?No. of Reads:??????????????? 35704 >>>>> No. of Writes:??????????????? 12398 >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 957293????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1455091???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 69951? RELEASEDIR >>>>> ????? 0.00????? 16.20 us?????? 5.43 us???? 100.95 >>>>> us??????????? 113???????? IPC >>>>> ????? 0.00???? 687.33 us????? 80.53 us 15414.03 us???????????? >>>>> 30 REMOVEXATTR >>>>> ????? 0.00???? 840.76 us????? 72.71 us 37648.39 us???????????? >>>>> 52?? FTRUNCATE >>>>> ????? 0.00???? 138.49 us????? 36.66 us 25280.24 us?????????? >>>>> 1385??????? STAT >>>>> ????? 0.00???? 363.20 us????? 63.45 us 15957.66 us??????????? >>>>> 810?? FGETXATTR >>>>> ????? 0.00??? 2208.82 us???? 127.23 us 267092.17 us??????????? >>>>> 137??????? LINK >>>>> ????? 0.00?? 37595.33 us??? 1677.46 us 218313.32 >>>>> us???????????? 12?????? FSYNC >>>>> ????? 0.00???? 675.61 us???? 106.82 us 114351.01 us??????????? >>>>> 698??? TRUNCATE >>>>> ????? 0.00??? 3995.66 us???? 147.00 us 109544.46 us??????????? >>>>> 188????? RENAME >>>>> ????? 0.00???? 453.11 us????? 53.37 us 104848.94 us?????????? >>>>> 1680???? SETATTR >>>>> ????? 0.00??? 6004.32 us???? 229.33 us 151322.58 us??????????? >>>>> 127?????? MKNOD >>>>> ????? 0.00????? 68.74 us????? 16.11 us 61050.93 us????????? >>>>> 11412??? FINODELK >>>>> ????? 0.00??? 3793.44 us????? 68.61 us 411523.70 us??????????? >>>>> 221??? SETXATTR >>>>> ????? 0.00?? 28669.99 us???? 227.28 us 138988.41 >>>>> us???????????? 30???? SYMLINK >>>>> ????? 0.00????? 56.19 us?????? 7.31 us 41484.37 us????????? >>>>> 22640?????? FLUSH >>>>> ????? 0.00???? 105.61 us????? 19.34 us 351055.72 us????????? >>>>> 21693????? STATFS >>>>> ????? 0.00?? 23642.90 us???? 244.19 us 234912.25 us??????????? >>>>> 178?????? MKDIR >>>>> ????? 0.00???? 537.38 us????? 31.73 us 296911.67 us?????????? >>>>> 9653?????? FSTAT >>>>> ????? 0.00???? 214.91 us????? 25.71 us 1712020.99 us????????? >>>>> 39054??????? READ >>>>> ????? 0.01???? 147.08 us?????? 1.47 us 241611.80 us????????? >>>>> 69951???? OPENDIR >>>>> ????? 0.01??? 3899.84 us???? 117.22 us 503422.21 us?????????? >>>>> 3180?????? RMDIR >>>>> ????? 0.01???? 327.13 us????? 55.69 us 420376.80 us????????? >>>>> 38366?????? WRITE >>>>> ????? 0.02??? 5580.36 us???? 210.11 us 1029702.30 us?????????? >>>>> 6169????? CREATE >>>>> ????? 0.05????? 92.69 us????? 29.94 us 687955.86 us??????? >>>>> 1159936??? READLINK >>>>> ????? 0.40???? 543.49 us????? 36.45 us 2430653.54 us??????? >>>>> 1449307??????? OPEN >>>>> ????? 0.48?? 87643.67 us????? 70.20 us 661999.51 us????????? >>>>> 10787??? FXATTROP >>>>> ????? 0.53??? 2342.57 us????? 20.91 us 4157542.43 us???????? >>>>> 447821????? UNLINK >>>>> ????? 0.53? 558416.10 us????? 24.40 us 1399487.63 us?????????? >>>>> 1889????????? LK >>>>> ????? 0.78? 145786.31 us????? 19.06 us 37490825.73 us????????? >>>>> 10687???? READDIR >>>>> ????? 0.99??? 2426.24 us????? 12.47 us 24176123.13 us???????? >>>>> 809430??? GETXATTR >>>>> ????? 1.45???? 676.75 us?????? 9.30 us 193177067.25 us??????? >>>>> 4260021???? ENTRYLK >>>>> ????? 1.76?? 38324.59 us????? 21.44 us 46693575.93 us????????? >>>>> 91439??? READDIRP >>>>> ????? 4.44??? 1200.37 us?????? 9.10 us 2559377.90 us??????? >>>>> 7352811???? INODELK >>>>> ????? 9.29??? 1848.94 us?????? 9.28 us 35299050.00 us??????? >>>>> 9989221????? LOOKUP >>>>> ???? 79.24?? 47792.42 us????? 76.77 us 10488824.26 us??????? >>>>> 3294642???? XATTROP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 3194806????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 5311???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us???????????? 54??? CI_XATTR >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 1907?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 3189457?? CI_FORGET >>>>> >>>>> ??? Duration: 209524 seconds >>>>> ?? Data Read: 4775365531 bytes >>>>> Data Written: 2354108664 bytes >>>>> >>>>> Brick: zoidberg:/mnt/gluster/b >>>>> ----------------------------------------- >>>>> Cumulative Stats: >>>>> ?? Block Size: 1b+?????????????????? 2b+?????????????????? 8b+ >>>>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>>>> No. of Writes: 53???????????????????? 1??????????????????? 60 >>>>> >>>>> ?? Block Size: 16b+????????????????? 32b+????????????????? 64b+ >>>>> ?No. of Reads: 0?????????????????? 213???????????????????? 2 >>>>> No. of Writes: 14????????????????? 1057??????????????????? 15 >>>>> >>>>> ?? Block Size: 128b+???????????????? 256b+ 512b+ >>>>> ?No. of Reads: 0??????????????????? 41?????????????????? 146 >>>>> No. of Writes: 59?????????????????? 715??????????????????? 80 >>>>> >>>>> ?? Block Size: 1024b+??????????????? 2048b+ 4096b+ >>>>> ?No. of Reads: 224???????????????????? 9??????????????????? 15 >>>>> No. of Writes: 163?????????????????? 282?????????????????? 404 >>>>> >>>>> ?? Block Size: 8192b+?????????????? 16384b+ 32768b+ >>>>> ?No. of Reads: 52??????????????????? 74?????????????????? 210 >>>>> No. of Writes: 7415????????????????? 1430????????????????? 6860 >>>>> >>>>> ?? Block Size: 65536b+????????????? 131072b+ 262144b+ >>>>> ?No. of Reads: 299???????????????? 15313???????????????????? 1 >>>>> No. of Writes: 2458???????????????? 12378???????????????????? 1 >>>>> >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 686733????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 837434???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 87338? RELEASEDIR >>>>> ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 >>>>> us??????????? 116???????? IPC >>>>> ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 >>>>> us???????????? 57?? FTRUNCATE >>>>> ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 >>>>> us???????????? 65?? FGETXATTR >>>>> ????? 0.00??? 2336.06 us????? 90.05 us 20509.81 us???????????? >>>>> 23 REMOVEXATTR >>>>> ????? 0.00???? 631.26 us???? 123.17 us 51803.77 us??????????? >>>>> 121??????? LINK >>>>> ????? 0.00???? 369.27 us????? 22.64 us 145090.48 us??????????? >>>>> 490????????? LK >>>>> ????? 0.00???? 360.85 us????? 45.14 us 46846.98 us??????????? >>>>> 540??????? STAT >>>>> ????? 0.00??? 1190.96 us???? 117.40 us 69781.85 us??????????? >>>>> 200??? TRUNCATE >>>>> ????? 0.00???? 245.72 us????? 54.91 us 123985.76 us?????????? >>>>> 1482???? SETATTR >>>>> ????? 0.00?? 42139.10 us??? 3313.56 us 312273.21 >>>>> us???????????? 12?????? FSYNC >>>>> ????? 0.00????? 55.83 us?????? 8.26 us 49649.35 us????????? >>>>> 10047?????? FLUSH >>>>> ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us?????????? >>>>> 9977??? FINODELK >>>>> ????? 0.00??? 3191.60 us???? 143.06 us 96214.75 us??????????? >>>>> 188????? RENAME >>>>> ????? 0.00?? 32862.86 us???? 215.38 us 83851.13 us???????????? >>>>> 25???? SYMLINK >>>>> ????? 0.00?? 11392.03 us???? 230.70 us 105471.90 us??????????? >>>>> 118?????? MKNOD >>>>> ????? 0.00???? 134.63 us????? 28.08 us 36326.43 us????????? >>>>> 12587?????? FSTAT >>>>> ????? 0.00???? 112.32 us????? 19.91 us 381241.34 us????????? >>>>> 21695????? STATFS >>>>> ????? 0.00???? 307.86 us????? 28.00 us 292666.83 us????????? >>>>> 16841??????? READ >>>>> ????? 0.00?? 29625.60 us???? 239.57 us 239532.26 us??????????? >>>>> 178?????? MKDIR >>>>> ????? 0.00???? 319.24 us????? 51.19 us 136353.39 us????????? >>>>> 33445?????? WRITE >>>>> ????? 0.01???? 175.83 us?????? 1.57 us 513635.22 us????????? >>>>> 87338???? OPENDIR >>>>> ????? 0.01???? 249.96 us????? 58.42 us 594774.86 us????????? >>>>> 85089??? SETXATTR >>>>> ????? 0.01?? 11001.25 us???? 119.87 us 612765.03 us?????????? >>>>> 3180?????? RMDIR >>>>> ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us?????????? >>>>> 6067????? CREATE >>>>> ????? 0.05????? 89.45 us????? 30.14 us 687961.60 us??????? >>>>> 1219542??? READLINK >>>>> ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us???????? >>>>> 831371??????? OPEN >>>>> ????? 0.51? 127078.45 us????? 78.55 us 694364.09 us?????????? >>>>> 9493??? FXATTROP >>>>> ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us??????? >>>>> 3764714???? ENTRYLK >>>>> ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us????????? >>>>> 22253???? READDIR >>>>> ????? 1.06??? 6913.71 us????? 11.58 us 36968532.72 us???????? >>>>> 364963??? GETXATTR >>>>> ????? 1.29?? 62130.12 us????? 36.89 us 8649871.11 us????????? >>>>> 49472??? READDIRP >>>>> ????? 2.89??? 1477.53 us????? 11.10 us 2807637.64 us??????? >>>>> 4650500???? INODELK >>>>> ????? 4.12?? 23681.31 us????? 20.98 us 23291414.50 us???????? >>>>> 414352????? UNLINK >>>>> ???? 16.09??? 5000.31 us?????? 9.04 us 23866016.88 us??????? >>>>> 7655621????? LOOKUP >>>>> ???? 72.04?? 68953.31 us????? 68.40 us 27542499.65 us??????? >>>>> 2486126???? XATTROP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 2433405????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 2453???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us???????????? 69??? CI_XATTR >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 1544?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 2430885?? CI_FORGET >>>>> >>>>> ??? Duration: 209524 seconds >>>>> ?? Data Read: 2051388524 bytes >>>>> Data Written: 2208261499 bytes >>>>> >>>>> Interval 0 Stats: >>>>> ?? Block Size: 1b+?????????????????? 2b+?????????????????? 8b+ >>>>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>>>> No. of Writes: 53???????????????????? 1??????????????????? 60 >>>>> >>>>> ?? Block Size: 16b+????????????????? 32b+????????????????? 64b+ >>>>> ?No. of Reads: 0?????????????????? 213???????????????????? 2 >>>>> No. of Writes: 14????????????????? 1057??????????????????? 15 >>>>> >>>>> ?? Block Size: 128b+???????????????? 256b+ 512b+ >>>>> ?No. of Reads: 0??????????????????? 41?????????????????? 146 >>>>> No. of Writes: 59?????????????????? 715??????????????????? 80 >>>>> >>>>> ?? Block Size: 1024b+??????????????? 2048b+ 4096b+ >>>>> ?No. of Reads: 224???????????????????? 9??????????????????? 15 >>>>> No. of Writes: 163?????????????????? 282?????????????????? 404 >>>>> >>>>> ?? Block Size: 8192b+?????????????? 16384b+ 32768b+ >>>>> ?No. of Reads: 52??????????????????? 74?????????????????? 210 >>>>> No. of Writes: 7415????????????????? 1430????????????????? 6860 >>>>> >>>>> ?? Block Size: 65536b+????????????? 131072b+ 262144b+ >>>>> ?No. of Reads: 299???????????????? 15313???????????????????? 1 >>>>> No. of Writes: 2458???????????????? 12378???????????????????? 1 >>>>> >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 686733????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 837434???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 87338? RELEASEDIR >>>>> ????? 0.00????? 22.69 us?????? 7.16 us???? 332.61 >>>>> us??????????? 116???????? IPC >>>>> ????? 0.00???? 119.93 us????? 70.56 us???? 272.28 >>>>> us???????????? 57?? FTRUNCATE >>>>> ????? 0.00???? 152.25 us????? 86.82 us???? 340.63 >>>>> us???????????? 65?? FGETXATTR >>>>> ????? 0.00??? 2336.06 us????? 90.05 us 20509.81 us???????????? >>>>> 23 REMOVEXATTR >>>>> ????? 0.00???? 631.26 us???? 123.17 us 51803.77 us??????????? >>>>> 121??????? LINK >>>>> ????? 0.00???? 369.27 us????? 22.64 us 145090.48 us??????????? >>>>> 490????????? LK >>>>> ????? 0.00???? 360.85 us????? 45.14 us 46846.98 us??????????? >>>>> 540??????? STAT >>>>> ????? 0.00??? 1190.96 us???? 117.40 us 69781.85 us??????????? >>>>> 200??? TRUNCATE >>>>> ????? 0.00???? 245.72 us????? 54.91 us 123985.76 us?????????? >>>>> 1482???? SETATTR >>>>> ????? 0.00?? 42139.10 us??? 3313.56 us 312273.21 >>>>> us???????????? 12?????? FSYNC >>>>> ????? 0.00????? 55.83 us?????? 8.26 us 49649.35 us????????? >>>>> 10047?????? FLUSH >>>>> ????? 0.00????? 57.52 us????? 14.62 us???? 820.59 us?????????? >>>>> 9977??? FINODELK >>>>> ????? 0.00??? 3191.60 us???? 143.06 us 96214.75 us??????????? >>>>> 188????? RENAME >>>>> ????? 0.00?? 32862.86 us???? 215.38 us 83851.13 us???????????? >>>>> 25???? SYMLINK >>>>> ????? 0.00?? 11392.03 us???? 230.70 us 105471.90 us??????????? >>>>> 118?????? MKNOD >>>>> ????? 0.00???? 134.63 us????? 28.08 us 36326.43 us????????? >>>>> 12587?????? FSTAT >>>>> ????? 0.00???? 112.32 us????? 19.91 us 381241.34 us????????? >>>>> 21695????? STATFS >>>>> ????? 0.00???? 307.91 us????? 28.00 us 292666.83 us????????? >>>>> 16841??????? READ >>>>> ????? 0.00?? 29625.60 us???? 239.57 us 239532.26 us??????????? >>>>> 178?????? MKDIR >>>>> ????? 0.00???? 319.22 us????? 51.19 us 136353.39 us????????? >>>>> 33445?????? WRITE >>>>> ????? 0.01???? 175.83 us?????? 1.57 us 513635.22 us????????? >>>>> 87338???? OPENDIR >>>>> ????? 0.01???? 249.95 us????? 58.42 us 594774.86 us????????? >>>>> 85089??? SETXATTR >>>>> ????? 0.01?? 11001.25 us???? 119.87 us 612765.03 us?????????? >>>>> 3180?????? RMDIR >>>>> ????? 0.04?? 14240.66 us???? 203.89 us 1376758.99 us?????????? >>>>> 6067????? CREATE >>>>> ????? 0.05????? 89.46 us????? 30.14 us 687961.60 us??????? >>>>> 1219542??? READLINK >>>>> ????? 0.39??? 1114.30 us????? 44.29 us 2412169.89 us???????? >>>>> 831371??????? OPEN >>>>> ????? 0.51? 127078.45 us????? 78.55 us 694364.09 us?????????? >>>>> 9493??? FXATTROP >>>>> ????? 0.63???? 399.16 us?????? 8.43 us 3815689.36 us??????? >>>>> 3764714???? ENTRYLK >>>>> ????? 0.86?? 91973.00 us????? 19.89 us 24258016.33 us????????? >>>>> 22253???? READDIR >>>>> ????? 1.06??? 6913.26 us????? 11.58 us 36968532.72 us???????? >>>>> 364963??? GETXATTR >>>>> ????? 1.29?? 62121.95 us????? 36.89 us 8649871.11 us????????? >>>>> 49472??? READDIRP >>>>> ????? 2.89??? 1477.58 us????? 11.10 us 2807637.64 us??????? >>>>> 4650500???? INODELK >>>>> ????? 4.12?? 23681.29 us????? 20.98 us 23291414.50 us???????? >>>>> 414352????? UNLINK >>>>> ???? 16.09??? 5000.02 us?????? 9.04 us 23866016.88 us??????? >>>>> 7655621????? LOOKUP >>>>> ???? 72.04?? 68956.14 us????? 68.40 us 27542499.65 us??????? >>>>> 2486126???? XATTROP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 2433405????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 2453???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us???????????? 69??? CI_XATTR >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 1544?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 2430885?? CI_FORGET >>>>> >>>>> ??? Duration: 209524 seconds >>>>> ?? Data Read: 2051388524 bytes >>>>> Data Written: 2208261499 bytes >>>>> >>>>> Brick: flexo:/mnt/gluster/b >>>>> -------------------------------------- >>>>> Cumulative Stats: >>>>> ?? Block Size: 1b+?????????????????? 8b+????????????????? 16b+ >>>>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>>>> No. of Writes: 117?????????????????? 195?????????????????? 367 >>>>> >>>>> ?? Block Size: 32b+????????????????? 64b+???????????????? 128b+ >>>>> ?No. of Reads: 130???????????????????? 0???????????????????? 3 >>>>> No. of Writes: 1012???????????????????? 9?????????????????? 134 >>>>> >>>>> ?? Block Size: 256b+???????????????? 512b+ 1024b+ >>>>> ?No. of Reads: 47???????????????????? 8?????????????????? 257 >>>>> No. of Writes: 729??????????????????? 73?????????????????? 168 >>>>> >>>>> ?? Block Size: 2048b+??????????????? 4096b+ 8192b+ >>>>> ?No. of Reads: 10??????????????????? 27?????????????????? 240 >>>>> No. of Writes: 294?????????????????? 466???????????????? 10472 >>>>> >>>>> ?? Block Size: 16384b+?????????????? 32768b+ 65536b+ >>>>> ?No. of Reads: 139?????????????????? 119?????????????????? 135 >>>>> No. of Writes: 1480????????????????? 6273????????????????? 3935 >>>>> >>>>> ?? Block Size:???????????? 131072b+ >>>>> ?No. of Reads:???????????????? 7892 >>>>> No. of Writes:??????????????? 12327 >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 955058????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1450749???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 81377? RELEASEDIR >>>>> ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 >>>>> us??????????? 108???????? IPC >>>>> ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 >>>>> us???????????? 44?? FTRUNCATE >>>>> ????? 0.00???? 198.38 us????? 81.68 us 1332.45 us???????????? >>>>> 30 REMOVEXATTR >>>>> ????? 0.00??? 4254.16 us???? 621.52 us 10602.99 >>>>> us????????????? 4?????? FSYNC >>>>> ????? 0.00???? 795.08 us???? 190.30 us 16518.64 us???????????? >>>>> 30???? SYMLINK >>>>> ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 >>>>> us??????????? 137??????? LINK >>>>> ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 >>>>> us??????????? 119?????? MKNOD >>>>> ????? 0.00???? 162.61 us????? 65.62 us 3864.77 us??????????? >>>>> 221??? SETXATTR >>>>> ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 >>>>> us??????????? 354??????? STAT >>>>> ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 >>>>> us??????????? 777????????? LK >>>>> ????? 0.00???? 461.72 us???? 114.54 us 16013.81 us??????????? >>>>> 188????? RENAME >>>>> ????? 0.00???? 150.40 us????? 75.59 us???? 749.54 >>>>> us??????????? 810?? FGETXATTR >>>>> ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 >>>>> us??????????? 698??? TRUNCATE >>>>> ????? 0.00??? 1176.00 us???? 194.89 us 78261.12 us??????????? >>>>> 178?????? MKDIR >>>>> ????? 0.00???? 439.83 us????? 54.38 us 75270.51 us?????????? >>>>> 1665???? SETATTR >>>>> ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us????????? >>>>> 11372??? FINODELK >>>>> ????? 0.00???? 131.57 us????? 32.68 us 2107.52 us?????????? >>>>> 7398?????? FSTAT >>>>> ????? 0.00???? 120.71 us????? 31.43 us???? 847.97 us?????????? >>>>> 9094??????? READ >>>>> ????? 0.00????? 60.75 us?????? 7.25 us 22520.98 us????????? >>>>> 22614?????? FLUSH >>>>> ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us????????? >>>>> 21656????? STATFS >>>>> ????? 0.00???? 323.63 us???? 166.56 us 23034.44 us?????????? >>>>> 6145????? CREATE >>>>> ????? 0.00???? 217.55 us????? 50.60 us 30734.99 us????????? >>>>> 38051?????? WRITE >>>>> ????? 0.01???? 146.39 us?????? 1.80 us 318612.45 us????????? >>>>> 81377???? OPENDIR >>>>> ????? 0.01??? 5641.37 us???? 108.84 us 468960.33 us?????????? >>>>> 3177?????? RMDIR >>>>> ????? 0.03???? 100.05 us????? 30.93 us 49756.93 us???????? >>>>> 432439??? READLINK >>>>> ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us???????? >>>>> 448381????? UNLINK >>>>> ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us????????? >>>>> 52185??? READDIRP >>>>> ????? 0.35???? 414.05 us????? 36.58 us 1038229.93 us??????? >>>>> 1444606??????? OPEN >>>>> ????? 0.49?? 78518.02 us????? 73.39 us 373999.31 us????????? >>>>> 10745??? FXATTROP >>>>> ????? 0.72???? 776.21 us????? 11.24 us 4271792.57 us??????? >>>>> 1605822??? GETXATTR >>>>> ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us????????? >>>>> 13532???? READDIR >>>>> ????? 0.79???? 320.44 us?????? 7.71 us 2369511.06 us??????? >>>>> 4257297???? ENTRYLK >>>>> ????? 1.97???? 464.02 us?????? 9.38 us 1038460.80 us??????? >>>>> 7333722???? INODELK >>>>> ???? 13.73??? 2372.50 us?????? 8.33 us 6258159.87 us??????? >>>>> 9979799????? LOOKUP >>>>> ???? 80.79?? 42372.04 us????? 61.49 us 2238308.13 us??????? >>>>> 3289103???? XATTROP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 2250268????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 3756???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us??????????? 167??? CI_XATTR >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 1908?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 2246352?? CI_FORGET >>>>> >>>>> ??? Duration: 209334 seconds >>>>> ?? Data Read: 1059777395 bytes >>>>> Data Written: 2336921014 bytes >>>>> >>>>> Interval 0 Stats: >>>>> ?? Block Size: 1b+?????????????????? 8b+????????????????? 16b+ >>>>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>>>> No. of Writes: 117?????????????????? 195?????????????????? 367 >>>>> >>>>> ?? Block Size: 32b+????????????????? 64b+???????????????? 128b+ >>>>> ?No. of Reads: 130???????????????????? 0???????????????????? 3 >>>>> No. of Writes: 1012???????????????????? 9?????????????????? 134 >>>>> >>>>> ?? Block Size: 256b+???????????????? 512b+ 1024b+ >>>>> ?No. of Reads: 47???????????????????? 8?????????????????? 257 >>>>> No. of Writes: 729??????????????????? 73?????????????????? 168 >>>>> >>>>> ?? Block Size: 2048b+??????????????? 4096b+ 8192b+ >>>>> ?No. of Reads: 10??????????????????? 27?????????????????? 240 >>>>> No. of Writes: 294?????????????????? 466???????????????? 10472 >>>>> >>>>> ?? Block Size: 16384b+?????????????? 32768b+ 65536b+ >>>>> ?No. of Reads: 139?????????????????? 119?????????????????? 135 >>>>> No. of Writes: 1480????????????????? 6273????????????????? 3935 >>>>> >>>>> ?? Block Size:???????????? 131072b+ >>>>> ?No. of Reads:???????????????? 7892 >>>>> No. of Writes:??????????????? 12327 >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 955058????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1450749???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 81377? RELEASEDIR >>>>> ????? 0.00????? 15.55 us?????? 5.35 us????? 31.88 >>>>> us??????????? 108???????? IPC >>>>> ????? 0.00???? 125.85 us????? 59.77 us???? 224.15 >>>>> us???????????? 44?? FTRUNCATE >>>>> ????? 0.00???? 198.38 us????? 81.68 us 1332.45 us???????????? >>>>> 30 REMOVEXATTR >>>>> ????? 0.00??? 4254.16 us???? 621.52 us 10602.99 >>>>> us????????????? 4?????? FSYNC >>>>> ????? 0.00???? 795.08 us???? 190.30 us 16518.64 us???????????? >>>>> 30???? SYMLINK >>>>> ????? 0.00???? 195.90 us???? 111.02 us???? 953.39 >>>>> us??????????? 137??????? LINK >>>>> ????? 0.00???? 286.07 us???? 179.64 us???? 437.82 >>>>> us??????????? 119?????? MKNOD >>>>> ????? 0.00???? 162.61 us????? 65.62 us 3864.77 us??????????? >>>>> 221??? SETXATTR >>>>> ????? 0.00???? 117.29 us????? 48.50 us???? 317.17 >>>>> us??????????? 354??????? STAT >>>>> ????? 0.00????? 70.35 us????? 23.10 us???? 317.32 >>>>> us??????????? 777????????? LK >>>>> ????? 0.00???? 461.72 us???? 114.54 us 16013.81 us??????????? >>>>> 188????? RENAME >>>>> ????? 0.00???? 150.43 us????? 75.59 us???? 749.54 >>>>> us??????????? 810?? FGETXATTR >>>>> ????? 0.00???? 267.80 us????? 99.67 us???? 994.58 >>>>> us??????????? 698??? TRUNCATE >>>>> ????? 0.00??? 1176.00 us???? 194.89 us 78261.12 us??????????? >>>>> 178?????? MKDIR >>>>> ????? 0.00???? 439.83 us????? 54.38 us 75270.51 us?????????? >>>>> 1665???? SETATTR >>>>> ????? 0.00????? 65.89 us????? 17.39 us???? 354.77 us????????? >>>>> 11372??? FINODELK >>>>> ????? 0.00???? 131.57 us????? 32.68 us 2107.52 us?????????? >>>>> 7398?????? FSTAT >>>>> ????? 0.00???? 120.74 us????? 31.43 us???? 847.97 us?????????? >>>>> 9094??????? READ >>>>> ????? 0.00????? 60.75 us?????? 7.25 us 22520.98 us????????? >>>>> 22614?????? FLUSH >>>>> ????? 0.00????? 73.03 us????? 22.41 us???? 750.17 us????????? >>>>> 21656????? STATFS >>>>> ????? 0.00???? 323.63 us???? 166.56 us 23034.44 us?????????? >>>>> 6145????? CREATE >>>>> ????? 0.00???? 217.50 us????? 50.60 us 30734.99 us????????? >>>>> 38051?????? WRITE >>>>> ????? 0.01???? 146.39 us?????? 1.80 us 318612.45 us????????? >>>>> 81377???? OPENDIR >>>>> ????? 0.01??? 5641.37 us???? 108.84 us 468960.33 us?????????? >>>>> 3177?????? RMDIR >>>>> ????? 0.03???? 100.06 us????? 30.93 us 49756.93 us???????? >>>>> 432439??? READLINK >>>>> ????? 0.13???? 483.47 us????? 16.81 us 2608013.14 us???????? >>>>> 448381????? UNLINK >>>>> ????? 0.21??? 7101.56 us????? 22.26 us 2554390.49 us????????? >>>>> 52185??? READDIRP >>>>> ????? 0.35???? 414.01 us????? 36.58 us 1038229.93 us??????? >>>>> 1444606??????? OPEN >>>>> ????? 0.49?? 78518.02 us????? 73.39 us 373999.31 us????????? >>>>> 10745??? FXATTROP >>>>> ????? 0.72???? 776.24 us????? 11.24 us 4271792.57 us??????? >>>>> 1605822??? GETXATTR >>>>> ????? 0.77?? 97829.38 us????? 18.84 us 9477446.11 us????????? >>>>> 13532???? READDIR >>>>> ????? 0.79???? 320.45 us?????? 7.71 us 2369511.06 us??????? >>>>> 4257297???? ENTRYLK >>>>> ????? 1.97???? 464.07 us?????? 9.38 us 1038460.80 us??????? >>>>> 7333722???? INODELK >>>>> ???? 13.72??? 2372.42 us?????? 8.33 us 6258159.87 us??????? >>>>> 9979799????? LOOKUP >>>>> ???? 80.79?? 42373.50 us????? 61.49 us 2238308.13 us??????? >>>>> 3289103???? XATTROP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 2250268????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 3756???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us??????????? 167??? CI_XATTR >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 1908?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 2246352?? CI_FORGET >>>>> >>>>> ??? Duration: 209334 seconds >>>>> ?? Data Read: 1059777395 bytes >>>>> Data Written: 2336921014 bytes >>>>> >>>>> Brick: flexo:/mnt/gluster/c >>>>> -------------------------------------- >>>>> Cumulative Stats: >>>>> ?? Block Size:????????????????? 1b+ >>>>> ?No. of Reads:??????????????????? 0 >>>>> No. of Writes:??????????????? 32843 >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 686562????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 834596???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 93757? RELEASEDIR >>>>> ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 >>>>> us??????????? 107???????? IPC >>>>> ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 >>>>> us???????????? 36??????? LINK >>>>> ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 >>>>> us???????????? 65?? FGETXATTR >>>>> ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 >>>>> us???????????? 88????? RENAME >>>>> ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 >>>>> us??????????? 153??? TRUNCATE >>>>> ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 >>>>> us??????????? 491????????? LK >>>>> ????? 0.01????? 51.73 us?????? 7.70 us 1174.64 us?????????? >>>>> 9339?????? FLUSH >>>>> ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us?????????? >>>>> 9011??? FINODELK >>>>> ????? 0.03???? 126.23 us????? 43.15 us 4760.14 us????????? >>>>> 10824?????? FSTAT >>>>> ????? 0.03???? 571.94 us????? 94.85 us 51697.03 us?????????? >>>>> 2594?????? MKDIR >>>>> ????? 0.05???? 264.87 us????? 81.58 us 53152.06 us?????????? >>>>> 9009??? FXATTROP >>>>> ????? 0.05???? 423.05 us????? 38.26 us 548686.55 us?????????? >>>>> 5754????? CREATE >>>>> ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us????????? >>>>> 32843?????? WRITE >>>>> ????? 0.06???? 526.39 us???? 130.84 us 41405.46 us?????????? >>>>> 5798???? SYMLINK >>>>> ????? 0.10???? 133.73 us????? 16.25 us 676494.72 us????????? >>>>> 34904???? READDIR >>>>> ????? 0.12??? 2054.48 us????? 99.28 us 88061.83 us?????????? >>>>> 2732?????? RMDIR >>>>> ????? 0.21???? 110.60 us?????? 1.19 us 16833.78 us????????? >>>>> 93757???? OPENDIR >>>>> ????? 0.72???? 205.89 us????? 33.28 us 261023.99 us???????? >>>>> 170292????? UNLINK >>>>> ????? 0.85??? 2971.45 us????? 21.24 us 257839.19 us????????? >>>>> 13840??? READDIRP >>>>> ????? 1.94???? 200.88 us????? 25.49 us 461654.32 us???????? >>>>> 468842?? FTRUNCATE >>>>> ????? 2.13???? 212.44 us????? 54.02 us 461878.34 us???????? >>>>> 486882 REMOVEXATTR >>>>> ????? 2.27???? 223.01 us????? 52.03 us 178897.40 us???????? >>>>> 493240??? SETXATTR >>>>> ????? 3.53???? 155.31 us?????? 7.96 us 67873.00 us??????? >>>>> 1101338??? GETXATTR >>>>> ????? 3.63???? 212.32 us????? 44.42 us 551264.02 us???????? >>>>> 829002??????? OPEN >>>>> ????? 5.77????? 76.36 us?????? 7.76 us 962634.73 us??????? >>>>> 3660425???? ENTRYLK >>>>> ????? 6.23???? 304.93 us????? 42.51 us 661394.43 us???????? >>>>> 990644???? SETATTR >>>>> ????? 7.81???? 382.20 us???? 107.42 us 846502.78 us???????? >>>>> 990766???? XATTROP >>>>> ???? 12.39???? 549.11 us???? 135.57 us 1659697.83 us??????? >>>>> 1093387?????? MKNOD >>>>> ???? 12.47???? 139.78 us?????? 9.76 us 552067.56 us??????? >>>>> 4325281???? INODELK >>>>> ???? 39.53???? 206.94 us?????? 8.56 us 964823.33 us??????? >>>>> 9259429????? LOOKUP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1856377????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 933300???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us??????????? 251??? CI_XATTR >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us??????????? 293?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 923077?? CI_FORGET >>>>> >>>>> ??? Duration: 209334 seconds >>>>> ?? Data Read: 0 bytes >>>>> Data Written: 32843 bytes >>>>> >>>>> Interval 0 Stats: >>>>> ?? Block Size:????????????????? 1b+ >>>>> ?No. of Reads:??????????????????? 0 >>>>> No. of Writes:??????????????? 32843 >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 686562????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 834596???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 93757? RELEASEDIR >>>>> ????? 0.00????? 18.50 us?????? 8.21 us????? 50.20 >>>>> us??????????? 107???????? IPC >>>>> ????? 0.00???? 205.83 us???? 126.30 us???? 412.06 >>>>> us???????????? 36??????? LINK >>>>> ????? 0.00???? 167.33 us????? 75.01 us???? 272.47 >>>>> us???????????? 65?? FGETXATTR >>>>> ????? 0.00???? 203.58 us???? 117.35 us???? 648.28 >>>>> us???????????? 88????? RENAME >>>>> ????? 0.00???? 203.20 us????? 66.60 us???? 809.96 >>>>> us??????????? 153??? TRUNCATE >>>>> ????? 0.00????? 67.06 us????? 23.64 us???? 555.59 >>>>> us??????????? 491????????? LK >>>>> ????? 0.01????? 51.71 us?????? 7.70 us 1174.64 us?????????? >>>>> 9339?????? FLUSH >>>>> ????? 0.01????? 62.41 us????? 15.13 us???? 636.35 us?????????? >>>>> 9011??? FINODELK >>>>> ????? 0.03???? 126.23 us????? 43.15 us 4760.14 us????????? >>>>> 10824?????? FSTAT >>>>> ????? 0.03???? 571.94 us????? 94.85 us 51697.03 us?????????? >>>>> 2594?????? MKDIR >>>>> ????? 0.05???? 264.87 us????? 81.58 us 53152.06 us?????????? >>>>> 9009??? FXATTROP >>>>> ????? 0.05???? 423.05 us????? 38.26 us 548686.55 us?????????? >>>>> 5754????? CREATE >>>>> ????? 0.05????? 78.14 us????? 19.17 us???? 678.32 us????????? >>>>> 32843?????? WRITE >>>>> ????? 0.06???? 526.39 us???? 130.84 us 41405.46 us?????????? >>>>> 5798???? SYMLINK >>>>> ????? 0.10???? 133.73 us????? 16.25 us 676494.72 us????????? >>>>> 34904???? READDIR >>>>> ????? 0.12??? 2054.48 us????? 99.28 us 88061.83 us?????????? >>>>> 2732?????? RMDIR >>>>> ????? 0.21???? 110.60 us?????? 1.19 us 16833.78 us????????? >>>>> 93757???? OPENDIR >>>>> ????? 0.72???? 205.89 us????? 33.28 us 261023.99 us???????? >>>>> 170292????? UNLINK >>>>> ????? 0.85??? 2971.45 us????? 21.24 us 257839.19 us????????? >>>>> 13840??? READDIRP >>>>> ????? 1.94???? 200.87 us????? 25.49 us 461654.32 us???????? >>>>> 468842?? FTRUNCATE >>>>> ????? 2.13???? 212.44 us????? 54.02 us 461878.34 us???????? >>>>> 486882 REMOVEXATTR >>>>> ????? 2.27???? 223.00 us????? 52.03 us 178897.40 us???????? >>>>> 493240??? SETXATTR >>>>> ????? 3.53???? 155.32 us?????? 7.96 us 67873.00 us??????? >>>>> 1101338??? GETXATTR >>>>> ????? 3.63???? 212.32 us????? 44.42 us 551264.02 us???????? >>>>> 829002??????? OPEN >>>>> ????? 5.77????? 76.36 us?????? 7.76 us 962634.73 us??????? >>>>> 3660425???? ENTRYLK >>>>> ????? 6.23???? 304.88 us????? 42.51 us 661394.43 us???????? >>>>> 990644???? SETATTR >>>>> ????? 7.81???? 382.20 us???? 107.42 us 846502.78 us???????? >>>>> 990766???? XATTROP >>>>> ???? 12.39???? 549.25 us???? 135.57 us 1659697.83 us??????? >>>>> 1093387?????? MKNOD >>>>> ???? 12.47???? 139.78 us?????? 9.76 us 552067.56 us??????? >>>>> 4325281???? INODELK >>>>> ???? 39.53???? 206.95 us?????? 8.56 us 964823.33 us??????? >>>>> 9259429????? LOOKUP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1856377????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 933300???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us??????????? 251??? CI_XATTR >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us??????????? 293?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 923077?? CI_FORGET >>>>> >>>>> ??? Duration: 209334 seconds >>>>> ?? Data Read: 0 bytes >>>>> Data Written: 32843 bytes >>>>> >>>>> Brick: bender:/mnt/gluster/c >>>>> --------------------------------------- >>>>> Cumulative Stats: >>>>> ?? Block Size:????????????????? 1b+ >>>>> ?No. of Reads:??????????????????? 0 >>>>> No. of Writes:??????????????? 37050 >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ?????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 928119????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1401524???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 70092? RELEASEDIR >>>>> ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 >>>>> us??????????? 100???????? IPC >>>>> ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 >>>>> us???????????? 38??????? LINK >>>>> ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 >>>>> us???????????? 30???? SYMLINK >>>>> ????? 0.00??? 2670.18 us???? 765.93 us 4151.64 us????????????? >>>>> 4?????? FSYNC >>>>> ????? 0.00???? 350.65 us???? 141.92 us 10573.64 us??????????? >>>>> 122????? RENAME >>>>> ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 >>>>> us??????????? 777????????? LK >>>>> ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 >>>>> us??????????? 810?? FGETXATTR >>>>> ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 >>>>> us??????????? 603??? TRUNCATE >>>>> ????? 0.01??? 1742.85 us???? 202.81 us 47471.71 us??????????? >>>>> 302?????? MKDIR >>>>> ????? 0.01????? 59.05 us????? 15.40 us 1813.56 us????????? >>>>> 10060??? FINODELK >>>>> ????? 0.01???? 118.54 us????? 36.65 us 4002.58 us?????????? >>>>> 6409?????? FSTAT >>>>> ????? 0.01????? 49.38 us?????? 8.08 us 1264.41 us????????? >>>>> 21719?????? FLUSH >>>>> ????? 0.03????? 74.76 us????? 20.86 us 2008.41 us????????? >>>>> 37050?????? WRITE >>>>> ????? 0.04???? 553.12 us????? 40.81 us 386485.29 us?????????? >>>>> 5901????? CREATE >>>>> ????? 0.06???? 534.01 us???? 100.86 us 527770.44 us????????? >>>>> 10057??? FXATTROP >>>>> ????? 0.12???? 138.96 us?????? 1.00 us 601208.70 us????????? >>>>> 70092???? OPENDIR >>>>> ????? 0.12???? 947.76 us????? 18.54 us 712568.33 us????????? >>>>> 10779???? READDIR >>>>> ????? 0.15??? 4099.07 us???? 124.69 us 421939.47 us?????????? >>>>> 3008?????? RMDIR >>>>> ????? 0.61???? 465.30 us????? 20.67 us 936959.86 us???????? >>>>> 109478????? UNLINK >>>>> ????? 1.57???? 182.41 us????? 49.54 us 73938.09 us???????? >>>>> 722669 REMOVEXATTR >>>>> ????? 1.61???? 171.41 us?????? 6.98 us 83809.88 us???????? >>>>> 789294??? GETXATTR >>>>> ????? 1.86???? 217.88 us????? 26.40 us 556446.19 us???????? >>>>> 718835?? FTRUNCATE >>>>> ????? 1.89???? 219.47 us????? 49.92 us 506767.78 us???????? >>>>> 722856??? SETXATTR >>>>> ????? 3.02???? 181.66 us????? 44.40 us 344656.69 us??????? >>>>> 1395957??????? OPEN >>>>> ????? 5.56???? 322.08 us????? 43.37 us 634069.61 us??????? >>>>> 1450160???? SETATTR >>>>> ????? 5.84???? 115.88 us?????? 7.67 us 747387.51 us??????? >>>>> 4233520???? ENTRYLK >>>>> ????? 6.53???? 378.33 us???? 122.84 us 766939.14 us??????? >>>>> 1449601???? XATTROP >>>>> ????? 9.75???? 115.48 us????? 10.70 us 556675.13 us??????? >>>>> 7097226???? INODELK >>>>> ???? 19.37??? 1104.98 us???? 101.28 us 1612346.75 us??????? >>>>> 1472813?????? MKNOD >>>>> ???? 41.85???? 295.61 us?????? 8.98 us 996805.54 us?????? >>>>> 11896091????? LOOKUP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 3531139????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1862470???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us??????????? 244??? CI_XATTR >>>>> ?????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us??????????? 632?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1668669?? CI_FORGET >>>>> >>>>> ??? Duration: 208643 seconds >>>>> ?? Data Read: 0 bytes >>>>> Data Written: 37050 bytes >>>>> >>>>> Interval 0 Stats: >>>>> ?? Block Size:????????????????? 1b+ >>>>> ?No. of Reads:??????????????????? 0 >>>>> No. of Writes:??????????????? 37050 >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 928119????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1401524???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 70092? RELEASEDIR >>>>> ????? 0.00????? 19.49 us?????? 8.28 us???? 180.61 >>>>> us??????????? 100???????? IPC >>>>> ????? 0.00???? 238.88 us???? 163.97 us???? 486.93 >>>>> us???????????? 38??????? LINK >>>>> ????? 0.00???? 306.43 us???? 196.67 us???? 788.36 >>>>> us???????????? 30???? SYMLINK >>>>> ????? 0.00??? 2670.18 us???? 765.93 us 4151.64 us????????????? >>>>> 4?????? FSYNC >>>>> ????? 0.00???? 350.65 us???? 141.92 us 10573.64 us??????????? >>>>> 122????? RENAME >>>>> ????? 0.00????? 67.02 us????? 20.66 us???? 453.90 >>>>> us??????????? 777????????? LK >>>>> ????? 0.00???? 158.32 us????? 64.22 us???? 532.21 >>>>> us??????????? 810?? FGETXATTR >>>>> ????? 0.00???? 229.87 us????? 66.47 us???? 826.31 >>>>> us??????????? 603??? TRUNCATE >>>>> ????? 0.01??? 1742.85 us???? 202.81 us 47471.71 us??????????? >>>>> 302?????? MKDIR >>>>> ????? 0.01????? 59.05 us????? 15.40 us 1813.56 us????????? >>>>> 10060??? FINODELK >>>>> ????? 0.01???? 118.54 us????? 36.65 us 4002.58 us?????????? >>>>> 6409?????? FSTAT >>>>> ????? 0.01????? 49.39 us?????? 8.08 us 1264.41 us????????? >>>>> 21719?????? FLUSH >>>>> ????? 0.03????? 74.77 us????? 20.86 us 2008.41 us????????? >>>>> 37050?????? WRITE >>>>> ????? 0.04???? 553.12 us????? 40.81 us 386485.29 us?????????? >>>>> 5901????? CREATE >>>>> ????? 0.06???? 534.01 us???? 100.86 us 527770.44 us????????? >>>>> 10057??? FXATTROP >>>>> ????? 0.12???? 138.96 us?????? 1.00 us 601208.70 us????????? >>>>> 70092???? OPENDIR >>>>> ????? 0.12???? 947.76 us????? 18.54 us 712568.33 us????????? >>>>> 10779???? READDIR >>>>> ????? 0.15??? 4099.07 us???? 124.69 us 421939.47 us?????????? >>>>> 3008?????? RMDIR >>>>> ????? 0.61???? 465.30 us????? 20.67 us 936959.86 us???????? >>>>> 109478????? UNLINK >>>>> ????? 1.57???? 182.41 us????? 49.54 us 73938.09 us???????? >>>>> 722669 REMOVEXATTR >>>>> ????? 1.61???? 171.41 us?????? 6.98 us 83809.88 us???????? >>>>> 789294??? GETXATTR >>>>> ????? 1.86???? 217.88 us????? 26.40 us 556446.19 us???????? >>>>> 718835?? FTRUNCATE >>>>> ????? 1.89???? 219.48 us????? 49.92 us 506767.78 us???????? >>>>> 722856??? SETXATTR >>>>> ????? 3.02???? 181.66 us????? 44.40 us 344656.69 us??????? >>>>> 1395957??????? OPEN >>>>> ????? 5.56???? 322.09 us????? 43.37 us 634069.61 us??????? >>>>> 1450160???? SETATTR >>>>> ????? 5.84???? 115.88 us?????? 7.67 us 747387.51 us??????? >>>>> 4233520???? ENTRYLK >>>>> ????? 6.53???? 378.33 us???? 122.84 us 766939.14 us??????? >>>>> 1449601???? XATTROP >>>>> ????? 9.75???? 115.48 us????? 10.70 us 556675.13 us??????? >>>>> 7097226???? INODELK >>>>> ???? 19.37??? 1105.06 us???? 101.28 us 1612346.75 us??????? >>>>> 1472813?????? MKNOD >>>>> ???? 41.84???? 295.54 us?????? 8.98 us 996805.54 us?????? >>>>> 11896091????? LOOKUP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 3531139????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1862470???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us??????????? 244??? CI_XATTR >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us??????????? 632?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1668669?? CI_FORGET >>>>> >>>>> ??? Duration: 208643 seconds >>>>> ?? Data Read: 0 bytes >>>>> Data Written: 37050 bytes >>>>> >>>>> Brick: bender:/mnt/gluster/a >>>>> --------------------------------------- >>>>> Cumulative Stats: >>>>> ?? Block Size: 1b+?????????????????? 2b+?????????????????? 8b+ >>>>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>>>> No. of Writes: 53???????????????????? 1??????????????????? 60 >>>>> >>>>> ?? Block Size: 16b+????????????????? 32b+????????????????? 64b+ >>>>> ?No. of Reads: 0?????????????????? 235???????????????????? 4 >>>>> No. of Writes: 14????????????????? 1033??????????????????? 15 >>>>> >>>>> ?? Block Size: 128b+???????????????? 256b+ 512b+ >>>>> ?No. of Reads: 4?????????????????? 115??????????????????? 81 >>>>> No. of Writes: 58?????????????????? 703??????????????????? 80 >>>>> >>>>> ?? Block Size: 1024b+??????????????? 2048b+ 4096b+ >>>>> ?No. of Reads: 181??????????????????? 26??????????????????? 74 >>>>> No. of Writes: 163?????????????????? 278?????????????????? 397 >>>>> >>>>> ?? Block Size: 8192b+?????????????? 16384b+ 32768b+ >>>>> ?No. of Reads: 131?????????????????? 161?????????????????? 267 >>>>> No. of Writes: 7264????????????????? 1398????????????????? 6742 >>>>> >>>>> ?? Block Size: 65536b+????????????? 131072b+ 262144b+ >>>>> ?No. of Reads: 443???????????????? 25528???????????????????? 0 >>>>> No. of Writes: 2426???????????????? 12240???????????????????? 1 >>>>> >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 684942????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 837092???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 87297? RELEASEDIR >>>>> ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 >>>>> us????????????? 8 REMOVEXATTR >>>>> ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 >>>>> us???????????? 97???????? IPC >>>>> ????? 0.00???? 152.77 us????? 78.90 us 1059.82 us???????????? >>>>> 46?? FTRUNCATE >>>>> ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 >>>>> us???????????? 65?? FGETXATTR >>>>> ????? 0.00???? 243.66 us???? 140.50 us 3392.16 us??????????? >>>>> 121??????? LINK >>>>> ????? 0.00??? 1216.47 us???? 194.37 us 23411.42 us???????????? >>>>> 25???? SYMLINK >>>>> ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 >>>>> us??????????? 107?????? MKNOD >>>>> ????? 0.00?? 36094.61 us?? 36094.61 us 36094.61 >>>>> us????????????? 1?????? FSYNC >>>>> ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 >>>>> us??????????? 188????? RENAME >>>>> ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 >>>>> us??????????? 745??????? STAT >>>>> ????? 0.00???? 531.71 us???? 113.57 us 53973.08 us??????????? >>>>> 200??? TRUNCATE >>>>> ????? 0.00???? 289.05 us????? 63.10 us 70107.41 us?????????? >>>>> 1444???? SETATTR >>>>> ????? 0.00??? 2430.15 us???? 232.78 us 68120.54 us??????????? >>>>> 177?????? MKDIR >>>>> ????? 0.00????? 52.68 us?????? 7.28 us???? 826.01 us?????????? >>>>> 9890?????? FLUSH >>>>> ????? 0.00????? 60.96 us????? 17.54 us 1393.95 us?????????? >>>>> 9792??? FINODELK >>>>> ????? 0.00????? 74.26 us????? 20.00 us 1522.39 us????????? >>>>> 21521????? STATFS >>>>> ????? 0.00???? 129.21 us????? 34.15 us 9068.67 us????????? >>>>> 13094?????? FSTAT >>>>> ????? 0.00???? 456.99 us???? 195.50 us 130182.02 us?????????? >>>>> 5967????? CREATE >>>>> ????? 0.00???? 112.06 us????? 28.14 us 9545.53 us????????? >>>>> 27474??????? READ >>>>> ????? 0.01???? 313.76 us????? 52.98 us 71635.74 us????????? >>>>> 32926?????? WRITE >>>>> ????? 0.01???? 135.84 us?????? 1.52 us 671373.08 us????????? >>>>> 87297???? OPENDIR >>>>> ????? 0.01??? 5225.28 us???? 131.28 us 951323.69 us?????????? >>>>> 3177?????? RMDIR >>>>> ????? 0.01???? 203.34 us????? 56.13 us 486756.90 us????????? >>>>> 85073??? SETXATTR >>>>> ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us????????? >>>>> 15090??? READDIRP >>>>> ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 >>>>> us??????????? 564????????? LK >>>>> ????? 0.05????? 92.18 us????? 29.71 us 39069.99 us???????? >>>>> 850938??? READLINK >>>>> ????? 0.23???? 453.79 us????? 41.84 us 655999.23 us???????? >>>>> 831212??????? OPEN >>>>> ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us????????? >>>>> 22482???? READDIR >>>>> ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us???????? >>>>> 415226????? UNLINK >>>>> ????? 0.54??? 1272.83 us?????? 9.66 us 1604368.93 us???????? >>>>> 686945??? GETXATTR >>>>> ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us??????? >>>>> 3739718???? ENTRYLK >>>>> ????? 0.60? 103888.49 us????? 89.32 us 761509.35 us?????????? >>>>> 9311??? FXATTROP >>>>> ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us??????? >>>>> 4649291???? INODELK >>>>> ????? 5.82??? 1240.20 us?????? 8.78 us 1526868.50 us??????? >>>>> 7625383????? LOOKUP >>>>> ???? 88.60?? 58005.98 us????? 67.31 us 1448810.06 us??????? >>>>> 2481914???? XATTROP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1396875????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 1854???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us???????????? 28??? CI_XATTR >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 1051?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1394996?? CI_FORGET >>>>> >>>>> ??? Duration: 208643 seconds >>>>> ?? Data Read: 3409342848 bytes >>>>> Data Written: 2180686507 bytes >>>>> >>>>> Interval 0 Stats: >>>>> ?? Block Size: 1b+?????????????????? 2b+?????????????????? 8b+ >>>>> ?No. of Reads: 0???????????????????? 0???????????????????? 0 >>>>> No. of Writes: 53???????????????????? 1??????????????????? 60 >>>>> >>>>> ?? Block Size: 16b+????????????????? 32b+????????????????? 64b+ >>>>> ?No. of Reads: 0?????????????????? 235???????????????????? 4 >>>>> No. of Writes: 14????????????????? 1033??????????????????? 15 >>>>> >>>>> ?? Block Size: 128b+???????????????? 256b+ 512b+ >>>>> ?No. of Reads: 4?????????????????? 115??????????????????? 81 >>>>> No. of Writes: 58?????????????????? 703??????????????????? 80 >>>>> >>>>> ?? Block Size: 1024b+??????????????? 2048b+ 4096b+ >>>>> ?No. of Reads: 181??????????????????? 26??????????????????? 74 >>>>> No. of Writes: 163?????????????????? 278?????????????????? 397 >>>>> >>>>> ?? Block Size: 8192b+?????????????? 16384b+ 32768b+ >>>>> ?No. of Reads: 131?????????????????? 161?????????????????? 267 >>>>> No. of Writes: 7264????????????????? 1398????????????????? 6742 >>>>> >>>>> ?? Block Size: 65536b+????????????? 131072b+ 262144b+ >>>>> ?No. of Reads: 443???????????????? 25528???????????????????? 0 >>>>> No. of Writes: 2426???????????????? 12240???????????????????? 1 >>>>> >>>>> ?%-latency?? Avg-latency?? Min-Latency Max-Latency?? No. of >>>>> calls???????? Fop >>>>> ?---------?? -----------?? ----------- -----------?? >>>>> ------------??????? ---- >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 684942????? FORGET >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us???????? >>>>> 837092???? RELEASE >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us????????? >>>>> 87297? RELEASEDIR >>>>> ????? 0.00???? 219.22 us???? 121.08 us???? 306.12 >>>>> us????????????? 8 REMOVEXATTR >>>>> ????? 0.00????? 21.70 us?????? 7.24 us???? 196.07 >>>>> us???????????? 97???????? IPC >>>>> ????? 0.00???? 152.77 us????? 78.90 us 1059.82 us???????????? >>>>> 46?? FTRUNCATE >>>>> ????? 0.00???? 295.73 us???? 167.10 us???? 606.96 >>>>> us???????????? 65?? FGETXATTR >>>>> ????? 0.00???? 243.66 us???? 140.50 us 3392.16 us??????????? >>>>> 121??????? LINK >>>>> ????? 0.00??? 1216.47 us???? 194.37 us 23411.42 us???????????? >>>>> 25???? SYMLINK >>>>> ????? 0.00???? 327.87 us???? 234.27 us???? 551.71 >>>>> us??????????? 107?????? MKNOD >>>>> ????? 0.00?? 36094.61 us?? 36094.61 us 36094.61 >>>>> us????????????? 1?????? FSYNC >>>>> ????? 0.00???? 219.63 us???? 147.34 us???? 783.10 >>>>> us??????????? 188????? RENAME >>>>> ????? 0.00???? 104.27 us????? 40.34 us???? 595.29 >>>>> us??????????? 745??????? STAT >>>>> ????? 0.00???? 531.71 us???? 113.57 us 53973.08 us??????????? >>>>> 200??? TRUNCATE >>>>> ????? 0.00???? 289.05 us????? 63.10 us 70107.41 us?????????? >>>>> 1444???? SETATTR >>>>> ????? 0.00??? 2430.15 us???? 232.78 us 68120.54 us??????????? >>>>> 177?????? MKDIR >>>>> ????? 0.00????? 52.69 us?????? 7.28 us???? 826.01 us?????????? >>>>> 9890?????? FLUSH >>>>> ????? 0.00????? 60.96 us????? 17.54 us 1393.95 us?????????? >>>>> 9792??? FINODELK >>>>> ????? 0.00????? 74.25 us????? 20.00 us 1522.39 us????????? >>>>> 21521????? STATFS >>>>> ????? 0.00???? 129.23 us????? 34.15 us 9068.67 us????????? >>>>> 13094?????? FSTAT >>>>> ????? 0.00???? 456.99 us???? 195.50 us 130182.02 us?????????? >>>>> 5967????? CREATE >>>>> ????? 0.00???? 112.45 us????? 28.14 us 9545.53 us????????? >>>>> 27474??????? READ >>>>> ????? 0.01???? 313.77 us????? 52.98 us 71635.74 us????????? >>>>> 32926?????? WRITE >>>>> ????? 0.01???? 135.84 us?????? 1.52 us 671373.08 us????????? >>>>> 87297???? OPENDIR >>>>> ????? 0.01??? 5225.28 us???? 131.28 us 951323.69 us?????????? >>>>> 3177?????? RMDIR >>>>> ????? 0.01???? 203.34 us????? 56.13 us 486756.90 us????????? >>>>> 85073??? SETXATTR >>>>> ????? 0.03??? 3521.00 us???? 145.28 us 3396306.00 us????????? >>>>> 15090??? READDIRP >>>>> ????? 0.04? 113758.50 us????? 26.72 us 1147074.44 >>>>> us??????????? 564????????? LK >>>>> ????? 0.05????? 92.18 us????? 29.71 us 39069.99 us???????? >>>>> 850938??? READLINK >>>>> ????? 0.23???? 453.79 us????? 41.84 us 655999.23 us???????? >>>>> 831212??????? OPEN >>>>> ????? 0.41?? 29425.34 us????? 19.20 us 1526187.08 us????????? >>>>> 22482???? READDIR >>>>> ????? 0.50??? 1954.70 us????? 21.51 us 1526772.28 us???????? >>>>> 415226????? UNLINK >>>>> ????? 0.54??? 1272.79 us?????? 9.66 us 1604368.93 us???????? >>>>> 686945??? GETXATTR >>>>> ????? 0.55???? 237.57 us?????? 7.91 us 1277047.91 us??????? >>>>> 3739718???? ENTRYLK >>>>> ????? 0.60? 103888.49 us????? 89.32 us 761509.35 us?????????? >>>>> 9311??? FXATTROP >>>>> ????? 2.60???? 909.04 us?????? 9.68 us 2272778.49 us??????? >>>>> 4649291???? INODELK >>>>> ????? 5.82??? 1240.10 us?????? 8.78 us 1526868.50 us??????? >>>>> 7625383????? LOOKUP >>>>> ???? 88.60?? 58006.02 us????? 67.31 us 1448810.06 us??????? >>>>> 2481914???? XATTROP >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1396875????? UPCALL >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 1854???? CI_IATT >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 >>>>> us???????????? 28??? CI_XATTR >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us?????????? >>>>> 1051?? CI_UNLINK >>>>> ????? 0.00?????? 0.00 us?????? 0.00 us?????? 0.00 us??????? >>>>> 1394996?? CI_FORGET >>>>> >>>>> ??? Duration: 208643 seconds >>>>> ?? Data Read: 3409342848 bytes >>>>> Data Written: 2180686507 bytes >>>>> >>>>> >>>>> ________ >>>>> >>>>> >>>>> >>>>> Community Meeting Calendar: >>>>> >>>>> Schedule - >>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>>>> Bridge: https://bluejeans.com/441850968 >>>>> >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>> >>> ________ >>> >>> >>> >>> Community Meeting Calendar: >>> >>> Schedule - >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>> Bridge:https://bluejeans.com/441850968 >>> >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200508/f6a1f7af/attachment-0001.html>