Hi all,
I'm running a 3 node gluster cluster in AWS (m4.xlarge, 4 x st1 500G disks)
and just starting out.
glusterfs 3.7.11 built on Apr 18 2016 12:57:30 --servers
glusterfs 3.7.13 built on Jul 8 2016 15:46:50 -- fuse clients
Volume Name: marketplace_nfs
Type: Distributed-Replicate
Volume ID: 261f1e1e-49c1-4e93-b7a5-082590fd606f
Status: Started
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: 10.90.3.4:/data/data1/marketplace_nfs
Brick2: 10.90.4.82:/data/data1/marketplace_nfs
Brick3: 10.90.5.26:/data/data1/marketplace_nfs
Brick4: 10.90.3.4:/data/data0/marketplace_nfs
Brick5: 10.90.4.82:/data/data0/marketplace_nfs
Brick6: 10.90.5.26:/data/data0/marketplace_nfs
Brick7: 10.90.3.4:/data/data3/marketplace_nfs
Brick8: 10.90.4.82:/data/data3/marketplace_nfs
Brick9: 10.90.5.26:/data/data3/marketplace_nfs
Brick10: 10.90.3.4:/data/data2/marketplace_nfs
Brick11: 10.90.4.82:/data/data2/marketplace_nfs
Brick12: 10.90.5.26:/data/data2/marketplace_nfs
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
network.ping-timeout: 15
cluster.quorum-type: none
performance.readdir-ahead: on
The idea is that at least 2 nodes have a copy of the data in the volume.
I mount this on the client
10.90.3.4:/marketplace_nfs on /data type fuse.glusterfs
(rw,default_permissions,allow_other,max_read=131072)
(10.90.3.4:/marketplace_nfs /data glusterfs
backupvolfile-server=nfs.market.customer.envato.internal 0 0)
When I execute on a fuse mounted client:
$ time find ./* -maxdepth 1 -type d |wc -l
6219
real 10m42.753s
user 0m0.730s
sys 0m0.467s
Locally on one of the gluster nodes (over a larger number of files).
time find /data/data*/marketplace_nfs/ -maxdepth 1 -type d |wc -l
24908
real 0m0.202s
user 0m0.058s
sys 0m0.126s
I am not sure of the performance I can reasonably expect. The question I
have is is this reasonable and if not, how can I improve performance?
Should I reduce the number of bricks per node? Should I increase the
replica count? Is there a fundamental problem in my configuration?
Any help appreciated,
Den
further information:
Each gluster node has similar top:
top - 03:19:57 up 7 days, 22:19, 1 user, load average: 12.84, 9.13, 6.56
Tasks: 142 total, 2 running, 140 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.4 us, 0.3 sy, 0.0 ni, 99.2 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
KiB Mem : 16431672 total, 239224 free, 644316 used, 15548132 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 15365572 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3782 root 20 0 2002532 83264 7736 S 206.2 0.5 101:34.25
/usr/sbin/glusterfsd -s 10.90.3.4 --volfile-id
marketplace_nfs.10.90.3.4.data-data1-marketplace_nfs -p
/var/lib/glusterd/vols/marketplace_nfs/run/10.90.3.4-data-data1-marketplace_nfs.pid
-S /var/run/gluster/aef8f955e6f50664936da6eee0158803.socket --brick-name
/data/data1/marketplace_nfs -l /va+
3820 root 20 0 2068064 83124 8120 S 175.0 0.5 99:59.40
/usr/sbin/glusterfsd -s 10.90.3.4 --volfile-id
marketplace_nfs.10.90.3.4.data-data3-marketplace_nfs -p
/var/lib/glusterd/vols/marketplace_nfs/run/10.90.3.4-data-data3-marketplace_nfs.pid
-S /var/run/gluster/a4ab633dc313d2e6fd8e53a37c77c30b.socket --brick-name
/data/data3/marketplace_nfs -l /va+
3801 root 20 0 1335868 72172 8128 S 6.2 0.4 9:12.61
/usr/sbin/glusterfsd -s 10.90.3.4 --volfile-id
marketplace_nfs.10.90.3.4.data-data0-marketplace_nfs -p
/var/lib/glusterd/vols/marketplace_nfs/run/10.90.3.4-data-data0-marketplace_nfs.pid
-S /var/run/gluster/ff9b43a6f3ea1b1ec0c04b9aee936dc8.socket --brick-name
/data/data0/marketplace_nfs -l /va+
1 root 20 0 128556 8576 5736 S 0.0 0.1 0:06.74
/usr/lib/systemd/systemd --switched-root --system --deserialize 21
2 root 20 0 0 0 0 S 0.0 0.0 0:00.10
[kthreadd]
Each glusterfs server is showing this behaviour, massive network sends and
little disk usage:
---total-cpu-usage---- --dsk/xvdg----dsk/xvdh----dsk/xvdi----dsk/xvdj-
-net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ: read writ: read writ: read writ|
recv send| in out | int csw
24 74 2 0 0 0| 0 0 : 0 1024B: 0 0 : 0 0 |
129k 14M| 0 0 |6533 6308
25 74 1 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
160k 14M| 0 0 |6973 6669
24 74 2 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
161k 15M| 0 0 |7300 7562
25 74 2 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
149k 14M| 0 0 |6642 6528
31 68 1 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
124k 14M| 0 0 |6683 6343
24 64 12 0 0 0| 0 0 : 0 0 : 0 0 : 0 1024B|
142k 13M| 0 0 |6800 6939
25 73 1 0 0 1| 0 0 : 0 0 : 0 0 : 0 0 |
154k 14M| 0 0 |7054 6484
26 70 4 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
132k 14M| 0 0 |6803 6539
25 69 5 0 0 1| 0 0 : 0 0 : 0 0 : 0 0 |
160k 14M| 0 0 |7323 9557
24 70 5 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
143k 14M| 0 0 |6769 6935
25 73 2 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
137k 14M| 0 0 |6976 6257
29 70 0 0 0 0| 0 1024B: 0 0 : 0 0 : 0 0 |
171k 14M| 0 0 |6836 5853
27 70 3 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
179k 14M| 0 0 |7081 7359
22 62 16 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
133k 13M| 0 0 |6499 6478
15 36 48 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
91k 8798k| 0 0 |4686 3832
15 44 41 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
132k 10M| 0 0 |5708 5834
19 53 28 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
140k 12M| 0 0 |6471 5599
27 70 3 0 0 1| 0 0 : 0 0 : 0 0 : 0 0 |
151k 13M| 0 0 |7134 7197
24 74 2 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
159k 14M| 0 0 |7682 8206
28 69 3 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
134k 13M| 0 0 |7194 8798
29 70 1 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
72k 14M| 0 0 |6365 6426
15 44 42 0 0 0| 0 0 : 0 0 : 0 0 : 0 0 |
61k 11M| 0 0 |4929 6895
0 0 100 0 0 0| 0 0 : 0 0 : 0 1024B: 0 0 |
25k 15k| 0 0 |1121 1259
0 0 100 0 0 0| 0 0 : 0 0 : 0 0 : 0 0
|4268B 3186B| 0 0 | 458 562
The profile info is:
$ cat /tmp/dataout1
Brick: 10.90.3.4:/data/data2/marketplace_nfs
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+
128b+
No. of Reads: 0 0
0
No. of Writes: 15 1
2
Block Size: 256b+ 512b+
1024b+
No. of Reads: 0 0
0
No. of Writes: 4 5
20
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 0 0
0
No. of Writes: 50 67
114
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 0 1
0
No. of Writes: 273 391
831
Block Size: 131072b+
No. of Reads: 14301
No. of Writes: 397404
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8687
FORGET
0.00 0.00 us 0.00 us 0.00 us 1800
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199314
RELEASEDIR
0.00 30.00 us 26.00 us 34.00 us 3
STATFS
0.01 121.00 us 94.00 us 155.00 us 9
UNLINK
0.01 31.45 us 14.00 us 51.00 us 44
READDIR
0.03 106.95 us 13.00 us 618.00 us 60
GETXATTR
0.08 32.99 us 16.00 us 1321.00 us 470
STAT
3.68 40.79 us 19.00 us 588.00 us 16548
INODELK
5.16 114.49 us 60.00 us 8674.00 us 8274
RMDIR
7.20 39.89 us 8.00 us 1114.00 us 33114
ENTRYLK
17.10 49.37 us 1.00 us 9456.00 us 63574
OPENDIR
31.39 115.54 us 15.00 us 4769.00 us 49862
LOOKUP
35.33 123.04 us 23.00 us 7915.00 us 52693
READDIRP
Duration: 684652 seconds
Data Read: 1874518431 bytes
Data Written: 52198234474 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.06 23.40 us 23.00 us 24.00 us 5
STAT
23.14 45.95 us 29.00 us 3089.00 us 929
OPENDIR
34.89 115.37 us 25.00 us 301.00 us 558
READDIRP
41.91 116.09 us 75.00 us 1106.00 us 666
LOOKUP
Duration: 58 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.3.4:/data/data0/marketplace_nfs
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+
128b+
No. of Reads: 0 0
0
No. of Writes: 19 1
1
Block Size: 256b+ 512b+
1024b+
No. of Reads: 0 0
0
No. of Writes: 3 8
15
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 0 0
0
No. of Writes: 55 71
92
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 0 1
1
No. of Writes: 274 413
838
Block Size: 131072b+
No. of Reads: 3
No. of Writes: 246923
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8688
FORGET
0.00 0.00 us 0.00 us 0.00 us 1792
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199385
RELEASEDIR
0.00 35.67 us 32.00 us 38.00 us 3
STATFS
0.01 31.00 us 17.00 us 56.00 us 44
READDIR
0.01 178.92 us 94.00 us 632.00 us 13
UNLINK
0.03 119.72 us 13.00 us 597.00 us 60
GETXATTR
0.08 37.79 us 18.00 us 1467.00 us 470
STAT
3.29 42.40 us 19.00 us 624.00 us 16548
INODELK
5.06 130.39 us 58.00 us 12961.00 us 8274
RMDIR
7.53 48.42 us 9.00 us 2416.00 us 33122
ENTRYLK
19.89 66.66 us 1.00 us 8525.00 us 63574
OPENDIR
31.02 125.13 us 24.00 us 6020.00 us 52812
READDIRP
33.08 140.87 us 16.00 us 8952.00 us 50033
LOOKUP
Duration: 684652 seconds
Data Read: 558591 bytes
Data Written: 32474329315 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.06 27.60 us 21.00 us 37.00 us 5
STAT
30.85 76.03 us 29.00 us 4056.00 us 929
OPENDIR
31.14 130.61 us 29.00 us 3018.00 us 546
READDIRP
37.95 130.47 us 52.00 us 1154.00 us 666
LOOKUP
Duration: 58 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.3.4:/data/data1/marketplace_nfs
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 128b+
256b+
No. of Reads: 0 0
0
No. of Writes: 11 3
2
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 0 0
0
No. of Writes: 5 6
47
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 0 0
0
No. of Writes: 62 100
263
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 0 3
168
No. of Writes: 412 838
334487
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8657
FORGET
0.00 0.00 us 0.00 us 0.00 us 1779
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199385
RELEASEDIR
0.00 54.33 us 42.00 us 66.00 us 3
STATFS
0.00 29.43 us 15.00 us 44.00 us 44
READDIR
0.00 5174.50 us 81.00 us 33257.00 us 8
UNLINK
0.01 44.76 us 19.00 us 26107.00 us 16548
INODELK
0.01 131.59 us 59.00 us 10776.00 us 8274
RMDIR
0.01 48.54 us 9.00 us 28337.00 us 33112
ENTRYLK
0.04 4452.26 us 18.00 us 132302.00 us 1263
STAT
0.24 548915.85 us 13.00 us 5224650.00 us 60
GETXATTR
0.93 2414.17 us 16.00 us 203291.00 us 51699
LOOKUP
1.48 3129.09 us 1.00 us 228575.00 us 63574
OPENDIR
97.28 68862.50 us 17.00 us 460340.00 us 189998
READDIRP
Duration: 684653 seconds
Data Read: 22303103 bytes
Data Written: 43952561838 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.02 5927.05 us 25.00 us 75030.00 us 20
STAT
0.73 5543.27 us 78.00 us 141158.00 us 668
LOOKUP
1.29 7060.32 us 27.00 us 144956.00 us 930
OPENDIR
97.96 105543.88 us 25.00 us 384946.00 us 4722
READDIRP
Duration: 58 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.3.4:/data/data3/marketplace_nfs
--------------------------------------------
Cumulative Stats:
Block Size: 1b+ 128b+
256b+
No. of Reads: 0 0
0
No. of Writes: 8 1
3
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 0 0
0
No. of Writes: 7 21
44
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 0 0
0
No. of Writes: 62 104
241
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 2 1
6
No. of Writes: 464 791
285704
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8659
FORGET
0.00 0.00 us 0.00 us 0.00 us 1762
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199385
RELEASEDIR
0.00 38.33 us 30.00 us 52.00 us 3
STATFS
0.00 31.25 us 15.00 us 111.00 us 44
READDIR
0.00 1703.00 us 97.00 us 15797.00 us 14
UNLINK
0.00 42.25 us 19.00 us 13149.00 us 16548
INODELK
0.01 121.61 us 61.00 us 4492.00 us 8274
RMDIR
0.01 40.81 us 9.00 us 5622.00 us 33124
ENTRYLK
0.05 123021.10 us 13.00 us 2265444.00 us 60
GETXATTR
0.12 13702.60 us 17.00 us 296082.00 us 1263
STAT
1.16 3452.46 us 15.00 us 272058.00 us 50064
LOOKUP
1.45 3379.65 us 1.00 us 325140.00 us 63574
OPENDIR
97.20 75905.73 us 21.00 us 533385.00 us 190235
READDIRP
Duration: 684652 seconds
Data Read: 974227 bytes
Data Written: 37555843577 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.08 20941.70 us 20.00 us 179031.00 us 20
STAT
0.98 5891.36 us 27.00 us 180270.00 us 929
OPENDIR
1.15 9582.95 us 81.00 us 194474.00 us 666
LOOKUP
97.79 113708.24 us 22.00 us 424985.00 us 4784
READDIRP
Duration: 58 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.4.82:/data/data1/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 128b+
256b+
No. of Reads: 0 0
0
No. of Writes: 11 3
2
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 0 0
0
No. of Writes: 5 6
47
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 0 0
0
No. of Writes: 62 100
263
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 0 2
43
No. of Writes: 412 838
334487
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8662
FORGET
0.00 0.00 us 0.00 us 0.00 us 1779
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199383
RELEASEDIR
0.00 56.33 us 35.00 us 67.00 us 3
STATFS
0.01 155.88 us 112.00 us 235.00 us 8
UNLINK
0.01 30.86 us 16.00 us 61.00 us 44
READDIR
0.03 120.55 us 13.00 us 343.00 us 60
GETXATTR
0.10 45.39 us 22.00 us 2903.00 us 470
STAT
3.27 41.38 us 20.00 us 1282.00 us 16548
INODELK
4.04 102.23 us 60.00 us 6847.00 us 8274
RMDIR
5.91 37.39 us 9.00 us 1601.00 us 33112
ENTRYLK
21.07 69.38 us 37.00 us 9144.00 us 63574
OPENDIR
30.59 121.72 us 36.00 us 7740.00 us 52616
READDIRP
34.97 141.63 us 87.00 us 5463.00 us 51699
LOOKUP
Duration: 684653 seconds
Data Read: 5784405 bytes
Data Written: 43952561838 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.04 22.60 us 22.00 us 43.00 us 5
STAT
27.35 131.09 us 36.00 us 2312.00 us 554
READDIRP
31.73 90.78 us 37.00 us 9144.00 us 928
OPENDIR
40.88 162.48 us 87.00 us 4127.00 us 668
LOOKUP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.4.82:/data/data0/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+
128b+
No. of Reads: 0 0
0
No. of Writes: 11 1
1
Block Size: 256b+ 512b+
1024b+
No. of Reads: 0 0
0
No. of Writes: 7 10
15
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 0 0
2
No. of Writes: 46 63
120
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 0 0
0
No. of Writes: 248 438
866
Block Size: 131072b+
No. of Reads: 171
No. of Writes: 350568
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8702
FORGET
0.00 0.00 us 0.00 us 0.00 us 1862
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199383
RELEASEDIR
0.00 33.45 us 15.00 us 66.00 us 44
READDIR
0.00 1804.00 us 43.00 us 5262.00 us 3
STATFS
0.00 678.89 us 90.00 us 5115.00 us 9
UNLINK
0.00 465.45 us 13.00 us 11085.00 us 60
GETXATTR
0.01 45.88 us 19.00 us 11892.00 us 16548
INODELK
0.01 105.35 us 63.00 us 2869.00 us 8274
RMDIR
0.02 38.91 us 9.00 us 18766.00 us 33114
ENTRYLK
0.10 6311.11 us 30.00 us 230926.00 us 1264
STAT
0.96 1596.82 us 101.00 us 304352.00 us 50059
LOOKUP
1.11 1456.50 us 30.00 us 309713.00 us 63574
OPENDIR
97.81 43329.13 us 27.00 us 1075779.00 us 188746
READDIRP
Duration: 684653 seconds
Data Read: 22434084 bytes
Data Written: 46064215951 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.03 2744.90 us 30.00 us 49925.00 us 20
STAT
0.66 2076.87 us 101.00 us 149921.00 us 666
LOOKUP
0.83 1897.86 us 30.00 us 204514.00 us 928
OPENDIR
98.48 44524.85 us 27.00 us 1075779.00 us 4669
READDIRP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.4.82:/data/data3/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 128b+
256b+
No. of Reads: 0 0
0
No. of Writes: 8 1
3
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 0 0
0
No. of Writes: 7 21
44
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 0 1
0
No. of Writes: 62 104
241
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 0 0
2099
No. of Writes: 464 791
285704
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8665
FORGET
0.00 0.00 us 0.00 us 0.00 us 1762
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199383
RELEASEDIR
0.00 28.00 us 24.00 us 31.00 us 3
STATFS
0.01 32.00 us 16.00 us 61.00 us 44
READDIR
0.01 110.00 us 84.00 us 138.00 us 14
UNLINK
0.04 117.45 us 13.00 us 636.00 us 60
GETXATTR
0.09 37.84 us 58.00 us 2402.00 us 470
STAT
3.58 40.91 us 19.00 us 3205.00 us 16548
INODELK
4.53 103.60 us 59.00 us 4561.00 us 8274
RMDIR
6.06 34.60 us 9.00 us 1404.00 us 33124
ENTRYLK
18.68 55.57 us 33.00 us 5689.00 us 63575
OPENDIR
32.75 117.30 us 30.00 us 6893.00 us 52807
READDIRP
34.25 129.40 us 103.00 us 5439.00 us 50064
LOOKUP
Duration: 684653 seconds
Data Read: 275135247 bytes
Data Written: 37555843577 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.06 27.80 us 58.00 us 58.00 us 5
STAT
28.17 70.89 us 33.00 us 4767.00 us 928
OPENDIR
30.52 126.15 us 30.00 us 3003.00 us 565
READDIRP
41.25 144.62 us 103.00 us 2822.00 us 666
LOOKUP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.4.82:/data/data2/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 16b+
64b+
No. of Reads: 0 0
0
No. of Writes: 12 1
4
Block Size: 128b+ 256b+
512b+
No. of Reads: 0 0
0
No. of Writes: 1 3
7
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 0 0
0
No. of Writes: 14 43
68
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 0 1
0
No. of Writes: 114 263
419
Block Size: 65536b+ 131072b+
No. of Reads: 2 352
No. of Writes: 824 322197
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8705
FORGET
0.00 0.00 us 0.00 us 0.00 us 1807
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199308
RELEASEDIR
0.00 29.33 us 22.00 us 36.00 us 3
STATFS
0.00 68.50 us 47.00 us 90.00 us 2
READLINK
0.00 28.57 us 14.00 us 46.00 us 44
READDIR
0.00 344.00 us 10.00 us 4695.00 us 60
GETXATTR
0.00 1442.17 us 93.00 us 8467.00 us 18
UNLINK
0.01 100.33 us 60.00 us 8658.00 us 8274
RMDIR
0.01 70.57 us 20.00 us 485119.00 us 16548
INODELK
0.03 86.06 us 9.00 us 889356.00 us 33132
ENTRYLK
0.12 9973.16 us 24.00 us 403909.00 us 1264
STAT
1.06 2120.37 us 89.00 us 411175.00 us 50520
LOOKUP
1.20 1900.07 us 30.00 us 374073.00 us 63575
OPENDIR
97.57 52219.09 us 28.00 us 1051061.00 us 188703
READDIRP
Duration: 684653 seconds
Data Read: 46382302 bytes
Data Written: 42340686672 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.06 6935.55 us 24.00 us 135183.00 us 20
STAT
0.37 1290.03 us 89.00 us 168901.00 us 668
LOOKUP
0.60 1503.80 us 30.00 us 185597.00 us 929
OPENDIR
98.97 49757.74 us 28.00 us 1051061.00 us 4625
READDIRP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.5.26:/data/data0/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+
128b+
No. of Reads: 0 0
0
No. of Writes: 11 1
1
Block Size: 256b+ 512b+
1024b+
No. of Reads: 0 0
0
No. of Writes: 7 10
15
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 0 1
0
No. of Writes: 46 63
120
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 2 0
1
No. of Writes: 248 438
866
Block Size: 131072b+
No. of Reads: 905
No. of Writes: 350568
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8710
FORGET
0.00 0.00 us 0.00 us 0.00 us 1862
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199383
RELEASEDIR
0.00 109.67 us 32.00 us 238.00 us 3
STATFS
0.01 37.95 us 20.00 us 82.00 us 44
READDIR
0.02 407.44 us 95.00 us 2742.00 us 9
UNLINK
0.03 114.35 us 17.00 us 417.00 us 60
GETXATTR
0.10 45.22 us 18.00 us 1975.00 us 470
STAT
3.23 39.57 us 17.00 us 512.00 us 16548
INODELK
4.09 100.09 us 61.00 us 5656.00 us 8274
RMDIR
6.11 37.33 us 9.00 us 1217.00 us 33114
ENTRYLK
20.72 65.99 us 1.00 us 6995.00 us 63574
OPENDIR
31.05 119.10 us 24.00 us 7180.00 us 52780
READDIRP
34.64 140.09 us 22.00 us 7989.00 us 50059
LOOKUP
Duration: 684616 seconds
Data Read: 118761609 bytes
Data Written: 46064215951 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.45 229.20 us 22.00 us 1051.00 us 5
STAT
29.27 133.60 us 25.00 us 5005.00 us 558
READDIRP
29.65 81.38 us 30.00 us 3405.00 us 928
OPENDIR
40.63 155.39 us 67.00 us 2723.00 us 666
LOOKUP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.5.26:/data/data2/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 16b+
64b+
No. of Reads: 0 0
0
No. of Writes: 12 1
4
Block Size: 128b+ 256b+
512b+
No. of Reads: 0 0
0
No. of Writes: 1 3
7
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 0 0
1
No. of Writes: 14 43
68
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 0 0
1
No. of Writes: 114 263
419
Block Size: 65536b+ 131072b+
No. of Reads: 1 10072
No. of Writes: 824 322197
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8714
FORGET
0.00 0.00 us 0.00 us 0.00 us 1807
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199308
RELEASEDIR
0.00 44.33 us 25.00 us 73.00 us 3
STATFS
0.01 35.36 us 17.00 us 64.00 us 44
READDIR
0.07 204.87 us 14.00 us 3290.00 us 60
GETXATTR
0.07 706.28 us 98.00 us 10710.00 us 18
UNLINK
0.09 34.82 us 18.00 us 2003.00 us 470
STAT
2.80 53.16 us 24.00 us 5770.00 us 9488
READLINK
3.50 38.14 us 19.00 us 2399.00 us 16548
INODELK
4.40 95.87 us 54.00 us 10825.00 us 8274
RMDIR
5.82 31.65 us 8.00 us 2535.00 us 33132
ENTRYLK
17.28 49.00 us 1.00 us 4676.00 us 63574
OPENDIR
32.61 116.38 us 21.00 us 8548.00 us 50519
LOOKUP
33.36 114.35 us 25.00 us 8772.00 us 52588
READDIRP
Duration: 684615 seconds
Data Read: 1320327387 bytes
Data Written: 42340686672 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.05 45.50 us 45.00 us 46.00 us 2
READLINK
0.07 24.20 us 21.00 us 28.00 us 5
STAT
21.85 43.76 us 28.00 us 829.00 us 928
OPENDIR
35.71 121.53 us 25.00 us 2901.00 us 546
READDIRP
42.33 117.76 us 55.00 us 296.00 us 668
LOOKUP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.5.26:/data/data1/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+
128b+
No. of Reads: 0 0
0
No. of Writes: 19 1
1
Block Size: 256b+ 512b+
1024b+
No. of Reads: 0 0
0
No. of Writes: 3 8
15
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 0 0
0
No. of Writes: 55 71
92
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 2 0
4
No. of Writes: 274 413
838
Block Size: 131072b+
No. of Reads: 15212
No. of Writes: 246923
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8683
FORGET
0.00 0.00 us 0.00 us 0.00 us 1792
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199383
RELEASEDIR
0.00 73.67 us 24.00 us 152.00 us 3
STATFS
0.00 28.59 us 14.00 us 55.00 us 44
READDIR
0.00 179.22 us 11.00 us 2137.00 us 60
GETXATTR
0.00 1016.15 us 93.00 us 8833.00 us 13
UNLINK
0.01 43.64 us 20.00 us 8567.00 us 16548
INODELK
0.01 105.54 us 60.00 us 18316.00 us 8274
RMDIR
0.02 37.47 us 10.00 us 2952.00 us 33122
ENTRYLK
0.07 3915.24 us 17.00 us 210047.00 us 1263
STAT
0.76 1109.99 us 19.00 us 206017.00 us 50033
LOOKUP
0.99 1141.17 us 1.00 us 240750.00 us 63574
OPENDIR
98.15 40529.31 us 23.00 us 1009517.00 us 177968
READDIRP
Duration: 684616 seconds
Data Read: 1994272237 bytes
Data Written: 32474329315 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.18 16199.70 us 21.00 us 169000.00 us 20
STAT
0.45 1231.90 us 57.00 us 80090.00 us 666
LOOKUP
0.51 990.99 us 28.00 us 117563.00 us 928
OPENDIR
98.86 42375.20 us 27.00 us 425098.00 us 4223
READDIRP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 10.90.5.26:/data/data3/marketplace_nfs
---------------------------------------------
Cumulative Stats:
Block Size: 1b+ 64b+
128b+
No. of Reads: 0 0
0
No. of Writes: 15 1
2
Block Size: 256b+ 512b+
1024b+
No. of Reads: 0 0
0
No. of Writes: 4 5
20
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 0 0
0
No. of Writes: 50 67
114
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 2 1
1
No. of Writes: 273 391
831
Block Size: 131072b+
No. of Reads: 14324
No. of Writes: 397404
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8669
FORGET
0.00 0.00 us 0.00 us 0.00 us 1800
RELEASE
0.00 0.00 us 0.00 us 0.00 us 199389
RELEASEDIR
0.00 47.00 us 30.00 us 79.00 us 3
STATFS
0.00 171.32 us 15.00 us 6207.00 us 44
READDIR
0.00 463.12 us 13.00 us 7289.00 us 60
GETXATTR
0.00 3234.22 us 99.00 us 24983.00 us 9
UNLINK
0.01 45.34 us 19.00 us 21238.00 us 16548
INODELK
0.01 104.78 us 54.00 us 15752.00 us 8274
RMDIR
0.01 38.32 us 9.00 us 22041.00 us 33114
ENTRYLK
0.14 11662.03 us 15.00 us 313625.00 us 1263
STAT
1.07 2314.18 us 29.00 us 288612.00 us 49863
LOOKUP
1.30 2209.79 us 1.00 us 363556.00 us 63573
OPENDIR
97.46 55158.65 us 25.00 us 602484.00 us 190328
READDIRP
Duration: 684616 seconds
Data Read: 1877692288 bytes
Data Written: 52198234474 bytes
Interval 9 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 926
RELEASEDIR
0.16 18884.20 us 21.00 us 172352.00 us 20
STAT
0.61 1582.79 us 27.00 us 232040.00 us 927
OPENDIR
0.75 2693.21 us 58.00 us 212646.00 us 666
LOOKUP
98.49 50363.67 us 26.00 us 438706.00 us 4708
READDIRP
Duration: 57 seconds
Data Read: 0 bytes
Data Written: 0 bytes
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160712/fdb9042a/attachment.html>