Hi,
we have set up a replicated gluster volume on 3 glusterservers, 2 of them act as
well as gluster clients using NFS-Mode.
It seems we have a performance problem especially in read access, compared by
Iozone-Test for GlusterFS and local filesystem (testfiles attached)
Does anyone no what FINODELK in the profiling status means?
gluster> volume info var_customers
Volume Name: var_customers
Type: Replicate
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: www1.example.com:/export/var_customers
Brick2: www2.example.com:/export/var_customers
Brick3: db1.example.com:/export/var_customers
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
nfs.addr-namelookup: off
performance.cache-size: 1GB
performance.cache-refresh-timeout: 10
nfs.disable: off
gluster> volume profile var_customers info
Brick: www1.example.com:/export/var_customers
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 1 470 51
No. of Writes: 3 2096 70
Block Size: 8b+ 16b+ 32b+
No. of Reads: 257 328 1909
No. of Writes: 56539 29510 21106
Block Size: 64b+ 128b+ 256b+
No. of Reads: 4392 3989 53665
No. of Writes: 7799 3798244 68347
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 48922 30482 57730
No. of Writes: 101337 77721 146879
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 125754 85601 85585
No. of Writes: 83477 57299 100724
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 215287 402116 16703
No. of Writes: 128264 651165 13
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 808254 FORGET
0.00 0.00 us 0.00 us 0.00 us 4232835 RELEASE
0.00 0.00 us 0.00 us 0.00 us 43827826 RELEASEDIR
0.00 30.00 us 30.00 us 30.00 us 1 FTRUNCATE
0.00 101.00 us 101.00 us 101.00 us 1 MKNOD
0.00 27.80 us 19.00 us 51.00 us 5 LK
0.00 64.33 us 24.00 us 132.00 us 3 SETXATTR
0.00 129.00 us 122.00 us 136.00 us 2 GETXATTR
0.00 32.66 us 21.00 us 72.00 us 232 READLINK
0.00 365.62 us 78.00 us 12105.00 us 50 SYMLINK
0.00 101.06 us 26.00 us 16858.00 us 55237 RMDIR
0.00 4.76 us 1.00 us 127434.00 us 1201187 INODELK
0.00 4.88 us 1.00 us 323110.00 us 1197308 ENTRYLK
0.00 126.49 us 33.00 us 94811.00 us 57339 TRUNCATE
0.00 173.33 us 22.00 us 1838409.00 us 75148 UNLINK
0.00 119.27 us 32.00 us 13839.00 us 115214 XATTROP
0.00 166.44 us 10.00 us 650082.00 us 90531 RENAME
0.00 304.93 us 121.00 us 1688.00 us 60335 STATFS
0.00 17.13 us 2.00 us 17742.00 us 1184450 FLUSH
0.00 195.37 us 7.00 us 537283.00 us 126802 MKDIR
0.00 200.94 us 9.00 us 1301745.00 us 160031 CREATE
0.00 23.13 us 4.00 us 256994.00 us 2181212 FSTAT
0.00 128.58 us 16.00 us 111526.00 us 497584 SETATTR
0.00 45.61 us 12.00 us 284287.00 us 1577385 OPEN
0.00 88.07 us 13.00 us 237483.00 us 885547 WRITE
0.00 338.71 us 5.00 us 2026980.00 us 492461 READDIR
0.00 139.54 us 17.00 us 1063367.00 us 1686287 FXATTROP
0.00 43.84 us 6.00 us 775509.00 us 11817763 OPENDIR
0.00 2492.07 us 6.00 us 2104998.00 us 228619 READ
0.00 1051.69 us 39.00 us 1728976.00 us 702295 FSYNC
0.01 34.73 us 5.00 us 1927494.00 us 60153264 STAT
0.01 31.48 us 2.00 us 1906788.00 us 96888660 ACCESS
0.01 783.05 us 4.00 us 31758218.00 us 4095330 READDIRP
0.20 79.04 us 10.00 us 2517754.00 us 601250517 LOOKUP
99.76 4941107.88 us 1.00 us 308307554.00 us 4853874
FINODELK
Duration : 5434814 seconds
Data Read : 42004988515 bytes
Data Written : 53775093991 bytes
Brick: www1.example.com:/export/var_customers
Interval 63 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 1 2 3
No. of Writes: 0 0 0
Block Size: 16b+ 32b+ 64b+
No. of Reads: 10 41 125
No. of Writes: 309 173 34
Block Size: 128b+ 256b+ 512b+
No. of Reads: 122 949 2738
No. of Writes: 2109 2585 4951
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2047 3770 6424
No. of Writes: 1459 673 661
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 6744 6530 8956
No. of Writes: 200 168 217
Block Size: 65536b+
No. of Reads: 28825
No. of Writes: 45193
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6136 FORGET
0.00 0.00 us 0.00 us 0.00 us 104031 RELEASE
0.00 0.00 us 0.00 us 0.00 us 914132 RELEASEDIR
0.00 5.27 us 2.00 us 9.00 us 22 ENTRYLK
0.00 157.00 us 157.00 us 157.00 us 1 UNLINK
0.00 38.04 us 8.00 us 547.00 us 26 FSTAT
0.00 5.23 us 1.00 us 44.00 us 252 INODELK
0.00 169.50 us 124.00 us 234.00 us 10 CREATE
0.00 70.59 us 30.00 us 156.00 us 69 SETATTR
0.00 15.14 us 4.00 us 73.00 us 348 FLUSH
0.01 76.03 us 29.00 us 155.00 us 89 WRITE
0.01 145.22 us 77.00 us 627.00 us 76 XATTROP
0.02 138.39 us 2.00 us 11687.00 us 190 FINODELK
0.02 161.03 us 44.00 us 9066.00 us 178 FXATTROP
0.03 64.33 us 23.00 us 167.00 us 546 OPEN
0.06 2113.37 us 58.00 us 18096.00 us 38 TRUNCATE
0.07 1020.89 us 513.00 us 6723.00 us 89 FSYNC
0.10 45.80 us 11.00 us 262.00 us 2940 OPENDIR
0.30 1744.12 us 11.00 us 23006.00 us 219 READ
0.42 40.12 us 7.00 us 2013.00 us 13427 STAT
0.69 1309.23 us 8.00 us 54238.00 us 675 READDIR
3.33 22.93 us 3.00 us 11924.00 us 186953 ACCESS
10.34 2915.11 us 8.00 us 27456.00 us 4566 READDIRP
84.59 103.58 us 14.00 us 51079.00 us 1050757 LOOKUP
Duration : 78200 seconds
Data Read : 2517835518 bytes
Data Written : 2991808746 bytes
Iozone-Test
iozone-localfile.txt - Iozone Test on local filesystem
iozone-glusterfile.txt - Iozone GlusterFS
Hope someone can help us.
Thank you very much
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120509/2f282e9f/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: iozone-localfile.txt
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120509/2f282e9f/attachment.txt>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120509/2f282e9f/attachment-0001.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: iozone-glusterfile.txt
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120509/2f282e9f/attachment-0001.txt>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120509/2f282e9f/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4897 bytes
Desc: not available
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120509/2f282e9f/attachment.p7s>
On Wednesday 09 May 2012 11:01:00 Jens Nauber | tyclipso. net wrote:> we have set up a replicated gluster volume on 3 glusterservers, 2 of them > act as well as gluster clients using NFS-Mode. > > It seems we have a performance problem especially in read access, compared > by Iozone-Test for GlusterFS and local filesystem (testfiles attached)Comparing a network-filesystem to a local filesystem is a bit unfair, isn't it? You "think" reading happens via local disk, but actually reading happens across the network when you use nfs-mount. Even when you use the fuse-mount locally, you still have a userspace-component (which has only a very little influence). And while reading of big contents all happens locally and is thus fast, accessing files (small or big, read or write) has an overhead as the attributes of the remote bricks are checked too to see if the volume is consistent and trigger a self-heal if needed. So while reading big files from local fuse-mount is fast, reading many small files still has the network-penalty of any distributed and/or replicated system. A better comparison is kernel-nfs against gluster with its userspace-nfs. And maybe also compare gluster to moosefs or ceph or lustre or any fs above drbd... Have fun, Arnold -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120509/1d26a628/attachment.sig>