Van Renterghem Stijn
2016-Feb-17 15:23 UTC
[Gluster-users] FW: Performance with Gluster+Fuse is 60x slower then Gluster+NFS ?
Hi,
I have setup a server with a new installation of Gluster.
The volume type is 'Replicate'.
1)
I mounted the volume with Fuse
IP1:/app /srv/data glusterfs
defaults,_netdev,backupvolfile-server=IP2,fetch-attempts=2 0 0
When I start my application, it takes 2h until the application is started
Below you can see the stats after the application is started. I can see a very
high LOOKUP value.
Can you explain this high value ? The volume type is replicate, so I should
think, I shouldn't have LOOKUPs ?
Interval2
Block Size: 1b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 342 25 575
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 143 898 118
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1 4 11
No. of Writes: 82 0 0
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 11 31 39
No. of Writes: 0 0 0
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 59 148 555
No. of Writes: 0 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 FORGET
0.00 0.00 us 0.00 us 0.00 us 201 RELEASE
0.00 0.00 us 0.00 us 0.00 us 54549 RELEASEDIR
0.00 47.00 us 47.00 us 47.00 us 1 REMOVEXATTR
0.00 94.00 us 74.00 us 114.00 us 2 XATTROP
0.00 191.00 us 191.00 us 191.00 us 1 TRUNCATE
0.00 53.50 us 35.00 us 74.00 us 4 STATFS
0.00 79.67 us 70.00 us 91.00 us 3 RENAME
0.00 37.33 us 27.00 us 68.00 us 15 INODELK
0.00 190.67 us 116.00 us 252.00 us 3 UNLINK
0.00 28.83 us 8.00 us 99.00 us 30 ENTRYLK
0.00 146.33 us 117.00 us 188.00 us 6 CREATE
0.00 37.63 us 12.00 us 73.00 us 84 READDIR
0.00 23.75 us 8.00 us 75.00 us 198 FLUSH
0.00 65.33 us 42.00 us 141.00 us 204 OPEN
0.01 45.78 us 11.00 us 191.00 us 944 FINODELK
0.01 80.34 us 31.00 us 211.00 us 859 READ
0.02 96.74 us 50.00 us 188.00 us 944 FXATTROP
0.02 55.84 us 24.00 us 140.00 us 1707 FSTAT
0.02 52.89 us 21.00 us 175.00 us 2183 WRITE
0.02 59.69 us 11.00 us 235.00 us 2312 GETXATTR
0.03 51.18 us 8.00 us 142.00 us 3091 STAT
0.46 48.66 us 1.00 us 179.00 us 54549 OPENDIR
1.13 135.93 us 18.00 us 16362.00 us 48124 READDIRP
98.29 70.46 us 16.00 us 2903.00 us 8104385 LOOKUP
Duration: 7560 seconds
Data Read: 91208567 bytes = 91MB
Data Written: 292007 bytes = 0,292MB
2)
I have tried some tuning options, but that didn't changed anything:
#gluster volume info app
Volume Name: app
Type: Replicate
Volume ID: f1b59aec-adf8-41f8-ad95-839ace247041
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: IP1:/exports/app/app
Brick2: IP2:/exports/app/app
Options Reconfigured:
cluster.readdir-optimize: on
server.event-threads: 8
client.event-threads: 8
cluster.lookup-optimize: on
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
auth.allow: client1,client2
nfs.rpc-auth-allow: client1,client2
nfs.export-volumes: on
nfs.addr-namelookup: off
nfs.disable: off
performance.readdir-ahead: on
performance.io-thread-count: 64
3)
I then have enabled NFS support.
I stopped the application and unmounted the volume. I then mounted it again with
nfs:
IP1:/app /srv/data nfs
rsize=4096,wsize=4096,hard,intr 0 0
I started the application again and it was running within 3minutes.
The stats with NFS where very different then with Fuse. It seems that they are
almost not logged.
Interval 11 Stats:
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 0 0
No. of Writes: 9 1 1
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 0 0 0
No. of Writes: 1 5 8
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1 RELEASEDIR
0.02 2.00 us 2.00 us 2.00 us 1 OPENDIR
0.57 34.00 us 19.00 us 49.00 us 2 READDIR
0.81 96.00 us 96.00 us 96.00 us 1 SETATTR
1.06 62.50 us 61.00 us 64.00 us 2 OPEN
1.39 164.00 us 164.00 us 164.00 us 1 TRUNCATE
1.39 41.25 us 30.00 us 52.00 us 4 GETXATTR
1.54 91.00 us 86.00 us 96.00 us 2 XATTROP
2.72 80.50 us 29.00 us 122.00 us 4 LOOKUP
2.81 33.30 us 17.00 us 56.00 us 10 INODELK
10.36 76.69 us 26.00 us 133.00 us 16 FLUSH
15.83 75.00 us 61.00 us 105.00 us 25 WRITE
17.22 48.55 us 13.00 us 78.00 us 42 FINODELK
44.28 124.83 us 62.00 us 161.00 us 42 FXATTROP
Duration: 580 seconds
Data Read: 0 bytes
Data Written: 60839 bytes
What is wrong with the Fuse client ?
Why does my application start in 120min with Gluster+Fuse and in 3min with
Gluster+NFS ?
Regards,
Stijn
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160217/9066b422/attachment.html>
Stefan Jakobs
2016-Feb-17 22:56 UTC
[Gluster-users] FW: Performance with Gluster+Fuse is 60x slower then Gluster+NFS ?
Van Renterghem Stijn:> Interval2 > Block Size: 1b+ 16b+ 32b+ > No. of Reads: 0 0 > 0 No. of Writes: 342 25 > 575 > > Block Size: 64b+ 128b+ > 256b+ No. of Reads: 0 0 > 0 No. of Writes: 143 898 > 118 > > Block Size: 512b+ 1024b+ > 2048b+ No. of Reads: 1 4 > 11 No. of Writes: 82 0 > 0 > > Block Size: 4096b+ 8192b+ > 16384b+ No. of Reads: 11 31 > 39 No. of Writes: 0 0 > 0 > > Block Size: 32768b+ 65536b+ > 131072b+ No. of Reads: 59 148 > 555 No. of Writes: 0 0 > 0 > > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop --------- ----------- ----------- ----------- ------------ > ---- 0.00 0.00 us 0.00 us 0.00 us 1 > FORGET 0.00 0.00 us 0.00 us 0.00 us 201 > RELEASE 0.00 0.00 us 0.00 us 0.00 us 54549 > RELEASEDIR 0.00 47.00 us 47.00 us 47.00 us 1 > REMOVEXATTR 0.00 94.00 us 74.00 us 114.00 us 2 > XATTROP 0.00 191.00 us 191.00 us 191.00 us 1 > TRUNCATE 0.00 53.50 us 35.00 us 74.00 us 4 > STATFS 0.00 79.67 us 70.00 us 91.00 us 3 > RENAME 0.00 37.33 us 27.00 us 68.00 us 15 > INODELK 0.00 190.67 us 116.00 us 252.00 us 3 > UNLINK 0.00 28.83 us 8.00 us 99.00 us 30 > ENTRYLK 0.00 146.33 us 117.00 us 188.00 us 6 > CREATE 0.00 37.63 us 12.00 us 73.00 us 84 > READDIR 0.00 23.75 us 8.00 us 75.00 us 198 > FLUSH 0.00 65.33 us 42.00 us 141.00 us 204 > OPEN 0.01 45.78 us 11.00 us 191.00 us 944 > FINODELK 0.01 80.34 us 31.00 us 211.00 us 859 > READ 0.02 96.74 us 50.00 us 188.00 us 944 > FXATTROP 0.02 55.84 us 24.00 us 140.00 us 1707 > FSTAT 0.02 52.89 us 21.00 us 175.00 us 2183 > WRITE 0.02 59.69 us 11.00 us 235.00 us 2312 > GETXATTR 0.03 51.18 us 8.00 us 142.00 us 3091 > STAT 0.46 48.66 us 1.00 us 179.00 us 54549 > OPENDIR 1.13 135.93 us 18.00 us 16362.00 us 48124 > READDIRP 98.29 70.46 us 16.00 us 2903.00 us 8104385 > LOOKUP > > Duration: 7560 seconds > Data Read: 91208567 bytes = 91MB > Data Written: 292007 bytes = 0,292MBHow did you collect these statistics? Thanks Stefan