Kris Laib
2016-Jan-27 16:51 UTC
[Gluster-users] Write speed issues with 16MB files and using Gluster fuse mount vs NFS
Hi all, We were initially planning on using NFS mounts for our gluster deployment to reduce the amount of client-side changes we had to make to swap out our existing NFS solution. I ran into an HA issue with NFS (see my other post from this morning), so started looking into the Fuse client as an alternative. What I found is that I can't seem to get above ~30MB/s with a gluster mount transferring 16MB files, even though a NFS mount form the same client to the same volume reaches ~165MB/s. This seems like a much more drastic difference that I've seen others mention, but I don't have a good feel for how to troubleshoot this. Any advice? Below shows what I'm seeing. nfs03-poc is the client, and gfs-slc02.mgmt and gfs-slc03.mgmt are the gluster nodes serving the data via RRDNS at gfs-slc.mgmt. Please let me know if any other info would be helpful. -- On the client side: [root at nfs03-poc fio]# cat GlusterConfig.fio [global] description=fio load testing...simulating disk activity for writing the WAL file blocksize=8192 filesize=16m fsync=0 iomem=malloc loops=1 numjobs=1 prio=0 runtime=1m rw=write size=200M startdelay=0 time_based [wal01] directory=/mnt/gluster/fio_test ioengine=sync nrfiles=1 [wal02] directory=/mnt/gluster/fio_test ioengine=sync nrfiles=1 [wal03] directory=/mnt/gluster/fio_test ioengine=sync nrfiles=1 .. .. .. [wal18] directory=/mnt/gluster/fio_test ioengine=sync nrfiles=1 [wal19] directory=/mnt/gluster/fio_test ioengine=sync nrfiles=1 [wal20] directory=/mnt/gluster/fio_test ioengine=sync nrfiles=1 [root at nfs03-poc fio]# iperf -c gfs-slc02.mgmt ------------------------------------------------------------ Client connecting to gfs-slc02.mgmt, TCP port 5001 TCP window size: 19.3 KByte (default) ------------------------------------------------------------ [ 3] local 10.124.10.18 port 58877 connected with 10.124.0.91 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 9.96 GBytes 8.56 Gbits/sec [root at nfs03-poc fio]# iperf -c gfs-slc03.mgmt ------------------------------------------------------------ Client connecting to gfs-slc03.mgmt, TCP port 5001 TCP window size: 19.3 KByte (default) ------------------------------------------------------------ [ 3] local 10.124.10.18 port 46043 connected with 10.124.0.88 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 8.69 GBytes 7.47 Gbits/sec [root at nfs03-poc fio]# umount /mnt/gluster/ [root at nfs03-poc fio]# mount -t nfs -o async gfs-slc.mgmt:/dr_opt_nwea /mnt/gluster/ [root at nfs03-poc fio]# mount | grep gluster gfs-slc.mgmt:/dr_opt_nwea on /mnt/gluster type nfs (rw,addr=10.124.0.151) [root at nfs03-poc fio]# fio GlusterConfig.fio -output=Gluster1.txt [root at nfs03-poc fio]# tail -n 1 Gluster1.txt.0% done] [0K/176.6M/0K /s] [0 /22.6K/0 iops] [eta 00m:00s] WRITE: io=9749.1MB, aggrb=166113KB/s, minb=7034KB/s, maxb=9929KB/s, mint=60015msec, maxt=60103msec [root at nfs03-poc fio]# umount /mnt/gluster/ [root at nfs03-poc fio]# mount -t glusterfs -o direct-io-mode=disable gfs-slc.mgmt:/dr_opt_nwea /mnt/gluster [root at nfs03-poc fio]# mount | grep gluster gfs-slc.mgmt:/dr_opt_nwea on /mnt/gluster type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) [root at nfs03-poc fio]# fio GlusterConfig.fio -output=Gluster2.txt [root at nfs03-poc fio]# tail -n 1 Gluster2.txt.0% done] [0K/33088K/0K /s] [0 /4136 /0 iops] [eta 00m:00s] WRITE: io=1756.8MB, aggrb=29975KB/s, minb=1498KB/s, maxb=1499KB/s, mint=60001msec, maxt=60012msec On the gluster server side: [root at gfs-slc02.mgmt ~]$ gluster --version glusterfs 3.7.5 built on Oct 7 2015 16:27:05 [root at gfs-slc02.mgmt ~]$ gluster volume info dr_opt_nwea Volume Name: dr_opt_nwea Type: Replicate Volume ID: 6ecf855e-e7e0-4479-bde9-f32e5e820365 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: gfs-slc02.mgmt:/data/glusterfs/dr_opt_nwea_brick1/brick1 Brick2: gfs-slc03.mgmt:/data/glusterfs/dr_opt_nwea_brick1/brick1 Options Reconfigured: performance.readdir-ahead: on nfs.export-volumes: on nfs.addr-namelookup: Off nfs.disable: off network.ping-timeout: 5 cluster.server-quorum-type: server cluster.server-quorum-ratio: 51% [root at gfs-slc02.mgmt ~]$ dd if=/dev/zero of=/data/glusterfs/dr_opt_nwea_brick1/brick1/ddtest bs=4k count=10000 10000+0 records in 10000+0 records out 40960000 bytes (41 MB) copied, 0.0311661 s, 1.3 GB/s [root at gfs-slc03.mgmt ~]$ dd if=/dev/zero of=/data/glusterfs/dr_opt_nwea_brick1/brick1/ddtest bs=4k count=10000 10000+0 records in 10000+0 records out 40960000 bytes (41 MB) copied, 0.0445559 s, 919 MB/s --- I've tried with the following volume settings as well, which makes little to no difference on the gluster mount test: performance.write-behind-window-size: 1MB performance.io-cache: on performance.client-io-threads: on performance.cache-size: 512MB performance.io-thread-count: 64 Ideas? Thanks, Kris -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160127/3bf5ce43/attachment.html>