I think I should provide some additional info
To be more explicit the volumes are replicated volume created with the
command
gluster volume create $VOL replica 2
dc1strg001x:/zfspool/glusterfs/$VOL/data
dc1strg002x:/zfspool/glusterfs/$VOL/data
I also decided to use "real" file for the testing and came up with
some
different results
linux-lts-raring.tar is just a tarfile of a whole bunch of binaries.
When I control the blocksize and use large ones I very much close the
performance gap with NFS.
When I do not control block sizes (rsync) I take ~%50 performance hit.
Someone told me that when I use the Gluster FUSE client against a
replicated volume that I am actually writing the data twice - once to
each brick - which would very much make sense that writes to NFS are
faster since data would be written only to one server and then they
would replicated between each other.
Does anyone have any overall suggestions about using the GlusterFS
client as a general purpose network store vs the NFS client?
My feeling right now is I am just going to have to try it with real
world load and see if the write performance loss is acceptable
Thanks!
root at vc1test001 /tmp 570# dd if=linux-lts-raring.tar
of=/mnt/backups_nfs/linux-lts-raring.tar bs=64M count=256
42+1 records in
42+1 records out
2851440640 bytes (2.9 GB) copied, 54.6371 s, 52.2 MB/s
root at vc1test001 /tmp 571# dd if=linux-lts-raring.tar
of=/mnt/backups_gluster/linux-lts-raring.tar bs=64M count=256
42+1 records in
42+1 records out
2851440640 bytes (2.9 GB) copied, 61.8533 s, 46.1 MB/s
root at vc1test001 /tmp 564# rsync -av --progress linux-lts-raring.tar
/mnt/backups_nfs/
sending incremental file list
linux-lts-raring.tar
2,851,440,640 100% 43.63MB/s 0:01:02 (xfr#1, to-chk=0/1)
sent 2,852,136,896 bytes received 35 bytes 44,219,177.22 bytes/sec
total size is 2,851,440,640 speedup is 1.00
root at vc1test001 /tmp 565# rsync -av --progress linux-lts-raring.tar
/mnt/backups_gluster/
sending incremental file list
linux-lts-raring.tar
2,851,440,640 100% 22.33MB/s 0:02:01 (xfr#1, to-chk=0/1)
sent 2,852,136,896 bytes received 35 bytes 23,282,750.46 bytes/sec
total size is 2,851,440,640 speedup is 1.00
On 2/26/16 9:45 AM, Mark Selby wrote:> Both the client and the server are running Ubuntu 14.04 with GlusterFS
> 3.7 from Ubuntu PPA
>
> I am going to use Gluster to create a simple replicated NFS server. I
> was hoping to use the Native FUSE client to also get seamless fail
> over but am running into performance issue that are going to prevent
> me from doing so.
>
> I have replicated Gluster volume on a 24 core server with 128GB RAM,
> 10GBe networking and Raid-10 served via ZFS.
>
> From a remote client I mount the same volume via NFS and the native
> client.
>
> I did some really basic performance tests just to get a feel for what
> penalty the user space client would incur.
>
> I must admit I was shocked at how "poor" the Gluster FUSE client
> performed. I know that small block sizes are not Glusters favorite but
> even at larger ones the penalty is pretty great.
>
> Is this to be expected or is there some configuration that I am missing?
>
> If providing any more info would be helpful - please let me know.
>
> Thanks!
>
> root at vc1test001 /root 489# mount -t nfs
> dc1strg001x:/zfspool/glusterfs/backups /mnt/backups_nfs
> root at vc1test001 /root 490# mount -t glusterfs dc1strg001x:backups
> /mnt/backups_gluster
>
> root at vc1test001 /mnt/backups_nfs 492# dd if=/dev/zero of=testfile
> bs=16k count=16384
> 16384+0 records in
> 16384+0 records out
> 268435456 bytes (268 MB) copied, 2.6763 s, 100 MB/s
>
> root at vc1test001 /mnt/backups_nfs 510# dd if=/dev/zero of=testfile1
> bs=64k count=16384
> 16384+0 records in
> 16384+0 records out
> 1073741824 bytes (1.1 GB) copied, 10.7434 s, 99.9 MB/s
>
> root at vc1test001 /mnt/backups_nfs 517# dd if=/dev/zero of=testfile1
> bs=128k count=16384
> 16384+0 records in
> 16384+0 records out
> 2147483648 bytes (2.1 GB) copied, 19.0354 s, 113 MB/s
>
> root at vc1test001 /mnt/backups_gluster 495# dd if=/dev/zero of=testfile
> bs=16k count=16384
> 16384+0 records in
> 16384+0 records out
> 268435456 bytes (268 MB) copied, 102.058 s, 2.6 MB/s
>
> root at vc1test001 /mnt/backups_gluster 513# dd if=/dev/zero of=testfile1
> bs=64k count=16384
> 16384+0 records in
> 16384+0 records out
> 1073741824 bytes (1.1 GB) copied, 114.053 s, 9.4 MB/s
>
> root at vc1test001 /mnt/backups_gluster 514# dd if=/dev/zero of=testfile1
> bs=128k count=16384
> 16384+0 records in
> 16384+0 records out
> 2147483648 bytes (2.1 GB) copied, 123.904 s, 17.3 MB/s
>
> root at vc1test001 /tmp 504# rsync -av --progress testfile1
> /mnt/backups_nfs/
> sending incremental file list
> testfile1
> 1,073,741,824 100% 89.49MB/s 0:00:11 (xfr#1, to-chk=0/1)
>
> sent 1,074,004,057 bytes received 35 bytes 74,069,247.72 bytes/sec
> total size is 1,073,741,824 speedup is 1.00
>
> root at vc1test001 /tmp 505# rsync -av --progress testfile1
> /mnt/backups_gluster/
> sending incremental file list
> testfile1
> 1,073,741,824 100% 25.94MB/s 0:00:39 (xfr#1, to-chk=0/1)
>
> sent 1,074,004,057 bytes received 35 bytes 27,189,977.01 bytes/sec
> total size is 1,073,741,824 speedup is 1.00
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users