Gluster users - Apr 2019 - performance - what can I expect

Hi all

I am currently testing gluster on a single server. I have three bricks, 
each a hardware RAID6 volume with thin provisioned LVM that was aligned 
to the RAID and then formatted with xfs.

i've created a distributed volume so that entire files get distributed 
across my three bricks.

first I ran a iozone benchmark across each brick testing the read and 
write perofrmance of a single large file per brick

i then mounted my gluster volume locally and ran another iozone run with 
the same parameters writing a single file. the file went to brick 1 
which, when used driectly, would write with 2.3GB/s and read with 
1.5GB/s. however, through gluster i got only 800MB/s read and 750MB/s 
write throughput

another run with two processes each writing a file, where one file went 
to the first brick and the other file to the second brick (which by 
itself when directly accessed wrote at 2.8GB/s and read at 2.7GB/s) 
resulted in 1.2GB/s of aggregated write and also aggregated read 
throughput.

Is this a normal performance i can expect out of a glusterfs or is it 
worth tuning in order to really get closer to the actual brick 
filesystem performance?

here are the iozone commands i use for writing and reading.. note that i 
am using directIO in order to make sure i don't get fooled by cache :)

./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 -s 
$filesize -r $recordsize > iozone-brick${b}-write.txt

./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 -s 
$filesize -r $recordsize > iozone-brick${b}-read.txt

cheers

Pascal

I just noticed i left the most important parameters out :)

here's the write command with filesize and recordsize in it as well :)

./iozone -i 0 -t 1 -F /mnt/gluster/storage/thread1 -+n -c -C -e -I -w 
-+S 0 -s 200G -r 16384k

also i ran the benchmark without direct_io which resulted in an even 
worse performance.

i also tried to mount the gluster volume via nfs-ganesha which further 
reduced throughput down to about 450MB/s

if i run the iozone benchmark with 3 threads writing to all three bricks 
directly (from the xfs filesystem) i get throughputs of around 6GB/s .. 
if I run the same benchmark through gluster mounted locally using the 
fuse client and with enough threads so that each brick gets at least one 
file written to it, i end up seing throughputs around 1.5GB/s .. that's 
a 4x decrease in performance. at it actually is the same if i run the 
benchmark with less threads and files only get written to two out of 
three bricks.

cpu load on the server is around 25% by the way, nicely distributed 
across all available cores.

i can't believe that gluster should really be so slow and everybody is 
just happily using it. any hints on what i'm doing wrong are very welcome.

i'm using gluster 6.0 by the way.

regards

Pascal

On 03.04.19 12:28, Pascal Suter wrote:
> Hi all
>
> I am currently testing gluster on a single server. I have three 
> bricks, each a hardware RAID6 volume with thin provisioned LVM that 
> was aligned to the RAID and then formatted with xfs.
>
> i've created a distributed volume so that entire files get distributed 
> across my three bricks.
>
> first I ran a iozone benchmark across each brick testing the read and 
> write perofrmance of a single large file per brick
>
> i then mounted my gluster volume locally and ran another iozone run 
> with the same parameters writing a single file. the file went to brick 
> 1 which, when used driectly, would write with 2.3GB/s and read with 
> 1.5GB/s. however, through gluster i got only 800MB/s read and 750MB/s 
> write throughput
>
> another run with two processes each writing a file, where one file 
> went to the first brick and the other file to the second brick (which 
> by itself when directly accessed wrote at 2.8GB/s and read at 2.7GB/s) 
> resulted in 1.2GB/s of aggregated write and also aggregated read 
> throughput.
>
> Is this a normal performance i can expect out of a glusterfs or is it 
> worth tuning in order to really get closer to the actual brick 
> filesystem performance?
>
> here are the iozone commands i use for writing and reading.. note that 
> i am using directIO in order to make sure i don't get fooled by cache
:)
>
> ./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 
> -s $filesize -r $recordsize > iozone-brick${b}-write.txt
>
> ./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 
> -s $filesize -r $recordsize > iozone-brick${b}-read.txt
>
> cheers
>
> Pascal
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users