I just noticed i left the most important parameters out :) here's the write command with filesize and recordsize in it as well :) ./iozone -i 0 -t 1 -F /mnt/gluster/storage/thread1 -+n -c -C -e -I -w -+S 0 -s 200G -r 16384k also i ran the benchmark without direct_io which resulted in an even worse performance. i also tried to mount the gluster volume via nfs-ganesha which further reduced throughput down to about 450MB/s if i run the iozone benchmark with 3 threads writing to all three bricks directly (from the xfs filesystem) i get throughputs of around 6GB/s .. if I run the same benchmark through gluster mounted locally using the fuse client and with enough threads so that each brick gets at least one file written to it, i end up seing throughputs around 1.5GB/s .. that's a 4x decrease in performance. at it actually is the same if i run the benchmark with less threads and files only get written to two out of three bricks. cpu load on the server is around 25% by the way, nicely distributed across all available cores. i can't believe that gluster should really be so slow and everybody is just happily using it. any hints on what i'm doing wrong are very welcome. i'm using gluster 6.0 by the way. regards Pascal On 03.04.19 12:28, Pascal Suter wrote:> Hi all > > I am currently testing gluster on a single server. I have three > bricks, each a hardware RAID6 volume with thin provisioned LVM that > was aligned to the RAID and then formatted with xfs. > > i've created a distributed volume so that entire files get distributed > across my three bricks. > > first I ran a iozone benchmark across each brick testing the read and > write perofrmance of a single large file per brick > > i then mounted my gluster volume locally and ran another iozone run > with the same parameters writing a single file. the file went to brick > 1 which, when used driectly, would write with 2.3GB/s and read with > 1.5GB/s. however, through gluster i got only 800MB/s read and 750MB/s > write throughput > > another run with two processes each writing a file, where one file > went to the first brick and the other file to the second brick (which > by itself when directly accessed wrote at 2.8GB/s and read at 2.7GB/s) > resulted in 1.2GB/s of aggregated write and also aggregated read > throughput. > > Is this a normal performance i can expect out of a glusterfs or is it > worth tuning in order to really get closer to the actual brick > filesystem performance? > > here are the iozone commands i use for writing and reading.. note that > i am using directIO in order to make sure i don't get fooled by cache :) > > ./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 > -s $filesize -r $recordsize > iozone-brick${b}-write.txt > > ./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 > -s $filesize -r $recordsize > iozone-brick${b}-read.txt > > cheers > > Pascal > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users
i continued my testing with 5 clients, all attached over 100Gbit/s omni-path via IP over IB. when i run the same iozone benchmark across all 5 clients where gluster is mounted using the glusterfs client, i get an aggretated write throughput of only about 400GB/s and an aggregated read throughput of 1.5GB/s. Each node was writing a single 200Gb file in 16MB chunks and the files where distributed across all three bricks on the server. the connection was established over Omnipath for sure, as there is no other link between the nodes and server. i have no clue what i'm doing wrong here. i can't believe that this is a normal performance people would expect to see from gluster. i guess nobody would be using it if it was this slow. again, when written dreictly to the xfs filesystem on the bricks, i get over 6GB/s read and write throughput using the same benchmark. any advise is appreciated cheers Pascal On 04.04.19 12:03, Pascal Suter wrote:> I just noticed i left the most important parameters out :) > > here's the write command with filesize and recordsize in it as well :) > > ./iozone -i 0 -t 1 -F /mnt/gluster/storage/thread1 -+n -c -C -e -I -w > -+S 0 -s 200G -r 16384k > > also i ran the benchmark without direct_io which resulted in an even > worse performance. > > i also tried to mount the gluster volume via nfs-ganesha which further > reduced throughput down to about 450MB/s > > if i run the iozone benchmark with 3 threads writing to all three > bricks directly (from the xfs filesystem) i get throughputs of around > 6GB/s .. if I run the same benchmark through gluster mounted locally > using the fuse client and with enough threads so that each brick gets > at least one file written to it, i end up seing throughputs around > 1.5GB/s .. that's a 4x decrease in performance. at it actually is the > same if i run the benchmark with less threads and files only get > written to two out of three bricks. > > cpu load on the server is around 25% by the way, nicely distributed > across all available cores. > > i can't believe that gluster should really be so slow and everybody is > just happily using it. any hints on what i'm doing wrong are very > welcome. > > i'm using gluster 6.0 by the way. > > regards > > Pascal > > On 03.04.19 12:28, Pascal Suter wrote: >> Hi all >> >> I am currently testing gluster on a single server. I have three >> bricks, each a hardware RAID6 volume with thin provisioned LVM that >> was aligned to the RAID and then formatted with xfs. >> >> i've created a distributed volume so that entire files get >> distributed across my three bricks. >> >> first I ran a iozone benchmark across each brick testing the read and >> write perofrmance of a single large file per brick >> >> i then mounted my gluster volume locally and ran another iozone run >> with the same parameters writing a single file. the file went to >> brick 1 which, when used driectly, would write with 2.3GB/s and read >> with 1.5GB/s. however, through gluster i got only 800MB/s read and >> 750MB/s write throughput >> >> another run with two processes each writing a file, where one file >> went to the first brick and the other file to the second brick (which >> by itself when directly accessed wrote at 2.8GB/s and read at >> 2.7GB/s) resulted in 1.2GB/s of aggregated write and also aggregated >> read throughput. >> >> Is this a normal performance i can expect out of a glusterfs or is it >> worth tuning in order to really get closer to the actual brick >> filesystem performance? >> >> here are the iozone commands i use for writing and reading.. note >> that i am using directIO in order to make sure i don't get fooled by >> cache :) >> >> ./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 >> -s $filesize -r $recordsize > iozone-brick${b}-write.txt >> >> ./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 >> -s $filesize -r $recordsize > iozone-brick${b}-read.txt >> >> cheers >> >> Pascal >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users
Amar Tumballi Suryanarayan
2019-May-01 12:55 UTC
[Gluster-users] performance - what can I expect
Hi Pascal, Sorry for complete delay in this one. And thanks for testing out in different scenarios. Few questions before others can have a look and advice you. 1. What is the volume info output ? 2. Do you see any concerning logs in glusterfs log files? 3. Please use `gluster volume profile` while running the tests, and that gives a lot of information. 4. Considering you are using glusterfs-6.0, please take statedump of client process (on any node) before and after the test, so we can analyze the latency information of each translators. With these information, I hope we will be in a better state to answer the questions. On Wed, Apr 10, 2019 at 3:45 PM Pascal Suter <pascal.suter at dalco.ch> wrote:> i continued my testing with 5 clients, all attached over 100Gbit/s > omni-path via IP over IB. when i run the same iozone benchmark across > all 5 clients where gluster is mounted using the glusterfs client, i get > an aggretated write throughput of only about 400GB/s and an aggregated > read throughput of 1.5GB/s. Each node was writing a single 200Gb file in > 16MB chunks and the files where distributed across all three bricks on > the server. > > the connection was established over Omnipath for sure, as there is no > other link between the nodes and server. > > i have no clue what i'm doing wrong here. i can't believe that this is a > normal performance people would expect to see from gluster. i guess > nobody would be using it if it was this slow. > > again, when written dreictly to the xfs filesystem on the bricks, i get > over 6GB/s read and write throughput using the same benchmark. > > any advise is appreciated > > cheers > > Pascal > > On 04.04.19 12:03, Pascal Suter wrote: > > I just noticed i left the most important parameters out :) > > > > here's the write command with filesize and recordsize in it as well :) > > > > ./iozone -i 0 -t 1 -F /mnt/gluster/storage/thread1 -+n -c -C -e -I -w > > -+S 0 -s 200G -r 16384k > > > > also i ran the benchmark without direct_io which resulted in an even > > worse performance. > > > > i also tried to mount the gluster volume via nfs-ganesha which further > > reduced throughput down to about 450MB/s > > > > if i run the iozone benchmark with 3 threads writing to all three > > bricks directly (from the xfs filesystem) i get throughputs of around > > 6GB/s .. if I run the same benchmark through gluster mounted locally > > using the fuse client and with enough threads so that each brick gets > > at least one file written to it, i end up seing throughputs around > > 1.5GB/s .. that's a 4x decrease in performance. at it actually is the > > same if i run the benchmark with less threads and files only get > > written to two out of three bricks. > > > > cpu load on the server is around 25% by the way, nicely distributed > > across all available cores. > > > > i can't believe that gluster should really be so slow and everybody is > > just happily using it. any hints on what i'm doing wrong are very > > welcome. > > > > i'm using gluster 6.0 by the way. > > > > regards > > > > Pascal > > > > On 03.04.19 12:28, Pascal Suter wrote: > >> Hi all > >> > >> I am currently testing gluster on a single server. I have three > >> bricks, each a hardware RAID6 volume with thin provisioned LVM that > >> was aligned to the RAID and then formatted with xfs. > >> > >> i've created a distributed volume so that entire files get > >> distributed across my three bricks. > >> > >> first I ran a iozone benchmark across each brick testing the read and > >> write perofrmance of a single large file per brick > >> > >> i then mounted my gluster volume locally and ran another iozone run > >> with the same parameters writing a single file. the file went to > >> brick 1 which, when used driectly, would write with 2.3GB/s and read > >> with 1.5GB/s. however, through gluster i got only 800MB/s read and > >> 750MB/s write throughput > >> > >> another run with two processes each writing a file, where one file > >> went to the first brick and the other file to the second brick (which > >> by itself when directly accessed wrote at 2.8GB/s and read at > >> 2.7GB/s) resulted in 1.2GB/s of aggregated write and also aggregated > >> read throughput. > >> > >> Is this a normal performance i can expect out of a glusterfs or is it > >> worth tuning in order to really get closer to the actual brick > >> filesystem performance? > >> > >> here are the iozone commands i use for writing and reading.. note > >> that i am using directIO in order to make sure i don't get fooled by > >> cache :) > >> > >> ./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 > >> -s $filesize -r $recordsize > iozone-brick${b}-write.txt > >> > >> ./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 > >> -s $filesize -r $recordsize > iozone-brick${b}-read.txt > >> > >> cheers > >> > >> Pascal > >> > >> _______________________________________________ > >> Gluster-users mailing list > >> Gluster-users at gluster.org > >> https://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > >-- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190501/19c6cf32/attachment.html>