Hello, I am testing GlusterFS for the first time and have installed the latest GlusterFS 3.5 stable version on Debian 7 on brand new SuperMicro hardware with ZFS instead of hardware RAID. My ZFS pool is a RAIDZ-2 with 6 SATA disks of 2 TB each. After setting up a first and single test brick on my currently single test node I wanted first to see how much slower will GlusterFS be compared to writting directly to the ZFS volume. For that purpose I have mounted my GlusterFS volume locally on the same server using FUSE. For my tests I have used bonnie++ with the command "bonnie++ -n16 -b" and I must say I am quite shocked to see that with this current setup GlusterFS slows down the whole file system with a factor of approximately 6 to 8. For example: ZFS volume Sequential output by block (read): 936 MB/sec Sequential input by block (write): 1520 MB/sec GlusterFS on top of same ZFS volume mounted with FUSE Sequential output by block (read): 114 MB/sec Sequential input by block (write): 312 MB/sec Now I was wondering if such a performance drop on a single GlusterFS node is expected? If not is it maybe ZFS which is messing up things? bonnie++ took 3 minutes to rune on the ZFS volume and 18 minutes on the GlusterFS mount. I have copied the bonnie++ results below just in case in CVS format: 1.96,1.96,ZFS,1,1422907597,31960M,,170,99,936956,94,484417,74,463,99,1520120,98,815.4,41,16,,,,,3376,26,+++++,+++,3109,22,3261,21,+++++,+++,3305,20,66881us,15214us,84887us,23648us,53641us,93322us,39607us,363us,298ms,136ms,18us,176ms 1.96,1.96,GFS,1,1422897979,31960M,,16,17,114223,20,92610,20,+++++,+++,312557,14,444.5,6,16,,,,,385,3,5724,5,916,4,357,3,2044,4,750,4,550ms,9715us,23094us,3334us,125ms,90070us,154ms,8609us,17570us,67180us,4116us,7879us Maybe they are a few performance tuning trick that I am not aware of? Let me know if I should provide any more information. In advance thanks for your comments. Best regards ML
Pranith Kumar Karampuri
2015-Feb-05 11:17 UTC
[Gluster-users] GlusterFS with FUSE slow vs ZFS volume
+Kiran Patil may know about this. Pranith On 02/03/2015 12:56 AM, ML mail wrote:> Hello, > > I am testing GlusterFS for the first time and have installed the latest GlusterFS 3.5 stable version on Debian 7 on brand new SuperMicro hardware with ZFS instead of hardware RAID. My ZFS pool is a RAIDZ-2 with 6 SATA disks of 2 TB each. > > After setting up a first and single test brick on my currently single test node I wanted first to see how much slower will GlusterFS be compared to writting directly to the ZFS volume. For that purpose I have mounted my GlusterFS volume locally on the same server using FUSE. > > For my tests I have used bonnie++ with the command "bonnie++ -n16 -b" and I must say I am quite shocked to see that with this current setup GlusterFS slows down the whole file system with a factor of approximately 6 to 8. For example: > > ZFS volume > > Sequential output by block (read): 936 MB/sec > Sequential input by block (write): 1520 MB/sec > > > GlusterFS on top of same ZFS volume mounted with FUSE > Sequential output by block (read): 114 MB/sec > Sequential input by block (write): 312 MB/sec > > > Now I was wondering if such a performance drop on a single GlusterFS node is expected? If not is it maybe ZFS which is messing up things? > > bonnie++ took 3 minutes to rune on the ZFS volume and 18 minutes on the GlusterFS mount. I have copied the bonnie++ results below just in case in CVS format: > > 1.96,1.96,ZFS,1,1422907597,31960M,,170,99,936956,94,484417,74,463,99,1520120,98,815.4,41,16,,,,,3376,26,+++++,+++,3109,22,3261,21,+++++,+++,3305,20,66881us,15214us,84887us,23648us,53641us,93322us,39607us,363us,298ms,136ms,18us,176ms > 1.96,1.96,GFS,1,1422897979,31960M,,16,17,114223,20,92610,20,+++++,+++,312557,14,444.5,6,16,,,,,385,3,5724,5,916,4,357,3,2044,4,750,4,550ms,9715us,23094us,3334us,125ms,90070us,154ms,8609us,17570us,67180us,4116us,7879us > > Maybe they are a few performance tuning trick that I am not aware of? > > Let me know if I should provide any more information. In advance thanks for your comments. > > Best regards > ML > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
On 02/02/2015 08:26 PM, ML mail wrote:> Hello, > > I am testing GlusterFS for the first time and have installed the latest GlusterFS 3.5 stable version on Debian 7 on brand new SuperMicro hardware with ZFS instead of hardware RAID. My ZFS pool is a RAIDZ-2 with 6 SATA disks of 2 TB each. > > After setting up a first and single test brick on my currently single test node I wanted first to see how much slower will GlusterFS be compared to writting directly to the ZFS volume. For that purpose I have mounted my GlusterFS volume locally on the same server using FUSE. > > For my tests I have used bonnie++ with the command "bonnie++ -n16 -b" and I must say I am quite shocked to see that with this current setup GlusterFS slows down the whole file system with a factor of approximately 6 to 8. For example: > > ZFS volume > > Sequential output by block (read): 936 MB/sec > Sequential input by block (write): 1520 MB/sec > > > GlusterFS on top of same ZFS volume mounted with FUSE > Sequential output by block (read): 114 MB/sec > Sequential input by block (write): 312 MB/sec > > > Now I was wondering if such a performance drop on a single GlusterFS node is expected? If not is it maybe ZFS which is messing up things? > > bonnie++ took 3 minutes to rune on the ZFS volume and 18 minutes on the GlusterFS mount. I have copied the bonnie++ results below just in case in CVS format: > > 1.96,1.96,ZFS,1,1422907597,31960M,,170,99,936956,94,484417,74,463,99,1520120,98,815.4,41,16,,,,,3376,26,+++++,+++,3109,22,3261,21,+++++,+++,3305,20,66881us,15214us,84887us,23648us,53641us,93322us,39607us,363us,298ms,136ms,18us,176ms > 1.96,1.96,GFS,1,1422897979,31960M,,16,17,114223,20,92610,20,+++++,+++,312557,14,444.5,6,16,,,,,385,3,5724,5,916,4,357,3,2044,4,750,4,550ms,9715us,23094us,3334us,125ms,90070us,154ms,8609us,17570us,67180us,4116us,7879us > > Maybe they are a few performance tuning trick that I am not aware of? > > Let me know if I should provide any more information. In advance thanks for your comments.Is ZFS using SA based extended attributes here? Since GlusterFS makes use of extended attributes for storing state, having SA support in ZFS would be necessary for good performance. Regards, Vijay