On 02/02/2015 08:26 PM, ML mail wrote:> Hello,
>
> I am testing GlusterFS for the first time and have installed the latest
GlusterFS 3.5 stable version on Debian 7 on brand new SuperMicro hardware with
ZFS instead of hardware RAID. My ZFS pool is a RAIDZ-2 with 6 SATA disks of 2 TB
each.
>
> After setting up a first and single test brick on my currently single test
node I wanted first to see how much slower will GlusterFS be compared to
writting directly to the ZFS volume. For that purpose I have mounted my
GlusterFS volume locally on the same server using FUSE.
>
> For my tests I have used bonnie++ with the command "bonnie++ -n16
-b" and I must say I am quite shocked to see that with this current setup
GlusterFS slows down the whole file system with a factor of approximately 6 to
8. For example:
>
> ZFS volume
>
> Sequential output by block (read): 936 MB/sec
> Sequential input by block (write): 1520 MB/sec
>
>
> GlusterFS on top of same ZFS volume mounted with FUSE
> Sequential output by block (read): 114 MB/sec
> Sequential input by block (write): 312 MB/sec
>
>
> Now I was wondering if such a performance drop on a single GlusterFS node
is expected? If not is it maybe ZFS which is messing up things?
>
> bonnie++ took 3 minutes to rune on the ZFS volume and 18 minutes on the
GlusterFS mount. I have copied the bonnie++ results below just in case in CVS
format:
>
>
1.96,1.96,ZFS,1,1422907597,31960M,,170,99,936956,94,484417,74,463,99,1520120,98,815.4,41,16,,,,,3376,26,+++++,+++,3109,22,3261,21,+++++,+++,3305,20,66881us,15214us,84887us,23648us,53641us,93322us,39607us,363us,298ms,136ms,18us,176ms
>
1.96,1.96,GFS,1,1422897979,31960M,,16,17,114223,20,92610,20,+++++,+++,312557,14,444.5,6,16,,,,,385,3,5724,5,916,4,357,3,2044,4,750,4,550ms,9715us,23094us,3334us,125ms,90070us,154ms,8609us,17570us,67180us,4116us,7879us
>
> Maybe they are a few performance tuning trick that I am not aware of?
>
> Let me know if I should provide any more information. In advance thanks for
your comments.
Is ZFS using SA based extended attributes here? Since GlusterFS makes
use of extended attributes for storing state, having SA support in ZFS
would be necessary for good performance.
Regards,
Vijay