Xavier Hernandez
2017-Jan-19 12:09 UTC
[Gluster-users] ZFS + GlusterFS raid5 low read performance
Hi Yann, On 19/01/17 09:35, Yann MAUPU wrote:> Hi everyone, > > I am currently working on a project for which I am using: > > * 3 storage nodes connected with Omnipath > * 6 sata 750GB HDD per node (total of 18 disks) > > I created a ZFS raidZ1 on each node (5 disks + 1) and used GlusterFS in > raid5 mode between the 3 nodes. > > Unfortunately, with (very) big files, I experience a quite low > read-performance, compared to write-perf (write=700MB/s while read=320 > MB/s). > > Do you know tuning/optimization parameters which could help me get > *better read-performance* ?You can try to set client.event-threads and server.event-threads to higher values. For example 4 or 8 (default is 2). gluster volume set <volname> client.event-threads 8 gluster volume set <volname> server.event-threads 8 Xavi> > > Here's more information on the configuration: > > _ZFS raidz1 on each node:_ > > /# zpool list////NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT////tank 4,06T 10,5M 4,06T - 0% 0% 1.00x ONLINE -/// > > /# zpool status//// pool: tank//// state: ONLINE//// scan: none requested////config://// NAME STATE READ WRITE CKSUM//// tank ONLINE 0 0 0//// raidz1-0 ONLINE 0 0 0//// sda ONLINE 0 0 0//// sdb ONLINE 0 0 0//// sdc ONLINE 0 0 0//// sdd ONLINE 0 0 0//// sde ONLINE 0 0 0//// sdf ONLINE 0 0 0////errors: No known data errors/// > > The command used to create the volume: > > # zpool create -f tank raidz sda sdb sdc sdd sde sdf > > When running IOR on each node, I get about /write perf = 460 MB/s/ and > /read perf = 430 MB/s/ (writing 1024 GiB with zfer_size=16MiB). > > > _GlusterFS raid5 through TCP (IPoIB) between the 3 nodes:_ > > # gluster volume create raid55 disperse 3 redundancy 1 sm01.opa:/tank/ec_point sm02.opa:/tank/ec_point sm03.opa:/tank/ec_point > > > There is a big difference between the read-performance on each ZFS node > and with Gluster. > > Thanks in advance :) > > Yann > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >
Yann Maupu
2017-Jan-23 09:45 UTC
[Gluster-users] ZFS + GlusterFS raid5 low read performance
Hi Xavier, Thanks a lot for your message, it really improved the performance. I updated both the recordsize to 256K on all nodes, and thread-count to 8 and now the read performance got even better than write. Regards, Yann On 19 January 2017 at 13:09, Xavier Hernandez <xhernandez at datalab.es> wrote:> Hi Yann, > > On 19/01/17 09:35, Yann MAUPU wrote: > >> Hi everyone, >> >> I am currently working on a project for which I am using: >> >> * 3 storage nodes connected with Omnipath >> * 6 sata 750GB HDD per node (total of 18 disks) >> >> I created a ZFS raidZ1 on each node (5 disks + 1) and used GlusterFS in >> raid5 mode between the 3 nodes. >> >> Unfortunately, with (very) big files, I experience a quite low >> read-performance, compared to write-perf (write=700MB/s while read=320 >> MB/s). >> >> Do you know tuning/optimization parameters which could help me get >> *better read-performance* ? >> > > You can try to set client.event-threads and server.event-threads to higher > values. For example 4 or 8 (default is 2). > > gluster volume set <volname> client.event-threads 8 > gluster volume set <volname> server.event-threads 8 > > Xavi > > >> >> Here's more information on the configuration: >> >> _ZFS raidz1 on each node:_ >> >> /# zpool list////NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP >> DEDUP HEALTH ALTROOT////tank 4,06T 10,5M 4,06T - 0% >> 0% 1.00x ONLINE -/// >> >> /# zpool status//// pool: tank//// state: ONLINE//// scan: none >> requested////config://// NAME STATE READ WRITE CKSUM//// >> tank ONLINE 0 0 0//// raidz1-0 ONLINE 0 >> 0 0//// sda ONLINE 0 0 0//// sdb >> ONLINE 0 0 0//// sdc ONLINE 0 0 >> 0//// sdd ONLINE 0 0 0//// sde ONLINE >> 0 0 0//// sdf ONLINE 0 0 0////errors: >> No known data errors/// >> >> The command used to create the volume: >> >> # zpool create -f tank raidz sda sdb sdc sdd sde sdf >> >> When running IOR on each node, I get about /write perf = 460 MB/s/ and >> /read perf = 430 MB/s/ (writing 1024 GiB with zfer_size=16MiB). >> >> >> _GlusterFS raid5 through TCP (IPoIB) between the 3 nodes:_ >> >> # gluster volume create raid55 disperse 3 redundancy 1 >> sm01.opa:/tank/ec_point sm02.opa:/tank/ec_point sm03.opa:/tank/ec_point >> >> >> There is a big difference between the read-performance on each ZFS node >> and with Gluster. >> >> Thanks in advance :) >> >> Yann >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170123/23ceceb3/attachment.html>