Ravishankar N
2016-Feb-11 00:42 UTC
[Gluster-users] [ovirt-users] ovirt glusterfs performance
+gluster-users Does disabling 'performance.write-behind' give a better throughput? On 02/10/2016 11:06 PM, Bill James wrote:> I'm setting up a ovirt cluster using glusterfs and noticing not > stellar performance. > Maybe my setup could use some adjustments? > > 3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt 3.6.2.6-1. > Each node has 8 spindles configured in 1 array which is split using > LVM with one logical volume for system and one for gluster. > They each have 4 NICs, > NIC1 = ovirtmgmt > NIC2 = gluster > NIC3 = VM traffic > > I tried with default glusterfs settings and also with: > performance.cache-size: 1GB > performance.readdir-ahead: on > performance.write-behind-window-size: 4MB > > [root at ovirt3 test scripts]# gluster volume info gv1 > > Volume Name: gv1 > Type: Replicate > Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2 > Status: Started > Number of Bricks: 1 x 3 = 3 > Transport-type: tcp > Bricks: > Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1 > Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1 > Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1 > Options Reconfigured: > performance.cache-size: 1GB > performance.readdir-ahead: on > performance.write-behind-window-size: 4MB > > > Using simple dd test on VM in ovirt: > dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct > 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s > > Another VM not in ovirt using nfs: > dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct > 1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s > > > Is that expected or is there a better way to set it up to get better > performance? > > Thanks. > > > Cloud Services for Business www.j2.com > j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox > > > This email, its contents and attachments contain information from j2 > Global, Inc. and/or its affiliates which may be privileged, > confidential or otherwise protected from disclosure. The information > is intended to be for the addressee(s) only. If you are not an > addressee, any disclosure, copy, distribution, or use of the contents > of this message is prohibited. If you have received this email in > error please notify the sender by reply e-mail and delete the original > message and any copies. (c) 2015 j2 Global, Inc. All rights reserved. > eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are > registered trademarks of j2 Global, Inc. and its affiliates. > _______________________________________________ > Users mailing list > Users at ovirt.org > http://lists.ovirt.org/mailman/listinfo/users
Nir Soffer
2016-Feb-11 07:55 UTC
[Gluster-users] [ovirt-users] ovirt glusterfs performance
On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N <ravishankar at redhat.com> wrote:> +gluster-users > > Does disabling 'performance.write-behind' give a better throughput? > > > > On 02/10/2016 11:06 PM, Bill James wrote: >> >> I'm setting up a ovirt cluster using glusterfs and noticing not stellar >> performance. >> Maybe my setup could use some adjustments? >> >> 3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt 3.6.2.6-1. >> Each node has 8 spindles configured in 1 array which is split using LVM >> with one logical volume for system and one for gluster. >> They each have 4 NICs, >> NIC1 = ovirtmgmt >> NIC2 = glusterHow do you ensure that gluster trafic is using this nic?>> NIC3 = VM trafficHow do you ensure that vm trafic is using this nic?>> I tried with default glusterfs settingsAnd did you find any difference?>> and also with: >> performance.cache-size: 1GB >> performance.readdir-ahead: on >> performance.write-behind-window-size: 4MB >> >> [root at ovirt3 test scripts]# gluster volume info gv1 >> >> Volume Name: gv1 >> Type: Replicate >> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2 >> Status: Started >> Number of Bricks: 1 x 3 = 3 >> Transport-type: tcp >> Bricks: >> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1 >> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1 >> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1 >> Options Reconfigured: >> performance.cache-size: 1GB >> performance.readdir-ahead: on >> performance.write-behind-window-size: 4MB >> >> >> Using simple dd test on VM in ovirt: >> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=directblock size of 1G?! Try 1M (our default for storage operations)>> 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s >> >> Another VM not in ovirt using nfs: >> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct >> 1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s >> >> >> Is that expected or is there a better way to set it up to get better >> performance?Adding Niels for advice.>> This email, its contents and attachments contain information from j2 >> Global, Inc. and/or its affiliates which may be privileged, confidential or >> otherwise protected from disclosure. The information is intended to be for >> the addressee(s) only. If you are not an addressee, any disclosure, copy, >> distribution, or use of the contents of this message is prohibited. If you >> have received this email in error please notify the sender by reply e-mail >> and delete the original message and any copies. (c) 2015 j2 Global, Inc. All >> rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox >> are registered trademarks of j2 Global, Inc. and its affiliates.Please avoid this, this is a public mailing list, everything you write here is public. Nir