Hi, I have setup oVirt hyperconverged with 3xDL325 with 3 NVMe SSDs on each server for each Gluster volumes: engine, vmstore and data in replica 3. I am getting almost 4x performance difference when I am benchmarking with DD on gluster disk: Below output is from one of the gluster host: time dd if=/dev/zero of=/gluster/vmstore/testfile bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 0.546909 s, 1.9 GB/s real 0m0.552s user 0m0.001s sys 0m0.165s Below output is from VM. # time dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 1.89253 s, 554 MB/s real 0m1.896s user 0m0.002s sys 0m0.247s Earlier I was getting around ~270MB in that VM, It improved after setting the engine-config -s LibgfApiSupported=true --cver=4.3 I am using default tuned profile(virtual-host). Backend network is 10GbE, tested with iPerf and it gives above 9.40 Gbits/se bandwidth # gluster v info vmstore Volume Name: vmstore Type: Replicate Volume ID: b1f0c1fb-8b8b-479a-9117-61d45692047a Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: daal-gluster.mbi.nus.edu.sg:/gluster/vmstore/brick1 Brick2: naan-gluster.mbi.nus.edu.sg:/gluster/vmstore/brick1 Brick3: dosa-gluster.mbi.nus.edu.sg:/gluster/vmstore/brick1 Options Reconfigured: server.event-threads: 4 client.event-threads: 4 performance.client-io-threads: off nfs.disable: on transport.address-family: inet performance.quick-read: off performance.read-ahead: off performance.io-cache: on performance.low-prio-threads: 32 network.remote-dio: enable cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off storage.owner-uid: 36 storage.owner-gid: 36 features.shard-block-size: 512MB performance.cache-size: 1GB performance.io-thread-count: 32 performance.write-behind-window-size: 4MB Is there any way I can improve my gluster disk?s performance? Thanks for helping out. Regards, Sandeep ________________________________ Important: This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify us immediately; you should not copy or use it for any purpose, nor disclose its contents to any other person. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190613/24a05a6f/attachment.html>