Thomas Jackson
2011-Sep-09 05:00 UTC
[Gluster-users] Slow performance - 4 hosts, 10 gigabit ethernet, Gluster 3.2.3
Hi everyone, I am seeing slower-than-expected performance in Gluster 3.2.3 between 4 hosts with 10 gigabit eth between them all. Each host has 4x 300GB SAS 15K drives in RAID10, 6-core Xeon E5645 @ 2.40GHz and 24GB RAM running Ubuntu 10.04 64-bit (I have also tested with Scientific Linux 6.1 and Debian Squeeze - same results on those as well). All of the hosts mount the volume using the FUSE module. The base filesystem on all of the nodes is XFS, however tests with ext4 have yielded similar results. Command used to create the volume: gluster volume create cluster-volume replica 2 transport tcp node01:/mnt/local-store/ node02:/mnt/local-store/ node03:/mnt/local-store/ node04:/mnt/local-store/ Command used to mount the Gluster volume on each node: mount -t glusterfs localhost:/cluster-volume /mnt/cluster-volume Creating a 40GB file onto a node's local storage (ie no Gluster involvement): dd if=/dev/zero of=/mnt/local-store/test.file bs=1M count=40000 41943040000 bytes (42 GB) copied, 92.9264 s, 451 MB/s Getting the same file off the node's local storage: dd if=/mnt/local-store/test.file of=/dev/null 41943040000 bytes (42 GB) copied, 81.858 s, 512 MB/s 40GB file onto the Gluster storage: dd if=/dev/zero of=/mnt/cluster-volume/test.file bs=1M count=40000 41943040000 bytes (42 GB) copied, 226.934 s, 185 MB/s Getting the same file off the Gluster storage dd if=/mnt/cluster-volume/test.file of=/dev/null 41943040000 bytes (42 GB) copied, 661.561 s, 63.4 MB/s I have also tried using Gluster 3.1, with similar results. According to the Gluster docs, I should be seeing roughly the lesser of the drive speed and the network speed. The network is able to push 0.9GB/sec according to iperf so that definitely isn't a limiting factor here, and each array is able to do 400-500MB/sec as per above benchmarks. I've tried with/without jumbo frames as well, which doesn't make any major difference. The glusterfs process is using 120% CPU according to top, and glusterfsd is sitting at about 90%. Any ideas / tips of where to start for speeding this config up? Thanks, Thomas
Pavan T C
2011-Sep-14 11:19 UTC
[Gluster-users] Slow performance - 4 hosts, 10 gigabit ethernet, Gluster 3.2.3
On Friday 09 September 2011 10:30 AM, Thomas Jackson wrote:> Hi everyone,Hello Thomas, Try the following: 1. In the fuse volume file, try: Under write-behind: "option cache-size 16MB" Under read-ahead: "option page-count 16" Under io-cache: "option cache-size=64MB" 2. Did you get 9Gbits/Sec with iperf with a single thread or multiple threads? 3. Can you give me the output of: sysctl -a | egrep 'rmem|wmem' 4. If it is not a problem for you, can you please create a pure distribute setup (instead of distributed-replicate) and then report the numbers? 5. What is the inode size with which you formatted you XFS filesystem ? This last point might not be related to your throughput problem, but if you are planning to use this setup for a large number of files, you might be better off using an inode size of 512 instead of the default 256 bytes. To do that, your mkfs command should be: mkfs -t xfs -i size=512 /dev/<disk device> Pavan> > I am seeing slower-than-expected performance in Gluster 3.2.3 between 4 > hosts with 10 gigabit eth between them all. Each host has 4x 300GB SAS 15K > drives in RAID10, 6-core Xeon E5645 @ 2.40GHz and 24GB RAM running Ubuntu > 10.04 64-bit (I have also tested with Scientific Linux 6.1 and Debian > Squeeze - same results on those as well). All of the hosts mount the volume > using the FUSE module. The base filesystem on all of the nodes is XFS, > however tests with ext4 have yielded similar results. > > Command used to create the volume: > gluster volume create cluster-volume replica 2 transport tcp > node01:/mnt/local-store/ node02:/mnt/local-store/ node03:/mnt/local-store/ > node04:/mnt/local-store/ > > Command used to mount the Gluster volume on each node: > mount -t glusterfs localhost:/cluster-volume /mnt/cluster-volume > > Creating a 40GB file onto a node's local storage (ie no Gluster > involvement): > dd if=/dev/zero of=/mnt/local-store/test.file bs=1M count=40000 > 41943040000 bytes (42 GB) copied, 92.9264 s, 451 MB/s > > Getting the same file off the node's local storage: > dd if=/mnt/local-store/test.file of=/dev/null > 41943040000 bytes (42 GB) copied, 81.858 s, 512 MB/s > > 40GB file onto the Gluster storage: > dd if=/dev/zero of=/mnt/cluster-volume/test.file bs=1M count=40000 > 41943040000 bytes (42 GB) copied, 226.934 s, 185 MB/s > > Getting the same file off the Gluster storage > dd if=/mnt/cluster-volume/test.file of=/dev/null > 41943040000 bytes (42 GB) copied, 661.561 s, 63.4 MB/s > > I have also tried using Gluster 3.1, with similar results. > > According to the Gluster docs, I should be seeing roughly the lesser of the > drive speed and the network speed. The network is able to push 0.9GB/sec > according to iperf so that definitely isn't a limiting factor here, and each > array is able to do 400-500MB/sec as per above benchmarks. I've tried > with/without jumbo frames as well, which doesn't make any major difference. > > The glusterfs process is using 120% CPU according to top, and glusterfsd is > sitting at about 90%. > > Any ideas / tips of where to start for speeding this config up? > > Thanks, > > Thomas > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Possibly Parallel Threads
- [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements
- [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements
- very high traffic without any load
- [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements
- [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements