Christopher Michaelis
2010-Dec-03 23:54 UTC
[Gluster-users] GlusterFS - Abysmal Speed on Fresh Install
Hi there, I'm new to the GlusterFS project - looks superb and very promising! I'm running into massive performance issues, however. I'm using the stock default configuration that GlusterFS put in place when I created the volume - it seems to reference io-cache, quick-read, etc in the volume files, which makes me think it's already pulling in these optimizations. This is a replicate setup - I've tried with 2, 3, and 4 nodes, and performance remains awful on any of them - network communication seems fine, with average speeds at 9-15MB/sec. # gluster --version glusterfs 3.1.1 built on Nov 29 2010 10:07:45 Repository revision: v3.1.1 With no other activity on the filesystems on any of my nodes: # time ls -al total 10072 drwx--x--x 10 root root 4096 Dec 3 16:50 ./ drwxr-xr-x 3 root root 4096 Dec 3 14:19 ../ -rw-r--r-- 1 root root 10240000 Dec 3 16:49 testfile real 0m1.347s user 0m0.000s sys 0m0.000s # time cp testfile testfile2 real 0m11.254s user 0m0.000s sys 0m0.000s # time diff testfile* real 0m5.792s user 0m0.004s sys 0m0.000s Read speed is marginally faster than write speed, but still horrible - e.g. if Apache is serving content off of a glusterfs mountpoint, it times out 95% of the time before it can read the files. I'm using mount -t glusterfs, with default mount options. Can anyone point me in the right direction to getting things nice and speedy here? I'd appreciate any feedback or help! I can provide any configuration files necessary, or even root login access to the box(es) via private e-mail if you want to poke around (these are just test boxes presently). Thanks, --Chris chris.michaelis at uk2group.com
Jacob Shucart
2010-Dec-06 22:41 UTC
[Gluster-users] GlusterFS - Abysmal Speed on Fresh Install
Chris, Can you tell me more about your environment such as a description of the hardware, network speed and topology, and the "gluster volume info" output? The speed you sent is much lower than I would expect in most environments. Since you're only copying a single file I don't know that caching will really help much. There are cache options you can set if you want to try with them. You can find them on: http://www.gluster.com/community/documentation/index.php/Gluster_3.1:_Sett ing_Volume_Options Specifically, you can set the size of files that get cached(the file you have there might be too big, so try setting a max file size that is big enough). Also, the cache could be expiring so you might want to increase the refresh timeout. -Jacob -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Christopher Michaelis Sent: Friday, December 03, 2010 3:55 PM To: gluster-users at gluster.org Subject: [Gluster-users] GlusterFS - Abysmal Speed on Fresh Install Hi there, I'm new to the GlusterFS project - looks superb and very promising! I'm running into massive performance issues, however. I'm using the stock default configuration that GlusterFS put in place when I created the volume - it seems to reference io-cache, quick-read, etc in the volume files, which makes me think it's already pulling in these optimizations. This is a replicate setup - I've tried with 2, 3, and 4 nodes, and performance remains awful on any of them - network communication seems fine, with average speeds at 9-15MB/sec. # gluster --version glusterfs 3.1.1 built on Nov 29 2010 10:07:45 Repository revision: v3.1.1 With no other activity on the filesystems on any of my nodes: # time ls -al total 10072 drwx--x--x 10 root root 4096 Dec 3 16:50 ./ drwxr-xr-x 3 root root 4096 Dec 3 14:19 ../ -rw-r--r-- 1 root root 10240000 Dec 3 16:49 testfile real 0m1.347s user 0m0.000s sys 0m0.000s # time cp testfile testfile2 real 0m11.254s user 0m0.000s sys 0m0.000s # time diff testfile* real 0m5.792s user 0m0.004s sys 0m0.000s Read speed is marginally faster than write speed, but still horrible - e.g. if Apache is serving content off of a glusterfs mountpoint, it times out 95% of the time before it can read the files. I'm using mount -t glusterfs, with default mount options. Can anyone point me in the right direction to getting things nice and speedy here? I'd appreciate any feedback or help! I can provide any configuration files necessary, or even root login access to the box(es) via private e-mail if you want to poke around (these are just test boxes presently). Thanks, --Chris chris.michaelis at uk2group.com _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users